The Tipping Point
The Analytics industry has made incredible strides since it’s nascent days in the mid-1990’s. We have evolved the architectures, established program management disciplines, improved on our techniques to define requirements, invested in tools to help automate the processes around data management, integration, quality, governance, visualization as well as the generation of predictions and prescription of insights. However, 25 years later the majority of organizations are not getting the economic value they expected and desperately need out of their Analytics Program investments. Time to market is still too long, the underlying data and analytic solutions still have too many issues so they are not operationalized, the expected economic value is still not being realized and everyone is frustrated.
The McKinsey Global Institute’s findings from the 2019 AI Frontier Summit found Augmented Intelligence (AI) techniques have the potential to create between $3.5 trillion and $5.8 trillion USD in measurable value annually across nine business functions in 19 industries. This constitutes about 40 percent of the overall $9.5 trillion to $15.4 trillion USD annual economic impact that could potentially be enabled by all analytical techniques. We need to morph all of that latent potential into realized value!
Most enterprises recognize the need to be data-driven, yet 60% of data projects fail to move past preliminary stages, and 87% of data science projects never make it to production. More surprisingly the number of data-driven companies has actually fallen from 37% to 31% since 2017, despite increased investment, according to Gartner. We’re going the wrong way.
Gartner also predicts digital innovation timelines will double for the enterprise. Through 2021, digital transformation initiatives will take large traditional enterprises, on average, twice as long and cost twice as much as anticipated. Smaller, more agile organizations, by contrast will have an opportunity to be first to market as larger organizations exhibit lackluster immediate benefits.
We are at a tipping point! We need a new operating system for our enterprise analytics program. 3rd Generation Business Intelligence calls for more ubiquitous adoption of Self-Service Analytics capabilities balanced with pragmatic and appropriate governance. We need to consistently demonstrate enterprise analytics program execution competency. Analytics and data-driven cultures, processes, portfolio management disciplines and Data Literacy education all need focused attention.
Let’s acknowledge it, continuous disruption at an increasingly accelerated rate is the new normal. It’s time to capitalize on that disruption through the establishment of a Lean-Agile Product Management mindset for our analytics programs. We need to cultivate continuous innovation, generate measurable economic value and relentlessly improve our portfolio of Analytic and Data Products. Enterprises that double-down on this new operating system will ultimately achieve Business Agility.
Business Agility requires that everyone involved in the definition, delivery, adoption and oversight of Data and Analytic Products—including business and technology leaders, development, IT operations, legal, marketing, finance, support, compliance, security, and others—all use Lean and Agile practices to continually deliver innovative, high-quality, value generating and differentiating Analytic and Data Products faster than the competition.
Developing the seven core competencies of the Scaled Agile Framework (SAFe ®) provides a flexible operating system built upon the time tested Lean, Agile, and SAFe practices. Enterprises that achieve Business Agility can deftly organize and swiftly reorganize their analytics program to sense and respond to market rhythms and external events without completely disrupting the existing enterprise organizational hierarchy.
Organizing the SAFe ® operating system around the analytic program and value streams instead of departments or projects, offers a way for enterprises to focus on Analytic and Data products that create measurable value for its business stakeholders and end-customers, through product / service differentiation, innovation, and growth.
The following infographic published by ScaledAgile represents the seven core competencies of the Scaled Agile Framework SAFe ® which when mastered will elevate the business agility of the enterprise. We have taken these core competencies and applied them to the execution of the enterprise analytic program. The enterprise will experience a continuous delivery of value generating, innovation differentiating portfolio of Analytic and Data products that meet the need! The delivery cycles will be greatly compressed. The quality of those Analytic and Data Products will be dramatically improved. The deployment of the Analytic and Data Products will be executed on a predictable cadence at a sustainable scale. The adoption and utilization of those Analytic and Data Products to their fullest is assured.
Business Agility is the ability to compete and thrive in the digital age by continuously sensing and quickly responding to market rhythms and emerging opportunities with innovative analytic-based business solutions. Business Agility also takes feedback from the organization, in the form of rip-tides or feed-back loops from the delivery teams and internal innovation events as opportunities to innovate. Let’s take a closer look at each of the seven core competencies as applied to the enterprise analytics program.
Team and Technical Agility
Agile development is the cornerstone of Business Agility. The Team and Technical Agility competency describes the critical skills and Lean-Agile principles, practices and enabling tools that high-performing Agile teams and Teams of Agile teams use to create high-quality Analytic and Data Products for their internal business stakeholders and the enterprise customers. As we’ll see with each of the the seven core competencies, Team and Technical Agility consists of three dimensions:
- Agile Teams – High-performing, self-sufficient, cross-functional teams anchor the competency by applying effective Agile principles and practices. These teams are comprised of data professionals (Data Engineers, Data Scientists, Data Analysts, Data Quality Analysts, Enterprise Architects, Production Operations, Security, etc), business professionals (Business Process Owners, Business Analysts, Managers, Directors, Operators, Data Stewards, etc) and where possible and applicable the representative voice-of-the-customer for the end-customers who know their need and will experience their need has been met. The embedded analytic solutions within the products and services delivered to the customer being a contributor to that perceived differentiation and realized satisfaction.
- Team of Agile Teams – Agile teams operate within the context of a SAFe Agile Release Train (ART), a long-lived, team of Agile teams that provides a shared vision and direction and is ultimately responsible for the continuous delivery on the enterprise objectives through Analytic and Data products that optimize business outcomes and meet customer needs.
- Built-in Quality – All Agile teams apply rigorously defined Agile practices and techniques to create a high-quality, well-designed portfolio of Analytic and Data Products that support current and future business needs. Following proven quality assurance principles including Behavior Driven Development (BDD), Test Driven Development (TDD) and highly automated testing of data pipelines, analytic algorithms and calculations will contribute to the relentless pursuit of continuous improvement and quality of those Analytic and Data Products.
Agile Product Delivery
Business Agility for analytic programs demands that enterprises rapidly increase their ability to deliver innovative Analytic and Data Products which foster market differentiation. To ensure the enterprise is continuously creating the right solutions for the right customers at the right time; a balance must be maintained between an internal execution focus to optimize enterprise performance and external customer focus to meet customer needs through analytics-enabled product and service differentiation. Agile Product Delivery is a customer-centric approach to ideating, defining, building, and releasing a continuous flow of value-generating Analytic and Data Products to enterprise stakeholders and end-customers. The three dimensions to Agile Product Delivery for Analytic and Data Product development are:
- Customer Centricity, Design Thinking and Behavioral Stories – Customer centricity puts the customer need at the very core of every analytic product and feature decision. Design Thinking techniques are used to ensure the Analytic and Data Product is desirable, feasible, viable, and sustainable. As new analytic ideas and hypotheses are flushed out through data discovery and analytic prototyping techniques, those that show promise of economic value are documented further with EPICS using behavioral storytelling techniques and placed into the Agile Product Delivery backlog. Where the development of those Analytic and Data Products necessitate technical tools and platforms, ENABLERS are used to document the enabling technology needs thus ensuring the Architectural Runway stays in front of the Analytic and Data Product delivery. The combination of Analytic and Data Product EPICS and ENABLERS are prioritized and loaded into the Product Increment (PI) objectives and delivery backlog to be worked upon during the next scheduled PI release.
- Develop on Cadence; Release on Demand – Developing on cadence helps manage the variability inherent in product development. Unknowns are always being revealed. The cadence surfaces them quickly so the impact on the desired MVP can be assessed and decisions on the features to deploy can be made. Decoupling the release of value to customers from the deployment of value on cadence enables and ensures customers can get what they need when they are ready for it.
- DataOps, Machine Learning Ops (MLOps) and the Continuous Delivery Pipeline – DataOps & MLOps, the analytics domain adaptation of DevOps, and the Continuous Delivery Pipeline creates the foundation that enables enterprises to release Analytic and Data Product value, in whole or in part, at the right time and when it’s needed. DataOps has a primary focus on delivering data pipelines that produce the Data Products which in turn enable the creation and delivery of Analytic Products. This is the Analytic Product demand-driven model for Data Products. Data that is fit for purpose for analysis and the supporting services to easily access the data for analysis. MLOps has a primary focus on delivering additional data pipelines and persisted data that is fit for purpose for Machine Learning (ML) analytic models, as well as the ML model code and execution telemetry metadata to enable ML model efficacy monitoring and alert triggering to refine those ML analytic models.
Enterprise Solution Delivery
Building and evolving large, complex enterprise Analytic and Data Products is a monumental effort. They may be supporting a wide-range of interrelated internal enterprise analytics decision support systems. They may be implemented as embedded services in a wide-range of products, tools or platforms sold to thousands or millions of end-customers.
Many such systems require hundreds or thousands of engineers. They demand sophisticated, rigorous practices for engineering, operations, and support. Moreover, over the decades, ensuring that these analytic systems are still operational, their purpose and mission evolve to continue to differentiate and produce value. That calls for continuous research on market rhythms, incubating new capabilities and techniques, ensuring technology upgrades and security patches are applied seamlessly and proactively, and other enhancements proposed from the internal rip-tides are addressed. As true with ‘living organisms’, natural selection happens. Therefore, the activities above are never really ‘done.’ Instead, they are released earlier and continuously enhanced over time for the effective life-cycle of the portfolio of Analytic or Data Products that make up the large solution ‘organism’.
The Enterprise Solution Delivery competency describes how to apply Lean-Agile principles and practices to the ideation, specification, development, deployment, operation, and evolution of the world’s largest and most sophisticated analytic software applications, networks, and cyber-physical data systems. It consists of three dimensions:
- Lean System and Solution Engineering applies Lean-Agile practices to align and coordinate all the activities necessary to specify, architect, design, implement, test, deploy, evolve, and ultimately decommission these systems.
- Coordinating Trains and Suppliers coordinates and aligns the extended set of value streams to a shared business and technology mission. It uses the coordinated Vision, Backlogs, and Roadmaps with common Program Increments (PI) and synchronization points. Many times these large systems are reliant on a partner eco-system providing source data products and enabling tools and platforms. Their products are part of the ‘organism’.
- Continually Evolve Live Systems ensures both the Analytic and Data Product development pipeline and the large analytic systems themselves support continuous delivery of value, both during and after release into the field.
Lean Portfolio Management
The three competencies above provide the technical practices needed to build, deploy, manage and support a meaningful portfolio of Analytic and Data products. But none of them directly address the more significant issue of why those Analytic and Data Products are required, how they are funded and governed, and what other analytic capabilities are necessary to deliver enterprise value. Traditional Program / Portfolio Management approaches must be modernized to support the new Lean-Agile Product Management way of thinking to deliver value through Analytic and Data Products. The Lean Portfolio Management competency aligns enterprise strategy and the analytic program execution by applying Lean and systems thinking. The fundamental shift is to fund the Analytics Value Stream to continuously deliver Analytic and Data Products. The size of the fund pool is based on Product Managers estimation of value generation balanced against the demonstrated amount of value generation velocity of the Agile Teams. The allocation of the funding pool is continuously facilitated through a participatory budgeting process. The prioritization of the budget allocation is based on the estimated amount of value and needed time to market to realize that value. The net result is a fundamental shift. Bringing a continuous flow of work to the people instead of bringing people to the work. Again, comprised of three dimensions:
- Strategy and Investment Funding ensures that the entire analytic portfolio is aligned to the proper enterprise strategies and it is funded to create and maintain the Analytic and Data Products necessary to meet business targets across the functional organization and / or to meet the needs of the customers that use the analytics-enabled products and services of the enterprise. It requires the cooperation of the internal business function owners, portfolio stakeholders, technologists, Voice of the Customer Evangelists (VoiCE) and Enterprise Architects. It also requires input from the customer for their needs to ensure customer centricity.
- Agile Portfolio Operations (APO) coordinates and supports decentralized program execution across the Agile teams. APO enables operational excellence through a relentless quest for improvement via continuous feedback loops and corrective action implementation. It requires the cooperation of the Agile Program Management Office/Lean-Agile Center of Excellence (APMO/LACE), Communities of Interest (CoI), Voice of the Customer Evangelists (VoiCE) providing input to Product Managers, Release Train Engineers (RTEs) and Scrum Masters.
- Lean Governance manages spending, audit and compliance (guardrails), forecasting expenses, and measurement. It requires the engagement of the Agile PMO/LACE, Business Owners, and Enterprise Architects.
Even with the technical and programmatic oriented competencies above, enterprises must be able to adapt quickly to respond to the challenges and opportunities that today’s rapidly evolving markets and environments present. This requires more flexibility and adaptability than the traditional hierarchical operating system is likely to be able to muster. Again, we turn to the second operating system for help. The SAFe approach to cultivating a Lean-Agile Product Management mindset helps businesses address these challenges. Enter Organizational Agility, which is too is comprised of three dimensions:
- Lean-Thinking People and Agile Teams – This state occurs when everyone involved in solution delivery is trained in Lean and Agile methods and embraces and embodies the values, principles, and practices. Agile teams are not limited to technical teams. Every organizational unit in the enterprise would benefit from these principles.
- Lean Business Operations – Teams apply Lean principles to understand, map, and continuously improve the portfolio of Analytic and Data Products to optimize business processes that produce and support the business’ internal execution and delivery of products / services.
- Strategy Agility – This state occurs when the enterprise consistently demonstrates the ability and adaptability needed to continuously anticipate / sense shifts in market rhythms and quickly adapt strategy when necessary.
Continuous Learning Culture
And even with mastery of the above, there will be no steady-state. Change is a constant. Startup companies continue to challenge the status quo. Juggernaut data-driven companies like Amazon and Google are entering entirely new markets such as banking and healthcare and they are quickly dominating. Expectations from new generations of workers, customers, and society as a whole challenge companies to think and act beyond the theater of balance sheets and quarterly earnings reports. To address the demand for continuous learning, growth of its people, and improvement in processes, the Continuous Learning Culture competency describes a set of values and practices that encourage individuals—and the enterprise as a whole—to continually improve their data literacy, market knowledge, analytic tool competence, personal performance, and innovation contribution. It is expressed in three dimensions:
Learning Organization – Employees at every level are learning and growing so that the organization can transform and adapt to an ever-changing world.
Innovation Culture – Employees are encouraged and empowered to explore and implement creative analytic ideas that enable future value delivery. High performing, data-driven organizations invest in various forums for their people to engage, ideate, validate and propose new innovations that meet customer needs through value-generating solutions for inclusion in the portfolio of Analytic and Data Products. Self-Service oriented data and analytic products with embedded augmented intelligence machine-learning technologies enable the business knowledge workers to continuously ideate, validate and prototype new analytic capabilities at scale.
Relentless Improvement – Every person in the enterprise focuses on continuously improving its analytic solutions, products, and processes. Analytic solutions will only generate value when they are adopted, operationalized and utilized by the organization’s decision-makers and the end-customers for whom the solutions were deployed. This critical last mile of success is ensured by continuously soliciting feedback to learn what is needed to achieve that success. Also, the Analytic and Data Products themselves are deliberately designed with machine-learning based capabilities to continuously generate the telemetry necessary to monitor efficacy and to continuously improve the performance of those deployed and operationalized Analytic and Data Products.
The organization’s executives, managers, and other leaders provide the foundation ultimately responsible for the adoption and success of the Lean-Agile Product Management mindset and for the mastery of competencies that lead to Business Agility. Only they have the authority to effect change and continuously improve the systems that govern how work is performed. Only they can create an environment that encourages high-performing Agile teams to flourish and produce economic value from the analytics program. Leaders, therefore, must internalize and exemplify leaner ways of thinking and operating so that team members will learn from their example, coaching, and encouragement. Leaders develop along three dimensions:
- Leading by Example – Leaders gain earned authority by modeling the desired behaviors for others to follow, inspiring them to incorporate the leader’s example into their developmental journey.
- Mindset and Principles – By embedding the Lean-Agile way of working in their beliefs, decisions, responses, and actions, leaders model the expected norm throughout the organization. They drive the urgency to be a data-driven enterprise.
- Leading Change – Leaders lead and serve, not merely support, the data-driven, analytics-driven transformation by creating the environment, preparing the people, and providing the necessary investment in education and enabling resources to realize the desired outcomes from analytics!
The Call to Action
Ok, by now, you might be contemplating, “Doing the same thing in the same way within your analytics program and expecting a different / better outcome is folly, at best. However, this new operating system advocates a significant change in the way our Analytics program is run.” Yes, you’re right. Here’s the beauty of the Lean-Agile Product Management Mindset approach for delivering continuously improved value out of your Analytics program. It is predicated on continuous learning, it is founded on a continuous flow of Minimum Viable Product (MVP) increments not only in the Analytic products produced but equally important through deliberate incremental implementation of changes in process, discipline and technique for running the analytic program itself.. Establish your current level of health across the seven core competencies of Business Agility. Decide which of those health readings you want to improve upon over the next 3, 6, 9, 12 months and layout the implementation details for the journey towards Business Agility tailored for your enterprise analytic program. Measure the performance of the program’s ability to generate economic value for every Product Increment (PI) release (quarterly). Reassess your Business Agility health and establish the next set of desired improvements on a regular cadence (quarterly). These results will inform the enterprise where the next wave of continuous improvements need attention and investment.