Project to Product: How to Survive and Thrive in the Age of Digital Disruption with the Flow Framework

Praise for

With the introduction of the Flow Framework, Mik has provided a missing element to any large-scale Agile transformation. I recommend that anyone involved in complex product delivery read this book and think about how they can apply this thinking to their value stream.

—Dave West, CEO Scrum.org and author of Nexus Framework for Scaling Scrum: Continuously Delivering an Integrated Product with Multiple Scrum Teams

During our transformation to 100% Agile BMW Group IT organization, we discovered early on that the former project portfolio approach did not sufficiently support our journey. Therefore, we started with a transition from project to product. The exchanges with Mik on the topic of product orientation and the Flow Framework was very helpful and a real inspiration for me. The fact that Mik is now sharing his vast knowledge in this book makes me particularly happy. It provides the motivation and the toolset necessary to help create a product portfolio based on a value driven approach. For me it is a must read—and indeed it is also a fun read.

—Ralf Waltram, Head of IT Systems Research & Development, BMW Group

Organizing software development as a group of loosely connected projects will never lead to good products. Kersten explains how to tie work products to value streams corresponding to features, defects, security, and (technical) debt....[Project to Product is a] major contribution to the theory of management in the age of software.

—Carliss Y. Baldwin, William L. White Professor of Business Administration at the Harvard Business School, Emerita, and co-author of Design Rules, Volume 1: The Power of Modularity

If you want to get rid of obsolete practices and succeed with the new digital, read this book.

—Carlota Perez, author of Technological Revolutions and Financial Capital: The Dynamics of Bubbles and Golden Ages

Every now and then, a body of work comes along with such timely precision that you think hallelujah! Mik’s book Project to Product is the perfect antidote for those businesses struggling with digital transformation, broken Agile implementations, and the onslaught of enterprise disruption. In fact, it’s a really important component in the world of flow which is at the forefront of business agility. Not only will this framework help your teams to ignite their software delivery cadence but to do it at scale with high quality, reduced costs, and increased value. And more importantly, with happy teams—and with the metrics to prove it.

—Fin Goulding, International CIO at Aviva and co-author of Flow: A Handbook for Change-Makers, Mavericks, Innovation Activists, Leaders: Digital Transformation Simplified, and 12 Steps to Flow: The New Framework for Business Agility

I had the pleasure of having an advance copy of Project to Product at my company over the summer—and what an eye opener it is. This book is spot on the journey Volvo Car Group is starting up right now. The insight Mik has in our industry and the way his book describes the Age of Software makes this our new go to book for our product journey in our digital landscape!

—Niclas Ericsson, Senior IT Manager, Volvo Car Corp

Project to Product is going to be one of the most influential reads of 2019 and beyond. One that connects work outcomes to business results. One that provides models to make better business decisions. One that gives technology leaders a framework to enable the change necessary for companies to remain relevant.

—Dominica DeGrandis, author of Making Work Visible: Exposing Time Theft to Optimize Work & Flow

Many large organizations are still applying a management model from the early 1900s optimized for manual labor to everything they do, including complex, unique, product development. With this book, Mik provides a great articulation of the importance of focussing on the work not the workers, on the value stream network, and lessons learned on what to avoid. Mik, who has many years experience working with hundreds of companies on this topic, shares his wisdom and insights via a Flow Framework, which is immensely valuable for organizations who recognize the need to move to better ways of working.

—Jonathan Smart, Head of Ways of Working, Barclays

Project to Product is a very insightful book, and the overall model Mik lays out for the Flow Framework is especially intriguing. Not only does Mik address the complexities of Agile transformation and moving to a product-based development, he also discusses how to get your architecture, process, and metrics integrated in a way to effectively measure value delivery. I got pretty excited about the Flow Framework and look forward to applying it to my own technology transformation activities.

—Ross Clanton, Executive Director, Technology Modernization, Verizon

25 NW 23rd Pl, Suite 6314

Portland, OR 97210

Copyright © 2018 by Mik Kersten, all rights reserved.

For information about permission to reproduce selections from this book, write to

Permissions, IT Revolution Press, LLC, 25 NW 23rd Pl., Suite 6314, Portland, OR 97210.

First Edition

Printed in the United States of America

22 21 20 19 18 1 2 3 4 5 6

Cover illustration by Rachel Masterson

Figure illustrations by Zhen Wang

Cover and book design by Devon Smith

Author photograph by Janine Coney

Library of Congress Catalog-in-Publication Data

is available upon request.

ISBN: 978-1-942-78839-3

eBook ISBN: 978-1-942-78840-9

Kindle ISBN: 978-1-942-78841-6

Web PDF ISBN: 978-1-942-78842-3

Publisher’s note: Many of the ideas, quotations, and paraphrases attributed to different thinkers and industry leaders herein are excerpted from informal conversations, correspondence, interviews, conference round tables, and other forms of oral communication with the author. Although the author and publisher have made every effort to ensure that the information in this book was correct at press time, the author and publisher do not assume and hereby disclaim any liability to any party for any loss, damage, or disruption caused by errors or omissions, whether such errors or omissions result from negligence, accident, or any other cause.

For information about special discounts for bulk purchases or for information on booking authors for an event, please visit our website at ITRevolution.com.

The Flow Framework, the Value Stream Diagrams and studies, and all related materials, to include tables, graphs, and figures, are copyrighted by Tasktop Technologies, Inc. 2017-2018, with all rights reserved, and are used with permission wherever they may appear within this book.

PROJECT TO PRODUCT

To my mother, who made me who I am,

and my father, who taught me who to be.

CONTENTS

ILLUSTRATIONS

FOREWORD

by Gene Kim

The mark of a great book is that it makes obvious what is wrong with the old worldview and replaces it with one that is simultaneously simpler and yet presents a better model of reality. The transition from Copernican to Newtonian physics has been long held as a great example of such a breakthrough. I believe Project to Product presents a new way to think that enables a new way of doing.

In today’s business landscape, with companies facing the threat of digital disruption, the old ways of planning and executing no longer seem enough to survive. For decades, great minds have been seeking a way to manage technology to achieve business goals—after all, we know there is something very, very wrong with the way we’re managing technology, we see the poor outcomes with our own eyes.

Project to Product makes the solid case that in the Age of Software, the methods that served us well for over a century are truly coming to an end: project management, managing technology as a cost center, traditional outsourcing strategies, and relying on software architecture as the primary means to increase developer productivity. And better yet, it provides a wonderful framework to replace it, namely the Flow Framework.

You’ll learn what it looked like when an organization spent over a billion dollars on a technology transformation that was doomed to fail from the beginning because it was trying to solve the wrong problem. You will learn how some of the fastest growing companies nearly died by ignoring technical debt that was accumulated in their need to cut corners to ship products quickly, which included Nokia’s massive Agile transformation that did nothing to stop its demise.

Dr. Mik Kersten brings the perspective of someone who got his PhD in software engineering only to discover that the massive productivity gains were not to be found there. Instead, those productivity gains can only be reaped when we change how teams across the entire business value stream work together, an epiphany common to so many of us in the DevOps community.

But he also brings the perspective of someone who built the large open-source software community around Mylyn in the Eclipse ecosystem, used by millions of Java developers. As founder and CEO of a software company, he brings a visceral understanding of what it’s like to live and die by the ability of business, product, and engineering leadership to work together effectively.

You’ll also follow in Dr. Kersten’s professional journey and relive his three biggest epiphanies—fans of The Phoenix Project will especially love the lessons learned from the BMW Group Leipzig manufacturing plant, which he rightly calls a pinnacle of the Age of Mass Production, and the profound lessons that the software industry can learn from it.

Project to Product is an incredible achievement. Dr. Kersten provides a better way to think about how business and technology organizations create value together, and provides the Flow Framework as a way for those leaders to plan and execute together, to innovate for their customers, and to win in the marketplace. To disrupt, instead of being disrupted. The upcoming Deployment Period Age of Software may bring the equivalent of an economic extinction event, so these capabilities are no longer optional for survival.

Every decade, there are a couple of books that genuinely change my worldview. You can tell which books they are, because more than one-third of the pages are bookmarked, indicating something I felt was truly an important a-ha moment or a reminder to myself to study further later. This is one such book.

I hope you find it as rewarding and life-changing as I did.

Gene Kim

Portland, OR

September 4, 2018

INTRODUCTION

The Turning Point

For the majority of our careers, those of us involved with enterprise IT have been dealing with change at a frenzied pace. Technology platforms, software development methodologies, and the vendor landscape have been shifting at a rate that few organizations have been able to match. Those that manage to keep up, such as Amazon, and Alibaba, are further driving change by redefining the technology landscape around their software platforms, causing the rest to fall even further behind.

This daunting and unrelenting pace of change has been seen as a hallmark of the digital disruption. But if we step back and look at the patterns of progress that came before, we begin to see ripples of the great surges of change and development of previous industrial and technological revolutions.

Over the course of three centuries, a pattern emerges. Starting with the Industrial Revolution, every fifty years or so a new technological wave combines with ecosystems of innovation and financing to transform the world economy.¹ Each of these technological waves has redefined the means of production so fundamentally that it triggered an explosion of new businesses followed by the mass extinction of those businesses that thrived in the culmination of the previous surge. Each wave has been triggered by the critical factor of production becoming cheap. New infrastructure is then built while financial capital drives the ecosystem of entrepreneurs and innovators who leverage the new techonological systems to disrupt and displace the incumbants of the last age.

Each of these technological revolutions has required existing businesses to master a new means for production, such as steam or the assembly line. For the digital revolution, the new means of production is software. If your organization has already mastered software delivery at scale, this book is not for you. The goal of this book is to provide everyone else with a new managerial framework that catalyzes the transition to the Age of Software.

Theories that explain the cycles of the last four technological revolutions and the first half of this one are proposed by Carlota Perez in Technological Revolutions and Financial Capital: The Dynamics of Bubbles and Golden Ages and by Chris Freeman and Francisco Louçã in As Time Goes By: From the Industrial Revolutions to the Information Revolution. Perez expands on the long wave or Kondratiev economic model by specifying two distinct periods within each cycle (Figure 0.1). The first half is the Installation Period, when a new technology and financial capital combine to create a Cambrian explosion of startups, disrupting entire industries of the previous age. At the end of the Installation Period is the Deployment Period of technological diffusion, when the production capital of new industrial giants starts taking over. Between these two periods is what Perez termed the Turning Point, historically marked by financial crashes and recoveries. This is when businesses either master the new means of production or decline and become relics of the last age.²

Figure 0.1: Technological Revolutions and the Age of Software.³

Fifty years have passed since NATO held the first conference on software engineering in 1968 and the Age of Software officially began. Today, the pace of change feels relentless because we are passing through the Turning Point. At the current rate of disruption and decline, half of S&P 500 companies will be replaced in the next ten years.⁴

These businesses, many of which were founded prior to the Age of Software, are starting to see a growing portion of their spending shift to technology as their market success is increasingly determined by software. However, the productivity of software delivery at enterprise organizations falls woefully behind that of the tech giants, and the digital transformations that should be turning the tide are failing to deliver business results.

The problem is not with our organizations realizing that they need to transform; the problem is that organizations are using managerial frameworks and infrastructure models from past revolutions to manage their businesses in this one. Managerial accounting, organizational hierarchies, and Lean manufacturing were critical to success in previous revolutions. But these managerial frameworks are no longer sufficient to successfully direct and protect a business in the Age of Software.

I had a chance to witness the pitfalls of this trap firsthand. Working with Nokia, I noticed that management was measuring the success of its digital transformation by how many people were trained on Agile software development methodologies and were onboarded onto Agile tools. These activity-based proxy metrics had nothing to do with business outcomes. As I will summarize in Part I, Nokia’s transformation efforts failed to address the core platform problems that made it so difficult for the company to adapt to the changing market. In spite of what appeared to be a well-planned transformation, management was not able to realize this until too late. I watched with frustration as Nokia lost the mobile market it had created, in spite of the heroic efforts of my colleagues, who were doing everything they could to save the company.

A few years later, I was invited to speak with IT leaders at a global bank. The bank was six months into its third attempt at a digital transformation, and this time, DevOps tools were added to the mix and expected to save the day. The budget for the transformation was approximately $1 billion, but shockingly, I realized their transformation plan was even more flawed in its approach than the one at Nokia. Every aspect of the transformation was being project managed to cost reduction alone and not to project overall business outcome with reduced cost as a key metric. As I learned more, I started getting a visceral image that a billion dollars of the world’s wealth was going to go up in flames without producing any value. There were still eighteen months left to right the ship, but I knew that with cost alone as the foundation of the transformation, it was too late to alter course. Nokia had left me with an image of a burning mobile platform that destroyed a tremendous amount of wealth and prosperity. I now had a vivid image of the bank’s digital transformation lighting fires of waste across its ranks.

That was the day I started this book. There was something so fundamentally wrong with the way business people and technologists worked and communicated that even leaders with the best of intentions could still lead their companies into predictable decline.

How is this possible when we now have five decades of software practice behind us? The Agile and DevOps movements have made great strides in adapting key production techniques from the Age of Mass Production to the technical practice of building software. For example, continuous delivery pipelines allow organizations to leverage the best practices of automated production lines. Agile techniques capture some of the best technical management practices of Lean manufacturing and adapt them to software delivery.

The problem is, with the exception of some tech giants run by former software engineers, these techniques are completely disconnected from the way that the business is managed, budgeted, and planned. Software delivery concepts near and dear to technologists, such as technical debt and story points, are meaningless to most business leaders who manage IT initiatives as projects and measure them by whether they are on time and on budget. Project-oriented management frameworks work well for creating bridges and data centers, but they are woefully inadequate for surviving the Turning Point of the Age of Software.

In this book, we will examine several digital transformation failures that caused organizations to lose their place in the market. We will then dig further into understanding the current state of enterprise software delivery by looking at a study I conducted with Tasktop, Mining the Ground Truth of Enterprise Toolchains, that analyzed the Agile and DevOps toolchains of 308 organizations to uncover the causes of this disconnect between business and technology.⁵ Project to Product will then provide you with a new management framework and infrastructure model, called the Flow Framework, for bridging this gap between business and technology.

The Flow Framework is a new way of seeing and measuring delivery and aligning all of your IT investments according to value streams that define the set of activities for bringing business value to the market, via software products or software as a service (SaaS). The Flow Framework displaces project-oriented management, cost center budgeting, and organizational charts as the primary methods of measuring software initiatives. These are replaced with flow metrics for connecting technology investment to business results. The Flow Framework allows you to scale the Three Ways of DevOps—flow, feedback, and continual learning (as outlined in The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations⁶)—beyond your technology organization and to your entire business.

With each technological revolution, a new kind of infrastructure has been established in order to support the new means of production. Canals, railways, electrical grids, and assembly lines were key infrastructure components that underpinned the technological ecosystems of previous cycles. Many digital transformations have gone wrong by over applying infrastructure concepts of the last revolution to this one. Production and assembly lines are great at reducing variability and reliably producing similar widgets, but software delivery is an inherently variable and creative endeavor that spans a complex network of people, processes, and tools. Unlike manufacturing, in modern software delivery the product development and design process are completely intertwined with the manufacturing process of software releases. Attempting to manage software delivery the way we manage production lines is another instance where frameworks from previous technological revolution are failing us in this one. The Flow Framework points to a new and better way.

What if we could see the flow of business value within our organizations in real time, all the way from strategic initiative to running software, the way the masters of the last age ensured they could see and collect telemetry for every step of the assembly line? Would we see a linear flow or a complex network of dependencies and feedback loops? As the data set of 308 enterprise IT toolchains we will examine in Chapter 8 demonstrates, we see the latter. This flow of business value within and across organizations is the Value Stream Network. In the Age of Software, Value Stream Networks are the new infrastructure for innovation. A connected Value Stream Network will allow you to measure, in real time, all software delivery investments and activities, and it will allow you to connect those flow metrics to business outcomes. It will empower your teams to do what they love doing, which is to deliver value for their particular specialty in the value stream.

A developer’s primary function and expertise is coding, yet studies summarized in this book have shown that developers spend more than half their time