Stopping the Software Factory

It’s only a good practice if you have the cultural discipline to make it work.

Stopping the factory used to be sacrilege — companies figured they were only making money if the line was moving — so when Toyota adopted this practice on their manufacturing line is was a huge shift. It was taken so they could resolve quality problems immediately, which saved them time and money because otherwise the problems occurred in many cars and had to be fixed later. It was better to pay the price of stopping the line now than the price of redoing the work later.

This was a huge cultural shift throughout the organization. Management had to recognize the cost of fixing quality later, and change their short-termer mindset around the moving line. The workers on the line had to learn not to let problems slide by, to ask for help immediately and not rest until the problem was truly resolved. Everyone had to learn to diagnose problems in the line without blaming individuals, otherwise trust was impossible and the real problems could not be diagnosed. (This is also where Toyota developed their ‘Five whys’ practice for getting to the bottom of problems.)

While we’re obsessed with individual techniques like Kanban, the team at Toyota knows that it’s culture that allows them to win, not the tools they use. That difference is also exactly why it took decades for the rest of the industry to catch up: You can adopt technology quickly, but culture? Not so much.

We experienced the same problem when we tried to adopt this practice in our software teams. People loved the ability to stop the factory, and they used it all the time. Often, it did actually increase the quality of the work we did. But overall, it made things worse for us, not better. It was causing arguments, bad relations, and most importantly, was reducing our long-term efficiency, not increasing it. It became nearly impossible to ship software.

What went wrong?

We had adopted this practice without developing the cultural discipline to make it work. People weren’t stopping their own lines to resolve systemic quality issues; instead, they were stopping the work of other people to question their decisions. A technique for improving our work became a technique for attacking our strategy. Disagree with the product that team is building? Stop the factory! Don’t like the database they chose? Pull the cord!

Yuck. That’s not how that’s supposed to work. It didn’t help that I’ve always pushed hard for people at the front line to be opinionated and educated about as much of the product as possible. Suddenly they’re applying those opinions to the work of others, and we’re not building higher quality products, we’re just arguing non-stop about what we should be building and how.

This required that we develop a couple of new cultural skills. When you read about the Toyota system, it’s striking how much of what they accomplished was essentially cultural technology — that is, tools for developing and sustaining culture — rather than manufacturing tech.

Our failed experiment at letting anyone stop the factory showed that first everyone needed to understand the value in stopping the line. We started to point out that they should be focused on their own work, not others’, and thus should only be able to stop their own line. Most importantly, remember back to the point of pulling the cord: You’ve discovered that your own work is compromised because of a quality problem in the system, and you need time and help to fix it. That had to be why we stopped the line, and nothing else.

That being said, people’s concerns about decision making in other parts of the organization needed to be heard. Expert teams should being given the power to build what they think is best. But even the best teams need outside reviewers, need other experts to question their assumptions and conclusions.

Our work cycle needed to include productive, constructive feedback opportunities. The best process like this I’ve ever seen is in Pixar founder Ed Catmull’s Creativity, Inc, where he describes their “Brain Trust” system for helping directors make great movies. This is such a great system that it’s worth reading yourself, but the aspect that stands out is the responsibility of the team giving feedback to the director:

Your job is to help them see the flaws in their proposal. You can not and must not propose solutions, and you can not demand that they agree with your complaint. Even if you’re the CEO. You are a collaborator helping someone do their best work, not a competing director trying to get your ideas into the movie.

See the parallels to the value in stopping the line? It’s about improving the quality of work, not making a power play, or being right about something.

Pixar has a critical second layer to this process: Management decides whether to ship the movie. This allows them to give the director full responsibility for the movie. If directors don’t agree with the team’s list of flaws, that’s fine, but in the end, you have to convince the management team that the movie works.

The thing I love about this is it gives you concentric circles of iteration and control. The movie team makes decisions constantly about what to do and why, then they loop out to the Brain Trust to get feedback on what they’ve done, and finally there’s a business cycle that determines whether the movie should ship. Everyone gets to contribute, but your business concerns are isolated from the work cycle, giving teams the freedom they need to build something great.

I was never able to figure out how to adapt their process to software, but I hold out great hope that there is a way, it just requires a lot of evolution. Pixar gets to take five years to make a great movie, but you never get that kind of window to make software. They also kept all of the iteration inside the building, where modern software should be built in layers, in collaboration with your users and the market.

Even without being able to directly adopt a better practice, we recognized the need for two layers of process. We thinned down people’s ability to stop the work of others, and at the same time built feedback loops at larger intervals and in different forums for strategic concerns to be heard. We also began building cross links between teams that allowed those with concerns to collaborate with those doing the work, rather than leaping in and shouting ‘stop!’.

There are many great inspirations for how to build better software, faster, but all require some level of adaptation. I am absolutely convinced we will see a software production system as coherent and clear as Lean Manufacturing, but I also think it’s going to take a long time, a lot of experimentation and failure to get from here to there.

Hopefully our experience, our successes and failures, can help you get there faster.