O'Reilly FYI

News from within O'Reilly

Applying Lean to Software Development, an Excerpt from The Art of Software Development

By Sara Peyton
January 27, 2009 | Comments: 7

9780596517311_lrg.jpgJust checking out The Art of Lean Software Development, which is based on the techniques that revolutionized Japanese manufacturing. Written by Curt Hibbs, Steve Jewett, and Mike Sullivan, this concise new title shows you how to adopt Lean practices one at a time. And in this excerpt the authors discuss the importation relationship between Lean and Agile.

Aren't Lean and Agile just two names for the same thing?

This is a common question and a common misconception. The short answer is no, they are not the same. But you want the long answer, don't you?

It's easy to see why this misconception exists. Lean and Agile share the same goals: to increase the productivity of software development while simultaneously increasing the quality of the resulting software. To compound the confusion, the practices that make up the various Agile methodologies also support the Lean principles.

Agile has a different perspective from Lean and, generally, has a narrower focus. Conversely, Lean takes a wider view, preferring to look at the entire business context in which the software development is done. Lean views Agile software development methodologies as valid supporting practices of Lean software development.

Lean Software Development
In 2003, Mary and Tom Poppendieck published the first thorough mapping of Lean principles to software development in their book, Lean Software Development: An Agile Toolkit for Software Development Managers (Addison-Wesley Professional), and they refined this mapping in their second book, Implementing Lean Software Development: From Concept to Cash (Addison-Wesley Professional, 2006).

Most subsequent writings about Lean Software Development have followed the Poppendiecks' lead and used the seven principles they identified:

-- Eliminate waste

-- Build quality in

-- Create knowledge

-- Defer commitment

-- Deliver fast

-- Respect people

-- Optimize the whole

Let's take a closer look at each of these seven principles.

Eliminate Waste
This is a rather large category, and rightfully so, since the main thrust of Lean is the elimination of waste. Remember DOTWIMP, the seven deadly wastes? Let's see what the software development equivalents of defects, overproduction, transportation, waiting, inventory, motion, and processing would be.

This one is easy. A defect is a defect, whether you're talking about manufacturing a physical product or developing software. Defects cause expensive rework, which is always non-value-added waste. The focus in a Lean environment is on preventing defects, whereas traditional development focuses on finding defects after they have already occurred. Defects are especially expensive when detected late.

In Lean, when a defect is found, the response is to find the root cause of the defect and make changes that will ensure the defect cannot recur. In software development, this means having a suite of automated tests that prevent defects from slipping into the software undetected. When a defect does slip through, a new test is created to detect that defect so that it cannot pass through undetected again.

Overproduction--extra features
Every line of code costs money. Over the lifetime of the software, the cost of writing the code is probably the smallest cost. The code must also be designed, documented, and maintained (changed, enhanced, debugged, etc.). This means that a cadre of current and future team members must repeatedly read and understand the code. The presence of the code must be taken into account in each future product change or enhancement.

The 80/20 rule applies in most software products: 80% of the user's real needs are provided by 20% of the product's features. This means the other 80% of a product's features are rarely (or never) used. At the XP 2002 Conference, the Standish Group reported that 45% of features were never used and that only 20% of features were used often or always. In fact, a 2001 paper titled "Improving Software Investments through Requirements Validation" (IEEE 26th Software Engineering Workshop) found that in 400 projects examined over a period of 15 years, less than 5% of the code was actually useful or used!

Even worse, when the Standish Group looked at only successful projects in their CHAOS study, they found that 64% of features were never or rarely used (see Figure 2-1). This is a huge waste that becomes an increasing drain on the project's resources over time. The time spent working on these features would be much better spent working on the real needs of the customer. If a feature does not address a clear customer need, it should not be created.

If you have worked on large projects, this might sound familiar:

Analysts create a document containing all of the product's requirements and hand it off to the architects.

Architects take the requirements and create the product design, which they hand off to the programmers.

The programmers write the code to implement the design and pass the results to the QA testers.

The QA testers validate the resulting product against the requirements.

This is a classic Waterfall process that is replete with handoffs. A tremendous amount of knowledge is lost through each handoff simply because it is not possible to record everything that was learned, discovered, created, and known in a written form. A large amount of tacit knowledge is not passed on.

This means that the architects won't understand the requirements as deeply as the analysts, and that the programmers will not understand the design as well as the architects. This incomplete understanding will lead to errors and omissions, which will require costly rework to correct.

Even worse, the knowledge loss is compounded with each handoff. For example, if you assume that 50% of the knowledge is lost in each handoff, the programmers only receive 25% of the original knowledge, losing a whopping 75% in only two handoffs!

Try to avoid handoffs whenever possible.

Decisions are made almost constantly in every software development project. If a developer has extensive knowledge of the system being created, he will already know most of the answers to most of his questions (or be able to deduce them). However, the developer can't know everything and will invariably need to ask questions of coworkers, customers, and other stakeholders. If these people are immediately available, there is no delay and development continues at full speed.

When a question can't be immediately answered, the stage is set for all kinds of wasteful results. The developer could:

Suspend the current task and move onto something else (which is task switching, a waste that is covered shortly).

Guess the answer, which results in rework when the guess is wrong--potentially a lot of rework if it is not discovered until much later.

Try to find the answer. Even when the developer tries to find the answer, if it's too much trouble, he'll end up guessing to save the hassle.

No matter which scenario you pick, there is waste. This is why the most productive arrangement is colocated Integrated Product Teams (IPTs) that include all team members, including the customer (or a customer representative).

Inventory--partially completed work
Simply stated, partially completed work is anything that has been started but not finished. This could be requirements (features) that haven't been coded, or code that hasn't been tested, documented, and deployed, or bugs that haven't been fixed. Rather than letting partially done work build up in queues, the Lean approach uses single-piece flow to take a feature through to deployment as rapidly as possible.

A feature is not complete until it is potentially deployable (other constraints may not allow actual deployment as often as we would like), fully documented, tested, and error-free.

Motion--task switching
Task switching and interruptions kill productivity. It takes time to get the brain focused on the task at hand so that you can understand the required factors and begin the process of problem solving. Interruptions restart this process. Task switching (a much longer interruption) causes you to have to "relearn" where you were before you can even begin to be productive again.

This is why single-piece flow is so productive. You can work completely through a feature or task without the waste of task switching.

(Over) processing--unneeded processes
Unneeded processes are pure waste. They get in the way of productivity without adding any value. They include procedures that accomplish nothing and documents that no one reads. They also include manual tasks that could be automated and procedures that make simple tasks hard.

Build Quality in
One of Taiichi Ohno's key manufacturing insights was that you cannot inspect a product for quality at the end of a production line. That approach detects problems, but it does nothing to correct them. Instead, each step in the process should be mistake-proof and self-inspecting. When a problem is detected, the entire assembly line stops until the root cause of the problem is found and corrected (so that it cannot occur again).

A famous example is the New United Motor Manufacturing, Inc. (NUMMI) automobile factory, a joint venture between Toyota and General Motors. Toyota told workers simply to do good work and to stop the line whenever something prevented them from doing their jobs. It took nearly a month to produce the first vehicle, but since each problem was solved once--permanently--the plant quickly became a leader in quality and productivity in the U.S.

Traditional software development has followed the same pattern as traditional U.S. car manufacturing: let defects slip through and get caught later by QA inspections. The Lean approach is to mistake-proof your code by writing tests as you code the features. These tests prevent subsequent changes to the code from introducing undetected defects. Chapter 4 discusses automated testing in more detail.

Create Knowledge
The point here is not to forget the lessons you have learned. Obviously, making the same mistakes over again or relearning how something works is a waste of time and effort. And it's not just you--your coworkers shouldn't have to learn something you have already figured out.

Find ways to record your team's knowledge so that you can easily locate it the next time you need it. It's hard to be specific about this because what makes sense and what will work for you is highly dependent upon the context. However, it is generally best to store a given piece of knowledge closest to its source.

For example, say you are adding a feature to a system and you have to read through the code to understand how a subsystem works. What you learn should be recorded somewhere. You could add it to a detailed design document, but it would be much more useful to record it as a comment in the code. After all, the next time someone needs to know this information, he is likely to be looking at the code--just like you.

As you architect, design, and code, you will constantly be considering alternatives and making decisions. When you do make a decision, consider recording why you chose one alternative over another. Sometimes this knowledge can be a useful timesaver in the future, but sometimes it can be overkill. Use your best judgment and try to maintain a useful balance.

Defer Commitment
The best decisions are made when you have the most information available. If you don't have to make a particular decision now, wait until later when you have more knowledge and information. But don't wait too long, either--lack of a decision should not hold up other aspects of the project.

Wait until the last responsible moment to make an irreversible decision.

Let's take the case where you must commit to an architectural choice. First, determine when the last responsible moment for that decision is. Use the interim time to accumulate knowledge about the real needs of the other components of the system. Use that time to explore the characteristics of alternative choices.

For example, would a simple dispatch queue work, or does the system require a publish-subscribe messaging engine? What choices lie between these two extremes? The best approach to take, if you can, is to try both alternatives and eventually choose the one that best meets the system's needs.

This is called set-based design. With set-based design you simultaneously pursue multiple solutions, eventually choosing the best one.

A classic example of this approach was the design of the Toyota Prius. The requirements for the Prius did not specify a hybrid engine; they only stated that the car should get exceptionally good gas mileage. Several competing engine designs were developed simultaneously. The hybrid engine was chosen at the last responsible moment (when there was just enough time to get the Prius into production by its target release date).

This might seem like waste, but in reality, making the wrong choice could have dramatically reduced the success of the Prius, resulting in wasteful lost opportunity.

Deliver Fast
Software development is an abstract endeavor. Yet most of us (including our customers) work best when dealing with concrete things. When we can see, touch, and feel something, it becomes real, and our brains can more easily think about what works and what doesn't. Our imaginations can dream of features or capabilities that we didn't realize we needed.

This is why software requirements are so volatile. The Waterfall approach would have you waiting until the end of the project to get customer feedback based on the actual use of the software, and that is why the Waterfall process is so prone to failure.

"Deliver fast" means developing features in small batches that are delivered to the customer quickly, in short iterations. These features can be implemented and delivered before the associated requirements can change. This means that the customer has an opportunity to use these features (which are now concrete) to provide feedback that can change the other requirements before they are implemented.

The completion of each short iteration provides an opportunity to change and reprioritize the requirements based on real feedback and use. The end result is a product that more closely meets the real needs of the customer, while simultaneously eliminating the tremendous amount of waste and rework created by the requirements churn--truly a win-win situation.

Respect People
This is a lofty altruism that is also the down-home truth. As Mary and Tom Poppendieck said in Implementing Lean Software Development, "Engaged, thinking people provide the most sustainable competitive advantage."

Respect for people means trusting them to know the best way to do their jobs, engaging them to expose flaws in the current process, and encouraging them to find ways to improve their jobs and the surrounding processes. Respect for people means recognizing them for their accomplishments and actively soliciting their advice.

Don't waste your most valuable resource--the minds of your team members!

Optimize the Whole
This is an important part of Lean thinking no matter where Lean is being applied, and Lean software development is no exception. Whenever you optimize a local process, you are almost always doing so at the expense of the whole value stream (this is suboptimizing).

If you don't have control over the entire value stream, you may be forced to suboptimize a piece of it. In general, though, you should always attempt to include as much of the value stream as you can when you try to optimize a process.

If you enjoyed this excerpt, buy a copy of The Art of Lean Software Development.

You might also be interested in:


Sounds intriguing, Sara, especially for a book that weighs in at less than 200 pages. When will it be available for review? Cheers.

Thank you. The book is ready for reviewing now. I'll send you a review copy.

I've been reading about the topic, Dasari Ravi talks about some other principles that are also related to Lean Software Development:

* Satisfy stake holders;
* Empowerment;
* Deploying Comprehensive testing;
* Refactoring;
* Learn by Experimentation;
* Measure Business impact;
* Optimize across organization.

A very informative article. I am definitely going to buy this book and read it.

This article got me thinking about how similar Agile and Lean really are. Check out my blog post ... http://blog.agilebuddy.com/2009/01/agile-and-lean-closer-than-you-think.html.


Thank you Jack for posting the linking to your excellent blog.

As a practitioner of a hybrid methodology (we took any idea that helped us and added some of our own), I'm not sure of the differences between the strict methodologies. Here is what worked for us:

I participated in an agile-scrum group that implemented much of these lean methodologies. This department was able to double programmer productivity almost immediately—but there was some temporary stress increase as the change from waterfall to agile was made—they were able to produce a User’s Guide for each feature while the programming and QA was happening, and they were able to release features most needed by customers in a truly agile and quick manner. They accomplished this by the following practices and some others:
An analyst prepares some ideas for a project’s features and then schedules a meeting (Feature Blitz) where all the development team [domain expert(s), programmer, code reviewer, SQA engineer, the analyst, and any manager that really wanted to attend] decide the features needed, their specs, their implementation, the proposed order they will be coded and list of dependencies, breaks them into iteration-sized mini-projects, and at the end of the meeting assigns people to each of the above roles for each mini-project. The specs are the notes and digital photos from this meeting. Next each member of the team enters estimates of hours for them to finish their role’s work for each iteration or mini-project. The managers, in Iteration Planning, then swap around any resources or mini-project to maximize the use of everyone’s time, keeping in mind that changes may require the new team member to meet with the analyst or others to get up-to-speed on what happened at the Feature Blitz and that the new team member may not be able to do the role’s work as fast as the one who helped decide the particular implementation in the Feature Blitz. Next there is a commitment meeting where the development team (the four roles besides domain expert) commits to their deadlines even if they have to do overtime, which rarely happened because people became good at estimating their work ability. Simultaneously during an iteration, the analyst writes the User’s Guide, the QA engineer wrote a test suite with corner cases and exceptions that programming may not have thought about, the programmer and code reviewer discuss the details of the implementation and do the coding. Obstacles and progress were discussed frequently among the whole team and communicated to management and other stakeholders. QA tested each code drop so a programming path that wasn’t going to work was caught quickly and little code had to be thrown out. QA had enough time to do their job properly and checked how the feature functioned with other features. Analysts, QA, programmers, and domain experts generated other enhancement ideas not only for the project, but for other projects as there was more time for exploratory testing, prototyping, and thinking about the entire product/program. At the end of the iteration, the released product was more closely bug-free than the waterfall methodology. Lastly, QA had test suites that could be assigned in the future to other QA members or customer support for re-testing prior to each product/program release.

We used a Quality Function Deployment (QFD) chart in Excel to prioritize our Backlog.
Each person (Programmer, Analyst, Code Reviewer, & QA) puts in their estimates for the stories they are assigned.
Each manager puts in the person's hours capacity for the iteration.
Stories are swapped around until everyone is at full capacity.
We had a commit rule that if you committed, you did overtime and got comp time another iteration/week. Everything was done on time, unless the roadblock was from another department, somebody had surgery, or some other insurmountable "event".

Thank you for your information and services so it worked.


Popular Topics

Browse Books


Or, visit our complete archives.

FYI Topics

Recommended for You

Got a Question?