Last week Jenny and I gave our new talk, Getting Agile Right [pdf], for the first time. We’re really excited, because it also marks our first public announcement of our current book project for  O’Reilly: a new book about agile development and project management. It’s aimed at people preparing for the PMI-ACP certification, but our goal is to help anyone who’s interested in agile really understand the ideas behind it.

We talk to a lot of teams and help  them understand what’s gone right and wrong with their projects, and look for common patterns of project problems and failure (that’s what our Why Projects Fail talk [pdf] is all about.) People starting with agile often us about something that we call “better-than-not-doing-it results.” Basically, they adopt a bunch of great agile practices, and they see a clear improvement. It was definitely worth going agile, but something feels a little hollow about it. They were expecting the “astonishing results” and “hyper-productive teams” that they’d read about in agile books and blog posts, but there’s a feeling that at its core, the team hasn’t really changed how they do things, they just made incremental improvements.

I recently read Lyssa Adkins’ excellent book, Coaching Agile Teams, and one of the really insightful things she points out is that “metaphor is a powerful thing.” Jenny and I put a lot of thought into coming up with a good metaphor to help explain what’s going on. We hit on a really good one: the story of the blind men and the elephant. I like the Jain version of the story (from Wikipedia):

A Jain version of the story says that six blind men were asked to determine what an elephant looked like by feeling different parts of the elephant’s body. The blind man who feels a leg says the elephant is like a pillar; the one who feels the tail says the elephant is like a rope; the one who feels the trunk says the elephant is like a tree branch; the one who feels the ear says the elephant is like a hand fan; the one who feels the belly says the elephant is like a wall; and the one who feels the tusk says the elephant is like a solid pipe.

A king explains to them: ”All of you are right. The reason every one of you is telling it differently is because each one of you touched the different part of the elephant. So, actually the elephant has all the features you mentioned.”

So what does this have to do with agile teams having trouble getting past better-than-not-doing-it results?

Teams that get better-than-not-doing-it results from agile are often the ones who were already able to get software out the door reasonably well before starting with agile, and were hoping that agile adoption would help them get a real improvement. The problem is that before the team started adopting agile practices, they were experiencing problems—not the serious, software crisis problems that caused their projects to fail outright, but problems that caused friction and discomfort on the team. The source of these problems is something that we call a “fractured perspective”: the developers think about developer stuff, project managers think about project manager stuff, and they throw the code over the wall to a business user who thinks about business stuff. Everyone is really busy thinking about his or her own project work. There isn’t a lot of communication between people, and they’re really functioning as individuals working separately towards compatible goals, not as a team.

That’s where the Blind Men and the Elephant story comes in—it’s a good metaphor for how a team with a fractured perspective adopts agile. Each person sees the practices that impact his or her work. Developers concentrate on, say, test-driven development, refactoring, and automated builds. Project managers like task boards, project velocity, and burndown charts. Business users use release planning and user stories to get a better grasp on what the team is doing. Team leads use daily standups and retrospectives to manage and improve the team. Everybody wants something different from the project, and they each see a few practices that do something specific to help them.

And that’s definitely going to improve things, because agile practices are generally really good. The problem is that since everyone—developers, project managers, business users, and team leads—sees the project from a different perspective, they’ll concentrate on only those practices that immediately appeal to them. There’s a paradoxical effect (we call it, “See! I was right all along”) where each person now sees only the part of agile that affects his specific project work, and draws the conclusion that agile is all about getting everyone else to come around to his point of view.

But agile is more than just practices. In fact, that’s one of the core ideas behind agile: principles over practices. So while the agile “elephant” is made up of many great practices, the whole thing is greater than the sum of the parts. And if you only see practices—especially if you’re only looking at the practices that directly affect your project work—then you’ll only see one small piece of agile. The “elephant” of agile is made up of the practices day to day, but it’s much bigger than just those practices.

A team whose members only see the practices and don’t think about the principles will miss out on the important interactions between people. Their perspective stays fractured, and they stay separate and don’t really function as an effective team. They’ll still get their work done, but they miss out on the great team interactions and collaboration that make agile really work.

This is built into agile. If it’s been a while since you’ve had a look at the agile manifesto, open it up again and have a look at the very first value:

Individuals and interactions over processes and tools

Processes, methodologies, and tools are still important (“…while there is value in the items on the right, we value the items on the left more.”). But even more important than specific practices are the individuals and interactions. It’s these values, and the twelve principles, that show us how the practices work together, and serve as a guide for how teams adopt those practices.

That’s one of the lessons of our “Getting Agile Right” talk [pdf]. It’s also going to be one of the big themes in our upcoming book, due out from O’Reilly next year, about agile development, project management, and the new PMI-ACP agile certification. We’ll continue to make posts to help connect these dots and learn more about agile.

Tomorrow at the Agile 2009 conference, Abby Fichtner and Nate Oster are doing a workshop called Where Does Developer Testing End and Tester Testing Begin?. Jenny and I hope you can make it, because they’ll be doing a giveaway of autographed copies Beautiful Teams. Check out my O’Reilly Community posts for more information:

• Agile testing and Beautiful Teams (giveaway)
• Agile testing: why good developers resist great habits

In that second post, I spend a little time talking about some of the reasons that programmers resist great practices like test-driven development. Writing that post reminded me of something that Jenny and I wrote about change in Applied Software Project Management:

Many project managers—especially ones who have a technical background—tend to ignore the fact that their organizations are made up of people who need to be convinced of the importance of a change before they will adopt it. Some of these people will have an emotional or even irrational response to any attempt at change; it could take a sea change in the organization before they agree to it.

Irrational attitudes about software development usually boil down to two basic beliefs. First, people believe that most or all software projects are delivered late and delivered with many bugs, and that this is just a fact of life. Second, they believe that their organization is unique, and that the problems they are experiencing are particular to their organization and have never been seen before in any other organization.

(This second belief may seem odd, considering the many thousands of software organizations around the world that have all used similar tools and techniques to fix very similar problems and make real, lasting improvements. It’s possible that the belief in uniqueness comes from the fact that the software being built truly is unique, in that it has never been built before; it’s not a leap to assume—incorrectly—that the software project and all of itsproblems are therefore also unique to that particular organization.)

Many times, resistance is not irrational at all. Anyone who has been through a change previously—possibly a passing management fad—that didn’t fix the problem (or failed outright) may be resistant to another change. It may seem unfair, but if people in your organization have previously gone through poorly planned changes, it will be harder for you to make changes of your own.

When you are introducing new tools, techniques, or practices in your organization, you may encounter resistance for a number of reasons. By exploring the feelings, fears, and justifications for resisting change that project managers commonly encounter, these reactions can be unraveled and understood.

— Stellman & Greene, Applied Software Project Management, p206 (O’Reilly 2005)

It’s really easy to forget about this when you’re pushing for a change, especially something that requires extra work and learning. (I had to learn that the hard way – hopefully you won’t have to!)

I got a great question from a software developer who also happens to be a fellow CMU alum.

I have a question related to managing scope creep with respect to “on-going”/iterative development processes.

I’m currently managing a project where we’re redesigning my application’s primary workflow. Simply put, the app is currently designed to have users to signoff all items and we’re redesigning it to be exception-based (only require certain items to be signed off).

As we’ve progressed down the path of planned iterative development, a lot of good/new ideas for future enhancements/requirements spring up. I find myself regularly working with my users and “working group” to prioritize and analyze if any of these new ideas need to be considered to build sooner rather than later (and thus triggering plan adjustments or delays).

I often feel like I’ll end up delivering a product that does deliver the initial vision, but still doesn’t make my users happy, as they’ve already shifted their expectations to wanted the “next” thing (aka phase 2).

Do you have any other tips about how to manage this process?

I’ve used things like taking a timeline-style roadmap and adjusting it by overlaying the new requirements and shifting the timeline out. Do you have an recommendations of ways to present this type of information?

Curious your thoughts. Thanks.

— Seth L.

Iterative software development can be a really useful—and highly effective—tool for software development, but it’s also one of the most abused tools I’ve seen. Even as recently as a few days ago, someone in charge of a software team that’s critical to his comapny came to me with the (incorrect) assertion that iteration just means diving into a prototype without talking to anyone or doing any investigation. Iteration done well can lead to very high quality software. But as Seth saw, iteration done poorly can lead to scope creep and serious planning issues.

Making your users “happy” means managing their expectations. They need to see exactly what you’re working on, and what’s coming next. If all people see is deadlines and they don’t have a sense of what’s going on, then they naturally start looking to the next deadline, because that’s all they see. The more visibility you can give them into the way you build software, the more understanding they’ll have – that’s just human nature.

There are a few things that work well for improving visibility into your project’s iterations. One of them is a task board, which typically involves sticking index cards with your user stories or scenarios on a whiteboard or wall. This means that you actually need to have user stories or scenarios. Scope creep is often an indication of a requirements problem, and getting consensus on at least the scenario level about exactly what’s going into the iteration. Having the index cards arrayed on a task board, with each index card showing the status (“planned”, “in development”, “completed”, “out of scope”) gives a lot more visibility into exactly what you’re building and how you’re building it. In a lot of ways, this is a sort of iterative development project plan.

Another way to prevent iteration-related scope problems is committing to delivering releasable code at the end of each iteration. Test-driven development (or, at least, developing complete unit tests) and continuous integration are effective ways to help make that happen. If your stakeholders are comfortable that you’ll deliver high-quality code at each iteration, they’ll feel less pressure to get the new features in immediately, and will be more willing to wait until the next iteration.

If this sounds like something that might help you, I definitely recommend reading the interview Jenny and I did with Mike Cohn for Beautiful Teams, who knows more about agile and iterative planning than pretty much everyone . He has a lot to say about effective iteration planning. There are pictures of task boards he’s used in the past.

That gets me back to the basic idea—one which I give Mike a whole lot of credit for helping people understand—that iterative development, especially in an Agile project, works best when we take the time to plan each iteration. It still faces the same problems: requirements need to be discovered, scope needs to be controlled, and progress needs to be communicated to everyone who cares about it… especially to anyone who can potentially make the developers’ lives more difficult. That’s why the most successful Agile development projects collect requirements and document them using user stories (or other techniques for writing down what the software needs to do). They plan their progress using task boards, forecast them using calculations and charts like project velocity and burn down rates, and constantly keep any business owners and other stakeholders up to date on their progress.

It’s a great way to develop software, and it’s been really effective for a lot of teams. And I think it shows that iterative development does not necessarily mean unplanned development.

When I sent this to the developer who sent me the question, he had an interesting follow-up, which I thought deserved a response:

So I’ve been digesting this a bit and I am curious to get your thoughts about adapting project management fundamentals into the often fluid process of app management. I manage a few virtual projects within my company and at times have struggled to keep things focused as business demands and/or interests shift over time. Similarly, the “iterative” approach has helped to clarify requirements while building out new flows/apps, but as you pointed out can be very tricky to get “right”.

It’s funny how often I hear people say, “Well, this project management stuff works in theory, but my project is fluid” (or “changing,” or “under too much pressure from the business,” or “critical,” etc.). It turns out that pretty much every project is challenging, and project management is set up specifically to deal with that kind of challenging project.

Here are two thoughts I had relating to this idea.

First, the iterative development model works very well, as long as you’re committed to delivering a high-quality product at the end of each iteration. Whether or not you develop using an iterative approach, you need to manage change: prevent unnecessary changes, and make sure you understand the impact of any change that you make. It also means that you need some sort of scope baseline, so that you know what is and is not a change. It’s faster and easier to update software on paper, before it’s written into code, so the more changes you can move to the “write it down and review it” phase of your project, the better.

And second, if your business is overly demanding, it often means that you could manage your stakeholders’ expectations better. Make sure you identify them – and write down their names and needs! – from the very beginning of the project. Talk to them… a lot. Make sure they’re in the loop. If possible, see them so often that they’re sick of seeing you. If they’re always aware of what you’re doing, there won’t be nearly as many surprises. Also, the more your stakeholders understand the details of the work that you’re doing, the more slack they’ll cut you when they ask you for changes. Often, when someone puts a lot of pressure on you to do the impossible, it’s because they don’t realize that’s what they’re asking.

Here’s a question that I get over and over again:

What’s the difference between user stories and use cases?

— Ron K.

Before I dive into an answer to that question, let’s rewind a little bit and talk about where user stories came from. I like them because they’re a great example of how the Agile movement changed the software world. Programmers used to just dive right into software projects and start coding. Whenever one of those pesky users started to tell us what they needed, we’d stop them and say something like, “Don’t worry, I totally get it. I know what you need.” The Agile folks figured out that “I know what you need” is a nasty little trap that programmers — especially good ones — fall into. We’d spend the whole project thinking that we understood our users’ needs, only to deliver software that they didn’t want. The Agile folks realized that if developers had to start working with users throughout the project to understand their needs if they wanted to avoid the code-and-fix trap.

Jenny and I teach you all about use cases and requirements in our first book, Applied Software Project Management (O'Reilly, 2005).

And that’s why I think the user story is one of the most useful tools to come out of the Agile movement. A user story — some people call it a scenario — expresses one very specific need that a user has. It’s usually written out as a couple of sentences. Most user stories are written in the language of the users, so any user should be able to read a user story and immediately understand what it means. A lot of time, user stories are written on index cards, although I’ve put them in Word documents, Excel spreadsheets and Wiki pages (depending on how the particular project is run).

A use case is similar to a user story, because it also describes one specific interaction between the user and the software. When I’m training people to improve the way they write down their project’s requirements, I often describe the use case as a “deceptively simple tool” that helps us find and write down all of the ways users interact with the software.

Looking at those definitions, I can definitely see why there’s confusion about the difference between user stories and use cases. If you look at the last two paragraphs, it might sound like I said the same thing twice! But while user stories and use cases are definitely similar, there are important differences between them. Each serves a distinct purpose, and I think they both have their place on a well-run software project.

I think the easiest way to understand the difference between use cases and user stories is to take a look at an example. Luckily, I’ve got one that I think helps make the difference clearer.

In our first book, Applied Software Project Management, Jenny and I spend a lot of time talking about how to develop use cases and use them to build better software. And as an example, we showed a use case for a software feature that everyone should be familiar with: a search and replace feature from a word processor. Comparing a user story for search and replace with a use case for the same feature helps highlight the differences.

It’s not hard to find lots of user story examples. There are lots of different ways you’ll see a user story formatted (although if you’re looking for a user story template, a 3×5 index card should be a good starting point!). So what would a user story for search and replace look like? I took a stab at writing one:

(One thing I like to do with user stories is to use “he” or “she”, rather than try to be gender-neutral. I think this makes the user in the story easier to connect with by personifying him a bit. It it also lets me write in a more conversational tone, which makes the user story friendlier and, I think, a bit easier to read and understand.)

Now, if you’re not familiar with user stories, you might think to yourself, “Wait a minute, my word processor’s search and replace feature does a lot more than that!” And that’s okay. A typical user story will have enough information to help the user understand what it is the software needs to accomplish, but it’s not meant to be a complete description of how the software works. I’m not going to try to give a long lesson in writing effective user stories here; I highly recommend reading Mike Cohn’s excellent articles and posts aboout user stories. (Mike, incidentally, is one of the software development veterans who contributed to our latest book, Beautiful Teams [O'Reilly, 2009]. He has some really fascinating things to say about Agile planning.)

So what would a use case sample look like for search and replace? Here’s the use case example Jenny and I built to demonstrate how use cases work:

Now, if I were a developer building a word processor or text editor, I’d actually be able to write a search and replace feature that implements that particular use case. (Just to be clear: there are many different use case formats; Jenny and I use this use case template in our book because it’s stripped down to the bare minimum sections that we think an effective use case should have.)

Here’s something about use cases that I think is interesting. While you were reading through our use case example, were you thinking of something that looks like the Replace dialog in Notepad or Microsoft Word, or the Find dialog in TextEdit? If so, take another look at the sample use case. It doesn’t have any words like “window,” “button,” “click,” “field” or “checkbox”. It’s all about what actions the user takes, and how the software responds. And there are many different ways that you could build software that implements the use case. Have you ever used the search and replace feature in vi? What about the search and replace feature in Emacs? They have very different user interfaces! Who knew there were so many ways you could implement search and replace? But if you compare each of them with this use case, they all follow the same basic course of events.

So now that we’ve gone through the use case and user story examples, what’s the difference between user stories and use cases? Here’s what I think are some of the key differences:

• User stories are about needs. When you write a user story, what you’re describing is a “raw” user need. It’s something that the user needs to do in his day-to-day job. If you never build any software for him, then that need will still exist!
• Use cases are about the behavior you’ll build into the software to meet those needs. A developer who needs to build working software should be able to read a use case and get a good sense of what the software needs to do. It typically has a lot of detail, and describes everything that the developer needs to build in order to meet the user’s need. That’s why it needs to have a lot more detail, and be clear and unambiguous.
• User stories are easy for users to read. When you write a user story, what you’re concentrating on is writing something that anyone can understand, in the language of the users. We all know that developers have a lot more patience for talking about details of the software they’re building than users do, which is why user stories have to be brief. A user story needs to express a complete thought in just a couple of sentences. (That’s also why it’s good to put them on index cards: somehow, that makes it clearer that it’s self-contained and independent of the other user stories.)
• User cases describe a complete interaction between the software and users (and possibly other systems). When you’re doing use case analysis, what you’re doing is designing a functional solution that meets the users’ needs. It needs to be something that developers can implement. It’s possible that one user story could spawn several use cases. And when you combine all of your use cases into one use case document, you’ll end up with a complete description of every interaction between the user and the software that you’re planning on building. And if your software has to interact with multiple systems, you may end up treating those other systems as actors in your use case.

Once you get a sense of how user stories and use cases differ, you can start to see what purpose they can serve on your project. And if you only use user stories, or if you only use cases, then maybe on your next project you can try using them both.

HF PMP: how does it fit with Lean Software Development…

Gian Franco asked:

How does PMP fit with practices as explained in Lean Software Development (by Poppendieck) or Agile practices in general?

I wrote a blog post about this a while back called “What About Agile?”

http://www.stellman-greene.com/2007/04/27/what-about-agile/

There are definitely a lot of things that the PMBOK(r) Guide and PMP exam cover that aren’t addressed at all with Agile — like whether a fixed price contract is better or worse for the seller than a cost-plus contract. But there’s one really important way that both the PMBOK(r) and Agile are very similar: they both recognize that managing change is an important part of a successful project. And in that respect, you’ll definitely find that your knowledge of Agile can help you when you study for the PMP exam.

And Gian had a follow-up question:

Well :), let me put it in another way…, In Lean SW development requirements churn is considered to increase costs, I don’t know much about PMP, but suppose it resembles to Prince2 (another comparable projectmanagement method) then the initial phases of specification might fall in this category of specifying too much too early.

The PMBOK(r) Guide was developed so that it can be applied to any kind of project, not just a software project. So projects that fit into its framework tend to develop all of the requirements up front, before any work begins. That’s because all of the plans for the submarine or office building need to be finalized and agreed upon before you hire the construction crew to break ground.

There will be changes that happen. But if those changes involve very large alterations to the blueprints and you need to tear out the last three floors you put in, it gets very, very expensive.

Oddly, the same is true of software in many cases. There are some changes that need to be made, and which you could have found had you “churned” through the requirements a little more before you started building the software, and now you’ve got a bunch of code you need to tear out — and it would have been a lot more efficient to take the time up front to figure out the requirements and then only build your software once. Unfortunately, many people don’t consider that “Agile”. (I think they’re wrong — I think that doing a lot of iteration before you even start writing code can be the most efficient and customer-focused way that you can build software… but that’s a story for another day.)

HF PMP: PMP is Agile!

Darya Akbari asked:

Can one say that PMP is not agile :) ?

That’s an interesting question. But I’d ask the opposite question: can an Agile shop fit into the PMBOK(r) Guide framework?

The reason is that the PMBOK(r) Guide doesn’t define one specific set of doing things. In fact, just the opposite. It says that you need to select a particular methodology based on what you know about your industry and past projects, the specific needs of the project, the deadline and milestones, etc. It doesn’t say that every project needs to include specific things. Instead, it includes things that typically happen on most projects — and practices that are most commonly found. And remember, the PMBOK(r) Guide doesn’t just apply to software: it needs to be general enough so that it can include practices for construction, industrial, civil engineering, electrical and other kinds of projects.

So can a typical Agile process fit into the PMBOK(r) Guide? As far as I can tell, the answer is yes. One hint is that when Jenny and I were working on “Head First PMP”, the PMBOK(r) Guide team members on our technical review team repeatedly stressed iteration and iterative development.

One of the strongest points in the PMBOK(r) Guide (ones that is stressed on the PMP exam) is that it really emphasizes collaboration with the stakeholders, and keeping them in the loop on all important decisions. Another thing that it really stresses is responding to change — and it’s very clear that the customers need to be involved in decisions about change.

There are definitely some things that Agile people might not agree with. It may seem very documentation-heavy, and very concerned with contracts. But the PMBOK(r) Guide was developed in a world where subcontracting is very important, and where a lack of documentation or attention to the contract can mean that the company can get sued and go out of business. I’ve spent a lot of time working in a consulting situation, and even the friendliest clients can turn into adversaries if you don’t have everything documented properly. But if you go back to the Agile manifesto, you’ll see “Individuals and interactions over processes and tools,” “Customer collaboration over contract negotiation,” and “Working software over comprehensive documentation”. While the PMBOK(r) Guide highly stresses individuals and interactions, customer collaboration, and working software (in the form of deliverables that meet the customer’s needs). But it needs to pay attention to processes and tools (since that’s what a framework is made of), contract negotiation (because a process is no good if it puts your company out of business), and comprehensive documentation (because it’s really hard to build a strip mall or highway overpass without it).

HF PMP question: SCRUM?

Rogerio Kioshi asked:

I’ve read about SCRUM. Does PMP have anything to do with this metodology?

The PMP exam won’t have any questions specifically about SCRUM. But if you have a good understanding of SCRUM, then that will give you a good leg up on studying for the PMP exam. The reason for this is that if you’ve spent time thinking about SCRUM, then you probably have a good understanding of a lot of the ideas behind team communication, project schedule constraints, and activity sequencing, and those are core concepts that you need to understand to become PMP certified.

You’ll still have studying to do, though!

Need advice on HOW to start a new Software Product

This isn’t a question about Agile specifically, but it’s still a great question. Paul Michael Laborte asked:

I hope this doesn’t sound too off topic. Anyway here it goes…

I’m currently a software developer with a Monday to Friday job.
Right now, I’m looking for opportunities which would allow me to have a 20% time (pretty much like google).

During that 20%, I’m thinking of developing my own software product.
I would be needing to wear different hats so I really think HFPMP would be of great value.

Aside from the contents of the book, would you have any other tips for entrepreneur aspirants like us

That’s a really interesting question. How do you start out a software project?

Well, if you want to get your project started out right, the first thing you need to do is figure out what it is you want to build. That might sound a little odd — of course you know what to build, right? Otherwise, why would you start a project? But if you look at a lot of projects that went off the rails at one point, one thing that you’ll see over and over again is that many problems can be traced back to the fact that one person wanted to build A, while another person wanted to build B.

There are a lot of tools and practices that can help you with this. My favorite is a Vision & Scope document — mostly because it’s very lightweight, only takes a few minutes to write, and it’s something that anyone can read if they want to learn what it is your project does. Also, it’s something that serves really well as a front page for a wiki or project website, and immediately brings people up to speed on it. Basically, the Vision & Scope tells you very briefly who needs the software, what their needs are, and how you’ll meet those needs (by explaining the features of the software you’ll be developing).

To be perfectly honest, Head First PMP may not really help you as much with this particular problem. But it’s something that Jenny and I wrote a whole lot about in our first book, Applied Software Project Management. We have a whole chapter on starting out a project using a Vision & Scope document.

Other things you need to think about when you’re starting out your project — which we also talk about in depth in Applied Software Project Management — are figuring out how your users are going to interact with the software, setting up a version control system, and doing test-driven development.

I definitely recommend taking an hour or two and really think about how you’ll handle those things. One way I’ve seen a lot of people do this is start a wiki for your project, and have a separate page that says how each of those things will be handled. It doesn’t have to be fancy or anything, and it shouldn’t take long to throw together. But just doing that will really help get your head straight about them, and set your project in the right direction from the beginning.

If you’ve got a burning question and you’re wondering what we think about it, send it to us using our “Contact Us” page! We love answering questions from our readers.