Don’t document your process!

Joe perfects his process for the perfect date

Yesterday, a Slashdot article asked an age-old question:

One of the worst problems [in my company] is a lack of process documentation. All knowledge is passed down via an oral tradition. Someone gets hit by a bus and that knowledge is lost forevermore. Now I know what I’ve seen in the past. There’s the big-binder-of-crap-no-one-reads method, usually used in conjunction with nobody-updates-this-crap-so-it’s-useless-anyway approach. I’ve been hearing good things about company wikis, and mixed reviews about Sharepoint and its intranet capabilities. And yes, I know that this is all a waste of time if there’s no follow-through from management. But assuming that the required support is there, how do you guys do process documentation?

This question seems to come up over and over again. The funny thing is that it almost always leads straight to a long discussion of the technique for gathering process documentation, and then a discussion of mechanism for storing it. That’s the question I think the reader thought he was asking: how to “copy down” the process by looking at how the people on the team build the software and putting it into a “complete” set of documents, and whether to use a wiki, Sharepoint, a version control system or some other repository to hold the docuemnts. And the discussion that followed from that Slashdot post should be pretty familiar to anyone who’s tried to solve this problem in real life. Some people talk about systems to store documents, others talk about the virtues of keeping them up to date, there’s a healthy dose of “write down what you’re currently doing” or suggestions for incident logging, an apparent epidemic of bus drivers who have it in for the one guy who knows how everything in the company works, and lots of talk of cataloging, updating and verifying.

I’ve been through that before, and I’ve bought into many of those ideas in the past. And you know what? It wasn’t particularly useful. I know that in many process circles, that’s a heretical idea. In fact, if you’d told me back in 1999 that it’s not useful, I would have laughed at you. Of course you’re supposed to start by documenting the complete process! How else do you know what you’re improving? There seems to be an unspoken rule that we’re supposed to be striving for a fully documented, constantly improving process. And in some shops, that does make sense. But it’s a very, very hard thing to do, and in practice it’s almost impossible to put in place from the ground up.

This is a really hard thing for a lot of software people to accept. Our nature, as programmer types, is to strive for complete systems. When we build software, we try to handle every possible special case. We’re overly pedantic and literal; it’s like exceptions and missing cases just get under our skin. So when we’re presented with the problem of how to improve the way a team or a company builds software, the first thing we want to do is come up with a system that describes the complete process for building software, mapping out every possible special case and exception that a project might come to. And it makes sense that we’d want to test that process to make sure it’s accurate, correct any changes, and put everything in a repository that gives us complete access to the one true way that we build software.

The problem is that people don’t really work that way. Process engineering suffers from a serious problem: it seems simple when you think about it in the abstract, but once you start trying to document precisely and completely how a team builds software, you run into an enormous number of special cases. Architecture is always finished and signed off before coding begins, right? Oh, wait, except for Joe’s project, where we did 30% of the architecture and started building the code for that while Louise and Bob worked on the next piece of architecture. Oh yeah, and then there’s that project that’s going to be broken into three phases, and we don’t really know how the third part is going to work.

I’ve seen that pattern many times, and it usually plays out in one of two ways. Either you end up with really general documentation that lays out a very general process that’s trivial to follow but doesn’t really provide any useful guidance (like a big chart that shows that testing comes after coding, which comes after design), or you end up with a tangled mess of special cases and alternative paths that seems to get updated every time there’s a new project. Both of those technically fulfill the goal of process documentation — which is great if your job was to document the process. But neither is particularly useful if your goal is to actually build better software.

There’s an easy solution to this problem: don’t document your software process. Or, at least, don’t start out by documenting the complete software process. Instead, take a step back and try to figure out what problems you’re facing. What about your process needs to be fixed? Do you have too many bugs? Do you deliver the software too late? Does your CFO complain that projects are too expensive? Do you deliver a build to your users, only to have them tell you that it looks fine and all, but wasn’t it supposed to do this other thing? Those are all different problems, and they have different solutions.

That’s what Jenny and I teach in our first book, Applied Software Project Management. We call it the “diagnose and fix” approach: first you figure out what problems are plaguing your projects, and then you put in very limited fixes that address the most painful problems that keep you from building better software. People don’t just wake up one day and say, “We’ve got to totally change the way we build software.” They don’t start documenting the software process because things are going just fine. People hate change, and they don’t start making changes to the way they build software unless they have a good reason. So look for that reason, find the pain that hurts the most, and make the smallest change that you can to fix that one problem without rocking the boat. Then find the next most painful thing, and put in the smallest change that you can to fix that. This is something you can keep doing indefinitely, in a way that doesn’t disrupt those parts of your projects that are working just fine. Because the odds are that there are plenty of things that the team is doing right! If it ain’t broke, don’t fix it.

So what about the question of how to actually document the process changes that you do want to make? That’s a very practical problem, and one that we had to handle in our book. After all, we do give you processes for planning, estimating, documenting, building and testing software. And we wanted to do it in a way that was programmer-friendly, with as little cognitive overhead as we possible.

We decided to use process scripts — that’s scripts like an actor reads, not scripts like a shell runs — to describe our processes. We developed these scripts based on use cases (which we talk about in detail in the book). If you take a look at the use case page from the book’s companion website, you can see an example of a use case, followed by a typical script that you’d follow to develop use cases for your project. That particular script is very iterative, because use case development (like many great software practices) should be a highly iterative process. We’ve got examples of many scripts for the various practices and processes: ones for planning projects, reviewing deliverables, and building and testing software.

As for storing these scripts, I’ve used all sorts of ways to do it in the past. I’ve used wikis, version control systems (both Subversion and VSS, depending on what’s in place at the company), even plain old folders full of MS Word documents. The actual mechanics of storing documents aren’t particularly interesting, and are pretty much interchangeable for process documentation. Processes shouldn’t change all that often, because change is very disruptive to a company. The changes should be small, incremental, and easily understood by the team… and the team should agree that they’re useful! Because the biggest problem with process changes — and several posts in the Slashdot thread bring this up — is that they don’t “stick”. But making those changes stick is easy. Just make small changes that the team buys into, and that actually get you to build better software.

That’s easier said than done, of course. Lucky for us, too! Otherwise there wouldn’t be a market for our books or training.

Speaking, training and writing

Training saves your kneecaps

We’ve been keeping ourselves busy! What’s that? You want to know more? Well, certainly. We’ve got lots of news:

  • Jenny and I are doing some guest blogging on the Head First Labs website, talking about what it’s like writing a Head First book (and whatever else we feel like talking about). I’ll be doing the posts this week, starting with one called “How We Made Head First C# Different”. I’ll probably get a little more technical near the end of the week — there’s only so much anyone wants to read about writing books. (Or is there?)
  • We’ve put up a new training page, because we’ve been getting a lot of questions about training. It’s a list of the various courses we offer on project management and software development. Right now, we’re mainly concentrating on training corporate teams — we’ll go into a company and do a few days of training for a team. We’ve been getting an increasing number of inquiries about putting together classes that are open to the public, though. If you’re interested in that, please drop us a note using our contacts page and we’ll let you know the next time we’re offering one.
  • Last week we were invited to do our “Why Projects Fail” talk for the PMINYC Breakfast Roundtable. After the talk, one of the audience members came up to me to thank us for doing a presentation that wasn’t boring. I thought that was pretty cool! Anyway, here are the slides from the talk.
  • We’ve been doing our “Why Projects Fail” talk at companies around town. If you work at a company in New York City and want some insight into why projects fail, you’ve got a brown-bag lunch program (or some other kind of program where your company brings speakers in to do a talk), and you can get a reasonably-sized audience together, get in touch with us — we’re usually happy to come in and do it as part of our New York outreach program. It’s generally pretty fun, and a good way to take your mind off of the job for an hour.

2007-10-09 presentation screenshot

Whoa…

Whoa!

Well, we didn’t see this coming. Check out what Tim O’Reilly says in his latest blog post:

I was probably most surprised when I saw Programming WCF Services on our list of top performing books for the year. If you’re steeped in open source, you might never have heard of Windows Communications Foundation, Microsoft’s approach to building SOA applications on Windows. And you might not care. But you’d be making a mistake. Don’t count Microsoft out of the Web services game yet! They still have a brilliant, passionate developer community, and as a company have tremendous resources, persistence, and talent. And now that they have real competition, I expect them to reinvent themselves. (For that matter, Head First C# was the top selling programming language title in Bookscan last week, except for “Javascript: The Definitive Guide”. And C# continues to gain significantly on Java in terms of book sales.)

Wait, what? Let me repeat that for you. Except for a book on Javascript, Head First C# was the top selling programming language title in Bookscan last week. That’s pretty amazing to me, because we only released it at the end of November, so word of mouth hasn’t even gotten a chance to spread yet.

We took a few risks with Head First C# — we took an approach that was somewhat different than any other programming book I’ve seen. I’ve learned at least a dozen and a half programming languages over the years, and I followed the same pattern every time: I thought of a project that I wanted to do, then I figured out how to do the project using the new language. (Like back in 1994 or so, when I wanted to learn Perl and also learn about web servers, so I built a web server in Perl.) But I’m definitely not unique in this approach: pretty much every good programmer I know takes the same approach to learning a new language or technology.

Which is why I thought it was so weird that I couldn’t even find a language learning book that asked you to solve programming problems. I’ve seen a lot of “hands-on” work, where you follow step-by-step instructions to type in code to get a certain result. And certainly, an enterprising person can definitely learn that way. (If they couldn’t, nobody would buy programming books!) And certainly, we have a lot of that in Head First C#. But I just don’t think that’s enough.

I guess I have a hidden agenda here. I’ve led a lot of programming teams over the years. In 2006 and 2007, I spent a lot of time working with a consulting company, and one thing you learn working with consulting teams is that you need to work with a lot of different people. Often, you’ll have at least one very junior member on a team. And what I’ve found is that a lot of really junior developers have the capacity to be good programmers, but don’t have any experience independently solving programming projects on our own. So I decided early on that one goal of Head First C# would be to give people that experience. I wanted to teach them how to solve programming problems, rather than just follow step-by-step instructions. In essence, I wanted to create programmers who I’d be happy to have on my team in the future. (Does that sound too self-serving? I hope not!)

So when Jenny and I started working on Head First C#, it took us a few chapters to figure out exactly how to do that. It was interesting going back to the beginning of the book after we finished, to do a second pass and incorporate comments from our (really great, and really patient) tech reviewers. It really did take us about five or six chapters to work out a good way of assigning problems. We ended up co-opting the “Exercise” element that you’ll find in the other Head First books. The Lose Weight Exercises in the other Head First books were generally pencil-and-paper oriented, although I think I remember a few step-by-step programming ones as well. That’s definitely the approach we took in Head First PMP, and it worked really well.

But with Head First C#, we took the “Exercise” element in a totally different direction. We basically used the Lose Weight Exercise to give a programming assignment, where we’d give a problem to solve. We’d include a screenshot, define some classes, maybe give a little of the code, and then leave it up to the reader to build working software. It was lucky that I spent so much of my career writing specifications, because I think we really had to push our writing skills to the limit. It’s far too easy to write an Lose Weight Exercise that’s ambiguous or hard to follow… and as we found out during the tech review, it’s really demoralizing to run across a poorly-written Lose Weight Exercise. And we had to be clear that there’s no single, “correct” solution to the exericse: if it works, then you got it right. I’d be surprised if a single reader comes up with exactly the same solution that we did on any of the projects in the last third of the book.

I hope that interactive approach is what’s paying off. Word of mouth is just starting to get out, and I’m really happy to say that it seems to be positive.