Software estimation considered harmful?

A few weeks ago I was in the midst of reading Steve McConnell’s Software Estimation: Demystifying the Black Art when I had the conversation with my friend Marc that I have written about previously. Marc, as I mentioned before, is the founder of and Chief Product Officer at a small startup called Wesabe. When I told him what I was reading, he asked, “Do you believe it?”

“Sure,” I said. By which I meant nothing in the book had struck me as patently bogus. A lot of it is good sense about the limits of estimation, the relation of estimation to planning, and why estimation is so hard. Parts II and III of the book present specific estimation techniques that, assuming one had the relevant inputs and historical data, seem likely to be capable of producing fairly accurate estimates. Now, the descriptions of some of these techniques made me think — wow, if that’s what it takes to produce good estimates, no wonder we all muck along with crappy ones. But it did for the most part seem like the kind of thing we Serious Software Professionals™ should be doing.

Marc — it turns out — is far more skeptical about the whole enterprise of software estimation. He tells me that at Wesabe they never make schedule estimates. He manages his developers, as he has explained in a blog entry, by trying to get them excited enough about the features he thinks should be added to Wesabe that they decide to go ahead and add them. Or they may get excited about their own ideas and add them instead. Marc retains final authority over what gets added to the product and a Wesabe developer who consistently gets excited about developing things that Marc refuses to allow into the product should probably make sure their resume is up to date. But he never asks them for estimates. He does encourage his developers to spend most of their time working on things that they can finish quickly and get into the product, but that seems to be as much about what he thinks most likely to keep his developers happy as anything else. When they need to, Wesabe developers will tackle bigger projects, still without estimating how long they will take. Marc’s point of view, I take it, is that the only reason to do these big projects is because you have to, and if you have to, it doesn’t really matter how long it’s going to take.

Now, Marc’s a smart guy and he’s been managing software developers for as long as I’ve known him (more than a decade) and I know he thinks a lot about how he does what he does. On the other hand, Steve McConnell’s also a smart guy whose books I’ve been a big fan of for about as long. So, how to reconcile these two points of view? Is Marc’s approach only tenable in a startup? Or maybe McConnell’s approach to estimation is only worthwhile in big organizations, that are doing more or less the same thing over and over again.

So I returned to reading Software Estimation with a new question in mind: Why estimate? The nearest McConnell comes to an answer is in section 1.7 “Estimation’s Real Purpose”, where he gives this definition of a good estimate:

A good estimate is an estimate that provides a clear enough view of the project reality to allow the project leadership to make good decisions about how to control the project to hit its targets.

I think the key word in McConnell’s definition is “targets”. The reason Marc can get away with not estimating is because he’s found a way to manage without setting targets. So the question “Why estimate?” is better expressed as “Why set targets?”

Sometimes we set targets in order to convince others, or ourselves, that something can be done. We may set targets to inspire ourselves to do more, though it’s not clear that’s a winning move, and even less so when managers set a target to “inspire” the folks who work for them. (See DeMarco and Lister’s Peopleware, chapter 3 and the discussion of Spanish Theory managers.) We may also set targets to give ourselves a feeling of control over the future, illusory though that feeling may be. After the fact, a target hit or missed can tell us whether or not we did what we set out to do. However if we missed a target, we can’t know whether that’s because the target was unrealistic or because we didn’t perform as well as we should have. Setting and hitting targets does make it look like we know what we’re doing but we need to keep in mind that targets rarely encompass all the things we care about — it’s much easier to set a target date for delivering software than a target for how much users will love it.

If the goal is simply to develop as much software as we can per unit time, estimates (and thus targets), may be a bad idea. In chapter 4 of Peopleware, DeMarco and Lister discuss a study done in 1985 by researchers at the University of New South Wales. According to Peopleware the study analyzed 103 actual industrial programming projects and assigned each project a value on a “weighted metric of productivity”. They then compared the average productivity scores of projects grouped by how the projects’ estimates were arrived at. They found that, as folks had long suspected, that programmers are more productive when working against their own estimates as opposed to estimates created by their boss or even estimates created jointly with their boss, averaging 8.0 on the productivity metric for programmer-estimated projects vs 6.6 and 7.8 for boss-estimated and jointly-estimated. The study also found that on projects where estimates were made by third-party system analysts the average productivity was even higher, 9.5. This last result was a bit of a surprise, ruling out the theory that programmers are more productive when trying to meet their own own estimates because they have more vested in them. But the real surprise was that the highest average productivity, with a score of 12.0, was on those projects that didn’t estimate at all.

There is, however, one other reason to estimate: to coordinate our work with others. The marketing department would like the developers to estimate what features will be included in the next release so they can get to work writing promotional materials. Or one developer wants to know when a feature another developer is working on will be ready so he can plan his own work that depends on it. Note however, that in cases like this, estimates are really just a tool for communication. Marketing needs to know what’s going to end up in the release and the developers, by virtue of being the ones building it, have the information and somehow that information has to be communicated from the developers to the marketers. But there are lots of ways that could happen. In a small company it might happen for free — everyone knows what everyone else is working on so the marketers will have as good an idea as anyone what’s actually going to be ready for the release. If water-cooler conversations are insufficient, then marketing and development could meet to talk about it on a regular basis. Or the developers could maintain an internal wiki about what’s going on in development. Some of these methods may work better than others but it’s not a given that using estimates is always the best way.

To decide whether estimates are a good way to communicate, we need a way to compare different methods of communication. I’d argue that all methods of communication can be modeled, for our present purposes, with the following five parameters:

• Preparation time required
• Communication time required
• Rate at which information is conveyed
• Rate at which misinformation is conveyed
• Other benefits and costs

The idea is that communication happens in this pattern: one or more people spend some amount of time preparing to communicate. This would include activities such as thinking, writing, estimating, etc. Then the communication proper happens, which takes some time. This may require time from both the sender and the receiver (conversations, meetings, presentations, etc.) or only the receiver (reading an email, looking at a web site).

After the communication is complete, some amount of information has been conveyed and also, sadly, some amount of misinformation. The misinformation may arise from faulty preparation, from misstatements by the sender, or from misunderstandings by the receiver. Obviously different methods of communication will be able to convey more or less information in a given amount of time and will be more or less prone to miscommunication.

Finally, different methods of communication can have benefits beyond the information conveyed and costs other than the time spent and the misinformation conveyed. For instance, chatting around the water-cooler may build team cohesion while highly contentious meetings may have the opposite effect. Another important kind of benefit is that the preparation and communication phases may itself improve the communicators’ understanding of what they are communicating about. For example, writing clearly on any topic invariably forces the writer to clarify their own thoughts and the give and take in a well-run meeting can help bring out and refine what were previously only amorphous notions.

Now we can look at estimation as a communication tool according to this model. The preparation time for good estimates is fairly high. This is why so many developers try to avoid estimating or give it short shrift. The communication time is low — since an estimate distills things to the essence of “these features by this date” or “this much effort for that much functionality” it can be quickly conveyed. However, exactly because estimates do distill things, they are a low bandwidth form of communication. Without a lot of other communication about what exactly the features are going to look like, a list of features and the dates when they will be done, leaves out a lot. Worse yet, estimates are notoriously prone to conveying misinformation, either because the estimate is inaccurate or because an accurate estimate is misunderstood. McConnell devotes a whole chapter, “Estimate Presentation Styles”, to discussing how to present estimates so they will be less likely to be misunderstood, but no presentation style can help the poor estimator who’s giving an estimate to an audience that hears “2 to 4 weeks” as “2 weeks”.

When it comes to other costs and benefits, it’s hard to say. If we assume for the sake of argument that it is possible to make good estimates, does the very act of preparing the estimate have its own benefits? Certainly, making a good estimate requires identifying all the work to be done, so a team that has gone through the exercise of estimating may identify all the work that actually needs to be done sooner than a team that hasn’t, which may allow the work to be done more efficiently. Another potential consideration is that in a company where non-developers expect developers to be able to provide reliable estimates, then meeting those expectations has the social benefit of giving the rest of the company confidence in their development team.

On the other hand, there’s no guarantee of obtaining those benefits. Estimating badly certainly has bad ancillary costs. Leaving aside the consequences in terms of the misinformation it generates, it’s just no fun to estimate when you know your estimates are bad. Either you feel guilty because you believe it’s possible to do well and that you should be better at it or you think it’s actually impossible and therefore whoever is asking you for the estimate is wasting your time. And if good estimates can increase the company’s confidence in their developers, bad estimating can destroy it. Or, worse yet, developers can be unfairly deemed unreliable because the rest of the company expects more precision in estimation than is actually possible. According to McConnell, at the beginning of a project even the best estimators can be up to 4x off in either direction. Which means unless folks can accept an initial estimate of “from three months to four years” and a promise to refine it over time, developers are bound to disappoint.

So we have: High preparation cost. Low communication costs. Low bandwidth with high potential for misinformation. Ambiguous ancillary costs and benefits. Clearly, if you’re going to estimate, you’d better do it well and accept the costs that entails. But if you doubt it’s possible to do well, either in principle, or in your particular situation, then it can be useful to think about other ways to make sure the necessary communication happens.

Advertisement

Software delivery on the web?

The other day I was talking to my friend Marc about software development. Marc is founder and Chief Product Officer of Wesabe a startup that, “makes it easy to better understand how you spend your money and links you to a community of people dedicated to helping each other make better financial decisions.” Wesabe is one of these new-fangled companies whose whole product is their web site, which they update, according to Marc, every day. The frequency of their updates came up when I was using version control as an example of something that developers need to understand in order to work well in groups — not just the mechanics of a particular version control system, but the how and why of things like release branches. Marc suggested that my thinking was way out of date — that because everyone is delivering their software on the web these days, release branches were an anachronism. He’s certainly right that if you release your software by updating your own web site then there’s no reason to make release branches — the point of release branches is to allow you to go back and fix bugs in old releases and if you’re delivering your software as a service there’s only one release and no direction to go but forward.

My question is: what percentage of software startups these days are planning to deliver their software as a service vs “traditional” delivery of software to be run by customers? Is Marc really right that the software-as-service model is so prevalent now that a book aimed at developers who want to learn the things they need to know to be productive members of a programming team can ignore older styles of delivering software?

Another question is, even if Marc is right that everyone is delivering their software as a service, to what extent they will be able to avoid having to support older versions of their software? It’s one thing for small companies, still in early days of fleshing out their feature sets, to continually upgrade and force their users to come along for the ride. But as they get more users and a more mature feature set, I suspect they will discover that not all their users will be willing to accept upgrades willy nilly. Presumably this will be less of an issue for companies whose real value is the social network of the users as opposed to the value being in the raw functionality of the software itself. In the former case the whole point is to have exactly one instance of the software, shared by all users while in the latter case, if the users are depending on particular functionality, they may not want it to change, even if the change is ostensibly an “upgrade”.

Have an opinion? Let me know.

Update: My former boss Bob Pasker, now a Venture Advisor and CTO-in-Residence at Azure Capital Partners, gives me this breakdown of the software delivery mechanism of companies he’s seen recently in his work at Azure:

• 50% delivered over the web, either as HTML or flash
• 25% require an install, either on server or on desktop
• 25% other (mobile, email, hardware, or both)

How to write a book

In my first entry on this blog I mentioned the Gigamonkey Four-Step Algorithm for Writing a Book. In the past week, I’ve twice had occasion to explain this algorithm to someone. Rather than wait for a third occasion, I figured I’d write it down. Here it is.

Step 1 Write the index. Write down, in no particular order, every word, phrase, name, and concept that you would expect to appear in a well-prepared index of your book, assuming the darn thing was already written.

Step 2 Write a hierarchical outline. This outline should contain, somewhere, all the items from your index. Follow the rule from junior-high essay writing that you can only create a sub-topic under a topic if it will have at least one sibling. But don’t worry if your outline gets ridiculously deeply nested. If you’re a forest-then-trees kind of person you can build your outline top down — define the top-level topics you want to talk about, then split those topics into sub-topics and those sub-topics into sub-sub-topics, and so on until you get down to a granularity where you can start inserting items from the index. Or you can work bottom-up — start directly from your index, lumping related items together, then lumping the lumps into bigger lumps. I found that alternating between top-down and bottom-up worked best for me.

Step 3 Write a a flattened outline. Books are a linear medium. While it can be immensely useful to organize your thoughts hierarchically, you are ultimately going to have to guide your reader from A to Z along some linear path. In his essay The Cognitive Style of PowerPoint Edward Tufte observes:

People have communicated about complex matters for centuries without hierarchical bullet lists. Richard Feynman wrote about much of physics — mechanics, optics, thermodynamics, quantum behavior — in a 600-page book with only 2 levels: chapters and headings within chapters. (Emphasis in original.)

Following Feynman’s lead, in this step you should produce on outline with no more than three-levels: chapters, sections, and paragraphs. Each paragraph-level outline item should be — again shades of junior high — a topic sentence for that paragraph. This will let you judge whether it’s actually going to be possible to move from paragraph to paragraph in a reasonable way — can you imagine a transition that’s going to get you from the topic of one paragraph to the topic of the next?

Note that there’s more to this step than simply flattening the outline from step two. The step two outline is a taxonomy of all the things you want to talk about, but it’s unlikely your readers are ready to digest a whole taxonomy in one go. You’ll likely find that in order to provide a comprehensible linear order for your reader, you’ll need to split certain taxonomic topics across different chapters or sections. For example, when writing Practical Common Lisp, my hierarchical outline contained top-level sections on functions, variables, and macros, three of the main elements of Common Lisp. Initially I thought each of these top-level categories would map to a chapter. However, it turned out it was impossible to write about all aspects of functions without at least some discussion of variables. Yet, there were also aspects of variables that could only be meaningfully discussed after discussing functions. And macros were similarly intertwined with the other two topics. Eventually I figured out that what I needed to do was to write a chapter in which I would discuss the basics — but just the basics — of functions, variables, and macros before moving onto dedicated chapters for each topic. It also turned out that it was useful to split my discussion of macros into two chapters. Thus the three top-level topics from my hierarchical outline had to be split: part of each became a section of chapter four and the remainder became the root of their own chapters except for the macros section which was split into two chapters. In other words, this is the step where you think about the structure of your book, whether you can march straight from A to Z or whether you need to occasionally circle back to clarify things that you couldn’t give full justice to the first time you mentioned them.

Step 4 Write the book. In theory, all that remains is to work through your step three outline, fleshing out each paragraph-level topic sentence with the full paragraph and providing a transition to the next paragraph. In practice, you’ll probably have to frequently pop back to step three if not all the way back to steps one and two. During the actual writing of Practical Common Lisp, I was constantly discovering things that, had I explained them already, would make whatever paragraph, section, or chapter I was currently working on completely straight-forward to write. But then I’d have to figure out how to work those things in somewhere earlier. And often when I actually tried to write those earlier sections, I’d discover some other concepts that I’d really need to introduce before I could write them. I’d know I was having a particularly bad day when I’d discover that I’d looped — that the thing I needed to discuss first, in order to be able to explain everything else I had pushed on the stack, was the very thing I had started with before pushing all these other things on the stack. At that point I’d have to go back to step three and re-outline the relevant sections in light of my new understanding of how things actually fit together. Maybe I could have avoided some of those bad days by better up-front outlining but I suspect not; it was only by actually getting down to the nitty gritty of writing paragraphs and wrestling with how to explain each thing that I could discover the flaws in my more general plan of attack. It was painful at times but I don’t think there was any way to avoid it.

Step four is also obviously the step when you deal with crafting your deathless prose. I don’t have a lot to say about that as good prose style seems more a matter for heuristics than algorithms. That said, if you’ve actually organized your material into some rational, linear order, and you can manage to say what you mean at the sentence and paragraph level, you’ll be doing a lot better than a lot of authors. I did find it useful to print my chapters on actual paper in a reasonable-size font with nice wide margins and double spacing between the lines and to sit down to edit them, away from the computer, with a red pen in hand.

But what do I know? I’ve only written one book. Check back in a few years when I’ve finished my next book and I’ll probably have a whole new algorithm. But for now, as far as I know, that’s how to write a book.

Hello

Hello. Welcome to my blog. If you know who I am you’re probably either my mother or have read my book Practical Common Lisp. Or possibly you’ve either seen the video of a talk I gave at Google’s New York office last May or heard the IT Conversations interview I did with Phil Windley of Technometria. Any of which — except in the case where you’re my mother — might lead you to expect that this blog is going to be about Lisp. Which, though I’ll probably have occasional things to say about Lisp, it mostly won’t. But I’m hoping you might stick around anyway. (Yeah, Mom, I know you will.)

So what can you expect if you do? You’ll get to see me exploring ideas that I hope will eventually turn into another book. Since Practical Common Lisp came out in April 2005, I’ve been consulting — mostly Lisp work, some not. I wound all that down last September in anticipation of the arrival of a baby, my daughter Amelia. Now, much as my wife’s memories of the pain of childbirth are fading, my own memories of the pain of writing a book have faded from visceral to merely intellectual. I remember saying, “I’m never ever going to do this again,” but I can no longer summon up the feeling that led me to say it. Which must mean it’s time to write another one.

At a recent get together of the Bay Area Lispniks I asked the folks at my table what computer-related book they felt was missing. One fellow suggested a book on the programming language D. Hmmmm, I think I’ve already done my bit trying to promote a deserving but unappreciated language to a wider audience. (And I don’t really know enough about D to say whether it’s really deserving. Though any language that starts from the premise that C++ needs to replaced is at least starting out on the right foot.) Another guy suggested a book on “programming for the data center” — i.e. how to write programs that are going to run on a Google-style room full of computers. That sounded a bit more promising but not something that I have any particular expertise in; someone at Google should write it. Then, speaking of Google, Peter Norvig suggested a book about how to program in groups. By which he meant a book that explains all the stuff that a programmer needs to know to be an effective member of a team; the stuff that a lone-hacker, either self-taught or fresh out of a comp sci program, typically has to learn on the job. Norvig claimed — and I tend to agree, based on my own bookshelf — that there isn’t really a good book on that topic. There are books about how to be a better programmer, as an individual. And there are books about project management from which an enterprising developer could back out some lessons about how to be an effective contributor to a team. But nothing that I’ve seen aimed at helping an individual developer develop the skills that are specific to working in the context of a development team.

This suggestion immediately pricked my interest: I’ve always been interested in the question of “what is the best way to develop software?” One way to look at my interest in Lisp is that it is part of an answer to this bigger question. But as much as I love Lisp, I recognize it’s not a complete answer — most software shops would probably get more bang for their buck from adopting techniques that allow groups of people to work together most effectively than they would from switching to a better programming language, even Lisp. So now I’m excited to start writing about that.

My basic plan is to talk to everyone I know who’s involved in developing software and anyone they’ll introduce me to and to read or reread anything and everything relevant I can get my hands on. Then I’ll be ready to apply the Gigamonkey Four-Step Algorithm for Writing a Book that I developed while writing Practical Common Lisp. If you have suggestions of people I should talk to or things I should read please drop me a note. Since all that talking and reading will take a while, it’ll probably be a while before I can get down to the writing phase proper. In the meantime I plan to use this blog as a place to think out loud and to try to keep up my writing chops. And if the stuff I write here helps either to drum up advance interest in the book or to attract folks interested in hiring me as a consultant, that’d be fine too. (Unlike when I wrote Practical Common Lisp, when I spent two years without working other than on the book, this time around I plan to try to combine writing with consulting, both because I’ve got a new mouth to feed and because the right kind of consulting gigs will actually give me more material for the book. See the section “Gigamonkeys Consulting” in the sidebar if you’re interested in hiring me.)