Comments on Why is estimating so hard? from Reddit can eventually lead you to a comment on another post titled, Engineering Management: Why are software development task estimations regularly off by a factor of 2-3?:

Let's take a hike on the coast from San Francisco to Los Angeles to visit our friends in Newport Beach. I'll whip out my map and draw our route down the coast...

The line is about 400 miles long; we can walk 4 miles per hour for 10 hours per day, so we'll be there in 10 days. We call our friends and book dinner for next Sunday night, when we will roll in triumphantly at 6 p.m. They can't wait!


Man, this is slow going! Sand, water, stairs, creeks, angry sea lions! We are walking at most 2 miles per hour, half as fast as we wanted. We can either start walking 20 hours per day, or we can push our friends out another week.  OK, let's split the difference: we'll walk 12 hours per day and push our friends out til the following weekend. We call them and delay dinner until the following Sunday. They are a little peeved but say OK, we'll see you then.


G-----n map doesn't show this s--t! We have to walk 3 miles inland, around some fenced-off, federally-protected land, get lost twice, then make it back to the coast around noon. Most of the day gone for one mile of progress. OK, we are *not* calling our friends to push back again. We walk until midnight to try to catch up and get back on schedule.


Okay, well, that's why you hire someone who has walked that route before to estimate it. The problem is, of course, that it's surprisingly rare to find programmers who have walked the exact route that your customers want. And if they have and it's available, that's usually called off the shelf software. As beager so eloquently puts it, when it comes to OTS software (commercial, OSS, whatever):

Ooh, the best part is, the wireframes have items that are SO PAINFULLY CLOSE to what plugins do, but are EVER SO SLIGHTLY different in a way that makes you have to roll your own.

That, my friends, is often, unfortunately, an entirely new path, if only because you haven't hired the authors of the plugins.

Perhaps more (less?) succinctly put:

Engineers attempt to scope work by estimating the time all known substeps will take to address. Any other kind of scoping seems unprofessional, since you are basically making things up. Yet everyone agrees that there will be a number of potentially large unforeseen issues (the unknown unknowns).

If all the steps were known and independent, estimation would work much closer to your intuition--some steps would be overestimated and others underestimated, and you would expect these mistakes to cancel each other out. This is the kind of estimation people are used to and good at. In reality many steps are sequential so one problem may cascade into others, and worse new work items continually arise which are added to the total.