Estimation for Software Projects

It’s a common practice to measure failure or success of a project based on the initial functionality requirements and initial cost and schedule estimated.

The Standish Group publishes its Chaos report for software projects which terms a project as a "Success" if it is completed on time, on budget, and satisfying all the initial requirements. Projects are deemed a "Challenged" if functionality is achieved but cost and schedule over runs occur and "Failed" if a project is cancelled while in execution.

However there are other factors e.g. Tom DeMarco’s  Estimation Quality Factor  and Boehm’s Cone of Uncertainty (COU) or combination of these two (a brief analysis of these two factors can be found in the Jan/Feb IEEE Software magazine The Rise and Fall of the Chaos Report Figures: http://www.computer.org/portal/web/csdl/doi/10.1109/MS.2009.154).  

The questions is, does your company measure project success/failure based on initial software estimation? If yes , what methods are used at your company to establish the matrix (Standish, EQF, COU, any other)? In your opinion what is the value of software estimation accuracy?

Failed software projects are always bad but there are additional complications when there is a contract in place to deliver the software.  Disputes over failed software can result in costly litigation that generally damages both the vendor and the buyer. According to observations of Capers Jones in "Conflict and Litigation Between Software Clients and Developers" (2007) , 5% of the projects his clients were involved in either had litigation pending or were currently involved in litigation over project failures.  His findings indicate that it is very large projects, over 10,000 Function Points that are most prone to litigation, which makes sense if you consider the relative increased complexity, as well as the large amount of money such a contract would involve.  Although one occasionally hears of pending litigation over software projects in the news, determining the background and outcomes of such litigation is difficult because they involve large customers and software contracting organizations, both of whom would rather keep their failures out of the spotlight.  This is likely a significant contributor to the fact that such litigations still occur since the industry is denied opportunities to examine and learn from the mistakes of others.

A report in Computer World [6] details a law suit filed by health insurer Highmark against KPGM for malpractice, fraud and brief of contract for a software system that Highmark had outsourced to KPMG.  While the article does not detail the outcome of this lawsuit, it makes it clear that the disagreement stemmed from misunderstandings and the failure of the two organizations to communicate and collaborate.

In light of this I have compiled a list of four responsible actions that both bidder and buyer should engage in when software projects are outsourced.  These may seem like common sense and the same old story, but clearly they bear repeating......

1. Create and Communicate Good Requirements ... misunderstood, constantly changing and new requirements are a common source of angst in software development projects.  Bidders need to understand the buyers requirements, document them fully, and incorporate the risk of requirements growth into their plans
2. Create and Maintain a credible cost estimate - the buyer needs to know what the effort should cost to evalute bids.  The bidder needs to properly assess the cost to them in order to make the project profitable and to avoid having the project cost them money
3. Understand and document risks; Plan up front for risk mitigation - bidders should work with buyers to understand their uncertainties.  These uncertainties need to be incorporated into the estimate and documented in the contract.
4. Create and maintain a cooperative relationship... when bidders and buyers collaborate and work nicely together, bumps in the road and set backs are handled sensibly, reducing the possiblities that things get out of hand.

Bad project estimates lower profitability.  Despite this fact many business leaders don’t invest in improving their estimating capability, buying into the fatalistic myth that this is as good as it gets.  This is patently wrong.  Project portfolios are prioritized based on the total expected Return on Investment (ROI) of projects.  Investments in the wrong project based on bad estimates could lead to lost revenue or delay of net benefit.

All around us we see reports of software projects which are over budget, delivered late or cancelled because they are taking too much time and money.  This very fact makes it easier for business leaders to throw up their hands and accept bad estimates instead of proactively looking to improve their estimating capability and their profitability.  It doesn’t have to be this way.  Any organization can turn this around with careful analysis of their own history combined with analysis of relevant industry benchmarks.  TruePlanning by PRICE Systems contains the methodology and cost estimating software  to help business leaders utilize history to improve project planning and avoid making bad decisions. 
 

Here’s a great article I happened upon while doing research for a paper I’m writing.  “Lessons Learned: IT’s Biggest Project Failures”  In this article we are treated to stories of IT projects that “first make people laugh and then” (hopefully) “make them think.”  As a long time student of the failed software project, I was neither surprise nor disappointed with the projects relayed.  The projects noted failed for reasons such as:

• Failure to perform a should-cost analysis before selecting a supplier
• Failure to recognize an unhealthy project before it was too late
• Unrealistic and aggressive timelines
• Scope creep.

Doing business with the Department of Defense (DOD) requires that you have disciplined company governance in specific areas as noted in the Defense Federal Acquisition Regulations (DFARs). In particular, DFARs 215.811 and 252.215-7003. DFARs 215.811 requires all DOD contractors, large and small, have adequate estimating systems to support their proposals. As part of a regulatory oversight requirement, the Defense Contracts Audit Agency (DCAA) will periodically perform contractor estimating system reviews. If you are a large defense contractor, you can expect your estimating system to be reviewed routinely. Smaller defense contractors can be audited at any time at the request of their customer. If DCAA finds that you have estimating system deficiencies, "Flash" reports are broadcasted to your customer and other defense agencies. If the deficiencies are serious, you may be suspended from submitting any further proposals until your estimating system is deemed adequate.

So what is an adequate estimating system? It is DOD policy that contractors have estimating systems that consistently produce well supported proposals acceptable as a basis for negotiating fair and reasonable prices. Estimating systems should be consistent and integrated with a contractor's related management systems, and be subject to applicable financial control systems. To be considered adequate, an estimating system must be established, maintained, reliable, and consistently applied. It must also produce verifiable, supportable and documented cost estimates.

DFARs 215.811-70 delineates attributes of an adequate estimating system. So if your estimating system is tested, you will be scored on following:

The Contractor…

1. Value models - enabling analysts to estimate the value and benefits of project once it is delivered, and over its lifetime
2. Business Case Analysis - enabling analysts to prepare and score a case for the project including the cost/benefit analysis, payback period, investment rate of return, strategic benefits, and other evaluation criteria scoring.  The business case tells the story of the project.
3. Analysis of Alternatives and Portfolio Analysis - enabling decision-makers to view Risk/Reward charts of alternatives and planners to reconcile total project estimates to budgets.

Last week I was asked to participate in Career Day at my son’s elementary school.  I was both honored and humbled.  Honored because the school felt that my career was something the children would be interested in and humbled because I was forced to concoct a story that would make cost estimating and analysis both understandable and interesting to children from kindergarten through grade eight.  Fortunately the format was such that I presented to each grade individually so at least I did not have to come up with one story to address all ages.

The experience turned out to be a lot more fun than I imagined.  It’s amazing how many kindergarteners in my son’s school have parents that are scientists (although to be fair their teacher informed me that thirty minutes earlier, when a policeman visited the classroom, many of their dad’s were also on the police force).  Lots of forensic scientists I suppose! 

And it wasn’t really as hard as I expected, explaining what PRICE Systems does.  I used images of the kinds of hardware and systems that TruePlanning for Products can be used to estimate. While some of the younger ones thought that I actually had flown on the Space Shuttle, the upper grades got it.  The experience actually helped me learn several new things about my job.  First of all, I learned that what I do actually is interesting to kids who are starting to think about what they might want to do with their lives. The little kids were fun but the older kids were attentive and asked good questions.  The fact that my work contains elements of science and technology exploration, with math and a little bit of detective work thrown in was actually appealing to many of them.  I was also reminded of how much I like my job and why.  Not only does my job create lots of new and interesting challenges, it also gives me opportunities daily to help people who do software project estimating, hardware project estimating and system level estimating.  I get to help these people be more successful at their jobs by providing tools and methodologies that make their estimates credible and believable.

Software cost estimation is hard.  I have learned this the hard way – as a software developer and later as the manager of software development projects. Today, as a builder of cost estimating software, I learn something everyday about software development that reinforces the fact that software cost estimation is hard.

I attend a lot of trade shows and I talk to a lot of software people about how they estimate software costs.  Many of them have no formal process, many don’t collect data as projects progress, many of them perform estimates off the cuff, and many of them, not surprisingly, are unhappy about the outcome of their software development projects.

Many of these same people are quick to point out, when presented with possible help in the form of TruePlanning for Software, why it will never work for them.  Nine times out of ten, the reason they give me has to do with the uniqueness of their product, their customer base, their market or their software development process.  They question the data behind the model and suggest that this data can not be successfully translated to properly model their specific situation.

And I can’t fight the uniqueness argument – it’s most likely true.  Every software project has factors that make it different from all the other software projects that have occurred in the past.  No two software projects are the same.  I can however argue about the suitability of the data behind my model. The majority of software projects are not defined by their uniqueness.  There is much more about most projects that is similar to other projects than is different.  A good software estimation tool like TruePlanning for Software capitalizes on this fact by allowing the user to take advantage of industry or organizational data for those many places where a project is not unique.  Further, it offers expert systems like guidance to transcend those areas where a software project veers from commonplace to extraordinary.

In software, as in life, we tend to put the most focus on the parts of a project that are challenging and interesting with much less focus on the parts that are ordinary.  If, in doing this, we are discounting the lessons we can learn from history, we are doing ourselves and our software projects a disservice.

A good point is made in a comment to my last post, Chris Carter says, "As estimators I think it is our duty to tell our customer (management) what we assess the possible range of outcomes to be so that they can make use of this information ".  I agree that we should always deliver an indication of accuracy every estimate.  Uncertainty and risk analysis is an integral feature of TruePlanning and we educate our clients on the value of estimate ranges to optimizing project and portfolio performance. The uncertainty-based probablistic confidence-level of an estimate that our products produce is an effective means of addressing the issue. Your cost estimating tools and cost estimating software should address uncertainty and confidence so that weapon system project planning and IT budget planning can be optimized.

Many times estimating software cost is the major challenge. I recently came across a strong case for estimating accuracy and the value of risk ranges in a recent Forrester Research report titled,  “Debunking IT Project Failure Myths,” by Lewis Cardin, a former CIO and currently senior analyst. The twelve-minute Podcast is worth a listen. Mr. Cardin found that among the top reasons projects failed were:

[Begin quote]
1. An unrealistic project plan, which dooms the best project.
All too often, when these projects go on the rails of the original project plan, PMs must spend more time on damage control with steering committees and project resources rather than on execution — doubling their work when it is least desirable to do so.

2. First-number syndrome, which makes business execs forget it’s an estimate.
When projects are first sized, which is likely to occur before they are approved, estimates of cost, time, and resources are preliminary with a wide confidence interval…. But business execs remember the number and forget how uncertain it is…[and] may see this simply as increasing costs, not as the inevitable result of greater knowledge.
[End quote]

I have seen these two factors kill many good projects and ruin the careers of many good people.  I served on the "Intenational Space Station Management and Cost Evaluation Task Force" (IMCE) for NASA and this was exactly the situation with that program.  My experience there prompted me to compile "Six Steps to Program Success" where, "Getting the estimate right the first time" is the most important step. It is so important that you create the right expectations at the beginning of project by presenting the possible outcomes, good, bad, and ugly.

Estimating accuracy is not just about one number. It applies to risk ranges and confidence-levels so that management can determine the right number that goes into the bid or budget.

Martin Woodward ‘s comparison of software development to Sudoku does an excellent job of explaining why software cost estimation is hard.  We constantly hear about software development projects that are unsuccessful because they have violated one or more aspects of the Project Management Triangle  - they are either delivered late, delivered over budget or fail to deliver the capability that users require. And while I firmly believe that software development is an engineering discipline, software project estimation is often more complicated than estimation for other engineering disciplines because software project output is not nearly as easy to quantify and because there are other intangible factors that impact the software development process - the same kind of intangibles that Woodward mentions in relation to a Sudoku puzzle.

Woodward also does a great job in describing how agile software development practices help with software project estimation.  When employing agile development practices, an organization effectively embarks on a new software development project with every iteration (an iteration is a small chunk of time -  usually one or two weeks long - during which the next 'release' of software is completed). The tenets of agile require frequent releases and frequent customer reviews.  These reviews often trigger reprioritization and re-estimation. The agile team develops its own system of software measurement and quickly becomes good at estimating the work required for any given iteration. Agile also promotes good development practices to ease recovery when intangibles threaten to unhinge the Project Management Triangle.  

If every software development project is a two week venture and the development team is very good at estimating the next iteration, why would an organization require a software estimation tool like TruePlanning by PRICE Systems?  This, of course, is an excellent question.  It is true that the software development team doesn’t need a software estimation tool but the organization still does.  The organization needs to be able to put together credible a credible product roadmap.  Customers and sales people still want to know what’s up next and when it’s going to be available.  And although the nature of agile development indicates that content might change, the knowledge of the agile development team will actually help the organization use a software estimation tool successfully. 

Agile is all about measurement.  Successful software cost estimation, even with a great tool, requires good measurement practices.  The tool is the framework for success, but good metrics customize the framework for a specific organization.  Agile organizations that can combine their data with a framework like TruePlanning  can optimize their ability for successful long term planning.

While working with clients who operate under fixed budgets I often hear, "Cost is not an issue when we plan and budget, so why is estimating important?".  I quickly stress that estimating is not just about cost.  It about balancing project scope, cost, time, benefits and risks. If your estimates are not accurate, your projects and portfolios are not optimized – and you are wasting money.

Projects are planned and managed within scope, time, and cost constraints. These constraints are referred to as the Project Management Triangle.  Each side represents a constraint.  One side of the triangle cannot be changed without impacting the others. The time constraint refers to the amount of time available to complete a project. The cost constraint refers to the budgeted amount available for the project. The scope constraint refers to what must be done to produce the project's end result. These three constraints are often competing constraints: increased scope typically means increased time and increased cost, a tight time constraint could mean increased costs and reduced scope, and a tight budget could mean increased time and reduced scope.

The discipline of project management is about providing the tools and techniques that enable the project team to forecast, anticipate and organize their work to meet these constraints.  The discipline of portfolio management is about using tools and techniques to provide management with accurate estimates of cost, scope and time for each ongoing and proposed project so that they can make the right decisions to optimize their objectives within overall constraints. Every day, project managers and business leaders make decisions based on estimates of the dynamics of the project management triangle.  Since each decision can determine whether a project succeeds or fails, accurate estimates are critical.  Projects launched without a rigorous initial estimate are five times more probable of experiencing delays and cancellations.   Even projects with sound initial estimates are doomed if they are not guided by informed decisions within the constraints of the triangle.  If you are working under a fixed budget (cost constraint), then an inaccurate estimate of the number of product features you can produce (scope) within a fixed period of time (schedule) will doom your project.  Inaccurate estimates across your projects de-optimize your portfolio. 

Unfortunately, most project management tools only balance cost and schedule. Tools like Microsoft Project, CA Clarity, Primavera, and others do not link cost and schedule with scope.



Project modeling and estimating tools, like TruePlanning by PRICE systems, maintain a persistent link among project scope, cost, schedule and risks.  With project models, project managers can instantly see the impact that changes in scope have to costs and schedules.  Any side of the Project Management Triangle can be constrained while "what if?" analyses are performed.  Accurate estimates optimize the balancing act.

Successful organizations recognize that estimating is not just about cost, but about balancing within constraints and optimizing results.  They respect the triangle.

Software size measurement continues to be a contentious issue within the software industry.  While the software engineering community has made significant progress in it’s quest for the right way to assign value, the journey has not been without missteps and detours.  And there is still more work to be done.

When I first started programming it never occurred to me to think about the size of the software I was developing.  This was true for several reasons.  First of all, back in the day, software had a tactile quality through the deck of punched cards required to run a program.  If I wanted to size the software I could touch, feel or eyeball the deck to get a sense of how much there was.  Secondly, I had no real reason to care how much code I was writing; I just kept writing until I got the desired results and then I moved on to the next challenge.  Finally, as an engineering student I was expected to learn how to program but I was never taught to appreciate the fact that developing software was an engineering discipline.  The idea of size being a characteristic of software was foreign to me – what did it really mean and what was the context?  And why would anyone care?

Twenty five years later, if you Google the phrase “software size” you will get more than one hundred thousand hits.  Clearly there is a reason to care about software size and there are lots of people out there worrying about it.  And still I am left to wonder – what does it really mean and what is the context?  And why would anyone care?

It turns out there are several very good reasons for wanting to measure software size.  Software size is an important component of productivity and quality metrics.  The ability to measure productivity and quality are an important part of any process improvement initiative.  The ability to measure size is essential to successful software project estimation.  Additionally, a good software size measure could conceivably lead to a better understanding of the value being delivered by a software application.  The problem is that there is no agreement among professionals as to the right units for measuring software or the right way to measure within the units we use today.  Read more about Software Measurement.