Project Estimating Software

It’s a common practice to measure failure or success of a project based on the initial functionality requirements and initial cost and schedule estimated.

The Standish Group publishes its Chaos report for software projects which terms a project as a "Success" if it is completed on time, on budget, and satisfying all the initial requirements. Projects are deemed a "Challenged" if functionality is achieved but cost and schedule over runs occur and "Failed" if a project is cancelled while in execution.

However there are other factors e.g. Tom DeMarco’s  Estimation Quality Factor  and Boehm’s Cone of Uncertainty (COU) or combination of these two (a brief analysis of these two factors can be found in the Jan/Feb IEEE Software magazine The Rise and Fall of the Chaos Report Figures: http://www.computer.org/portal/web/csdl/doi/10.1109/MS.2009.154).  

The questions is, does your company measure project success/failure based on initial software estimation? If yes , what methods are used at your company to establish the matrix (Standish, EQF, COU, any other)? In your opinion what is the value of software estimation accuracy?

Let’s start with a simple test. Which is greater: the number of six-letter English words that have "n" as the fifth letter or the number of six-letter words ending in "ing"?

If you are like most people you’re thinking the correct answer is six-letter words ending in "ing". But most people are wrong. And the reason is simple, people rely on what they can easily recall. Since it’s much easier to think of 6-letter words ending in "ing" the fact that people come to that conclusion isn't suprising. Psychologists refer to this as availability bias. This type of bias gives more importance to things that we can vividly remember and easily recall. All too often, bias like this plays a prominent a role in the development of estimates. (By the way, the group of six-letter words that have "n" as the fifth letter include all of the six-letter words ending in "ing")

Part of the power of parametric cost estimating is the ability to substantially diminish this bias and to fill in where data is lacking. A parametric approach uses sophisticated mathematical equations along with historical data from similar systems or projects in combination with existing organizational data to produce an estimate that is external to the types of biases that often plague estimates. Once organizations are estimating software, hardware or systems of systems using a parametric model, the accuracy of the model will increase as data is captured from other projects. This systematic collection of actuals compared to estimates over time leads to data driven decision making.

Parametric cost estimating is a widely used approach for bidding on a contract, input into a cost benefit analysis, or as the pre-planning tool for project implementation. Extensive literature reviews suggest that an effective parametric cost estimating methodology is becoming an essential tool for technology-driven organizations. The use of parametric estimating in budgeting, scheduling, and control of projects will enhance the ability of project management organizations to effectively and efficiently utilize valuable resources. The benefit of parametric cost estimating tools is its use as an estimating model for better determining potential resource requirements during the project pre-planning and conceptual phase. When software cost estimation is performed using a parametric approach with proven commercial framework, the benefits realized far outweigh the cost of doing the estimating.
 

In an article in last weeks Harvard Business , IT Costs: Do You Speak Their Language), John Sviokla discusses the fact that as the information business continues to grow it is increasingly important for organizations to understand the impact of IT as it relates to their operating costs. This certainly rings True to us here at PRICE Systems who have recognized this reality. TruePlanning 2009 has been developed by PRICE specifically to help organizations get their heads around the true costs of Information Technology.

Application development projects can represent significant expense to an organization and tend to be the riskier items in the IT budgets. But it is important to recognize that they represent only a small part of most organizations IT budgets. According to Gartner’s "IT Spending and Staffing Report 2008",on average organizations spend about 20% of their IT budgets on application development. The rest of their budget is spent on hardware, software, networking, maintenance, as well as data center and help desk activities. The figure below indicates the typical distribution of IT costs based on this report:

From this last report we see that over the last 7 years the distribution of investment dollars between on-going operation and growth and innovation has remained steady with operational expenditures dominating. On average, organizations have spent 65% of their budgets to run the business, 21% to grow the business with a mere 14% of the budget left to transform the business.

With technologies such as SaaS and cloud computing, there is potential for organizations to make decisions to change this picture but they first need a way of identifying what in their IT organization is driving operational costs. TruePlanning for Information Technology makes it possible to model both your appliation development projects and the IT infrastructure (at a micro or macro leve) to help identify the best choices for optimizing Information Technology

Bad project estimates lower profitability.  Despite this fact many business leaders don’t invest in improving their estimating capability, buying into the fatalistic myth that this is as good as it gets.  This is patently wrong.  Project portfolios are prioritized based on the total expected Return on Investment (ROI) of projects.  Investments in the wrong project based on bad estimates could lead to lost revenue or delay of net benefit.

All around us we see reports of software projects which are over budget, delivered late or cancelled because they are taking too much time and money.  This very fact makes it easier for business leaders to throw up their hands and accept bad estimates instead of proactively looking to improve their estimating capability and their profitability.  It doesn’t have to be this way.  Any organization can turn this around with careful analysis of their own history combined with analysis of relevant industry benchmarks.  TruePlanning by PRICE Systems contains the methodology and cost estimating software  to help business leaders utilize history to improve project planning and avoid making bad decisions. 
 

The US Department of Defense (DOD) continues to be plagued with cost overruns on major weapons systems.  Last month  Senators Carl Levin (D-Mich) and John McCain (R-Ariz) introduced the 2009 Weapon Systems Acquisition Reform Act intended to put measures in place to force the DOD to address the issues that cause overruns and schedule slippage.   Among other things,  this legislation would create the position of Director of Independent cost assessment for Major Defense Acquisition Programs (MDAPs) and require the DOD to perform trade-offs between cost, schedule and performance early in the program lifecycle. Read more about this legislation

The DOD's experience with cost overruns and schedule slippage is certainly not unique to them.  The size and complexity of the problems that they are trying to solve, along with the fact that their projects are funded with US tax dollars, makes their contract difficulties more dramatic and more public.  Everyone who builds large systems knows that the bigger their system the more risk there is that things will change between the time the project is first estimated and the time it is delivered.  And most changes that occur within a project involve adding requirements rather than removing them.  This is why cost, schedule and performance tradeoffs are so important – early and often in every project. 

And while there is no silver bullet that will solve the problems that the DOD and many other systems developers have with projects failing to meet goals  – there are tools and methodologies that will facilitate success with cost, schedule and performance tradeoffs.  TruePlanning for Products from PRICE Systems provides a framework in which systems developers can translate their own past performance to provide forward looking analyses that facilitate trade-offs between cost, performance and schedule. TruePlanning provides a trade space to help project managers create realistic expecatations for the cost and schedule of complex systems.

1. Value models - enabling analysts to estimate the value and benefits of project once it is delivered, and over its lifetime
2. Business Case Analysis - enabling analysts to prepare and score a case for the project including the cost/benefit analysis, payback period, investment rate of return, strategic benefits, and other evaluation criteria scoring.  The business case tells the story of the project.
3. Analysis of Alternatives and Portfolio Analysis - enabling decision-makers to view Risk/Reward charts of alternatives and planners to reconcile total project estimates to budgets.

Last week I was asked to participate in Career Day at my son’s elementary school.  I was both honored and humbled.  Honored because the school felt that my career was something the children would be interested in and humbled because I was forced to concoct a story that would make cost estimating and analysis both understandable and interesting to children from kindergarten through grade eight.  Fortunately the format was such that I presented to each grade individually so at least I did not have to come up with one story to address all ages.

The experience turned out to be a lot more fun than I imagined.  It’s amazing how many kindergarteners in my son’s school have parents that are scientists (although to be fair their teacher informed me that thirty minutes earlier, when a policeman visited the classroom, many of their dad’s were also on the police force).  Lots of forensic scientists I suppose! 

And it wasn’t really as hard as I expected, explaining what PRICE Systems does.  I used images of the kinds of hardware and systems that TruePlanning for Products can be used to estimate. While some of the younger ones thought that I actually had flown on the Space Shuttle, the upper grades got it.  The experience actually helped me learn several new things about my job.  First of all, I learned that what I do actually is interesting to kids who are starting to think about what they might want to do with their lives. The little kids were fun but the older kids were attentive and asked good questions.  The fact that my work contains elements of science and technology exploration, with math and a little bit of detective work thrown in was actually appealing to many of them.  I was also reminded of how much I like my job and why.  Not only does my job create lots of new and interesting challenges, it also gives me opportunities daily to help people who do software project estimating, hardware project estimating and system level estimating.  I get to help these people be more successful at their jobs by providing tools and methodologies that make their estimates credible and believable.

                     
Like many others, I was astonished last Thursday by the images on my browser of those 155 extremely lucky people standing in the Hudson River.  And they certainly were very lucky last Thursday.  If you’re destined to fly on a flight bound for collision with birds, you want it to be piloted by a hero like Captain Sullenberger.  The incident made me think about what a hero is and how we all have the opportunities to be heroic in our chosen professions.

According to Wikipedia, a hero refers to a character that, in the face of danger and adversity or from a position of weakness, displays courage and the will for self sacrifice.  As a project manager you may not encounter danger daily, at least of a physical nature, but you certainly deal with adversity on many levels.  And courage is essential!

An heroic project manager is one who makes project decisions based on facts and who stands behind those decisions.  He or she defends the project team from those who wish to impose unrealistic deadlines and stands up to those who want to abuse or ignore the Project Management Triangle.  In order to be heroic, a project manager needs to have the tools necessary to plan a project right the first time as well as manage it with agility as situations change.  TruePlanning from PRICE Systems is just such a tool.  TruePlanning is the worlds leading solution for software cost estimation and IT Project Planning.  It gives project mangers the cost estimates they need to plan projects successfully and the courage necessary to defend those numbers to the naysayers.

Although the use of TruePlanning is unlikely to lead to instant fame or enable people to walk on water, a project manager who uses it regularly will be a hero to the team!
 

Today, change is in the air.  As I write this, Barack Obama is about to be sworn in as our new U.S. President and the space community, among others, should be braced for change.  A recent LA times article reported that of the 74 questions asked of NASA by the Obama transition team, over half were on basic spending issues, including cost overruns.

The Obama team and the NASA Administrator Michael Griffin clearly do not see eye-to-eye.  Monday, it was announced that Mr. Griffin will step down from the post.  Griffin characterizes himself as an engineer and states that NASA shouldn't be evaluated by how well it estimated the cost of projects.

"We start these things out, and we admit up front we don't completely know how to do them. That is what makes them interesting," Griffin said recently.

"If we are to judge the worth of our work by our ability to estimate, then that is a standard I am not ready to apply or to accept," Griffin said.

Others are more realistic about NASA's obligation to create credible expectations for spending taxpayer money.

"Our space program is running inefficiently, and without sufficient regard to cost performance," wrote Alan Stern, a former NASA associate administrator who has been mentioned as a possible replacement for Michael Griffin, the current NASA administrator.

In a recent op-ed piece in the New York Times, Stern called the cost overruns a "cancer" that has cost the agency's science program about $5 billion over five years.

Agency officials said they had improved financial controls -- including forcing managers to better estimate costs.

Considering NASA's great technical achievements, one must wonder, "Is accurate estimating harder than rocket science?" 

I don't think so.  I believe the clue to accurately estimating the cost, uncertainty and risk associated with large, complex space projects can be found in a recent Defense AT&L article by Col. Brian Shimel, USAF.  Col. Shimel states,

"We cannot relieve ourselves of the need to plan for the future just because the future is uncertain. For our plans to be reasonably accurate and reliable, it is prudent we base them on rational analysis and not on wishful thinking.", and...

"We must think clearly about uncertainty and risk, and we must fight the temptation to discount those factors when communicating the real conditions of our management situation. We don't get in trouble because of risk and uncertainty. We get in trouble for not admitting to ourselves - and those who rely on us - all of the risk and uncertainty that inherently exist in everything we plan to do."

Our TruePlanning estimating and cost management software gives managers metrics and benchmarks of previous projects, and methods to realistically quantify uncertainty and risks.  TruePlanning's reports and charts expose over-optimism and show managers the real conditions of the situation.  I routinely recommend that government agencies quantify all the risks and uncertainties of each project in their portfolio and budget to the sum of the 70% confidence levels of each project, yielding 80% confidence that the projects can meet taxpayer expectations for the budget.

President Barack Obama just finished his acceptance speech.  He called upon us all to increase our service to the country.  Government managers can help by stepping up to the estimating challenge.

Accurate estimating is not rocket science.  There are sophisticated cost estimating models to accurate estimate software development costs, hardware development costs, IT project costs, and operation and support costs - we just need to be realistic, responsible and courageous about presenting the results for our analysis. Contact us and we'll help you get started today.

This was a fun and gratifying week at PRICE Systems. In our Mt. Laurel, NJ headquarters we had our annual holiday gift wrapping of presents we donated for needy children and families in the area.  The PRICE team gathered in our classroom and the wrapping began with no pre-instruction or guidance - but much merriment.  What a study in production process and learning!  Wrapping that first present was awkward.  It had been a while since I wrapped last.  How to measure the paper?  Where to cut?  How to keep the cut straight?  Which way do I fold first? How to fold?  What to do if I have too much paper on the ends?  Where do I put the tape?  ...and so on.  Needless to say that first present took some time, I wasted a lot of paper and tape, and the final product had some wrapping defects.  After reflecting on the results, I looked around at others to see what they were doing, asked some questions, processed that information along with my own experiences, and set out on the second package.  I did much better.  I wrapped faster, I wasted less, and the final product was an improvement over the first.  With each new gift I got better and better.  I "learned" how to be more productive, reducing the effort and materials with each present.  By the eight present, I started to focus on how I could reduce materials.  How can I use less paper?  How can I use the least amount of tape and still keep the paper form coming off?  I was improving the overall process.

Productivity increases like mine occur with each and every production process and it is an essential part of cost and schedule estimating.  The method most commonly used to model this effect is called "learning curve", and we use it in all of our models. You can find hundreds of references and equations on the web, but generally the model uses a "learning curve parameter" (LC) to estimate the cost for a certain production quantity.  The LC is expressed as a percentage, and (1-LC) represents the amount the production cost is reduced every time the quantity doubles.  For instance, if I say an 80% learning curve applies to the process, then every time the quantity doubles the cost to produce is reduced by 20%.  So if the first unit cost $100, then the second would be $80, and the fourth would be $64, and the eighth would be $51.20.  For the same starting point, a 90% learning curve represents less learning and higher costs, while a 70% learning curve steeper learning and lower costs. 





If I had timed my wrapping and measured the materials used, I could have used simple math to determine the learning curve that applied. I suspect it was close to 70%.  Determining the learning curve for everything from paper clips to space ships is more complicated, but that is one of the things we do at PRICE.  We perform operations research and analyze data to keep our models as accurate as possible, and embed that research in our cost management software.  Accurate learning curve estimates help our clients optimize their investments, prepare better budgets, plan more effectively.  And as part of this most recent study, we made  a lot of kids happier for the holidays.  Happy Holidays to you. 

A recent Gartner report indicates that industry enthusiasm for SOA is waning.  The reasons cited are the lack of enough people with the proper skill sets to perform SOA deployments and the lack of a good business case for SOA.  It’s an interesting but not really unexpected direction.  SOA has been surrounded by significant hype, ensuring that organizations surveyed would be anxious to profess their desire to start a SOA project.  But as the rubber hits the road, these organizations are realizing that SOA may not be the answer to all of their organizational woes.

Many will argue that SOA is nothing new – but rather a repackaging of an object oriented approach to software reuse.  They insist that the reason for this repackaging is motivated by a strong marketing push to sell a new generation of tools to folks that were not successful achieving reuse with earlier versions of tools.  Although I agree that the notion of SOA is not new, I do believe that technology has been advanced to a level where enough abstraction can be applied to the process to make reuse a possibility.  I do agree with the cynics that the heavy marketing push for SOA has not helped businesses make thoughtful decisions about whether it is right for them.

Now that enthusiasm has been tempered, organizations are starting to look at the costs, benefits and ROI of SOA.  This is a good thing.  A more thoughtful approach across the industry will balance the effects of aggressive marketing and ensure that the right SOA projects are attempted.  Of course, the challenge of determining ROI is daunting, particularly without a lot of data. Determining the benefits of a relatively untested paradigm is problematic.  If the SOA vendors are to be believed the benefits are tremendous.  Experience of course tells us that this is optimistic.  But common sense indicates that there is a great deal of potential for benefit for businesses that meet one or more of the criteria listed in the article “Your SOA needs a business case” – in other words if your business must respond to frequent changes in the marketplace or with business rules and relationships.

Understanding, or predicting, the benefits SOA will bring to the organization is only half the challenge.  The costs associated with a deployment of capabilities in a service based framework are also an important consideration.  The TruePlanning Suite, available from PRICE Systems, can help in this area.  TruePlanning allows for IT Project cost estimation that will help address costs associated with deploying the right infrastructure for SOA, the costs associated with development of services, and the costs associated with the composition of applications using services.

Technology readiness is a critical cost driver of development programs.  Many high technology programs fail because initial cost and schedule expectations were based on the assumption that the technologies employed were proven,  when actually they were not. Space programs have the most dubious history in this regard.  I once listened to a Lockheed Martin executive explain how the X33 space shuttle was a great vehicle, but was canceled because it was, "two inventions short of meeting the requirements". Canceled after over one billion dollars were spent.




Starting development projects that have constrained budgets and schedule with unproven technology is a bad idea, one that has not gone unnoticed.  The Defense Acquisition Performance Assessment 2006 (DAPA) report states,

"Technology maturity or “knowledge-based” development has been a subject of considerable discussion between the Department and the Congress. However, although there is agreement concerning the advantages of ensuring that technology is mature prior to proceeding to development and production, there are no clearly definable measures of technology readiness.", and,

"Incorporating high-risk technology in systems generally leads to significant cost and schedule impacts."

So, have decision-makers learned their lesson by now. Recent news suggests not.  It was just published that NASA's Mars Rover mission was delayed two years and is over budget because,

"problems developed in the design and operation of the 31 actuators that control the mechanics of the craft, including the steering mechanism and its robotic arm"

Probably a few inventions short.

Technology should be matured in a lab environment with a sustained level of effort before it is employed in a development project with cost and schedule constraints.  Anything else is extremely risky and will most likely fail.

The PRICE TruePlanning models each have a method evaluating the cost and risks associated with technology maturity.  Our cost management software helps our clients decide among  alternatives with varying technology maturity and let them know the risk of overrunning cost and schedule.  Most of the time our analysis can dissuade executives from starting a project "two inventions short".

Software cost estimation is hard.  I have learned this the hard way – as a software developer and later as the manager of software development projects. Today, as a builder of cost estimating software, I learn something everyday about software development that reinforces the fact that software cost estimation is hard.

I attend a lot of trade shows and I talk to a lot of software people about how they estimate software costs.  Many of them have no formal process, many don’t collect data as projects progress, many of them perform estimates off the cuff, and many of them, not surprisingly, are unhappy about the outcome of their software development projects.

Many of these same people are quick to point out, when presented with possible help in the form of TruePlanning for Software, why it will never work for them.  Nine times out of ten, the reason they give me has to do with the uniqueness of their product, their customer base, their market or their software development process.  They question the data behind the model and suggest that this data can not be successfully translated to properly model their specific situation.

And I can’t fight the uniqueness argument – it’s most likely true.  Every software project has factors that make it different from all the other software projects that have occurred in the past.  No two software projects are the same.  I can however argue about the suitability of the data behind my model. The majority of software projects are not defined by their uniqueness.  There is much more about most projects that is similar to other projects than is different.  A good software estimation tool like TruePlanning for Software capitalizes on this fact by allowing the user to take advantage of industry or organizational data for those many places where a project is not unique.  Further, it offers expert systems like guidance to transcend those areas where a software project veers from commonplace to extraordinary.

In software, as in life, we tend to put the most focus on the parts of a project that are challenging and interesting with much less focus on the parts that are ordinary.  If, in doing this, we are discounting the lessons we can learn from history, we are doing ourselves and our software projects a disservice.

Jacob has a great comment to my recent post on getting it "right the first time".  He notes that requirements are often fuzzy and that estimates rely on peoples' opinion of cost and duration - and that the are often wrong.  He asked what silver bullet we have. Our silver bullet is a proven discipline that makes people better estimators and sheds light on fuzzy requirements.

It starts with the people. Becoming an accurate estimator requires the same step-by-step approach as any learned skill such as golf or tennis or swimming (things I am currently struggling to do well). First you need the right tools, then you need to learn the right way, then you need to be mentored by experts as you practice what you learned until it becomes second nature. Recently, I was struggling with my breathing while attempting to swim. Lap-after-lap I was breathing hard, swallowing water - it was a miserable sight.  Then I consulted an expert that got my head turning the right way and got me into an rhythm of breathing every third stroke.  My swimming quickly improved by 40% - more laps in less time.

We do the same for our clients.  First we arm them with the right tools.  Our TruePlanning software provides a proven step-by-step discipline to do an estimate the right way,  TruePlanning asks the right questions and covers all the bases, including risks.  Then we train people not just how to use our software but how to be great estimators.  Sizing methods, interview techniques, and requirements elicitation are part of our training.  Then our experts are by their side as they hone their skills through email, live-meetings, a 24/7 hot-line, and on-site visits.  Throughout we provide people with benchmark measurements of other similar efforts and tips to guide them. Suddenly fuzzy requirements become clearer. People are doing more accurate estimates in less time. Typically our clients improve their estimating accuracy by 50% within three months and several of our clients now produce estimates that average within 5% of actual scope, cost, and schedule.  Our clients leverage our cost estimating software and cost management software to estimate software costs and schedules, IT infrastructure costs, and custom hardware to improve their overall project and portfolio management.  By the way, I just finished a mile in the pool and improved by another 10% - practice makes perfect.