Estimate Software Costs

TruePlanning 2010 SR1 estimation software is now available as an upgrade for existing PRICE customers. The most significant update to this version of TruePlanning is the capability to use both parametric estimating models as well as analogous data to produce estimates. This capability validates and increases the defensibility of estimates.

TruePlanning provides a framework that allows content driven parametric models to be estimated in one system. Most notably, hardware, software, IT and Systems of Systems (SoS). No other commercially available estimating tool can make that claim. However, whereas in previous versions estimates relied on parametric models which are based on industry benchmarks, now users can also incorporate their own data for analysis. Customers can import organizational data to TruePlanning, run regression analysis on that data, and further validate their estimates. So now estimates can be backed by internal organizational data as well as external benchmark data, which is a powerful way to defend an estimate.

To view complete details of this new feature as well as other enhancements visit our New Release Info page. 

Today, PRICE Systems, Senior Research Analyst, Bob Koury, will be presenting on Will Cost/Should Cost management.

The presentation will focus on two main requirements mandated in the Ash Carter memo (mentioned here several times): Developing Should Cost/Will Cost targets and establishing Affordability as a requirement.  An example will be provided of how parametric estimating models were used to establish “Should Cost” targets and how they can be used by a budget authority (government or Industry) to be an informed consumer of contractor or sub-contractor bids. The demonstration portion of this webinar will focus on a process for using a parametric estimating model to produce the reasonable affordability targets.

To sign up for the webinar go here: https://www2gotomeeting.com/register/142598427

In last month’s blog I wrote about Ash Carter’s (Under Secretary of Defense for Acquisition, Technology & Logistics) Memorandum for Acquisition Professionals, Better Buying Power: Guidance for Obtaining Greater Efficiency and Productivity in Defense Spending (14 September 2010).

I concluded the TruePlanning unified framework and comprehensive cost models, is a tool very well suited to provide the types of analysis outlined in the memorandum. In terms of Should Cost and Independent Cost Estimates (ICE), TruePlanning estimation software provides the industry standard capability to conduct Should Cost and calibration (actual program history) for ICE. Most interestingly and to the point of the Carter memo, TruePlanning has the capability of breaking the “self fulfilling” prophesy of business-as-usual.

On November 3^rd, Ash Carter issued a follow up memorandum outlining specific actions to implement the September 14^th Guidance. For this month’s post, I wanted to point out some specific comments about Milestone A and B actions and specifically how TruePlanning can help.

At Milestone A the memorandum specifics establishment of an “affordability target to be treated by the Program Manager (PM) like a Key Performance Parameter (KPP)…This analysis should show results of capability excursions around expected design performance points to highlight elements that can be used to establish cost and schedule trade pace.

At Milestone B, the memorandum requires ”a system engineering tradeoff analysis showing how cost varies as the major design parameters and time to complete are traded off against each other.” Thus, Integration between parametric cost models and performance models is needed to allow full exploration and optimization of the trade space, treating cost as another design parameter

Did you know that TruePlanning through its integration with Phoenix ModelCenter is a powerful tool to provide exactly this type of analysis? ModelCenter contains a native plug-in to interface with TruePlanning directly. Through the ModelCenter interface, analysis can easily perform Design of Experiments (DOE), Carpet Plot Analysis (trading performance KPP’s against cost) and Optimization Analysis.  If you are interested in seeing some examples, especially system engineering trade off analysis, please get in contact with me and I will send you a few presentations.

In his August blog-entry here, Zach Jasnoff outlined typical client perspectives for the different types of analyses that TruePlanning can accommodate. Working on a large project, we’ve experienced situations that, realistically, can happen where the initial intent and model structuring later have the boundaries of model appropriateness stretched. An Analysis of Alternatives (AoA), for example, is meant to measure deltas between baseline and its alternatives. If common costs “wash” then they can be excluded… which becomes an issue when treated as a Rough Order Magnitude for customer budgeting. 

Likewise, if a ROM or Independent Cost Estimate (ICE) of acquisition costs is extended to Program Lifecycle Costs, significant gap analysis is in someone’s future. TruePlanning can handle these tasks. But obviously it’s always best to establish use of an estimate and its underlying model, beyond initial application. TruePlanning has enormous flexibility. We analysts endeavor to do the same.  Clients, particularly those new to parametric cost estimation, just need coaching and synchronization up-front sometimes. They may think they have “the estimate” when it only covers certain configurations, activities, phases or cost-categories. Expanding a model is simple enough; creating and defending changes can be tough for the customer internally. 

I’m continuing to realize that when in doubt, make it clear what an estimate & model is meant for, and what it’s not.

John Swaren
Solutions Consultant, PRICE Systems

Yesterday I had the pleasure of speaking at the New England SCEA Chapter December luncheon  [link to Flyer .pdf].  The attendees were a great mix of experienced, seasoned cost estimators and young, new talent, eager to learn techniques to apply on the job. 

My topic was the program management value of combining estimating Rules of Thumb with more rigorous cost estimating models and databases [link to presentation .pdf].  Rule of Thumb estimating is used every day by program managers to help guide their projects.  Oversight authorities rarely have the resources to perform detailed program estimates, so they rely on simple Rule of Thumb-like models to ensure estimates are within the budget tolerance.  I stressed the importance of using external and internal benchmarks to aid model building.  One of my favorite examples of this is the Tailor Rule of Thumb.  

This is a simple approximation that was used by tailors to determine the wrist, neck, and waist circumferences of a person through one single measurement of the circumference of that person's thumb. The rule states that twice the circumference of a person's thumb is the circumference of their wrist, twice the circumference of the wrist is the circumference of the neck, and twice around the neck is the person's waist. This external benchmark uses average values (2x) for the wide population of the time.  I decided to update this with internal benchmarks, or measurements of my own.  I carried a tape measure for a week and measured everyone who agreed – friends, family, and co-workers.  Based my internal benchmarks, I discovered that a multiplier of 2.3 made the model extremely accurate – it is no secret that we eat more than average New Jersey.  In this way, I combined an external benchmark with internal benchmarks to determine an estimating Rule of Thumb with which I am comfortable. 

We empower our customers with models and tools to arrive at the same comfort level for complex projects.  Most of the luncheon attendees work every day with rigorous models and databases, so they left the luncheon for a new appreciation for the value of Rules of Thumb.

I was recently struck by Ash Carter’s (Under Secretary of Defense for Acquisition, Technology & Logistics) Memorandum for Acquisition Professionals, Better Buying Power: Guidance for Obtaining Greater Efficiency and Productivity in Defense Spending (14 September 2010). Within this broad sweeping memo, Ash Carter outlines 23 principal actions in five major areas aimed at increasing efficiency in Defense acquisition.  The first major area covered is “Target Affordability and Control Cost Growth”.

Within this major area, program managers must treat affordability as a requirement before milestone authority is granted to proceed (starting with Milestone A). This means developing affordability targets and treating cost as a Key Performance Parameter. 

What I find really interesting is the critique of Will Cost vs. Should Cost. The memo is critical of Will Cost, or Independent Cost Estimates (ICE). As Dr. Carter points out, “the ICE reflecting business-as-usual management in past programs, becomes a self fulfilling prophesy. The forecast budget is expected, even required, to be fully obligated and expended.” To combat this “vicious cycle”, the memorandum now requires “each major program to conduct a Should Cost analysis justifying each element of program cost and showing how it is improving year by year or meeting other relevant benchmarks for value.”

In my career experience, parametric estimating models such as TruePlanning play a major role in targeting affordability and controlling cost growth. In terms of affordability analysis, TruePlanning contains a unified framework of all elements of program cost (hardware, software, IT and Systems) and has built-in capacity to interface with engineering optimization tools such as Model Center. Through this interface cost can be treated as a Key Performance Parameter (KPP) for optimization and engineering trade off analysis.

In terms of Should Cost and ICE, TruePlanning provides the industry standard capability to conduct Should Cost and calibrated (actual program history) for ICE. Most interestingly and to the point of the Carter memo, TruePlanning has the capability of breaking the “self fulfilling” prophesy of business-as-usual. Using a calibrated TruePlanning model for ICE, estimators can change key engineering and programmatic parameters and see the impact on cost. For example, parameters such as requirements stability, engineering complexity and team composition can be quickly changed to assess a new program’s reality while still taking into account past performance history.

As the Carter memorandum points out … “the ability to understand and control future cost from a program’s inception is critical to achieving affordability requirements.” Because of TruePlanning unified framework and comprehensive cost models, it is a tool very well suited to provide the types of analysis outlined in the memorandum.

Zach Jasnoff
Solutions Architect, PRICE Systems

After some recent meetings with clients I am sensing some confusion on how to estimate software reuse. I think part of the problem is in the definition of reuse, so let's start with a definition and then address the estimating issue. Software reuse is defined as “the use of existing software, or software knowledge, to build new software.” This definition came from Wikipedia. From a estimating software costs perspective the above definition is part of the problem. The definition should read: "Use of existing software with no changes for operation in the new software program.” 

If the existing software is going to be changed, re-written or modified to operate in the new program, it should be modeled as software adaptation. In this case the software will require some amount of re-testing, and software integration. Software that is being re-used has been fully tested during the original development program. In this case the reused software does not have to go through full up integration and testing. Rather it will go through a regression testing (computer aided testing) to insure it is still operating correctly.

If the software is going to be re-used as is, then clients should use the TruePlanning  “Software COTS” cost object to model and estimate. The purpose of modeling is for integration control. External integration should be set to a value of 0 to 1.0. A value of zero implies that there is no additional integration effort. Values less than one reduce the amount of model calculated software-software integration effort. The reused software can be modeled as a software component. However, do not include software adaptation or new software with the reuse data. This will overstate the integration effort. 

Some of us remember taking the Iowa tests during our early school days. The Iowa Tests of Basic Skills (ITBS) are standardized tests provided as a service to schools by the College of Education of The University of Iowa. The tests, administered to students in grades K-8, became a national standard for measuring scholastic aptitude – I was educated in Pennsylvania.

Now out of Iowa comes another test of sorts, something called an Integrity Index Score based upon a proprietary algorithm of an organization called Iowa Live. Iowa Live calls itself, “a growing network of volunteer Citizens & professionals for improving Iowa.” The organization has addressed a variety of issues in the education, health, and government areas. In July, Iowa Live published its comparison of two commercially marketed parametric estimating software products, one of which was PRICE TruePlanning. The comparison was for both estimation of software projects and estimating hardware. If you’d like to see the results, visit the Iowa Live website, where you can also learn more about the theory and derivation of the Integrity Index.  

Bruce Fad
VP Professional Services, PRICE Systems

When I glanced at the Washington Post on Sunday, the following headline screamed out:

Defense cuts could slow D.C. economy for years

The article basically covers how Defense Secretary Robert M. Gates is calling for reducing spending on "support contractors" by 10 percent each of the next three years as the Defense budget shrinks. As Washington DC is a hub for these types of companies, the impact is expected to be significant. According to the article, more than a quarter of national defense spending contains of outlays for service contracts. Among the largest companies affected are CACI, SAIC, Lockheed-Martin, Booz Allen Hamilton and ManTech.

As the Defense budget shrinks and more companies are chasing fewer projects, the need for greater accuracy in cost estimating increases. At the same time, the WSRA adds 20,000 new positions in cost estimating, contracts and oversight.

This is an interesting conundrum, cutting DoD service contracts 10% over three years while the government is adding 20,000 new positions in cost estimating, contract and oversight

One year after the WSRA was signed into law, an interesting observation from the National Defense Magazine Blog:

Nancy Spruill, director of acquisition resources and analysis at the office of the undersecretary of defense for acquisition, technology and logistics points outs “The long pole in the tent is the cost estimates,” she said. “There's a lot of programs that need cost estimates as they're moving through the process today. … Doing additional ones has been difficult, especially a while ago, when they didn't have the staff.”

Thus, in an era of shrinking defense budgets, greater oversight and more competition for fewer projects, parametric cost estimating models such as TruePlanning can play a major role in providing accurate cost and risk estimates for both government and contractors. This is especially true in Source Selection where determining the most cost-effective offering is critical. For more details on using TruePlanning for Source selection, see my previous two blog posts.

Ahhhh, the 80s… a challenging (but often confusing) time in an evolving computing world.  Working in 1985 as a software estimator as well as SQA engineer in a quality assurance department that “audited” real-time projects using new concepts like OOD & OOP… well, you get the picture.  It was a great time to get immersed into great work.  And the good news:  that company’s process as well as its developers were bullish on a young estimation/ quality types asking plenty of questions… as long as they were of the Yes-No variety.  And ask I did.  Writing and using those checklists was great OJT, above & beyond adding their value of verifying/ enabling healthy development activities. 

Less than two years later, SEI’s Walt Humphrey formalized his Capability Maturity Model with its five levels of software process maturity.  These “CMM” levels are still used by many organizations and estimating tools, including within True Planning’s “Organizational Productivity Calculator.”  Could the CMM assessment be supplemented to include cost-driver questions for purposes of parametric estimation?  It worked in my SQA audit days. 

In fact many estimators (if only because they can’t get on an SME’s schedule twice) take the opportunity to ask qualitative and quantitative questions.  Integrating both lines into one interview should still be received as a healthy value-add.  Whether estimation is resident within a quality or financial function is moot.  We have the entrée to support both costing/ planning and productivity assessment.  A good example of an integrated approach is the David Group’s Project Profile Worksheet, cited in the text “Function Point Analysis” by David Garmus & David Herron.  Less comprehensive checklists are likely out there too, including methodologies proprietary to individual companies. 

The thought today is that the future is now and structured approaches, called audits or otherwise, will enable good practices as well as illuminate new approaches too.  In Wall Street, regulations are only good in controlling past (known) bad behaviors.  In just as creative Software, process rules help developers maximize their productivity towards current & future good behaviors.

The key cost driver when estimating software costs is the size of the product. The problem is that there is no perfect technique available to measure and quantify the size of software. The two major techniques in use today are Source Lines of Code and Function Points.  Today we will talk about Source Lines or Code or SLOC.

Source Lines of Code measures logical lines of code. It takes some of the uncertainty out of physical line of code measures by counting only complete statements (which can cross over more than on physical line). SLOC excludes comments and blank lines.

SLOC has several advantages:

·         Easily counted by automated tools, which can be configured with an organizations definition to produce consistent counts. Link to free tool from USC is provided below.

http://sunset.usc.edu/research/CODECOUNT/download/2010/UCC_Release_Notes_v.2010.07.pdf

·         Provide visibility into technical progress as projects proceed through design, code and testing.

·         Can be used to develop historical databases of past project size, which enable the more accurate prediction of size of future similar projects.

·         Design, Code and Test change to modified software can be specifically quantified

SLOC has several disadvantages:

·         It’s very difficult to predict SLOC on new programs prior to design

·         Different programming languages will require different amounts of SLOC to implement the same functions. This makes summary measures of size on multi-programming language products difficult

·         SLOC is difficult to map back to product features or requirements. Which makes usage of EVM difficult to apply to SLOC based SW projects. 

·         SLOC is a meaningless number to non Software Engineers (without detailed explanation)

Over the past several weeks several users have inquired about the best way to model legacy software that is being modified when estimating software costs. The software component within the TruePlanning Software model has an input parameter call “adapted Code Size.” This input parameter accounts for existing or legacy software that will be modified or changed to meet a new requirement. Tied with the size input parameter is Percent Design/Code/Test adapted. Although the model will calculate a percentage for each input, I would recommend that user’s analyze the calculate values and override the calculation where required. The percentage for each parameter is to reflect the amount of modification being accomplished. For example, calculated values below five percent or above forty percent are probably unrealistic. 

Values below five percent reflect one single function being modified, which is normally not the case. Values above forty percent generally result in the entire software package being re-developed. Hopefully this will clear up this software input parameter. If you have further questions, please email me at james.otte@pricesystems.com

Jim Otte
Solutions Architect, PRICE Systems

Recently I came across the word “off-label”.  It is the term used by the medical community when a drug is used to treat a condition for which it has not been approved by the Food and Drug Administration.   We sometimes use TruePlanning for “off-label” purposes. A good example would be using the TruePlanning Calibration tool to answer such questions as, what is the maximum number of source lines of code (SLOC) I can get and remain within my budget?  I call this TruePlanning Optimization. Here is an example answering the SLOC question.

First begin by performing a normal cost estimate for your software project. If the cost is under your budget and the project contains all the software functionality you would like, you are “home free”. If you have excess budget and would like to see how much more functionality you can add or if you are over budget and need to consider eliminating some functionality to stay within budget, then the following is for you.

Let’s say that the requirements call for 10,000 SLOC of C++ code. Using the TruePlanning Software and TruePlanning Systems catalogs, the total cost is $873,299, but our budget is only $750,000. How many SLOC can we purchase given our budget? This question can be answered quickly and easily using the TruePlanning Calibration tool. Open the TruePlanning Calibration tool and select a New calibration. On the Calibration Settings screen, for the Inputs Cost Selection, choose the Software Component cost object and New Code Size. For the Output Cost Selection, choose System Folder and Estimated Cost. For the Target Value, enter the $750,000 and adjust the Tolerance and Maximum Iterations.

Now, on the Calibrations screen, click on Calibrate and the answer appears as the Calibrated Input. As shown below, we can purchase 8,594 SLOC with our $750,000 budget. Next the big question is, what functionality can we safely eliminate in going from the 10,000 SLOC to the 8,594 SLOC?

As you can see, the medical folks are not the only ones who use things “off-label”.    The more you use TruePlanning, the more likely you are to discover new, creative uses for TruePlanning. Off-label is not a bad word or concept.

You need 3 things for your software estimates to be successful

And I will add a fourth one in after I talk about the first 3

1.       You need qualified and experienced people to generate the estimates. They have to know how to estimate and they have to understand what the problem is that the project is going to solve…..at least well enough to estimate it. This can be one person or many depending on the difficulty of the business area. The harder it is, the better having more brains look at the problem. But not to the point where it can slow you down. A team of 2 to 5 people can be faster and more efficient that a team of 8 to 12 and it’s easier to reach consensus.

2.       You need your own data as a reference point in the estimate. As a comparison or an analogy.   Your own data makes selling and explaining the estimate easier. It provides context for it that can enable management to give a quicker and more reasonable answer. 

3.       The estimate must be used as part of the decision making process. If it’s not used its wasted time and effort. When things get tight the estimation will go away.

4.       Automated project estimating software tools to speed up the software cost estimation process and to enable some more what if analysis can be indispensible,

David Seaver,  Solutions Architect, PRICE Systems

I have been fortunate in my career to have been associated with some great mentors. Each individual has provided me a bit of a golden nugget to carry with me as I tried to navigate my way through the professional waters. My first “civilian” manager, after I left the service and joined industry, provided me a list of the Laws of Analysis (I had just started a position as an operations research analyst). He explained that this list was a mix of serious and tongue in cheek snippets of wisdom. I looked at the list and thought … “He must be kidding, followed by, Wow this is not politically correct!” But what I realized over time was even the farcical statements were a bit of reverse truth. I have hung this list in every office I have occupied since it was given to me. I provide it here for your consideration and entertainment. Hopefully you too will recognize a golden nugget or two:

1.       Never B.S. yourself

2.       Never use a round number – even though only one significant figure counts

3.       Never tell a small lie

4.       Answers are always more credible if printed on paper with small holes on the sides
(I know this is a throw-back to early mainframe computers and printers – so I am old look beyond it)

5.       When briefing always take someone along to share the blame

6.       Always remind yourself it’s a science

7.       Everything takes twice as long as it should plus a month

8.       The farther a system is from reality the more cost effective it is

9.       Large committees produce only large confusion

10.   If more than one person is responsible for a miscalculation no one will be at fault

11.   If someone isn’t mad at you, you’re not trying hard enough or been on vacation

12.   Never enough time to do it right – always enough time to do it over
 

I had expected to present my webinar,  “Best Practices for Cost Effectiveness Studies using TruePlanning” in early August. As you might know, I was planning to show a real world example from a recent engagement with a government customer. Unfortunately, since the Source Selection has not concluded with a downselect, I was not able to obtain the public release in time. However, for this month’s blog I will continue share some of the highlights of the webinar.

In last month’s blog we explored the uses of TruePlanning during Source Selection from the Supplier’s (or Contractor’s) perspective. This month, I would like to share with you some of the uses of TruePlanning cost estimating software from the Customer’s (or Government’s) perspective:

• Analysis of Alternatives (AoA) – Is proposed technical baseline cost-effective against other competing alternatives?
• Cost Realism – Are the performers bidding within an accurate range
• Data Driven Estimating – Are the performers bidding based on appropriate, traceable historical data points.
• Independent Cost Estimate (ICE) – Using the performer’s technical configuration, what does a completely independent look say about the performer’s bid?
• Risk Analysis – Is our bid over conservative, how much risk are we willing to take and how much cost exposure can we absorb?
  
• Schedule Estimating – Can we really do the job within the schedule constraints?
• Growth Estimating – What other configurations, materials or technologies might we consider?

While TruePlanning is useful in estimating project costs and all of the above types of analysis, I have found it most useful in developing AoAs. During Source Selection, TruePlanning provides a structured, repeatable framework that can rapidly develop cost estimates across a range of alternatives. These results are often incorporated directly into Operational Effectiveness models along with Measure of Effectiveness and Measures of Performance. In some cases, clients using tools such as Phoenix Model Center can directly link TruePlanning with optimization tools as well. 

Over the past several weeks several users have inquired about the best way to estimate costs associated with porting existing software to a new hardware environment. Normally for this situation some of the existing software will require some amount of adaptation to operate on a new server. However, a large portion of the existing software will only require integration into the new environment.  

Estimating software costs associated with the above will require the use of several cost objects:

- Systems cost object if program management, Quality Assurance, configuration, and    documentation costs are to be included in the estimate

- Assembly cost object to estimate integration costs

- Software component cost object to estimate software adaptation costs

- Software COTS cost object to model all existing software requiring only integration effort 

The main input parameter that should be reviewed with care in the system cost object is project complexity. The default value for this input is nominal. Recommend the low value be reviewed since the team size will be small.

The main input parameter that should be reviewed with care in the assembly cost object is system complexity. If the software is just be ported to the new environment, recommend a low value be selected, since the requirements definition system design has been completed previously. 

The software component input parameters will be treated in a normal fashion. All adapted software will be modeled as adapted new design, code and test. 

The main input parameters that should be reviewed with care in the software COTS cost object are software size, integration team maturity, and external integration. This cost object should be used for all software that will only be integrated in the new environment. Functional size should be used if the size is unknown. The input parameters for integration maturity and external integration must be reduced. The recommended value is 0.05, which is the lowest value accepted by the model. This value will account for much lower integration activity for the existing software.

If you have any further questions, please contact a PRICE Systems Support.