Estimating Trends

TruePlanning 2010 SR1 estimation software is now available as an upgrade for existing PRICE customers. The most significant update to this version of TruePlanning is the capability to use both parametric estimating models as well as analogous data to produce estimates. This capability validates and increases the defensibility of estimates.

TruePlanning provides a framework that allows content driven parametric models to be estimated in one system. Most notably, hardware, software, IT and Systems of Systems (SoS). No other commercially available estimating tool can make that claim. However, whereas in previous versions estimates relied on parametric models which are based on industry benchmarks, now users can also incorporate their own data for analysis. Customers can import organizational data to TruePlanning, run regression analysis on that data, and further validate their estimates. So now estimates can be backed by internal organizational data as well as external benchmark data, which is a powerful way to defend an estimate.

To view complete details of this new feature as well as other enhancements visit our New Release Info page. 

Last week I gave a webinar which detailed the PRICE perspective on Should Cost & Will Cost Management. The responses I have received have been very positive and also informing. For those of you who could not attend you can view the recorded version of that webinar here.

Below is a brief summation of that presenation and some key takeaways.

The Under Secretary of Defense issued a memo late last year. The thrust of the memo was the current need for greater efficiency and productivity in defense spending. His guidance contained 23 principal actions for improving the efficiency of the department organized into five initiatives: (1) Target Affordability and Control Cost Growth; (2) Incentivize Productivity and Innovation in Industry; (3) Promote Real Competition; (4) Improve Tradecraft in Services Acquisition; and (5) Reduce Non-Productive Processes and Bureaucracy.

One of the principals outlined in the first initiative is the idea of driving productivity growth through Will Cost / Should Cost management. In my view, Should Cost is "the realm of the possible" and Will Cost is "the domain of the probable". Simply put Will Cost is a forecast of a program’s cost …” based upon reasonable extrapolations from historical experience.”[1] Whereas, Should Cost is an analysis of what the program ought to cost given concerted efforts in economy and efficiency by the contractor. “A should cost analysis results in an, …” Approximation of a contract-price, developed by the customer’s accounting, engineering, procurement, and other costing staff.” [2]

Should Cost analysis for DoD was first applied by the US Air Force in the early 1960s. Over time this practice matured to where in 1972 it was declared “A Multimillion-Dollar Savings” in an article written by Major David N. Burt in the Air University Review (September-October 1972). Major Burt describes the evolution of the practice commencing in a thorough discussion of a “new” alternative approach. He goes on to describe a five phase approach spanning one to four or more months consisting of a very large team which includes actual on-site (contractor) investigation of contractor practices and operations. There are issues associated with both of these concepts.
 
First, a Will Cost estimate is a business as usual view which contains all of the characteristics of both good and bad production and management practices. Estimates based on Will Cost have the effect of perpetuating previous inefficiencies by providing a flawed benchmark. Secondly, a Should Cost analysis as described by Major Burt is both time consuming and costly to implement. Furthermore, Secretary Carter, in his November 3, 2010 memorandum to Military Departments and Directors of Defense Agencies declared his desire …”to establish “Should Cost” targets as management tools for all ACAT I programs… and to establish by January 1, 2011 …”’Should Cost’ estimates for ACAT II and III programs… for component MS decisions.”

Clearly Secretary Carter is serious in regards to achieving productivity growth and soon. However, given the timelines, the cost of a traditional “Should Cost” analysis, and the problems associated with a “Will Cost” approach,  how can the Military Departments and Defense Agencies meet the “beyond objectives” outlined in the November 3rd memorandum?

One solution may be utilizing a parametric approach. A parametric estimating approach will reduce the time and resources required while also providing an external benchmark of industry standards for similar systems. Given the pace with which agencies are expected to move its probably an approach many should consider.

Bob Koury
Senior Research Analyst, PRICE Systems

Today, PRICE Systems, Senior Research Analyst, Bob Koury, will be presenting on Will Cost/Should Cost management.

The presentation will focus on two main requirements mandated in the Ash Carter memo (mentioned here several times): Developing Should Cost/Will Cost targets and establishing Affordability as a requirement.  An example will be provided of how parametric estimating models were used to establish “Should Cost” targets and how they can be used by a budget authority (government or Industry) to be an informed consumer of contractor or sub-contractor bids. The demonstration portion of this webinar will focus on a process for using a parametric estimating model to produce the reasonable affordability targets.

To sign up for the webinar go here: https://www2gotomeeting.com/register/142598427

In last month’s blog I wrote about Ash Carter’s (Under Secretary of Defense for Acquisition, Technology & Logistics) Memorandum for Acquisition Professionals, Better Buying Power: Guidance for Obtaining Greater Efficiency and Productivity in Defense Spending (14 September 2010).

I concluded the TruePlanning unified framework and comprehensive cost models, is a tool very well suited to provide the types of analysis outlined in the memorandum. In terms of Should Cost and Independent Cost Estimates (ICE), TruePlanning estimation software provides the industry standard capability to conduct Should Cost and calibration (actual program history) for ICE. Most interestingly and to the point of the Carter memo, TruePlanning has the capability of breaking the “self fulfilling” prophesy of business-as-usual.

On November 3^rd, Ash Carter issued a follow up memorandum outlining specific actions to implement the September 14^th Guidance. For this month’s post, I wanted to point out some specific comments about Milestone A and B actions and specifically how TruePlanning can help.

At Milestone A the memorandum specifics establishment of an “affordability target to be treated by the Program Manager (PM) like a Key Performance Parameter (KPP)…This analysis should show results of capability excursions around expected design performance points to highlight elements that can be used to establish cost and schedule trade pace.

At Milestone B, the memorandum requires ”a system engineering tradeoff analysis showing how cost varies as the major design parameters and time to complete are traded off against each other.” Thus, Integration between parametric cost models and performance models is needed to allow full exploration and optimization of the trade space, treating cost as another design parameter

Did you know that TruePlanning through its integration with Phoenix ModelCenter is a powerful tool to provide exactly this type of analysis? ModelCenter contains a native plug-in to interface with TruePlanning directly. Through the ModelCenter interface, analysis can easily perform Design of Experiments (DOE), Carpet Plot Analysis (trading performance KPP’s against cost) and Optimization Analysis.  If you are interested in seeing some examples, especially system engineering trade off analysis, please get in contact with me and I will send you a few presentations.

In his August blog-entry here, Zach Jasnoff outlined typical client perspectives for the different types of analyses that TruePlanning can accommodate. Working on a large project, we’ve experienced situations that, realistically, can happen where the initial intent and model structuring later have the boundaries of model appropriateness stretched. An Analysis of Alternatives (AoA), for example, is meant to measure deltas between baseline and its alternatives. If common costs “wash” then they can be excluded… which becomes an issue when treated as a Rough Order Magnitude for customer budgeting. 

Likewise, if a ROM or Independent Cost Estimate (ICE) of acquisition costs is extended to Program Lifecycle Costs, significant gap analysis is in someone’s future. TruePlanning can handle these tasks. But obviously it’s always best to establish use of an estimate and its underlying model, beyond initial application. TruePlanning has enormous flexibility. We analysts endeavor to do the same.  Clients, particularly those new to parametric cost estimation, just need coaching and synchronization up-front sometimes. They may think they have “the estimate” when it only covers certain configurations, activities, phases or cost-categories. Expanding a model is simple enough; creating and defending changes can be tough for the customer internally. 

I’m continuing to realize that when in doubt, make it clear what an estimate & model is meant for, and what it’s not.

John Swaren
Solutions Consultant, PRICE Systems

Recently a cost research project on missiles was completed. The research resulted in performance based equations for air-to-ground and surface-to-air missiles were developed. The performance based equations can be used for early concept estimation  on missile development and production costs. The question though is “What is the process for developing this type of estimating relationship?” This will be the first of a series of BLOGs on this topic. 

The first task is to define what is a “Performance Based Equation?” Bruce Fad covered this definition in a previous “Data Driven BLOG” so please review his post for the details. The second step is data collection of not only cost and technical data, but also collection of data for potential independent variables. I would recommend collecting data for several independent variables. This avoids starting over with the data collection effort, when the initial independent variable does not correlate with the dependent variable. In the case of the missile data, various potential independent variables were identified: Range, speed, payload, guidance system, and propulsion system. The independent variable data was not used for the next step. Rather the next step was to set up a calibration TruePlanning input file. The calibration results were for manufacturing complexity and were ported over to MS Excel for completing the analysis. This will be discussed in a follow-up post.

Jim Otte
Solutions Architect, PRICE Systems
 

Unfortunately this is a true story.
 

I decided to have a couple of gold fish in a nice glass bowl, having done my sums I estimated the cost should be:

• Bowl £20.00
• Gravel £5.00
• Two gold fish £8.00
• Food £4.00

Making a total of £37.00, ok I can afford this, off I go to the pet shop.

The pet shop advised “fish don’t like bowls”, plus “it’s cruel to keep them in one”, but as a child I often won fish at the fun fair and they lasted years, so undeterred off I went and purchased a bowl and 2 lovely gold fish from somewhere else. All is good.

After two weeks I watched one gold fish do a fantastic somersault and then die, off to the pet shop and buy another fish, £4.00, not too bad.

Two weeks later while cleaning the much-cherished bowl it shattered and cut my finger so badly I end up lying on the kitchen floor, finger in the air to try and stop the flow of blood, the fish at this point are swimming in an old sandwich box.  Obviously needing help I phone my son to buy me another bowl, he arrives with an all singing all dancing tank, daylight, moon light, filters the lot. Cost £50.00 with the added insult of “You shouldn’t be trusted with glass or sharp objects”.
 

Next day my finger has swollen so badly I attend the A&E department to have my gold ring cut off, cost to have ring repaired £40.00.

All in all what should have cost £39.00 has now cost £131.00 but hey my gold fish now live in a palace! And yes, I do talk to them.

I do wonder what parametric cost estimating could have taken into account the cost drivers, risk and benefits of this venture.

By K.T. Price

UK

I was recently struck by Ash Carter’s (Under Secretary of Defense for Acquisition, Technology & Logistics) Memorandum for Acquisition Professionals, Better Buying Power: Guidance for Obtaining Greater Efficiency and Productivity in Defense Spending (14 September 2010). Within this broad sweeping memo, Ash Carter outlines 23 principal actions in five major areas aimed at increasing efficiency in Defense acquisition.  The first major area covered is “Target Affordability and Control Cost Growth”.

Within this major area, program managers must treat affordability as a requirement before milestone authority is granted to proceed (starting with Milestone A). This means developing affordability targets and treating cost as a Key Performance Parameter. 

What I find really interesting is the critique of Will Cost vs. Should Cost. The memo is critical of Will Cost, or Independent Cost Estimates (ICE). As Dr. Carter points out, “the ICE reflecting business-as-usual management in past programs, becomes a self fulfilling prophesy. The forecast budget is expected, even required, to be fully obligated and expended.” To combat this “vicious cycle”, the memorandum now requires “each major program to conduct a Should Cost analysis justifying each element of program cost and showing how it is improving year by year or meeting other relevant benchmarks for value.”

In my career experience, parametric estimating models such as TruePlanning play a major role in targeting affordability and controlling cost growth. In terms of affordability analysis, TruePlanning contains a unified framework of all elements of program cost (hardware, software, IT and Systems) and has built-in capacity to interface with engineering optimization tools such as Model Center. Through this interface cost can be treated as a Key Performance Parameter (KPP) for optimization and engineering trade off analysis.

In terms of Should Cost and ICE, TruePlanning provides the industry standard capability to conduct Should Cost and calibrated (actual program history) for ICE. Most interestingly and to the point of the Carter memo, TruePlanning has the capability of breaking the “self fulfilling” prophesy of business-as-usual. Using a calibrated TruePlanning model for ICE, estimators can change key engineering and programmatic parameters and see the impact on cost. For example, parameters such as requirements stability, engineering complexity and team composition can be quickly changed to assess a new program’s reality while still taking into account past performance history.

As the Carter memorandum points out … “the ability to understand and control future cost from a program’s inception is critical to achieving affordability requirements.” Because of TruePlanning unified framework and comprehensive cost models, it is a tool very well suited to provide the types of analysis outlined in the memorandum.

Zach Jasnoff
Solutions Architect, PRICE Systems

Margaret Wolfe Hungerford in 1878 in her book Molly Bawn coined the phrase …”Beauty is in the eye of the beholder”. This concept of the “value of beauty” has been expressed by others such as:

 

 Benjamin Frankin in Poor Richards Almanack 1741 when he wrote;

                “Beauty, like supreme dominion

                Is but supported by opinion”

David Hume in Moral and Political 1742

“Beauty in things exists merely in the mind which contemplates them.”

So what does this have to do with Cost Benefit? Well Merriam-Webster dictionary defines benefit as something that provides useful aid. Inherent in the term “useful” is the idea of value. Value in the context of useful only becomes meaningful when someone considers or contemplates it as being valuable. The degree to which something is evaluated as useful becomes a mental process of a stakeholder or user of the item. You might say that value is in the eye of the user! This becomes important in our efforts as cost estimators especially when we are involved in cost-benefit analysis. The mental evaluation of the stakeholder assigns meaning to the cost we calculate. Payback, and return on investment are related to this same mental evaluation. These kinds of metrics are a way of quantifying a stakeholders opinion about value and the usefulness of a program, project, or system. We as cost analysts should never lose sight of this truth.

Bob Koury
Senior Cost Research Analyst, PRICE Systems

The midterm elections are finally over. The themes of reduced spending and lower taxes showed up in force at the ballot box. But what does that mean for the defense industry?

The U.S. Secretary of Defense, Robert Gates, caused quite a stir when he announced his proposals for reigning in defense spending. There are the expected assortment of eliminations (U.S. Joint Forces Command and Business transformation Agency to name two), reductions (in service support contracts, number of senior civilian executive and general/admiral military officers, and funding for intelligence community advisory contracts), freezes (of automatic replacement of departing contractors with full-time government personnel and oversight reports), and consolidations (IT assets). All of these have created so much emotional reaction that I almost missed one component of his proposal that looks like an adder – to attach a cost estimate to every new initiative proposed hereafter. It’s hard to find more detailed explanation on this. I’m reading it as: let’s figure out what our ideas are likely to cost us before we fall in love with them. And that sounds like a fiscally good idea, as well as one that needs a community of smart cost estimators, capable of producing meaningful cost-benefit analyses.

After some recent meetings with clients I am sensing some confusion on how to estimate software reuse. I think part of the problem is in the definition of reuse, so let's start with a definition and then address the estimating issue. Software reuse is defined as “the use of existing software, or software knowledge, to build new software.” This definition came from Wikipedia. From a estimating software costs perspective the above definition is part of the problem. The definition should read: "Use of existing software with no changes for operation in the new software program.” 

If the existing software is going to be changed, re-written or modified to operate in the new program, it should be modeled as software adaptation. In this case the software will require some amount of re-testing, and software integration. Software that is being re-used has been fully tested during the original development program. In this case the reused software does not have to go through full up integration and testing. Rather it will go through a regression testing (computer aided testing) to insure it is still operating correctly.

If the software is going to be re-used as is, then clients should use the TruePlanning  “Software COTS” cost object to model and estimate. The purpose of modeling is for integration control. External integration should be set to a value of 0 to 1.0. A value of zero implies that there is no additional integration effort. Values less than one reduce the amount of model calculated software-software integration effort. The reused software can be modeled as a software component. However, do not include software adaptation or new software with the reuse data. This will overstate the integration effort. 

Some of us remember taking the Iowa tests during our early school days. The Iowa Tests of Basic Skills (ITBS) are standardized tests provided as a service to schools by the College of Education of The University of Iowa. The tests, administered to students in grades K-8, became a national standard for measuring scholastic aptitude – I was educated in Pennsylvania.

Now out of Iowa comes another test of sorts, something called an Integrity Index Score based upon a proprietary algorithm of an organization called Iowa Live. Iowa Live calls itself, “a growing network of volunteer Citizens & professionals for improving Iowa.” The organization has addressed a variety of issues in the education, health, and government areas. In July, Iowa Live published its comparison of two commercially marketed parametric estimating software products, one of which was PRICE TruePlanning. The comparison was for both estimation of software projects and estimating hardware. If you’d like to see the results, visit the Iowa Live website, where you can also learn more about the theory and derivation of the Integrity Index.  

Bruce Fad
VP Professional Services, PRICE Systems

When I glanced at the Washington Post on Sunday, the following headline screamed out:

Defense cuts could slow D.C. economy for years

The article basically covers how Defense Secretary Robert M. Gates is calling for reducing spending on "support contractors" by 10 percent each of the next three years as the Defense budget shrinks. As Washington DC is a hub for these types of companies, the impact is expected to be significant. According to the article, more than a quarter of national defense spending contains of outlays for service contracts. Among the largest companies affected are CACI, SAIC, Lockheed-Martin, Booz Allen Hamilton and ManTech.

As the Defense budget shrinks and more companies are chasing fewer projects, the need for greater accuracy in cost estimating increases. At the same time, the WSRA adds 20,000 new positions in cost estimating, contracts and oversight.

This is an interesting conundrum, cutting DoD service contracts 10% over three years while the government is adding 20,000 new positions in cost estimating, contract and oversight

One year after the WSRA was signed into law, an interesting observation from the National Defense Magazine Blog:

Nancy Spruill, director of acquisition resources and analysis at the office of the undersecretary of defense for acquisition, technology and logistics points outs “The long pole in the tent is the cost estimates,” she said. “There's a lot of programs that need cost estimates as they're moving through the process today. … Doing additional ones has been difficult, especially a while ago, when they didn't have the staff.”

Thus, in an era of shrinking defense budgets, greater oversight and more competition for fewer projects, parametric cost estimating models such as TruePlanning can play a major role in providing accurate cost and risk estimates for both government and contractors. This is especially true in Source Selection where determining the most cost-effective offering is critical. For more details on using TruePlanning for Source selection, see my previous two blog posts.

Ahhhh, the 80s… a challenging (but often confusing) time in an evolving computing world.  Working in 1985 as a software estimator as well as SQA engineer in a quality assurance department that “audited” real-time projects using new concepts like OOD & OOP… well, you get the picture.  It was a great time to get immersed into great work.  And the good news:  that company’s process as well as its developers were bullish on a young estimation/ quality types asking plenty of questions… as long as they were of the Yes-No variety.  And ask I did.  Writing and using those checklists was great OJT, above & beyond adding their value of verifying/ enabling healthy development activities. 

Less than two years later, SEI’s Walt Humphrey formalized his Capability Maturity Model with its five levels of software process maturity.  These “CMM” levels are still used by many organizations and estimating tools, including within True Planning’s “Organizational Productivity Calculator.”  Could the CMM assessment be supplemented to include cost-driver questions for purposes of parametric estimation?  It worked in my SQA audit days. 

In fact many estimators (if only because they can’t get on an SME’s schedule twice) take the opportunity to ask qualitative and quantitative questions.  Integrating both lines into one interview should still be received as a healthy value-add.  Whether estimation is resident within a quality or financial function is moot.  We have the entrée to support both costing/ planning and productivity assessment.  A good example of an integrated approach is the David Group’s Project Profile Worksheet, cited in the text “Function Point Analysis” by David Garmus & David Herron.  Less comprehensive checklists are likely out there too, including methodologies proprietary to individual companies. 

The thought today is that the future is now and structured approaches, called audits or otherwise, will enable good practices as well as illuminate new approaches too.  In Wall Street, regulations are only good in controlling past (known) bad behaviors.  In just as creative Software, process rules help developers maximize their productivity towards current & future good behaviors.

The key cost driver when estimating software costs is the size of the product. The problem is that there is no perfect technique available to measure and quantify the size of software. The two major techniques in use today are Source Lines of Code and Function Points.  Today we will talk about Source Lines or Code or SLOC.

Source Lines of Code measures logical lines of code. It takes some of the uncertainty out of physical line of code measures by counting only complete statements (which can cross over more than on physical line). SLOC excludes comments and blank lines.

SLOC has several advantages:

·         Easily counted by automated tools, which can be configured with an organizations definition to produce consistent counts. Link to free tool from USC is provided below.

http://sunset.usc.edu/research/CODECOUNT/download/2010/UCC_Release_Notes_v.2010.07.pdf

·         Provide visibility into technical progress as projects proceed through design, code and testing.

·         Can be used to develop historical databases of past project size, which enable the more accurate prediction of size of future similar projects.

·         Design, Code and Test change to modified software can be specifically quantified

SLOC has several disadvantages:

·         It’s very difficult to predict SLOC on new programs prior to design

·         Different programming languages will require different amounts of SLOC to implement the same functions. This makes summary measures of size on multi-programming language products difficult

·         SLOC is difficult to map back to product features or requirements. Which makes usage of EVM difficult to apply to SLOC based SW projects. 

·         SLOC is a meaningless number to non Software Engineers (without detailed explanation)

Over the past several weeks several users have inquired about the best way to model legacy software that is being modified when estimating software costs. The software component within the TruePlanning Software model has an input parameter call “adapted Code Size.” This input parameter accounts for existing or legacy software that will be modified or changed to meet a new requirement. Tied with the size input parameter is Percent Design/Code/Test adapted. Although the model will calculate a percentage for each input, I would recommend that user’s analyze the calculate values and override the calculation where required. The percentage for each parameter is to reflect the amount of modification being accomplished. For example, calculated values below five percent or above forty percent are probably unrealistic. 

Values below five percent reflect one single function being modified, which is normally not the case. Values above forty percent generally result in the entire software package being re-developed. Hopefully this will clear up this software input parameter. If you have further questions, please email me at james.otte@pricesystems.com

Jim Otte
Solutions Architect, PRICE Systems