PRICE Systems

The GAO just published a report saying that strong leadership is key to planning and executing stable weapons programs. And, evidence is presented to back the claim, including result from study of a subset of the 21% of the 2008 defense programs that were deemed stable - on track with original estimates of cost and schedule. What kind of strong leadership made these programs stable? Things like experience, continuity, and open and honest communication, knowledge-based planning, disciplined execution of plans, and establishment of realistic cost and schedule estimates that account for risk are all cited. 

Here’s what bothers me about this when I think of it in the context of WSARA (Weapons Systems Acquisition Reform Act of 2009) – what’s in the act to increase the incidence of strong leadership on weapons programs? I don’t see anything, and I’m aware of the planned 20% increase to the stable of acquisition professionals in DoD. That’s good and may result in more strong leaders, but in 20 years. Problem is, we need them now and even if a handful are scarfed up in the 20% influx, they won’t be in a position to lead. For now, keeping Mr. Gates in charge of DoD is probably our best hope for more strong leadership – he has shown the willingness to replace those who don’t exhibit the strength we need for stability. I just hope he has enough bench strength when he needs to make a replacement.

Bruce Fad
VP of Professional Services, PRICE Systems

This week I was thinking how useful the Export Import feature in TruePlanning can be. First, the Excel Import spreadsheet gives you an easy and convenient way to gather your data. In addition, it gives you an easy and convenient way to check and validate the data. When observing the data in a column format, it is so easy to spot and correct anomalies.

Second, the Excel Import spreadsheet gives you an easy and convenient way to build your Product Breakdown Structure. No more inserting one cost object at a time. The Excel Import feature does it all for you.

Third, the Excel Import spreadsheet supplies all the necessary parametric inputs for your estimate. No need to type one at a time into the value field of the Input Sheet. And the bonus is that TruePlanning automatically calculates the cost. 

In addition to supplying inputs for the cost objects, the Excel Import spreadsheet will also create, populate, and assign Worksheet sets and Escalation tables.   How easy can it be?

PRICE Systems supplies pre-formatted Import Templates available to users. Just contact PRICE Systems (info@pricesystems.com) and we will e-mail them to you, along with directions for their use. Happy estimating!!          

Now that we know the background on the original concept of TRL's (Technology Readiness Levels - reference Arlene Minkiewicz's earlier blog post), we now want to address estimating costs associated with different TRL levels. It is important to realize that a model cannot estimate TRL costs by simply changing an input parameter. Rather the only way to estimate costs associated with different TRL levels is to model the scenario. For example, if you are estimating costs for TRL level 2 phase the input parameters would be very different than if estimating costs for TRL level 6. 

PRICE Systems has prepared a white paper, along with web presentation on just this topic. The web presentation can be found on PRICE University, and is available to all clients. If interested in the white paper, please contact Amanda Rivera, at 856 608-7201 or send an email to info@pricesystems.com with "TRL Paper" as the subject.  

Jim Otte
Solutions Architect, PRICE Systems

Currently we are exploring the best approach to including a more comprehensive cost estimate for Total Ownership Costs (TOC) into TruePlanning. The current version of the software has focused on development and production costs with some life cycle costing including. The life cycle costs included are focused on the system specific O&S costs such as initial spares for priming the supply pipeline, maintenance, replenishment spares, etc. It is a system view as opposed to a program view of TOC.

As we better understand the need to conduct affordability studies it has become clear that design decisions which can be assessed in regards to their total ownership cost will result in more affordable systems. In order to do this we need to include a greater range of cost elements such as construction of system support facilities, O&M of infrastructure and system facilities, some personnel cost which are enterprise level not necessarily systems specific and so forth. What is not clear is how these elements differ from different customer perspectives.

For example, our current lifecycle costs have an original equipment manufacturer (OEM) flavor to it. Whereas I speculate that a PEO, PM, or service staff analyst might have a need for a broader set of elements. Feedback in this regard from our customers would be very beneficial.

Bob Koury
Senior Cost Research Analyst, PRICE Systems

One of the great features of the TruePlanning cost management software is the fact that it makes it easy to handle complications of inflation and estimating projects performed in different countries and currencies. The costs associated with doing work in different countries, and the relative value of different currencies is constantly changing. To address this, the cost research team at PRICE does an annual economic update performed by the cost research team, and this blog will introduce some of basic concepts and research that goes into maintaining this feature every year.

The price of goods and services changes over time, and this value is measured by inflation rates (if prices have gone up) or deflation rates (if prices have gone down). These inflation/deflation rates are constantly updated for many different currencies by the International Monetary Fund (IMF). In TruePlanning, we use past and predicted inflation rates published by the IMF  to determine the cost in today’s currency of doing work in the future, for whatever currency you want to use. Within the TruePlanning tool, these factors can be applied to help produce a cost estimate of work that takes place over a long period of time, with the cost estimate shown in today’s currency values.

In addition to inflation/deflation rates for different currencies, we also recognize that the price of goods and services may be different from one country to the next (i.e. a U.S. Dollar exchanged and spent in India will buy more haircuts than a dollar spent in the United States). The degree of these price differences can be measured by looking at Purchasing Power Parity (PPP) data of a country. Standard PPP values are calculated or compiled and published by the Organization for Economic Co-operation and Development (OECD). The PPP rates published by the OECD are used in the TruePlanning tool to estimate costs of performing projects in different countries, as the cost of doing the same amount of work in different countries can vary significantly.

By automating the application of these and other economic complications of cost estimating, the TruePlanning tool helps create a process in which the effects of these economic factors can be applied consistently and correctly for all your cost estimates.

Last month I blogged about the importance of cost realism, its roots and how as estimators we must always reflect the truth, no matter how unpopular. This month I want to share with you a recent experience on a Source Selection. As part of the Source Selection team, my role was to conduct a Cost Realism estimate on each of the performers submitting bids. I want to share with you a few insights from that experience.

One of the first rules I always follow is to never ask engineers to provide data that you can readily find. I rarely will use parametric data forms, as you will get to be unpopular fast with those you task with the honor of filling them out. Rather, I dig into the technical and cost volumes to derive the configuration, technology, weight statement, rates/overheads and other juicy information to populate TruePlanning.

Once I have the TruePlanning cost management software populated with all of the data I can derive from existing documents, I will then ask the subject matter experts in each area to discuss the other qualitative/quantitative factors about each performer. I can usually guide this conversation to derive inputs such as requirements stability, engineering complexity, integration and others. I find it effective to hold this meeting remotely using Go-To-Meeting so everyone can see my desktop displaying the TruePlanning model and no one has to leave their desk. When inputs are critical cost drivers, I usually pop up the wizards or generators so the engineers can see the choices available.

Once the model is fully populated and all inputs agreed to, I will then produce a very well documented estimate with all assumptions as the final deliverable. In this particular case the Source Selection team was very pleased with the result and asked if we could extrapolate a new configuration based on some technical changes proposed by the performer.

Using our well qualified TruePlanning model, we ran the extrapolation and reported to the Source Selection team that we expect to see a large impact. Initially, no one believed the results; it could not be so they said! However, several weeks later, when the performer’s new estimate arrived, we were within 1% of the revised bid. During this time period, the Source Selection team was able to successfully prepare a negotiating stance based on the expected bid coming in as predicted.

In all my years as an estimator, I have only seen estimates go up as more is known about the technical configuration. Bottom line, as estimators we are the “front line” for telling management the hard to hear truth. In this case, at least they were prepared!

Earlier this week I conducted a webinar intended to make PRICE users aware of the Cost Research Services available to them as part of the license fee they pay to use PRICE products. I thought I would recap the highlights of this webinar for those of you who might have missed it.

At PRICE we understand that cost estimating tools, while useful and valuable, do not always present the complete solution. Every single cost estimation projects presents new and unique challenges.  We think it's important that in addition to solid, time trusted cost estimating models, our users have access to the many years of experience we have as seasoned costestimators, subject matter experts and operations researchers.

This service is nothing new.  For the 30 years that PRICE has been in existence, we have worked as partners with ourcustomers to optimize their use of our models and methodologies.  This was just an opportunity to formalize the offerings and remind the community what services are available.

So what does the Cost Research team at PRICE have to offer the cost estimating commmunity?  On average our researchers have more than 24 years of experience with hardware estimating, software estimating, operations research or some combination of the three. We are constantly engaged in cost research projects addressing market needs as they arise.  The results of these studies vary depending on the need they attempt to address.  In some cases, data collection indicates custom models should be developed.  These can be developed and deployed in TruePlanning, the flexible cost estimating framework.  Some results are published as updates to tables or calculators in the PRICE Software and Hardware cost estimation models.  White papers, webinars and the PRICE blog are all means we use to communicate the results of cost research studies.  

PRICE's Cost Research Team is available 24/7 to address users cost estimating question on an as needed basis.  Some issues require more attention than a single phone call.  Users are encouraged to schedule working sessions with one or more of our analysts to take a deeper dive into cost estimating issues that perplex or intrique them.

Some areas the team is currently studying include Total Ownership Costs, Joint Confidence Level, Performanced based models for technologies such as FPGAs and ASICs, semi-rigid cables, and  Operations and Support costs for space systems. 

The most important thing the cost research team at PRICE wants to do is make our clients better estimators while adding value to the cost estimating community as a whole.  We can't do this without input from our clients.  Share your cost estimating challenges with us.  Call, email or comment on our blog.  856-608-7201 or info@pricesystems.com


 

From my perspective as a cost researcher, the calibration tool is one of the most powerful analysis capabilities built into the TruePlanning cost management software . One way I can use this tool is to go back to an old estimate for a project that is now completed, and analyze the correctness of the previously entered input values. With this analysis, I can find ways to improve our methods of soliciting input values from the user to ensure the best values are entered the next time. This way, the TruePlanning models keep getting “smarter” as new information becomes available.

TruePlanning models calculate a cost estimate based on a set of inputs about the project being estimated. After the project is completed and we have actual cost data available to us, we can go back to TruePlanning and explore the various reasons why an estimated cost may have been different from the actual cost.

Let’s say we are estimating a project involving a brand new electronic technology, and we are uncertain of the values we selected to describe its complexity. The calibration tool allows us to enter the actual costs of the project, and reverse-engineer the value of an input we were uncertain about (i.e. manufacturing complexity for electronics) that would lead to the correct answer. This allows us to identify incorrect input values so we can identify problems, then review and refine our methods used to pick the value in the first place. 

Over the years, we have used this (and many other) TruePlanning analysis capabilities to further our understanding of our cost estimates and our cost estimating methods. There are many analysis tools built into TruePlanning, and many ways to slice and dice information about a cost estimate. This makes it quick and easy to standardize the cost estimation methods used, and continually review and refine our methods to get the most accurate cost estimate possible. With more analysis tools and more information becoming available all the time, TruePlanning will keep getting smarter.

I am excited about the efforts described in Derek Kaufman’s article on the AFMC website.  For the complete story please see this link http://www.afmc.af.mil/news/story.asp?id=123202181. The DoD is investing in the rapid build up of a new foundation of acquisition workers focused on estimating costs and negotiating prices through continuous learning. 

Mr. Assad notes that Wright-Patterson Air Force Base, Ohio, was once the preeminent DoD recognized leader at estimating costs and negotiating prices with defense contractors. It's a skill that has been allowed to atrophy, he noted.   "We need to build that pricing capability back up," he said. "It's extremely important for us to have world-class pricing talent, so that we make sure that we get a reasonable deal for our taxpayers."

For over 30 years, PRICE Systems has passionately supported this same vision.

Every year we train hundreds of analysts and acquisition professional across the globe. We proudly offer training on topics such as source selection support, software cost estimating fundamentals, data collection and total ownership costs.  And recently we introduced an on-line acquisition training center, known as PRICE University, where students can gain continuous learning credits through self paced study, right from their own desktops!

If you haven’t had the chance to hone your skills as Mr. Assad suggests, I invite you to start today by taking a quick tour of PRICE University - http://university.pricesystems.com/index.jsp

I have been frequently asked to do crosschecks on other people’s software cost estimates which are potentially done in a variety of tools from spreadsheets to SLIM.

One of the common operator errors I see from other users is not understanding what activities and resources are included in the outputs of the particular tool that they are estimating with.

This is akin to deciding between two cars and not knowing if both come with the same sets of features (stereo, AC, heated seats).   With software estimation tools you need to know what work is getting estimated by the tool. (Requirements, Design, Code, Test, Documentation, System Test) and what resources are included in the effort estimates to accomplish those task/activities. (Programmers, Analysts, Architects, Systems Engineers, Testers, Tech Writers, Project Managers, QA and CM)

A common mistake I see is generating an estimate with a tool then adding in Program Management effort and cost when the tool utilized already includes those activities and resources in the estimate.

That’s why I drive a pearl black TruePlanning 2 door coupe.

With so many acquisition programs over budget and behind schedule, the term “Cost Realism” is suddenly very popular. In my experience as an estimator on many major acquisition programs, two things have remained certain over years (besides death and taxes). First, the probability of the program ever achieving the original cost estimate is exactly zero and second, the more information that is known about a program, the more it will exceed its original cost estimate.   

With that said, the move to Cost Realism is so important because it recognizes these two fundamental facts. Much of the interest in Cost Realism is driven by the Weapon Systems Acquisition Reform Act of 2009. According to US Senator Carl Levin “Report after report has indicated that the key to successful acquisition programs is getting things right from the start with sound systems engineering, cost estimating, and developmental testing early in the program cycle. The WSARA also calls for a “Director of Independent Cost Assessment to ensure that cost estimates for major defense acquisition programs are fair, reliable, and unbiased.”

If we go a bit further and look at definition of Cost Realism in the Parametric Cost Estimating Handbook we find  that “no one expects a cost estimate to precisely predict what a hardware or software project costs or a time and material service will cost. So, cost realism is not about the exact cost estimate. It's about the system of logic, the assumptions about the future, and the reasonableness of the historical basis of the estimate. That is, it's about the things that make up the foundation of the estimate.”

So while we recognize that at its core cost estimating can never be exact, cost realism seeks to ensure estimates are closely tied to the fundamental realities of the program as we know them at the time, without regard to politics or undue optimism.   As I finish this blog, I’m struck by the line in a Few Good Men “You can’t handle the truth!” As estimators, we need to always reflect the truth, and that is what cost realism is all about.

Software project falures coupled with rapidly changing business needs are forcing organizations to revisit the way they go about buiilding software.  Agile development has emerged as one possible solution to the woes of the software industry.  Agile enthusiasts claim significant increases in productivity and quality, while detractors cite instances where the reverse is true.  It seems to me that probably both are right  - some of the time anyway. 

Agile means many different things to different organization.  There is a long list of agile tenets but not every method of agile applies all of them.  And some organizations have cultures which adapt well to agile methods while others don't.  All of these things affect the 'success' of applying agile practices to your organization.

Personally I don't think that most agile practices inherently improve productivity.  The long term application of agile within a cohesive development team should definitely improve their productivity but this would probably be true of the same team if they were applying some other philosophy.

I do, however  think there are agile practices specifically focused on improving the quality of the software that is delivered.  My list follows.....

Test Driven Development  No code is written until there is a test.  Business Analyst build tests that coders use to determine if the code meets the requirement.  Only enough code is written to make the test pass.  As code is refactored (improved) over time, the test remains to ensure that subsequent changes do not degrade the initial requirement.

Continuous Integration and Automated Testing  Builds are run with each change to the code base or at regulalry scheduled intervals during the day.  Suites of automated tests are run against each build.  When tests fail making them pass because a number 1 priority of the test

Pair Programming  All production code is written by two coders  one at the key board and the other navigating, correcting and thinking of better solutions.  Sort of like just in time peer reviews.  Pairing occurs regularly throughout the development - with no set pairs but rather pairs that make sense at the moment.  Driver and navigator roles should shift often. 

Customer involvement  Customers (or their surrogates) actually participate on the development team helping the business analysis develop the right tests and testing and reviewing the frequently released versions of the software

While these practices are included as tenets of agile, a shop need not be 'agile' (in the purest sense of the word) in order to incorporate one or more of these quality focused practices into their software development processes.

To read more about Agile practices and software quality check out my article in Software Tech News 

In September of 2009 the United States Government Accountability Office (GAO) submitted a report[1] discussing the lack of robust Analysis of Alternatives for weapons systems. The report indicated that …

“Cost, schedule, and performance problems in the Department of Defense’s (DOD) weapon system programs are serious.

Why is it that DoD weapons programs experience a simultaneous cost growth and performance degradation? I believe the answer is found in unrealistic cost estimates and schedule estimates mostly driven by pressure to win a program within a certain budget constraint. Excessive requirements change either through poor requirements development or an ever growing list of wants are also contributing factors. This problem is best address by a systematic affordability process which incorporates cost estimating, schedule, technology risk, and performance.

The first time I remember hearing the term, “data driven estimating,” was at an ISPA/SCEA conference in Phoenix about 6 years ago. I thought I had a pretty good concept of what that meant. Then, I started hearing it more frequently about 6 to 9 months ago, and I wasn’t as sure of my understanding. So, I went to the Internet and Googled, “Data Driven Estimating.”  The first 3 pages yield links to statistical, engineering, and technical estimating founded on data; e.g. packet routing, data delivery rate, radar and rain gage data, spectral density, Markov chains. A few relevant hits surfaced, but none really define what data driven cost estimating means, with one possible exception: “Lessons Learned and Results from Applying Data-Driven Cost Estimation to Industrial Data Sets,” a presentation to the 6th International Conference on the Quality of Information and Communications Technology (QUATIC 2007). What did it say about data and estimating? This: “….few estimation methods can deal with imperfect data.” Wow! That’s not real encouraging.

There was one other hit that I found interesting, probably because it was the closest thing I saw to my interpretation of data driven estimating. The site is: MyBusinessMusings (MBM); http://www.mybusinessmusings.com/. Here’s the part that connected with me:  “In order to become data driven, business leaders need real-time, actionable data in a usable form. That’s how you empower a significant cultural shift: provide the right information at the right time and in the right format.  Anything less doesn’t enable the data-driven organization. …. We all want to be data driven, but the cost of usable data may prohibit…..”

I see a connection to parametric estimating – here it goes: the data we have for cost estimating is imperfect, and the cost to make it at least readily usable by traditional accounting methods at the right time is prohibitive. Mathematical modeling of traditional accounting (parametric estimating) is the best response to the imperfection situation. That’s the way I see it.

Despite the plethora of literature on Technology Readiness Levels (TRLs) it remains a difficult concept.  I thought I would share my interpretation,

For most of us the concept of technology readiness is hard to grasp. This is because in general, our experiences with technology are with fully matured technology. In 1961, President Kennedy challenged US scientists, mathematicians and engineers when he announced that within the decade of the 1960s the US would ‘land a man on the Moon, and return him safely to Earth’. At the time, there were no solutions to solve problems such as reaching earth’s orbit and traveling to the moon, let alone for how man would be able to survive in space. Some very smart people and creative thinking was needed to invent solutions that didn’t exist in order to make Kennedy’s dream a reality.

One of the many things that these smart people discovered was that programs envisioned when the technology is very new or non-existent are much harder to plan for than programs using technology that has been used successfully on other programs. As the possibilities for the space program greatly outsized the budget for space programs this fact became increasingly problematic. In 1974 Stan Sadin of NASA developed Technology Readiness Levels as a methodology to assess the technology readiness of a proposed spacecraft design. Eventually the methodology was institutionalized by NASA and later similar methodologies were developed by the US Department of Defense (DoD) and other organizations responsible for the acquisition of complex aerospace and defense programs. TRLs are used by NASA and the Departments of Defense worldwide to support go/no-go decisions at various acquisition milestones.

Here's how I like to simplify TRLs.....

Level 1 => Back of the napkin sketch
Level 2 => Idea confirmed as both good and useful
Level 3 => Idea proven possible
Level 4 => Idea proven to present a realistic solution
Level 5 => Alpha version of technology
Level 6 => Beta version of technology
Level 7 => Release candidiate ready for operational test
Level 8 => Technology has gone gold
Level 9 => Technology used successfully in target environment

The newest version of TruePlanning has been released and distributed to customers. The new features were designed to make it easier to estimate entire systems, not just individual components  or sub-systems. TruePlanning is an excellent solution in this regards.

Systems that require you to estimate software costs, hardware costs, and the integration of multiple pieces of each can all be done in one framework. There are  specific features in this release, like input inheritance, that allow you to make changes at the top Project level which then flow throughout the entire system.  This saves an estimator or an engineer lots of time as typically they have to dedicate time to making sure that there is consistency from the top level of a systems throughout.

Other enhancements include improved help and guidance. While using TruePlanning’s built-in calculators to come up with good inputs, you are also informed through pop-up messages and alerts  about what inputs need to be entered or should be entered based on what you have already told the software. We’ve added more charts and graphs that give you visual clarity of how specific inputs of the models effect certain estimated costs. There is also a new Help System that is easy to navigate so you can  locate information that can help answer questions you may have as you are building your estimate.

For more information visit our New Release Info page.  

Here are some choices to the question:

1>By comparing with old analoguos data ?
2>Creating and presenting the basis of estimate (BOE) documentation?
3>Illustrating a track back to cost estimating relationships (CERs) and their referenced historical databases?
4>Demonstrate that the estimate came from a reputable parametric tool like TruePlanning?
5> Any other approach?
6> All of the above or some combination of these approaches?
7> You don't have this problem i.e. you never had a need to prove the credibility of your estimates?

If the answer is point number 7, how do would you rate the above criteria over the others?

Recently Dale Shermon of PRICE Systems gave a webinar presentation “Delivering Programs with TOC and CAIV” which looks at the application of parametric cost estimation in the context of determining a delivery strategy.  He then walks through the Cost as an Independent Variable (CAIV) process that is often employed within a Total Ownership Cost (TOC) approach to estimating.

Often, once a large defense contract is won by the business development team the program delivery team then has the job of figuring out “Ok, what exactly have we just won and how do we deliver it?” Dale’s presentation does an excellent job of taking you through the thought processes involved at that point and shows you how parametric cost estimation is competent and trusted resource during this phase of the acquisition.

To view the webinar click here.

That’s a question that members of the cost estimating community will try to answer at next week’s  43^rd Annual DoDCAS symposium. The conference is centered around the “theory and implementation” of the WSARA 2009 (Weapons System Acquisition Reform Act). Simply put, the cost estimating community needs to first and foremost understand what the real life implications of the WSARA will be. At least in theory that is.  

According to GAO, nearly 70% of the Pentagon’s 96 biggest weapons programs were over budget in 2008 and another government report found $295 billion in waste and cost overruns in defense contracts. Clearly things need to change.

As the Deputy Secretary of Defense, William Lynn, correctly states, overly optimistic estimating is at the root of cost overruns.

“We rely on overoptimistic cost estimates. These estimates are success oriented…they assume every step in the development process will go as planned. To produce weapon systems efficiently, it is critical to have budget stability. But it is impossible to attain stability in our modernization budgets if we underestimate the cost of our weapons systems from the start.”

Lynn also states the need to bolster personnel by 20,000 positions over the next five years with a heavy focus on cost estimators, systems engineers, and acquisition managers. 

Naturally, PRICE Systems, is eager to support the acquisition community as it attempts to implement the key initiatives of the WSARA. We will have our some of our cost research experts on hand to present and discuss best practices in cost estimating and they will be reporting back on what they learned and observed at the conference.

Stay tuned.