DO-178C Costs Versus Benefits
Download Full 10+ Page DO-178C Costs vs. Benefits Whitepaper
“DO-178 is the worst standard in the world …except for all the others.” (Vance Hilderman, 2004)
The above rewrite of Winston Churchill’s famous quote about Democracy has some truths: DO-178 is the benefactor, or bane, of avionics projects the world over. Many users complained of added costs associated with DO-178B, and DO-178C streamlines some of those costs but offsets others with increased rigor. Truly, DO-178C is never cheap, certainly not on the first project. And in clear cases outlined herein, DO-178C can increase costs above DO-178B, which already increased costs by 20-40% itself. But is DO-178C really “too” expensive? Doesn’t it actually reduce costs over DO-178B for companies who were doing it “right”? Does it reduce long-term costs at the expense of increased development cost? Will it improve safety and reliability and if so, to what degree? Exactly what benefits are received from complying with DO-178C? In what areas is DO-178C more expensive than DO-178B? These important questions are answered in this Vance Hilderman Whitepaper.
DO-178 has increasingly evolved into the de-facto standard for virtually all forms of commercial avionics except for experimental aircraft. In the past decade, DO-178B and now DO-178C are mandatory for most Military avionics. The new DO-178C made both minor and major changes; costs will definitely increase for those users taking shortcuts with the former DO-178B. However, DO-178 possesses attributes common to all safety critical domains: planning, consistency, determinism, thorough documentation and testing, and proof of the preceding attributes. Truly, DO-178 relies upon significant verification (reviews, tests, and analysis) to assess avionics quality. However, avionics quality comes from a quality process, design, and implementation, not just from testing.
The DO-178 (software) and DO-254 (hardware) standards presume that hardware and software must operate in harmonic unison, each with proven reliability. But avionics software and hardware are part of an overall “Ecosystem” including Safety, Systems, and ancillary guidelines as depicted below:
The DO-178 (software) and DO-254 (hardware) standards presume that hardware and software must operate in harmonic unison, each with proven reliability. But avionics software and hardware are part of an overall “Ecosystem” including Safety, Systems, and ancillary guidelines as depicted below:
Previously, hardware was considered “visible” and tested at the systems level with integrated software; hence hardware was exempt from DO-178B’s objectives. But that exemption resulted in functionality being moved from software to hardware for the purpose of avoiding software certification. Also, hardware complexity has evolved such that hardware is often as complex, or more so, than software due to the embedded logic within the PLDs, ASICs and FPGAs. Now, everyone recognizes that hardware and software comprise an inextricable chain with the quality equal to that of the weakest link, hence the mandate to also apply DO-254 to avionics hardware; the weak links are continually removed through the avionics certification evolution. ARP-4754A requires explicit consideration of this entire system throughout the avionics development lifecycle, and DO-178C thus aligns with ARP-4754A.
In DO-178 “software” pertains to all drivers, BSP, RTOS, libraries, graphics, and the application software – in other words, any executable aspect that is loaded into memory during execution. Software testing means ensuring that the lowest level detailed requirements are accurately implemented, paths are covered according to their criticality level, and full traceability is provided. Increasingly, tools are being used to automate DO-178 processes. For example, almost all DO-178 projects today use a commercial traceability tool for avionics software. While traceability tools are not formally required, virtually every DO-178 compliant project uses them to meet DO-178’s top-to-bottom and bottom-to-top traceability. While DO-178 does not require such tools (you can always provide traceability manually), a DO-178 compliant traceability tool greatly reduces the cost of compliance.
Criticality Levels and Cost.
DO-178 and DO254 entail five different levels of criticality, ranging from Level A (most critical) to Level E (least critical). Each avionics system is assigned one or more levels of criticality, and each component within that system must meet or exceed its assigned criticality level. As the criticality level increases, so does the degree of rigor associated with documentation, design, reviews, implementation, and verification. Accordingly, cost and schedule increase as well. The following graph, and associated table below, depict the actual cost deltas associated with each level of criticality: as a baseline, the additional costs of DO-178C are contrasted with medium-quality software development, e.g. high quality financial or consumer software (where the applicable true CMMI level of development was between Level 2 and 3).
Graph: Delta Cost and Schedule of DO-178C per DAL
A popular myth is that DO-178C is expensive. Indeed, DO-178C is not “cheap” as clearly the additional costs can be seen above. Level D certified software still has generally full planning, high and low level requirements, implementation, reviews, and full functional testing of all high-level requirements with traceability applied. Additionally, configuration management, quality assurance, and DER liaison are applied to Level D. Yet the costs of Level D are just 15% higher than medium-quality consumer/financial software developed per typical CMMI Level 2-3 processes. Why? Because Level D is comprised almost entirely of normal industry-standard software engineering principals. Also, many companies new to DO-178C believe their prior, existing efforts including planning, requirements, designs, test, and reviews must be re-done: not true. In fact, a DO-178C Gap Analysis activity will analyze the existing processes and work to re-use such processes while identifying the “gaps” to fulfill DO-178’s requirements. (The author has performed dozens of successful DO-178 Gap Analysis activities which take 2-4 weeks to perform and form the basis of the data herein.)
DO-178 also requires reviews (generally independent for Level B and mostly independent for Level A) of all software development processes and artifacts. DO-178 reviews must be per approved DO-178 checklists to ensure all required aspects are properly reviewed. To save money and time, many companies use automated project management/review tools.
Another myth is that the most significant software cost escalation occurs when moving from DO-178 Level B criticality to Level A. Untrue, but for an interesting reason. The singular largest difference between a Level A system and the Level B system is the 100x greater reliability required by the Level A system per ARP-4761A. However, that 100x reliability must come primarily from the system/hardware architecture and not the software. How? Added redundancy: the only way to meet DO-178C’s Level A reliability is via increased hardware/system redundancy which of course greatly increases total product cost. But the software cost difference, the subject herein, between Level A and Level B software is quite minor as seen in the figures above.
The following DO-178C Costs chart shows a typical breakdown of expenditures for a Level C project; Level A of course would have a greater percentage allocated to Verification.
Benefits of DO-178C.
DO-178C is certainly neither free nor cheap, as cited above. However, DO-178C can actually be cost-effective, when implemented properly. Particularly when evaluated over a product lifetime or subsequent product versions when DO-178C efficiency and benefits are most notable. Why then are so many entities adopting DO-178? Because of the actual DO-178 benefits.
The following describes the most commonly obtained benefits from DO-178C based upon AFuzion’s 200+ person-years’ experience consulting on over 250 DO-178 avionics projects.
➤ Greater upfront requirements clarity: DO-178C mandates thorough and detailed software requirements. Such detail, and the necessary discipline, force answers to be provided up-front instead of being deferred. Assumptions are drastically minimized. Consistency of requirements and their testability is greatly enhanced. Iterations and rework due to faulty and missing requirements are greatly reduced. Yes, other standard and guidelines such as CMMI mandate such upfront requirements; however, DO-178C is unique in its enforcement of requirement detail.
➤ Fewer coding iterations: Code iterations, or churn, are the bane of software engineering. In many cases, ten, twenty, and even thirty versions of evolving code files exist on new products. Nonsense. With strong engineering processes and discipline per DO-178C, code should be largely correct the first time it is written and should not require dozens of updates to get it right, particularly when using well-advised software modeling. Models and code should be reviewed by analyzing implementation versus DO-178c Documented high and low level requirements.
➤ Fewer bugs found during module testing: Since DO-178C mandates thorough and testable requirements, in addition to code reviews for Level C and above, far fewer bugs will be found during module testing. Independent code reviews required for Level B and A also further that aim. And DO-178C code reviews require the following completed and configured items prior to writing models or code:
o Standards & checklists
o High and low level requirements
o Design
o Traceability for the above
Another myth is that the most significant software cost escalation occurs when moving from DO-178C Level B criticality to DO-178C Level A. Untrue, but for an interesting reason. The singular largest difference between a DO-178C Level A system and the Level B system is the 100x greater reliability required by the DO-178C Level A system per ARP-4761A. However, that 100x reliability must come primarily from the system/hardware architecture and not the software. How? Added redundancy:, not from DO-178C explicitly, but via the associated ARP4761A Functional Hazard Assessment (FHA) process which precedes DO-178C: the only way to meet DO-178C’s Level A reliability is via increased hardware/system redundancy which of course greatly increases total product cost. But the DO-178C software cost difference, the subject herein, between Level A and Level B software is quite minor as seen in the figures above.
The cost differential within DO-178C is the most significant between Level D and Level C. Why? Level C requires the following key objectives which Level D does not and which results in Level C requiring 35% more effort than Level D:
➤ Testing of low-level software requirements, particularly for DO-178C versus DO-178B
➤ Ensuring 100% coverage of all source code statements
➤ Assessment of requirements, design, and code to standards
➤ Greater rigor placed upon reviews and evidence thereof ( see AFuzion’s DO-178C checklists here: https://afuzion.com/plans-checklists/)
➤ In many cases more rigorous configuration management per DO-178C CC1 and CC2
DO-178C’s DAL B requires additional structural coverage (decision-condition, e.g. all branches in the source code), additional independence in reviews, and tighter configuration management. At first glance, it would seem that Level B should be significantly more expensive, e.g. 50% – 70%, than Level C. In theory, it seems to make sense, but as in many areas of life, common sense overcomes theory. In Level B (and C) there must be detailed, low-level software requirements and they must be thoroughly tested. Remember, DO-178C requires detailed low level requirement verification beginning with Level C and those low level requirements will cover the vast majority of software logic decisions. During D0-178C requirements-based testing, most (80-90%+) of the branches in the source code are already covered and hence require no additional structural coverage testing if test capture and coverage tools are appropriately used during that functional testing. Therefore, the seemingly significant cost increase associated with Level B versus C structural coverage is already mitigated by DO-178C’s greatly enhanced requirements-based testing. Also, quality software engineering organizations already incorporate a semi-automated and streamlined process which includes independent reviews and tight configuration management; ergo the added cost of those aspects for Level B is largely mollified. The reader is well-advised to undergo upfront DO-178C Training and DO-178C Process Improvement to leverage these cost reduction techniques. (For free DO-178C training videos, watch here on the AFuzion YouTube channel: https://www.youtube.com/watch?v=RMzLRzcahJE&list=PL0es63yi1vbw9Tt1GWX8TtiFm4R2qRryu
Now, “efficient” followers of DO-178B will find even greater cost impact in adhering to DO-178C. Their “efficiency” may have been due to taking liberties with DO-178B’s intended, but less enforced, low-level requirement detail. Such DO-178B (but not DO-178C) shortcuts enabled less detailed functional testing with many fewer logic branches verified. While acceptable for DO-178B Level C, it’s unacceptable for DO-178C Level C which mandates greater detailed of low level requirements. As a result of that greater detail, DO-178C inadvertently reduced the difference between Level C and Level B because the decision-condition structural coverage objective of Level B is largely covered already in Level C due to those more detailed low level requirements. Also, DO-178Cdevelopers making extensive use of Parameter Data Items (objects or logic external to the main application program) are now required to fully document, review, trace, and test all that data under DO-178C; something they “should” have been doing under the intent of DO-178B. Voilà. The following figure summarizes DO-178C costs (copyright excerpt from AFuzion’s DO-178C Training):
Figure: DO-178C Cost Allocation: Typical DO-178C Project
(Excerpt from DO-178C Optimization Training by AFuzion)
Additional DO-178C benefits are summarized below:
➤ More thorough testing via DO-178C: 100% high- and low-level requirements coverage, greater robustness coverage with clarity, and software structural coverage are deterministic verification aspects of DO-178C. Parameter Data Items must be particularly well documented and verified under DO-178C. Unused software structures (dead code) must be removed in DO-178C DAL C and above, and deactivated code documented and verified for Levels C through A The probability of software errors encountered in the field is vastly reduced, particularly at DO-178C Levels A and B.
➤ Improved hardware and system testing: system testing per ARP4754A is typically challenging for embedded systems because the development environment is quite removed from the target environment. Also, testing is often performed via different engineering teams. DO-178C mandates that software components which could be affected by hardware be tested on the hardware, e.g. hardware/software interfaces, interrupts, timing, board-level components, BSP/RTOS, etc. And the improved determinism and quality of all these components via DO-178C belies improved hardware integration. Since DO-178C requires complete traceability, it is quite possible to show via traceability that system requirements were actually fully tested during software testing and therefore need not be repeated by additional testing at the system level per ARP4754A (which can be combined and overlap with DO-178C). For a list of DO-178C development tools, see here: http://certegic.com/aviation-development-information/adi-development-tools/
➤ Fewer DO-178C in-the-field defects: In-the-field defects and customer returns are hugely expensive in both the short- and long-term. Products developed under DO-178 have been demonstrated to result in 80%-90% fewer returns.
Improved software reusability via DO-178C: Via thorough and consistent documentation required by DO-178C, modularization, enforcement of documented modern engineering principles, and reviews to ensure all the above was achieved, reusability is greatly improved within DO-178C. In software, reusability is the DO-178C Holy Grail, but the reality is that unless a software component is at least 80% reusable (i.e. unchanged), then it is quicker and less risky to simply start from scratch. In reality, most software is less than 50% “reusable”. With DO-178C, and enforcement of design/coding standards, coupled with independent reviews and traceability, most modules should be at least 70% re-usable. And when modeling and C++ are employed in DO-178C per DO-331 and DO-332 (see those respective AFuzion whitepapers), DO-178C reusability increases even more. Software reusability is a common DO-178C gap; for a video on closing DO-178C gaps, view here: https://www.youtube.com/watch?v=7doZAZ5CTMk&list=PL0es63yi1vbw9Tt1GWX8TtiFm4R2qRryu&index=4&t=1
Now, DO-178C increases initial software engineering costs by 25% – 40%. The top ten ways to reduce DO-178C costs are listed below (download the rest of this free AFuzion DO-178C Cost / Benefit paper to read).
For the remaining 10+ DO-178C Benefits, please download the remainder of this whitepaper below:
Download Full 10+ Page DO-178C Costs vs. Benefits Whitepaper