Best Approaches to Assessment During the Sustainment Phase
Best Approaches to Assessment During the Sustainment Phase Charlie Vono charlesvono.com 8 Jan 19 charlesvono.com Charlie Vono EDUCATION B.S. Astronautical Engineering, USAF Academy, 1976 M.S. Systems Management, USC, 1985 Crawford Slip Method under Dr. C.C. Crawford M.S. Mechanical Engineering, USU, 1995 Air University, 1996
US AIR FORCE Retired Colonel KC-135 Aircraft Commander Inertial Upper Stage Software Systems Chief F-16 Battle Damage Repair Engineer Ogden Air Logistics Center Staff Pacific Command Reserve Forces Division Chief TRW & NORTHROP GRUMMAN Retired ICBM Engineer and Technical Manager, propulsion & guidance Weapon system sustainment expert Other Interests AIAA Associate Fellow, SAME, INCOSE Utah Engineers Council AIAA Distinguished Lecturer Husband of the Turtle Lady Author My W eb Site
Common Issues with LongLived Complex Systems Unforeseen failure modes emerge undetected Original vendors and suppliers are gone Design documentation fails to fully explain why Operators develop expectations for a capability Priority for resources has diminished or disappeared Managers and their teams are set in their ways Why Management Models? Good Management Models help keep the individual, team, and management working the right tasks and issues at the right times To do this, they are: Self-improving anti-fragile, or at least robust
Constant -- unaffected by changing laws, regulations, or fads Applicable to the very complex systems employed today Memorable -- easily called to mind Practical -- easy to apply, common lexicon Integrated -- internally consistent eat-drink-love.com The Complex System Sustainment Management Model Readiness Factors Reliable Available Accurate
Sustaining Your System Enablers Observe: Assessment Program People: Everyone a Leader Identify: Risk Management IT: Handling Big Data Fix: Long-Range Planning Processes: Disciplined Approaches 6 5-Step Approach Use your free data Look to your repair depots
Set up an age surveillance program Establish processes for special testing Analyze your data to create information Affordable and Effective Assessment Use Your Free Data Examples of free data measurements taken while operating parts availability rates bench stock depletion rates failure rates repair depot testing. Defer to your teams best ways to collect and filter
If data kept in more than one place, create a way to reconcile it Err on the side of more data; encourage process improvement Collect the meta data too If people enter data, motivate them to enter it correctly Look to Your Repair Depots Repair shops are focused on productivity and throughput sub-optimal to your goals of observing your system to identify and mitigate risks with sufficient lead time contract with your repair depots for CLFA to ensure you get the data you need, when you need it CLFA is MIL-HDBK-2155 FRACAS in a repair depot FAILED
Depot System Operational "close the loop" between the failure noted and the repair made Did this bad part create this fielded failure? Remove and Replace New part from vendor Test and place in supply Depot supply Bin in repair shop Discard bad part
Depot supply Test and place in supply FAILED Bin in repair shop Depot System Operational Fix via remove and replace Induct and Test Test and Sell Off CLFA
New part from vendor Did this part cause this failure? Test and place in supply Depot supply Bin in repair shop Discard bad part Depot supply Test and place in supply FAILED Bin in repair shop
Depot System Operational Fix via remove and replace Induct and Test Test and Sell Off Advanced CLFA Sufficient diagnostics to find emerging failure modes, but For affordability, work to eliminate from scrutiny those failure modes and repairs that are well known or become well known after CLFA has functioned for a few years Use findings of failure review board to improve depot processes and equipment CLFA requires process discipline, info management
Good tracking prevents poor components from circulating CLFA is a contract and also a partnership that benefits both Age Surveillance Program Anticipating a downward trend in a failure mode when that subsystem or part is not exercised often enough to provide the trend Analyze system for expected failure modes Analyze modes to predict their symptoms Exercise representative subsystems often enough to detect trends From design data, use, maintenance, repair data Weigh impacts on production Establish Processes for Special Testing Readiness Factors
Reliable Available Accurate To deal with knowledge gaps apparent over time (readiness factors) Key skills: test plan authors and testing leads Similar enough to previous so data points can be combined Changes from one test to another thoroughly documented Test plans vetted/understood with community & decision-makers Testing lead rules test with iron hand Analyze Your Data to Create Information Readiness Factors
Reliable Available Accurate Do it right the first time and establish credibility with your decision-makers Good decision-makers know that no model is perfect, systems are hard to observe, and mistakes are easily made. Reporting needs to convey an understanding of this uncertainty and what, if anything, can be done to decrease it. Most likely to go wrong: inability to convey the trend, why the trend is important, why you think it is real, why it needs to be identified immediately, or someday soon, as a risk 3 key roles: statistician, engineer, communicator Double-checking between different kinds of observations Good observations depend heavily upon excellent configuration tracking Advanced Analysis
Other sources of error poor test plans, ineffective test directors, changing test conditions from year to year, more than one emerging failure mode confused as one, and locked-in thinking that uses data to confirm a strongly held belief Periodically recheck your assessment program to see if it is looking at your entire system, looking at sufficient subsystems to ensure lead times, learning from new techniques from other assessment programs and doing all of this across the spectrum of the readiness factors that your operators or warfighters need to complete their mission. Before you ever started this path of creating and executing an adequate assessment program, your organization should have reinvigorated its risk identification system. In this process, readiness factors important to the mission, such as reliability and survivability should have been defined. The best understanding of these readiness factors is always improving and should be a part of your periodic re-assessment of your assessment program. Completing the circle, any deficiencies discovered in your assessment program should be written up as risks identified and discussed at your risk identification meetings. The Complex System Sustainment Management Model Readiness Factors
Reliable Available Accurate Sustaining Your System Enablers Observe: Assessment Program People: Everyone a Leader Identify: Risk Management IT: Handling Big Data
Fix: Long-Range Planning Processes: Disciplined Approaches 17 Mega Trends in Systems, Systems Engineering, and Sustainment Charlie Vono charlesvono.com 9 Jan 19 Assessment: Observation of the system to find changes in performance, determine if those changes affect the system readiness parameters, and characterize them if they do. Assessor: Team member trained in the skill of observing the system and drawing conclusions about trends. Assessors should work in teams with a minimum of a system expert and a statistician. Test chiefs, test plan authors, test report writers, and data base experts are examples of other skills critical to good assessment. Assessor is an example of sustainment knowledge that needs preserved via good organizational structure where the individuals are spread throughout the organization, but form an association or guild to share their knowledge of assessment. Availability: A readiness factor that measures how many systems are available when needed. For instance, combat aircraft may have availability requirements stated like
this: 85% of all aircraft must be ready for flight crews 2 hours after notification of a mission. A power gird may need 18 megawatts of alternate power available within microseconds. Capabilities Baseline: Once the system is deployed, the operator begins to perceive and depend upon capabilities of the system that might not be captured in any design documentation. This becomes an important baseline for the sustainer. CLFA: A Closed Loop Failure Analysis (CLFA) program applies FRACAS to a repair depot during the sustainment phase of a system. An effective CLFA program is a formal contract between the sustaining organization and the repair depot to ensure sufficient diagnostic information is created and delivered to support the sustainment assessment program. The information includes verification that the failure that was fixed was reasonably the one that occurred to bring the component to the depot. The information can be used to improve the readiness factors of the system and the effectiveness of the depot equipment. CLFA programs typically required significant changes to the depot data and hardware routing systems. Complex Funding System: Complex systems are usually associated with complex systems for providing sustainment funding. They have many funding sources that interact in sometimes unpredictable ways with decision-makers and office staff. This creates the need for experts in all the sources of funding, their rules, and their interactions. Typically, the rules and funding sources change over time. Complex System: Systems are considered complex when they can enter states unpredictably. Before deployment, design engineers attempt to determine all states of the system, including states that are entered at failure. Safety critical components are designed to fail in safe modes. After deployment, new failure states emerge and may place the system in unanticipated states. Complicated System: A system with many, many components interacting in many, many ways. Cowboy: A sustainment slang term for the hero that rides in and saves the day during a crisis. Not a bad thing, but we want to minimize crises. Farmer: A sustainment slang term for the bulk of the organization which is, hopefully, following process to find emerging failure modes early and avoid flashy crises. FRACAS: A Failure Reporting and Corrective Action System (FRACAS) is a process typically applied to a system production line to ensure early achievement of reliability and maintainability. An effective FRACAS program reports failures of system components (including software), analyses to determine causes of failure, takes corrective actions, and verifies results. See Mil-HDBK-2155. See also CLFA.
Fundamental Theorem of Sustainment: A fundamental theorem is a statement that is necessary to create the associated domain of knowledge. In sustainment, the fundamental theorem is: An effective sustainment organization will always find ways to affordably detect threats to the system in time to correct them before the mission is impacted. Impact: In a sustainment risk analysis, the effects on the system and mission due to an emerging failure mode or inadequate sustainment process. Information Management System (IMS): Software, hardware, and processes designed to stow all kinds of data, metadata, and information; the tools needed to transform this big data into useful information; and the means to retrieve and report the products. Integrated Product Team (IPT): A team of teams in two ways. First, the full team is composed of lower level teams who focus on system components or engineering specialties that need to preserve their knowledge of the system and of sustainment. Second, lower level teams are composed of all the needed experts from multiple companies. The entire organization can be referred to as an IPT structure. Each individual team within this structure can be referred to as an IPT. For example, a guidance system IPT could be part of a larger rocket IPT which is part of a larger system IPT. Lead Time Ahead: A phrase meant to capture the need to consider when the risk might be realized versus the time it will take to mitigate it. Design and development schedules are fixed by many other factors. In the sustainment phase, projects and programs to mitigate future risks are primarily created based on the timing of risk realization and mitigation. Leadership: The ability to focus others on the systems mission and instill a desire for competence while performing your tasks with competence. Likelihood: In a risk analysis, how likely the system and mission will feel the impact of a realized risk. Or, in other words, how likely it is the risk will actually occur. Mission: The reason the system is employed. The military warfighters or civil system operators mission is the sustainers mission. Sustainer mission statements that use the word sustain remove themselves too far from their actual mission. Sustainers must see themselves as part of the weapon system warfighters or civilian system operators team. Process Discipline: The actions of your people as they follow organizational processes. Improvements can only occur if the teams respect the processes and improve them instead of ignoring them. Audits that focus on improvements instead of blame and processes to quickly change processes support this organizational goal.
Readiness Factors: Two to six independent characteristics that, if violated, will affect the systems ability to perform its mission. For instance, the vast majority of systems must be both reliable when used and available when needed. Some must provide accuracy while others need to deliver persistence over a defined location. Readiness factor requirements (e.g. 85% reliable) are often measured across many individual systems and aggregated. This improves the precision of the estimate, usually to the benefit of the mission. Reliability: A readiness factor that measures how well the system meets the mission once employed. A system might have several modes depending on a mission. For instance, an aircraft might score 100% reliable for delivery of cargo but 0% reliable for delivery of fuel in-flight if the refueling boom breaks. Risk Integrator: Every sustainment organization needs a few risk integrators well-schooled in working with assessment engineers and other IPT members to formalize risks, present them at monthly risk meetings, and teach their IPT members how to do the same. Risk integrator is a great position to give to a rising star who can work well with the system experts on the team. Sustainment: Support of the system to ensure continued mission capability. Some view logistics as sustainment, or supply as sustainment, or depot activities as sustainment. Others raise expert engineers or astute program managers to be the most important element of sustainment. In this handbook, sustainment encompasses all the skills required to provide support of the deployed system. Experts in funding sources are just as important as expert repair techs. Sustainment Risk: A risk that can be shown to impact the mission via the system readiness factors. System: A set of interacting components. In this handbook, the system includes everything required for the operator to employ the hardware and embedded software to achieve the mission. For instance, manned strategic bombers are designed and deployed to carry out the military doctrine of strategic bombardment against a nations ability to wage war. World-wide lighter than air Wi-Fi vehicles are designed and deployed to ensure internet coverage in even the most remote parts of Earth. Time-dependence: In a sustainment risk analysis, a required factor to help determine risk prioritization. See LIKELIHOOD in this chapter.
Proposed source-to-source transformation for performing vulnerability analysis. Newsome and Song (2005) Performed Taint analysis on compiled binaries through Valgrind to detect buffer overflow attacks. Yin and Song (2009) Performed dynamic taint analysis on VEX/Vine IR. James - 1. modify the...
which allow two or more locations to interact via two-way video and audio transmissions simultaneously. e.g. Skype,OoVoO,… CHAT (IRC ) When two or more computer users can see and respond . to instant messages [in real time (live)]. A ....
Credit Invoice: is received by the purchaser for defective or damaged merchandise.This invoice allows the purchaser to reduce the amount owing to the vendor on account. Purchase Returns & Allowances: is the contra account to purchases and is used to...
Consistent and strong evidence. Do you get your point across? Elements of Good Writing. Element. Description of Element. Lead. Focus. Opening of paper (first line, first paragraph, or first several paragraphs) MUST capture the reader's interest and state the purpose...
Social Darwinism developed in 1870 by Herbert Spencer a. influenced by Darwin's theory of survival of the fittest b. adapted this theory to the business environment c. Andrew Carnegie was a prime example i. a poor Scottish immigrant ii. he...
NAB/CTIA/NATE/PCIA petition to include notice as part of ASR process - environmental challenges to be treated as Petitions to Deny LMCC to file comments noting difficulties created for frequency coordinators, suggesting solutions Utilities to spend $100s billions over next 10-20...
Used in persuasion to demonstrate the logic of one idea by showing how it is similar to an accepted idea. ("Pupils are more like oysters than sausages. The job of teaching is not to stuff them and then seal them...
Ready to download the document? Go ahead and hit continue!