FAQ
Bolded words or phrases are key terms or acronyms defined under References or concepts discussed in NASA-STD-7009 or NASA-HDBK-7009.
To communicate, promote, and provide a medium for collaboration on the best practices in the development and use of models and simulations (M&S).
The initial content for this website starts from the NASA Standard for Models and Simulations, NASA-STD-7009, which is broadly applicable to all types of M&S.
The intent, for this information on the practice of modeling and simulation, is to be applicable to a broad range of modeling and simulation types and applications, not a single type or application. However, the practice of a single type or specific application of modeling, which might be broadened to a wider range of types or applications, is also valuable.
Planning the development and use of an M&S is much like planning any other (hardware or software or system) project. Whether implicit or explicit, the development and subsequent use follows a set of ideally sequential phases with the potential for recursive loops to earlier phases, depending on many potential influences including the success of development and interim testing.
For more, see: M&S Planning
M&S Criticality is always with respect to the real-world decisions made from M&S results. A critical decision is defined as “the selection of a course-of-action related to design, development, manufacturing, ground, or flight operations that may significantly impact human safety or mission success, as measured by program/project-defined criteria” (NASA-STD-7009).
For more, see: M&S Criticality
The intended use of an M&S is “the expected purpose and application of an M&S” and is best determined as early as possible in the M&S life cycle as a standard-bearer for development, even if it is modified at a later time. The intended use is also a concise communication mechanism of what the M&S does.
For more, see: M&S Intended Use
Reviewing and testing of an M&S can occur frequently throughout its development phases. From a rigorous perspective, a review of model development could occur at the end of each phase, to ensure all the processes and products of each phase are acceptably complete or ready to support the subsequent phase. A more common approach includes formal reviews at the completion of model design, called conceptual (or design) validation, and at the completion of verification and empirical validation. Conceptual validation is typically a review between the modelers and the customers or stakeholders of the model, ideally prior to the start of model construction, to ensure the conceptual model or model design adequately represents the real world from the perspective of the intended uses of the M&S.
Model testing, after the completion of Model Construction, includes the independent tasks of verification and validation. Verification is the process of determining the extent to which an M&S is compliant with its requirements and specifications as detailed in its conceptual models, mathematical models, or other constructs. Validation is the process of determining the degree to which a model or a simulation is an accurate representation of the real world from the perspective of the intended uses of the M&S. Specifically accomplishing verification prior to empirical validation is, arguably, the most effective and efficient. Attempting empirical validation prior to full model verification is a highly risky practice, is to be avoided, and likely induces inefficiencies requiring repeated verification and validation cycles.
With the successful completion of verification and validation, the M&S is ready for official release and use.
Releasing an M&S is more than simply making it available for use. At the successful completion of M&S testing (i.e., the distinctly independent tasks of verification and validation), the permissible uses of the M&S are documented, along with or as part of the M&S User’s Guide.
Additionally, a set of ‘test suites’ are now available (for future regression tests) along with procedures for M&S use. Both the ‘test suites’ and ‘M&S Use Procedures’ are derived from those used in M&S verification and validation testing.
An M&S User’s Guide is a document to help install, use, or troubleshoot an M&S. The user’s guide typically include the M&S Statement of Intended Use, a list of model assumptions and abstractions, descriptions of how the model works, and the permissible uses of the model, including the verified and validated limits of model operation (use).
NASA-HDBK-7009, Appendix C, shows a content outline for an M&S User’s Guide.
An M&S Use Assessment determines the acceptability of an M&S for a Proposed Use by comparison with the Permissible Uses of the M&S.
For more, see: M&S Use
The NASA Standard for Models and Simulations (NASA-STD-7009) requires the reporting of supplementary information, along with the M&S results, to substantiate the credibility of the M&S results and understand the potential risks associated with accepting the results of the M&S use. The supplementary information includes:
● M&S Criticality Assessment
● Analysis Caveats
● M&S Results Uncertainty
● M&S Results Credibility Assessment
● M&S Risk Assessment
● M&S Documentation
For more, see: Reporting M&S Results
The credibility of M&S-based results, i.e., “the quality to elicit belief or trust in M&S results,” is not something that can be assessed directly. However, key factors of credibility may be assessed that inform a customer’s (decision maker’s) credibility assessment. These eight key factors are:
● Data Pedigree
● Verification
● Validation
● Input Pedigree
● Uncertainty Characterization
● Results Robustness
● M&S History
● M&S Process / Product Management.
For more, see: M&S Results Credibility
From an M&S perspective, a risk exists in the potential shortfalls in the M&S with respect to sufficiently representing the RWS. In other words, if an M&S does not sufficiently represent the real world system for which it is providing results, then some amount of risk is incurred when using the M&S results to make real world decisions. An assessment of M&S risk is, therefore, warranted.
For more, see: M&S Risk
The sources of M&S risk may originate in any phase of the M&S Life Cycle
Refer to the NASA-HDBK-7009:
● Section 5.6.3.2.2 M&S Risk Assessment.
● Appendix E – M&S Risk Assessment.