Towards Developing Metrics to Evaluate Digital Engineering

Abstract

Model-based systems engineering (MBSE) is an increasingly accepted practice in the Systems Engineering (SE) community, however, little has been done to empirically show that MBSE provides value. Furthermore, as the industry continues in the direction of digital transformation, MBSE will become a critical component of the larger Digital Engineering (DE) approach. This paper presents a measurement framework for selecting and developing appropriate metrics to assess the value/benefits of MBSE and subsequently DE. Utilizing expected benefits identified in a review of MBSE literature, a causal map was hypothesized to show how expected benefits (potential metrics) influence and relate to each other. This was done in order to systematically determine which benefits would be the most impactful to measure. The hypothesized causal model was presented for feedback to subject-matter experts from a working group developing the first DE measurement framework. This group is a joint effort with industry, academia, and the USA government to develop DE metric standards. Once the causal map was finalized, a case study was used to partially validate the causal model. Based on the causal map and subsequent analysis, we can recommend the first metrics to be employed for DE/MBSE based on the most influential nodes of the causal model. The potential metric candidates include: system quality, defects, time, rework, ease of making changes, system understanding, Effort, accessibility of information, collaboration, project methods/processes, and use of DE/MBSE tools. We believe a concerted effort across the industry to focus on measuring these variables is the most effective way to establish proof of the value of MBSE and DE.