Model Driven Engineering for Resilience of Systems with Black Box and AI-based Components

Modern complex cyber-physical systems heavily rely on humans and AI for mission-critical operations and decision making. Unfortunately, these components are often "black boxes" to the operator, either because the decision models are too complex for human comprehension (e.g. deep neural networks) or are intentionally hidden (e.g. proprietary intellectual property). In these cases, the decision logic cannot be validated and therefore trust is forced. Development of system modeling techniques for past influences when data/internals of specific critical components cannot be accessed is a challenge, as is the case with AI and human components. This is a recognized contemporary concern for industrial operators and government agencies, since stakeholders of large engineering projects typically do not want to share design data or model access. From the client point of view, there is a need for modeling system resilience (safety and security) when there is lack of complete trust/control in the AI process or over human factors and fail-safes - layers of defense should be deployed. Prior work has presented a methodology for assessing and supporting development of resilience for mission critical systems that include AI components and humans. Zero Trust and Defense-in-Depth (DiD) principles within the methodology protect critical components, taking into account interfaces and influences during different lifecycle phases and system configurations. However, the methodology does not cover analysis of past influences of critical components which is a very important but laborious task that could be supported by model driven engineering methods. This work extends that of prior methodologies and presents ways to systematically model the past influences for critical human and AI components. The concept is based on metamodels for interaction/dependency modeling and then on the definition of metrics in order to establish and even reduce the search space across past influences, and add controls when a search path is not followed.