Reliability of Convolutional Neural Networks: Failure Metrics with Metamorphic Test Cases

The data-driven software development paradigm and the uncertain nature of deep neural networks bring new challenges for reliability engineering. In autonomous driving and other safety-critical domains, high reliability must be assured. However, failure metrics measuring reliability post deployment can be costly. To explore more efficient ways of assessing reliability, we leverage metamorphic testing where a follow-up test case (TC) can be automatically generated to account for unseen conditions once a source TC is given. We further determine source TCs by investigating various TC prioritization strategies based on risk, fault, similarity, and coverage. Our experiment on a convolutional neural network's image classifications shows that TC priorities determined by code coverage are prominent in exposing reliability issues, combined with the metamorphic TCs generated by semantic input data changes. The results also show that, in order to safely operate the neural networks, reliability engineers shall pay attention to the potential network structural changes and the evolving requirements-level risks.