Machine Learning Infused Software Testing for Mobile Device Development

Automated test systems execute a large set of test cases across mobile devices that are distributed throughout the global test labs. The test cases generate a huge amount of data which consists of various logs that are captured from the mobile devices under teat and the automated test systems. The logs that are associated with the failed test cases are typically manually analyzed to determine if the failure is a true product failure or an issue with the test environment. This binary classification is performed for each test failure. The failures classified as test environment failures are further classified into subcategories. The outcome of this analysis helps ensure the test case failures are properly triaged and the respective team can take the appropriate action. Our study demonstrates the potential application of data mining techniques to automatically categorize failed test cases using the test logs. These logs contain explicit details such as time stamps, log levels, the tested module, error codes, and test outcomes. Moreover, they also have implicit information such as line velocity, error velocity, test duration, and sentiment profile. We utilize heuristics to extract both the explicit and implicit information as features. These features are associated to training labels using a subset of the logs. Our experimentation involved multiple models, including Naive Bayes, Random Forest, Decision Tree, and Decision Table. We show that with an initial one-time labeling effort, the best models for binary classification achieve around 90% accuracy for binary classification. We further show that multiclass classification achieves around 85% accuracy for most of the classes.