Large Language Models: The Next Frontier for Variable Discovery within Metamorphic Testing

Metamorphic testing involves reasoning on necessary properties that a program under test should exhibit regarding multiple input and output variables. A general approach consists of extracting metamorphic relations from auxiliary artifacts such as user manuals or documentation, a strategy particularly fitting to testing scientific software. However, such software typically has large input-output spaces, and the fundamental prerequisite extracting variables of interest is an arduous and non-scalable process when performed manually. To this end, we devise a workflow around an autoregressive transformerbased Large Language Model (LLM) towards the extraction of variables from user manuals of scientific software. Our end-toend approach, besides a prompt specification consisting of fewshot examples by a human user, is fully automated, in contrast to current practice requiring human intervention. We showcase our LLM workflow over a real case, and compare variables extracted to ground truth manually labelled by experts. Our preliminary results show that our LLM-based workflow achieves an accuracy of 0.87, while successfully deriving 61.8% of variables as partial matches and 34.7% as exact matches.