Random Input and Automated Output Generation in Submitty

"Fuzzing," testing a codebase against a set of randomly generated inputs, has become a promising model of testing across the industry due to its ability to reveal difficult to detect bugs. Separately, the use of randomized inputs when testing student code submissions removes the potential for student "hard-coding" behavior. Motivated by these factors, we present a solution for the automated generation of testcase inputs and expected outputs within Submitty, an open source automated grading system from Rensselaer Polytechnic Institute. We detail an enhanced workflow that allows instructors to provide our testing system with an assignment-specific input generation script and an assignment solution. The input generation script is run at student test-time, providing students with either entirely generated inputs or a combination of generated and hand-crafted testcases. The instructor solution is run against the same inputs to produce expected results. This model of testcase specification carries the benefit of simple regeneration of expected output files if an assignment's specification changes after submissions open or between semesters. We present preliminary results of the use of random input generation in our large introductory programming courses, and evaluate the ability of random inputs to curb student hardcoding behavior as it relates to an "early submission incentive" system, which grants students an extension for achieving a target assignment score early in the week an assignment is due. We examine random input generation's ability to reveal bugs in student submissions from previous semesters.