Basically, add a command-line option that when activated, on every testcase run, hashes the testcase, grader_flags, grader, output validator and output validator flags, and submission.
This is useful in problems with slow solutions, lots of testdata, where you want to add testcases and see if the verdict of some given submissions change.