It's painful to debug interactive output validators

[Matistjati]

The reason is mostly that it's nontrivial for a casual user to reproduce the exact way in which we run interactive problems.
For example, I recently got a nonobvious RTE. Even when running with -l debug, the logs don't help.

DEBUG Grader flags: ['ignore_sample']
DEBUG run "/opt/venvs/kattis-problemtools/lib/python3.12/site-packages/problemtools/run/../support/default_grader ignore_sample < /tmp/tmputpeck_g > /tmp/tmpt8m08dev 2> /dev/null"
DEBUG Grade on testcase group data is RTE (0.0)
DEBUG Grading 2 results:
RTE 0.0
RTE 0.0

DEBUG Grader flags: ['ignore_sample']
DEBUG run "/opt/venvs/kattis-problemtools/lib/python3.12/site-packages/problemtools/run/../support/default_grader ignore_sample < /tmp/tmpdy3k4hft > /tmp/tmpmjs6llqc 2> /dev/null"
DEBUG Grading 2 results:
RTE 0.0
RTE 0.0

DEBUG Grader flags: ['ignore_sample']
DEBUG run "/opt/venvs/kattis-problemtools/lib/python3.12/site-packages/problemtools/run/../support/default_grader ignore_sample < /tmp/tmpkdgy474a > /tmp/tmpa5rlun1t 2> /dev/null"
ERROR AC submission randomized_only_assume_1_simd.cpp (C++) got RTE [testcase: testcase secret/g6/001-simple-1, CPU: 0.00s @ testcase secret/g1/001-simple-1]

[gkreitz]

I agree that debugging interactive problems is a pain. I'm not sure what would be a good way to fix that.

I'm struggling a bit to make sense of this ticket. The issue title says it's difficult to debug interactive output validators, but the log files seem to show an interactive submission getting RTE.

Do you have any thoughts on what would have helped you debug here? Being able to dump the conversation between validator and submission? Being able to dump just what the validator wrote (so you can then run the submission with that as input)? Just capturing stderr from the components and displaying it as a debug log?

[Matistjati]

In this instance, what would've helped would be to capture stderr from both the validator and the submission, and displaying them. It would also be nice to have a short script to run an output validator along with a submission interactively. But this probably doesn't belong in problemtools.

[Tagl]

What about a command line flag to specify you want to step through the interaction process? Haven't thought about the feasibility of this as is but it would be nice to be able to do that and simply specify a testcase or have it do that for each test case and then you can simply Ctrl+D to ask it to finish or Ctrl+C to cancel.

[gkreitz]

It would also be nice to have a short script to run an output validator along with a submission interactively. But this probably doesn't belong in problemtools.

I think a tool to help debug interactive (or, probably more generally, submissions and validators) would be a fine addition to problemtools. I always find it annoying to debug validators, as one needs to set up a directory, run the program with a couple of command line arguments, and then check the directory. And interactive is even worse (worse command lines, more complexity).

Scope creeping this issue a lot, and brain dumping some loose ideas for what I think a "debug" script could do:

• Just help the user build the command lines ("Give me a command line to run the validator on this file", "Give me the command line to run interactive with this submission and this test case")
• Compile and run a submission on a test case, printing output to stdout (debug run data/secret/01.in submissions/accepted/foo.cc)
• Run an output validator on an output file, specifying the test case (debug validate data/secret/01.in < foo.txt), parsing score and summarizing other feedback files
• Combine the two steps above, running a submission on a test case and validating the output
• Run an interactive submission and grader on a test case. Ideally also dumping/capturing stderr.

Longer term, I want to move towards a proxy for interactive grading (cf. #113). That would allow us to do many more things with interactive (we could maybe do some hacks before, but I think I'd prefer to get the proxy in first). I love the idea from @Tagl of being able to step through the interaction process.

So, to summarize, I think we're currently a bit limited in how much we can do for interactive problems beyond just making it more convenient to run them (possibly also extracting stderr). However, I think a debug script could also be useful for non-interactive problems (above I only talk about "normal" problems - I assume multi-pass will also be fun to debug, but I haven't worked on any myself yet). And this ticket is very timely, as I'm thinking about a proxy, and I wouldn't have thought of the feature to step through an interaction without the discussion here.