common/tasks: Add diff-quiz task for verifying code understanding #65
Conversation
Add a tasks directory designed primarily for AI agents to execute. These are called "skills" in Claude Code and "commands" in OpenCode, but they're simply structured markdown files. The first task is perform-forge-review.md, which defines an AI-augmented human-approved code review workflow. The key design principle is that the AI builds a review in a local JSONL file, which the human can inspect and edit before submission. The review is submitted as a pending/draft review, allowing the human to make final edits in the forge UI before publishing. Assisted-by: OpenCode (Claude Sonnet 4) Signed-off-by: Colin Walters <walters@verbum.org>
Add a new diff-quiz task that generates quizzes to verify human understanding of code changes. This is particularly relevant when AI tools assist in generating code, helping ensure developers understand what they're submitting. Features: - Three difficulty levels (easy, medium, hard) - Markdown or interactive bash script output formats - Separate grading step with constructive feedback - Early exit support for efficient verification Assisted-by: OpenCode (claude-sonnet-4-20250514) Signed-off-by: Colin Walters <walters@verbum.org>
Summary of ChangesHello @cgwalters, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the capabilities for AI agents by introducing a mechanism to assess human comprehension of code changes and streamlining the process of AI-assisted code reviews across various Git platforms. The core idea is to ensure that developers using AI tools maintain a deep understanding of the code they are contributing, thereby improving code quality and maintainability. It also provides a structured approach for AI agents to interact with and manage pull/merge request reviews. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces two new agent tasks: diff-quiz for verifying code understanding and perform-forge-review for AI-assisted code reviews. The changes include detailed documentation for both tasks and a suite of shell scripts to support the review workflow on GitHub, GitLab, and Forgejo. While the PR title only mentions diff-quiz, the perform-forge-review task and its scripts are also a significant and valuable addition.
The documentation is thorough and well-structured. The scripts are robust, using set -euo pipefail and including good validation.
My main feedback is about the default filenames used in the scripts, which are not unique and could cause issues with concurrent reviews. I've left specific suggestions on each of the affected scripts to make the defaults safer. This change would significantly improve the robustness of the review workflow.
| if [[ -z "$REVIEW_FILE" ]]; then | ||
| REVIEW_FILE=".git/review.jsonl" | ||
| fi |
There was a problem hiding this comment.
The default review file path .git/review.jsonl is not unique. As documented in perform-forge-review.md, using a non-unique filename can lead to comments from different reviews being mixed up if multiple reviews are performed concurrently. This could lead to incorrect reviews being submitted.
To prevent this, it's safer to make the --review-file argument mandatory. Please add a validation check for it in the Validate required arguments section (lines 90-106) and remove this default assignment.
| OWNER="$1" | ||
| REPO="$2" | ||
| PR_INDEX="$3" | ||
| REVIEW_FILE="${4:-.git/.forge-review.jsonl}" |
There was a problem hiding this comment.
The default review file path .git/.forge-review.jsonl is not unique. This can lead to comments from different reviews being mixed up if multiple reviews are performed concurrently.
Since this script knows the pull request index, a safer default would be to include it in the filename.
| REVIEW_FILE="${4:-.git/.forge-review.jsonl}" | |
| REVIEW_FILE="${4:-.git/.forge-review-$PR_INDEX.jsonl}" |
| OWNER="$1" | ||
| REPO="$2" | ||
| PR_NUMBER="$3" | ||
| REVIEW_FILE="${4:-.git/.forge-review.jsonl}" |
There was a problem hiding this comment.
The default review file path .git/.forge-review.jsonl is not unique. This can lead to comments from different reviews being mixed up if multiple reviews are performed concurrently.
Since this script knows the pull request number, a safer default would be to include it in the filename.
| REVIEW_FILE="${4:-.git/.forge-review.jsonl}" | |
| REVIEW_FILE="${4:-.git/.forge-review-$PR_NUMBER.jsonl}" |
|
|
||
| PROJECT_ID="$1" | ||
| MR_IID="$2" | ||
| REVIEW_FILE="${3:-.git/.forge-review.jsonl}" |
There was a problem hiding this comment.
The default review file path .git/.forge-review.jsonl is not unique. This can lead to comments from different reviews being mixed up if multiple reviews are performed concurrently.
Since this script knows the merge request IID, a safer default would be to include it in the filename.
| REVIEW_FILE="${3:-.git/.forge-review.jsonl}" | |
| REVIEW_FILE="${3:-.git/.forge-review-$MR_IID.jsonl}" |
➡️ 
Gotta say, medium difficulty is definitely making sure you know things... |
|
This is evil and I love it. |
2 participants
Depends on #64
name: diff-quiz
description: Generate a quiz to verify human understanding of code changes. Use when reviewing PRs or commits to ensure the submitter understands what they're proposing.
Diff Quiz
This task generates a quiz based on a git diff to help verify that a human
submitter has meaningful understanding of code they're proposing — particularly
relevant when agentic AI tools assisted in generating the code.
Purpose
When developers use AI tools to generate code, there's a risk of submitting
changes without fully understanding them. This quiz helps:
The quiz is designed to be educational, not adversarial.