Add robustness testing integration with BenchDrift for m-programs #15
Conversation
…ting - Add variation_types parameter to run_benchdrift_pipeline() to allow users to customize which semantic variation types to generate (generic, cluster_variations, persona, long_context) - Update test/1_test_robustness_testing.py to demonstrate variation_types usage - Add docs/ROBUSTNESS_TESTING.md with comprehensive documentation for robustness testing workflow - Enables fine-grained control over robustness testing configurations 🤖 Generated with Claude Code Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
delucs21
left a comment
delucs21
left a comment
There was a problem hiding this comment.
Reviewed with some necessary changes before merging
| ### Step 1: Install BenchDrift | ||
| Install BenchDrift from source (required for robustness testing pipeline): | ||
| ```bash | ||
| git clone https://github.com/ritterinvest/BenchDrift.git |
There was a problem hiding this comment.
This repo returns 404. I proceeded with testing using the internal repo, but BenchDrift needs to be in a publicly accessible repo.
There was a problem hiding this comment.
I suggest renaming this file for consistency. Perhaps something like "test_benchdrift_robustness.py"
There was a problem hiding this comment.
I suggest renaming this to be more specific - something like "benchdrift_model_client_adapter.py"
| import logging | ||
| import tempfile | ||
| from typing import List, Dict, Any, Callable, Optional, Tuple | ||
|
|
| 'unified_file': temp_output_filename, | ||
| 'input_problems': temp_input_filename, | ||
| 'batch_size': 2, | ||
| 'max_workers': 4, |
There was a problem hiding this comment.
max_workers is hardcoded instead of using the passed parameter.
|
|
||
| # --- Core API Functions --- | ||
|
|
||
| def run_benchdrift_pipeline( |
There was a problem hiding this comment.
Documentation defines variation_types parameter for run_benchdrift_pipeline, but it is missing from the function implementation. Following the doc raises a TypeError.
| def run_benchdrift_pipeline( | ||
| baseline_problem: str, | ||
| ground_truth_answer: str, | ||
| m_program_callable: Optional[Callable[[str, Dict[str, Any]], Any]] = None, |
There was a problem hiding this comment.
m_program_callable type hint implies 2 arguments, but invocations define 1. Should this be Callable[[str], Any] instead?
2 participants
Summary
This PR adds the ability to test Mellea m-program robustness by integrating with BenchDrift- semantic variation generation and evaluation pipeline. Users can now systematically evaluate how consistently their m-programs answer semantically equivalent variations of a problem.
What This Enables
Key Components
run_benchdrift_pipeline(): Orchestrates BenchDrift's 3-stage pipeline (generate variations → execute m-program → evaluate)MelleaModelClientAdapter: Bridges Mellea m-programs to BenchDrift's test frameworkanalyze_robustness_from_probes(): Computes robustness metrics from test results