- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces GAL, a framework that combines adversarial benchmark generation with automated algorithm discovery. The two agent work in an iterative loop, allowing the system to discover new weakness and develop solutions. This paper seems like a case study report, because it does not provide sufficient results to validate the effectiveness of the proposed framework. Also, I am curious about whether this paper is produced by automated research?
Critical strengths
-
I think the idea of automatically discovering weaknesses and using them to guode further improvements is interesting. Somehow, this paper, from my perspective, is not completed yet.
-
The authors evaluate this framework on four different tasks, but almost any of them only provides a case analysis, which can not be used to validate the effectiveness of the proposed framework.
Critical weaknesses
-
The overall idea is relatively straightforward, where one agent searches for failure cases, and another agent tries to improve the algorithm based on those failures. Both parts are implemented using existing tools, such as GEPA and coding agents. Similar ideas have also been explored in previous work, including Hacker-Fixer Loops.
-
The paper does not provide enough comparisons to show why the proposed framework is necessary. How does GAL compare with simply using an LLM to generate challenging test cases, or using a coding agent to improve algorithms on a fixed benchmark? Without these baselines, it is difficult to understand what GAL actually contributes beyond combining existing techniques.
-
Although the paper covers four inference tasks, most results are presented as individual examples rather than systematic evaluations. Besides, the complete adversarial loop is only demonstrated on two tasks with two iterations each.
-
In Figure 7, the evolved algorithms improve substantially on the generated adversarial benchmarks, but some original benchmark scores remain unchanged or become worse. This makes the claim of general algorithm improvement less convincing.
Questions for authors
-
Could authors provide comparisons with simpler baselines, such as random adversarial search, direct LLM-based test generation, and algorithm optimization without the discriminator?
-
What is the overall success rate of the discriminator across different tasks? How many candidates are generated, and how many valid failure cases are found? Could the authors report the average performance gap and variance across multiple independent runs, rather than mainly showing selected examples?
-
Do the discovered algorithms improve on independently generated adversarial examples that were not used during the optimization process?
-
In Figure 7, some standard benchmark results become worse after algorithm evolution. How do the authors balance performance on newly discovered adversarial examples with performance on existing benchmarks?
-
Why is the full GAL loop only evaluated for two iterations on two tasks? Could the authors provide more iterations or repeated experiments to show that the framework can consistently discover new failure cases and improve the algorithms?
5 comments
Success rate and variance of discriminator (Q2)
Figure 6 in the paper shows the gap plots (avg and std-dev) against $ cost for various models and setups for both KV compression and context extension. These plots show results for 10 independent runs of discriminator for these two domains. The following tables show the data for these plots.
KV Compression via FastKVZip
| model | max avg gap | std-dev |
|---|---|---|
| gepa-gpt-5.6-luna | 0.585 | 0.243 |
| gepa-gpt-5.6-terra | 0.600 | 0.269 |
| gepa-gpt-5.6-sol | 0.295 | 0.090 |
| gepa-gemini-3.5-flash-lite | 0.310 | 0.257 |
| gepa-gemini-3.6-flash | 0.300 | 0.105 |
| gepa-gemini-3.1-pro-preview | 0.365 | 0.206 |
Context Extension via DCA
| model | max avg gap | std-dev |
|---|---|---|
| gepa-gpt-5.6-luna | 1.000 | 0.000 |
| gepa-gpt-5.6-terra | 1.000 | 0.000 |
| gepa-gpt-5.6-sol | 0.955 | 0.142 |
| gepa-gemini-3.5-flash-lite | 0.700 | 0.256 |
| gepa-gemini-3.6-flash | 0.960 | 0.070 |
Even in the least performing model, the discriminator ends up discovering an average gap of 0.295 for KV compression with FaskKVZip and 0.70 for context extension with DCA.
Additionally, in the five-round loops, every round produced a break. During full GAL loops the number of candidates evaluated before discriminator succeeded were as following:
| Round | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| Context extension (DCA) | 1 | 8 | 36 | 10 | 7 |
| KV compression (Compactor) | 7 | 22 | 43 | 8 | 5 |
For sparse video generation, we present statistics over all 725 search candidates instead of the selected example in the paper. A failure counts as valid only if the dense video still matches its prompt, so that the damage is due to sparse attention.
| Objective (target) | Candidates | Valid failures | LPIPS > 0.5 | PSNR < 18 dB |
|---|---|---|---|---|
| Divergence × coherence (SVG1) | 242 | 90% | 76% | 82% |
| Divergence × VBench damage (SVG1) | 223 | 66% | 25% | 44% |
| Pixel MSE (SVG1) | 136 | 95% | 43% | 62% |
| Pixel MSE (SVG2) | 124 | 4% | 0% | 0% |
Improvements on independently generated adversarial examples (Q3)
We perform this experiment in the KV compression setting. At the beginning, we generate multiple adversarial benchmark candidates and evaluate the candidates produced by Gen1 and Gen2 on these held-out adversarial benchmarks. In the table below, B1 and B2 are the adversarial benchmarks used in GAL rounds 1 and 2, respectively, while C1–C13 are adversarial candidates which were created at Base and that were never shown to the generator. As we can see, the improvements generally transfer to other held-out adversarial datasets.
| Algorithm | B1 | B2 | C1 | C2 | C3 | C4 | C5 | C6 | C7 | C8 | C9 | C10 | C11 | C12 | C13 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Base | 0.25 | 0.45 | 0.35 | 0.45 | 0.25 | 0.00 | 0.00 | 0.00 | 0.00 | 0.05 | 0.10 | 0.05 | 0.45 | 0.30 | 0.50 |
| Gen1 | 0.85 | 0.25 | 0.35 | 0.35 | 0.05 | 0.10 | 0.00 | 0.05 | 0.10 | 0.10 | 0.10 | 0.05 | 0.65 | 1.00 | 1.00 |
| Gen2 | 0.95 | 1.00 | 0.55 | 0.75 | 0.60 | 0.35 | 0.05 | 0.20 | 0.45 | 0.75 | 0.55 | 0.70 | 0.85 | 0.75 | 1.00 |
Note that, in practice, a generator may produce candidates that perform worse on unseen adversarial examples. However, in our experiments, the improvements generally transfer across the board to other held-out adversarial benchmarks.
Extension to 5 GAL iterations and more standard benchmark evaluation (Q4 and Q5)
In the following tables, we present extensions of our experiment over two axis
(1) We run additional GAL loops upto 5 loops in total. (We would like to clarify that GAL is a bottleneck by the evaluation we have to do in every sub iteration in discriminator and Generator. For example, a single GAL loop takes up to 20 hours in KV Compression.)
(2) We also expand the tables with more standard benchmarks not part of B0 which is shown to the discriminator and generator to show that these improvements do not come at the cost of existing performance. Note that in an ideal invocation of GAL, B0 should be representative of all evaluation data that Algorithm saturates. But practically this is not possible due to cost and time constraints.
In the following results all benchmarks other than [B0-B5] are heldout existing benchmarks.
KV compression with Compactor
| Algorithm | B0 (RULER-HARD) | B1 | B2 | B3 | B4 | B5 | LongBench |
|---|---|---|---|---|---|---|---|
| no-press | 0.8823 | 1.00 | 0.95 | 1.00 | 1.00 | 1.00 | 0.4344 |
| Gen0 (Compactor) | 0.7652 | 0.35 | 0.30 | 0.00 | 0.00 | 1.00 | 0.3768 |
| Gen1 | 0.7684 | 0.85 | 0.25 | 0.00 | 0.20 | 1.00 | 0.3784 |
| Gen2 | 0.7581 | 0.95 | 1.00 | 0.05 | 0.05 | 1.00 | 0.3914 |
| Gen3 | 0.7557 | 1.00 | 0.95 | 0.90 | 0.30 | 1.00 | 0.3759 |
| Gen4 | 0.7800 | 1.00 | 1.00 | 1.00 | 0.95 | 0.40 | 0.3717 |
| Gen5 | 0.7847 | 1.00 | 1.00 | 1.00 | 0.85 | 0.85 | 0.3836 |
RULER-HARD: 7 task sets - 350 examples and LongBench: 21 task sets - 1050 examples
The per-task results reported in the paper are based on a sample of 20 examples per task and represent just one evaluation run, leaving room for sampling-related variation. The table above reports results using a larger sample of 50 examples per task, showing more consistent improvements or, at minimum, comparable performance.
Context extension with DCA
| Algorithm | B0 (Simple Niah) | B1 | B2 | B3 | B4 | B5 | RULER, 13 tasks (390 rows) | LongBench, 15 English tasks (550 rows > 8k) | Passkey, 16k and 32k (50 each) | Needle, all depths, 16k and 32k | L-Eval Coursera / SciFi (average) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| No extension | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | |||||
| Gen0 (DCA) | 0.97 | 0.40 | 0.68 | 0.30 | 0.38 | 0.78 | 0.601 | 45.38 | 1.00 | 1.00 | 58 / 68 (63) |
| Gen1 | 1.00 | 0.95 | 0.70 | 0.42 | 0.38 | 0.80 | 0.713 | 46.12 | 1.00 | 1.00 | 60 / 78 (69) |
| Gen2 | 1.00 | 0.93 | 0.90 | 0.50 | 0.33 | 0.33 | 0.709 | 46.03 | 1.00 | 1.00 | 60 / 74 (67) |
| Gen3 | 0.97 | 0.88 | 0.93 | 0.85 | 0.35 | 0.12 | 0.685 | 44.91 | 1.00 | 1.00 | 58 / 76 (67) |
| Gen4 | 1.00 | 0.97 | 0.93 | 0.88 | 0.85 | 0.00 | 0.657 | 45.18 | 1.00 | 1.00 | 58 / 76 (67) |
| Gen5 | 1.00 | 1.00 | 0.95 | 0.80 | 0.82 | 0.93 | 0.692 | 45.41 | 1.00 | 1.00 | 58 / 76 (67) |
We hope these results address reviewer’s concerns and would be glad to answer any further questions.
We thank the reviewer for the time and effort. To address the reviewer's concerns we ran a broader set of experiments to support our experimental hypothesis. We appreciate that they find the idea of letting discovered weaknesses guide algorithm improvement interesting and want to clarify that the paper was not produced by automated research. The idea, implementation, experiments and writing are the authors’ own; we used AI assistants for coding and for polishing the text (i.e blobs of text were written manually and AI was asked to take an edit pass to fix grammar), as stated in the paper.
The premise of GAL is that as systems increasingly saturate established benchmarks, scientific progress requires benchmarks to evolve alongside them. Our experiments with both non-evolutionary and agentic benchmark-generation baselines demonstrate the limitations of existing approaches and motivate the need for a system that can continually advance benchmark generation. Our results show that GAL achieves better discriminator outcomes at lower cost than both non-evolutionary and agentic benchmark-generation baselines.
Importantly, although GAL leverages existing methods for optimization and evolution, its underlying approach to driving system advancement is not derived from any one of these individual techniques. Rather, GAL introduces a new paradigm in which the process of evaluating and improving systems is itself continuously evolving. We believe this approach can contribute to sustaining scientific progress, particularly in ML and, more broadly, across data-driven fields.
We acknowledge that other works are emerging in similar directions. However, these works are concurrent with ours and generally address more specific, distinct problems. For instance, Hacker-Fixer Loops (mentioned by a reviewer), which appeared in June 2026 (concurrent work), focuses on hardening the verifiers of fixed agent benchmarks such as KernelBench and Terminal-Bench. In contrast, GAL focuses on improving the algorithm under test by continually discovering new adversarial inputs against which it is evaluated. Thus, while both explore iterative improvement, they address fundamentally different aspects of the research process: improving the reliability of evaluation mechanisms versus advancing the capabilities of the systems by advancing evaluation datasets.
To support the above claims, we present the evidence below:
Comparison with simpler baselines (Q1)
1. Random adversarial search, direct LLM-based test generation
We already compare discriminator results with GEPA based evolution to Coding Agent based evolution in the paper (Figure 6 left for KV compression). We also compare with plain LLM discriminator under same model and cost budget below -
2. Algorithmic optimization without discriminator
The premise of GAL is that as systems saturate for standard benchmarks, there is a need for pushing the benchmarks ahead in order to further drive and evaluate system improvements. Hence, while system evolution may perform well on existing benchmarks, GAL’s value is in identifying new benchmarks when algorithm evolution alone saturates.
(continued)