- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
AtomicPO, sys prompt optimization for open ended gen. keeps a prompt pool, pairwise compares responses,extracts atomic strengths/weaknesses for both sides not just win/lose. refinement keeps the top prompt’s strengths + uses a comparative prompt and the prompt’s own weaknesses revealed against it. same model does everything, no refs/rubrics,just 4 generic criteria.
Tested on WritingBench,ResearchQA,HealthBench,PRBench w Qwen3 / Gemma4, judged by GPT-OSS-120B. 3 runs. best on all 8 combos + several ablations
Critical strengths
design makes sense,winning overall doesnt mean no issues on every dimension. comparative prompt gives the optimizer smth concrete to learn from.
gains are consistent, beats baselines everywhere in tab 2,~1.4/1.5 over PDO. 3 runs w variance. final judge is a diff model which is good
Critical weaknesses
- contribution of atomic format not fully isolated.
w/ text feedback (tab12) seems to differ from tab11 in more than format,~100 words,improvement focused,question independent vs pros/cons for both sides. so content/specificity/length change together. I noticed code also has EVALUATE_OPEN_PROMPT_WITH_REFLECTION_ATOMIC_TEXT — if that was the one used, the concern is smaller and maybe just the appendix needs updating. margins are small (+0.4), and gemma4 w/o strength 56.0 vs full 55.8 (authors already note this). overall I think the evidence supports the full system more than the individual parts
- mapping between results and setup
sec3.3 selects by # atomic weaknesses,optimization_enginev3.py seems to join them into one string and count records, which may pick different prompts
A.3 GEPA 1500 calls, test_gepa_healthbench_general_gemma30b.py L35 has 750
healthbench_train_prompt_pdo_run2.json has 400 items, paper mentions 100
could well be older scripts or resampling,not assuming anything unfair. just hard to confirm tab2/5 setup from current materials
Questions for authors
which template was used for tab5 text feedback? would a list vs prose only control be possible?
could you share code version/configs/sample idx for tab2/5, and clarify 750/1500, 400/100, records vs weaknesses? do conclusions hold under the stated setup?
from tab10 outputs look ~25-39% longer than PDO on ResearchQA/HealthBench. a length matched comparison or stratified analysis w CIs would help. not saying gains are from length, just to help separate the two answers to these would help me reassess my score
Thank you for pointing this out. After checking the script, we found that we actually use EVALUATE_OPEN_PROMPT_WITH_REFLECTION_ATOMIC_TEXT for w/ text feedback. We updated the appendix to correct this detail.
When AtomicPO selects comparative prompts, AtomicPO ranks the number of records with non-empty atomic weakness and does not count the number of atomic weakness points in each record. We find our current description can lead to misunderstanding and we will update the description.
The script for GEPA is the old version since we initially want to match the number of calls to LLM judges between baselines. We later change the script to match the number of responses generated.
It is a mistake of our resampling script and we run the experiments for PDO, AtomicPO and its ablated versions to address the issue. We updated the results on the HealthBench and do not find big differences.
To investigate whether AtomicPO's performance improvements are primarily driven by longer outputs, we conduct an additional length-controlled comparison with PDO.
Specifically, we first record the output length of AtomicPO for each test query. We then regenerate PDO's responses by appending the instruction "The response should be around X tokens," where X is the corresponding AtomicPO output length. If a generated PDO response exceeds 120% of the corresponding AtomicPO output length, we truncate it to the AtomicPO output length. We conduct this experiment using the first run of each benchmark and report the performance and average output length below. The results differ slightly from Tables 2 and 10, which report averages across three runs.
From the table, we can observe that the length of outputs of PDO almost matches the length of outputs of AtomicPO with the matching process. AtomicPO still outperforms PDO in this setting.