- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes CARAT, a framework comprising two metrics (CARAT-A for Accessibility and CARAT-C for Coherence) designed to evaluate whether a specific task is suitable for reflective prompt optimization. To avoid biases from specific optimizers, the authors introduce Multi-Root Full-Retention Sampling (MRFS) to construct an unbiased prompt-by-instance response matrix. CARAT-A measures whether prompt preferences remain stable across different case samples, and CARAT-C measures whether the solved instances are compatible under a shared prompt. Extensive experiments on 10 datasets across 5 prompt optimization methods demonstrate that CARAT features persist across task distributions and show high consistency independent of the search algorithms used.
Critical strengths
- Novel Perspective on Task Diagnosis: Shifting the focus from proposing new prompt optimization algorithms to diagnosing the inherent "suitability" of the task itself is a highly valuable and underexplored perspective.
- Decoupled and Objective Sampling: The introduction of MRFS—which retains all generated prompts regardless of their immediate performance—provides an objective, optimizer-agnostic basis for extracting the proposed metrics, effectively mitigating the survivorship bias seen in typical prompt optimization reports.
Critical weaknesses
- Ambiguity in Mixed/Real-world Task Interpretation: Real-world applications are typically multifaceted and lack the singular characteristics of clean academic datasets. While the paper includes a "Mixture" task (HotpotQA + MMLU-Pro), it lacks a deep qualitative discussion on what the resulting readings (CARAT-A: 0.295, CARAT-C: 0.926) actually imply for engineering decisions.
- Conflating Task Evaluation with Method Evaluation: CARAT is inherently an evaluation scheme for datasets (tasks), not for optimization methods. A low CARAT-A score under MRFS cold-start conditions indicates a rugged task landscape. However, it does not directly prove that a sophisticated optimization method (which might utilize external knowledge or different reflectors) is unsuccessful. The framing in Observation 4 risks conflating task difficulty with optimizer failure.
- Missing Matrix Size Convergence Analysis: The detection cost is not explicitly profiled in the main text, though the method appears computationally lightweight (a significant plus). However, the paper lacks an empirical convergence study demonstrating how large the response matrix ( prompts cases) needs to be for the CARAT metrics to stabilize.
- Presentation and Clarity: The paper jumps abruptly into dense mathematical formulations (Section 4) without providing intuitive, concrete examples. The writing and flow of figures need substantial optimization to clearly illustrate how CARAT serves as a practical gating mechanism in an actual pipeline.
Questions for authors
- Matrix Size Convergence and Extreme Cases: Could you provide an experiment showing the convergence of CARAT-A and CARAT-C as the number of sampled prompts () scales from 1 to 100+? Furthermore, regarding extreme edge cases: if (a single query), there is theoretically an optimal solution. CARAT-C mathematically defaults to (perfect coherence), but CARAT-A becomes undefined due to partition size limits (). Could you explicitly discuss how the metrics align with these boundary assumptions in the text?
- Interpretation of Multifaceted Tasks: For the "Mixture" dataset, how should practitioners interpret its readings compared to its constituent datasets? In real-world scenarios where tasks inherently combine multiple skills, how can CARAT precisely inform whether to continue optimizing a single prompt or to switch to a routing/multi-agent architecture?
- Task Metric vs. Optimizer Efficacy: Since CARAT diagnoses the inherent difficulty of a dataset under a cold-start background, how can we definitively distinguish whether a low-performance outcome is due to the task's low accessibility (rugged landscape) or the specific optimization method's weak search policy?
I am currently leaning towards a Weak Reject (Score 2) due to the lack of convergence analysis and the blurred boundary between task and optimizer evaluation. However, I am open to raising my score to an Accept if Question 1 and Question 3 are robustly addressed during the rebuttal.
9 comments
Response to Reviewer 6krG (2/4): minimum matrix size (W3)
Response to Reviewer 6krG (2/4)
A. Convergence of the readings (continued)
A2. How large the response matrix needs to be (Weakness 3)
"the paper lacks an empirical convergence study demonstrating how large the response matrix (m prompts × N cases) needs to be for the CARAT metrics to stabilize."
We suggest 24 prompts × 150 cases as the minimum matrix. At that size the differences between tasks are 3.5 times (CARAT-A) and 4.1 times (CARAT-C) the spread of a single reading; at one draw's size (48 × 300) they are 6.6 and 9.0 times.
We ran the study on the recorded matrices, over a grid of 2 to 144 prompts and 10 to 300 cases. At each size we drew 100 random sub-matrices per task, keeping each revision with its root prompt, and computed both readings on each. A matrix is large enough when its readings tell tasks apart, so at each size we report the signal-to-noise ratio: the SD of the readings between tasks divided by the typical SD of a single reading at that size.
| Prompts × cases | CARAT-A: signal-to-noise | CARAT-C: signal-to-noise |
|---|---|---|
| 12 × 150 | 2.7 | 2.5 |
| 24 × 150 (suggested minimum) | 3.5 | 4.1 |
| 24 × 300 | 4.2 | 5.6 |
| 48 × 150 | 5.3 | 5.3 |
| 48 × 250 | 6.2 | 7.8 |
| 48 × 300 (one draw's size) | 6.6 | 9.0 |
| 96 × 300 | 12.0 | 19.3 |
The suggested minimum. At 24 prompts × 150 cases the ratio is 3.5 for CARAT-A and 4.1 for CARAT-C, and it is at least that at every larger size on the grid. With 12 prompts on the same cases it falls below 3 on both. From 48 × 150 it is at least 5 on both.
What sets the size. More prompts sharpen both readings: at 300 cases, going from 24 to 48 to 96 prompts raises the ratio from 4.2 to 6.6 to 12.0 for CARAT-A and from 5.6 to 9.0 to 19.3 for CARAT-C. More cases matter most for CARAT-C: at 48 prompts, going from 150 to 300 cases raises its ratio from 5.3 to 9.0.
Fresh cases. These sizes come from resampling the recorded matrices, so close to the full matrix part of the sharpening is mechanical. The check on cases outside the matrix is in the paper: readings from 200 cases persist on the 100 held-out cases (Obs. 1, Table 1). More than two revision rounds or 300 cases cannot be read from the recorded draws.
Revision. App. D adds the grid of prompts × cases with this table, and §5 states the suggested minimum in one sentence.
Response to Reviewer 6krG (3/4): task or optimizer (Q3, W2); boundary cases (Q1)
Response to Reviewer 6krG (3/4)
B. Task or optimizer (Question 3, Weakness 2)
"how can we definitively distinguish whether a low-performance outcome is due to the task's low accessibility (rugged landscape) or the specific optimization method's weak search policy?"
A low reading under MRFS points to the task; a high MRFS reading with a low outcome from one optimizer's run points to that method's search.
Why the MRFS reading stands for the task. MRFS starts from independent roots and retains every valid proposal, including revisions that do not improve (§1), so it is tied to no starting prompt or search policy (§6.2). The five optimizers validate this: the MRFS task order is the most shared signal, its average agreement with the other procedures is 80.4% for CARAT-A and 92.0% for CARAT-C (Table 2). On the eight common tasks it contradicts none of the task orders the five optimizers share, while each optimizer contradicts up to three (App. D). One MRFS measurement is what a user needs.
What a method's weak search looks like. A method shifts the level of its own readings, consistently across tasks (Kendall's W 0.54, Obs. 4), and does not reorder the tasks.
"The framing in Observation 4 risks conflating task difficulty with optimizer failure."
Observation 4. It concerns the methods, not the tasks, and the reviewer's concern holds for one part of it: it ranked the methods. The five optimizers keep nearly the same accessibility order across the eight common tasks (W 0.54), while the order of their final prompts' accuracy does not repeat (W 0.07). We now state this as a property of the prompt space each method's search visits. It carries no better-or-worse conclusion about the methods, and a method may do better under a different setup. The revision withdraws the ranking: the mean-rank row of Table 3, "six of eight" and the last sentence of Obs. 4 are removed.
CARAT-C. The same holds for coherence. CARAT-C is 1 exactly when one prompt in the collection solves every case solved anywhere in it (§4.2), so a prompt that combines the others' successes, found by any search, would raise the reading. Prompts pooled across all five optimizers and seeds track the MRFS readings of CARAT-C (r = 0.98) and keep the task order on all 45 task pairs (Obs. 3). We also regenerated every answer of one draw from an empty cache on 150 cases. For each pair of prompts, we took the cases one solved and the other missed in one set of answers and checked whether that advantage returned in the other set, in both directions. The smaller returning advantage is at most 0 in expectation unless the prompts truly solve different cases. On the task with the lowest CARAT-C it averages 0.10 (95% CI [0.04, 0.13]): the prompts do succeed on different cases, and none of the tested searches found a prompt that combines those successes.
Revision. Obs. 4 and Table 3 are revised as above. App. D adds the fresh-answer control.
C. Boundary cases (Question 1)
"if N = 1 (a single query) … CARAT-C mathematically defaults to 1 (perfect coherence), but CARAT-A becomes undefined due to partition size limits (h ≥ 2, n ≥ 1). Could you explicitly discuss how the metrics align with these boundary assumptions in the text?"
§4.3 and App. A already state these boundaries (declared boundaries; boundary conventions; Prop. A.8), and the revision spells out the single-case and single-prompt cases in both. With one case or one prompt the readings are 1 or undefined by definition, and neither is evidence about the task.
| Case | CARAT-A | CARAT-C | Stated in |
|---|---|---|---|
| One case (N = 1) | Undefined: it needs h ≥ 2 and h + n ≤ N, so at least three cases (four with the default sizes) | 1 if a prompt solves the case, as the reviewer says; undefined if none does | App. A, boundary conventions; Prop. A.8; §4.2 |
| One prompt | Undefined: there is no prompt pair | 1 if the prompt solves any case | App. A, boundary conventions; Prop. A.8 |
| No reversal among the usable pairs | +∞, with no clipping and no added constant | Not affected | §4.3; App. A, boundary conventions |
| No case solved by any prompt | Undefined: the prompts are identical | Undefined | App. A, boundary conventions; §4.2 |
What the boundary values mean. CARAT-C is 1 exactly when one prompt solves every solved case (Prop. A.8). With one case or one prompt nothing can conflict, so CARAT-C = 1 there does not show coherence.
No reversal in practice. No draw in the paper has a reversal rate of 0; the lowest is 0.005. It occurs only in small subsets of a draw: 204 and 22 of the 6,000 subsets at 4 and 8 prompts in part A, and none from 12 prompts. The averages in part A leave those subsets out.
Revision. §4.3 and the boundary conventions of App. A now spell out the single-case and single-prompt cases.
Response to Reviewer 6krG (1/4): summary; convergence as prompts grow (Q1)
Response to Reviewer 6krG (1/4)
We thank the reviewer for the careful review and for stating which questions decide the score. We answer Question 1 and Question 3 first (A, B, C), then the remaining points (D to F).
In brief
A. Convergence of the readings (Question 1, Weakness 3)
A1. As the number of prompts grows (Question 1)
Both readings settle by 24 prompts, half of a draw. Beyond 100 prompts CARAT-A does not move and CARAT-C falls by 0.003 on average.
We ran it on the recorded draws. For each size we took 200 random subsets of prompts from each of the three MRFS draws of every task. "Difference" is the mean reading of the subsets minus the published reading, averaged over the ten tasks, with its SD across tasks in brackets. "SD across subsets" is the typical spread of single subsets of one draw; the last row gives the SD across the three published draws for comparison (Table 14). Task order is the number of task pairs, out of 45, in the published order.
† Subsets with no reversal have an infinite CARAT-A and are left out of the mean and the SD (part C).
The level. At 24 prompts the mean reading is within 0.001 (CARAT-A) and 0.003 (CARAT-C) of the published one. CARAT-A is close on average from 4 prompts (−0.026) and level at 8 (−0.001), but single tasks still differ there: the SD across tasks is 0.055 at 4 prompts and 0.030 at 8, against 0.003 at 24. CARAT-C reads higher in small collections (+0.014 at 8 prompts) and comes down as prompts are added.
The spread. Single subsets of 8 or 16 prompts vary more than the published draws do (CARAT-A 0.131 and 0.070 against 0.068). At 24 prompts they vary less (0.046 against 0.068 for CARAT-A; 0.008 against 0.009 for CARAT-C).
The task order. It settles before the level does: 4 prompts already give about 40 (CARAT-A) and 43 (CARAT-C) of the 45 task pairs.
Beyond 100 prompts. Taking each task's three draws together on the same cases gives 115 to 144 prompts. With all of them, CARAT-A differs from the published reading by 0.000 on average (SD across tasks 0.009), and CARAT-C by −0.003 (0.003; at most −0.011). The task order agrees on 44 (CARAT-A) and 45 (CARAT-C) of the 45 task pairs.
These are subsets and unions of the recorded draws, so the spread narrows as a subset approaches the full draw.
Revision. App. D adds the table.