acceptodds
Under review as a conference paper at ICLR 2027

Should I Optimize Prompts for This Task? Measuring Task Suitability with CARAT

Abstract

Reflective prompt optimization, in which a reflector model revises instructions from task feedback, improves a language model on specific tasks efficiently and without updating its weights, but its outcomes are unstable. Repeated runs of the same setup with different random seeds can shift held-out accuracy considerably, and prompts with nearly the same accuracy can still solve different cases, so a single run's returned prompt says little about the task itself. We introduce CARAT, two task signals for a fixed answerer, reflector and dataset-represented task: CARAT-A measures whether prompt preferences persist across disjoint case samples, and CARAT-C measures whether the observed successes are compatible with a shared prompt. We also propose Multi-Root Full-Retention Sampling (MRFS), which explores the prompt space from independent roots under a fixed proposal budget and retains every valid proposal. Across ten tasks with Qwen3-8B as answerer and reflector, we validate the signals against collections from five prompt optimization methods over multiple seeds: both signals persist across the entire task distribution and are consistent across methods, and MRFS yields the task ordering the optimizers' collections share most. CARAT thus quantifies the task-suitability question that run-level evaluations leave open, putting prompt optimization results on a firmer footing. Code is available at https://anonymous.4open.science/r/carat-1B00.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Prompt optimization, Large language models, Task suitability, Prompt selection, Generalization

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_6krGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes CARAT, a framework comprising two metrics (CARAT-A for Accessibility and CARAT-C for Coherence) designed to evaluate whether a specific task is suitable for reflective prompt optimization. To avoid biases from specific optimizers, the authors introduce Multi-Root Full-Retention Sampling (MRFS) to construct an unbiased prompt-by-instance response matrix. CARAT-A measures whether prompt preferences remain stable across different case samples, and CARAT-C measures whether the solved instances are compatible under a shared prompt. Extensive experiments on 10 datasets across 5 prompt optimization methods demonstrate that CARAT features persist across task distributions and show high consistency independent of the search algorithms used.

Critical strengths
  • Novel Perspective on Task Diagnosis: Shifting the focus from proposing new prompt optimization algorithms to diagnosing the inherent "suitability" of the task itself is a highly valuable and underexplored perspective.
  • Decoupled and Objective Sampling: The introduction of MRFS—which retains all generated prompts regardless of their immediate performance—provides an objective, optimizer-agnostic basis for extracting the proposed metrics, effectively mitigating the survivorship bias seen in typical prompt optimization reports.
Critical weaknesses
  • Ambiguity in Mixed/Real-world Task Interpretation: Real-world applications are typically multifaceted and lack the singular characteristics of clean academic datasets. While the paper includes a "Mixture" task (HotpotQA + MMLU-Pro), it lacks a deep qualitative discussion on what the resulting readings (CARAT-A: 0.295, CARAT-C: 0.926) actually imply for engineering decisions.
  • Conflating Task Evaluation with Method Evaluation: CARAT is inherently an evaluation scheme for datasets (tasks), not for optimization methods. A low CARAT-A score under MRFS cold-start conditions indicates a rugged task landscape. However, it does not directly prove that a sophisticated optimization method (which might utilize external knowledge or different reflectors) is unsuccessful. The framing in Observation 4 risks conflating task difficulty with optimizer failure.
  • Missing Matrix Size Convergence Analysis: The detection cost is not explicitly profiled in the main text, though the method appears computationally lightweight (a significant plus). However, the paper lacks an empirical convergence study demonstrating how large the response matrix (mm prompts ×N\times N cases) needs to be for the CARAT metrics to stabilize.
  • Presentation and Clarity: The paper jumps abruptly into dense mathematical formulations (Section 4) without providing intuitive, concrete examples. The writing and flow of figures need substantial optimization to clearly illustrate how CARAT serves as a practical gating mechanism in an actual pipeline.
Questions for authors
  1. Matrix Size Convergence and Extreme Cases: Could you provide an experiment showing the convergence of CARAT-A and CARAT-C as the number of sampled prompts (mm) scales from 1 to 100+? Furthermore, regarding extreme edge cases: if N=1N=1 (a single query), there is theoretically an optimal solution. CARAT-C mathematically defaults to 11 (perfect coherence), but CARAT-A becomes undefined due to partition size limits (h≥2,n≥1h \ge 2, n \ge 1). Could you explicitly discuss how the metrics align with these boundary assumptions in the text?
  2. Interpretation of Multifaceted Tasks: For the "Mixture" dataset, how should practitioners interpret its readings compared to its constituent datasets? In real-world scenarios where tasks inherently combine multiple skills, how can CARAT precisely inform whether to continue optimizing a single prompt or to switch to a routing/multi-agent architecture?
  3. Task Metric vs. Optimizer Efficacy: Since CARAT diagnoses the inherent difficulty of a dataset under a cold-start background, how can we definitively distinguish whether a low-performance outcome is due to the task's low accessibility (rugged landscape) or the specific optimization method's weak search policy?

I am currently leaning towards a Weak Reject (Score 2) due to the lack of convergence analysis and the blurred boundary between task and optimizer evaluation. However, I am open to raising my score to an Accept if Question 1 and Question 3 are robustly addressed during the rebuttal.

9 comments
AuthorsOpenReview ↗

Response to Reviewer 6krG (1/4): summary; convergence as prompts grow (Q1)

Response to Reviewer 6krG (1/4)

We thank the reviewer for the careful review and for stating which questions decide the score. We answer Question 1 and Question 3 first (A, B, C), then the remaining points (D to F).

In brief

  • Convergence in prompts (A1). At 24 prompts, half of a draw, the mean reading is within 0.001 (CARAT-A) and 0.003 (CARAT-C) of the published one, and single subsets vary less than the published draws do. Beyond 100 prompts CARAT-A does not move and CARAT-C falls by 0.003 on average.
  • Matrix size (A2). We suggest 24 prompts × 150 cases as the minimum matrix: there the differences between tasks are 3.5 times (CARAT-A) and 4.1 times (CARAT-C) the spread of a single reading. At one draw's size (48 × 300) they are 6.6 and 9.0 times.
  • Task or optimizer (B). A low reading under MRFS points to the task; a high MRFS reading with a low outcome from one optimizer's run points to that method's search. A method shifts the level of its own readings and does not reorder the tasks.
  • Boundary cases (C). §4.3 and App. A already state them: with one case or one prompt the readings are 1 or undefined by definition, and with no reversal CARAT-A is infinite. The revision spells out the single-case and single-prompt cases.
  • Using a reading (D, E). CARAT-A says how many cases a task needs before a prompt can be selected. CARAT-C says whether the prompts' successes are compatible.
  • Revision. App. D gains the convergence table, the grid of prompts × cases, the fresh-answer control and the table of gap recovered by number of cases; §4 opens with the motivation and worked calculations and spells out the boundary cases; Obs. 4 no longer ranks the methods.

A. Convergence of the readings (Question 1, Weakness 3)

A1. As the number of prompts grows (Question 1)

"Could you provide an experiment showing the convergence of CARAT-A and CARAT-C as the number of sampled prompts (m) scales from 1 to 100+?"

Both readings settle by 24 prompts, half of a draw. Beyond 100 prompts CARAT-A does not move and CARAT-C falls by 0.003 on average.

We ran it on the recorded draws. For each size we took 200 random subsets of prompts from each of the three MRFS draws of every task. "Difference" is the mean reading of the subsets minus the published reading, averaged over the ten tasks, with its SD across tasks in brackets. "SD across subsets" is the typical spread of single subsets of one draw; the last row gives the SD across the three published draws for comparison (Table 14). Task order is the number of task pairs, out of 45, in the published order.

PromptsCARAT-A: difference from published (SD across tasks)CARAT-A: SD across subsetsCARAT-A: task orderCARAT-A: task order, single drawCARAT-C: difference from published (SD across tasks)CARAT-C: SD across subsetsCARAT-C: task orderCARAT-C: task order, single draw
1undefined1 if it solves any case
2−0.279 (0.193)†0.295†33.527.6+0.056 (0.047)0.02938.934.5
4−0.026 (0.055)†0.215†40.537.2+0.029 (0.027)0.02742.740.3
8−0.001 (0.030)†0.131†42.139.9+0.014 (0.013)0.01744.143.0
16+0.001 (0.006)0.07043.141.2+0.005 (0.005)0.01144.543.7
24+0.001 (0.003)0.04643.641.7+0.003 (0.002)0.00844.744.0
Published draws0.0680.009

† Subsets with no reversal have an infinite CARAT-A and are left out of the mean and the SD (part C).

The level. At 24 prompts the mean reading is within 0.001 (CARAT-A) and 0.003 (CARAT-C) of the published one. CARAT-A is close on average from 4 prompts (−0.026) and level at 8 (−0.001), but single tasks still differ there: the SD across tasks is 0.055 at 4 prompts and 0.030 at 8, against 0.003 at 24. CARAT-C reads higher in small collections (+0.014 at 8 prompts) and comes down as prompts are added.

The spread. Single subsets of 8 or 16 prompts vary more than the published draws do (CARAT-A 0.131 and 0.070 against 0.068). At 24 prompts they vary less (0.046 against 0.068 for CARAT-A; 0.008 against 0.009 for CARAT-C).

The task order. It settles before the level does: 4 prompts already give about 40 (CARAT-A) and 43 (CARAT-C) of the 45 task pairs.

Beyond 100 prompts. Taking each task's three draws together on the same cases gives 115 to 144 prompts. With all of them, CARAT-A differs from the published reading by 0.000 on average (SD across tasks 0.009), and CARAT-C by −0.003 (0.003; at most −0.011). The task order agrees on 44 (CARAT-A) and 45 (CARAT-C) of the 45 task pairs.

These are subsets and unions of the recorded draws, so the spread narrows as a subset approaches the full draw.

Revision. App. D adds the table.

AuthorsOpenReview ↗

Response to Reviewer 6krG (2/4): minimum matrix size (W3)

Response to Reviewer 6krG (2/4)

A. Convergence of the readings (continued)

A2. How large the response matrix needs to be (Weakness 3)

"the paper lacks an empirical convergence study demonstrating how large the response matrix (m prompts × N cases) needs to be for the CARAT metrics to stabilize."

We suggest 24 prompts × 150 cases as the minimum matrix. At that size the differences between tasks are 3.5 times (CARAT-A) and 4.1 times (CARAT-C) the spread of a single reading; at one draw's size (48 × 300) they are 6.6 and 9.0 times.

We ran the study on the recorded matrices, over a grid of 2 to 144 prompts and 10 to 300 cases. At each size we drew 100 random sub-matrices per task, keeping each revision with its root prompt, and computed both readings on each. A matrix is large enough when its readings tell tasks apart, so at each size we report the signal-to-noise ratio: the SD of the readings between tasks divided by the typical SD of a single reading at that size.

Prompts × casesCARAT-A: signal-to-noiseCARAT-C: signal-to-noise
12 × 1502.72.5
24 × 150 (suggested minimum)3.54.1
24 × 3004.25.6
48 × 1505.35.3
48 × 2506.27.8
48 × 300 (one draw's size)6.69.0
96 × 30012.019.3

The suggested minimum. At 24 prompts × 150 cases the ratio is 3.5 for CARAT-A and 4.1 for CARAT-C, and it is at least that at every larger size on the grid. With 12 prompts on the same cases it falls below 3 on both. From 48 × 150 it is at least 5 on both.

What sets the size. More prompts sharpen both readings: at 300 cases, going from 24 to 48 to 96 prompts raises the ratio from 4.2 to 6.6 to 12.0 for CARAT-A and from 5.6 to 9.0 to 19.3 for CARAT-C. More cases matter most for CARAT-C: at 48 prompts, going from 150 to 300 cases raises its ratio from 5.3 to 9.0.

Fresh cases. These sizes come from resampling the recorded matrices, so close to the full matrix part of the sharpening is mechanical. The check on cases outside the matrix is in the paper: readings from 200 cases persist on the 100 held-out cases (Obs. 1, Table 1). More than two revision rounds or 300 cases cannot be read from the recorded draws.

Revision. App. D adds the grid of prompts × cases with this table, and §5 states the suggested minimum in one sentence.

AuthorsOpenReview ↗

Response to Reviewer 6krG (3/4): task or optimizer (Q3, W2); boundary cases (Q1)

Response to Reviewer 6krG (3/4)

B. Task or optimizer (Question 3, Weakness 2)

"how can we definitively distinguish whether a low-performance outcome is due to the task's low accessibility (rugged landscape) or the specific optimization method's weak search policy?"

A low reading under MRFS points to the task; a high MRFS reading with a low outcome from one optimizer's run points to that method's search.

Why the MRFS reading stands for the task. MRFS starts from independent roots and retains every valid proposal, including revisions that do not improve (§1), so it is tied to no starting prompt or search policy (§6.2). The five optimizers validate this: the MRFS task order is the most shared signal, its average agreement with the other procedures is 80.4% for CARAT-A and 92.0% for CARAT-C (Table 2). On the eight common tasks it contradicts none of the task orders the five optimizers share, while each optimizer contradicts up to three (App. D). One MRFS measurement is what a user needs.

What a method's weak search looks like. A method shifts the level of its own readings, consistently across tasks (Kendall's W 0.54, Obs. 4), and does not reorder the tasks.

"The framing in Observation 4 risks conflating task difficulty with optimizer failure."

Observation 4. It concerns the methods, not the tasks, and the reviewer's concern holds for one part of it: it ranked the methods. The five optimizers keep nearly the same accessibility order across the eight common tasks (W 0.54), while the order of their final prompts' accuracy does not repeat (W 0.07). We now state this as a property of the prompt space each method's search visits. It carries no better-or-worse conclusion about the methods, and a method may do better under a different setup. The revision withdraws the ranking: the mean-rank row of Table 3, "six of eight" and the last sentence of Obs. 4 are removed.

CARAT-C. The same holds for coherence. CARAT-C is 1 exactly when one prompt in the collection solves every case solved anywhere in it (§4.2), so a prompt that combines the others' successes, found by any search, would raise the reading. Prompts pooled across all five optimizers and seeds track the MRFS readings of CARAT-C (r = 0.98) and keep the task order on all 45 task pairs (Obs. 3). We also regenerated every answer of one draw from an empty cache on 150 cases. For each pair of prompts, we took the cases one solved and the other missed in one set of answers and checked whether that advantage returned in the other set, in both directions. The smaller returning advantage is at most 0 in expectation unless the prompts truly solve different cases. On the task with the lowest CARAT-C it averages 0.10 (95% CI [0.04, 0.13]): the prompts do succeed on different cases, and none of the tested searches found a prompt that combines those successes.

Revision. Obs. 4 and Table 3 are revised as above. App. D adds the fresh-answer control.

C. Boundary cases (Question 1)

"if N = 1 (a single query) … CARAT-C mathematically defaults to 1 (perfect coherence), but CARAT-A becomes undefined due to partition size limits (h ≥ 2, n ≥ 1). Could you explicitly discuss how the metrics align with these boundary assumptions in the text?"

§4.3 and App. A already state these boundaries (declared boundaries; boundary conventions; Prop. A.8), and the revision spells out the single-case and single-prompt cases in both. With one case or one prompt the readings are 1 or undefined by definition, and neither is evidence about the task.

CaseCARAT-ACARAT-CStated in
One case (N = 1)Undefined: it needs h ≥ 2 and h + n ≤ N, so at least three cases (four with the default sizes)1 if a prompt solves the case, as the reviewer says; undefined if none doesApp. A, boundary conventions; Prop. A.8; §4.2
One promptUndefined: there is no prompt pair1 if the prompt solves any caseApp. A, boundary conventions; Prop. A.8
No reversal among the usable pairs+∞, with no clipping and no added constantNot affected§4.3; App. A, boundary conventions
No case solved by any promptUndefined: the prompts are identicalUndefinedApp. A, boundary conventions; §4.2

What the boundary values mean. CARAT-C is 1 exactly when one prompt solves every solved case (Prop. A.8). With one case or one prompt nothing can conflict, so CARAT-C = 1 there does not show coherence.

No reversal in practice. No draw in the paper has a reversal rate of 0; the lowest is 0.005. It occurs only in small subsets of a draw: 204 and 22 of the 6,000 subsets at 4 and 8 prompts in part A, and none from 12 prompts. The averages in part A leave those subsets out.

Revision. §4.3 and the boundary conventions of App. A now spell out the single-case and single-prompt cases.

Reviewer_qA4KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the issue of task suitability for reflection-based prompt optimization. Since different random seeds and optimization methods can yield varying outcomes, relying solely on accuracy is insufficient to determine whether a given task is well-suited for prompt optimization. To this end, the authors propose the CARAT framework, which employs two metrics: one to assess the stability of comparisons between different prompts (CARAT-A), and another to evaluate the consistency of the problems addressed by various prompts (CARAT-C). Furthermore, the authors introduce the Multi-start Retain-All Sampling (MRFS) method, which initiates the process from multiple independent starting prompts, performs a limited number of refinement iterations, and retains all valid candidates to construct a unified prompt performance matrix. In their experiments, the authors utilize Qwen3-8B as both the solver and the reflection model, evaluating the proposed metrics across 10 tasks and comparing them against five existing prompt optimization methods. The results demonstrate that CARAT-A and CARAT-C exhibit strong stability across different test samples and maintain consistent performance even when applied to candidate sets generated by various optimization methods. Moreover, the task rankings derived via MRFS align closely with the overall outcomes from several existing optimization approaches, suggesting that this method effectively reflects the suitability of different tasks for prompt optimization.

Critical strengths
  1. Novel evaluation metrics. CARAT addresses an underexplored aspect of prompt optimization: task suitability. Its two components, Accessibility (CARAT-A) and Coherence (CARAT-C), capture complementary aspects of prompt optimization that cannot be fully reflected by accuracy alone.

  2. Comprehensive empirical evaluation. The authors evaluate the proposed metrics on 10 tasks and compare them with five prompt optimization methods. Experiments across multiple random seeds and test samples provide evidence that the metrics are reasonably consistent across different evaluation settings and optimization approaches.

Critical weaknesses
  1. Cross-model generalizability remains unverified. All experiments use Qwen3-8B as both the answerer and the reflector. It remains unclear whether the CARAT metrics and resulting task rankings would remain consistent across different model architectures, scales, and answerer–reflector combinations.

  2. Limited prompt-space coverage and potential computational costs. Each MRFS run starts with only 16 initial prompts, with at most two rounds of refinement per prompt. This limited exploration may not adequately capture the broader prompt space, making the resulting CARAT metrics sensitive to the sampled candidate collection. In addition, evaluating every retained prompt on all evaluation instances requires substantial model inference, which may limit the scalability of the approach to larger datasets and candidate collections.

Questions for authors
  1. Could you include results for CARAT using updated models (e.g., Qwen3.8-27B, Kimi-K3)? Relying solely on the Qwen3 series limits the scope of the experimental findings and offers little insight into the performance of current, state-of-the-art models.
  2. Given that evaluating all prompts across all benchmarks incurs a high computational cost, could you consider calculating the similarity between prompts and selecting a subset of sufficiently distinct prompts for subsequent operations?
  3. How sensitive are CARAT-A and CARAT-C to the sampling budget allocated for MRFS—specifically regarding the initial number of prompts and the number of modification rounds? Could the authors provide experimental evidence demonstrating that the current budget is sufficient to yield reasonably reliable metric estimates?
6 comments
AuthorsOpenReview ↗

Response to Reviewer qA4K (1/3): summary; two further models (Q1, W1)

Response to Reviewer qA4K (1/3)

We thank the reviewer for the constructive review. We ran the two analyses the reviewer asks for: CARAT on further models (A) and the sensitivity to the MRFS budget (B). The similarity-based subset and scalability follow (C, D).

In brief

  • Further models (A). With Qwen3.8-27B and with DeepSeek-V4.1-Flash, readings on 200 evaluation cases persist on 100 held-out cases: CARAT-A differs by 0.029 and 0.038 on average, CARAT-C by 0.008 on both.
  • Budget (B). A smaller budget still tells the tasks apart: with 24 prompts (8 roots) the differences between tasks are 4.2 times (CARAT-A) and 5.6 times (CARAT-C) the spread of a single reading, against 6.6 and 9.0 times at the paper's budget.
  • Distinct subset (C). We tested it: a most-dissimilar subset is not closer to the full reading than a random subset of the same size.
  • Scalability (D). CARAT reads a task from a sample of its cases: readings from 200 cases hold on 100 held-out cases (Obs. 1), so the cost is set by the sample and does not grow with the dataset.
  • Revision. §6 reports the held-out check on both further models (Observation 1, Table 1); App. D gains the budget table and the subset test; App. B.5 states that the cost is set by the sample of cases.

A. Further models (Question 1, Weakness 1)

"Could you include results for CARAT using updated models (e.g., Qwen3.8-27B, Kimi-K3)? Relying solely on the Qwen3 series limits the scope of the experimental findings…"

On two further models, one larger and one from a different model family, readings again persist on held-out cases.

We ran one MRFS draw per task with Qwen3.8-27B and with DeepSeek-V4.1-Flash, each as answerer and reflector, and repeated the paper's held-out check (Obs. 1): readings on 200 evaluation cases against 100 held-out cases. ARC-Challenge and GSM8K are left out, because both models solve over 95% of them and they need no prompt optimization.

ModelCARAT-A: mean difference (spread between tasks)CARAT-C: mean difference (spread between tasks)
Qwen3.8-27B0.029 (0.247)0.008 (0.032)
DeepSeek-V4.1-Flash0.038 (0.170)0.008 (0.026)

Over 100 further splits, each task's mean difference is within 0.015 of zero on both models.

Task order. CARAT is defined for a declared answerer, reflector and task (§3.1), and changing either model "requires a new measurement" (App. D.3), so the task order need not match Qwen3-8B's. For CARAT-A the rank correlation with Qwen3-8B's order is 0.93 on DeepSeek and 0.55 on Qwen3.8-27B; for CARAT-C it is 0.48 and 0.52.

Both runs use one model in both roles; mixed answerer–reflector pairs are not tested.

Revision. §6 reports the held-out check on both models beside Qwen3-8B's (Observation 1, Table 1); App. D gives the per-split results.

AuthorsOpenReview ↗

Response to Reviewer qA4K (2/3): MRFS budget sensitivity (Q3, W2); distinct-prompt subset (Q2)

Response to Reviewer qA4K (2/3)

B. Sensitivity to the MRFS budget (Question 3, Weakness 2)

"How sensitive are CARAT-A and CARAT-C to the sampling budget allocated for MRFS—specifically regarding the initial number of prompts and the number of modification rounds? Could the authors provide experimental evidence demonstrating that the current budget is sufficient to yield reasonably reliable metric estimates?"

A smaller budget still tells the tasks apart. With 24 prompts (8 roots) the differences between tasks are 4.2 times (CARAT-A) and 5.6 times (CARAT-C) the spread of a single reading; at the paper's budget of 48 prompts (16 roots) they are 6.6 and 9.0 times.

Current budget yield reasonably reliable task property across methods. Every task has three independent MRFS draws, and Table 14 reports each reading with its standard deviation across draws. §6.2 shows that the MRFS task order agrees with the orders from five optimizers' collections more than any of those agree with each other (80.4% for CARAT-A, 92.0% for CARAT-C; Table 2).

Smaller budgets. We drew random sub-collections of the recorded prompts, keeping each revision with its root, and computed both readings on all 300 cases (100 sub-collections per task and size). By design a root with its two revisions gives three prompts, so the number of roots is a third of the number of prompts. The signal-to-noise ratio is the SD of the readings between tasks divided by the typical SD of a single reading at that size.

Prompts (roots)CARAT-A: signal-to-noiseCARAT-C: signal-to-noise
12 (4)3.22.9
24 (8)4.25.6
48 (16; the paper's budget)6.69.0
96 (32)12.019.3

With fewer cases as well, 24 prompts × 150 cases still gives 3.5 and 4.1, and we suggest that as the minimum matrix.

Revision rounds. Sampling 16 whole roots with their revisions, the ratio is 3.9 (CARAT-A) and 6.5 (CARAT-C) with the roots alone, 5.1 and 7.9 after one revision round, and 6.4 and 8.5 after two. Sixteen whole roots average 45 prompts, so these last values sit slightly below the table's row for 48 prompts. Each round raises the ratio on both metrics. The level moves little after the first round: with one round CARAT-A reads 0.019 lower and CARAT-C 0.005 higher than the published readings on average.

These are sub-collections of the recorded draws, so the largest size, which uses most of the recorded prompts, is flattered; more than two revision rounds cannot be read from them.

Revision. App. D adds the budget table.

C. A subset of distinct prompts (Question 2)

"could you consider calculating the similarity between prompts and selecting a subset of sufficiently distinct prompts for subsequent operations?"

We tested it: a most-dissimilar subset is not closer to the full reading than a random subset of the same size.

At 8, 16 and 24 prompts we chose the most mutually dissimilar prompts, by word overlap of the instruction text, and compared them with random subsets of the same size. The dissimilar subset was closer to the full-draw reading on both metrics on 1, 0 and 0 of the 10 tasks.

A uniform cut gives the saving the reviewer is after: 24 prompts still tell the tasks apart (part B). MRFS retains every prompt so that the collection is not itself selected, and a similarity filter would reintroduce a selection step.

Revision. App. D reports the test.

AuthorsOpenReview ↗

Response to Reviewer qA4K (3/3): scalability (W2)

Response to Reviewer qA4K (3/3)

D. Scalability to larger datasets and collections (Weakness 2)

"evaluating every retained prompt on all evaluation instances requires substantial model inference, which may limit the scalability of the approach to larger datasets and candidate collections."

CARAT scales to larger datasets because it reads a task from a sample of its cases: readings computed on 200 cases hold on 100 cases they never used (Obs. 1). Its cost is set by the sample, not by the size of the dataset.

Larger datasets. The paper tests exactly this. Each task's 300 evaluation cases are split at random into 200 cases on which both metrics are computed and 100 held-out cases from the same distribution (§5.2). The two sides differ by 0.024 (CARAT-A) and 0.017 (CARAT-C) on average, against between-task SDs of 0.312 and 0.069 (Table 1), and 100 further random splits agree (App. D.2). The same holds on the two further models (part A). A reading therefore describes the task's distribution, and a larger dataset needs no larger sample: an evenly sampled subset is enough.

What does grow with the dataset. The cost that grows is the optimizer's. A prompt optimizer scores its candidate prompts on its selection cases at every step, so using more of a larger dataset multiplies its evaluations. CARAT is read once, from the sample.

Larger collections. The cost grows with the number of prompts, and it need not. Part B shows that 24 prompts (8 roots), half of a draw, still tell the tasks apart: the differences between tasks are 4.2 times (CARAT-A) and 5.6 times (CARAT-C) the spread of a single reading on 300 cases. The matrix of prompts × cases in our reply to Reviewer 6krG (part A2) gives this ratio at every size. At 24 prompts × 150 cases it is 3.5 and 4.1, which we suggest as the minimum matrix: a quarter of one draw's prompt–case pairs.

The cost itself. One draw evaluates up to 14,400 prompt–case pairs (48 prompts on 300 cases; App. B.5). Once that matrix is recorded, both metrics are computed from it with no further model inference.

Revision. App. B.5 states that the cost is set by the sample of cases and points to Obs. 1.

Comments on the paper

9 comments
AuthorsOpenReview ↗

General response: new results and changes in the revised PDF

General response to all reviewers

We thank the three reviewers for careful and constructive reviews. We ran the analyses the reviews ask for. The detailed replies are posted under each review; this note lists the new results and the changes in the revised PDF.

New results

  • CARAT-A predicts prompt selection (zL37; 6krG). CARAT-A is read on 200 cases, a prompt is selected on 12 to 100 cases and evaluated on 100 held-out cases. Higher CARAT-A goes with a lower selection error, more frequent selection of a top-quartile prompt and lower regret as a share of the gap between the average and the best prompt (rank correlations −0.67, 0.73 and −0.89 at 50 cases), at each of six budgets. On the four highest-reading tasks 12 cases recover 63–82% of that gap; on the three lowest, 100 cases recover 0–10%. This is what a reading tells an engineer: how many cases a task needs before a prompt can be selected.
  • Readings reproduce on fresh answers (zL37; 6krG). With the collection fixed and every answer regenerated from an empty cache, CARAT-A changes by at most 0.029 and CARAT-C by at most 0.012. On the task with the lowest CARAT-C, prompts succeed on different cases and the same differences reappear in the fresh answers (0.10, 95% CI [0.04, 0.13]).
  • Two further models (qA4K). With Qwen3.8-27B and DeepSeek-V4.1-Flash, readings on 200 cases persist on 100 held-out cases: CARAT-A differs by 0.029 and 0.038 on average, CARAT-C by 0.008 on both.
  • Convergence and matrix size (6krG; qA4K). At 24 prompts, half of a draw, the mean reading is within 0.001 (CARAT-A) and 0.003 (CARAT-C) of the published one; beyond 100 prompts CARAT-A does not move and CARAT-C falls by 0.003. We suggest 24 prompts × 150 cases as the minimum matrix: there the differences between tasks are 3.5 times (CARAT-A) and 4.1 times (CARAT-C) the spread of a single reading, against 6.6 and 9.0 times at one draw's size.
  • What CARAT-A adds to simpler summaries (zL37). Against the reference-ordered share it reads how far ahead the better prompts are; against accuracy spread it reads whether prompts solve the same cases.
  • Task or optimizer (6krG). A low reading under MRFS points to the task; a high MRFS reading with a low outcome from one optimizer's run points to that method's search.

Changes in the revised PDF

  • §4 opens with the motivation and introduces each metric with a worked calculation before its definition. §4.2 describes low CARAT-C as incompatibility observed among the collected prompts. §4.3 spells out the single-case and single-prompt boundary cases. §4.4 gains a short guide to reading the two metrics together and the reading among top-quartile prompts.
  • §5 reports the two further models and states the suggested minimum matrix.
  • §6 Observation 1 and Table 1 report the held-out check on both further models. Observation 4 no longer ranks the methods, and the mean-rank row of Table 3 is removed.
  • Figure 2: the caption points to the worked calculations.
  • App. A adds the single-case and single-prompt cases to the boundary conventions. App. B.5 states that the cost of a reading is set by the sample of cases.
  • App. D adds the decision validation, the comparison with the two summaries and the constructed checks, the fresh-answer control, the convergence table and the grid of prompts × cases, the budget table, the distinct-subset test, the reading among top-quartile prompts, the reading of the Mixture task and the per-split results of the further models.
AuthorsOpenReview ↗

Response to Reviewer zL37 (1/5): summary

Response to Reviewer zL37 (1/5)

We thank the reviewer for a careful and constructive review. We ran both analyses the reviewer asks for. We answer the two weaknesses first (A, B) and the two related points after (C, D).

In brief

  • Decision validation (A). Higher CARAT-A goes with a lower selection error, more frequent selection of a top-quartile prompt and lower regret as a share of the gap between the average and the best prompt (rank correlations −0.67, 0.73 and −0.89), at each of six budgets from 12 to 100 cases. A reading taken on the full collection also predicts selection among as few as 4 of its prompts.
  • Fresh-answer control (B). With the collection fixed and every answer regenerated from an empty cache, CARAT-A changes by at most 0.029 and CARAT-C by at most 0.012. On the task with the lowest CARAT-C, prompts succeed on different cases, and the same differences reappear in the fresh answers.
  • Simpler summaries (C). Against the reference-ordered share, which §4.1 reports with every reading, CARAT-A reads how far ahead the better prompts are; against accuracy spread, it reads whether prompts solve the same cases. Read on the same cases, the three track the selection outcomes about equally.
  • Revision. App. D gains the validation table, the comparison with the two summaries and the control; §4.4 adds the reading among the top-quartile prompts; low CARAT-C is described as incompatibility observed among the collected prompts.
AuthorsOpenReview ↗

Response to Reviewer zL37 (2/5): CARAT-A predicts selection error and regret (W1, Q1)

Response to Reviewer zL37 (2/5)

A. Does CARAT-A predict selection error or regret as the selection case budget changes? (Weakness 1, Question 1)

"They do not show that higher CARAT A predicts more reliable selection of a strong prompt or lower selection regret at a fixed evaluation budget." / "Within the existing finite candidate collections, does CARAT A predict independently evaluated prompt selection error or regret as the selection case budget changes?"

Yes at every budget: for selection error, for selecting a top-quartile prompt, and for regret as a share of the gap between the average and the best prompt.

We ran the decision validation the reviewer asks for, within the existing collections. CARAT-A is read on 200 evaluation cases. A prompt is selected on 12 to 100 of those cases (the budget) and evaluated on 100 held-out cases. The collections are the 27 MRFS draws of the nine tasks with enough cases for this split.

Selection error. Higher CARAT-A goes with a lower selection error: the selected prompt's held-out accuracy falls below the best prompt's less often (rank correlation ρ −0.67 [−0.80, −0.31] at a budget of 50 cases). It also goes with more frequent selection of a top-quartile prompt (0.73 [0.34, 0.86]). Both hold at each of the six budgets (table).

Regret. We take the reviewer's regret to be the accuracy the selected prompt loses against the best prompt on the held-out cases. As a share of the gap between the average and the best prompt, regret falls with CARAT-A at every budget (ρ −0.89 [−0.94, −0.63]); one minus this share is the gap recovered. On the three lowest-reading tasks a prompt selected from 50 cases recovers 0–3% of the gap; on the four highest, 77–92%.

Why regret is read as a share of the gap. Regret in accuracy points is the width of the gap between prompts multiplied by the share of that gap the selection fails to recover. Accessibility concerns the second factor: whether the cases available for selection lead to the better prompts. The width of the gap is how far apart the prompts' accuracies lie, which is a different property of the collection.

Selection among fewer prompts. A reading taken on the full collection also predicts selection among a subset of its prompts, and more strongly as the subset shrinks. We kept each draw's readings from all of its prompts, drew 200 random subsets of 16, 8 and 4 prompts, selected among each subset on 50 or 100 cases, and evaluated on the 100 held-out cases. With 50 cases, the rank correlation of CARAT-A with selection error goes from −0.67 with all prompts to −0.88 with 4 prompts, and with regret as a share of the gap from −0.89 to −0.95 (second table).

The comparison with the simpler summaries is in part C; read on the same cases they track these outcomes about equally.

Revision. App. D reports the validation with the two tables below, and D.3's "requires separate validation" now points to it.

CARAT-A against selection outcomes, by budget. Rank correlation (ρ) across 27 MRFS draws (nine tasks, three draws each). A prompt is selected on the stated number of cases and evaluated on 100 held-out cases.

Cases available for selection1225375075100
Selection error−0.65−0.70−0.67−0.67−0.65−0.53
Top-quartile prompt selected0.760.760.760.730.710.72
Regret as a share of the gap−0.89−0.92−0.90−0.89−0.91−0.89

Readings from the full collection against selection among a random subset of its prompts. Rank correlation (ρ) across the 27 MRFS draws, with 200 random subsets per draw and size. A prompt is selected among the subset on 50 or 100 cases and evaluated on 100 held-out cases.

OutcomePrompts to select amongCARAT-A, 50 casesOrdered share, 50 casesAccuracy spread, 50 casesCARAT-A, 100 casesOrdered share, 100 casesAccuracy spread, 100 cases
Selection errorAll−0.67−0.67−0.62−0.53−0.52−0.48
16−0.76−0.76−0.70−0.78−0.76−0.71
8−0.81−0.80−0.75−0.80−0.79−0.74
4−0.88−0.88−0.83−0.89−0.88−0.83
Regret as a share of the gapAll−0.89−0.89−0.89−0.89−0.87−0.87
16−0.91−0.91−0.89−0.89−0.88−0.86
8−0.94−0.92−0.91−0.92−0.91−0.90
4−0.95−0.94−0.92−0.94−0.93−0.91

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.