acceptodds
Under review as a conference paper at ICLR 2027

Amortizing Cross-Example Responses from Large Language Models

10 views

Abstract

An example placed in context can change how a large language model (LLM) responds to other inputs. We show that this cross-example response can provide useful supervision when outcome labels are scarce. For each tabular record, we place the same record in context with each possible label and measure the resulting responses on a set of unlabeled probes. A small student learns to predict this response from the original features without outcome labels; the scarce labels enter only through the final classifier. The predictor distilled from the signed response reaches 63.61% balanced accuracy and outperforms raw logistic regression, Extra Trees, TabPFN-3.5, and TabICLv2. Matched controls show that the gain comes from the signed change induced by the hypothetical label. Altering the learned predictor while leaving its predictions on every labeled example unchanged substantially reduces held-out accuracy. The response data therefore supply predictive structure that the labeled examples alone do not determine. On a distinct low-label task with a different LLM backbone, the signed-response student improves balanced accuracy by 7.11 points over raw logistic regression and attains the highest point estimate among the tested methods, including XGBoost. In network-flow classification, useful information instead remains distributed across individual probe responses and is largely lost by scalar averaging. The LLM is used only offline to produce the response supervision; once the student is trained, new records require no LLM calls.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, in-context learning, knowledge distillation, privileged supervision, tabular learning, few-shot learning, behavioral supervision

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eiS4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how cross-example responses from large language models can provide additional supervision for low-label tabular prediction. Instead of directly asking an LLM to predict the label of a candidate record, the method places the same tabular record in context under different hypothetical labels and measures how this intervention changes the LLM's responses on a separate set of unlabeled probe records. The resulting response is then used as a supervision target for a lightweight student trained from raw tabular features, while scarce task labels are only used to fit the final downstream task head. As a result, no LLM inference is required at deployment time. The experiments cover Bank Marketing, Reading, Credit-g, and NetFlow, using Gemma-3-12B, Llama-3.1-8B-Instruct, and Qwen3-14B as teachers. Bank and Reading use a scalar response representation, whereas NetFlow retains a probe-resolved response profile. The paper further analyzes the proposed supervision signal through matched controls, support-null interventions, teacher substitution, and several failure-case studies.

Critical strengths
  1. Cross-example response is an interesting and reasonably novel supervision object. The main novelty is not the general paradigm of distilling a large teacher into a small model, but rather using the response induced by hypothetical-label interventions as the supervision target. Instead of directly using the teacher's p(y∣x)p(y\mid x) or hidden representation, the method captures how one example changes the LLM's behavior on other examples.

  2. The paired intervention and matched controls are well designed. For the same candidate, the method changes only the hypothetical label and uses a symmetric / antisymmetric decomposition to separate shared response level from label-induced change. On Bank, the signed response substantially outperforms symmetric mean, unsigned activity, random projection, shuffled supervision, and direct candidate margin, providing useful evidence that the predictive signal is specifically associated with the hypothetical-label-induced response change.

  3. The deployment-side amortization is practically appealing. Although offline response acquisition relies on 8B–14B teacher LLMs, the deployed students are extremely small, ranging from Ridge models with only tens of parameters to a small MLP with a few thousand parameters. New records therefore require no LLM calls at inference time, which may be attractive in settings where substantial offline computation is acceptable but deployment latency and throughput are important.

Critical weaknesses

1. The core response functionals are partly retrospectively outcome-informed.

The main response functionals for Bank, Credit-g, and canonical NetFlow were developed historically with access to panel outcomes. This makes those experiments better suited for retrospective analysis and mechanism study than for strict prospective validation of a fully specified method. The issue is important because the response functional is itself task-dependent: Bank / Reading use a signed scalar first moment, Credit-g uses a symmetric mean, and NetFlow requires a probe-resolved magnitude profile. For a genuinely unseen task, the paper still lacks an outcome-independent rule for selecting the response representation.

2. Prospective unseen-task validation remains limited.

The strongest prospective evidence mainly comes from Reading, while the separate NetFlow evaluation is more mixed. More completely unseen tasks with the full pipeline fixed before outcome access would be needed to support a broader methodological claim.

3. The evaluated tabular regime is relatively narrow.

The main experiments focus on relatively low-dimensional, binary, and extreme low-label settings. In particular, the strongest Bank and Reading results rely on compressing the LLM-derived response into a single scalar before fitting a simple downstream head. NetFlow already shows that such scalar compression may be insufficient even when the raw input dimension is small. It therefore remains unclear how well the approach extends to more complex tabular settings involving higher-dimensional features, stronger interactions, high-cardinality categoricals, or multiclass prediction.

4. Stronger and resource-matched LLM-derived supervision baselines are missing

The predictive baselines are reasonably strong, but the matched controls mainly compare variants within the proposed response family. The paper does not sufficiently compare cross-example supervision against alternatives such as teacher soft-label distillation, pseudo-labeling, embedding distillation, or LLM-generated features / rules. This comparison is especially important because the proposed method uses a substantial offline LLM budget; ideally, competing supervision methods should be evaluated under similar token or query budgets.

5. The method appears sensitive to teacher and task setting.

Teacher choice materially affects the final performance, and different tasks require different response structures. The paper does not yet identify which task properties determine when cross-example supervision is likely to work well, which limits its predictability on unseen tasks.

Questions for authors
  1. Can the authors evaluate the method on multiple completely unseen tasks, with the teacher, probe-bank construction, response functional, student architecture, and downstream procedure fixed before outcome access?

  2. Bank and Reading can use a single scalar response, whereas NetFlow requires a probe-resolved representation. Is there an outcome-independent criterion for determining when scalar compression is appropriate?

  3. How does the method perform on more complex tabular regimes, such as higher-dimensional inputs, stronger feature interactions, high-cardinality categorical features, or multiclass prediction?

  4. Can the authors include stronger alternative LLM-derived supervision baselines, such as teacher soft-label distillation, pseudo-labeling, embedding distillation, or generated semantic features / rules, under a comparable LLM token/query budget?

  5. The direct-margin baseline appears to suffer from calibration difficulty. Can the authors include a stronger calibrated direct-teacher baseline to more directly test the value of cross-example supervision over direct teacher knowledge transfer?

  6. Can the method be evaluated on a synthetic or newly collected dataset to reduce concerns about potential overlap with teacher pretraining data?

  7. How does the method extend to multiclass classification, and how does response-acquisition cost scale with the number of classes?

2 comments

Thank you for the careful review and helpful comments. We address your concerns below.

Q1/Q2. We do not have a validated rule for choosing the response functional on a new task. Credit-g illustrates the problem: response-only BA is 60.54% for Q-bar and 50.68% for D-bar, the opposite of Bank (App. E.2).

The scalar restriction appears useful when the labeled support leaves many linear directions undetermined. Bank has 53–54 encoded features but support rank only 32–34 (Table 22). NetFlow has ten features and is outside that setting. This helps explain the observed behavior, though it does not tell us which functional to choose in advance. The support-only selector is one attempt. Its mixed results and retrospective status are now stated in Section 5 and Scope.

Teacher choice also matters. The supplementary experiments cover five families: Gemma-3, Llama-3.1, Qwen3, DeepSeek V4 Flash and GPT-4o-mini. Crimes provides a positive result with a closed-weight teacher, though it remains exploratory. On Bank, Llama-3.1-8B gives a useful teacher response (62.95% BA), yet its student reaches only 56.98% (Table 28). On Adult, changing Qwen3-14B to DeepSeek V4 Flash raises student BA by 6.47 points (Table 32). Both students remain below the raw-LR comparator in Table 30. Section 5 now reports these teacher-sensitivity results more prominently.

Q3/Q7. High-cardinality categorical features and multiclass classification have not been tested. The revised Scope paragraph states this. We also have no implemented multiclass functional. One possible extension would use C hypothetical labels and at least C cyclic label-token mappings to reduce order bias. Acquisition would require O(C²K) contexts per candidate, potentially O(C × C! × K) with full permutations. This remains a proposal.

Q4/Q5. We have several relevant controls, though they do not cover all requested alternatives. The matched direct-margin student reaches 50.68% BA. The frozen 3,840-dimensional hidden-state student reaches 50.00% (Table 18). With the same unlabeled donors, self-training LR reaches 54.06%, self-training Extra Trees 53.91%, and label spreading 50.73% on Bank (Table 7).

NetFlow gives the opposite result for self-training Extra Trees: 82.92% against our 82.04% on canonical NetFlow, and 88.10% against 84.18% on the separate evaluation. This comparison is now in the main text.

We have not evaluated generated rules, token-matched LLM supervision budgets, or a direct-margin head with fixed semantic orientation. These remain limitations, as stated in the revised Scope paragraph.

Q6. Reading was posted on Mendeley in January 2025. We cannot establish that it falls outside Qwen3's pretraining data. The poor direct-margin result does not exclude contamination of the differential response either. Both limitations are now included in Scope.

Thank you for your responses. Although I acknowledge the novelty of the proposed method, my concerns about the narrow evaluated tabular regime and the presentation that needs improvement remain. Therefore, I keep my original score.

Reviewer_i5eSOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper propose to use an LLM's cross example response as supervision for tabular classification with few labels. A record xx with artificial label aa is shown to LLM as context, which is then queried with unlabeled probes. The response of probes to the change of label aa is summarized and then used as a regression target for student model. The student model further fit 50 labeled records without requiring LLM calls. The authors provide experimental results showing their method outperforms exiting methods.

Critical strengths
  1. Using the effect of a hypothetical in-context label on the LLM's answers for other inputs as a supervision signal is a new and interesting idea.
  2. The bank experiments are comprehensive and the authors are transparent to the experimental results by including cases where the method does not help.
Critical weaknesses
  1. The paper gives no reason why the response should be related to the label, and no way to tell in advance when it will help. The paper never explains why the shift in the probes' predictions under a flipped in-context label should predict the record's own label. The paper should compare with the LLM's direct zero-shot or few-shot prediction with a fixed orientation, distilled into the same student.
  2. The main Bank result was developed with access to the test outcomes. Appendix I.1 states that the response summaries for Bank, Credit-g and NetFlow "were developed historically with access to panel outcomes" and support only retrospective analyses. The authors should state which design choices were made with access to the target outcomes and report the Bank result on records that played no part in those choices.
  3. The method section is hard to follow and mixes the method with experimental choices. Basic quantities are not defined where they are used. The margin and the answer mapping in Eq. 1 only become clear from the Reading prompt in Appendix A.4, and the prompt for Bank is never shown. Half a page of derivation (Line 163-169) justifies a plain average over probes, and leftover notation (P = 2D, Line 173) has no effect because all targets are standardized. Eq. 3, Eq. 5 and Section 3.2 choose the response summary, the student and the head input separately for Bank, Reading, Credit-g and NetFlow before these datasets are introduced in Section 4, and the experimental constant of 50 labels appears in the title of Section 3.3. Nonstandard terms such as "donor set" (the transfer set in distillation), "support", "bank", "panel" and "historical" are used without definition. A self-contained description of the method, with the dataset-specific choices moved to the experimental setup, would make the paper much easier to assess.
Questions for authors

N/A

3 comments

Thank you for the careful review and helpful comments. We address your concerns below.

W1. The direct-margin baseline is already included (Fig. 3a, Table 19, App. E.4–E.5). With the same teacher, donors, Ridge student and 50-label head, it reaches 50.68% BA against 63.61% for the signed response. The difference is +12.93 [8.98, 16.84].

The direct margin does carry ranking information. Its fixed-orientation AUROC is 64.76%. The signed response reaches 64.23% after its consistently negative fitted orientation (Table 23). The difficulty is transferring the direct margin through a head trained on only 50 labels. Its fitted slope changes sign on two of five supports, and the target-to-support mean shift is 5.9 support SDs, compared with 0.36 for the signed-response student. We have not tested a direct-margin head constrained to the semantic orientation. The existing comparison therefore covers this matched calibration procedure, not every possible direct-teacher baseline.

We agree that the paper does not explain why the response should predict the candidate's outcome. We do not have a general mechanism for that. The experiments test what information the response supplies. PCA-1, reconstructed almost perfectly (R²=0.99995), gives 50.00% BA. Shuffled supervision gives 50.84%, the symmetric mean 50.03%, and a fixed random response projection 55.31% (Sec. 5.2, App. E.4–E.6).

The coefficient intervention gives a more direct test. We erase or reflect the component of the learned direction outside the labeled support's row space. All support logits remain unchanged to numerical precision (around 10⁻¹⁴), but held-out BA drops by 6.63 and 8.23 points. None of the 256 random directions or ten PCA directions produces losses as large: their maximum losses are 4.01/5.52 and 5.21/6.20 points, respectively (Sec. 5.3, App. E.5, E.7).

W2. The Bank choices made with outcomes visible include D-bar over Q-bar and |D|, scalar over profile, the scalar-only head, and Ridge alpha=1 (an earlier cache used 0.1). We cannot provide a prospective Bank result. The revised paper identifies Bank as retrospective and presents Reading as the prespecified transfer test.

W3. We revised Sections 3 and 4. Candidate, probe bank, donor set, support and panel are now defined when first used, with a terminology table and a worked Reading example in Appendix A.2. Dataset-specific choices and B=50 have moved to Section 4. The permutation argument and redundant P=2D notation have been removed from the main method.

We also recovered the prompt instructions from the supplementary artifacts. Appendix A.3 now gives the exact archived Bank system instruction and both answer mappings. The full Bank row renderer and complete chat transcripts were not preserved, which we state explicitly. Appendix A.4 provides the recovered Reading system instructions, both label mappings and the recorded message-construction procedure.

Follow-up to W1: Additional direct-margin calibration experiment

We ran the missing semantic-orientation control using the existing Bank cache, with the same five supports and evaluation targets. The original unconstrained direct-margin head reaches 50.68% BA and reverses direction on two supports. Constraining the logistic slope to be positive gives 52.34%. Using the direct-margin student's natural zero threshold gives 57.92%, recovering 7.24 points over the original baseline.

The signed-response student still reaches 63.61%, an advantage of +5.69 pp [2.43, 8.91] over the stronger zero-threshold baseline. The comparison averages the three response banks within each support before pairing.

Thus, part of the original 12.93-point gap was due to calibration, but correcting the direct-margin orientation does not eliminate the signed-response advantage. We will include this stronger baseline in the revised paper.

Thanks to the authors for the detailed response. The new baseline closes more than half of the original gap, and the remaining advantage is shown only on Bank, where the design choices were made with the test outcomes visible. The paper still gives no explanation of why the response should predict a record's label, or of when the method will help on a new task. My concerns about the motivation of this paper therefore remain, and I believe the paper in its current version does not meet the acceptance bar, so I keep my recommendation to reject.

Reviewer_nyEvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies cross-example LLM responses as supervision for low-label tabular classification. It assigns hypothetical labels to a candidate record and measures the resulting responses on unlabeled probes. A small student learns response-derived features from raw inputs, followed by a classifier fitted with 50 labeled examples. Experiments examine predictive utility, response structure, and distillation fidelity. Deployment requires no LLM calls.

Critical strengths

(1) The paper targets an important problem and proposes an interesting methods.

(2) The paper includes a relatively extensive set of experiments.

Critical weaknesses

(1) Comparison with related work and computational cost. The paper would benefit from a clearer comparison with closely related methods, particularly FeatLLM (ICML 2024) and Latte (IJCAI 2025). It remains unclear how the proposed cross-example response mechanism compares with these alternatives under similar label budgets, access to unlabeled data, and computational settings. The cost of obtaining the responses is also worth considering. For the Bank experiment, the paper reports 161.9M input tokens and 5.92 hours of response acquisition on an A100. These figures cover the complete historical response archive, including evaluation-response measurements, rather than only the responses needed to train a deployed student. Avoiding LLM inference at deployment is useful, but the overall benefit depends on whether the necessary upfront costs can be justified by subsequent use and the savings achieved. A more direct analysis of total costs, separating training requirements from diagnostic experiments and comparing with methods that also avoid deployment-time LLM calls, would help assess the practical value of the method.

(2) Retrospective method development and generalization. A concern is that some response functionals were developed with access to the corresponding panel outcomes (Appendix I.1). The reported results may overestimate how well the method would work on a new task. In the prespecified Reading experiment, the improvement over XGBoost is not statistically significant. In the separately allocated NetFlow evaluation, the improvement over raw RBF-SVM is also not statistically significant, and the proposed systems perform worse than Extra Trees and TabICLv2. The results also remain task-dependent. The automatic response-selection experiment improves average performance across eight datasets, but does not consistently outperform the matched raw-feature control. Moreover, it uses previously analyzed task panels, so it does not establish that the selection procedure will work reliably on entirely new datasets. A broader evaluation with the teacher-selection and response-selection procedures fixed before examining evaluation outcomes would help establish when the method offers a practical advantage.

Questions for authors

Please see weaknesses.

2 comments

Thank you for the careful review and helpful comments. We address your concerns below.

The Bank cost figures need separating. The reported 161.9M tokens and 5.92 hours cover all 486,912 historical contexts. Of these, 294,912 (60.6%) are evaluation-target responses used for teacher analysis. A single student with 200 donors and one bank needs 51,200 contexts, about 37 minutes on an A100 if scaled from the recorded run. Three banks for one support require 153,600 contexts, about 112 minutes. The five support conditions share a 250-donor union, requiring 192,000 contexts across three banks (Table 54). These times are estimates; we did not run donor-only timing. The revised Section 4 now includes this breakdown.

Bank shows some label efficiency. Raw LR with 250 labels reaches 58.68% BA, below the 63.61% obtained by our student with 50 labels (Table 41). This does not establish a cost break-even point. NetFlow is less convincing practically. Extra Trees and TabICLv2 outperform our method, and the evaluation student uses 1,268 donors. We use NetFlow mainly to study the response representation. We make no general cost advantage claim over methods that also avoid LLM inference at deployment.

We added Latte to Section 2 and Table 55 alongside FeatLLM. Neither method was run, so Table 55 remains a conceptual comparison. FeatLLM uses labeled examples to prompt the LLM for rules. Our response student uses no outcome labels, and the teacher's direct prediction of the candidate label is not its training target. The missing empirical comparison is now stated in Scope.

For Reading, the difference against XGBoost is +2.72 [−0.99, 6.31] (Table 11). The interval includes zero. Our student has a higher point estimate and much smaller variation across supports (SD 0.74 versus 8.01, Table 10). The paired intervals against the other six comparators exclude zero, without multiplicity correction. There is also sensitivity to probe count. At K=8, 16 and 32, BA is 65.83, 63.75 and 64.07, below raw LR's 66.09. The prespecified K=64 result reaches 73.20% (Table 38). Section 5 now discusses both the uncertainty against XGBoost and the probe-count sensitivity.

On the separate NetFlow evaluation, Extra Trees and TabICLv2 are ahead by 4.26 [3.01, 5.56] and 3.59 [2.37, 4.93] points (Table 16). These negative comparisons are now explicit in Section 5. Our finding there concerns the LR/RBF readouts and the information retained across probes.

The support-only selector in App. E.9 improves mean BA by 2.04 points over the matched raw-only selector across eight datasets. Four improve, one ties and three decline (Table 31). Crimes is worth mentioning separately. With GPT-4o-mini, the selector reaches 66.29% against 58.06% for the raw-only control, a gain of +8.24 [4.24, 12.15]. Selection used support labels, with target predictions locked before scoring in that run. Still, these panels had been analyzed historically. The revised paper reports these results as exploratory, without treating them as prospective validation.

Thank you for the detailed responses. The clarifications about the cost figures solve my concern. However, the experimental comparisons with related work and concerns about retrospective method development and generalization still exist. Thus, I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.