- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how cross-example responses from large language models can provide additional supervision for low-label tabular prediction. Instead of directly asking an LLM to predict the label of a candidate record, the method places the same tabular record in context under different hypothetical labels and measures how this intervention changes the LLM's responses on a separate set of unlabeled probe records. The resulting response is then used as a supervision target for a lightweight student trained from raw tabular features, while scarce task labels are only used to fit the final downstream task head. As a result, no LLM inference is required at deployment time. The experiments cover Bank Marketing, Reading, Credit-g, and NetFlow, using Gemma-3-12B, Llama-3.1-8B-Instruct, and Qwen3-14B as teachers. Bank and Reading use a scalar response representation, whereas NetFlow retains a probe-resolved response profile. The paper further analyzes the proposed supervision signal through matched controls, support-null interventions, teacher substitution, and several failure-case studies.
Critical strengths
-
Cross-example response is an interesting and reasonably novel supervision object. The main novelty is not the general paradigm of distilling a large teacher into a small model, but rather using the response induced by hypothetical-label interventions as the supervision target. Instead of directly using the teacher's or hidden representation, the method captures how one example changes the LLM's behavior on other examples.
-
The paired intervention and matched controls are well designed. For the same candidate, the method changes only the hypothetical label and uses a symmetric / antisymmetric decomposition to separate shared response level from label-induced change. On Bank, the signed response substantially outperforms symmetric mean, unsigned activity, random projection, shuffled supervision, and direct candidate margin, providing useful evidence that the predictive signal is specifically associated with the hypothetical-label-induced response change.
-
The deployment-side amortization is practically appealing. Although offline response acquisition relies on 8B–14B teacher LLMs, the deployed students are extremely small, ranging from Ridge models with only tens of parameters to a small MLP with a few thousand parameters. New records therefore require no LLM calls at inference time, which may be attractive in settings where substantial offline computation is acceptable but deployment latency and throughput are important.
Critical weaknesses
1. The core response functionals are partly retrospectively outcome-informed.
The main response functionals for Bank, Credit-g, and canonical NetFlow were developed historically with access to panel outcomes. This makes those experiments better suited for retrospective analysis and mechanism study than for strict prospective validation of a fully specified method. The issue is important because the response functional is itself task-dependent: Bank / Reading use a signed scalar first moment, Credit-g uses a symmetric mean, and NetFlow requires a probe-resolved magnitude profile. For a genuinely unseen task, the paper still lacks an outcome-independent rule for selecting the response representation.
2. Prospective unseen-task validation remains limited.
The strongest prospective evidence mainly comes from Reading, while the separate NetFlow evaluation is more mixed. More completely unseen tasks with the full pipeline fixed before outcome access would be needed to support a broader methodological claim.
3. The evaluated tabular regime is relatively narrow.
The main experiments focus on relatively low-dimensional, binary, and extreme low-label settings. In particular, the strongest Bank and Reading results rely on compressing the LLM-derived response into a single scalar before fitting a simple downstream head. NetFlow already shows that such scalar compression may be insufficient even when the raw input dimension is small. It therefore remains unclear how well the approach extends to more complex tabular settings involving higher-dimensional features, stronger interactions, high-cardinality categoricals, or multiclass prediction.
4. Stronger and resource-matched LLM-derived supervision baselines are missing
The predictive baselines are reasonably strong, but the matched controls mainly compare variants within the proposed response family. The paper does not sufficiently compare cross-example supervision against alternatives such as teacher soft-label distillation, pseudo-labeling, embedding distillation, or LLM-generated features / rules. This comparison is especially important because the proposed method uses a substantial offline LLM budget; ideally, competing supervision methods should be evaluated under similar token or query budgets.
5. The method appears sensitive to teacher and task setting.
Teacher choice materially affects the final performance, and different tasks require different response structures. The paper does not yet identify which task properties determine when cross-example supervision is likely to work well, which limits its predictability on unseen tasks.
Questions for authors
-
Can the authors evaluate the method on multiple completely unseen tasks, with the teacher, probe-bank construction, response functional, student architecture, and downstream procedure fixed before outcome access?
-
Bank and Reading can use a single scalar response, whereas NetFlow requires a probe-resolved representation. Is there an outcome-independent criterion for determining when scalar compression is appropriate?
-
How does the method perform on more complex tabular regimes, such as higher-dimensional inputs, stronger feature interactions, high-cardinality categorical features, or multiclass prediction?
-
Can the authors include stronger alternative LLM-derived supervision baselines, such as teacher soft-label distillation, pseudo-labeling, embedding distillation, or generated semantic features / rules, under a comparable LLM token/query budget?
-
The direct-margin baseline appears to suffer from calibration difficulty. Can the authors include a stronger calibrated direct-teacher baseline to more directly test the value of cross-example supervision over direct teacher knowledge transfer?
-
Can the method be evaluated on a synthetic or newly collected dataset to reduce concerns about potential overlap with teacher pretraining data?
-
How does the method extend to multiclass classification, and how does response-acquisition cost scale with the number of classes?
2 comments
Thank you for your responses. Although I acknowledge the novelty of the proposed method, my concerns about the narrow evaluated tabular regime and the presentation that needs improvement remain. Therefore, I keep my original score.
Thank you for the careful review and helpful comments. We address your concerns below.
Q1/Q2. We do not have a validated rule for choosing the response functional on a new task. Credit-g illustrates the problem: response-only BA is 60.54% for Q-bar and 50.68% for D-bar, the opposite of Bank (App. E.2).
The scalar restriction appears useful when the labeled support leaves many linear directions undetermined. Bank has 53–54 encoded features but support rank only 32–34 (Table 22). NetFlow has ten features and is outside that setting. This helps explain the observed behavior, though it does not tell us which functional to choose in advance. The support-only selector is one attempt. Its mixed results and retrospective status are now stated in Section 5 and Scope.
Teacher choice also matters. The supplementary experiments cover five families: Gemma-3, Llama-3.1, Qwen3, DeepSeek V4 Flash and GPT-4o-mini. Crimes provides a positive result with a closed-weight teacher, though it remains exploratory. On Bank, Llama-3.1-8B gives a useful teacher response (62.95% BA), yet its student reaches only 56.98% (Table 28). On Adult, changing Qwen3-14B to DeepSeek V4 Flash raises student BA by 6.47 points (Table 32). Both students remain below the raw-LR comparator in Table 30. Section 5 now reports these teacher-sensitivity results more prominently.
Q3/Q7. High-cardinality categorical features and multiclass classification have not been tested. The revised Scope paragraph states this. We also have no implemented multiclass functional. One possible extension would use C hypothetical labels and at least C cyclic label-token mappings to reduce order bias. Acquisition would require O(C²K) contexts per candidate, potentially O(C × C! × K) with full permutations. This remains a proposal.
Q4/Q5. We have several relevant controls, though they do not cover all requested alternatives. The matched direct-margin student reaches 50.68% BA. The frozen 3,840-dimensional hidden-state student reaches 50.00% (Table 18). With the same unlabeled donors, self-training LR reaches 54.06%, self-training Extra Trees 53.91%, and label spreading 50.73% on Bank (Table 7).
NetFlow gives the opposite result for self-training Extra Trees: 82.92% against our 82.04% on canonical NetFlow, and 88.10% against 84.18% on the separate evaluation. This comparison is now in the main text.
We have not evaluated generated rules, token-matched LLM supervision budgets, or a direct-margin head with fixed semantic orientation. These remain limitations, as stated in the revised Scope paragraph.
Q6. Reading was posted on Mendeley in January 2025. We cannot establish that it falls outside Qwen3's pretraining data. The poor direct-margin result does not exclude contamination of the differential response either. Both limitations are now included in Scope.