acceptodds
Under review as a conference paper at ICLR 2027

Beyond Prediction: A Component-Level Study of Concept Recovery in In-Context Learning

Abstract

In-context learning (ICL) enables language models to make accurate predictions from demonstrations, but prediction success alone does not reveal the extent to which the underlying concept has been recovered. Existing evaluations typically assess recovery at the level of the whole concept, making it difficult to identify which components are recovered and to separate model limitations from ambiguity in the demonstrations. We introduce a **component-level framework for concept recovery** that decomposes concepts into their defining components and uses controlled structural interventions to evaluate both component-level and whole-concept recovery under identifiable demonstrations. Across four instruction-tuned LLMs and three composition forms, we find that models often achieve high prediction accuracy despite poor whole-concept recovery, with systematic over-specification of relevant attributes and level constraints. Individual structural components are generally insufficient for whole-concept recovery, whereas jointly specifying complementary components, particularly relevant attributes and their level constraints, substantially improves recovery. However, this improvement is fragile: a single plausible but unnecessary level constraint can substantially disrupt recovery. Finally, joint supervision of recovery and implicit prediction substantially improves recovery while preserving prediction performance, with the learned capability transferring across composition forms. Overall, our component-level analysis provides a more diagnostic view of what models recover from demonstrations than the existing holistic view.

Primary area:
interpretability and explainable AI
Keywords:
LLM interpretability, In-context learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ny9XOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines whether accurate in-context prediction implies recovery of the underlying task concept. It introduces a component-level evaluation framework that represents concepts through relevant attributes, their allowed value levels, and, where appropriate, a composition threshold. The framework is instantiated in a controlled STAGGER-style environment with AND, OR, and m-of-K rules. Using version-space analysis, the authors assess whether the demonstrations uniquely identify their target concepts. Experiments with four instruction-tuned LLMs compare implicit prediction with explicit recovery, characterize component-level errors, and study interventions that supply correct or perturbed structural information. Finally, fine-tuning experiments with Qwen2.5-7B test whether structured recovery can be learned while preserving implicit prediction and transferring to unseen composition forms.

Critical strengths

Identifiability-aware evaluation. Appendix C explicitly analyzes whether the demonstration sets uniquely determine the target concept within the specified hypothesis class. This is consequential because otherwise poor recovery could simply reflect ambiguity in the evidence. The version-space enumeration and identification-threshold analysis make the central prediction–recovery comparison more interpretable.

Useful error diagnosis and evidence of learnability. Table 1 identifies a concrete failure pattern—substantial over-inclusion of supposedly relevant attributes—rather than treating all incorrect recovered rules alike. The paired prediction comparison and concept-application controls help interpret this gap. Tables 5 and 6 additionally show large improvements from supervised recovery training, while joint training retains strong implicit-prediction performance. These results make the diagnostic framework potentially useful beyond merely reporting a new metric.

Critical weaknesses

The full-information interventions in Table 3 leave nothing to infer from the demonstrations. Because the composition form in Equation (1) is already given, providing both the relevant attributes A and their allowed levels L completely specifies an AND or OR concept; adding the threshold m does the same for m -of- K . High recovery under R ( A , L ) and R ( A , L , m ) therefore shows that the model can format and copy a rule already stated in the prompt, not that it combines partial cues with the demonstrations to induce the concept. Running the same prompt without demonstrations would immediately test whether the examples play any role at all. Separately, Appendix D.1 states that the K intervention tells the model there are exactly K relevant attributes, whereas Appendix B.1 allows | A | < K for AND concepts; in those cases, the prompt gives the model an incorrect attribute count.

The decoy experiment in Section 3.2 and Table 4 is ambiguously defined and has conflicting baseline numbers. Adding an allowed level to each relevant attribute changes the ground-truth Boolean function rather than inserting a harmless distractor. When recovery drops, it is impossible to tell whether the model is distracted by irrelevant information or simply following the modified rule it was given. The text is also unclear on whether the corrupted level sets replace the true ones or appear alongside them. Finally, the unperturbed R ( A , L ) baseline numbers on OR concepts do not match between Table 3 and Table 4 (e.g., Qwen2.5-72B shifts from 0.94 to 0.88, and Llama-3.1-70B shifts from 0.88 to 1.00), which needs to be explained before drawing conclusions from the perturbation results.

The identifiability guarantee in Appendix C does not match the output space allowed by the recovery prompt. Appendix B.1 generates concepts from a restricted family of ordinal threshold predicates, and the version-space analysis in Appendix C proves uniqueness only within that restricted class. Yet the recovery prompt in Appendix D.1 allows the model to output arbitrary subsets of levels for each attribute without stating the ordinal-threshold constraint. A demonstration set that uniquely pins down a concept within the generator's hypothesis class can still be compatible with multiple non-ordinal level subsets in the model's output space. Either the prompt needs to explicitly state the predicate restriction, or identifiability needs to be checked over the full grammar the model is permitted to produce.

The fine-tuning transfer claims and statistical tests are overstated. In Tables 5 and 6, training and evaluation share the exact same attribute vocabulary, level space, and output format, varying only the composition operator. This shows transfer across operators within a single synthetic generator, not general concept-recovery ability. The fine-tuning experiments also test a single model (Qwen2.5-7B) with no variance reported across random seeds, and Appendix D.7 omits basic training details such as epochs/steps, effective batch size, and checkpoint selection. Finally, Table 2 runs McNemar's test at the individual query level even though multiple queries are evaluated on the same sampled concept and demonstration set, violating the independence assumption; variance and significance should be computed at the concept or episode level instead.

Questions for authors
  1. In Table 3, when A and L (and m for m-of-K) are given, which component of the target concept still has to be inferred from the demonstrations? Do you have an existing control that supplies the same information without demonstrations?
  2. How exactly is the K intervention implemented for AND concepts with |A| < K? Does the prompt communicate nominal predicate capacity or the number of effective relevant attributes?
  3. In Table 4, is one extra level introduced per relevant attribute or per concept, and are the correct level constraints still shown alongside the perturbed ones? What accounts for the differences in the unperturbed R(A,L) results between Tables 3 and 4?
  4. Does the identifiability guarantee in Appendix C hold for the arbitrary allowed-level subsets apparently permitted by the recovery prompt in Appendix D.1, or is the restricted ordinal-predicate family explicitly communicated to the model?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ny9X - Part I

Thank you very much for your valuable feedback! We have revised the manuscript and provide our detailed responses below.

  • W1-1 and Q1: The full-information interventions in Table 3 leave nothing to infer from the demonstrations.

    • We thank the reviewer for raising this concern and would like to clarify the misunderstanding here. Table 3 is intended to examine whether individual structural components are sufficient to recover the whole concept, and how recovery changes when multiple components are provided together.
      • Partial structural information is helpful for the concept recovery. In Table 3, we observe that providing only partial structural information can improve recovery over the demonstrations-only condition. For example, providing only the ground-truth relevant-attribute set AA improves recovery on AND concepts to 0.81 for Qwen and 0.84 for Llama, although the level constraints LL remain unspecified. Similarly, providing (A,L)(A,L) for mm-of-KK also improves recovery for some models, even though the threshold mm is not provided and must still be determined. These partial-information conditions show that providing individual or complementary structural components can help to recover the whole concept with demonstrations.
      • Progressive revelation of more relevant attributes can help the concept recovery. To further examine how the amount of provided information affects recovery, we include a fine-grained analysis in Appendix D.6. We progressively reveal subsets of ground-truth relevant attributes and average recovery over all possible subsets of each size. The results show that, for AND concepts, recovery generally improves as more relevant attributes are revealed. This is consistent with our findings in Table 3. We have revised the manuscript to include the full results and present the results for AND below.
      • Regarding the suggested no-demonstration control, we note that removing the demonstrations while retaining the complete concept specification would only test whether models can reproduce an explicitly provided concept. Such a control would not assess concept recovery or the role of demonstrations in inferring missing structural components. Our focus is instead on whether models can use demonstrations to recover unspecified components when only partial structural information is provided. We therefore focus our analysis on the partial-information conditions, where the demonstrations remain necessary for determining the missing components.

Table 1: Progressive revelation of structural information for concept recovery. RjR_j denotes the condition in which jj of the ∣A∣|A| ground-truth relevant attributes are revealed to the model, with results averaged over all (∣A∣j)\binom{|A|}{j} possible subsets. Qexp(Rj)Q_{\mathrm{exp}}(R_j) measures behavioral recovery on held-out queries under this condition. Thus, R0R_0 reveals none of the relevant attributes, whereas R∣A∣R_{|A|} reveals all of them. Implicit Prediction denotes direct query prediction from demonstrations without providing the relevant attributes. Levels for each attribute are not provided and blank entries correspond to undefined conditions.

ModelKKHHQexp⁡(R0)Q_{\exp}(R_0)Qexp⁡(R1)Q_{\exp}(R_1)Qexp⁡(R2)Q_{\exp}(R_2)Qexp⁡(R3)Q_{\exp}(R_3)QimpQ_{\text{imp}}
11000.760.970.95
12000.641.000.95
Qwen2.5-72B21000.610.750.880.90
22000.620.700.840.94
31000.550.690.760.720.82
32000.640.690.820.880.86
11000.561.000.95
12000.631.000.92
Llama-3.1-70B21000.500.780.890.86
22000.570.740.850.86
31000.540.700.810.830.76
32000.530.710.790.740.75
  • W1-2 and Q2: Clarification of the distinction between ∣A∣|A| and KK in Appendix D.1.
    • We thank the reviewer for catching this ambiguity. The distinction arises from our treatment of non-effective predicates in different concept forms. As explained in Appendix B.1, we exclude non-effective predicates for OR and mm-of-KK, but retain them for AND because vacuous predicates do not affect the conjunction. Consequently, ∣A∣=K|A|=K for OR and mm-of-KK, whereas ∣A∣≤K|A|\leq K for AND.
    • Importantly, the KK intervention in Appendix D.1 provides the model with the number of effective relevant attributes, ∣A∣|A|, rather than the nominal arity KK. The model is then asked to identify these relevant attributes from the candidate features. Thus, when ∣A∣<K|A|<K for AND concepts, the prompt communicates the smaller number of effective relevant attributes, and the model is not required to select all KK candidate attributes. We have revised Appendix D.1 to make this distinction explicit.
AuthorsOpenReview ↗

Response to Reviewer ny9X - Part II

  • W2-1: The decoy results may reflect following the modified specification rather than distraction by irrelevant information.

    • We clarify that the decoy intervention does not replace the ground-truth level constraints with modified constraints. Instead, it retains the ground-truth levels and adds one plausible but unnecessary level for each relevant attribute. The purpose is to examine whether models can distinguish the concept-defining levels from these additional candidates using the demonstrations.

      • The augmented specification retains useful ground-truth information. Comparing R(A,L~)R(A,\tilde L) in Table 4 with the attribute-only condition R(A)R(A) in Table 3, we find that the former yields higher recovery in multiple model--form conditions. Unlike R(A)R(A), the augmented specification provides the ground-truth level constraints together with one additional decoy level for each relevant attribute. Thus, despite the presence of decoys, the provided level information can still offer useful guidance for recovering the target concept.
      • As shown in Table 4, adding one decoy level per relevant attribute reduces whole-concept recovery. This indicates that models are sensitive to plausible but unnecessary structural information, even when the correct levels remain among the provided candidates.
      • Overall, we interpret these results as evidence that models can benefit from partial structural information but remain sensitive to plausible yet unnecessary additions. The experiment therefore probes the difficulty of distinguishing concept-defining information from irrelevant structure, rather than treating the augmented specification as a new ground-truth concept.
  • W2-2 and Q3: The decoy experimental setting.

    • In the level-perturbation condition (Table 4 in the paper), we add exactly one decoy level to each relevant attribute. Specifically, for each attribute jj, we construct L~j=Lj∪ℓ~j\tilde L_j=L_j\cup{\tilde{\ell}_j}, where ℓ~j∉Lj\tilde{\ell}_j\notin L_j is the first level outside the ground-truth set. Thus, the original ground-truth levels are retained rather than replaced. The model sees a single augmented candidate set L~j\tilde L_j for each attribute, without being told which level is the decoy. The prompt instructs the model to identify the true level constraints from the demonstrations, effectively requiring it to exclude the additional decoy level. The correct level constraints therefore remain present in the provided sets. The detailed design is presented in Sec. 3.3, and we have revised the Sec. 4.3 to avoid the ambiguity.
  • W2-3: Inconsistent values between Tables 3 and 4.

    • We thank the reviewer for identifying these inconsistencies. We rechecked the experimental logs and found errors in the reported OR results in Table 3. We have corrected these entries to match the corresponding experimental results. The corrected values are Qwen2.5-72B: 0.88 and Llama-3.1-70B: 1.00. These corrections do not change our findings or conclusions.
  • W3 and Q4: Identifiability in the output space allowed by the recovery prompt.

    • We thank the reviewer for the thoughtful question. The ordinal order of each predicate used to generate concepts is not sepcified in the recovery prompt in Appendix D.1. The prompt only specifies the three available levels, and our parser and scorer allow arbitrary level subsets.
      • We extend the identifiability analysis in Appendix C to the full output space allowed by the prompt. For each attribute, we consider either omitting it or assigning it one of the six non-empty proper subsets of {LOW,MEDIUM,HIGH}. The full set is excluded for OR and mm-of-KK concepts because it makes an OR concept always positive or changes the threshold semantics of an mm-of-KK concept. For AND concepts, it is behaviorally equivalent to omitting the attribute. We perform this analysis on the same 48 concepts and demonstration streams used in our experiments.
      • The results are identical to those obtained in the ordinal hypothesis class: 48/48 concepts are uniquely identified up to behavioral equivalence at (H=200), and 47/48 at (H=100), with the same single exception. In Appendix C.3, we check the demonstration streams used in our experiments, and the expanded analysis confirms that the target concepts remain identifiable even when considering the broader output space allowed by the recovery prompt. Therefore, the demonstrations provided to the LLMs in Appendix D.1 contain sufficient information to identify the target concepts, and recovery errors cannot be attributed to ambiguity in the demonstrations.
AuthorsOpenReview ↗

Response to Reviewer ny9X - Part III

  • W4-1: Performance gains on the fine-tuned model.
    • We conduct additional evaluations to examine generalization beyond the original representation. We evaluate on re-skinned mm-of-KK concepts by changing attribute names and value representations while keeping the concept grammar fixed. Joint training retains strong performance even when both types of changes are applied (Table 2 below). We further evaluate the same adapters on MONK-1/2/3 and UCI Mushroom, which involve different attributes, concept structures, and data distributions. Joint training improves implicit prediction on all four datasets and explicit concept application on three (Table 3 below). These results provide further evidence that the learned capabilities extend beyond the original synthetic setting. We have revised the manuscript and added these results in Appendix D.10.

Table 2: Evaluation of fine-tuned Qwen2.5 adapters under changes to attribute vocabulary and value representation. Orig: original attribute names and three ordinal levels; Names: renamed attributes; Values: four nominal values instead of ordinal levels; Both: renamed attributes and four nominal values. Trained-adapter results are mean ±\pm standard deviation across three runs.

TrainingQimpQ_{\mathrm{imp}} (Orig)QimpQ_{\mathrm{imp}} (Names)QimpQ_{\mathrm{imp}} (Values)QimpQ_{\mathrm{imp}} (Both)QexpQ_{\mathrm{exp}} (Orig)QexpQ_{\mathrm{exp}} (Names)QexpQ_{\mathrm{exp}} (Values)QexpQ_{\mathrm{exp}} (Both)
No training0.640.640.640.640.570.570.600.600.480.480.500.500.500.500.500.50
Recovery only0.50±0.000.50_{\pm 0.00}0.50±0.000.50_{\pm 0.00}0.51±0.010.51_{\pm 0.01}0.50±0.000.50_{\pm 0.00}0.99±0.010.99_{\pm 0.01}0.89±0.030.89_{\pm 0.03}0.71±0.060.71_{\pm 0.06}0.65±0.020.65_{\pm 0.02}
Joint0.96±0.010.96_{\pm 0.01}0.95±0.020.95_{\pm 0.02}0.85±0.000.85_{\pm 0.00}0.79±0.010.79_{\pm 0.01}0.99±0.010.99_{\pm 0.01}0.93±0.020.93_{\pm 0.02}0.77±0.040.77_{\pm 0.04}0.70±0.020.70_{\pm 0.02}

Table 3: Evaluation of fine-tuned Qwen2.5 adapters on MONK-1/2/3 and UCI Mushroom. Trained-adapter results are mean ±\pm standard deviation across five seeds.

DatasetTrainingQimpQ_{\mathrm{imp}}QexpQ_{\mathrm{exp}}
No training0.670.670.480.48
MONK-1Recovery only0.50±0.010.50_{\pm 0.01}0.60±0.030.60_{\pm 0.03}
Joint0.69±0.010.69_{\pm 0.01}0.61±0.040.61_{\pm 0.04}
No training0.540.540.490.49
MONK-2Recovery only0.54±0.030.54_{\pm 0.03}0.50±0.010.50_{\pm 0.01}
Joint0.69±0.050.69_{\pm 0.05}0.49±0.030.49_{\pm 0.03}
No training0.810.810.500.50
MONK-3Recovery only0.55±0.040.55_{\pm 0.04}0.76±0.040.76_{\pm 0.04}
Joint0.94±0.030.94_{\pm 0.03}0.72±0.060.72_{\pm 0.06}
No training0.710.710.430.43
MushroomRecovery only0.54±0.020.54_{\pm 0.02}0.59±0.030.59_{\pm 0.03}
Joint0.95±0.010.95_{\pm 0.01}0.72±0.090.72_{\pm 0.09}
  • W4-2: Training seeds and training details.

    • Multiple runs. We report the mean and standard deviation across three training runs. The two additional reruns use explicitly fixed seeds to control LoRA initialization, data order, and dropout. As shown in Tables 5 and 6, the results exhibit relatively small variation across runs.
    • Training details. We have revised Appendix D.7 to provide the missing training details, including the LoRA configuration, optimizer and learning-rate schedule, effective batch size, number of training epochs and optimizer steps, and checkpoint selection.
  • W4-3: Questions about McNemar's test.

    • We replace the original tests with a concept-level paired analysis.
      • For each of the 64 concepts, we compute the accuracy difference between implicit and explicit prediction. We then estimate 95% confidence intervals using a KK-stratified bootstrap and compute p-values using concept-level sign-flip permutation tests, with Holm correction across the four models.
      • As shown below, the prediction–recovery gap remains statistically significant across all four models (p<10−4)(p<10^{-4}), with mean differences ranging from +0.16 to +0.24. The detailed breakdown in Appendix D.4 further shows that the gap is positive for both K=3 and K=4, and occurs on the vast majority of individual concepts.

Table 4: Paired comparison of implicit and explicit predictions. Δ=Qimp−Qexp\Delta = Q_{\mathrm{imp}} - Q_{\mathrm{exp}} denotes the mean paired accuracy difference across 6464 concepts, with 95%95\% CIs estimated via KK-stratified bootstrap. The pooled row averages paired differences across models for each concept.

ModelQimpQ_{\mathrm{imp}}QexpQ_{\mathrm{exp}}Δ\Delta95% CI
Qwen2.5-72B0.840.61+0.23[0.20, 0.26]
Llama-3.1-70B0.730.57+0.16[0.12, 0.19]
Gemma-2-27B0.730.58+0.16[0.13, 0.19]
Mistral-24B0.770.53+0.24[0.21, 0.27]
Pooled (64 concepts)----+0.19[0.17, 0.22]
Reviewer_7ux9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors analyze whether LLMs that achieve high accuracy on in-context learning also recover the concept behind the demonstrations. Target concepts are generated in a synthetic environment, where a version-space formulation (all possible rules consistent with the labeled examples) verifies that the demonstrations uniquely identify each concept. Implicit prediction accuracy beats the accuracy of explicitly recovered rules across four instruction-tuned models.

Component-level analysis demonstrates that the difference in accuracy is connected to over-specification. Concretely, models list almost all offered attributes as relevant, while level-constraint and threshold errors allow too many levels per attribute and require too many conditions. Individual components are mostly not enough for a successful recovery, whereas providing the relevant attributes together with their level constraints enables near-complete recovery. The authors show that recovery from demonstrations is learnable (demonstrated with LoRA fine-tuning of Qwen2.5-7B).

Critical strengths
  • The paper provides an exhaustive identifiability control. Every rule in the hypothesis class is enumerated and checked against demonstrations. Target concepts are uniquely identified in 47/48 streams at H=100 and 48/48 at H=200. This is an improvement over prior rule-induction work, where recovery failures could be attributed to ambiguous evidence. Additionally, most probes become identifiable before the full concept does, so this offers a potential explanation for the gap between the prediction and recovery.

  • The diagnostic setting is clean and well specified. Every concept component has an explicit ground truth, which enables the component-level error analysis (e.g., whether level constraints are too broad or thresholds are too high; Tables 8 and 9).

Critical weaknesses
  • The joint-specification experiments mostly reflect copying the provided information. For AND and OR concepts, (A, L) is the complete target concept, and (A, L, m) as well for m-of-K, and the prompt shows it in the exact output format. The almost perfect R(A, L) and R(A, L, m) values in Table 3 show that models can simply copy the answer, and it's not clear how much the joint specification really enables recovery. Similarly, the decoy experiments show a drop in recovery, which could simply mean that the model trusts the provided information over the demonstrations.

  • The gap between prediction and recovery may partly come from the measurement itself. If the model applies the recovered rules on its own to compute QexpQ_\text{exp}, then the application error could explain part of the discrepancy, since the model can make a mistake even when applying the correct rule. Over-inclusion could also be inflated, since the attributes listed with all three levels are counted, which don't have an effect on AND rules but make OR rules always positive.

Questions for authors
  1. Can concepts be recovered when the provided information is not complete, or when it has to be checked against the demonstrations?
  2. Is QexpQ_\text{exp} computed by the LLM or with a deterministic executor?
7 comments
AuthorsOpenReview ↗

Response to Reviewer 7ux9 - Part I

Thanks a lot for your valuable comments! We have revised the manuscript and provide point-by-point responses below.

  • W1-1: The joint-specification experiments mostly reflect copying the provided information.

    • We thank the reviewer for raising this concern and would like to clarify the misunderstanding here. Table 3 examines whether providing individual structural components can help recover the remaining components, and how recovery changes when multiple components are provided together.
      • Partial structural information is helpful for the concept recovery. In Table 3, we observe that providing only partial structural information can improve recovery over the demonstrations-only condition. For example, providing only the ground-truth relevant-attribute set AA improves recovery on AND concepts to 0.81 for Qwen and 0.84 for Llama, although the level constraints LL remain unspecified. Similarly, providing (A,L)(A,L) for mm-of-KK also improves recovery for some models, even though the threshold mm is not provided and must still be determined. These partial-information conditions show that providing individual or complementary structural components can help to recover the whole concept with demonstrations.
      • Progressive revelation of more relevant attributes can help the concept recovery. To further examine how the amount of provided information affects recovery, we include a fine-grained analysis in Appendix D.6. We progressively reveal subsets of ground-truth relevant attributes and average recovery over all possible subsets of each size. The results show that, for AND concepts, recovery generally improves as more relevant attributes are revealed. This is consistent with our findings in Table 3. We have revised the manuscript to include the full results and present the results for AND below.

Table 1: Progressive revelation of structural information for concept recovery. RjR_j denotes the condition in which jj of the ∣A∣|A| ground-truth relevant attributes are revealed to the model, with results averaged over all (∣A∣j)\binom{|A|}{j} possible subsets. Qexp(Rj)Q_{\mathrm{exp}}(R_j) measures behavioral recovery on held-out queries under this condition. Thus, R0R_0 reveals none of the relevant attributes, whereas R∣A∣R_{|A|} reveals all of them. Implicit Prediction denotes direct query prediction from demonstrations without providing the relevant attributes. Levels for each attribute are not revealed and blank entries correspond to undefined conditions.

ModelKKHHQexp⁡(R0)Q_{\exp}(R_0)Qexp⁡(R1)Q_{\exp}(R_1)Qexp⁡(R2)Q_{\exp}(R_2)Qexp⁡(R3)Q_{\exp}(R_3)QimpQ_{\text{imp}}
11000.760.970.95
12000.641.000.95
Qwen2.5-72B21000.610.750.880.90
22000.620.700.840.94
31000.550.690.760.720.82
32000.640.690.820.880.86
11000.561.000.95
12000.631.000.92
Llama-3.1-70B21000.500.780.890.86
22000.570.740.850.86
31000.540.700.810.830.76
32000.530.710.790.740.75
  • W1-2: The decoy results may reflect that the model trusts the provided information over the demonstrations.

    • The recovery drop in Table 4 shows that models are sensitive to plausible but unnecessary structural information. The results do not suggest that models simply trust the provided specification without using the demonstrations.
      • As shown in Table 4, adding one decoy level per relevant attribute reduces whole-concept recovery. This indicates that models are sensitive to plausible but unnecessary structural information, even when the correct levels remain among the provided candidates.
      • The decoy specification retains useful ground-truth information. Comparing R(A,L~)R(A,\tilde L) in Table 4 with the attribute-only condition R(A)R(A) in Table 3, we find that the former yields higher recovery in multiple model--form conditions. Unlike R(A)R(A), the decoy specification provides the ground-truth level constraints together with one additional decoy level for each relevant attribute. Thus, despite the presence of decoys, the provided level information can still offer useful guidance for recovering the target concept.
      • Overall, our results indicate that models' recovery is sensitive to plausible but unnecessary structural information, while the provided specification can still offer useful guidance. We therefore interpret the decoy results as evidence of limited robustness in distinguishing concept-defining information from plausible but unnecessary structure, rather than as evidence that models trusts the provided information over the demonstrations.
AuthorsOpenReview ↗

Response to Reviewer 7ux9 - Part II

  • W2-1: The prediction–recovery gap may reflect errors in applying concepts.

    • We agree that QexpQ_{\mathrm{exp}} alone cannot distinguish errors in recovering a concept from errors in applying the recovered concept. We complement this behavioral measure with component-level analyses, whole-concept recovery metrics, and a direct-application control.
      • Evaluating and localizing concept recovery errors. Two types of errors can contribute to low QexpQ_{\mathrm{exp}}: (1) recovering an incorrect concept and (2) incorrectly applying a correctly recovered concept. For the first error, we compare recovered concepts against the ground-truth structural components using the Accuracy. As shown in Table 1, models exhibit systematic over-specification, tending to include additional attributes, satisfying levels, or required predicates. We further use R(⋅)R(\cdot) to evaluate whole-concept recovery: a recovered concept is counted as correct only if it agrees behaviorally with the ground-truth concept on all held-out queries. Therefore, R(⋅)R(\cdot) is a stircter metric than QexpQ_{\mathrm{exp}}. These two analyses provide evidence that concept recovery errors contribute to the prediction–recovery gap.
      • Evaluating concept application errors. For the second type of error, we conduct an additional experiment in Appendix D.5, where we directly provide the target concept to the models without requiring them to recover it from demonstrations. As shown in Table 11, models generally achieve high held-out query accuracy across different concept representations. These results suggest that the prediction–recovery gap is unliekly caused by difficulty in applying the correct recoverd concepts.
      • Overall, these evaluations support our finding that recovering a concept is more challenging than applying a correct concept.
  • W2-2: Attributes with three levels can cause unfair evaluation.

    • We would like to clarify that attributes with all three levels included are treated differently: they are excluded for OR and mm-of-KK, but retained for AND concepts. This design follows directly from the semantics of the respective logical form (Appendix B.1):
      • OR and mm-of-KK. An attribute whose three levels are all included corresponds to an always satisfied predicate. For OR, this makes the entire concept always positive, regardless of the other attributes. For mm-of-KK, it contributes an unconditional satisfied predicate to the threshold condition. Such attributes are therefore excluded as non-effective attributes.
      • AND. In contrast, an always-satisfied predicate does not influence the entire concept like OR and mm-of-KK. We therefore allow such attributes under AND.
    • We note that having fewer effective levels does not make OR concepts trivially easier to recover. In Table 1, we measures level recovery accuracy conditioned on correctly identfied relevant attributes. We can see that the accuracy is particularly low for OR. Therefore, even when attribute identification is controlled, models still struggle to recover the level constraints of OR concepts.
  • Q1: Can concepts be recovered when the provided information is not complete?

    • Our experiments show that if provided sufficient demonstrations, partial structural information can help LLMs to recover some concepts.

      • Demonstrations provide sufficient information for concept inference. As described in Appendix D.1, our prompt explicitly instructs LLMs to use the observed demonstrations to infer the underlying concept. Moreover, our theoretical analysis in Appendix B establishes that the demonstrations in our experimental setting are sufficient to uniquely identify the target concepts, the issue existing works usually ignored.
      • Models can recover unspecified components from partial structural information. As shown in Table 3, providing partial concept information can help the recovery of unspecified components. For example, when only the relevant-attribute set AA is provided for AND concepts, models still need to infer the corresponding level constraints LL from demonstrations. Similarly, for mm-of-KK concepts, providing (A,L)(A,L) still leaves the threshold mm to be inferred. These results demonstrate that models can recover missing component information beyond what is explicitly provided.
  • Q2: QexpQ_{\text{exp}} computed by LLMs or a deterministic executor?

    • We are sorry for the confusion here. QexpQ_{\mathrm{exp}} is computed using a deterministic executor, not by asking the LLM to directly predict the labels. The procedure is as follows:
      • We prompt the LLM to recover an explicit concept from the demonstrations.
      • We apply the recovered concept to the held-out queries using a deterministic executor.
      • We compare the predicted labels against the ground-truth labels to compute QexpQ_{\mathrm{exp}}.

Thank you for your response. The clarification that a deterministic executor applies rules resolves my concern about where the errors come from. However, I still have some major concerns about the paper. The results show recovery that goes beyond copying only for AND. For OR and m-of-K, I'm not convinced that joint specification really enables recovery. The demonstrations rule out the decoy levels, so I still believe a plausible explanation is that the models rely on the provided information instead of the demonstrations. It is unclear how many failed recoveries contain the decoy level or copy the specification. Additionally, you explained how ground-truth concepts manage attributes with all three levels, but I'm confused about how such attributes are scored in the models' outputs. If I'm understanding it correctly, they don't change anything under AND, which would artificially inflate the over-inclusion results. And under OR, these types of attributes make the rule label everything positive, which could explain why explicit-rule accuracy hovers around 0.5.

Reviewer_WPJHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the gap between successful in-context prediction and explicit recovery of the underlying task concept. The authors propose a component-level evaluation framework that decomposes concepts into relevant attributes, level constraints, and composition thresholds, while explicitly verifying that the demonstrations uniquely identify the target concept. Experiments across four LLMs show that models can achieve reasonably high prediction accuracy while recovering poor explicit concepts, with a systematic tendency toward over-specification. The paper further studies structural-information interventions and supervised training for concept recovery.

Critical strengths
  1. The paper identifies a clear limitation of holistic rule-recovery evaluation: it cannot localize which concept components are recovered incorrectly.
  2. Explicitly controlling concept identifiability is an important strength, since it separates model recovery errors from ambiguity in the demonstrations.
  3. The prediction–recovery gap is clear and consistent across models, and the component-level analysis reveals a strong and interpretable over-specification pattern.
  4. The experimental study is generally thorough, including controlled interventions, additional datasets, inference-time baselines, and supervised recovery experiments.
Critical weaknesses
  1. The strongest structural-intervention result is partly tautological. For AND/OR concepts, the target concept is defined by ((A,L)), while the (R(A,L)) condition directly provides the ground-truth attributes and their level constraints. Similarly, ((A,L,m)) essentially specifies the complete m-of-K concept. Therefore, the near-perfect recovery in these conditions mainly demonstrates that the model can reproduce a provided specification, rather than recover the concept from demonstrations.
  2. The paper operationalizes concept recovery through explicit structured output. Consequently, the observed prediction–recovery gap does not necessarily imply that the model lacks an internal representation of the concept; it may partly reflect difficulty in explicitly articulating that representation in the required format. The supervised recovery results may similarly improve the reporting interface in addition to concept inference itself.
Questions for authors

see weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer WPJH - Part I

Thank you so much for your comments! We have revised the manuscript and provide our responses below.

  • W1: Tautological concept recovery in Table 3.

    • We thank the reviewer for raising this concern and would like to clarify the misunderstanding here. Table 3 is intended to examine whether individual structural components are sufficient to recover the whole concept, and how recovery changes when multiple components are provided together.
      • Partial structural information is helpful for the concept recovery. In Table 3, we observe that providing only partial structural information can improve recovery over the demonstrations-only condition. For example, providing only the ground-truth relevant-attribute set AA improves recovery on AND concepts to 0.81 for Qwen and 0.84 for Llama, although the level constraints LL remain unspecified. Similarly, providing (A,L)(A,L) for mm-of-KK also improves recovery for some models, even though the threshold mm is not provided. These partial-information conditions show that providing individual or complementary structural components can help to recover the whole concept with demonstrations.
      • Progressive revelation of more relevant attributes can help the concept recovery. To further examine how the amount of provided information affects recovery, we include a fine-grained analysis in Appendix D.6. We progressively reveal subsets of ground-truth relevant attributes and average recovery over all possible subsets of each size. The results show that, for AND concepts, recovery generally improves as more relevant attributes are revealed. This is consistent with our findings in Table 3. We have revised the manuscript and present the results for AND below.

Table 1: Progressive revelation of structural information for concept recovery. RjR_j denotes the condition in which jj of the ∣A∣|A| ground-truth relevant attributes are revealed to the model, with results averaged over all (∣A∣j)\binom{|A|}{j} possible subsets. Qexp(Rj)Q_{\mathrm{exp}}(R_j) measures behavioral recovery on held-out queries under this condition. Thus, R0R_0 reveals none of the relevant attributes, whereas R∣A∣R_{|A|} reveals all of them. Implicit Prediction denotes direct query prediction from demonstrations. Levels for each attribute are not revealed and blank entries correspond to undefined conditions.

ModelKKHHQexp⁡(R0)Q_{\exp}(R_0)Qexp⁡(R1)Q_{\exp}(R_1)Qexp⁡(R2)Q_{\exp}(R_2)Qexp⁡(R3)Q_{\exp}(R_3)QimpQ_{\text{imp}}
11000.760.970.95
12000.641.000.95
Qwen2.5-72B21000.610.750.880.90
22000.620.700.840.94
31000.550.690.760.720.82
32000.640.690.820.880.86
11000.561.000.95
12000.631.000.92
Llama-3.1-70B21000.500.780.890.86
22000.570.740.850.86
31000.540.700.810.830.76
32000.530.710.790.740.75
  • W2-1: The prediction–recovery gap may reflect LLMs' difficulty in articulating the underlying concept in the specific format.

    • We agree that the prediction–recovery gap may partly reflect the difficulty of articulating a concept in a specific format, even when a model can successfully apply that concept. Indeed, this motivates our work:
      • As illustrated in Fig. 1, prior work takes a holistic view of concept recovery, evaluating whether a model can produce the complete concept. Such an evaluation does not distinguish errors in identifying the concept from difficulties in generating its explicit format. To isolate the format issue, our component-level analysis reduces the burden of open-ended rule generation: we provide the candidate attributes and their available levels, as well as the compositional form, and ask the model to identify the relevant attributes and levels by probing the demonstrations. In this constrained setting, the model does not need to generate the candidate attributes, invent their values, or infer the compositional form. It just needs to determine which provided components are actually part of the target concept.
      • Importantly, the resulting errors are not arbitrary failures to produce the required format. As shown in Tables 1, 8, and 9, models exhibit systematic over-specification: they tend to include additional attributes, satisfying levels, or required predicates that are not part of the target concept. This component-level analysis therefore allows us to move beyond the holistic recovery view and localize specific behavioral errors in recovering a concept.
      • Therefore, our goal is not to infer whether a model possesses an internal representation of the concept, but to provide a more fine-grained behavioral analysis of what models can and cannot recover from demonstrations.
AuthorsOpenReview ↗

Response to Reviewer WPJH - Part II

  • W2-2: Question about performance gains on the fine-tuned models.

    • We have incorporated explicit controls against memorization into our evaluation and conducted additional experiments to examine the generalizability of the learned capabilities. The results demonstrate that the gains from fine-tuning extend beyond memorizing training instances or reproducing the original structured output format.

      • First, our main evaluation controls for memorization and demonstrates improvements in both concept recovery and application. As described in Appendix D.9, the training and evaluation sets in Table 5 use disjoint relevant-attribute combinations, ensuring that evaluation concepts contain combinations not encountered during training. Moreover, R(∅)R(\varnothing) requires the recovered concept to agree with the ground-truth concept on all held-out queries. Under this strict criterion, joint training achieves high whole-concept recovery, while also substantially improving implicit prediction. Since implicit prediction does not require explicit structured output, these gains show that joint training improves the model's ability, rather than merely to articulate the output in the required format.
      • Second, the learned capabilities generalize to composition forms not observed during training. We train the adapters exclusively on mm-of-KK concepts and evaluate them on AND and OR concepts without additional fine-tuning. As shown in Table 6, joint training achieves high whole-concept recovery and implicit prediction. These results demonstrate that the learned recovery and application capabilities transfer across different compositional structures.
      • Third, we conduct additional evaluations to test generalization across representations and tasks. On re-skinned mm-of-KK concepts, we change attribute names and value representations while keeping the concept grammar and output format fixed. Despite these changes, the jointly trained model consistently outperforms the baselines, even when both types of changes are applied (Table 2 below). We further evaluate the same adapters on MONK-1/2/3 and UCI Mushroom, which involve different attributes, concept structures, and data distributions. Joint training improves implicit prediction on all four tasks and explicit concept application on three of them (Table 3 below).
      • Taken together, these results show that the gains from fine-tuning extend beyond just memorizing training instances or reproducing the original output format. We have added the additional evaluation results in Appendix D.10 and present them below.

Table 2: Evaluation of fine-tuned Qwen2.5-7B adapters under changes to attribute vocabulary and value representation. Orig: original attribute names and three ordinal levels; Names: renamed attributes; Values: four nominal values instead of three ordinal levels; Both: renamed attributes and four nominal values. Concept grammar and output format are unchanged. QimpQ_{\mathrm{imp}} denotes direct prediction accuracy, and QexpQ_{\mathrm{exp}} denotes accuracy obtained by executing the recovered concept on the same held-out probes. Results are mean ±\pm standard deviation across three runs.

TrainingQimpQ_{\mathrm{imp}} (Orig)QimpQ_{\mathrm{imp}} (Names)QimpQ_{\mathrm{imp}} (Values)QimpQ_{\mathrm{imp}} (Both)QexpQ_{\mathrm{exp}} (Orig)QexpQ_{\mathrm{exp}} (Names)QexpQ_{\mathrm{exp}} (Values)QexpQ_{\mathrm{exp}} (Both)
No training0.640.640.640.640.570.570.600.600.480.480.500.500.500.500.500.50
Recovery only0.50±0.000.50_{\pm 0.00}0.50±0.000.50_{\pm 0.00}0.51±0.010.51_{\pm 0.01}0.50±0.000.50_{\pm 0.00}0.99±0.010.99_{\pm 0.01}0.89±0.030.89_{\pm 0.03}0.71±0.060.71_{\pm 0.06}0.65±0.020.65_{\pm 0.02}
Joint0.96±0.010.96_{\pm 0.01}0.95±0.020.95_{\pm 0.02}0.85±0.000.85_{\pm 0.00}0.79±0.010.79_{\pm 0.01}0.99±0.010.99_{\pm 0.01}0.93±0.020.93_{\pm 0.02}0.77±0.040.77_{\pm 0.04}0.70±0.020.70_{\pm 0.02}

Table 3: Evaluation of fine-tuned Qwen2.5-7B adapters on MONK-1/2/3 and UCI Mushroom. Results are mean ±\pm standard deviation across five seeds.

DatasetTrainingQimpQ_{\mathrm{imp}}QexpQ_{\mathrm{exp}}
No training0.670.670.480.48
MONK-1Recovery only0.50±0.010.50_{\pm 0.01}0.60±0.030.60_{\pm 0.03}
Joint0.69±0.010.69_{\pm 0.01}0.61±0.040.61_{\pm 0.04}
No training0.540.540.490.49
MONK-2Recovery only0.54±0.030.54_{\pm 0.03}0.50±0.010.50_{\pm 0.01}
Joint0.69±0.050.69_{\pm 0.05}0.49±0.030.49_{\pm 0.03}
No training0.810.810.500.50
MONK-3Recovery only0.55±0.040.55_{\pm 0.04}0.76±0.040.76_{\pm 0.04}
Joint0.94±0.030.94_{\pm 0.03}0.72±0.060.72_{\pm 0.06}
No training0.710.710.430.43
MushroomRecovery only0.54±0.020.54_{\pm 0.02}0.59±0.030.59_{\pm 0.03}
Joint0.95±0.010.95_{\pm 0.01}0.72±0.090.72_{\pm 0.09}

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.