acceptodds
Withdrawn from ICLR 2027

Information Organization and Evidence Efficiency in Policy Induction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

How does information organization affect the amount of evidence foundation models need to induce policies from chronological state-action observations? We investigate whether organization inspired by interfaces designed to support human judgment improves this evidence efficiency. Using public market records and study-defined hidden policies, we construct chronological state-action observations in two text representations: flat tables (FLAT) and panels inspired by professional trading interfaces (PANELS). Models infer four policy settings from supplied candidates within a common rule structure. At each evidence budget, both representations present the same observations in the same chronological sequence. We introduce Evidence-to-Criterion (E2C), the first tested evidence budget at which an induced policy meets a prespecified action-reproduction criterion shared across representations on separate evaluation states. We summarize the cumulative criterion-attainment rate across evidence budgets using the area under the curve (AUC). On this task, FLAT achieves significantly higher E2C AUC than PANELS for both evaluated foundation models. The findings extend input-representation comparisons to evidence efficiency in policy induction and support evaluating information organization alongside content when designing inputs for foundation models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
foundation models, information organization, input representations, policy induction, evidence efficiency, model evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_V9LuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates whether organization of information inspired by interfaces designed to support human judgment (e.g., professional trading interfaces) could also help LLMs in doing similar tasks that require it as represented by flat tables (FLAT) and panels inspired by professional trading interfaces (PANELS). For the only two LLMs the paper evaluates in its results without any strong baselines to compare with, using FLAT seems to be better scoring in cumulative attainment/E2C than PANELS and consistently across evidence budget.

Critical strengths

I do like (and see the practical value) in the particular research question that authors have aimed for, which is investigating how informative interfaces that help support professional human work (e.g. professional trading interfaces as the authors cite) could also help LLMs perform certain tasks. This alone is a very interesting RQ. However, the execution of the experiments and the level of technical rigor is not enough for ICLR. I do hope the authors could still continue this line of work and put more effort in improving the rigor of the paper before sending out to peer review.

Critical weaknesses

The paper has no substantive and definitive discussion of the results wherein the main results section is just one page (page 8). This alone merits a clear rejection of the paper. Moreover, the authors should note that Appendices are optional for reviewers to look at and not something that the authors can redirect the reviewers to at all points of the paper. The Appendix should also not contain critical details that should have been in the main content of the paper.

The paper uses only two commercial models for its main results: GPT-5.6 Luna and Claude Sonnet 5 and there are also no baseline comparisons. These two are also grounds for clear rejection of the paper for lack of technical rigor and does not meet the empirical standards of ICLR. I highly and strongly recommend the authors to read previously accepted papers from ICLR/NeurIPS/ICML to get a level of how rigorous technical experiments are conducted and accepted in these venues. Simply running your methods on two models is simply not enough, much less selecting commercial models.

Aside from the two major flaws highlighted above, as I read through the paper, I found the overall writing of the paper to be quite poor and can be significantly improved in terms of readability. I acknowledge that the authors did mention they use AI for the paper and take responsibility for it, but I still highly encourage the authors to edit out any AI written text in the main paper given that they are very poorly written and are obvious to readers.

Questions for authors

N/A

Reviewer_6aUBOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how information organization affects the ability of LLMs to infer policies. Specifically, the authors use cryptocurrency trading policies as a test case and introduce a metric called E2C. They compare the policy induction performance of LLMs when the information is presented in two different formats: flat tables (FLAT) and panels inspired by professional trading interfaces (PANELS). The results show the importance of how input information is organized for LLMs.

Critical strengths
  1. The question of how input formats affect the capabilities of LLMs is important.
  2. The experiments are sufficient within the setting of cryptocurrency trading policies.
Critical weaknesses
  1. I am not clear about the application value of the policy induction ability studied in this paper, especially in the cryptocurrency trading setting. The paper also does not discuss this issue. One potentially useful application I can think of is in strategic or interactive settings, such as recommendation systems trying to infer users' behavior patterns.

  2. The paper only studies policy induction in the context of cryptocurrency trading, and the task is limited to inferring four policy parameters. (This also raises another question: Can real human behavior really be described by only four parameters?) However, the title does not reflect this limited scope. The authors should consider other settings with clear practical value, or revise the title to better reflect the actual content of the paper.

  3. The technical contribution of this paper is limited. Moreover, this cannot be balanced by strong practical value, as discussed in Point 1.

  4. The main conclusion of this paper is simply that input formats affect the policy induction ability of LLMs, which seems trivial. I would be more interested in understanding why FLAT is better suited for LLMs than PANELS. The authors should provide a deeper analysis of this question.

  5. The writing of this paper needs improvement. First, the paper contains many defensive statements (see experiment results in the appendix). Second, the paper does not have a Related Work section, and its contributions are not clearly presented or organized. As a result, I find it difficult to understand how this paper is positioned within the existing literature and what its main contributions are.

Questions for authors

pls see weaknesses

Reviewer_acNNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the way observations are laid out changes how many observations a foundation model needs to recover a hidden rule. The task is a small limit order policy with four hidden settings, buy and sell price levels and buy and sell cancel timeouts, five candidates each (Table 1), simulated on public Hyperliquid order book data for BTC, ETH and SOL. The model sees the same chronological observations either as a flat TSV table (FLAT) or as a text grid which looks like a trading screen (PANELS), gets the rule structure and the candidate values, and has to return the four settings. The metric, Evidence-to-Criterion, is the first budget out of 40, 78, 153 and 300 observations at which the induced policy beats a set of prespecified comparison policies on held out states, summarized as an AUC on the log budget axis. On 108 units (12 policies times 9 window sets) FLAT beats PANELS for GPT-5.6 Luna (mean difference −0.069) and Claude Sonnet 5 (−0.082), with cluster bootstrap CIs excluding zero. Supplementary comparisons show the text grid beats an image of the same grid, and FLAT is equivalent to a message-style format.

Critical strengths

The experimental control is done properly. Both representations get the same observations in the same order at every budget, the recovery criterion was fixed before the experiment, and the probe comparison policies flip exactly one setting, with all 2,376 policy-unit pairs checked (Appendix A.3). All 419 answers which attained the criterion had Macro-F1 of 1 on both evaluation sets (Section 4.1), so attainment actually means the four settings were recovered. The direction holds across two models, both bootstrap axes and both variance estimators (Table 8). So I trust the direction of the result.

Critical weaknesses

The comparison is between two bundles and the paper does not isolate what in PANELS hurts. PANELS is about twice the bytes of FLAT (Table 7), lists trades and fills newest first instead of oldest first (Section 2.3), and gives price rank by row position while FLAT has an explicit LEVEL column. The two hidden submission settings are exactly price ranks (Table 1). So in FLAT the model reads the rank off a labeled row and in PANELS it has to count rows in the DOM at submission time. Section 5 admits this is not separated. But the question the paper asks in the introduction, whether organization built for human work helps models, is the broad one, and what was tested is one grid against one table. I would like to see PANELS with a rank column in the DOM, FLAT without the LEVEL column, and a breakdown of which setting the failed PANELS answers got wrong. Without this the finding does not generalize beyond these two formats.

Questions for authors

For the units where PANELS failed at 40 or 78 and FLAT passed, which of the four settings were wrong? If it is mostly the two submission levels, then the result is about the LEVEL column versus counting DOM rows, not about panels.

1 comment

Thank you for this question. Among units where both formats were queried at the same budget and FLAT passed while PANELS failed, only the submission levels were incorrect in 7 of 11 cases for GPT-5.6 Luna and 5 of 8 cases for Claude Sonnet 5 at 40 snapshots. At 78 snapshots, all 16 Luna cases and 11 Sonnet cases had errors only in the submission levels.

We therefore tested PANELS with explicit rank labels. Adding these labels improved Luna's mean E2C AUC, bringing the mean difference within our equivalence margin. However, FLAT retained its advantage for Sonnet, including when compared with a FLAT rerun conducted closer in time. An additional experiment with Qwen also showed a remaining FLAT advantage after rank labels were added. These results suggest that explicit rank labeling can help, but that it does not consistently close the gap between PANELS and FLAT.

Reviewer_RxvYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how information organization affects the evidence efficiency of foundation models in policy induction. The authors construct a task using public market data and simulated order activity, where models must infer four hidden policy parameters from chronological observations. They compare two text representations: flat tables (FLAT) and trading-interface-inspired panels (PANELS), while keeping the underlying observations consistent. The authors introduce Evidence-to-Criterion (E2C), measuring the first tested observation budget at which an induced policy satisfies predefined action-reproduction thresholds. The main finding is that FLAT consistently outperforms PANELS, with mean E2C AUC differences. The paper presents an interesting empirical observation, but its generalizability and the mechanisms responsible for the performance differences remain insufficiently established.

Critical strengths
  1. The paper examines how information organization influences the amount of evidence needed for policy induction. Moving beyond final-task accuracy to consider evidence efficiency is a worthwhile direction, particularly for applications involving structured observations.
  2. The authors use paired evaluation units, identical underlying observations, predefined policy candidates, and common evaluation criteria across representations.
  3. The paper reports confidence intervals, accounts for dependence across policies and market windows, and investigates sensitivity to alternative variance estimators and bootstrap procedures.
  4. The policy definitions, observation construction, evaluation procedures, and statistical methods are described in detail.
Critical weaknesses
  1. The policy-induction task is highly constrained. The models are given the complete policy structure and candidate values and only need to identify four discrete parameters, corresponding to 625 possible combinations. Moreover, only 12 hidden policies are evaluated. This is closer to parameter identification than general policy induction, where the model must discover unknown rules, structure, or continuous decision boundaries. It remains unclear whether the findings extend to more complex policies or other decision-making tasks.
  2. The representation comparison does not isolate the underlying mechanisms. Although FLAT and PANELS contain the same logical information, they differ in several dimensions, including field labeling, record ordering, spatial organization, reading instructions, and token counts. For example, FLAT explicitly provides price ranks through a LEVEL column, whereas PANELS requires inferring ranks from row positions. These differences could individually influence the difficulty of parameter recovery. The main comparison establishes an effect between the implemented representations, but not which organizational factors cause it.
  3. E2C is evaluated at only four discrete budgets, and evaluation stops upon first attainment. Consequently, the metric captures first success rather than stable success across increasing evidence budgets. This distinction matters because model outputs can vary across repeated calls.
  4. The evidence budgets count chronological snapshots, although many snapshots contain repeated records rather than independent evidence. It is unclear how many distinct informative submission, cancellation, and fill events appear at each budget, or whether the available observations uniquely identify the hidden policy. A simple exhaustive-search baseline over the 625 candidate policies could establish an empirical reference for identifiability and help distinguish insufficient evidence from model failures in interpreting the representation. Additionally, all 419 first-attaining answers achieve perfect Macro-F1 on both evaluation sets, suggesting that direct parameter-recovery accuracy would be an informative complementary metric.
  5. Although the analysis contains 108 evaluation units, these reuse only 12 policies and nine market-window sets. The statistical appendix also reports elevated false-positive rates for some synthetic dependence structures under the selected bootstrap procedure. Furthermore, all experiments use a single market-based environment and a common rule structure, making the broader implications for foundation-model reasoning and professional interfaces uncertain.
Questions for authors
  1. Which specific representation differences explain the FLAT advantage? Could the authors conduct targeted ablations that independently vary explicit price-rank labeling, record ordering, panel layout, and field descriptions? In particular, would adding explicit LEVEL labels to PANELS reduce the performance gap?
  2. How sensitive are the results to token usage and input length? Table 7 indicates substantial differences in input tokens across representations, although the reported maxima are not paired comparisons. Could the authors report matched token-count distributions at each evidence budget and investigate whether the relative performance persists under comparable token budgets?
  3. How identifiable are the hidden policy parameters at each evidence budget? Could the authors report the number of informative order events, the number of candidate policies consistent with the observed history, and recovery accuracy for each of the four parameters?
  4. How robust is E2C to model stochasticity and budget selection? The repeated-run results show considerable variation in effect magnitude. Could the authors evaluate additional repeated runs and report success probabilities at each fixed budget, including budgets after first attainment? How much does the E2C AUC difference change under alternative budget grids?
  5. Does the observed advantage generalize beyond this restricted policy family? Have the authors considered experiments involving unknown policy structures, more parameters, continuous settings, or another decision-making domain?
  6. What additional insight does E2C provide beyond standard recovery-accuracy learning curves?
1 comment

Thank you for your thoughtful questions. We would like to share a few findings from our follow-up analyses.

Q1. Adding explicit rank labels to PANELS improved the mean E2C AUC for GPT-5.6 Luna, bringing the mean difference within our equivalence margin. For Claude Sonnet 5, however, we did not find clear evidence of improvement, and FLAT retained its advantage. Thus, rank labels helped Luna but did not consistently close the gap between PANELS and FLAT.

Q2. PANELS used more input tokens and UTF-8 text bytes than FLAT in every matched unit–budget pair. Across the four budgets, the median PANELS/FLAT token ratio ranged from 1.31 to 1.69 for Luna and from 1.14 to 1.41 for Sonnet, while the corresponding text-byte ratios ranged from 2.09 to 2.96. However, RAW was smaller than FLAT, and the mean E2C AUC differences between them met our supplementary equivalence criterion for both models. These results suggest that larger inputs do not necessarily lead to worse performance, although they do not isolate the causal contribution of input length to the PANELS–FLAT gap.

Q3. Checking all 625 candidate policies for consistency with the observed records identified a unique policy in 31 of 108 evaluation units at 40 snapshots, 85 at 78 snapshots, and all 108 at 153 and 300 snapshots. Models still made errors at these later budgets, showing that recovery failures persisted even when the observations uniquely identified the policy.

Q4. The repeat analysis used a subset of 24 units, so changes in individual responses could have a greater effect on the mean than in the full 108-unit analysis. Although the effect size varied, FLAT maintained its average advantage across all three runs for both models. In follow-up experiments, we repeated the FLAT–PANELS comparison on all 108 units and again found an advantage for FLAT, with confidence intervals excluding zero. Together, these findings suggest that FLAT's advantage is consistent in direction across runs, even though its magnitude varies.

Q5. We are considering expert-inspired displays in medical and industrial settings, as well as broader decision-making tasks. In preliminary tests, the models struggled with less constrained policy recovery tasks, making it difficult to meaningfully compare representations. We therefore provided the rule structure and candidate values so that we could study representation differences in a tractable setting. We see recovery of unknown policy structures as a direction for future work.

Q6. E2C measures how much evidence is needed to obtain a policy that meets the behavioral criterion. A fresh response to a longer input can fail even after an earlier response succeeded, but that later failure does not undo the earlier attainment. By stopping at first success, E2C captures when each representation first reaches the criterion without requiring further, longer-input calls. In contrast, fixed-budget accuracy curves measure the success rate of fresh responses at each budget.

Comments on the paper

1 comment

Thank you for your careful reading and thoughtful questions. Our follow-up analyses have shown that the manuscript would benefit from substantial restructuring, so we have decided to withdraw the submission. Your feedback has helped us identify areas that need further development. We hope to share a more complete version of this work in the future.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.