acceptodds
Withdrawn from ICLR 2027

Structure or Agency? Prior-Guided Agency Allocation for Multimodal Document Mining

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

As large language model (LLM) agents gain stronger planning and tool-use capabilities, a common trend is to grant them greater autonomy. Yet many real-world tasks already benefit from reliable procedural knowledge, making some model-controlled decisions unnecessary and error-prone. This raises a central question: which decisions should be controlled by the model, and which should be determined by task structure? To answer this question, we characterize the structure–agency trade-off through Agency Complexity, which measures the decision space exposed to model control, and two competing risks: Decision Error from incorrect model choices and Structural Bias from constraints that exclude successful trajectories. Guided by this analysis, we propose Prior-Guided Agency Allocation (PGAA), a principled framework for allocating control between reliable procedural priors and model judgment, formalized through a Human-Prior Decision Graph. To evaluate PGAA, we construct MineDoc, a challenging and representative benchmark for Web multimodal document mining, combining industrial procedural knowledge with instance-dependent judgments in retrieval, captioning, and LLM-as-a-Judge verification. It includes internal real-world business data and two public-data test sets simulating business scenarios, derived from Encyclopedic-VQA data associated with iNaturalist 2021 and Google Landmarks Dataset v2, respectively. We instantiate PGAA through a Filter–Produce–Reflect-ReAct mechanism and evaluate it on MineDoc. Under matched models, tools, inputs, and inference budget limits, PGAA improves reliability over fixed workflows and autonomous baselines while reducing inference cost relative to autonomous execution. Analyses of agency allocation and prior quality further support the trade-off, suggesting that effective agent design allocates control where model judgment is most valuable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agency allocation, structured agents, multimodal document mining

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_o7s9OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

There is a trade-off betweenthe Decision Error and the Structural Bias when the agents perform the specific tasks. This paper proposes Prior-Guided Agency Allocation (PGAA) a principled framework for allocating control between reliable procedural priors and model judgment. MineDoc was constructed to evaluate the effectiveness of PGAA. Results show that PGAA improves reliability over fixed workflows and autonomous baselines while reducing inference cost.

Critical strengths

The basic idea is interesting. There is a trade-off between the Decision Error and the Structural Bias when the agents perform the specific tasks. This paper tried to find a better way to balance this trade-off. This is a meaningful topic.

Critical weaknesses
  1. The experiments are weak. The authors construct MineDoc to evaluate their methods, which is also the key contribution of this paper. However, the construction pipeline of this dataset is unclear. There is limited description of this dataset in the paper. Only one backbone (Qwen 3.5) is used for the experiments. And all the baselines are built by the authors. The specific settings are unclear.

Except for the experiment settings, the results are also weak. The proposed method doesn't show substantial improvements compared with the simple baselines. The results of ReAct in Table 2 (GLDv2, 89.9%) can be easily higher than the proposed method in Table 1 (83.0%).

  1. The method is over-simplified on assumptions. The math derivation parts make some simple ideas complicated and constrained. The theory is not really used in the method. The FPR structure is decided by the authors' experience rather than the proposed rules.

  2. The claim in abstract "Analyses of agency allocation and prior quality further support the trade-off, suggesting that effective agent design allocates control where model judgment is most valuable." is unsupported by any experiments and results.

Questions for authors
  1. Can this method be applied to other tasks (such as coding)?
  2. How is MineDoc constructed? Specific construction workflow or dataset examples could be helpful.
Reviewer_Cg3QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the balance between task structure and model autonomy in LLM agents. It uses “autonomy complexity,” “decision errors,” and “structural bias” to describe how control is allocated and proposes PGAA. The authors build MineDoc, a benchmark for web document mining across different formats, and implement the framework through a ReAct mechanism with filtering, generation, and reflection. The reported results show that PGAA achieves higher reliability than fixed workflows and autonomous baselines when models, tools, inputs, and inference budget limits are matched. It also reduces inference costs compared with autonomous execution.

Critical strengths
  1. The paper treats autonomy allocation as an agent design problem and provides a clear view of the balance between structure and autonomy.
  2. PGAA has a systematic design. It combines procedural knowledge and model judgment through decision graphs based on human prior knowledge.
  3. MineDoc covers real industrial settings and public data, which gives it practical value for evaluation.
  4. The experiments show that the method improves both reliability and inference efficiency. Further analysis also supports the main ideas.
Critical weaknesses
  1. The results for the same baseline are inconsistent across tables. For ReAct + Prior on Landmarks, Table 1 reports a Pass@Query of 66.2%, while Table 4 in the appendix reports a Query Accuracy of 62.8%. If these metrics are equivalent, the results conflict. If they are different, the paper needs to explain the difference in their definitions.
  2. The paper introduces a new dataset but does not clearly report its basic size. Section 4.1 does not clearly state the number of examples in each test set or the overall data scale. It also does not provide enough detail about the annotation process. Readers cannot judge how many examples support these gains or whether the gains are stable. This also makes the results difficult to reproduce.
  3. The experiments are too limited to support the claim of a general framework. All experiments use the same backbone model, Qwen3.5-122B-A10B-FP8, and focus on a single type of document mining task. They show that the system works in the current setting, but they do not show whether the same allocation principle holds across different models and tasks.
  4. The evaluation mainly relies on MineDoc, which the authors build themselves. It remains unclear whether the framework also works on widely used general benchmarks. The paper needs experiments on such benchmarks to show that the method and conclusions apply more broadly.
Questions for authors

See above.

Reviewer_GwqoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies agency allocation for LLM agents: which execution decisions should be fixed by task structure and which should be left to the model. To this end, the author formalizes the execution as Human-Prior Decision Graph whose nodes are either deterministic Operator Nodes or model-controlled Decision Nodes, and proposes Agency Complexity measures the decision space exposed to the model. The framework is instantiated as FPR–ReAct on MineDoc, a new multimodal web document-mining benchmark (one private set plus two public sets). Experiments show that FPR–ReAct achieves the best query-level reliability at lower token and call cost than ReAct.

Critical strengths
  1. This paper investigates an interesting question about the application boundary of autonomous agents: which parts of a task should be executed through a predetermined workflow, and which parts can be delegated to the agents themselves. This question is important yet remains underexplored in prior work.
  2. This paper presents a clear conceptual framework: Operator/Decision Nodes, Agency Complexity, Decision Error, and Structural Bias are all defined in a simple and accessible manner.
  3. The experimental results in Tables 1 and 2 demonstrate that autonomous agents equipped with a fixed workflow achieve better performance, which supports the paper's central claim.
Critical weaknesses
  1. Weak generalization. My main concern is generalization ability and application scope of proposed method. The experiments demonstrate that FPR–ReAct achieves better performance by combining fixed workflow and adaptive model decision. However, realistic industry scenarios have serveral workflows, it is difficult to identify which part can not be replaced by LLMs. Therefore, we suggest the author gives some experiments with different tasks and models to show the generalization ability of proposed method.
  2. MineDoc reproducibility is weak. MineDoc-Private is explicitly unreleasable; the two public sets are described only as "derived from Encyclopedic-VQA associated with iNaturalist-21 / GLDv2," with query construction, validity annotation, and candidate-pool sourcing undocumented; no code release is mentioned.
  3. Counter-results in Table 2. In Table 2, Fix-first-5 outperforms FPR–ReAct on Landmarks (89.9% vs. 83.0%), and Fix-3 outperforms it on iNaturalist (77.1% vs. 76.2%). These results somewhat undermine the claimed effectiveness of FPR–ReAct, and the authors should provide an explanation for them.
Questions for authors

See my questions in weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.