Structure or Agency? Prior-Guided Agency Allocation for Multimodal Document Mining

As large language model (LLM) agents gain stronger planning and tool-use capabilities, a common trend is to grant them greater autonomy. Yet many real-world tasks already benefit from reliable procedural knowledge, making some model-controlled decisions unnecessary and error-prone. This raises a central question: which decisions should be controlled by the model, and which should be determined by task structure? To answer this question, we characterize the structure–agency trade-off through Agency Complexity, which measures the decision space exposed to model control, and two competing risks: Decision Error from incorrect model choices and Structural Bias from constraints that exclude successful trajectories. Guided by this analysis, we propose Prior-Guided Agency Allocation (PGAA), a principled framework for allocating control between reliable procedural priors and model judgment, formalized through a Human-Prior Decision Graph. To evaluate PGAA, we construct MineDoc, a challenging and representative benchmark for Web multimodal document mining, combining industrial procedural knowledge with instance-dependent judgments in retrieval, captioning, and LLM-as-a-Judge verification. It includes internal real-world business data and two public-data test sets simulating business scenarios, derived from Encyclopedic-VQA data associated with iNaturalist 2021 and Google Landmarks Dataset v2, respectively. We instantiate PGAA through a Filter–Produce–Reflect-ReAct mechanism and evaluate it on MineDoc. Under matched models, tools, inputs, and inference budget limits, PGAA improves reliability over fixed workflows and autonomous baselines while reducing inference cost relative to autonomous execution. Analyses of agency allocation and prior quality further support the trade-off, suggesting that effective agent design allocates control where model judgment is most valuable.