- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces InfoAgent, a training-free framework for generating and repairing infographics grounded in factual evidence. Its main contribution is the Infographic Visual Description (IVD), an executable representation that links factual content, evidence provenance, spatial regions, element dependencies, execution routes, and verification obligations.
Critical strengths
Strengths
a. Coherent end-to-end framework. The paper integrates evidence provenance, typed execution, verification, and localized repair within a single framework. The IVD provides a practical representation that connects these components across the generation and repair pipeline.
b. Comprehensive evaluation. The experiments include controlled baselines, mechanism ablations, independent external evaluation, and repair-locality analysis. This provides stronger evidence for the proposed method than relying only on qualitative visual examples.
Critical weaknesses
Weaknesses
-
Limited whole-infographic reliability. Despite the reported improvements, only 28.5% of outputs satisfy the complete external checklist on InfoGraphicBench-Evidence, and 41.0% of IGenBench outputs still fail at least one atomic question. These results indicate that the complete system still has substantial reliability limitations.
-
Limited evaluation under imperfect evidence. InfoGraphicBench-Evidence uses shared, frozen evidence bundles, while real-world retrieval may return missing, conflicting, or outdated evidence. The current evaluation does not show how robust evidence binding and repair remain under these more realistic conditions.
Questions for authors
-
Repair objective: Eq. 4 requires improvement in critical obligations, and Section 3.4 stops repair once all critical obligations pass. How are the remaining non-critical failures handled? Could the authors provide a failure breakdown explaining the gap between 61.5% CritPass and 28.5% Full?
-
Verification reliability: The binding and visual-semantic checkers achieve 84.9 F1 and 82.7 F1, respectively. How often do false PASS or UNKNOWN decisions prevent a correct repair? Could the authors report checker-specific confusion matrices and the final unresolved-obligation rates in the main evaluation?
-
Efficiency and controlled comparison: What are the average latency, number of model calls, and inference costs for IVD compilation, layered rendering, verification? Does the 3.5-point Full improvement over GenClaw remain when the methods are compared under matched inference budgets?
-
Novelty What is the fundamental methodological novelty of InfoAgent beyond integrating existing structured generation, evidence grounding, and localized repair techniques, and how do the ablations establish this contribution?
6 comments
Response to cyuJ (2/4): non-critical failures and the Full/CritPass gap
Q1. Non-critical failures and the Full/CritPass gap. Non-critical obligations enter candidate ranking through F+U after critical non-pass, but need not trigger another repair round once all internal critical obligations pass. Every previously passed critical obligation must remain PASS. P(H) protects previously passed obligations outside the original dependency closure, including non-critical ones; passed non-critical obligations inside that closure lack equivalent hard protection.
Under the evaluator definition, both Full and CritPass exclude additional critical errors. Full additionally requires every non-critical checklist item to PASS. Full therefore implies CritPass. The request-level cross-tab on the same final artifacts and evaluation version is:
| CritPass PASS | CritPass non-PASS | |
|---|---|---|
| Full PASS | 57 | 0 |
| Full non-PASS | 66 | 77 |
Each of the 66 CritPass-only outputs has no additional critical error under this definition and at least one non-critical checklist FAIL or UNKNOWN. Non-critical is not synonymous with cosmetic: these requirements include secondary content, relations, and layout.
The primary unmet-item category for these 66 requests is:
| Primary non-critical failure category | Requests |
|---|---|
| Secondary content or explanatory detail | 24 |
| Non-critical relation or binding | 13 |
| Secondary layout or reading-flow requirement | 23 |
| Decorative or stylistic requirement | 6 |
| Total | 66 |
A separate, co-occurring mechanism analysis found 41 requests that stopped after internal critical PASS, 28 with omitted or misconfigured non-critical obligations, 9 with within-closure non-critical regression, and 12 with at least one external non-critical UNKNOWN. These dimensions overlap and are not forced to sum to 66. For example, a request can stop after critical PASS and also contain an omitted non-critical obligation.
The results therefore explain the gap at two different levels. At the metric level, it consists entirely of non-critical checklist non-passes because additional critical errors already block CritPass. At the system level, termination after internal critical PASS is the most frequent recorded factor, while obligation omissions and repair regressions also occur. These overlapping labels do not isolate their causal contributions. It would be incorrect either to call all 66 cases decorative failures or to attribute them to additional critical errors.
Response to cyuJ (3/4): checker reliability, unresolved obligations, and lineage
Q2. Verification reliability and unresolved obligations. The originally reported 84.9/82.7 binding/visual-semantic F1 scores were computed on covered decisions, excluding UNKNOWN. Critical UNKNOWN is non-pass and never certified; it enters repair only when the execution trace exposes a typed, bounded target. Those F1 scores alone do not measure how often false PASS or UNKNOWN prevents repair.
We completed an audit of 1,000 checker decisions with externally determinate reference labels across all 200 main runs; another 37 externally undetermined decisions were retained separately. The table summarizes false PASS, false FAIL, and online UNKNOWN counts by checker. It is an error summary rather than a complete confusion matrix. Wrong-target/scope errors remain system verification failures; missing obligations are not checker decisions. Each decision was assessed against an external judgment on the exact artifact and target checked. The new sample is separate from the submitted 600-decision audit.
| Checker | Audited decisions | False PASS | False FAIL | UNKNOWN |
|---|---|---|---|---|
| Rendering-tree | 300 | 2 | 4 | 0 |
| Evidence support | 250 | 12 | 15 | 16 |
| Binding | 230 | 17 | 19 | 14 |
| Visual semantic | 220 | 24 | 21 | 25 |
| Total | 1,000 | 55 | 59 | 55 |
Among the 55 false PASS cases, all externally confirmed violations, 49 triggered no patch; 52 remained unresolved and 3 were corrected through another check. Of the 55 UNKNOWN decisions, 32 corresponded to externally confirmed violations and 23 to externally passing content. Of the 32 violation-related UNKNOWN cases, 26 remained unresolved and 6 were corrected. Together, 78/87 (89.7%) externally confirmed violation cases associated with audited false PASS or UNKNOWN decisions remained unresolved at the end. This is conditional on the audited decision categories, not an overall unresolved-error rate.
The 23 UNKNOWN decisions on externally passing content are excluded from that rate and analyzed separately for unnecessary repair, rollback, and extra cost. The 37 externally undetermined decisions also remain separate. Across all 55 UNKNOWN decisions, 31 had no bounded target and 24 entered repair; these counts do not specify their cross-tabulation with external truth. These are observed control paths, not claims that correcting the checker would necessarily have repaired the violations. For covered-decision F1, excluding the 55 online UNKNOWN decisions leaves 945 eligible decisions among the 1,000 externally determinate cases.
The final certificates over all 200 runs contain 3,200 active obligations. Their final online states are shown below. These counts describe internal decisions; a PASS entry can still conceal an externally identified violation.
| Checker | Final obligations | FAIL | UNKNOWN |
|---|---|---|---|
| Rendering-tree | 1,280 | 32 | 0 |
| Evidence support | 800 | 54 | 38 |
| Binding | 640 | 47 | 29 |
| Visual semantic | 480 | 41 | 51 |
| Total | 3,200 | 174 | 118 |
Hence, 292/3,200 final active obligations (9.1%) remain FAIL or UNKNOWN. By criticality, 216/2,050 critical obligations (10.5%) and 76/1,150 non-critical obligations (6.6%) remain non-pass. These are obligation-level rates and are not interchangeable with the 77/200 image-level CritPass non-pass rate.
Obligation lineage and acceptance. We also completed the audit of all 412 accepted repair transitions. The initial 3,160 obligations became 3,200 final obligations through 96 additions, 24 equivalent two-to-one merges, and 32 duplicate removals. The audit recorded zero predicate, argument, or scope weakenings, zero criticality demotions, and zero acceptance decisions affected by these merges or duplicate removals. Semantic-equivalence labels were assessed from obligation meaning rather than ID preservation. The audit found no accepted candidate whose reduction in critical non-pass was obtained through deletion or demotion. This inventory ledger describes obligation changes; it is distinct from final-state rates over the initial obligation lineages.
The budget-controlled reruns used the submitted acceptor. No lineage-guard or revised-controller result is included in any number reported here. The error summary, final certificate states, and lineage audit address different quantities: decision errors in the audited sample, active obligations left non-passing, and changes to the declared obligation set.
Response to cyuJ (1/4): reliability and imperfect evidence
Thank you for the detailed questions. We completed the evidence perturbations, request-level Full/CritPass breakdown, checker and obligation-lineage audits, and stage-cost reconstruction reported below. The revision and anonymous artifacts contain the corresponding results, and we summarize their conclusions and limits below.
Whole-infographic reliability. We agree that 57/200 T200 outputs passing Full and I-ACC=59.0% are substantial limitations: 41.0% of IGenBench outputs fail at least one atomic question. Our results support relative improvement and inspectable unresolved obligations, not complete reliability.
Imperfect evidence. We completed two new 80-request diagnostic arms: missing-only and outdated-only. We distinguish the submitted S4 results from the new perturbation results below. In the new arms, generation receives condition-specific evidence, while ReqCov and Full use independent references aligned to the request's target time. NonSup. measures support against the evidence actually supplied; it is not target-time factual correctness. Injected evidence is not the factual reference for ReqCov/Full. Each new run and scoring configuration has a separate immutable version identifier in the released manifest.
Submitted Table S4:
Results from the completed perturbation runs:
The submitted conflicting pool includes temporal and definition mismatches. In the new arms, missing-only evidence gives Full of 14/80 and ReqCov of 73.9, compared with 17/80 and 79.6 under outdated-only evidence. These are point estimates for the specified perturbations, not evidence that missing information is generally more damaging than outdated information. In 23 outdated-only outputs, a claim was supported by the supplied source but failed the requirement at the request's target time. Source support and temporal correctness therefore measure different properties.
We distinguish deficiencies introduced by the perturbation from errors in subsequent evidence selection, qualification, compilation, and repair. When additional acquisition is unavailable, rendering or binding repair cannot itself provide missing source support. This limits what repair can accomplish under incomplete or temporally mismatched evidence. Replies 2–4 report the failure decomposition, checker audit, efficiency, and methodological comparison.