acceptodds
Desk rejected at ICLR 2027

The Screen Is Not the State: Causally Grounded Artifact Critics for Office Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Office agents produce editable documents, spreadsheets, and presentations whose correctness depends on more than visible content. Required formulas, heading styles, or speaker notes may be absent even when a file appears correct. Such failures can violate one requirement while leaving others satisfied and different implementations may satisfy the same requirements. Reliable evaluation therefore requires both access to native file state and an understanding of which rubric judgments should change or remain stable after an edit. We introduce CARE, a rubric-conditioned artifact critic that combines native and rendered evidence with programmatically verified relation supervision. Its **locality** objective encourages sensitivity on affected rubrics and consistency on unaffected rubrics and **invariance** stabilizes predictions across reward-preserving implementations. Inference requires no golden artifact. Experiments on a verified DOCX, XLSX, and PPTX corpus show reduced drift across three held-out splits. In an ablation study, adding locality to pairwise reward modeling reduces unaffected-rubric drift from to , and adding invariance reduces reward-preserving candidate-score drift from to . These results show that relational supervision can improve response stability without necessarily improving classification or candidate selection.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
representation learning, language model, critic model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_k2rhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CARE is a judge for Office files. It scores how well each file satisfies task rubrics. It is trained on controlled file edits, so that scores respond to relevant changes and stay stable when rubric satisfaction does not change.

Critical strengths
  1. This paper tackles a meaningful evaluation problem. Files that appear identical can still differ in required formulas, styles, or speaker notes.
Critical weaknesses
  1. The practical advantage over SFT is still not clear. On all three test splits, full CARE achieves lower macro-F1 than Artifact-SFT. Invariance improves consistency, but this gain comes at the cost of weaker classification performance. Greater score consistency alone does not make a better judge.
  2. The data and preprocessing pipeline is hard to reproduce. The paper mentions "registered, redistributable Office sources" but does not identify them. It also provides limited detail on how task-specific edits are built and how relevant file information is chosen for each rubric.
  3. The OfficeVal results contain a consequential inconsistency. The introduction reports 0.06705 for full CARE and 0.06999 without invariance, whereas Table 2 and §6.5 report the opposite. This flips the conclusion about whether invariance helps or hurts candidate selection.
  4. The OfficeVal candidate pool offers limited evidence about stronger outputs. Even choosing the best candidate with the official verifier reaches only 0.09225. It remains unclear whether CARE can reliably tell apart smaller quality differences among higher-quality files.
Questions for authors

See Weaknesses 1–4.

Reviewer_byd9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

For DOCX, XLSX, and PPTX outputs made by Office agents, you cannot judge correctness only from what is on screen. For example, a heading can look correct but not use the real heading style, or a total can show the right number but have no SUM formula. The authors call this case, where rendering is the same but rubric reward is different, artifact aliasing. Main findings are four. Native structure improves observability. For open models without training, adding structure to renders increases macro-F1. Render-only judges give a tie on all 1,141 alias pairs. Locality and invariance reduce drift. Adding locality to Pairwise-RM reduces unaffected drift from 0.226 to 0.044 (source split), and adding invariance reduces preserving drift from 0.096 to 0.031. But with invariance, macro-F1 drops from 0.853 to 0.801 on source and from 0.814 to 0.764 on executor-style. The critic depends strongly on the candidate’s own native structure. Structure-only input is much better than render-only, and replacing it with another candidate’s structure drops macro-F1 a lot. OfficeVal Best-of-4 selection: on 84 tasks, full CARE gets 0.06999, CARE w/o invariance 0.06705, random 0.05663, and oracle 0.09225 (based on Table 2).

Critical strengths
  • The problem definition is clear and practical. The idea “the screen is not the state” and the split into reward-changing and reward-preserving relations to say how a critic should respond is a clean and useful framing.

  • The relation-based supervision design is good. The locality loss directly supervises which rubrics should change and which should stay, and points to something aggregate ranking misses. The observation that Pairwise-RM increases unaffected drift is also interesting.

  • Data verification is careful. Only groups where two independent verifiers agree are kept, and alias checking uses both pixel-level comparison and OCR.

  • The trade-off is not hidden. The paper reports that invariance improves stability but lowers classification performance.

  • The input intervention experiment is convincing. Performance collapsing when another candidate’s structure is used is good evidence that the critic really uses each candidate’s native state.

Critical weaknesses
  • OfficeVal results contradict each other inside the paper. The Abstract and Introduction say full CARE is 0.06705 and the no-invariance version is 0.06999, and that "the full objective lowers the selection score". But Table 2, Section 6.5, and the Conclusion say full CARE is the best at 0.06999. The direction of the key downstream result is flipped, so we cannot know which one is right.

  • The OfficeVal results have weak statistical meaning. The scores are very low (0.057–0.092), and differences between methods are around 0.003. There are no seed variance, CIs, or significance tests on 84 tasks. With this size of difference, it is hard to conclude "CARE is useful for selection".

  • That render-only cannot solve aliases is obvious by definition. If the input is the same, the output is the same, so a tie is expected and alias accuracy 0.5 is a tautology. Also, render-only open models get accuracy 0.07–0.28 and macro-F1 around 0.1, which looks less like an alias problem and more like a weak baseline with prompt or parsing failures. There is no strong proprietary judge (GPT, Claude, etc.) or a text-only LLM baseline that sees only the XML.

  • There is a shortcut risk from synthetic data. Candidates are made by deterministic edits with six fixed roles, and the input includes diffs from the initial file. Then the critic may solve the task by classifying edit-type patterns instead of understanding rubrics. The very small gain on real agent outputs (OfficeVal) also supports this concern.

  • The benefit of locality is limited. Locality mostly undoes the unaffected drift created by the rank loss. Artifact-SFT itself already has lower drift (source 0.027 vs 0.044, executor 0.040 vs 0.063). So the benefit of locality compared to SFT is not clear.

  • The cost of invariance is large. Preserving drift goes down, but macro-F1 drops about 5pp. There is no analysis of what point is good between stability and accuracy, or a trade-off curve when λ changes.

  • It is not explained well why a critic is needed when a verifier exists. Data construction already has a task-aware verifier and an oracle. An experiment is needed showing when a learned critic is valuable compared to this verifier (for example, new tasks or rubrics without a verifier). On OfficeVal the gap to the oracle is also large.

  • Information about data and setup is lacking. Source origin, number of tasks and rubrics, distribution and results per format (DOCX/XLSX/PPTX), how rubrics are written and checked, and the design rules of the deterministic selector are almost not explained. In particular, we need to check whether the way the selector maps rubrics to entities could include hints of the answer.

Questions for authors
  • For OfficeVal, the numbers in the Abstract/Introduction and in Table 2/Section 6.5 are opposite. Which is actually higher, full CARE or CARE w/o invariance?

  • Can you add seed variance, bootstrap CIs, and paired tests to the OfficeVal results? Is the 0.003 difference significant?

  • Render-only open models have very low accuracy (0.07–0.28). What is the output parsing failure rate? How do a strong proprietary judge or a text-only LLM baseline that sees only the XML perform?

  • Is it possible that the critic learned edit patterns as a shortcut because of the six fixed candidate roles and diff input? Do you have classification performance evaluated with human labels on outputs made by real agents?

  • What is the pure benefit of locality compared to Artifact-SFT? Did you try SFT + locality (without rank)?

  • What does the trade-off curve between drift and macro-F1 look like when the invariance weight λ changes?

Reviewer_vbydOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CARE, a rubric-conditioned critic specifically designed for Office artifacts. It combines the native file structure, rendering evidence, local response constraints, and invariance constraints to predict the scoring items of the documents. The authors constructed reward-changing and reward-preserving samples through programmatic editing and evaluated the classification, sorting, and scoring stability on three data partitions: source, task-mutation, and synthetic executor-style. Finally, they verified the selection effect on a fixed Best-of-4 candidate pool of 84 OfficeVal tasks.

Critical strengths

None. Both of the two main contributions of the paper have not fully reached the level of supporting acceptance: Firstly, although the combination of the original file state and rendering evidence has experimental support, existing works have already addressed the structured evaluation of Office artifacts. This paper has not fully defined the substantive increment of its substantive increment in the evaluation mechanism; Secondly, the local response and invariance constraints do indeed improve the corresponding drift indicators, but the invariance constraints simultaneously reduce the classification performance on multiple test sets, and the downstream selection experiments are not sufficient to prove the stability of its benefits.

Critical weaknesses

1.The boundaries of innovation for the core method have not yet been established. This paper cites relevant works such as counterfactual data, contrast sets, consistency learning, and reWordBench, but the main mechanism of CARE still involves imposing directional constraints on affected scoring items, consistency constraints on unaffected scoring items, and invariance constraints on candidate rewards to be maintained. The paper does not fully demonstrate the irreplaceable capabilities of these designs compared to existing paired ranking and consistency learning objectives. The existing ablation experiments show that different losses affect corresponding indicators, but they cannot directly prove that this method surpasses the conventional combination of existing goals.

2.The experimental design is insufficient to support the universal validity of the method, and the core results are inconsistent. The tra2ining data is mainly constructed through programmatic editing. Each task generates preset types of candidate files, and the supervisory labels are determined based on the consistent results of the task-aware validator and the independent package-level oracle. This design can verify the scoring responses under controlled editing, but it is not sufficient to demonstrate that the method can handle the open-ended errors that occur during the real Office Agent generation process. The external validation is only conducted on the fixed four candidate pools of 84 OfficeVal tasks, without providing task-level uncertainty or significance analysis. More importantly, the summary writes the scores of the complete CARE and the version without invariance constraints as 0.06705 and 0.06999 respectively, while the corresponding results given in Table 2 and Section 6.5 are exactly the opposite. This issue needs to be clearly corrected; otherwise, the downstream selection benefits and their conclusions cannot be verified.

Questions for authors

1.What are the essential differences between CARE and existing counterfactual supervision, pairwise ranking, and consistency learning? Under the same data, model, and training budget, compared to standard contrastive loss or consistency loss, how much measurable increment does the local response and invariance design of this paper bring?

2.Do the training set and test set generated by programmatic editing share the editing operator or generation template? On the basis of leaving out the editing mechanism, unseen error types, and real Agent-generated Office artifacts, can the same rating stability benefits still be obtained?

3.Does programmatic supervision and validator consistency filtering limit the data distribution? Please report the number of samples before and after filtering, the proportion of validator inconsistency, and the independent validation results on naturally generated errors to distinguish the model's fitting to the preset editing rules and its ability to judge the quality of actual artifacts.

4.What is the complete CARE score of OfficeVal: 0.06705 or 0.06999? Please correct the contradiction between the abstract and the main text and report the paired confidence intervals or significance tests on 84 tasks to verify whether the benefit of the complete CARE compared to the version without invariance constraints is reliable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.