acceptodds
Under review as a conference paper at ICLR 2027

Taking the PULSE of Enterprise Workflows: Evaluating Persistent Project-State Understanding

Abstract

Enterprise agents must maintain an up-to-date understanding of work distributed across heterogeneous artifacts, evolving over time, and unevenly visible to different collaborators. We formulate this challenge as persistent project-state understanding (PPSU) and introduce PULSE, a benchmark of 1,573 questions over 30 enterprise project histories. PULSE constructs deterministic, time-indexed project state and documented collaborator views from source-linked records, and evaluates three families of questions: open loops, project evolution, and view gaps. Across retrieval agents, graph- and memory-augmented retrieval, long-context, and oracle conditions, we find substantial headroom for current systems. Retrieval and extraction account for the largest share of the gap, but providing explicit project state on top of gold evidence yields further substantial gains, showing that PPSU cannot be reduced to retrieval alone. Further analysis shows that questions requiring information to be reconciled across multiple parts of the project history remain difficult even when the relevant evidence is provided, while project-state synthesis and perspective conditioning remain persistent challenges. These results motivate enterprise agents that explicitly maintain and reconcile temporally grounded state across sources, actors, and time. We will publicly release PULSE upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
enterprise agents, enterprise data, benchmarks, evaluation, retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3eKcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formulates the challenge of agent adaptation in evolving enterprise settings as persistent project-state understanding and introduces PULSE, a benchmark of 1,573 questions over 30 project histories. The benchmark introduces three categories of questions, namely, open-state, project evolution and view gaps. This allowed failure attribution to be associated with the query category by the retrieval and memory agents. The in-depth analysis shows that current enterprise systems need substantial improvement in various aspects, specifically, the retrieval remains a major challenge though explicit project-state information substantially improves performance even when evidence is available. The work also shows that enterprise agents require temporally grounded state representations, although perspective-dependent reasoning and higher-level project synthesis remains a major challenge.

Critical strengths
  1. Evaluating LLMs in evolving enterprise environments is an interesting and real-world problem. The authors attempt to formulate this challenge as persistent project-state understanding, targeting enterprise-agent evaluation over heterogeneous, evolving, and perspective-dependent work histories.

  2. The benchmark PULSE is created in a structurally consistent way, containing 1,573 questions across 30 enterprise project histories, covering open loops, project evolution, and collaborator-specific information gaps. Its deterministic, time-indexed state construction preserves provenance links between source artifacts, state transitions, and reference answers. The structured creation of the benchmark enables systematic assessment of the agentic capabilities.

  3. Controlled experiments in the paper is strong, providing good insights into retrieval-based agents, long-context evaluation, gold-evidence oracle, and explicit-state oracle. The authors show that the gaps in current enterprise systems lie mainly in retrieving the correct information, followed by project-state reconstruction, which is needed to adapt to changing environments

  4. An interesting insight from the work is the distinction between global project state and the information documented as available to individual collaborators. The view-gap questions assess whether agents can identify missing updates, stale information, and differences between individual and team-level perspectives.

Critical weaknesses
  1. There can be a potential bias when the answering model and the judge model come from the same family. To mitigate this bias, it is necessary to perform cross-family model evaluation and human evaluation on a selected sample.

  2. The choice of categories created at different stages is not clear. For example, why did the author select only the specified six typed stores? How do these categories help PULSE? Are the categories mentioned in Figure 2 exhaustive?

  3. The process of benchmark curation is hard to follow, especially source-linking, inducing project state snapshots, and documented view construction. I understood that the motivation for choosing them comes from the Related Work section, but how they were targeted is confusing.

  4. The motivation behind choosing Open Loop, Evolution, and View Gaps as the categories for question generation is not clear.

Questions for authors

See Weakness Section.

3 comments
AuthorsOpenReview ↗

Response to Same-Family Judge Bias (Weakness 1) and Why Six Typed Record Stores (Weakness 2)

We thank the reviewer for insightful questions. Below, we respond to each concern.

  1. Same-family judge bias (Weakness 1)

We acknowledge the potential for same-family evaluation bias and we ran a cross-family evaluation to check it. It is in a new appendix section, "C5. Cross-Family Judge Check" (Line 1279, marked in green). Moreover, only 2 of the 18 question types rely on LLM-based judging. The remaining 16 are evaluated using objective criteria. Thus, the potential same-family bias is limited to a small portion of our evaluation.

We re-judged E8 and E9 with Claude Sonnet 5.5, run through Claude Code. Because of time and API constraints, we covered the first five products (28 E8 and 30 E9 questions). Only the judge changes. The prompts, the parsing of verdicts and the scoring code are identical to the gpt-5.6-luna run. We judged five answer sets: the four GPT-answered conditions and a Claude Code (Opus 5.5) agent from our new answering-model pilot (see item 1 in our response to Reviewer F7SD).

  • The two judges agree closely. They rank the five systems in exactly the same order on both E8 and E9. Per-question correlations are high: Pearson r = 0.90 for E8 and 0.91 for E9.
  • E8 is uniformly stricter under Claude. Claude scores every system 9.2–15.3 points lower. The shift is similar across systems, so the gaps our conclusions rely on are preserved.
  • E9 is essentially unchanged. System-level Paired F1 differs by at most 1.4 points between the two judges, so the pairing decisions do not depend on the judge's model family.
  • No sign of self-preference. Under the GPT judge, Claude Code's answers lead the GPT-answered ReAct agent by 18.2 points on E8. Under the Claude judge, they lead by 12.2 points. The GPT judge thus gives the other family's answers a larger advantage, not a smaller one.

E8 Judge score and E9 Paired F1 under each judge (first five products; only the judge changes):

SystemE8 Judge: GPTE8 Judge: ClaudeE9 Paired F1: GPTE9 Paired F1: Claude
ReAct (gpt-5.6-sol)36.527.283.985.3
Claude Code (Opus 5.5)54.739.492.993.4
Long context47.133.093.093.5
Citation oracle49.836.781.181.5
State oracle84.575.390.990.3

Per-item agreement between the two judges (pooled over the five answer sets):

ItemnPearson rSpearman ρMean abs. diff.
E8 question score1400.900.8812.3
E8 single field5600.890.9214.3
E9 question F11500.910.893.2
  1. Rationale for the six typed record stores (Weakness 2)

We considered a broad range of project management concepts, including dependencies, decisions, priorities, and deadlines. However, constrained by the available HERB data and the need for reliable labels, we retained only categories satisfying three criteria: (1) State-bearing: having an explicit lifecycle (e.g., open → discharged); (2) Deterministically extractable: having both creation and closure evidence recorded in HERB; and (3) Source-linked: having every state transition traceable to specific artifacts.

The resulting six stores cover complementary aspects of project state: what is produced (work items), what is promised (commitments), what blocks progress (blockers), changes of direction (reversions), external demand (customer bugs), and who knows what (visibility). While these stores do not exhaust all potentially interesting management concepts, they represent the categories that HERB can reliably support under our construction criteria.

AuthorsOpenReview ↗

Response to Concerns on Pipeline Construction (Weakness 3) and Question Families Design (Weakness 4)

  1. Clarity of the construction pipeline (Weakness 3)

We agree that the pipeline is hard to show in one figure or one section. Each record type has its own rules for what opens it, what closes it, and who can be shown to know about it. Showing all of this in the main text would make it harder to read. We therefore split the explanation into two levels:

  • A worked-example figure in Section 3 (new figure, Line 182, marked in green), for the intuition. It traces one commitment through all three steps. A chat message opens the commitment, and a later pull request closes it (source-linking). The commitment is open in the snapshot before the pull request and closed in the snapshot after it (snapshots). An employee who took part in the chat but never saw the pull request knows of the promise but not its closure, so it is stale in that employee's documented view (documented view).
  • Detailed rules in the appendix, for completeness. A new appendix section, "A2. Record Lifecycle and View Definitions" (Line 742, marked in blue, since it is also Response item 1 to Reviewer gEpY), gives a table of how each record type is opened and closed, formal definitions of open state at a cutoff and of documented views, and a step-by-step trace of a commitment made in a meeting.
  1. Rationale for the three question families (Weakness 4)

PPSU asks what an agent must know to understand ongoing work. We organize this into three conceptually orthogonal dimensions:

  • Open Loops (State): What remains unresolved at a given time? This captures a snapshot of project state and requires determining whether commitments or blockers have been closed.
  • Evolution (Time): How has the project state changed over an interval? This captures the delta between two points in time, requiring tracking transitions such as supersession and reversal.
  • View Gaps (Perspective): How does project state appear from different collaborators' perspectives? This requires projecting the same project state or its evolution onto partial, person-specific information.

Together, these dimensions capture three distinct challenges in understanding ongoing work: reconstructing the current state, tracking how it changes, and reasoning about how it is perceived differently. They also correspond to the three gaps in prior work discussed in §2: deep search over evidence, temporal tracking of atomic facts, and treating partial information merely as an access-control problem. Each question family isolates a capability that retrieval alone cannot provide.

Reviewer_3eKcOpenReview ↗

Reply to authors' response

I thank the authors for the clarification, additional analysis provided above, and revising the paper. It has improved my understanding. The paper is a good contribution to the agentic benchmark with temporally evolving enterprise tasks. I will update my score.

Reviewer_gEpYOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how agents track project state across records and time. PULSE has 1,573 questions over a reported 30 project histories from HERB. These histories are synthetic. The questions ask what work remains open, how the project changed, and what each person is documented to know. The authors build structured labels with fixed rules. They compare retrieval agents, memory systems, long context, and two oracle settings. They find that better evidence helps, and that giving explicit state information raises scores further.

Critical strengths

I vote for the defined problem. Team work requires knowing what is still open and what other people have seen. I think the combination of time and personal views adds value beyond finding a relevant document.

Critical weaknesses
  • I think one shown label may be wrong. The paper lists onForceX_planning_4-t012 as open on September 9, 2026, and unclosed by March 6, 2027. Under a zero-based reading of HERB dialogue turns, this is a July 17 promise to add partnership target industries to a technical document. The same author publishes a document covering this change on July 20. This is before the cutoff. I cannot confirm an error without the exact turn mapping and closing rules. But I find this a concrete reason to ask for a human check of label meaning.
  • I do not think the state-oracle gain is enough to establish the proposed system direction. The oracle gets state chosen for the question, with some needed lookup fields already resolved. I understand why the authors use an oracle. But the 11.2-point gain does not tell me which part of that added information helps. I would need a matched structured-input control and a system that builds state from raw records to judge the architecture claim. The simple enumeration control is too weak for that purpose.
  • I am not satisfied with the evidence for ongoing state maintenance. The current task asks a model to rebuild state at query time. It does not test updates as new records arrive, error correction, forgetting, or long-term cost. The authors list online work as a future direction. I think the current evidence is stronger for offline reconstruction than for claims about persistent online systems.
  • I cannot understand the data coverage from the text. The main paper says 30 projects. Appendix A.1 refers to “two of the five products.” This may be an editing mistake or a subset, but it needs an explanation. I also want the evaluation to account for shared histories across questions. The 1,573 questions are not 1,573 independent project cases.
Questions for authors
  • What exact turn does onForceX_planning_4-t012 refer to? Why does latest_onforcex_technical_specifications_document, dated July 20, not close it? Please explain the linking rule and any human label audit.
  • Can you add a control with the same input length and structure, without supplying the target state? Do you have a non-oracle baseline that extracts state from raw history?
  • Does PULSE use 30 products or a smaller subset? Please give record and question counts by product, with uncertainty estimates that account for questions sharing the same history.
3 comments
AuthorsOpenReview ↗

Response to Label Correctness and Human Audit (Weakness 1; Question 1)

We thank the reviewer for their detailed read of our paper. Our responses to the noted concerns and questions are as follows:

  1. Label correctness and human audit (Weakness 1; Question 1)

Thank you for bringing this up. Upon further inspection, we confirmed that the label is correct, but your comment highlights an edge case in our evaluation pipeline, and we appreciate the opportunity to clarify.

Why the label is correct: Although a revised version of the Technical Specifications Document was indeed released on July 20 (following the July 17 review meeting), a content-level inspection reveals that its "Go-to-Market Strategy" section, the specific target of this commitment, is identical to the previous version. Because the promised addition of partnership target industries was never actually implemented or integrated into the updated text, our pipeline correctly determined that onForceX_planning_4-t012 was never fulfilled and thus remains unclosed.

We acknowledge that the appendix's previous description of commitment closure was outdated. Our current pipeline determines closure by comparing document versions, rather than simply checking whether the new version mentions the commitment.

During benchmark construction, we encountered lots of branching scenarios and corner cases, which may not have been sufficiently documented in the original appendix. To address this, we have added a new appendix section, "A2. Record Lifecycle and View Definitions" (Line 764, marked in blue). It gives formal definitions and a table of the opening and closing criteria for each record type, together with how records are assigned to each employee's documented view.

We did not run a separate human annotation study, and we acknowledge this as a limitation. Instead, we checked labels by hand throughout benchmark design. In each design iteration, we inspected generated questions and gold answers, traced every labeling defect to its cause, and turned each defect class into an automated validation gate. This yields 19 gates, covering temporal correctness (no gold answer uses evidence from after the cutoff), grounding (every graded item is backed by a cited artifact), well-posed gold answers, and answer leakage. All 30 products pass all 19 gates. Please see our response to Reviewer F7SD (item 6) and the new appendix section "A4. Validation Gates" (Line 931) for details.

AuthorsOpenReview ↗

Response to Statle-Oracle Gain (Weakness 2; Question 2) and Scope of claims (Weakness 3)

  1. What drives the state-oracle gain (Weakness 2; Question 2)

We would like to clarify what “state” means in PULSE. In PULSE, each record is defined by its evidence links (Appendix A.1): the utterance that opened it, the artifact that later closed or superseded it, and the artifact that is currently authoritative. A record’s state at a cutoff (open, closed, superseded, or reverted, and who is documented to have seen it) follows from these links. The identifiers of all linked artifacts already appear in the raw gold artifacts given to both oracle conditions. The citation oracle therefore has the same underlying content. What it lacks is the resolution of these links: deciding which artifact closes which commitment, which version supersedes which, and what holds at the cutoff. Recovering these links from raw evidence is exactly the state-construction step PULSE is designed to measure. The resolved lookup fields (e.g., an item’s current replacement or evidence holder) are attributes derived from the same links, so they are part of the project state rather than additional external information. The scoping to the question’s cutoff and window likewise follows directly from the time-indexed definition of state.

The state oracle is a reference condition. It supplies the complete project state, under PULSE’s definition and scope, on top of the same gold evidence. Its purpose is to measure how much performance becomes available once state construction is solved.

We agree that the appropriate form of project state is an important open question. It is likely to differ across industries, organizations, and the access an agent has to enterprise systems. This paper does not try to define or propose an optimal representation. We intentionally define PULSE records and format in a general manner, and the state definition is one auditable instantiation, but by no means universal. In the revision, we softened the wording in Sections 5 (Line 481) and 6 (Line 510, marked in blue) so that it reads as evidence that making project state explicit helps, not as support for a particular system architecture.

We do not include a dedicated baseline that reconstructs PULSE-style project state from raw history. Designing such a system is a method contribution, and we consider it out of scope for this benchmark paper; PULSE is meant to enable and measure such systems. The closest existing systems, such as Graphiti and A-MEM, are already evaluated. Their performances are far below the state oracle, suggesting that general-purpose memory structures do not recover project state and that building such a state extractor is a promising direction for future work.

Regarding input structure, all agents receive JSON-formatted inputs, ensuring consistency in the input format. We acknowledge that representation choices may still influence performance, but the observed gain should be interpreted as the potential benefit of providing resolved state, rather than evidence that our particular state representation is optimal.

  1. Scope of claims: offline reconstruction vs. persistent maintenance (Weakness 3)

We agree with this distinction. PULSE evaluates query-time, time-indexed reconstruction of project state. It does not evaluate online maintenance, such as incremental updates, error correction, forgetting, or long-term cost. That setting is out of scope here, and we list it as future work. As discussed in 2, PULSE’s role is to show why state tracking matters: even with gold evidence, models struggle to reconstruct state, and making it explicit yields substantial gains. Accurate reconstruction is therefore a prerequisite for persistent maintenance.

AuthorsOpenReview ↗

Response to Concerns on Data Coverage and Statistical Independence (Weakness 4; Question 3)

  1. Data coverage inconsistency (Weakness 4; Question 3)

Thank you for catching this. It was an editing error in Appendix A.1. PULSE uses all 30 products, consistent with the main paper. The sentence "two of the five products" and the record counts in that section were left over from an earlier version of the paper. We have corrected Appendix A.1 (Line 730, marked in blue). All record counts now cover the 30 products: 1,837 work items, 2,017 commitments, 223 blockers, 89 reversions, 420 customer bugs, and 18,264 visibility relations.

  1. Non-independence of questions, record/question count (Weakness 4; Question 3)

We agree that the 1,573 questions are not 1,573 independent cases, since questions from the same product share one project history. We added two analyses:

  • Counts by product: A new appendix table, "C7. Per-Product Breakdown" (Line 1382, marked in blue), lists for each of the 30 products the number of questions per family, the number of records per store, and the item-level F1 of ReAct, long context, and the two oracles. Products contain 37 to 64 questions (median 52.5).
  • Uncertainty that accounts for shared histories: All uncertainty estimates now treat the product as the unit. We use a product-level cluster bootstrap: each replicate resamples whole products, with all of their questions, rather than individual questions (10,000 replicates). We test paired differences with a Wilcoxon signed-rank test over the 30 per-product scores. The details are in the new appendix "C6. Statistical Analysis" (Line 1327, see also our response to Reviewer F7SD, item 5).

Finally, regarding per-product consistency, the main findings hold in every product. The state oracle outperforms the citation oracle in 30 of 30 products, and the citation oracle outperforms ReAct in 30 of 30 products, with a gap of 23.6 to 47.7 points. Our conclusions therefore do not depend on a few project histories.

ConditionAll F1 [95% CI]
ReAct31.4 [29.4, 33.2]
Long context52.9 [50.0, 55.7]
Citation oracle66.9 [65.1, 68.6]
State oracle78.1 [76.4, 79.9]
GapΔ F1 [95% CI]Products where it holdsp
Long context − ReAct+21.5 [+18.4, +24.6]30/30< 0.001
Citation oracle − Long context+14.0 [+11.7, +16.6]30/30< 0.001
State oracle − Citation oracle+11.2 [+9.2, +13.3]30/30< 0.001
Reviewer_F7SDOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formulates persistent project-state understanding (PPSU) as an evaluation problem for enterprise agents that reason over heterogeneous and temporally evolving work histories. Building on the existing HERB dataset, the authors construct PULSE, a benchmark of 1,573 questions over 30 enterprise project histories organised into three families: open loops (O), project evolution (E), and view gaps (V). Each question is grounded in deterministic, time-indexed project-state oracles with source-linked evidence. The paper benchmarks four retrieval/memory systems (ReAct, HippoRAG, Graphiti, A-MEM) alongside diagnostic reference conditions, namely: long context, citation oracle, and state oracle, to progressively isolate retrieval failures from state-construction failures. The central finding is that while retrieval is the dominant bottleneck (best retrieval system: 31.4 F1 vs. state oracle: 78.1 F1), explicit project-state representation yields substantial further gains even when gold evidence is supplied, demonstrating that PPSU cannot be reduced to retrieval alone.

Critical strengths

There are two main strengths I found in this paper:

  1. The progressive ablation design (retrieval system --> long context --> citation oracle --> state oracle) is the paper's core methodological contribution. By holding the answering model and evaluation fixed while progressively removing different sources of difficulty, the paper cleanly separates retrieval, evidence selection, and state construction as distinct bottlenecks. This goes beyond the typical "systems leave headroom" finding of most benchmark papers and provides actionable guidance for where future effort should be directed. The 11.2-point gap between citation oracle (66.9) and state oracle (78.1) is a commendable demonstration that PPSU is not reducible to retrieval.

  2. The design principle of deriving multiple question families from the same underlying project history in which the same records yield different gold answers under open-loop, evolution, and view-gap rules is methodologically clean. Every gold label is traceable to source evidence, and the supporting evidence explicitly includes artifacts needed to rule out stale or invalid alternatives, not just those supporting the correct answer. This addresses a real weakness of many QA benchmarks where negative evidence is implicit.

Critical weaknesses

Two major weaknesses:

  1. Only a single answering model (gpt-5.6-sol) was used for all experimental conditions. The paper's central claim that state construction is a separable bottleneck beyond retrieval may not generalise to other model families. Models with stronger long-context reasoning (e.g., those trained explicitly on structured state tracking) might close the citation-oracle --> state-oracle gap substantially. Similarly, the retrieval/memory systems evaluated (four systems) represent a limited slice of the design space: no dense-retrieval baselines, no hybrid systems, and no systems designed specifically for multi-hop temporal reasoning. Without broader system coverage, the benchmark findings risk being a characterisation of a particular configuration rather than a general property of the PPSU problem.

  2. Heavy dependence on the HERB dataset limits community uptake and generalisability. PULSE extends HERB, which provides the underlying enterprise artifacts. While the authors promise to release PULSE upon acceptance, the benchmark's construction pipeline and any future extensions are gated by access to HERB data. The 30 project histories, while diverse enough for initial findings, are drawn from a single enterprise dataset with particular characteristics (presumably enterprise software development workflows involving chat, meetings, docs, and PRs). Whether the identified difficulty patterns (e.g., view-gap questions being hardest, cross-store composition predicting difficulty) generalise to other enterprise domains (legal, healthcare, manufacturing) is an open question the paper does not address. Furthermore, the deterministic rule-based extractors used to build the six record stores are tuned to the HERB data format — adapting PULSE to new domains would require substantial re-engineering of these extractors.

Questions for authors

Detailed description of my impression of this paper:

The paper is well-structured and well-written. Figure 1 adequately illustrates the four-stage construction pipeline, and the schematic example that distinguishes source-linked records from project state and documented views makes the benchmark design clearly graspable. The progressive experimental structure (Sections 4.1 --> 4.2 --> 4.3) mirrors the benchmark's diagnostic architecture cleanly. Table 1 is dense but well-organised, and the consistent use of the O/E/V family notation throughout helps the reader track results across question types.

The paper's diagnostic decomposition of enterprise-agent failure modes: retrieval vs. evidence selection vs. state construction, provides considerable contribution. It demonstrates that providing gold evidence improves F1 from 31.4 to 66.9 but that an additional 11.2 absolute points remain locked behind state construction (66.9 --> 78.1). Further, the authors observe that cross-store composition (record-store count) becomes predictive of difficulty only after retrieval is controlled (Table 2). These are valuable findings for the community; the insights in Table 2 have previously been masked in aggregate evaluations. However, the overarching finding that current systems struggle with enterprise workflows, while clearly quantified here, is consistent with what prior work (HERB, DRBench) has already suggested.

The benchmark construction is methodologically sound. The deterministic, rule-based extraction pipeline and the use of topic-boundary cutoffs to avoid mid-conversation snapshots reflect careful design. The progressive ablation structure (retrieval --> evidence selection --> state construction) via the long-context, citation-oracle, and state-oracle conditions is the paper's strongest methodological contribution and cleanly separates confounded failure modes. However, I have several concerns:

(i) Single answering model. All experiments use gpt-5.6-sol with gpt-5.6-luna as the judge. This makes it impossible to determine whether the identified bottlenecks (retrieval vs. state construction) are properties of the benchmark or properties of a single model family. Even one additional frontier model would substantially strengthen the findings.

(ii) Fairness of the retrieval comparison. The paper acknowledges (line 265) that the comparison between Graphiti/A-MEM and ReAct/HippoRAG does not isolate representation from ranking, retrieval budget, or implementation differences. Given that Graphiti (18.3 F1) and A-MEM (16.6 F1) underperform the heuristic shortcut (13.1 F1) on view-gap questions by only a modest margin, it is unclear whether the graph/memory approaches were given a fair evaluation or simply received insufficient tuning for this benchmark.

(iii) No statistical reporting. No error bars, confidence intervals, variance across projects, or significance tests are reported. With 30 projects of presumably varying complexity, variance across projects is a relevant signal — for example, whether the retrieval bottleneck is uniformly distributed or concentrated in a subset of project histories.

Specific questions to the authors:

  1. The gap between the citation oracle (66.9 F1) and state oracle (78.1 F1) is your key evidence that state construction is a separable bottleneck. Have you tested whether this gap persists across different answering models? If a model with stronger structured-reasoning capabilities narrows this gap substantially, it would reframe the bottleneck as a model limitation rather than a fundamental challenge.

  2. The six typed record stores are constructed using "deterministic, rule-based extractors" (Section 3, line 158). What is the inter-annotator agreement or validation coverage for these extractors? Specifically, were any of the extracted records validated by human annotators, and if so, what was the precision/recall of the automatic extraction?

  3. The Graphiti and A-MEM systems substantially underperform ReAct and HippoRAG. You acknowledge that this comparison does not isolate the source of the gap (line 265). Did you perform any ablation to determine whether the underperformance is due to the retrieval interface (these systems return fewer source-level identifiers) or the quality of the retrieved evidence? A retrieval-recall-matched comparison would clarify whether graph/memory structure helps or hurts when retrieval volume is controlled.

  4. View-gap questions (V) remain the hardest family even under the state oracle (65.8 F1 vs. 78.1 O and 96.6 E). What is driving this persistent difficulty? Is it primarily the perspective-conditioning step (projecting state onto individual views), or are the view-gap questions intrinsically harder because they require reasoning about absence (what information someone has not seen)?

  5. Table 1 reports evidence precision of 2.5% for the long-context condition and 6.7% for ReAct, despite high recall. This suggests both systems return large amounts of irrelevant evidence. What is the average number of evidence items retrieved per question across conditions, and how does this relate to the structured answer schema's capacity?

5 comments
AuthorsOpenReview ↗

Response to General Concerns on Model Generality (Weakness 1; Concern (i); Question 1) and Retrieval Coverage (Weakness 1)

We thank the reviewer for their detailed reading of our paper, comprehensive feedback, and overall positive assessment. Our responses to noted concerns and questions are as follows:

  1. Generality across answering models (Weakness 1; Concern (i); Question 1)

Thank you for this suggestion. We ran a pilot study with a second model family, using the first five products (282 questions, due to the time and api resources limitation), and added it as a new appendix section, "C4. Pilot Study: Claude Code as the Answering Agent" (Line 1238, marked in red).

  • A stronger end-to-end agent performs better. We used Claude Code (Opus 5.5), not Claude API with ReAct. It uses the same input, but a more mature harness. Thus, this is a stronger end-to-end agent comparison, not an isolated model comparison. We use it as an additional strong agent pilot analysis.
  • The gain is not just from retrieving more evidence. Recall improves moderately (47.1 to 55.7), while precision improves substantially (7.2 to 31.2), suggesting that better evidence selection and agent reasoning also contribute.
  • The main conclusion remains unchanged. Claude Code still trails the state oracle, leaving substantial headroom. Understanding project state remains a challenge. | System | O | E | V | All | Field acc. | Judge | Paired F1 | P | R | |:---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | ReAct (gpt-5.6-sol) | 16.0 | 62.4 | 16.4 | 28.5 | 24.3 | 36.5 | 83.9 | 7.2 | 47.1 | | Claude Code (Opus 5.5) | 49.3 | 82.1 | 35.9 | 52.7 | 71.4 | 54.7 | 92.9 | 31.2 | 55.7 | | Long context | 47.4 | 80.9 | 39.8 | 53.3 | 70.2 | 47.1 | 93.0 | 2.7 | 99.7 | | Citation oracle | 64.8 | 84.4 | 52.4 | 65.1 | 74.1 | 49.8 | 81.1 | – | – | | State oracle | 74.8 | 97.1 | 63.2 | 76.1 | 88.3 | 84.5 | 90.9 | – | – |
  1. Coverage of retrieval/memory systems (Weakness 1)

Thank you for raising this. The evaluated systems cover a broader design space than our original description suggested. We have revised §4 (Line 255, Line 264), the appendix on evaluated systems (Line 1108, Line 1145), and the limitations (Line 1642) to state this explicitly (marked in red).

Dense and hybrid retrieval are already covered: The ReAct agent's content search uses a hybrid retriever. It combines dense retrieval (text-embedding-3-large) with sparse BM25 retrieval. We did not add a single-shot dense RAG baseline, because HERB already runs this comparison and shows that ReAct agent performs better.

Multi-hop and temporal systems are also represented: HippoRAG is designed for multi-hop retrieval. Graphiti, the engine behind Zep, is a temporal knowledge graph. Each fact it stores carries a validity interval, which makes it the evaluated system closest to tracking state at a cutoff. However, we agree that systems designed jointly for multi-hop and temporal reasoning (e.g., TG-RAG) would be a valuable addition. We have added this to the limitations. We will release the benchmark and evaluation harness so that such systems can be evaluated under identical conditions.

​​Our central finding does not depend on the retriever: The citation oracle supplies the evidence a perfect retriever would return. Even so, it reaches only 66.9 item-level F1, compared with 78.1 for the state oracle on the same evidence. A better dense, hybrid, or temporal retriever could move a system toward the citation oracle, but not past it. The gap between the two oracles therefore reflects state reconstruction, not the choice of retriever.

AuthorsOpenReview ↗

Response to Dataset Dependence and Generalizability (Weakness 2) and Statistical Reporting (Concern (iii))

  1. Dependence on HERB (Weakness 2)

We acknowledge that PULSE builds upon the HERB dataset and relies on its underlying enterprise artifacts. However, since HERB itself is publicly available, we do not believe this dependency poses a significant barrier to community adoption or further extensions. Furthermore, we will release both the benchmark and the complete code for constructing it from HERB, making it straightforward for others to reproduce, adapt, and extend our work.

The method is not fully dependent on HERB's storage format. A dedicated loader reads HERB's raw files and normalizes them into a common representation of chat topics, meeting turns, pull requests with reviews, and versioned documents. All subsequent extraction rules, snapshot construction, question generation, and validation gates operate on this representation, relying on signals commonly available in software teams, such as review states, shared PR links, and revert titles. Building a similar benchmark from another software-development dataset would therefore primarily require implementing a new loader, along with minor adjustments to dataset-specific conventions (e.g., channel naming and document version names).

  1. Generalization beyond the HERB domain (Weakness 2)

Software development is a broad and active domain for agent research, as reflected by existing datasets and benchmarks (e.g., SWE-bench [Jimenez et al., 2024], EnterpriseBench [Vishwakarma et al., 2025]). While PRs are specific to software development, other artifacts in PULSE, such as documents, meeting transcripts, and messages, are common across enterprise domains. Even PR-like records have counterparts in other domains, such as medical record updates and design iterations. We acknowledge that adapting PULSE to different domains or even organizations may require changes to the extractors due to differences in workflows and tools. We chose software development as a starting point given its growing relevance to agent applications, with the hope that our findings and benchmark design can inform future extensions to other domains.

  1. Statistical reporting (Concern (iii))

We added uncertainty estimates, significance tests, and an analysis of variation across products in a new Appendix, “C6. Statistical Analysis” (Line 1327) and updated Section 4 (Line 293) accordingly (marked in red). These analyses do not change our conclusions: performance consistently improves along the progression ReAct → Long Context → Citation Oracle → State Oracle, with positive gains across all products.

AuthorsOpenReview ↗

Response to Validation of the Extractors (Question 2)

  1. Validation of the rule-based extractors (Question 2)

We did not run a separate expert annotation study, and we acknowledge this as a limitation. Instead, we validated the extractors through the design process itself, and we now document this in a new appendix section, "A4. Validation Gates" (Line 931, marked in red).

We built the benchmark over several design iterations. In each iteration, we inspected the generated questions and gold answers by hand, and traced every defect we found back to its cause in the rules. We then turned each defect class into an automated check, so that the same error cannot return in a later build or in a new product. This process yields 19 validation gates. The generator refuses to release a product unless every gate passes.

They fall into four groups:

  • Temporal correctness. Every question states its cutoff, and no gold answer or citation uses evidence from after that cutoff.
  • Grounding. Every graded item is backed by a cited artifact, and every commitment points to a checkable deliverable (a pull request, document version, or bug fix).
  • Well-posed gold answers. No gold set is empty, no two question types ask for the same answer, and no two gold rows are so similar that a correct prediction could be matched to the wrong one.
  • No answer leakage. The question and its response template never reveal internal identifiers, decoy artifacts, or gold values.

All 30 products pass all 19 validation gates, with zero failures. The rules also drop a record whenever the corpus does not support it, rather than guess. For example, a spoken promise is kept as a commitment only if it has an identifiable speaker and a checkable target. This trades coverage for label precision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.