- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper formulates the challenge of agent adaptation in evolving enterprise settings as persistent project-state understanding and introduces PULSE, a benchmark of 1,573 questions over 30 project histories. The benchmark introduces three categories of questions, namely, open-state, project evolution and view gaps. This allowed failure attribution to be associated with the query category by the retrieval and memory agents. The in-depth analysis shows that current enterprise systems need substantial improvement in various aspects, specifically, the retrieval remains a major challenge though explicit project-state information substantially improves performance even when evidence is available. The work also shows that enterprise agents require temporally grounded state representations, although perspective-dependent reasoning and higher-level project synthesis remains a major challenge.
Critical strengths
-
Evaluating LLMs in evolving enterprise environments is an interesting and real-world problem. The authors attempt to formulate this challenge as persistent project-state understanding, targeting enterprise-agent evaluation over heterogeneous, evolving, and perspective-dependent work histories.
-
The benchmark PULSE is created in a structurally consistent way, containing 1,573 questions across 30 enterprise project histories, covering open loops, project evolution, and collaborator-specific information gaps. Its deterministic, time-indexed state construction preserves provenance links between source artifacts, state transitions, and reference answers. The structured creation of the benchmark enables systematic assessment of the agentic capabilities.
-
Controlled experiments in the paper is strong, providing good insights into retrieval-based agents, long-context evaluation, gold-evidence oracle, and explicit-state oracle. The authors show that the gaps in current enterprise systems lie mainly in retrieving the correct information, followed by project-state reconstruction, which is needed to adapt to changing environments
-
An interesting insight from the work is the distinction between global project state and the information documented as available to individual collaborators. The view-gap questions assess whether agents can identify missing updates, stale information, and differences between individual and team-level perspectives.
Critical weaknesses
-
There can be a potential bias when the answering model and the judge model come from the same family. To mitigate this bias, it is necessary to perform cross-family model evaluation and human evaluation on a selected sample.
-
The choice of categories created at different stages is not clear. For example, why did the author select only the specified six typed stores? How do these categories help PULSE? Are the categories mentioned in Figure 2 exhaustive?
-
The process of benchmark curation is hard to follow, especially source-linking, inducing project state snapshots, and documented view construction. I understood that the motivation for choosing them comes from the Related Work section, but how they were targeted is confusing.
-
The motivation behind choosing Open Loop, Evolution, and View Gaps as the categories for question generation is not clear.
Questions for authors
See Weakness Section.
3 comments
Response to Concerns on Pipeline Construction (Weakness 3) and Question Families Design (Weakness 4)
- Clarity of the construction pipeline (Weakness 3)
We agree that the pipeline is hard to show in one figure or one section. Each record type has its own rules for what opens it, what closes it, and who can be shown to know about it. Showing all of this in the main text would make it harder to read. We therefore split the explanation into two levels:
- A worked-example figure in Section 3 (new figure, Line 182, marked in green), for the intuition. It traces one commitment through all three steps. A chat message opens the commitment, and a later pull request closes it (source-linking). The commitment is open in the snapshot before the pull request and closed in the snapshot after it (snapshots). An employee who took part in the chat but never saw the pull request knows of the promise but not its closure, so it is stale in that employee's documented view (documented view).
- Detailed rules in the appendix, for completeness. A new appendix section, "A2. Record Lifecycle and View Definitions" (Line 742, marked in blue, since it is also Response item 1 to Reviewer gEpY), gives a table of how each record type is opened and closed, formal definitions of open state at a cutoff and of documented views, and a step-by-step trace of a commitment made in a meeting.
- Rationale for the three question families (Weakness 4)
PPSU asks what an agent must know to understand ongoing work. We organize this into three conceptually orthogonal dimensions:
- Open Loops (State): What remains unresolved at a given time? This captures a snapshot of project state and requires determining whether commitments or blockers have been closed.
- Evolution (Time): How has the project state changed over an interval? This captures the delta between two points in time, requiring tracking transitions such as supersession and reversal.
- View Gaps (Perspective): How does project state appear from different collaborators' perspectives? This requires projecting the same project state or its evolution onto partial, person-specific information.
Together, these dimensions capture three distinct challenges in understanding ongoing work: reconstructing the current state, tracking how it changes, and reasoning about how it is perceived differently. They also correspond to the three gaps in prior work discussed in §2: deep search over evidence, temporal tracking of atomic facts, and treating partial information merely as an access-control problem. Each question family isolates a capability that retrieval alone cannot provide.
Reply to authors' response
I thank the authors for the clarification, additional analysis provided above, and revising the paper. It has improved my understanding. The paper is a good contribution to the agentic benchmark with temporally evolving enterprise tasks. I will update my score.
Response to Same-Family Judge Bias (Weakness 1) and Why Six Typed Record Stores (Weakness 2)
We thank the reviewer for insightful questions. Below, we respond to each concern.
We acknowledge the potential for same-family evaluation bias and we ran a cross-family evaluation to check it. It is in a new appendix section, "C5. Cross-Family Judge Check" (Line 1279, marked in green). Moreover, only 2 of the 18 question types rely on LLM-based judging. The remaining 16 are evaluated using objective criteria. Thus, the potential same-family bias is limited to a small portion of our evaluation.
We re-judged E8 and E9 with Claude Sonnet 5.5, run through Claude Code. Because of time and API constraints, we covered the first five products (28 E8 and 30 E9 questions). Only the judge changes. The prompts, the parsing of verdicts and the scoring code are identical to the gpt-5.6-luna run. We judged five answer sets: the four GPT-answered conditions and a Claude Code (Opus 5.5) agent from our new answering-model pilot (see item 1 in our response to Reviewer F7SD).
E8 Judge score and E9 Paired F1 under each judge (first five products; only the judge changes):
Per-item agreement between the two judges (pooled over the five answer sets):
We considered a broad range of project management concepts, including dependencies, decisions, priorities, and deadlines. However, constrained by the available HERB data and the need for reliable labels, we retained only categories satisfying three criteria: (1) State-bearing: having an explicit lifecycle (e.g., open → discharged); (2) Deterministically extractable: having both creation and closure evidence recorded in HERB; and (3) Source-linked: having every state transition traceable to specific artifacts.
The resulting six stores cover complementary aspects of project state: what is produced (work items), what is promised (commitments), what blocks progress (blockers), changes of direction (reversions), external demand (customer bugs), and who knows what (visibility). While these stores do not exhaust all potentially interesting management concepts, they represent the categories that HERB can reliably support under our construction criteria.