acceptodds
Under review as a conference paper at ICLR 2027

PersonalArc: Harnesses at Both Ends of Agent Learning

Abstract

Personal-context agents must answer from task-scoped evidence while learning from prior executions. This creates two linked challenges: controlling how evidence enters a training trace, and deciding whether a learned answer should replace the base answer. We introduce PersonalArc, a framework that places restricted harnesses at both ends of agent learning. A task-conditioned harness declares stage inputs, artifact types, tool scopes, and resource limits before execution. Workspace-grounded mutations are audited and distilled into content-only stage supervision. At inference time, a back-end fusion harness governs evidence access, rubric-blind composition, and final-answer selection. It adopts the fused candidate when both comparison orders prefer it and numeric retention passes, returning the base answer exactly on rejection. Together, the two harnesses link experience acquisition and answer deployment, turning audited executions into reusable supervision while keeping final-answer adoption under explicit control. We evaluate the full evaluation sets of four benchmarks, with three generating backbones in the main comparison. Averaged across these backbones, PersonalArc improves Yes Rate and Coverage by 27.0% and 18.2% relative to Base Search, respectively, while increasing F1 by 6.77 points. It also outperforms the evaluated aggregation baselines at every tested token ceiling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Personal-Context Agents, Agent Learning, Harness Engineering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5M4rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

PersonalArc addresses how personal-context agents learn from prior executions while deciding when a learned answer should replace the Base answer. Its BUILD stage defines task-scoped evidence and execution boundaries; LEARN audits workspace-grounded mutations and turns executions into content-only stage supervision; FUSE adopts a fused answer only when both comparison orders favor it and numeric retention passes. The paper evaluates four benchmarks with three main generating backbones and reports improvements over Base Search, including a 6.77-point average F1 gain on LongMemEval-S. These results measure benchmark-informed adaptation: synthetic supervision uses references from the full evaluation sets, and evaluation is on the original tasks.

Critical strengths
  1. A clear connection between learning and deployment. PersonalArc uses explicit evidence boundaries for training traces and a defined rule for adopting a fused answer or returning Base verbatim. This makes the proposed system more concrete and auditable than an unconstrained learn-and-revise pipeline.

  2. Consistent gains in the reported setting. The paper evaluates four benchmarks with three main generating backbones and reports improvements over Base Search, including a 6.77-point average F1 gain on LongMemEval-S. This supports the method’s usefulness for benchmark-informed adaptation, though not yet for unseen tasks.

Critical weaknesses
  1. The student's contribution is not clearly established.

    • Figure 1 is crowded and does not make the relationship between BUILD, LEARN, FUSE, Base, and student easy to follow.
    • More importantly, Table 2 reports a 0.25-point gain in Orig. Coverage for Mutation stage SFT over Base Search, compared with 1.61 points for the full system. The experiments do not isolate the student's contribution while holding fusion and selection fixed. This makes it difficult to assess the paper's central claim that audited supervision improves final-answer adoption.
  2. The τ-bench evaluation protocol needs clarification.

    • τ-bench evaluates the final database state as well as the response. The paper states that Base and student execute in isolated environments before fusion, but does not specify which action trace and database state are evaluated when Fusion is adopted or Base is returned verbatim.
    • Without this explanation, the reported pass1/pass4 gains are difficult to connect to the proposed final-answer selection mechanism on tool-using tasks.
Questions for authors
  1. What does the student add beyond Base? Table 2 shows a small gain from Mutation stage SFT relative to Base Search, while the full system gains more. Can you report Base and student answer quality on the same tasks, and compare their use under the same fusion and selection procedure? This would help determine whether the gain comes from learned supervision or primarily from fusion and selection.

  2. How is τ-bench scored after fusion? Base and student execute in isolated environments, but τ-bench scores the final database state as well as the response. When Fusion is adopted or Base is returned, which execution's actions and database state are used for pass1/pass4? This determines whether those results validate the proposed final-answer selection mechanism.

Reviewer_fyLCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PersonalArc, a framework for learning from agent executions and controlling the use of learned answers. It has three components: (i) a task-conditioned execution harness with restricted tools, typed artifacts, and evidence-access rules; (ii) synthetic task generation, auditing, and stage-level supervised fine-tuning; and (iii) an answer-fusion mechanism that combines base and fine-tuned model outputs and selects the fused answer using order-swapped LLM judgments and numeric-retention checks.

The authors evaluate the system on CL-bench Life, HippoCamp, τ-bench retail, and LongMemEval-S, using three generating backbones in the main comparison. They report consistent improvements over search, agentic, aggregation, and supervised-learning baselines.

The problem is relevant, and the implementation is reasonably comprehensive. My main concerns are the evaluation protocol, limited methodological novelty, and insufficient isolation of the proposed components. The training procedure uses information derived from the evaluation tasks, making it difficult to determine whether the gains reflect generalizable learning. The paper also combines several established techniques without clearly establishing the contribution of its unified harness framework.

Overall assessment

The paper addresses a relevant problem and presents a detailed integrated system with promising results. The authors evaluate several benchmarks and model backbones and provide useful component and cost analyses.

However, the experimental protocol is a major limitation. The student receives supervision derived from the evaluation questions, personal histories, reference answers, and rubrics before being tested on the original tasks. The reported improvements therefore do not establish generalization to unseen requests or personal contexts.

The methodological contribution is also unclear. The framework combines established approaches to harness generation, synthetic-data training, trajectory distillation, and answer fusion. The experiments do not sufficiently isolate the benefits of the proposed shared evidence contracts.

Finally, the fusion mechanism relies on LLM preference judgments and surface-form numeric retention. The paper does not adequately measure whether these checks prevent factual regressions or improve on simpler selection methods.

The results support the effectiveness of an integrated system under benchmark-informed adaptation. Stronger held-out evaluations and targeted component ablations are needed to establish the claimed contribution to reusable agent learning.

Critical strengths

Strengths

  1. The paper addresses relevant problems in personal-context agents: ensuring that training traces use authorized evidence and deciding when a learned answer should replace an existing answer.

  2. The implementation is detailed. The authors specify restricted execution graphs, artifact dependencies, tool scopes, validation procedures, auditing, and fallback mechanisms.

  3. The evaluation covers four benchmarks and three generating backbones. The reported improvements are meaningful. For example, Qwen3.6-27B HippoCamp accuracy increases from 44.06% to 55.94%, while LongMemEval-S accuracy increases from 57.20% to 63.40%.

  4. The paper includes component comparisons, token-budget analyses, repeated evaluations, and a second judge for HippoCamp. These provide useful evidence about performance under the authors' evaluation setup.

  5. The authors explicitly disclose their use of evaluation-derived supervision and describe the results as benchmark-informed adaptation.

Critical weaknesses

Weaknesses

1. The evaluation protocol does not establish generalization

This is my main concern.

Section 4.1 states that the authors construct synthetic supervision using the full evaluation sets, including their questions, personal files or histories, reference answers, and rubrics. They then evaluate the adapted models on the original tasks.

The synthetic mutations may change question wording or constraints, but they retain the same underlying workspaces. Answer-preserving mutations can retain the original answers and evidence. Constraint-changing mutations can expose closely related facts and relationships from the same personal histories.

Consequently, the model may learn information specific to the evaluation questions and their underlying sources.

The authors enforce task-level evidence isolation during execution. However, this does not address information acquired during fine-tuning. A model can retain facts from evaluation workspaces in its parameters even when the inference environment restricts direct file access.

The reported improvements could arise from several factors:

  • Learning reusable procedures for retrieval, reasoning, and task execution.
  • Learning factual information specific to the evaluation workspaces.
  • Adapting to question types and answer patterns derived from the evaluation set.
  • Combining the base model's answer with a student already trained on related evaluation content.

The experiments do not distinguish these explanations.

Training on previous interactions from the same personal workspace is a reasonable setting for personal agents. However, the evaluation should then measure performance on new questions that were excluded from training and synthetic-task construction.

The current protocol does not establish this separation. It also does not test whether the learned procedures transfer to entirely new users or workspaces.

This affects the baseline comparisons. Generic methods such as ReAct, Self-Consistency, and Mixture-of-Agents do not receive equivalent reference-informed adaptation. Their comparison with PersonalArc measures end-to-end system performance under different training conditions. It provides limited evidence about the benefit of the proposed harness itself.

I would expect two evaluations:

  • Held-out questions: Train on earlier tasks from a workspace and evaluate on new questions, excluding their answers, rubrics, and synthetic derivatives from training.
  • Held-out workspaces: Train on separate personal histories and evaluate on entirely unseen users or task sources.

The distinction matters because the paper's central claim concerns learning reusable behavior from agent experience. The current results establish improvements under same-source, reference-informed adaptation. They do not show how much of that improvement transfers to genuinely unseen tasks.

2. Limited methodological novelty relative to existing work

The framework combines several ideas already explored in the literature.

Front-end harness. AutoHarness, Meta-Harness, Self-Harness, and Agentic Harness Engineering already study executable harness generation, execution constraints, workflow adaptation, and learning from execution feedback.

PersonalArc uses a restricted DSL, typed stage artifacts, declared dependencies, tool scopes, and runtime validation. These are reasonable engineering choices. However, the paper does not clearly identify which limitation of prior harness methods requires this particular contract structure.

Learning stage. The use of teacher executions, synthetic task variation, successful-trajectory selection, and supervised distillation has substantial precedent in agent-training research.

PersonalArc's potentially distinctive feature is its construction of content-only stage supervision using the evidence and artifacts visible at each execution stage. This could improve faithfulness or generalization by aligning training inputs with deployment conditions.

However, the paper does not directly test whether this approach is better than conventional trajectory-level supervised fine-tuning using the same teacher executions.

Back-end fusion. Candidate ranking, answer refinement, and generative fusion have already been explored, particularly in LLM-Blender and related work. Presentation-order bias in LLM judgments is also a known issue.

PersonalArc adds a specific selection rule based on agreement across both comparison orders, numeric retention, and deterministic fallback to Base. The paper does not establish that this rule provides a significant improvement over simpler selection or verification methods.

The strongest potential contribution is the shared contract framework linking training-trace acquisition and final-answer deployment. However, the authors need to establish what benefit arises specifically from this integration.

As presented, the novelty lies mainly in combining and formalizing existing mechanisms. The empirical evidence does not establish a sufficiently distinct contribution from the unified framework.

3. The ablations do not isolate the proposed mechanisms

The authors provide several component comparisons, including original-stage SFT, mutation-stage SFT, terminal-repair SFT, fusion, two-way selection, and the complete system.

These comparisons are useful, but several important effects remain confounded.

For example, on Qwen3.6-27B LongMemEval-S, accuracy increases from 57.20% with Base Search to 61.20% with Mutation SFT, and then to 63.40% with PersonalArc.

Mutation-based supervision therefore accounts for a substantial part of the total gain. The full system adds 2.20 percentage points over Mutation SFT.

The result supports the usefulness of the combined configuration. It does not establish how much of the improvement comes from the restricted harness, stage-aligned supervision, auditing, fusion, or selection.

The following experiments are particularly important:

Restricted harness versus conventional workflow. Use the same model, tools, training data, and execution budget. Compare the proposed typed evidence-contract harness against a simpler workflow without those additional constraints. This would test the contribution of the front-end harness.

Content-only stage SFT versus trajectory SFT. Train on the same successful teacher executions, with matched supervision volume. This would establish whether the stage-visible evidence interface provides an advantage.

Auditing ablation. Compare independently audited training traces against traces selected using simpler validity and success filters. This would establish the value of the proposed audit.

Fixed-candidate fusion comparison. Use exactly the same Base and student answers while varying only the fusion and selection strategy. This would isolate the back-end mechanism.

The matched-control experiment does address some training-budget differences. Equation 9 matches source sets, training-row counts, optimizer updates, and approximately matched loss-token volumes.

However, these controls do not make the training examples equivalent in their information content or degree of overlap with evaluation answers.

Appendix B.2 also reports only 29 training rows from six sources for each HippoCamp control/mutation corpus. This is a small adaptation set. The paper should explain how these retained corpora relate to the 1,149 synthesized HippoCamp tasks reported in Table 9.

The task-generation, filtering, trace-selection, and training pipeline needs clearer accounting.

4. The fusion and selection mechanism is not adequately validated

The back-end harness combines Base and student answers, generates a fused candidate, and selects it when two checks pass:

  1. An LLM comparator prefers Fusion over Base in both presentation orders.
  2. Fusion retains a sufficient proportion of the numeric and date tokens present in Base.

Both checks have limitations.

Agreement between two presentation orders can reduce order sensitivity, but the same comparator may prefer an incorrect answer in both orders. LLM preferences can be influenced by completeness, fluency, verbosity, and other features unrelated to factual correctness.

The numeric-retention check is also a surface-level measure. Preserving dates and numbers does not establish that they remain associated with the correct events or entities.

For example, an answer could retain every date and monetary amount while reversing the chronology of transactions or the direction of a price change.

The authors acknowledge this problem in the receipt-based example in Section 4.5, where swapping dates changes the price trend while preserving the numerical values.

The default retention threshold is θ = 0.25. This permits a fused candidate to retain only one-quarter of the counted numeric/date items, provided the comparison checks also pass. The threshold sweep shows sensitivity of aggregate performance but does not establish the factual reliability of this choice or its calibration on independent data.

The deterministic fallback to Base is useful, but it only preserves Base when Fusion is rejected. An accepted fused answer may still introduce errors or remove correct details.

The paper should report the actual selection outcomes:

  • Number and proportion of fused candidates accepted and rejected.
  • Incorrect Base answers corrected by accepted Fusion.
  • Correct Base answers degraded by accepted Fusion.
  • Beneficial revisions rejected by the gate.
  • Harmful revisions accepted despite passing both checks.
  • The separate effects of order-swapped consensus and numeric retention.

These results should be compared against simpler pairwise selection and evidence-grounded verification methods using fixed candidate outputs.

The current experiments show that the overall selection configuration improves aggregate benchmark performance. They do not demonstrate that the proposed checks reliably detect factual regressions.

5. Robustness analysis does not address the main validity concerns

The paper evaluates several generating backbones, uses complete benchmark sets, includes token-budget sweeps, and reports repeated evaluations and judge sensitivity.

These are useful checks, but they are conducted within the same reference-informed adaptation setting.

Evaluating more models under this protocol does not establish generalization beyond the evaluation sources. Repeated evaluations on the same tasks also do not establish performance on unseen personal histories.

The HippoCamp judge comparison illustrates another concern.

Under the primary judge, PersonalArc obtains 325 correct answers versus 256 for Base. Under the second judge, the corresponding counts are 339 and 299.

The improvement persists, but its magnitude changes from 69 to 40 additional correct tasks. This suggests meaningful sensitivity to the evaluator.

The paper would benefit from independent human verification of a sample of improved and degraded answers, particularly because an LLM comparator is also part of the proposed inference mechanism.

The results should also include paired confidence intervals or suitable statistical tests. Where multiple tasks share personal histories or profiles, the analysis should account for source-level dependence.

Finally, the paper provides limited testing of challenging evidence conditions, including contradictory documents, missing information, swapped entities, incorrect temporal relationships, and cases where the learned student contradicts a correct base answer.

These cases are directly relevant to the claimed benefits of evidence contracts and conservative answer selection.

6. Efficiency claims need clearer cost accounting

The inference-cost experiments are useful, but the full computational cost of the framework is not reflected in the reported comparisons.

On CL-bench Life, PersonalArc uses 24.5k tokens and 44 seconds per task, compared with 15.2k tokens and 23 seconds for Base Search.

This represents approximately 61% more tokens and 91% more latency than Base.

PersonalArc is cheaper than Mixture-of-Agents at this operating point, which uses 27.1k tokens and 57 seconds. The token-budget comparisons also show favorable performance against the evaluated aggregation baselines.

However, the online measurements exclude synthetic-task generation, teacher execution, auditing, and adapter training.

These costs are relevant when assessing the value of the complete learning pipeline. Their importance depends on how frequently the system is retrained and how many subsequent tasks benefit from each adaptation.

The paper should report the offline cost and provide an amortized quality-cost comparison under reasonable deployment assumptions.

The evidence supports favorable inference-time trade-offs against certain aggregation baselines. It does not establish an overall computational advantage for the full framework.

Clarity and presentation

The paper is generally organized, and the BUILD–LEARN–FUSE structure helps explain the system. However, the amount of framework-specific terminology makes the actual technical contribution harder to identify.

Several equations formalize validation checks, routing decisions, and fallback rules. These are useful implementation specifications, but they do not establish theoretical guarantees about factual correctness, evidence faithfulness, or generalization.

The paper also switches between original CL-bench Life score summaries and separately defined strict Yes Rate and rubric-micro Coverage metrics. The authors explain the distinction, but the inconsistent reporting makes comparisons difficult. A single primary scoring convention would improve clarity.

The relationship between synthesized tasks, accepted traces, retained supervision rows, and training updates should also be reported more clearly.

The receipt-based case study is illustrative but provides limited evidence about the effectiveness of the proposed mechanism. Showing the actual Base, student, fused, and selected answers would make it possible to assess whether the harness corrected an observed failure.

I found the presentation unnecessarily complicated in places, but I do not see a basis for drawing conclusions about inappropriate AI use. The substantive concerns are methodological and empirical.

Questions for authors

Questions for the authors

  1. Can the authors provide question-disjoint and workspace-disjoint evaluations, excluding evaluation questions, answers, rubrics, and their derivatives from training?

  2. How much semantic and factual overlap exists between the accepted synthetic supervision and the original evaluation questions and answers?

  3. What improvement does the typed evidence-contract harness provide over a simpler task-conditioned workflow with matched tools, data, models, and budgets?

  4. Does content-only stage supervision outperform conventional trajectory SFT when trained on the same teacher executions?

  5. What is the contribution of independent auditing relative to simpler success and validity filtering?

  6. For fixed Base and student candidates, what are the acceptance rates, factual regression rates, and correction rates under the proposed selection rule? How do these compare with consensus-only, retention-only, and simpler evidence-grounded selection?

  7. How was θ = 0.25 selected? Does the threshold generalize to independent tasks and workspaces?

  8. How do the synthesized-task totals relate to the much smaller retained training corpora? Can the authors report the full filtering and selection breakdown?

  9. How does the quality-cost comparison change when offline synthesis, teacher execution, auditing, and training costs are included?

Reviewer_sunAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper explores various ways to improve RAG over personal data.

  1. They parse questions into a structured series of steps that the agent must run.

  2. They train on synthetic data generated from the test data.

  3. They ensemble predictions from their base model and trained model during inference.

They demonstrate results across various related benchmarks and using three base models.

Critical strengths
  1. The paper provides extensive coverage of related benchmarks and baselines.

  2. There is additionally extensive analysis, for example showing that PersonalArc is superior to self consistency and mixture of agents at similar token budgets. That being said, the approach has many components, so more analysis may be warranted. For instance, latency breakdown by component, particularly the harness generation step. Or whether fusion across two base models can match the quality of base + SFT.

  3. Using a pre-defined, albeit adaptively predicted, workflow for tasks can have benefits for monitoring or controllable adjustments, although it could be better justified via empirical results at least from the quality perspective.

Critical weaknesses
  1. The approach is quite expensive, requiring multiple model executions, generation of a structured workflow (aka harness), and generative answer ensembling. While the authors provide several useful ablations, it's unclear whether the performance improvements justify the additional inference cost. For instance, PersonalArc nearly doubles latency compared to Base Search (44s vs. 23s in Table 3). The paper would benefit from a more direct comparison of quality versus cost, including stronger single-agent baselines given equivalent inference budgets.

  2. The training protocol is potentially leaking information from test. All of the test set is used to construct the trained set. Instead, a cleaner setup would use a fraction of the test set, e.g. 5-10 examples (which should be removed from testing).

3a. There is limited analysis on whether the harness predictions are effective. For instance, does the harness frequently include the same steps across a variety of questions? Could a fixed harness across all steps match in quality?

3b. Additionally, the structured steps used seem quite rigid. Are they necessary, or would an instruction describe the workflow be sufficient? Would the method generalize to a setup using only instructions instead of the structured steps?

Questions for authors

You may be interested in "Conditional Generation with a Question-Answering Blueprint" (https://arxiv.org/abs/2207.00397) which designs workflow-like inference for general QA.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.