Weaknesses
1. The evaluation protocol does not establish generalization
This is my main concern.
Section 4.1 states that the authors construct synthetic supervision using the full evaluation sets, including their questions, personal files or histories, reference answers, and rubrics. They then evaluate the adapted models on the original tasks.
The synthetic mutations may change question wording or constraints, but they retain the same underlying workspaces. Answer-preserving mutations can retain the original answers and evidence. Constraint-changing mutations can expose closely related facts and relationships from the same personal histories.
Consequently, the model may learn information specific to the evaluation questions and their underlying sources.
The authors enforce task-level evidence isolation during execution. However, this does not address information acquired during fine-tuning. A model can retain facts from evaluation workspaces in its parameters even when the inference environment restricts direct file access.
The reported improvements could arise from several factors:
- Learning reusable procedures for retrieval, reasoning, and task execution.
- Learning factual information specific to the evaluation workspaces.
- Adapting to question types and answer patterns derived from the evaluation set.
- Combining the base model's answer with a student already trained on related evaluation content.
The experiments do not distinguish these explanations.
Training on previous interactions from the same personal workspace is a reasonable setting for personal agents. However, the evaluation should then measure performance on new questions that were excluded from training and synthetic-task construction.
The current protocol does not establish this separation. It also does not test whether the learned procedures transfer to entirely new users or workspaces.
This affects the baseline comparisons. Generic methods such as ReAct, Self-Consistency, and Mixture-of-Agents do not receive equivalent reference-informed adaptation. Their comparison with PersonalArc measures end-to-end system performance under different training conditions. It provides limited evidence about the benefit of the proposed harness itself.
I would expect two evaluations:
- Held-out questions: Train on earlier tasks from a workspace and evaluate on new questions, excluding their answers, rubrics, and synthetic derivatives from training.
- Held-out workspaces: Train on separate personal histories and evaluate on entirely unseen users or task sources.
The distinction matters because the paper's central claim concerns learning reusable behavior from agent experience. The current results establish improvements under same-source, reference-informed adaptation. They do not show how much of that improvement transfers to genuinely unseen tasks.
2. Limited methodological novelty relative to existing work
The framework combines several ideas already explored in the literature.
Front-end harness. AutoHarness, Meta-Harness, Self-Harness, and Agentic Harness Engineering already study executable harness generation, execution constraints, workflow adaptation, and learning from execution feedback.
PersonalArc uses a restricted DSL, typed stage artifacts, declared dependencies, tool scopes, and runtime validation. These are reasonable engineering choices. However, the paper does not clearly identify which limitation of prior harness methods requires this particular contract structure.
Learning stage. The use of teacher executions, synthetic task variation, successful-trajectory selection, and supervised distillation has substantial precedent in agent-training research.
PersonalArc's potentially distinctive feature is its construction of content-only stage supervision using the evidence and artifacts visible at each execution stage. This could improve faithfulness or generalization by aligning training inputs with deployment conditions.
However, the paper does not directly test whether this approach is better than conventional trajectory-level supervised fine-tuning using the same teacher executions.
Back-end fusion. Candidate ranking, answer refinement, and generative fusion have already been explored, particularly in LLM-Blender and related work. Presentation-order bias in LLM judgments is also a known issue.
PersonalArc adds a specific selection rule based on agreement across both comparison orders, numeric retention, and deterministic fallback to Base. The paper does not establish that this rule provides a significant improvement over simpler selection or verification methods.
The strongest potential contribution is the shared contract framework linking training-trace acquisition and final-answer deployment. However, the authors need to establish what benefit arises specifically from this integration.
As presented, the novelty lies mainly in combining and formalizing existing mechanisms. The empirical evidence does not establish a sufficiently distinct contribution from the unified framework.
3. The ablations do not isolate the proposed mechanisms
The authors provide several component comparisons, including original-stage SFT, mutation-stage SFT, terminal-repair SFT, fusion, two-way selection, and the complete system.
These comparisons are useful, but several important effects remain confounded.
For example, on Qwen3.6-27B LongMemEval-S, accuracy increases from 57.20% with Base Search to 61.20% with Mutation SFT, and then to 63.40% with PersonalArc.
Mutation-based supervision therefore accounts for a substantial part of the total gain. The full system adds 2.20 percentage points over Mutation SFT.
The result supports the usefulness of the combined configuration. It does not establish how much of the improvement comes from the restricted harness, stage-aligned supervision, auditing, fusion, or selection.
The following experiments are particularly important:
Restricted harness versus conventional workflow. Use the same model, tools, training data, and execution budget. Compare the proposed typed evidence-contract harness against a simpler workflow without those additional constraints. This would test the contribution of the front-end harness.
Content-only stage SFT versus trajectory SFT. Train on the same successful teacher executions, with matched supervision volume. This would establish whether the stage-visible evidence interface provides an advantage.
Auditing ablation. Compare independently audited training traces against traces selected using simpler validity and success filters. This would establish the value of the proposed audit.
Fixed-candidate fusion comparison. Use exactly the same Base and student answers while varying only the fusion and selection strategy. This would isolate the back-end mechanism.
The matched-control experiment does address some training-budget differences. Equation 9 matches source sets, training-row counts, optimizer updates, and approximately matched loss-token volumes.
However, these controls do not make the training examples equivalent in their information content or degree of overlap with evaluation answers.
Appendix B.2 also reports only 29 training rows from six sources for each HippoCamp control/mutation corpus. This is a small adaptation set. The paper should explain how these retained corpora relate to the 1,149 synthesized HippoCamp tasks reported in Table 9.
The task-generation, filtering, trace-selection, and training pipeline needs clearer accounting.
4. The fusion and selection mechanism is not adequately validated
The back-end harness combines Base and student answers, generates a fused candidate, and selects it when two checks pass:
- An LLM comparator prefers Fusion over Base in both presentation orders.
- Fusion retains a sufficient proportion of the numeric and date tokens present in Base.
Both checks have limitations.
Agreement between two presentation orders can reduce order sensitivity, but the same comparator may prefer an incorrect answer in both orders. LLM preferences can be influenced by completeness, fluency, verbosity, and other features unrelated to factual correctness.
The numeric-retention check is also a surface-level measure. Preserving dates and numbers does not establish that they remain associated with the correct events or entities.
For example, an answer could retain every date and monetary amount while reversing the chronology of transactions or the direction of a price change.
The authors acknowledge this problem in the receipt-based example in Section 4.5, where swapping dates changes the price trend while preserving the numerical values.
The default retention threshold is θ = 0.25. This permits a fused candidate to retain only one-quarter of the counted numeric/date items, provided the comparison checks also pass. The threshold sweep shows sensitivity of aggregate performance but does not establish the factual reliability of this choice or its calibration on independent data.
The deterministic fallback to Base is useful, but it only preserves Base when Fusion is rejected. An accepted fused answer may still introduce errors or remove correct details.
The paper should report the actual selection outcomes:
- Number and proportion of fused candidates accepted and rejected.
- Incorrect Base answers corrected by accepted Fusion.
- Correct Base answers degraded by accepted Fusion.
- Beneficial revisions rejected by the gate.
- Harmful revisions accepted despite passing both checks.
- The separate effects of order-swapped consensus and numeric retention.
These results should be compared against simpler pairwise selection and evidence-grounded verification methods using fixed candidate outputs.
The current experiments show that the overall selection configuration improves aggregate benchmark performance. They do not demonstrate that the proposed checks reliably detect factual regressions.
5. Robustness analysis does not address the main validity concerns
The paper evaluates several generating backbones, uses complete benchmark sets, includes token-budget sweeps, and reports repeated evaluations and judge sensitivity.
These are useful checks, but they are conducted within the same reference-informed adaptation setting.
Evaluating more models under this protocol does not establish generalization beyond the evaluation sources. Repeated evaluations on the same tasks also do not establish performance on unseen personal histories.
The HippoCamp judge comparison illustrates another concern.
Under the primary judge, PersonalArc obtains 325 correct answers versus 256 for Base. Under the second judge, the corresponding counts are 339 and 299.
The improvement persists, but its magnitude changes from 69 to 40 additional correct tasks. This suggests meaningful sensitivity to the evaluator.
The paper would benefit from independent human verification of a sample of improved and degraded answers, particularly because an LLM comparator is also part of the proposed inference mechanism.
The results should also include paired confidence intervals or suitable statistical tests. Where multiple tasks share personal histories or profiles, the analysis should account for source-level dependence.
Finally, the paper provides limited testing of challenging evidence conditions, including contradictory documents, missing information, swapped entities, incorrect temporal relationships, and cases where the learned student contradicts a correct base answer.
These cases are directly relevant to the claimed benefits of evidence contracts and conservative answer selection.
6. Efficiency claims need clearer cost accounting
The inference-cost experiments are useful, but the full computational cost of the framework is not reflected in the reported comparisons.
On CL-bench Life, PersonalArc uses 24.5k tokens and 44 seconds per task, compared with 15.2k tokens and 23 seconds for Base Search.
This represents approximately 61% more tokens and 91% more latency than Base.
PersonalArc is cheaper than Mixture-of-Agents at this operating point, which uses 27.1k tokens and 57 seconds. The token-budget comparisons also show favorable performance against the evaluated aggregation baselines.
However, the online measurements exclude synthetic-task generation, teacher execution, auditing, and adapter training.
These costs are relevant when assessing the value of the complete learning pipeline. Their importance depends on how frequently the system is retrained and how many subsequent tasks benefit from each adaptation.
The paper should report the offline cost and provide an amortized quality-cost comparison under reasonable deployment assumptions.
The evidence supports favorable inference-time trade-offs against certain aggregation baselines. It does not establish an overall computational advantage for the full framework.
Clarity and presentation
The paper is generally organized, and the BUILD–LEARN–FUSE structure helps explain the system. However, the amount of framework-specific terminology makes the actual technical contribution harder to identify.
Several equations formalize validation checks, routing decisions, and fallback rules. These are useful implementation specifications, but they do not establish theoretical guarantees about factual correctness, evidence faithfulness, or generalization.
The paper also switches between original CL-bench Life score summaries and separately defined strict Yes Rate and rubric-micro Coverage metrics. The authors explain the distinction, but the inconsistent reporting makes comparisons difficult. A single primary scoring convention would improve clarity.
The relationship between synthesized tasks, accepted traces, retained supervision rows, and training updates should also be reported more clearly.
The receipt-based case study is illustrative but provides limited evidence about the effectiveness of the proposed mechanism. Showing the actual Base, student, fused, and selected answers would make it possible to assess whether the harness corrected an observed failure.
I found the presentation unnecessarily complicated in places, but I do not see a basis for drawing conclusions about inappropriate AI use. The substantive concerns are methodological and empirical.