acceptodds
Under review as a conference paper at ICLR 2027

Behavior Quotient Learning for Low-Rank Adaptation of LLM Agents

Abstract

LLM-based agents require diverse interaction capabilities to complete complex tasks. Existing approaches often assign these capabilities to multiple LoRA adapters, increasing storage requirements and routing overhead during inference. A single LoRA adapter simplifies deployment but poses two challenges under a fixed rank budget. First, different training trajectories may still reinforce the same behaviors, making part of the supervision redundant. Second, a low-rank approximation can closely match the desired parameter changes while distorting the intended effects on agent decisions. To address these challenges, we propose BQ-LoRA, which adapts LLM agents through two modules, i.e., behavior quotient balancing (BQB) and decision-preserving compression (DPC). BQB constructs a local quotient manifold from representations used to predict agent decisions. Trajectory gradients are then reweighted by their local density in the quotient tangent space to reduce the dominance of redundant behavioral effects. DPC projects the balanced direction onto the fixed-rank tangent space and compresses the resulting target to fit the original rank budget. This compression jointly controls parameter approximation error and first-order distortion of decision representations. Experimental results demonstrate the effectiveness of BQ-LoRA in improving agent performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model Agents, Low-Rank Adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TGEkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes BQ-LoRA, a framework for adapting LLM agents with a single fixed-rank LoRA adapter. It combines behavior quotient balancing (BQB), which compares projected trajectory updates through their first-order effects on decision representations at shared reference histories and downweights redundant responses, with decision preserving compression (DPC), which compresses the resulting update under the rank constraint while controlling both parameter approximation error and first-order representation distortion. The paper formalizes a local behavioral equivalence relation and provides bounds on representation error, with a connection to conditional token distributions at fixed reference prefixes. Evaluation uses Qwen3.5-4B and Qwen3.5-9B on AppWorld and BrowseComp-Plus, comparing against full fine-tuning, single- and multi-adapter methods, and trajectory curation. The reported results show improved task performance and fewer interactions, supported by component ablations, alternative weighting and compression comparisons, and sensitivity analyses. Resource measurements indicate inference costs close to standard LoRA, with increased training time and memory.

Critical strengths
  1. A coherent approach to behavior-aware training under a fixed rank budget. BQB and DPC use the same decision-response geometry to guide trajectory weighting and low-rank compression. This connects the treatment of redundant supervision with preservation of the resulting update effects, while retaining a single deployable adapter. This integration is consequential because it addresses a practical agent-adaptation problem without requiring capability-specific routing at inference.

  2. Consistent empirical improvements supported by relevant ablations. BQ-LoRA achieves the highest reported mean across all eight performance metrics for both backbone sizes. On AppWorld Test-C, TGC improves over the strongest corresponding baseline by 3.03 and 3.20 percentage points for the 4B and 9B models, respectively. Component ablations, together with comparisons against text/parameter-based weighting and SVD compression, provide substantive evidence that both proposed modules contribute to the gains. These results strengthen the case for practical value beyond standard LoRA optimization.

Critical weaknesses
  1. The implemented bandwidth rule does not adequately substantiate the balancing rationale. Appendix E.2 sets the bandwidth squared to the median strictly positive squared response distance. Consider a batch containing two groups of identical responses, with 60 and 4 trajectories respectively, separated by distance D>0D>0. This rule gives σ2=D2\sigma^2=D^2, so cross-group affinity remains e−1/2e^{-1/2}, regardless of separation. Even with α=1\alpha=1, the groups receive approximately 90.66% and 9.34% of the total weight, rather than approaching equal contributions. This does not contradict the conditional theoretical bounds, but exposes a gap between the bandwidth choice and the intended balancing behavior. Without controlled duplication experiments or diagnostics of actual affinities and weights, the paper leaves its central mechanism insufficiently validated.

  2. The incremental contribution over established gradient-based data selection remains insufficiently demonstrated. Prior work already uses gradient information to identify useful and redundant supervision. In particular, GTP explicitly performs joint selection to reduce redundancy and evaluates on ALFWorld agent data. Although Table 6 includes a parameter-distance weighting baseline, it does not establish an advantage over such established joint-selection methods. The decision-response metric and its integration with compression are identifiable distinctions, but stronger comparisons are needed to determine their additional value. This omission materially limits my confidence in the significance of the proposed contribution.

Questions for authors
  1. Clarification of the bandwidth rule. Is the median-positive-distance rule in Appendix E.2 the exact implementation, or are additional scaling or numerical safeguards used? For two internally identical response groups of unequal sizes, this rule keeps cross-group affinity at e−1/2e^{-1/2} regardless of separation. How do the authors reconcile this behavior with the balancing rationale, and do observed affinities and group weights demonstrate meaningful redundancy correction during training?

  2. Clarification of the weighting ablation. In Table 6, does “Parameter Weighting” use the same projected trajectory directions, normalization, density estimator, balancing exponent, and optimization procedure as BQB, changing only the distance metric? These details are necessary to determine whether the comparison isolates the contribution of the decision-response geometry.

  3. Evidence needed to assess the incremental contribution. How does BQB improve upon gradient-based joint-selection methods such as GTP, which already addresses redundancy and evaluates on agent data? Can the authors provide a comparison under comparable data and computational budgets, or explain any concrete obstacle to such a comparison?

6 comments

To W1

The default bandwidth discounts redundant responses. Equal total weight across groups requires the additional small-overlap condition in Section 4.3 and α=1\alpha=1. In the 60-to-4 example every positive distance equals DD. The rule therefore gives σT2=D2\sigma_T^2=D^2 and a cross-group affinity of e−1/2e^{-1/2} for every D>0D>0. The four trajectories account for 6.25% of the batch. Their weight increases to 8.46% at the 4B default α=0.75\alpha=0.75 and 9.34% at α=1\alpha=1, exactly as the calculation in the question predicts for the latter setting.

The same bandwidth rule behaves differently for tightly clustered responses that are distinct. For example, fix the distance between groups at 4. If each within-group distance is 1, then 1,776 of the 2,016 pairs are within-group. The median positive squared distance is now 1 and cross-group affinity falls to e−8e^{-8}. The table gives the share of total weight assigned to the four trajectories.

Within-group distanceσT\sigma_TCross-group affinityShare at α=0.75\alpha=0.75 (%)Share at α=1\alpha=1 (%)
040.6065318.469.34
110.00033531.2846.34

The relevant separation is the distance relative to the batch bandwidth. Exact duplicates contribute zero distances, which are excluded from the median that sets the bandwidth. The response geometry determines the amount of correction in each batch.

For a direct duplication test, we copied the update for a Spotify task about following classical artists, holding its 64 reference histories and the other 63 directions fixed. Freezing each original weight before renormalization controls for the fact that this trajectory already had a below-average weight. With eight copies, the group receives 8.20% with frozen weights and 7.70% after densities are recomputed. The additional 0.50-point reduction isolates the response to duplication itself. The total reduction relative to uniform weighting is 3.57 points.

CopiesDirectionsUniform, %Frozen weights, %Recomputed BQB, %
1641.561.101.10
2653.082.182.16
4675.974.284.15
87111.278.207.70

To W2

GTP already addresses redundancy by jointly selecting training examples from gradient information and evaluates that procedure on ALFWorld. The additional contribution of BQ-LoRA is to measure overlap by the decision changes induced by an update, then use the same metric to preserve those changes under rank compression. GTP makes a subset decision for the student using teacher-gradient snapshots. BQB revisits the weights at each update as the policy changes, and DPC retains the balanced update in a single adapter.

A concrete AppWorld comparison illustrates the distinction. From 576 candidates, joint GTP selection retained a vacation-photo sorting trajectory and omitted a Spotify library cleanup trajectory. At the first BQ update, both trajectories contributed, with weights of 0.795 and 0.853 on a scale with mean one. Their projected gradients had cosine similarity 0.253, while their effects on the same reference decisions had cosine similarity 0.971. The example shows the information available to BQB when it weights the two records. The aggregate comparison below tests the value of the complete method.

For each backbone, both methods start from the paper training pools of 642 AppWorld trajectories and 668 BrowseComp-Plus trajectories. GTP jointly selects half the candidates for a rank-32 LoRA student. The student and the full-pool methods receive the same number of optimizer updates. The AppWorld columns report task completion and the retrieval columns report answer accuracy, all in percent.

BackboneMethodTest-NTest-CBM25Embedding
4BLoRA52.9835.4920.8940.22
4BGTP59.1342.3728.0045.11
4BBQ-LoRA60.9144.0429.1146.44
9BLoRA60.7145.0826.2246.22
9BGTP68.8551.7232.0050.67
9BBQ-LoRA70.4453.3233.1151.78

GTP improves on full-data LoRA in all four columns for each backbone. BQ-LoRA adds 1.67 Test-C points on 4B and 1.60 on 9B over that gradient-based baseline. These differences compare the complete methods at matched student updates. Table 6 provides the complementary test of the weighting metric alone.

To Q1

Appendix E.2 gives the training rule exactly. In each batch we take the interpolated median of strictly positive squared response distances over unordered pairs to obtain σT2\sigma_T^2. We use a bandwidth multiplier of one. The diagonal and zero distances are excluded from the median, while self-affinity and the affinity of identical responses both contribute one to the density. We raise density to the power −α-\alpha and normalize the weights to mean one. There is no bandwidth floor or weight clipping. If all distances vanish, all trajectories receive weight one.

In the two-group example, increasing DD increases σT\sigma_T in the same proportion. Cross-group affinity therefore stays at e−1/2e^{-1/2}. This explains why the 4B default moves the minority share from 6.25% to 8.46%. For training data, the tails of the affinity distribution are more informative than its median, which is largely set by the bandwidth rule. The table shows the off-diagonal 10th and 90th percentiles. Effective size is the number of equally weighted trajectories that would give the same weight concentration.

UpdateAffinity P10Affinity P90Max weightEffective size / 64
40.0950.9149.1429.77
50.3050.7951.8561.27
60.1710.8269.9617.68
70.2720.82411.9721.54
80.1090.8407.0335.19
90.0380.90911.0523.34

The default gives nearly uniform weights on some updates and substantial reweighting on others. Effective size was 61.27 of 64 at step 5 and 17.68 at step 6. To check the direction of the correction, we also duplicated a single Spotify update while keeping the references fixed. Eight copies contributed 11.27% of the expanded batch by count. Their aggregate weight was 8.20% with the original weights frozen and 7.70% with densities recomputed. The training statistics establish when weights concentrate, and the controlled copies identify a reduction specifically attributable to redundancy.

Reviewer_sZnZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies behavioral redundancy and low-rank compression distortion when fine-tuning LLM agents with a single fixed-rank LoRA adapter. The authors propose BQ-LoRA, which uses behavior quotient balancing (BQB) to adjust trajectory weights based on their effects on decision representations. Decision preserving compression (DPC) then attempts to retain these effects during low-rank compression. Experiments are conducted on Qwen3.5-4B and 9B using AppWorld and BrowseComp-Plus to evaluate task performance and interaction counts. The results show some improvements, but the practical training cost remains unclear. In particular, the computation of shared reference responses, the handling of optimizer states, and the substantial increase in training memory need further explanation.

Critical strengths
  1. Measuring redundancy through the actual effects of training updates on model behavior seems more relevant than simply comparing trajectory text. This offers a different perspective from common trajectory selection methods;
  2. BQB and DPC both focus on changes in decision representations. One adjusts the contributions of different updates, while the other tries to preserve these changes during compression. The two components fit together reasonably well. The tangent space projection also makes the update independent of the particular LoRA factorization;
  3. The additional computation is confined to training, and inference still uses a single LoRA adapter. The reported inference memory and decoding latency are close to those of standard LoRA.
Critical weaknesses
  1. BQB requires evaluating each trajectory's update direction on the reference decisions from the entire batch. With a batch size of 64, this seems computationally expensive. The authors mention reusing gradients and intermediate quantities from linearization, but provide few implementation details. On AppWorld, training time per step increases from 3.91 seconds for standard LoRA to 6.62 seconds for BQ-LoRA. It is hard to tell from the current description how the method achieves this relatively modest overhead;
  2. The algorithm refactorizes the updated matrix into new LoRA factors at every step, but the optimizer used in practice is not specified. If AdamW is used, the existing momentum and variance estimates may no longer align with the new factor coordinates. This makes the method harder to reproduce and raises questions about whether the implementation follows the update procedure described in the paper;
  3. A considerable part of the paper is devoted to constructing a local quotient manifold, yet the actual behavioral distances are computed directly using Jacobian-vector products. Explicit quotient manifold coordinates are not needed for this computation. It is therefore unclear how much the full geometric construction contributes to the algorithm. The constant rank condition is also assumed without discussing whether it holds during training;
  4. BQ-LoRA reaches 48.4 GiB of peak training memory on BrowseComp-Plus, compared with 16.6 GiB for standard LoRA and 38.4 GiB for MoRAgent. Such a large memory overhead deserves more attention than a figure in the appendix. For settings with limited training resources, the current experiments do not make it clear whether the performance gains justify this additional cost.
Questions for authors
  1. Is AdamW or SGD used in the actual training? If AdamW is used, what happens to the momentum and variance estimates when the LoRA factors are refactorized at each step?
  2. What accounts for the additional 32 GiB of training memory on BrowseComp-Plus? Does most of it come from per-trajectory gradients, intermediate activations for reference decisions, or response vectors and their caches? Could the authors provide a breakdown of the memory usage?
10 comments

To W1

Equation 7 evaluates 64 directions on 64 shared histories using 64 trajectory gradients. Each response includes the coupled effects of perturbations through successive Transformer layers. The first two full-reference 4B updates covered all 4,096 combinations. Figure 3(a) reports 3.91 seconds per AppWorld update for LoRA and 6.62 seconds for BQ-LoRA on Qwen3.5-4B. The pairs share computation rather than requiring 4,096 separate forward calls.

Reuse each trajectory gradient for both weighting and updating. The gradients of the two LoRA factors determine the projected direction in low-rank form. Constructing that direction requires no dense gradient for the corresponding backbone weight matrix. Once the 64 weights are known, linearity lets us combine the same factor gradients to form the update target. A second loss backward pass over the training batch is unnecessary.

Reuse common prefixes and batch the directions. Consider four contexts with an identical 4,096-token prefix and different 512-token suffixes. Processing them independently visits 18,432 token positions per direction. Computing the prefix once and branching at the first differing token reduces this to 6,144 positions per direction. Exact reuse requires identical tokens, positions and masks at fixed model weights in a deterministic forward pass. Each branch inherits both the model state and its directional derivative, including the attention or recurrent state. Each occurrence retains its original weight in Equation 5. On any given context, a block of eight directions can share the ordinary forward computation while propagating eight distinct perturbations. Thus eight vectorized calls cover 64 directions on that context. A held-out 4B check gave a relative difference of 5.89×10−65.89 \times 10^{-6} between single-direction and batched FP32 derivatives.

Communicate distance totals and keep the hidden responses local. Prefix groups can be distributed by token workload across 8 NVIDIA H20 GPUs. The response pass reads decision hidden states directly, skipping the vocabulary projection. Each GPU accumulates squared distances in blocks of hidden coordinates with the normalization from Equation 5. Summing these local totals gives Equation 7. At batch size 64, the upper triangle of the distance matrix contains 2,080 values including the diagonal, a payload of 16.25 KiB in FP64 per device before collective overhead. This bounds communication for the distance reduction. Blocking also limits peak response storage.

Reuse the target bases throughout compression. At adapter rank 32, each layer target has rank at most 64. Building its bases once confines subsequent SVDs to cores of at most 64×6464 \times 64. The decision penalty still uses the joint full-model derivative. Backtracking trials need only objective values, so they can be scored in small groups, stopping at the first acceptable group. The adjoint is needed only when forming a new solver gradient.

The shared histories also preserve directional information. For the vacation-photo and Spotify cleanup tasks, the response cosine was 0.971, and the squared distance was 4.13% of the sum of the squared norms. Computing each response only on its own history would give 100%.

To W2

AdamW is used only for factor warmup. We discard both moment buffers before the first geometric update. This timing matters because the factor coordinates can change even when the represented matrix stays fixed. For example, simultaneously replacing BB and AA by −B-B and −A-A leaves BABA unchanged but reverses the factor coordinates in which the first moment was recorded. Retaining that moment across refactorization would require an explicit change of coordinates.

After warmup, momentum-free Riemannian SGD updates the effective adaptation matrix. DPC compresses the projected target to the prescribed rank. The SVD factors store that result for the next forward pass, and the next batch supplies fresh supervised gradients. The following 4B AppWorld checkpoints count how many of the 496 factor tensors, two for each of 248 adapted layers, carry AdamW state.

StepFactor tensorsTensors with AdamW state
3496496
44960
94960

Step 3 ends warmup. At step 4, the first geometric update, all 496 factors remain and the optimizer state is empty. It is still empty at step 9. We will state the optimizer switch immediately after the warmup line in Algorithm 1 so that the state used at the SVD step is explicit.

To W3

The quotient construction specifies which update differences can be discarded without changing the first-order response on the reference decisions. For a fixed batch, let JX=DΦT(X)J_X=D\Phi_{\mathcal T}(X). Two updates in the same tangent space induce identical responses precisely when their difference lies in ker⁡JX\ker J_X. BQB uses that equivalence to measure redundancy. DPC uses the same principle to penalize a compression residual in proportion to the response it changes. A Jacobian-vector product evaluates this criterion directly, so Algorithm 1 needs no explicit quotient coordinates.

Local constancy of the Jacobian rank is an assumption for the smooth quotient interpretation. Training evaluates the differential at the current iterate. If its rank changes, the kernel can change dimension, and the response distances and DPC penalty are still defined by the current differential. The tangent projection requires the effective LoRA matrices to have their prescribed ranks. We will distinguish this adapter rank condition from the local assumption on the Jacobian in Section 4.3.

We will write Equation 7 using the projected directions U~i=Proj⁡X(Ui)\widetilde U_i=\operatorname{Proj}_X(U_i), consistently with Section 4.4 and Equation 25. The distance used for weighting is:

dij2=∥JX[U~i−U~j]∥22.d_{ij}^2 = \left\|J_X\left[\widetilde U_i-\widetilde U_j\right]\right\|_2^2 .

We will add the two intermediate steps to the proof of Proposition 5 in Appendix C.4. For Equation 55, choose indices ii and jj at which uu attains its maximum and minimum. With eie_i and eje_j the corresponding coordinate vectors, the inner product bound gives:

max⁡kuk−min⁡kuk=⟨ei−ej,u⟩≤∥ei−ej∥2∥u∥2≤2∥u∥2.\max_k u_k-\min_k u_k=\langle e_i-e_j,u\rangle\leq\|e_i-e_j\|_2\|u\|_2\leq\sqrt{2}\|u\|_2 .

For A(z)=log⁡∑kezkA(z)=\log\sum_k e^{z_k}, the Hessian quadratic form in direction uu is the variance of uku_k under p=softmax⁡(z)p=\operatorname{softmax}(z). Centering at the midpoint of the range bounds that variance by one quarter of the squared range, hence by ∥u∥22/2\|u\|_2^2/2. For Equation 56, set δ=z′−z\delta=z'-z and p′=softmax⁡(z′)p'=\operatorname{softmax}(z'). The integral Taylor remainder gives:

=\int_0^1(1-t)\,\delta^\top\nabla^2 A(z+t\delta)\delta\,dt \leq\tfrac14\|\delta\|_2^2 .$$
Reviewer_BDLzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents BQ-LoRA for adapting LLM agents with a single fixed-rank adapter. Behavior Quotient Balancing reweights trajectory updates according to their first-order effects on decision representations, while Decision Preserving Compression incorporates representation distortion into low-rank compression. Experiments on AppWorld and BrowseComp-Plus benchmarks demonstrate improved task performance and fewer interactions, while maintaining decoding latency close to standard LoRA.

Critical strengths
  1. The key idea of the proposed BQ-LoRA is reasonable. Comparing the effects of training updates rather than trajectory content provides a useful perspective on behavioral redundancy. Using the same representation geometry for weighting and compression gives the method a coherent design.

  2. The experiments include relevant optimization, multi-adapter, and trajectory-curation baselines.

  3. The deployment benefit is concrete: the proposed method retains a single adapter without additional routing. The reported cost analysis also makes the extra training overhead explicit.

Critical weaknesses
  1. The connection between representation preservation and actual agent behavior is not sufficiently validated in the main text. The behavior map uses fixed histories and recorded reasoning, whereas deployment involves model-generated reasoning and changing histories. Preserving this local surrogate does not directly establish preservation of action choices or task outcomes.

  2. The main-text ablations support the two modules but do not isolate their distinctive mechanisms. Removing DPC does not distinguish the benefit of tangent projection from that of the decision-distortion term. Likewise, BQB is not compared with simpler weighting based on parameter-gradient similarity.

  3. Behavioral rarity is not necessarily training utility. Inverse-density weighting can increase the influence of unusual updates even when their effects are unhelpful. Successful source trajectories alone do not establish that every distinct update deserves greater weight.

  4. While the motivation is clear, the technical presentation is difficult to follow without consulting the appendix. Theorem 1 leaves key quantities and assumptions outside the main text, and Equation (13) specifies an optimization objective without a concise explanation of the actual solver.

Questions for authors
  1. How well do the proposed response distances predict changes in action distributions or task outcomes on held-out histories? Could you provide a concrete example where dissimilar trajectories produce equivalent behavioral updates?

  2. Could you highlight controlled comparisons against gradient-similarity weighting and tangent projection followed by ordinary truncated SVD (γ=0\gamma=0)? These would clarify the contribution of the behavior geometry.

  3. How are reference histories and the bandwidth σT\sigma_T selected and refreshed? How sensitive is BQB to reference coverage and isolated, high-norm trajectory updates?

  4. Does the advantage persist across smaller and larger ranks, and under matched total training time? Please also clarify the total adapter parameter counts for the multi-LoRA baselines.

10 comments

To W1

AppWorld evaluates each agent on its own reasoning and the observations returned by its actions. Equation 5 fixes the prefix to compare two updates at the same decision point. For example, one Spotify task was to queue every song by Astrid Nightshade with more than 980 plays. After the first 4B update, DPC made three song queries, added the qualifying songs and finished in 15 turns. SVD made 14 song queries and reached the 50-turn limit without adding any songs.

The final-update comparison uses the same uncompressed rank-64 target for DPC and truncated SVD, with both adapters compressed to rank 32. For each task, we select one target-generated history before the current assistant turn, independently of task success. Each model generates its own reasoning and Python action. Action agreement counts identical valid programs after parsing. In a separate rollout from the initial prompt, each model executes its actions against the environment. Success retention is the fraction of tasks solved by the target that the compressed agent also solves. TGC is the fraction of all tasks solved.

The table covers all 57 Dev, 168 Test-N and 417 Test-C tasks. Values are percentages, summarized across three paired runs.

SplitMethodAction agreementSuccess retentionTGC
DevSVD77.19±3.5177.19 \pm 3.5174.59±3.2874.59 \pm 3.2839.77±2.6839.77 \pm 2.68
DevDPC88.89±2.6888.89 \pm 2.6893.32±2.3493.32 \pm 2.3443.86±1.7543.86 \pm 1.75
Test-NSVD81.55±2.7381.55 \pm 2.7385.28±1.7485.28 \pm 1.7458.33±1.1958.33 \pm 1.19
Test-NDPC90.28±2.0990.28 \pm 2.0995.19±1.0695.19 \pm 1.0660.91±1.2460.91 \pm 1.24
Test-CSVD78.82±2.0478.82 \pm 2.0482.91±1.6282.91 \pm 1.6241.73±1.2041.73 \pm 1.20
Test-CDPC89.53±1.6889.53 \pm 1.6894.71±1.1494.71 \pm 1.1444.04±1.0844.04 \pm 1.08

On Test-C, DPC preserves more of the target behavior even when every agent follows its own interaction history. With SVD normalized to one, mean conditional token KL on the same histories is 0.69±0.030.69 \pm 0.03 for DPC. In the action forks, agreement rises from 78.82% to 89.53%. Across the three complete Test-C rollouts per task, the target succeeds on 568 task instances. DPC retains 538 of those successes, compared with 471 for SVD. The smaller gap in overall TGC is consistent with these counts because SVD also solves more tasks on which the target fails. Retention directly tests whether a task that succeeded before compression still succeeds after the agent generates and executes its own actions.

To W2

Appendix E.4 already includes separate ablations for trajectory weighting and rank compression. Table 6 varies the distance used to weight trajectories while keeping the projected gradients and density rule fixed. In Table 7, all compression methods use the same tangent projection. The identity-metric control also uses the same solver and iteration budget as DPC, allowing us to isolate the contribution of the decision metric. The table below reports 4B Test-C task completion and BrowseComp-Plus retrieval results in percent. Accuracy scores the final answer. Recall measures the fraction of annotated evidence retrieved per question.

ControlVariantTest-CBM25 acc.Emb. acc.BM25 recall
WeightingParameter distance38.13±1.2038.13 \pm 1.2024.00±1.7624.00 \pm 1.7642.00±2.0042.00 \pm 2.0026.70±1.5426.70 \pm 1.54
WeightingResponse distance41.33±0.9741.33 \pm 0.9727.56±2.0427.56 \pm 2.0444.22±2.1444.22 \pm 2.1428.00±2.1028.00 \pm 2.10
CompressionProjected SVD39.09±1.1039.09 \pm 1.1025.33±2.0025.33 \pm 2.0042.67±2.0042.67 \pm 2.0025.86±1.9025.86 \pm 1.90
CompressionIdentity metric39.09±1.1039.09 \pm 1.1025.33±1.7625.33 \pm 1.7642.67±1.7642.67 \pm 1.7625.76±1.8025.76 \pm 1.80
CompressionDecision metric40.77±1.1040.77 \pm 1.1027.11±1.6827.11 \pm 1.6843.56±1.3943.56 \pm 1.3927.40±1.8027.40 \pm 1.80

The identity-metric solver controls for extra optimization steps. Its objective is proportional to squared Frobenius error, which truncated SVD already minimizes. The two controls agree on task completion and answer accuracy. BM25 evidence recall differs by just 0.10 points. With the decision metric, Test-C improves by 1.68 points over either control and BM25 answer accuracy by 1.78 points. In the separate weighting comparison, using response distance adds 3.20 Test-C points over parameter distance. Each gain is computed within its own ablation.

For example, one Spotify task was to download tracks that were both liked and in the song library. With DPC fixed, response weighting issued seven download requests and passed. Parameter weighting issued 17 and failed. Across all 57 Dev tasks, eight succeeded only with response weighting and two only with parameter weighting.

To W3

BQB assigns more weight to trajectories whose effects on decisions are underrepresented in the batch. We test whether this extra weight helps on held-out tasks by doubling the weight of one trajectory at a time. At each of six checkpoints, we pair eight high-weight trajectories with eight typical-weight trajectories of comparable gradient norm. In each comparison, the other 63 weights stay at one before normalization. We fix the aggregate step norm at 0.03 and use the same DPC solver. Doubling the weight lowers validation loss in 38 of 48 high-weight cases, compared with 25 of 48 matched controls. Gain is the reduction in mean negative log-likelihood on the same 64 validation records.

Selected directionLower loss / 48Mean gain (10−410^{-4})
High weight38+2.7+2.7
Typical weight25+0.4+0.4
High weight, reversed10−2.2-2.2

Reversing the high-weight direction turns the mean gain negative. For a concrete example, a Spotify task asks for the most-played song by Velvet Echo. Sorting by descending play count gives the requested song. Sorting in ascending order rewards the opposite choice. To test this failure mode at task level, a frozen warmup model selects low-density candidates before corruption. All methods receive the same incorrect action labels at the selected turns, retain clean reference histories and use DPC. The shuffled control keeps the same set of weights as BQB but redistributes them among trajectories of similar gradient norm.

Incorrect labels / batchUniform + DPCShuffled + DPCBQB + DPC
0/6440.77±1.1040.77 \pm 1.1041.01±1.2041.01 \pm 1.2044.04±1.0844.04 \pm 1.08
1/6440.37±1.1340.37 \pm 1.1340.61±1.2340.61 \pm 1.2343.25±1.1343.25 \pm 1.13
4/6439.65±1.1839.65 \pm 1.1839.81±1.2739.81 \pm 1.2741.01±1.2741.01 \pm 1.27

With clean supervision, BQB reaches 44.04% AppWorld Test-C completion, compared with 41.01% after shuffling and 40.77% with uniform weights. Applying each weight to its original trajectory improves completion by 3.03 percentage points over shuffling the same weights. The gain over uniform weighting is 3.27 points with clean supervision and 1.36 points with four incorrect labels per batch. The gain is largest when the distinctive trajectories carry correct supervision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.