- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes λ-RLM, a fixed recursive workflow for long-context tasks. It replaces model-generated control code with predefined operators for splitting inputs, processing parts, and combining results. Across five benchmarks and nine models, the paper reports accuracy gains, especially for smaller models, and a 4× average speedup. It also analyzes termination, model-call counts, and accuracy under stated assumptions.
Critical strengths
- The paper addresses a useful and interesting question: when does restricting a model's control improve reliability and efficiency?
- The experiments are comprehensive and cover nine different models with diverse tiers and behaviors. The paper reports accuracy gains, especially for smaller models, and a 4× average speedup. Table 7 supports the value of restricting control within the tested system.
- Fixed workflows make execution easier to inspect, and the paper acknowledges weaker gains on stronger models and open-ended tasks.
- The method works with existing model weights, so its gains do not require additional model training.
- The accuracy analysis distinguishes tasks that need every part correct from retrieval tasks that need one relevant part.
Critical weaknesses
- Some of the largest gains do not match how the method is described, so it is unclear what produced them. According to Algorithm 2, if the input fits in the model's context window, λ-RLM does not split it. Apart from a short task-type call, it sends the whole input to the model in one call, which is what the Direct baseline also does. So on inputs that fit, λ-RLM and Direct should get about the same score. The results show large gaps instead. On OOLONG-Pairs, for models with a 128K window, almost all inputs fit, yet Direct scores near zero while λ-RLM scores around 30–65 (Table 3). On S-NIAH, Qwen3-8B with Direct averages 3.2 over up to five input lengths, so it can score at most about 16 at any length. Yet Table 5 reports a leaf accuracy of 0.70 for the same model on single 32K pieces. The two numbers are not the same metric, but the gap is too large to be explained by that alone. Together, these results suggest that the λ-RLM calls and the Direct calls differ in prompt, answer parsing, or scoring. Two reporting problems add to my concern. First, some numbers are reported both as an average over all input lengths and as the result for one input length, which is a bit confusing to me. For example, Qwen3-8B's 48.3 on OOLONG is the Table 3 average over all lengths, but Tables 6, 7, and 12 label it as the 131K result. Second, the paper uses paired two-sided permutation tests over five seeds. If the five seed results are the paired units, the smallest possible p-value of an exact test is 2/32 ≈ 0.06, so no result should be significant at 0.05.
- It is unclear whether the gains come from fixed control or from the hand-written plan for each task. The authors write a plan by hand for each task, and λ-RLM mostly runs it as a single split-then-combine step. The gains are largest where the plan already does much of the work. On OOLONG-Pairs, the model only labels records, and fixed code builds the pairs. On CodeQA, where the plan is generic, λ-RLM scores lower than RLM for all three strong models. The paper does not separate these two factors. Its ablation and PlanGen baseline each use only one model and one task, and it does not cite or compare with similar split-then-combine methods such as LLM×MapReduce and Chain-of-Agents. Testing on tasks without a ready-made plan, such as LongHealth or Harvey's Legal Agent Benchmark (LAB), would also show whether the method generalizes.
- The paper's motivation is only weakly supported by the reported evidence. The paper motivates the method with two claims: (1) Free-form RLM fails because of code and control errors and (2) Shorter chunks avoid context rot. Neither is well-supported. RLM's failures are never broken down by cause (for example, malformed code, runaway recursion, or wrong answers), so we cannot tell whether simpler fixes, such as retrying on error, would close the gap. For context rot, shorter chunks actually hurt. In Table 6, cutting leaves from 26.2K to 8.2K tokens drops OOLONG accuracy from 48.3 to 1.4.
- The 4× speedup is not shown under comparable resources. Although Section 6 says call counts are reported, the paper has no call counts for RLM and no token counts for any method. Only a few λ-RLM leaf counts appear (e.g., Tables 5 and 6). There is also no matched compute budget, and it is unclear whether the models ran on local vLLM or through free APIs, where latency depends on queueing and rate limits, which affects every task. In addition, the BrowseComp+ and S-NIAH plans filter inputs with cheap previews before the model reads them, but no ablation removes this filter, so part of the speedup on these two tasks may come from the filter.
- The conceptual novelty is also limited: the authors themselves state that the λ-calculus is only a specification language, that an ordinary verified function could give the same guarantees, and that the gains come from restricting control rather than from the notation.
Questions for authors
- Table 7(d) gives free-form code access to the combinators on Qwen3-8B × OOLONG. How does RLM perform when given the same combinators and a plan description on the other models and tasks? Does PlanGen use the λ-RLM executor to run its plan?
- On OOL-Pairs, how many pairs does each method produce, and how many does the task require? Does the gain remain when all methods have room to return all required pairs? Pair F1 penalizes both wrong pairs and missing pairs. Can you split the near-zero OOL-Pairs scores into early stopping, input limits, formatting errors, and wrong classification? Without this, the gap does not show how much better the method reasons.
- What limits on parallel requests, steps, and generated text apply to each method? How many calls and tokens does each use? Were the models served on local vLLM or through free APIs? What recursion depth is reached on each task, especially CodeQA?
- Does the model combine two child results per call, or all child results in one call? What happens when a leaf or composition prompt exceeds the context limit?
- What share of RLM failures are code errors, timeouts, or runaway recursion, and what share are wrong answers?
7 comments
Response to Reviewer STfL (continued 1)
Q1. How does free-form RLM perform with the same combinators and a plan description? Does PlanGen use the λ-RLM executor?
We gave free-form RLM both the λ-RLM combinators and a plan description on the same 20 Llama-3.1-8B OOLONG-Pairs questions at 32K. Both variants scored near zero. Plain RLM’s 2.3% mean pair F1 came almost entirely from partial recovery on one question. The assisted variant scored 0.0% at a mean latency of 893 s. Nine assisted runs produced no runnable code, and ten more ended with a code exception and empty answer, an iteration cap, or a time limit. The model still had to translate the plan into executable control code (Appendix D.12, Table 29). In the separate Qwen3-8B × OOLONG ablation, adding combinator access raised free-form RLM’s score from 24.1 to 31.7, while full λ-RLM scored 48.3 (Table 7).
The original PlanGen baseline runs its generated plan as written, outside the λ-RLM executor. The LongHealth generated-plan variant validates and freezes its plan before running it through the λ-RLM executor.
Q2. How many pairs are required and returned, and does the gain persist when all methods have enough output capacity?
For the sampled dense queries, the median gold answer contains 193, 850, 4,698, 15,679, and 63,915 pairs at 8K, 16K, 32K, 64K, and 128K, respectively. Every sampled query at 32K and above requires more than 4,096 output tokens to enumerate all gold pairs. At 128K, the median sampled gold list requires approximately 511K output tokens (Appendix D.7, Table 18). In that bucket, λ-RLM achieves 46.6 F1 versus 0.1 for Direct on Llama-3.1-8B (Appendix D.6, Table 17). On split inputs, λ-RLM predicts record labels and constructs the pairs symbolically, allowing the final list to exceed the per-call generation cap.
In the 20-question RLM diagnostic at 32K, one run recovered 2,487 of 5,356 gold pairs. Another reached the iteration cap and returned 50 pairs, two of which were correct. The remaining 18 recovered zero correct pairs (Appendix D.11, Table 28). The traces show no confirmed failure due to the output-token limit. The single API timeout may have been related to output length, and its cause remains uncertain. Code and control failures accounted for 17 of 20 runs, including two input-context overflows. These traces suggest that λ-RLM’s advantage over free-form RLM would likely persist in this diagnostic if both could return the complete pair list, since increasing output capacity alone would leave the predominant failure modes in place.
For Direct, we expect the gap to narrow if the model can process the full input and generate the complete pair list. The current cap limits attainable recall on dense queries. The size of the reduction remains unmeasured, as do returned-pair counts and failure breakdowns across all methods and buckets.
Q3. What were the serving setup, execution limits, and actual resource use?
All Table 3 runs used the free tier of a hosted inference API, with the same endpoint, backbone, and decoding settings within each model–task cell. Each model call could generate up to 4,096 output tokens. Direct used one call for inputs that fit. RLM had a depth cap of 2, 10 iterations per call, and up to 16 concurrent calls. λ-RLM calls ran sequentially (Appendix D.1).
Tables 30–31 report resource use for the targeted diagnostics. On the Llama-3.1-8B S-NIAH subset, plain RLM averaged 15 calls and 60,719 tokens per question. λ-RLM averaged 32,089 tokens without filtering and 20,293 with filtering. On CodeQA, RLM averaged 13 calls and 40,255 tokens at the 10-iteration limit. Raising the limit to 30 increased these means to 27 calls and 169,308 tokens, and accuracy on the 23 questions rose from 21.7% to 30.4%. Counting the root as depth 0, the deepest observed RLM call in these S-NIAH and CodeQA diagnostics was at depth 1.
Q4. How are child outputs composed, and what happens on context overflow?
For neural composition, the executor concatenates all child outputs at a node and sends them to the model in one call. Symbolic composition, including count merging, search-result selection, and pair construction, uses no model call. The planner limits leaf size and caps neural fan-in using the composition budget. Theorem 1 assumes that the resulting prompts fit within the applicable limit. The runtime behavior for an oversized formatted leaf or composition prompt remains unspecified in the manuscript.
Q5 and W5 continued in next comment
Response to Reviewer STfL continued 2)
Q5. What caused the RLM failures?
On Llama-3.1-8B × OOLONG-Pairs at 32K, the primary causes across 20 RLM runs were code errors in 5/20 (25%), control failures in 12/20 (60%), an API timeout in 1/20 (5%), and wrong answers after error-free execution in 2/20 (10%). The control failures comprised six iteration-cap failures, four runaway execution loops stopped at the time limit, and two input-context overflows. The diagnostic used a recursion depth cap of one. Restarting ten runs with an error signal improved none of them, and their mean F1 remained 0.0% (Appendix D.11, Table 28).
W5. Conceptual contribution. λ-RLM makes recursive inference analyzable before execution begins. Once a plan is instantiated, the executor enforces its splitting, stopping, and composition rules. The model solves bounded subproblems and performs neural composition where specified. Its outputs can affect the answer, while the executor retains control over how the computation proceeds. Under the stated assumptions, one termination and model-call analysis applies across all admissible plans. The λ-calculus states the invariants explicitly, so any runtime that enforces them can inherit the same guarantees.
Response to Reviewer STfL
Thanks for the insightful comments. We have addressed and comments. Please find our response to weakness (W) and questions (Q) below.
W1. Short-circuit behavior and experimental reporting. The short-circuit was active in the Table 3 runs. λ-RLM uses a 32K prompt-length threshold, so 64K and 128K inputs trigger decomposition even when they fit within the backbone’s native context window. For prompts at or below the threshold, λ-RLM makes one task-selection call and one solver call using the selected task template. Direct sends the raw benchmark prompt in one call. Both methods use the same final-answer parser and scorer. Algorithms 2 and 3 now show the templated call explicitly. Table 17 reports single-seed per-bucket scores for Llama-3.1-8B, while Table 3 reports five-seed aggregates. Routing in the nominal 32K bucket depends on the actual prompt length.
For Qwen3-8B on S-NIAH, 3.2 for Direct and 24.6 for λ-RLM are end-to-end F1 scores averaged across input lengths. The 0.70 in Table 5 is λ-RLM’s solver accuracy on the relevant 32K leaf containing the evidence. End-to-end evaluation also requires locating that evidence in the full input. A matched comparison of raw and templated prompts on the same leaves would quantify the template’s contribution.
The 131K label identifies the example used to illustrate the planner settings. The repeated 48.3 is the bucket-averaged Table 3 score. We clarified the captions and revised the win–tie–loss wording to report 36 higher five-seed means for λ-RLM and nine for RLM. With five paired seeds, the smallest attainable exact two-sided permutation p-value is 0.0625. Appendix D.10 provides standard deviations and p-values in Table 22 and individual seed scores in Tables 23–27.
W2. Fixed control and hand-designed plans. On Qwen3-8B × OOLONG, Table 7 holds the combinator library fixed. Fixing the plan before execution yields 48.3, compared with 40.2 when the model can reselect the plan at each turn. This 8.1-point difference measures the effect of plan fixity in that setting. The OOLONG-Pairs gains also include the benefit of symbolic pair construction, discussed in Q2.
On LongHealth, a benchmark outside the six predefined task types, the same executor achieved 76.0% with a generic split–map–reduce plan and 76.7% with a validated model-generated plan on Mistral-7B. RLM achieved 63.2%, and Chain-of-Agents, the strongest split-and-combine baseline, achieved 68.0% (Appendix D.8, Table 19). Both λ-RLM variants ran without a hand-written LongHealth plan. Table 21 also shows that λ-RLM outperforms fixed chunk–map–reduce, LLM×MapReduce, and Chain-of-Agents on OOLONG and OOLONG-Pairs at 32K with both Llama-3.1-8B and Mistral-7B. On CodeQA, λ-RLM with the generic
summariseplan has the higher mean on five of nine models, while RLM leads on all three strong models.W3. Execution failures and leaf size. Code or control errors were the primary cause in 17 of 20 free-form RLM runs in the Llama-3.1-8B OOLONG-Pairs diagnostic. Restarting ten runs that showed an error signal left their mean F1 at 0.0% (Appendix D.11). Q5 gives the breakdown.
In the OOLONG ablation, reducing leaf size from 26.2K to 8.2K lowered measured leaf accuracy from 0.85 to 0.76. Shorter chunks can split records across boundaries. For the representative 131K input, the same fan-out change increased the leaf count from five to sixteen, so the aggregate depended on more leaf outputs being correct. The observed score across buckets fell from 48.3 to 1.4 (Tables 6 and 12). This motivates choosing a leaf size that preserves the evidence needed for the task.
W4. Scope of the speedup. The 4.0× figure is the ratio of mean end-to-end wall-clock latencies across the 45 Table 3 cells under the same hosted setup. Timing includes intermediate calls, network time, and API queueing.
On 32 S-NIAH questions in the 32K bucket with Llama-3.1-8B, λ-RLM without preview filtering was 2.46× as fast as RLM. Enabling filtering provided a further 1.67× speedup within λ-RLM (Appendix D.13, Table 30). Uncapped RLM used nearly twice as many tokens per question as unfiltered λ-RLM (60,719 versus 32,089) and scored lower (8.5% versus 16.7% F1). Capping RLM near λ-RLM’s mean token use reduced RLM’s F1 further to 5.1%, with realized usage of 26,706 tokens per question. This comparison is specific to the S-NIAH diagnostic. Benchmark-wide RLM call and token totals were unavailable for Table 3, and the filter’s separate effect on BrowseComp+ remains unmeasured.
Response to W5 and Questions continued in next comments.