acceptodds
Under review as a conference paper at ICLR 2027

PAIR: Memory-Efficient and Fast Inference with Reading–Reasoning Fusion

Abstract

For context-based question answering, using one large model for both evidence extraction and answer reasoning requires substantial memory and computation. Existing approaches trade accuracy for efficiency within a single model only. We propose a multi-model approach for efficient inference: our method, PAIR, employs two differently sized models for evidence extraction and answer reasoning, respectively. A smaller Reader model processes and retains the full context, while a larger Reasoner model sees only the question and generated sequence without directly attending to the context. These capabilities are fused at the logit level: at each token, both Reader and Reasoner produce a prediction, and their logits are combined into one, behaving as a single next-token-prediction model. Fine-tuning further boosts their performance. On HotpotQA, PAIR with a 32B Reasoner retains 96% of that same 32B model’s full-context accuracy while using roughly one quarter of its KV memory. At 45.7K-token contexts, PAIR achieves up to 3.2 higher end-to-end throughput than the full-context 32B model on the same GPU. PAIR also improves the accuracy-memory and accuracy-throughput trade-offs over evaluated token-reduction baselines. Its accuracy-preserving behavior generalizes across Qwen2.5, OLMo-3, and Gemma-3 model families.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Efficient language model inference, Efficient LLM inference, Retrieval-augmented generation, Model collaboration, KV cache compression, KV cache optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_e35POpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PAIR, a multi-model approach for efficient long-context inference. PAIR uses a smaller Reader to process the full context, while a larger Reasoner only sees the question and generated sequence. At each decoding step, their logits are combined using a weighted sum, and the resulting next token is appended to a shared generated sequence, which serves as the communication channel between the two models. This allows context-specific evidence extracted by the Reader to become indirectly available to the Reasoner for subsequent reasoning. The models are further adapted through role-specific fine-tuning for collaborative generation.

Critical strengths
  • Simple and interpretable method : PAIR introduces a simple and intuitive decomposition of long-context inference: a smaller Reader retains the full context, while a larger Reasoner operates only on the question and generated sequence, with their predictions combined through logit-level fusion. Despite its simplicity, the method achieves a strong accuracy–efficiency trade-off, retaining most of the full-context large model’s accuracy while substantially reducing KV-cache memory and improving throughput.

  • Empirical validation across different serving regimes and model configurations : The evaluation goes beyond theoretical KV-memory savings and includes end-to-end throughput, matched-batch comparisons, scaling with context length and batch size, and two-GPU branch-parallel execution. The authors also evaluate multiple Reader–Reasoner size ratios and three model families, providing useful evidence that the observed accuracy preservation is not specific to a single model pair.

Critical weaknesses
  • The main result does not fully cover closely related input-compression approaches. While the paper identifies CoGenesis as the closest architectural precedent to PAIR, the distinction is primarily discussed in terms of motivation and deployment setting, and no direct empirical comparison is provided. Similarly, input-compression methods such as LLMLingua/LongLLMLingua are discussed in related work but are absent from the empirical comparison. Comparing against sequential context compression followed by large-model inference would provide a broader assessment of PAIR’s end-to-end accuracy–efficiency trade-off.

  • Although Appendix G.1 explicitly distinguishes resident KV-cache storage from peak GPU memory, the paper does not report total serving-memory usage, including the additional Reader weights. Reporting this at matched batch sizes across context lengths would clarify the practical memory savings.

Questions for authors
  • Could you provide direct empirical comparisons with CoGenesis and with input-compression methods such as LLMLingua/LongLLMLingua followed by large-model inference?
  • How much total GPU memory does PAIR require relative to the full-context baseline across different context lengths?
4 comments
AuthorsOpenReview ↗

Response to W1/Q1

W1 / Q1. CoGenesis and input compression

W1. The main result does not fully cover closely related input-compression approaches. While the paper identifies CoGenesis as the closest architectural precedent to PAIR, the distinction is primarily discussed in terms of motivation and deployment setting, and no direct empirical comparison is provided. Similarly, input-compression methods such as LLMLingua/LongLLMLingua are discussed in related work but are absent from the empirical comparison. Comparing against sequential context compression followed by large-model inference would provide a broader assessment of PAIR's end-to-end accuracy--efficiency trade-off.

Q1. Could you provide direct empirical comparisons with CoGenesis and with input-compression methods such as LLMLingua/LongLLMLingua followed by large-model inference?

We compare PAIR against three alternatives: Sequential, CoGenesis, and LongLLMLingua. In Sequential, the adapted Reader generates a complete response, including reasoning and the answer, which is then passed to the unadapted 32B for final answering. LongLLMLingua uses the base 7B to compress the context to approximately 25% of its tokens before the unadapted 32B generates an answer.

We also evaluate a CoGenesis-style learned fusion gate as an alternative to PAIR's fixed fusion. Since CoGenesis was developed for personalized generation and does not specify a training setup for long-context QA, we use PAIR's S-solo-adapted Reader and explicit reasoning step. We train the fusion gate on the L-fusion corpus using the same distillation procedure as PAIR. This comparison evaluates a CoGenesis-style fusion approach adapted to long-context QA, rather than the original CoGenesis training setup.

Method (F1)HotpotQAMuSiQueLooGLELoCoMo
Full-context 32B0.7490.4700.4880.558
PAIR (ours)0.7180.4360.4370.499
Sequential0.7150.4180.4260.495
CoGenesis0.7070.4130.4280.500
LongLLMLingua (25%)0.6930.3920.4270.369
Reader alone0.6940.4020.4260.484

PAIR achieves higher F1 than Sequential and LongLLMLingua on all four benchmarks. It also outperforms CoGenesis on three benchmarks and trails by only 0.001 F1 on LoCoMo.

We additionally compare end-to-end throughput and time to first token (TTFT) on 96 HotpotQA questions using one GPU, with both models loaded for methods that require them.

MethodBatch sizeSeconds / answerTTFT (s)
PAIR (ours)201.005.5
Sequential201.0612.1
Full-context 32B121.7910.4
LongLLMLingua (25%)162.8835.8

Sequential achieves similar throughput to PAIR, but its TTFT is 2.2 times higher because the 32B must wait for the Reader to finish generating its response. LongLLMLingua incurs an even larger delay from context compression. The TTFT values for Sequential and LongLLMLingua are calculated from their measured stage timings.

AuthorsOpenReview ↗

Response to W2/Q2

W2 / Q2. Peak GPU memory

W2. Although Appendix G.1 explicitly distinguishes resident KV-cache storage from peak GPU memory, the paper does not report total serving-memory usage, including the additional Reader weights. Reporting this at matched batch sizes across context lengths would clarify the practical memory savings.

Q2. How much total GPU memory does PAIR require relative to the full-context baseline across different context lengths?

Peak GPU memory depends on batch size, context length, and the stage of inference at which the maximum allocation occurs. Therefore, our original evaluation compares end-to-end throughput at the largest tested feasible batch size under the same GPU memory budget. This accounts for PAIR's additional Reader weights while measuring the practical benefit of its lower per-sequence memory requirements.

To make the memory crossover explicit, we additionally measured peak GPU memory at matched batch sizes across four context lengths. We used the HotpotQA batch-scaling workload with 40, 80, 120, and 160 documents (approximately 6.0K, 11.6K, 17.1K, and 22.9K tokens) on a single GH200 GPU with 95.08 GiB of usable memory. Both methods use the FKV backend, with bf16 model weights and bf16 KV caches. LoRA adapters are merged into the model weights.

We report the maximum PyTorch-allocated memory (torch.cuda.max_memory_allocated()) observed during the complete 96-question workload, including prefill, question processing, CUDA Graph operations, and decoding. The measurements include model weights, KV caches, and temporary allocations. The peak occurs during context prefill, when preallocated KV buffers coexist with temporary activations.

The largest tested feasible batch sizes for 40, 80, 120, and 160 documents were 10, 6, 4, and 3 for the full-context 32B, compared with 16, 10, 6, and 6 for PAIR. These are not necessarily the maximum supported batch sizes; PAIR did not encounter OOM at its largest tested batches.

DocsBatch32B (GiB)PAIR (GiB)
40267.578.5
40887.084.7
80271.579.3
80693.585.1
120275.880.1
120491.683.7
160279.880.9
160391.984.5

Immediately after model loading, the full-context 32B occupies 61.027 GiB, compared with 75.244 GiB for PAIR, reflecting 14.217 GiB of additional Reader weights. This makes PAIR more memory-intensive at small batch sizes. Linear fits to the batch-sweep measurements indicate that the full-context 32B requires approximately 2.8–4.0× more incremental memory per sequence than PAIR. The estimated crossover batch sizes are approximately 7, 4, 3, and 2 for 40, 80, 120, and 160 documents, respectively. Consequently, PAIR supports larger tested batches under the same GPU memory budget, explaining the throughput advantage reported in our original evaluation.

Thank you for the additional comparisons and memory measurements. The baseline comparisons and memory crossover analysis help me better understand where PAIR offers practical benefits. It would be helpful to include these results in the revised paper.

Appreciate your constructive feedback. We have uploaded a revised manuscript with the KL-based fusion-weight selection procedure in Section 4.3, additional training details in Appendix B, fusion-weight calibration results in Appendix C, and matched-batch peak GPU memory measurements and memory crossover analysis in Appendix G. We have also updated the evaluation results to reflect the KL-selected fusion weights.

Reviewer_mUbvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PAIR separates long-context question answering between a small Reader, which retains the complete context, and a larger Reasoner, which sees only the question and shared generated prefix. Their next-token logits are fused, with sequential teacher-based adaptation of the Reader and Reasoner. The reported Qwen2.5 configuration retains 89-96% of the large model's answer F1 using 22.7-25.2% of its persistent KV storage, while improving end-to-end throughput on the tested workloads. Experiments also examine model families, context lengths, adaptation, and quantization.

Critical strengths
  1. The accuracy-efficiency tradeoff is practically interesting and supported by several complementary measurements. On HotpotQA, PAIR achieves 0.7178 F1 versus 0.7492 for the 32B model at 24.8% of its KV storage (Table 6). Timing includes prefill and decoding, and the paper separates feasible-batch throughput from matched-batch scaling. Appendix G explicitly includes both models' caches and distinguishes persistent KV storage from peak device memory.
  2. The evaluation includes useful breadth and qualifications. The accuracy improvement over the smaller reference extends to Qwen2.5, OLMo-3, and Gemma-3 (Table 11). The authors disclose that the OLMo pair provides no KV saving, and Appendix K reports both successful Reader-int8 composition and unsuccessful Reader-int4 quantization. These results help identify where the proposed tradeoff applies.
Critical weaknesses
  1. The experiments do not sufficiently isolate the contribution of the collaboration and adaptation recipe. Section 2 identifies CoGenesis as an architectural precedent with asymmetric context access and token-level fusion, but no direct comparison is reported. A matched sequential evidence-extraction/answering pipeline would also test whether per-token fusion is needed. Furthermore, Table 1's “Reader FT” remains a fused configuration: it does not evaluate the adapted Reader alone. This distinction matters because plain fusion underperforms the small reference on LooGLE, while most of the LoCoMo improvement already appears with Reader-only adaptation. The ablation establishes that Reasoner adaptation helps the tested fused system; without a Reasoner-only adaptation condition, it does not establish that adapting both models is necessary.
  2. The checkpoint and fusion-weight selection protocol is unclear. Section 4.2 states that the same trained weights are used across benchmarks. Appendix C.1, however, says changing the fusion weight requires separately adapting the Reasoner, and Table 3 selects 0.70 for HotpotQA and 0.85 for LoCoMo. The paper does not identify disjoint selection and final-evaluation splits or clearly explain whether these deployed settings use different adapters. This matters for the generalization claim: LoCoMo improves from 0.461 to 0.499 across the sweep and includes only ten source conversations. Excluding evaluation examples from fine-tuning does not by itself establish separation from hyperparameter selection.
Questions for authors
  1. How do the adapted Reader alone and a matched CoGenesis-style or sequential Reader-to-Reasoner system compare under the same training data, generation budget, and hardware?
  2. Which adapters are deployed for each benchmark, and which examples or conversations are used for fusion-weight selection, teacher-KL diagnostics, and final evaluation? Can the authors report an explicitly held-out selection protocol?
  3. What are peak total GPU memory and weight precision at matched batch sizes? This would clarify the practical crossover from adding Reader weights, especially relative to the higher-accuracy full-model int4 operating points in Table 13.
5 comments
AuthorsOpenReview ↗

Response to W1/Q1

W1 / Q1. Collaboration, adaptation, and matched baselines

W1. The experiments do not sufficiently isolate the contribution of the collaboration and adaptation recipe. Section 2 identifies CoGenesis as an architectural precedent with asymmetric context access and token-level fusion, but no direct comparison is reported. A matched sequential evidence-extraction/answering pipeline would also test whether per-token fusion is needed. Furthermore, Table 1's “Reader FT” remains a fused configuration: it does not evaluate the adapted Reader alone. This distinction matters because plain fusion underperforms the small reference on LooGLE, while most of the LoCoMo improvement already appears with Reader-only adaptation. The ablation establishes that Reasoner adaptation helps the tested fused system; without a Reasoner-only adaptation condition, it does not establish that adapting both models is necessary.

Q1. How do the adapted Reader alone and a matched CoGenesis-style or sequential Reader-to-Reasoner system compare under the same training data, generation budget, and hardware?

Result I: Training both models gives the strongest fused result.

We have additionally evaluated the Reader alone and separately trained each component of the fused system. “Reader alone” means the S-solo trained 7B answers independently, without a Reasoner or fusion. “Reader FT + fusion” instead combines that trained Reader with the base 32B. The Reasoner-only setting trains the 32B using L-fusion with the base Reader.

ConfigurationTrainingHotpotQA F1
Plain fusionNeither model0.676†
Reader FT + fusionReader only0.693†
Reasoner FT + fusionReasoner only0.700
PAIR (ours)Both models0.718
Reader alone (without fusion)Reader only0.694

Note. † These two values have been corrected from the original submission. While checking the additional experiments, we found that the newer evaluation harness stopped at <|im_end|> but not at <|endoftext|>, unlike the original fusion decoder. Some outputs therefore continued after the intended end token, and the answer extractor included the extra text. We restored the original stopping rule and regraded the existing generations consistently. Plain fusion changed from 0.672 to 0.676, and Reader FT + fusion from 0.663 to 0.693. All other reported results, including PAIR (0.718), remained unchanged. We did not retrain models or change evaluation questions.

Result II: Additional baselines

We compare PAIR against three alternatives: Sequential, CoGenesis, and LongLLMLingua. In Sequential, the adapted Reader generates a complete response, including reasoning and the answer, which is then passed to the unadapted 32B for final answering. LongLLMLingua uses the base 7B to compress the context to approximately 25% of its tokens before the unadapted 32B generates an answer.

We also evaluate a CoGenesis-style learned fusion gate as an alternative to PAIR's fixed fusion. Since CoGenesis was developed for personalized generation and does not specify a training setup for long-context QA, we use PAIR's S-solo-adapted Reader and explicit reasoning step. We train the fusion gate on the L-fusion corpus using the same distillation procedure as PAIR. This comparison evaluates a CoGenesis-style fusion approach adapted to long-context QA, rather than the original CoGenesis training setup.

Method (F1)HotpotQAMuSiQueLooGLELoCoMo
Full-context 32B0.7490.4700.4880.558
PAIR (ours)0.7180.4360.4370.499
Sequential0.7150.4180.4260.495
CoGenesis0.7070.4130.4280.500
LongLLMLingua (25%)0.6930.3920.4270.369
Reader alone0.6940.4020.4260.484

PAIR achieves higher F1 than Sequential and LongLLMLingua on all four benchmarks. It also outperforms CoGenesis on three benchmarks and trails by only 0.001 F1 on LoCoMo.

We additionally compare end-to-end throughput and time to first token (TTFT) on 96 HotpotQA questions using one GPU, with both models loaded for methods that require them.

MethodBatch sizeThroughput (s/answer)TTFT (s)
PAIR (ours)201.005.5
Sequential201.0612.1
Full-context 32B121.7910.4
LongLLMLingua (25%)162.8835.8

Sequential achieves similar throughput to PAIR, but its TTFT is 2.2 times higher because the 32B must wait for the Reader to finish generating its response. LongLLMLingua incurs an even larger delay from context compression. The TTFT values for Sequential and LongLLMLingua are calculated from their measured stage timings.

AuthorsOpenReview ↗

Response to W2/Q2

W2 / Q2. Fusion weights and checkpoint selection

W2. The checkpoint and fusion-weight selection protocol is unclear. Section 4.2 states that the same trained weights are used across benchmarks. Appendix C.1, however, says changing the fusion weight requires separately adapting the Reasoner, and Table 3 selects 0.70 for HotpotQA and 0.85 for LoCoMo. The paper does not identify disjoint selection and final-evaluation splits or clearly explain whether these deployed settings use different adapters. This matters for the generalization claim: LoCoMo improves from 0.461 to 0.499 across the sweep and includes only ten source conversations. Excluding evaluation examples from fine-tuning does not by itself establish separation from hyperparameter selection.

Q2. Which adapters are deployed for each benchmark, and which examples or conversations are used for fusion-weight selection, teacher-KL diagnostics, and final evaluation? Can the authors report an explicitly held-out selection protocol?

To select λ\lambda without relying on downstream evaluation accuracy, we now apply the teacher-forced KL heuristic originally derived for theoretical analysis in Section 4.3 (Theoretical Analysis Behind Fusion Weight Selection) as our actual weight-selection procedure. This changes how we select λ\lambda, not the model training or evaluation setup. Specifically, we generate outputs using a full-context teacher and estimate the relative prediction errors of the unadapted Reader and Reasoner along those outputs.

We perform this label-free calibration directly on the evaluation inputs, using 100 sampled questions per benchmark, or five source conversations for LoCoMo. These are not held-out calibration examples, but no ground-truth answers or downstream accuracy measurements are used. The Reader adapter is shared across benchmarks, while Reasoner adapters are pretrained at λ\lambda intervals of 0.05 using the same task-agnostic L-fusion corpus. We select the Reasoner adapter corresponding to the nearest calibrated λ\lambda, without benchmark-specific training.

Checkpoint selection is also independent of downstream evaluation accuracy. Each adapter is trained for a fixed number of steps, without selecting checkpoints based on benchmark performance.

QA F1 using the original accuracy-selected fusion weights and the new KL-selected weights. Both settings are evaluated on the same test sets.

BenchmarkOriginal λ\lambdaOriginal F1KL-selected λ\lambdaKL-selected F1
HotpotQA0.700.7180.700.718
MuSiQue0.700.4380.650.436
LooGLE0.700.4440.750.437
LoCoMo0.850.4990.850.499

The KL-based selection preserves nearly the same accuracy without tuning on downstream evaluation performance.

AuthorsOpenReview ↗

Response to Q3

Q3. Total peak GPU memory

Q3. What are peak total GPU memory and weight precision at matched batch sizes? This would clarify the practical crossover from adding Reader weights, especially relative to the higher-accuracy full-model int4 operating points in Table 13.

Peak GPU memory depends on batch size, context length, and the stage of inference at which the maximum allocation occurs. Therefore, our original evaluation compares end-to-end throughput at the largest tested feasible batch size under the same GPU memory budget. This accounts for PAIR's additional Reader weights while measuring the practical benefit of its lower per-sequence memory requirements.

To make the memory crossover explicit, we additionally measured peak GPU memory at matched batch sizes across four context lengths. We used the HotpotQA batch-scaling workload with 40, 80, 120, and 160 documents (approximately 6.0K, 11.6K, 17.1K, and 22.9K tokens) on a single GH200 GPU with 95.08 GiB of usable memory. Both methods use the FKV backend, with bf16 model weights and bf16 KV caches. LoRA adapters are merged into the model weights.

We report the maximum PyTorch-allocated memory (torch.cuda.max_memory_allocated()) observed during the complete 96-question workload, including prefill, question processing, CUDA Graph operations, and decoding. The measurements include model weights, KV caches, and temporary allocations. The peak occurs during context prefill, when preallocated KV buffers coexist with temporary activations.

The largest tested feasible batch sizes for 40, 80, 120, and 160 documents were 10, 6, 4, and 3 for the full-context 32B, compared with 16, 10, 6, and 6 for PAIR. These are not necessarily the maximum supported batch sizes; PAIR did not encounter OOM at its largest tested batches.

DocsBatch32B (GiB)PAIR (GiB)
40267.578.5
40887.084.7
80271.579.3
80693.585.1
120275.880.1
120491.683.7
160279.880.9
160391.984.5

Immediately after model loading, the full-context 32B occupies 61.027 GiB, compared with 75.244 GiB for PAIR, reflecting 14.217 GiB of additional Reader weights. This makes PAIR more memory-intensive at small batch sizes. Linear fits to the batch-sweep measurements indicate that the full-context 32B requires approximately 2.8–4.0× more incremental memory per sequence than PAIR. The estimated crossover batch sizes are approximately 7, 4, 3, and 2 for 40, 80, 120, and 160 documents, respectively. Consequently, PAIR supports larger tested batches under the same GPU memory budget, explaining the throughput advantage reported in our original evaluation.

We also measured the full-context 32B with int4 KV quantization and bf16 weights. It uses less peak memory than PAIR at all tested matched batches (e.g., 87.6 vs. 91.7 GiB at 80 documents, batch 10), although it uses eager execution rather than FKV. As discussed in Appendix K, KV quantization can also be combined with PAIR, as demonstrated by our Reader-int8 experiments. The memory advantage reported above is therefore relative to the bf16 full-context baseline, not the int4 KV baseline.

Reviewer_W1U7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PAIR a long-context reasoning framework. It delegates evidence extraction to a small model with full-context access and answer reasoning to a large model that operates without long-context input. By fusing logits using fixed weights, it achieves savings in KV cache memory and increased throughput on benchmarks.

Critical strengths
  • Decoupling reading from reasoning, it avoids the need for large models to perform pre-filling on long texts or maintain a KV cache; an effective approach.
  • Retaining the full context within a smaller model overcomes the limitations of token truncation.
Critical weaknesses
  • Lacks conceptual novelty. The method relies on a patch work of static logit weighting and distillation.
  • It is heavily dependent on explicit evidence restatement and requires the Q&A pairs to share the same vocabulary.
  • Evaluated on only 45.7k samples and necessitates model retraining.
  • The parameter λ\lambda requires task-specific tuning.
  • The paper fails to specify the LoRA hyperparameters and the construction process for the S-solo dataset.
Questions for authors
  1. Why are the LoRA training hyperparameters and the specific construction method for the S-solo synthetic data omitted from Appendix B?
  2. Does PAIR fail in tasks where it is not possible to first generate a concise restatement of the evidence (e.g., long-document summarization or code comprehension)?
  3. When facing tasks with an unknown distribution, how can the weight λ\lambda be determined without a validation set?
3 comments
AuthorsOpenReview ↗

Responses to W1, W2/Q2, and W3

W1. Conceptual novelty

W1. Lacks conceptual novelty. The method relies on a patch work of static logit weighting and distillation.

PAIR demonstrates that reading and reasoning can be separated in long-context QA. Making this separation work requires more than combining existing techniques. We develop a logit fusion strategy, role-specific adaptation, and task-agnostic distillation to enable effective collaboration between the Reader and Reasoner. We also implement efficient decoding with fixed-KV (FKV) backend to realize the computational benefits of this separation. Together, these contributions produce a practical inference system that preserves 89–96% of the full-context 32B model's QA F1 with only 22.7–25.2% of its persistent KV storage, while improving inference throughput.

 

W2 / Q2. Evidence restatement and generalization beyond QA

W2. It is heavily dependent on explicit evidence restatement and requires the Q&A pairs to share the same vocabulary.

Q2. Does PAIR fail in tasks where it is not possible to first generate a concise restatement of the evidence (e.g., long-document summarization or code comprehension)?

PAIR does not require vocabulary overlap between the question and answer or a separate evidence-restatement step. Instead, the Reader and Reasoner jointly generate a shared sequence through token-level logit fusion. The explicit reasoning step is part of this generation process, not a prerequisite for summarizing the evidence. We further evaluate PAIR on API code generation below.

To evaluate a different task, we tested WAPIIBench, which requires generating executable JavaScript API calls from natural-language instructions and real-world OpenAPI specifications. The Reader processes the API documentation, while the Reasoner generates code from the task instruction. We evaluated 395 tasks using the benchmark's official execution-based evaluator. The average retrieved context length is approximately 790 tokens.

MethodFully correct API calls (395 tasks)
Full-context 32B198 (0.501)
PAIR (ours)203 (0.514)
SpecPrefill172 (0.435)
SnapKV130 (0.329)
ExpectedAttention105 (0.266)
Reader alone104 (0.263)

PAIR uses λ=0.65\lambda=0.65, selected through teacher-forced KL on 100 sampled tasks. It achieves comparable accuracy to the full-context 32B model and outperforms the other evaluated baselines. This demonstrates that the reading/reasoning separation also works for API code generation.

 

W3. Evaluation scale and retraining

W3. Evaluated on only 45.7k samples and necessitates model retraining.

The cited 45.7K refers to context length in tokens, not the number of evaluation samples. PAIR does not require task-specific retraining. Both models are adapted using task-agnostic distillation, and their adapters are reused across downstream benchmarks.

AuthorsOpenReview ↗

Response to W4/Q3

W4 / Q3. Fusion-weight selection and unknown task distributions

W4. The parameter λ\lambda requires task-specific tuning.

Q3. When facing tasks with an unknown distribution, how can the weight λ\lambda be determined without a validation set?

We understand the concern. To address it, we have studied two additional methods: (i) label-free in-task calibration, and (ii) pre-calibration.

New Method I: Label-free Calibration

To select λ\lambda without relying on downstream evaluation accuracy, we now apply the teacher-forced KL heuristic originally derived for theoretical analysis in Section 4.3 (Theoretical Analysis Behind Fusion Weight Selection) as our actual weight-selection procedure. This changes how we select λ\lambda, not the model training or evaluation setup. Specifically, we generate outputs using a full-context teacher and estimate the relative prediction errors of the unadapted Reader and Reasoner along those outputs.

We perform this label-free calibration directly on the evaluation inputs, using 100 sampled questions per benchmark, or five source conversations for LoCoMo. These are not held-out calibration examples, but no ground-truth answers or downstream accuracy measurements are used. The Reader adapter is shared across benchmarks, while Reasoner adapters are pretrained at λ\lambda intervals of 0.05 using the same task-agnostic L-fusion corpus. We select the Reasoner adapter corresponding to the nearest calibrated λ\lambda, without benchmark-specific training.

QA F1 using the original accuracy-selected fusion weights and the new KL-selected weights. Both settings are evaluated on the same test sets.

BenchmarkOriginal λ\lambdaOriginal F1KL-selected λ\lambda (new in rebuttal)KL-selected F1 (new in rebuttal)
HotpotQA0.700.7180.700.718
MuSiQue0.700.4380.650.436
LooGLE0.700.4440.750.437
LoCoMo0.850.4990.850.499

The KL-based selection preserves nearly the same accuracy without tuning on downstream evaluation performance.

Teacher-forced KL allows us to select λ\lambda for a new task using unlabeled examples, without requiring a labeled validation set. When target-task examples are unavailable, we instead use a general default estimated from independent benchmarks.

New Method II: Pre-calibration for Completely Unknown Tasks

To estimate such a default, we constructed a calibration set independent of our evaluation benchmarks. We sampled 100 questions from each of nine public benchmarks: Natural Questions, 2WikiMultiHopQA, MultiHop-RAG, QASPER, FinQA, TAT-QA, ContractNLI, ConditionalQA, and Doc2Dial. These cover factual retrieval, multi-hop reasoning, scientific QA, financial reasoning, legal documents, policies, and document-grounded dialogue.

We combined their source documents into a corpus of 2.82 million passages, each approximately 200 words long. For each question, BM25 retrieves the top 20 passages as the Reader's context, while the Reasoner receives only the question. We apply the same QA prompt and teacher-forced KL procedure as in our main experiments. We average prediction errors within each benchmark and then across the nine benchmarks to give each category equal weight. These 900 questions are used only for calibration, not for training.

The estimated prediction errors are KLS=0.424\mathrm{KL}_S=0.424 and KLL=0.939\mathrm{KL}_L=0.939, yielding λ^=KLL/(KLS+KLL)=0.689\hat{\lambda}=\mathrm{KL}_L/(\mathrm{KL}_S+\mathrm{KL}_L)=0.689. We round this estimate to 0.700.70 as a general default when target-task calibration examples are unavailable.

AuthorsOpenReview ↗

Response to W5/Q1

W5 / Q1. Missing training details

W5. The paper fails to specify the LoRA hyperparameters and the construction process for the S-solo dataset.

Q1. Why are the LoRA training hyperparameters and the specific construction method for the S-solo synthetic data omitted from Appendix B?

We construct S-solo using 5,280 template-generated synthetic examples and 2,000 real-data examples. For model adaptation, both the Reader and Reasoner use rank-16 LoRA with α=32\alpha=32 and dropout 0.05, applied to all seven attention and MLP projection modules. Both stages use AdamW with a learning rate of 10−410^{-4}, weight decay 0.01, and an effective batch size of 8. The Reader is trained for 900 optimizer steps on S-solo using cross-entropy against Teacher-generated responses. The Reasoner is trained for 600 steps on L-fusion using the fused-logit loss while the adapted Reader remains frozen. We will include the full training configuration and S-solo construction procedure in Appendix B.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Evaluation Protocol Clarification

For the additional rebuttal experiments, we consistently used updated stopping and answer-extraction rules to address an end-token handling issue observed in the Reasoner-only ablation. The original paper experiments used the previous stopping and extraction rules.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.