acceptodds
Withdrawn from ICLR 2027

Retriever Optimization for Low-Resource Telecom Domain-Specific Question Answering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Telecom question answering is a challenging setting for retrieval-augmented generation (RAG): evidence is fragmented across standards, research papers, encyclopedic resources, and domain web documents, while answers often depend on technical tables, equations, and specialized protocol language. In low-resource telecom subdomains, fully adapting the generator can over-specialize the model and degrade general capability, making retriever-centric adaptation especially attractive. To this end, we investigate which components should be optimized and which retriever training objectives are most effective for developing a domain-specific QA agent in a low-resource setting. We argue that retrieval is the most effective adaptation target compared with model fine-tuning. Theoretically, we show that when domain-specific data are scarce, retriever-side query-encoder tuning can generalize better than LoRA due to its lower estimation complexity. We then identify two particularly relevant retriever objectives: the latent-document RAG likelihood, which optimizes generation utility, and the contrastive InfoNCE objective, which improves semantic retrieval geometry. Rather than treating these objectives as static alternatives, we leverage them jointly through a retriever optimization method designed to maximize downstream QA performance in the telecom domain. Specifically, we introduce an adaptive temperature framework that inserts learnable temperatures into the retrieval distribution of the RAG objective and the softmax normalization of the InfoNCE objective, thereby shaping the gradient signal that each objective sends to the query encoder during training. We further add a query-distillation regularizer that constrains the learned query encoder to remain compatible with the frozen base document space, preserving the strong retrieval geometry of the base model while still allowing domain adaptation. Across telecom-specific retrieval and generative QA benchmarks, we show empirically that this proposed retriever optimization method improves both evidence retrieval and answer generation for a domain-specific RAG agent.

Primary area:
optimization
Keywords:
Retrieval-augmented generation, Domain-specific question answering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PpRzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ARMOR improves the query encoder in a telecom RAG system while leaving the document encoder, index, and answer model unchanged. It combines RAG likelihood and InfoNCE, learns a temperature for each loss, and penalizes query embeddings that move away from the original encoder's outputs. The experiments cover three telecom domains, and a capacity argument is used to motivate retriever tuning instead of generator tuning.

Critical strengths
  1. Updating only the query encoder avoids rebuilding the document index, and holding out source documents gives a useful test of whether the update generalizes (Sections 2.1 and 5.1).
  2. Table 2 shows that learned temperatures and query distillation work better together than either change alone on ISAC, which gives a concrete reason to study the combination.
Critical weaknesses
  1. The theory uses dS and dR as both parameter counts and sequence lengths, so I cannot see how it supports the claim that fewer trainable parameters give retriever tuning a better bound (Section 2.2; Appendix E.2).
  2. The answer-score changes from Base RAG are small (+0.0226, −0.0044, and +0.0025 on ISAC, JCC, and SAGIN), and without uncertainty estimates it is unclear whether these changes would persist across training runs or test samples (Table 1).
Questions for authors
  1. Are dS and dR parameter counts or sequence lengths, and how does the fourfold parameter reduction relate to the bound after this notation is corrected?
  2. Could you provide the training settings and uncertainty estimates needed to assess the small answer-score differences?
  3. How does ARMOR compare with tuned fixed temperatures and generator tuning on the same test documents?
Reviewer_y4CAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies low-resource, domain-specific RAG for telecommunications QA and argues for adapting only the retriever’s query encoder while keeping the generator and document index fixed. It introduces ARMOR, which jointly optimizes a RAG latent-document likelihood and an InfoNCE contrastive objective using learned temperatures for each softmax and regularizes the adapted query encoder toward the frozen base query encoder to remain compatible with the fixed document space.

Critical strengths

The query-only tuning formulation is well-motivated for operational constraints (frozen doc index and generator), and the method’s core idea—learned, objective-specific temperatures for RAG likelihood and InfoNCE paired with base-compatibility regularization—is a coherent design tailored to this setting.

The evaluation follows a source-document-level split to reduce leakage, a stronger setting than standard example-level splits.

The approach is likely to be of practical interest to teams building domain-specific RAG under low supervision and limited re-indexing budgets.

Critical weaknesses

Learned temperatures act as implicit weights, but the mixture is still a simple sum of two losses; no comparison is provided against established adaptive weighting methods (e.g., uncertainty weighting, GradNorm, CAGrad), which weakens claims about the specific benefits of temperature learning vs. other adaptive schemes.

Main results rely on an LLM-as-judge (“GPT-5.2”) whose identity and availability are unclear; no inter-judge agreement, calibration, or human validation is reported on the main test sets, and significance intervals are absent despite small test sizes (150 examples/domain).

Positives are defined as top-3 chunks from the matched source document; labeling and alignment errors in this heuristic can bias R@k and complicate interpretation (e.g., relevant evidence in another doc counted as false negative).

Some implementation details are deferred (negative mining policy, temperature initialization/floor choices, training schedules, and hyperparameter sensitivity), and λ_q is fixed to 1 without justification or sensitivity analysis.

The description of the judge and prompts is said to be in the appendix, but the main text over-relies on this opaque component for headline claims.

Questions for authors

Why evaluate only the first 150 test examples per domain for generation? Can you report full-test results and/or multiple random subsets with CIs to reduce sampling variance concerns?

How are negatives constructed for InfoNCE (in-batch vs. mined) and how sensitive are results to this choice? Could you include an ablation over negative mining strategies and batch size?

Can you add direct comparisons on the same Tele-Eval split against RAFT, REPLUG, and a recent RL-based retriever optimizer (e.g., HARR), as well as simple but strong baselines like BM25, hybrid sparse+dense, and cross-encoder reranking?

Please report sensitivity to λ_q (currently fixed at 1) and initial/floor temperatures. Does a learned global mixture coefficient (or uncertainty/GradNorm weighting) change the outcome relative to temperature-only adaptivity?

The positive-passage alignment selects top-3 chunks within the matched source document. How accurate is this heuristic? Have you audited labeling noise and measured how often “external” positives (relevant chunks in other documents) exist but are scored as false negatives?

Beyond telecom, have you attempted standard ODQA datasets (e.g., NQ, TriviaQA) or another technical domain to test generality? If not, what adaptations would be required?

Reviewer_NMf8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies retriever adaptation for telecom question answering with limited supervision. It freezes the generator, document encoder, and retrieval index, and updates only the query encoder. The proposed method, ARMOR, combines a document-level answer likelihood objective with InfoNCE, learns a separate temperature for each objective, and regularizes query embeddings toward those of the base encoder. The evaluation covers three telecom subdomains, with additional component ablations, data-size experiments, generator comparisons, and TeleQnA results.

Critical strengths

The fixed-generator, fixed-index setting allows an existing RAG system to adapt through updates to the query encoder alone, which is a realistic deployment constraint for a system that cannot be retrained end to end.

Critical weaknesses

[W1]The paper's claim that query-encoder adaptation outperforms generator adaptation relies solely on Figure 1, which Appendix D.1 states used a different data pipeline from the main experiments; Table 1 lacks any generator-adaptation baseline, making it unclear whether this conclusion holds in the primary setting. There is also an inconsistency in RAFT's classification (generator vs. query-encoder tuning in the caption vs. bar label), requiring clarification. Moreover, ARMOR underperforms base RAG in some cases, indicating that robustness claims need stronger empirical support.

[W2]Table 2 supports using the two components together, but it does not distinguish the benefit of learning temperatures dynamically from that of selecting suitable fixed temperatures. The temperatures, mixture weights, and selection procedure for the static baselines are not adequately reported. A reasonably selected fixed configuration with the same query regularization is therefore an important missing comparison. The gradient interpretation in Appendix B also needs qualification: lowering temperature affects examples differently depending on their margins. The explicit 1/τ factor alone does not establish a stronger contribution from an objective. The reported trajectories and ablations do not yet establish the proposed balancing mechanism.

[W3]Section 2.2 introduces dS and dR as trainable parameter dimensions, but Appendix E.2–E.3 uses them as sequence lengths in the input matrices and token-pooling operations. Parameter count and sequence length are distinct quantities. Consequently, the reported fourfold reduction in trainable parameters does not establish the dimensional condition in Remark 1. The N−3/2 complexity term in Eq. (8) also needs a clear derivation or correction.

[W4]Relative to Base RAG, the main answer-quality gain occurs on ISAC; the JCC and SAGIN results are close to the frozen baseline. The two smaller generators in Figure 4 do not benefit, and the lower-supervision region of Figure 2 does not show a general advantage. The authors acknowledge these limitations, but the paper does not yet identify when adapting with ARMOR is preferable to retaining the base retriever. This limits the practical scope of its low-resource claim.

Questions for authors

Please check the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.