acceptodds
Under review as a conference paper at ICLR 2027

SCM-Bench: An Auditable Benchmark for Context Efficiency in Multi-Turn Dialogue

Abstract

Context efficiency in multi-turn dialogue concerns the answer produced from a particular history, not only the amount of history retained. SCM-Bench defines a versioned target-turn task and an auditable framework for studying history retention, selection, compression, and model routing. Deployment decisions use the current request and strictly preceding legal history; evaluation uses frozen requirements, anonymous answers, and complete legal history. The framework distinguishes reference-history budgets from provider-native generation usage and shared-matrix construction from policy deployment. Its historical profile contains 3,893 deployment-eligible inputs from 261 sessions, but candidate selection is not a completed formal evaluation. Eight context interfaces and independent versus selected-context routing support controlled comparisons without implying learned joint optimization. The proposed analysis preserves failures, missing outcomes, and constraint applicability. Available development re-evaluations do not supply a qualified quality–efficiency release.

Primary area:
datasets and benchmarks
Keywords:
LLM routing, context selection, model routing, multi-turn dialogue, datasets and benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_58nCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SCM-Bench, a benchmark task and evaluation framework for studying context efficiency in multi-turn dialogue. It defines versioned target-turn records and a protocol for comparing history selection, compression, and model routing in terms of answer quality and resource use.

Critical strengths
  1. The paper raises a relevant and practical question about evaluating context efficiency in multi-turn dialogue.

  2. The proposed framework covers a broad range of evaluation needs, including data versioning, information boundaries, resource accounting, and answer assessment.

Critical weaknesses
  1. Version management, frozen experimental materials, resource attribution, and information separation are useful for reproducibility and evaluation integrity. However, these are largely standard experimental practices. The paper does not clearly demonstrate how their integration provides new evaluation capabilities or methodological advances beyond existing evaluation frameworks.

  2. The current development diagnostics and retrospective re-evaluation of previous answers do not establish whether the framework enables more reliable comparisons between strategies or produces meaningful new research findings. Although the authors acknowledge these experimental limitations, the central research question remains insufficiently addressed.

Questions for authors
  1. Could the authors clarify the main methodological contributions of the proposed framework beyond standard reproducibility and evaluation practices, particularly version management, frozen experimental materials, resource attribution, and information separation? What new evaluation capabilities does the framework provide compared with existing approaches?

  2. Could the authors provide additional experimental evidence demonstrating that the framework enables more reliable comparisons between strategies or leads to meaningful new research findings, rather than primarily supporting development diagnostics and retrospective re-evaluation?

  3. Could the authors clarify how the reliability of the review system is established? In particular, how are invalid votes and evaluation protocol errors identified and handled, and what evidence supports the validity of the resulting quality labels?

  4. Could the authors explain how the 3,893 target-turn inputs were validated beyond structural filtering? Specifically, how are task quality, dependence on conversation history, and scoring validity assessed, and what criteria will be used to select the formal evaluation subset?

  5. Could the authors improve the readability of Figure 1 by aligning the connecting arrows, increasing the text size, and simplifying the overall layout?

Reviewer_mBd3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SCM-Bench compares the effects of context policies and model routing on answer quality and resource use at fixed target turns. Deployment receives only the current query and permitted history, while assessment uses full history, previously frozen requirements, and anonymized answers. The framework defines eight context interfaces and four context/routing combinations, separates reference-history budgets from actual generation usage, and retains failures, unknown outcomes, and constraint inapplicability. The reported evidence includes a historical profile of 3,893 eligible inputs, development re-evaluation of 400 existing answers for 40 targets, and judge-qualification diagnostics that do not pass the overall gate. The authors do not claim established quality preservation, deployment savings, or generalization to unseen conversations.

Critical strengths
  • The comparison unit and information flow are clearly defined. The current query remains fixed, history is restricted to preceding messages on the true ancestral branch, and evaluation requirements and historical labels are withheld from deployment policies. Static target-turn replay is also explicitly distinguished from online dialogue evolution. These choices help explain what a context intervention changes.
  • Resource accounting addresses common omissions. Required summarization, generative routing, answer generation, and retries are included, while shared offline matrix construction is distinguished from the costs each policy would incur in deployment. Reference-history tokens, provider-native usage, and latency are also kept separate. These are practically useful conventions.
  • Unknown outcomes are treated thoughtfully. Assigned units remain in the analysis, unknown and inapplicable outcomes are separated, bounds represent unresolved labels, and resource savings use a ratio of totals. The paper correctly distinguishes identification intervals from sampling confidence intervals and nonsignificance from evidence of non-inferiority.
  • Adverse diagnostics are reported candidly. The manuscript retains 192/1,049 invalid votes, 62/400 unknown task outcomes, failed role gates, and an overall evidence-binding failure. Even aggregate task decisions of 120/120 are not presented as evaluator qualification. This gives a concrete illustration of the difference between correct aggregate decisions and a trustworthy evaluation procedure.
Critical weaknesses
  • A complete formal benchmark object is still missing. Candidate sets, input profiles, and development re-evaluations exist, but a formally evaluated population and final version manifest are not identified; Fresh and Validated Challenge are both empty. The paper explicitly provides no qualified paired results answering its intended comparison questions. The issue is research completion: the design has not yet become a sufficiently complete benchmark deliverable.
  • Quality assessment is neither fully integrated nor qualified. The v1 scoring protocol invokes the third judge only under specified conditions, whereas the V5 aggregator requires exactly three distinct role, model, and family identities. The authors acknowledge that these protocols remain unintegrated. Existing qualification also fails the overall evidence-binding gate, and all 62 unknown task outcomes involve critical protocol errors. This directly affects whether the benchmark can produce trustworthy labels; higher aggregate agreement alone would not resolve it.
  • Complete execution records do not yet substantiate the resource definitions. The manuscript describes missing historical and formal runtime objects, and existing answer-token comparisons do not include a complete policy ledger. Readers can understand how usage should be accounted for, but cannot yet inspect a complete comparison demonstrating those conventions in practice. The mapping among required calls, actual model identities, usage, and quality versions remains a substantive gap.
  • Important parts of the statistical analysis are still proposed. The existing comparator clusters by session, while cross-session lineage and inference for partially identified endpoints need further support. Non-inferiority margins, significance levels, comparison families, and resampling settings remain unfrozen. This does not invalidate the bounds or ratio-of-totals formulas, but it leaves no complete common procedure through which independent implementations would reach the same formal inference.
Questions for authors
  • Which already existing inputs, requirements, and version manifest constitute the smallest usable benchmark? If these are in the referenced appendices or companion materials, please identify them and distinguish candidate, development, and formal-evaluation status.
  • Which evaluator version is actually bound to the reported execution? How are v1's conditional third judge and V5's three-identity requirement reconciled, which rules do existing qualification results correspond to, and has any version passed all gates?
  • Does any subset of the 400 answers or other reported records have a complete required-call ledger consistently bound to its inputs, policies, and quality version? Please identify existing coverage, or confirm that only definitions and partial records are currently available.
  • Are the target population, cross-session lineage clustering, unknown-endpoint inference, and non-inferiority and multiplicity settings fixed in an existing version? Which components remain proposed?
Reviewer_aY5wOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SCM-Bench to test how the history shown to a model affects its next answer and the resources needed to generate it. The protocol replays a fixed conversation turn, varies the history or the model, and scores each answer against requirements drawn from the full preceding conversation. The authors identify 3,893 eligible requests from 261 sessions. A development study re-scores 400 earlier answers to 40 requests across five history conditions and two models.

Critical strengths
  • Scoring answers against the full conversation is a useful design choice. It allows the benchmark to detect earlier user constraints lost when history is shortened.

  • The proposed cost accounting includes summarization and retries, avoiding misleading efficiency comparisons based only on final prompt length.

Critical weaknesses
  • The scoring procedure is not sufficiently validated. Protocol errors left 62 of 400 pilot outcomes unresolved, and the evaluation panel failed its evidence-binding check despite matching all 120 expected task decisions. Without independent human validation, the reliability of the answer-quality scores remains uncertain.

  • The empirical evidence remains preliminary. The available results do not pair validated answer quality with complete usage records, making it difficult to assess whether shorter histories preserve earlier user requirements at lower cost.

Questions for authors
  • How many of the 400 pilot answers have independent human ratings? Among those reviewed, how often did reviewers agree with the panel’s pass/fail decisions, and how did they score cases the panel left unresolved?

  • For requests with verified scores and complete usage records in both settings, how often did the recent-history answer miss an earlier requirement that the full-history answer met? How did total token use differ when all calls and retries were counted?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.