acceptodds
Under review as a conference paper at ICLR 2027

KITE: SPECIALIZING FROZEN MULTIMODAL MODELS WITH A SHARED SIDE-INPUT EXPERT

Abstract

Specializing a multimodal model with external evidence raises a twofold design challenge: how to integrate that evidence without consuming the host’s input sequence, while preserving its behavior when specialization is inactive. We introduce Kite, a shared side-input expert that reads external evidence and host marker states at sparse depths and writes additive updates at those markers. Marker-free requests bypass the branch, preserving host outputs bit-exactly under the same execution configuration; a training-free, prompt-level router controls invocation. We instantiate Kite for Earth observation using frozen DINOv3 features and a frozen 35B host, training a 1.88B-parameter side expert. On a pre-locked new-site recognition test, Kite exceeds a host LoRA matched across the full training chain by +9.8 percentage points. On a separate pre-locked test, mismatched imagery reduces accuracy from 0.429 to 0.106, supporting evidence-sensitive recognition. Benchmark adaptation confined to the expert raises the official VRSBench-VQA ten-category mean from 0.627 to 0.772 and reaches 0.913 on RSVQA-LR. These gains incur additional side computation. We report its cost alongside controlled ablations and a pre-registered negative result on depth-structured evidence use.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Multimodal Language Models, Earth Observation, Specialist Model Grafting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZgwWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Kite, a construction for specialising a frozen multimodal host with external evidence while preserving host behaviour exactly when the specialist is inactive. Frozen DINOv3-7B features are routed as side input, outside the host token sequence, to a shared 1.88B-parameter trunk attached at five MoE stations of a frozen Qwen3.6-35B-A3B host. Prompts carry markers; at each station, marker states query the evidence via cross-attention and the trunk output is written back additively at marker positions through zero-initialised interfaces. Marker-free inputs bypass the branch by control flow, giving bit-exact host preservation (Theorem 1), verified empirically (Δ=0), with prefix KV-cache locality and runtime removability. Training is three-stage: EO expert training on a generated corpus with the host's native vision off; native-path scheduling with a sigmoid gate; benchmark fine-tuning. Deployment invocation uses a prompt-level router; the learned gate is a diagnostic. On a pre-locked new-site test, Kite reaches 0.429 versus 0.258 for a full-chain-matched host LoRA (+9.8pp on corner identification, n=458); mismatched imagery drops accuracy to 0.106, supporting evidence dependence. Benchmark results: VRSBench 0.627→0.772 (seven-category 0.814 vs published Earth-OneVision 0.803), RSVQA-LR 0.913, RSVQA-HR below dedicated specialists, EarthBench agent gains +17/+22pp with mixed tool-trajectory metrics. The paper also reports matched-LoRA ablations (some paired comparisons significantly favour LoRA), evidence-mismatch controls, station-count ablations, a pre-registered negative result on progressive evidence consumption (falsified and reported), a channel-competition analysis motivating "establish then fuse", runtime and memory costs, and a rerun reproducing headline numbers.

Critical strengths

S1. Verifiable exact preservation. Inertness is enforced by control flow rather than by a zero-valued additive branch, so equality is bitwise on a fixed configuration; prefix locality means KV caches computed by the host remain valid for Kite and vice versa; removal is a runtime switch; the marker occupies an unused vocabulary row so logits are unchanged. The property is verified (Δ=0) across every trained model, and the one condition that can break it in deployment — router false-fire — is quantified rather than hidden. Auditability of the inactive path is a real, checkable deployment property that the adapter literature generally does not provide.

S2. Evaluation discipline well above field norm. The pre-registered negative result fixes acceptance criteria before the experiments and reports the falsification; LoRA controls are matched across the full training chain (corpus, stages, step budgets) rather than taken off the shelf; test manifests are locked before evaluation; paired bootstrap CIs are used throughout; a full-recipe rerun reproduces the headline numbers (0.763 vs 0.764; OA 0.845 vs 0.844); leakage audits found 21–41% image overlap with training data, which was fixed and disclosed; scoring protocols are separated and their swings documented. These choices materially raise my confidence in the reported numbers, including the unfavourable ones.

S3. Transferable empirical findings. First, channel competition: an evidence branch co-trained from scratch alongside a strong native vision path is absorbed (write-back scale 0.0918→0.0002), whereas establish-then-fuse preserves it (0.0996→0.0977), with controls (input dropout does not help; a fully trained expert keeps its scale). This is a concrete, mechanism-backed lesson for grafting new input channels onto strong hosts. Second, the pre-registered architecture hypothesis (progressive evidence consumption) was tested with reusable probes and shown false: stations behave as a single on/off unit, with real but unstructured per-station causality. Both findings outlast this particular instantiation.

Critical weaknesses

W1. The title and abstract sell a general construction; every quantitative result comes from one host (Qwen3.6-35B-A3B), one evidence encoder (DINOv3-7B), and one modality (optical EO imagery). No second host or second evidence type appears; SAR and multispectral are explicitly deferred, and the boundary conditions of the channel-competition finding (does establish-then-fuse matter only when the native path is strong?) are unexplored. As written, this is convincingly an EO-specialization system paper; the general-method framing is the least supported part.

W2. Zero-initialized gated write-back is Flamingo; side networks are Ladder Side-Tuning; invocation markers and prefix reuse are Activated LoRA. The paper itself concedes that conditional activation is not unique to Kite and locates the distinction in "one shared expert reading external evidence and host states at sparse depths." Theorem 1 is a control-flow identity. This would be acceptable if the delta were validated across settings, but with n=1 (W1) the methodological novelty is thin.

W3. The +9.8pp headline is corner identification, a 20-class task, on a 458-question subset; under matched benchmark rows and steps on VRSBench, the joint model wins under the official protocol (0.815 vs 0.786) but loses significantly under the authors' frozen-host judge with shared extractors (−0.035, CI excluding zero), RSVQA-LR counting is at parity with LoRA, and on RSVQA-HR the model trails published specialists by a wide margin. The paper reports all of this honestly, but no single unified protocol establishes superiority over matched LoRA, and official protocols here credit a large share of answers by substring matching.

W4. The pipeline requires a 2.27M-row generated corpus with twelve template families, machine-checked acceptance rules and version freezing, three-stage scheduling, a diagnostic gate, and a prompt-level router with its own failure modes (1.3% false-fire under an EO-like header on general prompts, which forfeits exactness in deployment). Nothing in the paper indicates this budget transfers to a new domain at a cost below simply LoRA-tuning the host. The practical case for the construction — exactness plus an auditable evidence pathway — is real, but the paper would be stronger if it claimed that value proposition directly rather than the broader one.

Questions for authors

Q1. Do any results exist on a second host or second evidence modality, however small (a different LLM host, or multispectral tiles)? Relatedly, do you have evidence on when establish-then-fuse is necessary, e.g., whether it matters when the host's native vision path is weak?

Q2. On the 932-question LoRA-comparison subset, how do Kite and matched LoRA compare on the non-corner fields? This would show whether the specialist advantage extends beyond the narrowest task. Also, most configurations are trained once; how sensitive are the small-n paired comparisons (n=458) are to the training seed?

Q3. In deployment, exact preservation holds only when the router withholds markers. What is the operational contract when the router fires incorrectly on a general request — is there an acceptance-time mechanism restoring host behavior, or is bit-exactness conditional on router accuracy?

2 comments
AuthorsOpenReview ↗

Recognition Results, Preservation Guarantees, and Evaluation Scope

Thank you for the review. Given the time constraints of the rebuttal period, we prioritized adding non-corner results, evidence controls, and three-judge scoring across all 37,409 VRSBench questions.

Q1 / W1: Other hosts, modalities, and native-vision strength

Our study focuses on a 35B host and optical Earth Observation (EO) imagery. To directly investigate the interaction between visual pathways, we conducted fixed-checkpoint intervention experiments (§5.3, Appendix E.4). These controls characterize the dynamic interaction between the host visual path and the expert module during inference. Our current experiments isolate and validate the core fusion mechanism at the 35B scale; given time constraints and training costs, determining when the expert must be established prior to fusion is left to future work.

Revised paper: §5.3 (p. 9) and Appendix E.4 (pp. 21–22) describe the intervention experiments; §6 (p. 10) states the scope limitations.

Q2 / W3: Non-corner fields and training seeds

On 466 retained non-corner judgments, Kite answers 176 correctly (37.77%) versus LoRA's 162 (34.76%), an advantage of +3.00 pp.

Across all 924 retained judgments including corner identification (excluding eight data-leakage instances from the initial 932-question set), Kite answers 339 correctly (36.69%) versus LoRA's 280 (30.30%), an overall gain of +6.39 pp.

On 458 corner questions, Kite yields a +9.8 pp gain (35.6% vs. 25.8%); on an independent 198-question subset, the visual-path mismatch rate drops from 0.429 to 0.106.

Regarding reproducibility and seed sensitivity: our paired confidence intervals establish significance across evaluation samples. While 35B multi-seed training is computationally infeasible within the rebuttal period (stated in §6), §4.2 confirms that Kite's gains remain robust across training configurations.

Revised paper: §5.1 (p. 8) adds full non-corner counts; Table 3 (p. 9) separates the recognition subsets. §6 (p. 10) states the seed sensitivity limitation; §4.2 (p. 7) reports reproducibility results from the rerun.

Q3: Router false positives and operational guarantees

We distinguish Kite's architectural guarantee from its empirical routing behavior as follows:

Bitwise preservation: When the router inserts no evidence markers, the expert module is strictly bypassed. Under the stated runtime configuration, the system output remains bitwise identical to the base host model.

If a false-positive insertion occurs, the expert module is activated, returning a specialist prediction rather than the default host output. No mechanism currently checks the answer and restores the host output after an erroneous invocation. If a missed invocation occurs, the system returns the host model's own answer.

The deterministic bypass guarantee and empirical routing accuracy are two distinct concepts.

Paper locations: §3.3, Theorem 1 (p. 4); §4.4, "Autonomous invocation" (p. 8).

W2: Prior components and methodological contribution

Kite builds upon established techniques including zero-initialized branches, side networks, and conditional activation. It combines evidence outside the host sequence with a single shared expert that reads evidence and host marker states at sparse depths.

Its demonstrated value lies in enabling specialist adaptation with an explicit, auditable inactive bypass path, while providing empirical findings on visual-path competition. Under our tested criteria, we found no evidence of progressive depth specialization.

Paper locations: §2 (pp. 2–3); §3.2 (shared expert formulation); Appendix A (depth-specialization tests).

W3: Scoring sensitivity

To eliminate evaluation bias, we evaluated both raw and extracted outputs using three distinct LLM judge models (GPT-4o-mini, the frozen 35B host, and DeepSeek-V4.1-Flash): Kite consistently outperforms LoRA across all judges and response formats, with Avg10 improvements spanning +3.23 to +3.67 pp. This cross-model consistency confirms that Kite's advantage is robust to evaluator choice and answer formatting.For transparency, we retain protocol-specific results and comparisons with dedicated RSVQA-HR specialists.

Revised paper: §5.1 and Table 2 (p. 8); Appendix E.3 (pp. 20–21). Benchmark-specific results remain in §4.2 and Appendix G.

W4: Cost and practical positioning

The value of this method lies in providing an auditable external-evidence interface and exact preservation when idle. The expert module trains 1.88B parameters versus 84.7M for LoRA; estimated visual-path compute costs are 6.8–27 TFLOPs versus 3.8 TFLOPs for LoRA. LoRA is lower in cost. These cost metrics are intended to characterize deployment trade-offs rather than demonstrate superior adaptation efficiency.

Paper locations: §5.1 (p. 8); Appendix E ("What the graft costs").

We thank the reviewer again for the constructive comments and would be glad to address any further questions or concerns.

AuthorsOpenReview ↗

Appreciation for your review and feedback

We sincerely appreciate your supportive evaluation and valuable input, which has helped us further improve the quality of our manuscript. As the discussion period is drawing to a close, we remain happy to provide further clarification if helpful. Thank you again for your time and consideration!

Reviewer_3UdzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces KITE, a shared side-input expert that specializes a frozen multimodal model by reading external evidence and host marker states at sparse depths, then writing additive updates only at marker positions. External evidence remains outside the host’s token sequence, and marker-free requests bypass the expert to preserve host behavior under the same execution configuration. The authors instantiate the approach for Earth observation with a staged training recipe and evaluate it through remote-sensing VQA benchmarks, independent-site recognition tests, and controlled comparisons with LoRA, architectural variants, and mismatched evidence.

Critical strengths
  1. The independent-site recognition results provide meaningful evidence of specialist capability. KITE improves over a LoRA baseline matched across the full training chain by 9.8 percentage points, and a separate image-mismatch test substantially reduces recognition accuracy. Together, these results support a real, evidence-sensitive recognition gain rather than merely better adaptation to benchmark answer formats.

  2. The expert can be disabled without changing the host’s behavior on marker-free requests under the same execution configuration. This is useful when general and specialist requests share a deployment. Explicitly bypassing the branch provides a stronger basis for preserving existing behavior than relying on a learned gate to remain near zero or measuring retention on a few general-purpose benchmarks.

Critical weaknesses
  1. It remains unclear how much the shared, multi-depth architecture contributes beyond the visual features, added capacity, and training recipe. The full-chain LoRA control matches data and training steps, but uses a different visual pathway and substantially fewer trainable parameters. The trunk-free connector has only one station and lacks large-corpus training. Meanwhile, the strongest single-station comparison starts from a five-station checkpoint and removes four stations before continuation; the from-scratch single- and five-station models perform similarly. These comparisons show that the complete system works, but do not establish its advantage over a simpler, sufficiently trained branch using the same visual features.

  2. The VRSBench advantage over LoRA is sensitive to how answers are scored: official scoring favors KITE, while shared answer extraction followed by frozen-host judging favors LoRA. Replacing the joint model’s external evidence with mismatched imagery also barely changes its mean accuracy. This does not invalidate the official results or imply that the model ignores images, since its native visual pathway remains available. However, it leaves uncertain whether the additional branch improves answer quality by supplying useful external evidence or mainly by adding adaptation capacity. The strongest support for the evidence pathway therefore remains the narrower independent-site recognition task.

Questions for authors
  1. Was a multi-station connector without the shared trunk trained using the same DINOv3 features through the large-corpus stage? If so, could you report its independent-site recognition results and training budget? If not, please clarify any specific implementation or training obstacle to this control.

  2. For the joint model, could you provide two targeted checks on the existing VRSBench diagnostic sample: performance with side-evidence features zeroed while keeping markers and native vision unchanged, and a small, model-blinded manual evaluation of cases where official and extracted-answer scoring disagree about the KITE–LoRA comparison?

6 comments
AuthorsOpenReview ↗

Multilayer Connector, Evidence Controls, and Scoring Analysis

Thank you for the thoughtful review. We added a five-station connector, 12 evidence conditions and three-judge scoring on all 37,409 VRSBench questions.

Given the time required for 35B experiments and manual review, full-pipeline training and blinded review remain incomplete. We appreciate your understanding.

Q1: A multilayer connector trained on the large corpus.

We extended the earlier connector to five stations and completed its benchmark evaluation. Under the same scoring pipeline, official ten-category mean accuracy (Avg10) is:

ModelAvg10 (%)
Shared five-station expert72.65
Five-station connector, no trunk71.82
Single-station expert70.01
Native-path LoRA69.40

The shared expert achieved the highest test score, outperforming the other variants by 0.83, 2.64, and 3.25 percentage points (pp), respectively. All these models were trained on the benchmark; minor differences in training configurations are detailed in the Appendix.

Due to time constraints, we prioritized completing this critical benchmark comparison within the limited rebuttal period. The earlier recognition controls are retained in the Appendix. The single-station continuation is initialized from a five-station checkpoint to test its dependence on this initialization.

Paper changes: §5.2, p. 8, and Table 4, p. 9, add the benchmark controls. Appendix E.1, Table 9, p. 20, retains the recognition controls. Appendix E.2, p. 20, gives category scores; Figure 7, p. 21, adds training diagnostics separately from test results. Appendix E.3 describes training differences.

Q2(a): Zero side evidence with markers and native vision unchanged.

We added 12 interventions using one fixed benchmark-trained joint checkpoint and all 37,409 questions from 9,349 images. Native vision and markers are retained in the comparisons below, with common short-answer prompts and decoding settings:

DINO / EO conditionAvg10 (%)
Correct features74.46
Zero features64.57
Fixed training-image mean63.07
Mismatched features, mean of three permutations63.73
Entire EO writeback off66.95

Correct versus zero features improves Avg10 by 9.89 pp (paired image-bootstrap 95% CI [9.22, 10.56]). Correct, zero, mean and mismatched features all execute the EO branch. Turning off its entire writeback gives 66.95%, so enabling it with correct inputs adds 7.51 pp [6.76, 8.29]. Invalid features perform below EO-off, showing that the branch can also interfere with the native pathway.

Both official-rule scoring and semantic-only judging employ DeepSeek-V4.1-Flash, and both consistently support the above conclusions.

Paper changes: §5.3, “Visual-evidence dependence,” and Table 5, p. 9, add these controls. The earlier experiment remains under “Original diagnostic setting.” Appendix E.4, pp. 21–22, defines all 12 conditions and reports paired statistics. §6, p. 10, distinguishes the evaluation settings.

Q2(b): Blinded manual review of scoring disagreements.

While manual review and the diagnosis of the original ranking reversal are still underway, as complementary evidence on evaluator sensitivity, we evaluated the fixed benchmark predictions across three semantic judges using the identical extractor and scoring rules. The extractor receives neither reference answers nor model identities. The Avg10 results (%) are as follows:

Semantic judgeRaw KiteRaw LoRAExtracted KiteExtracted LoRA
GPT-4o-mini72.65269.40372.65069.418
Frozen host72.97669.30972.97469.316
DeepSeek-V4.1-Flash74.79671.28574.79471.283

The frozen host is Qwen3.6-35B-A3B. Kite leads in all six conditions by 3.23–3.67 pp; the DeepSeek gap is 3.511 pp for either answer form. Extraction changes only 7 Kite and 39 LoRA responses. This establishes stable ordering for these predictions, not an explanation of the original reversal or a replacement for blinded manual assessment.

Paper changes: §5.1 and Table 2, p. 8; Appendix E.3, pp. 20–21, provides the scoring protocol. The original scoring discrepancy remains reported in §5.1. §6, “Limitations,” p. 10, explicitly states that blinded human assessment remains unevaluated.

We once again thank the reviewer for the constructive feedback and would be glad to address any further questions or concerns.

AuthorsOpenReview ↗

A Gentle Reminder

Dear Reviewer, thank you again for your careful review and constructive suggestions. We have posted our responses and updated the manuscript with additional experiments and results addressing your concerns. We would greatly appreciate it if you could take a look when you have a chance. We would be happy to clarify any remaining questions and welcome your further feedback.

Thank you for the response and the additional experiments. The new results suggest that most of the gain comes from injecting DINOv3 features at more layers and adding more parameters, both of which are expected to help.

However, what is specific to this paper is not well supported: the five layers are chosen without explanation and, by Appendix A, play no distinct roles; the shared trunk adds about 1.7B parameters for only 0.8 pp; and bit-exact preservation follows directly from skipping the branch. I therefore find the contribution of the method unclear and will keep my score.

Reviewer_SmCLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes Kite, a modular approach to specializing a frozen multimodal model using external visual evidence. A shared expert trunk reads evidence features and question-conditioned marker states at several host depths, then writes additive updates only at designated marker positions. External patches remain outside the host’s input sequence, while marker-free requests bypass the branch and preserve the original host’s behavior under the stated execution conditions. A training-free prompt-level router controls invocation. The Earth-observation instantiation combines a frozen 35B host and frozen DINOv3 encoder with a 1.88B-parameter trainable side expert reused at five depths. Training first establishes the specialist pathway, then introduces native vision, followed by benchmark-specific adaptation. Validation includes locked new-site recognition tests, public VQA benchmarks, routing and preservation checks, and architectural and evidence-mismatch ablations. Kite improves new-site recognition over a full-training-chain LoRA control by 9.8 percentage points and shows substantial evidence sensitivity on a separate recognition test. Benchmark adaptation reaches 0.772 on VRSBench’s official ten-category mean and 0.913 on RSVQA-LR’s three-type mean. The paper also reports substantial computational overhead and negative findings concerning depth specialization and side-evidence dependence in the joint model.

Critical strengths
  1. A concrete modular design with a precisely defined preservation boundary. Kite provides a separate evidence interface while retaining a frozen host and limiting direct updates to a fixed number of marker positions. Marker-free execution bypasses the specialist, making the extension removable and allowing its inactive behavior to be audited. The paper distinguishes this property from zero initialization and explicitly limits the guarantee to the appropriate execution conditions and input positions (§§3.2–3.3). The reported zero logit differences against an independently loaded host, including native-vision inputs, support the implementation (§4.4). This is a useful deployment property for adding specialist capabilities to an established model.
  2. Meaningful specialist gains supported by independent-site and evidence-mismatch evaluations. The strongest empirical evidence is the new-site recognition evaluation. On the 458-question comparison, Kite achieves 35.6% accuracy versus 25.8% for host LoRA matched across the full training chain, a difference of +9.8 percentage points with a 95% interval of [+6.6, +13.1]. On a separate, stricter 198-question subset, replacing the correct side imagery with mismatched imagery reduces accuracy from 0.429 to 0.106 (Table 2). Together, these results support a useful image-dependent specialist capability. The site-separation audits and explicit reporting of unfavorable comparisons and failed mechanism hypotheses further strengthen the credibility and informativeness of the study.
Critical weaknesses
  1. The current controls do not sufficiently isolate the benefit of the defining multi-depth architecture. The strongest station-count comparison starts both continuation arms from a checkpoint pretrained with five stations; the single-station arm then retains only station 31 (Table 3; Appendix B.1). The resulting gap therefore establishes that the pretrained five-station system depends on its existing stations, but does not establish an advantage over a single-station expert trained through the same complete chain while using one station throughout. This distinction matters because the from-scratch comparisons show no detected advantage for five stations: 0.08 versus 0.09 on development recognition, and 0.066 versus 0.056 on the independent test (§5.2). The full-chain LoRA comparison supports the complete system’s utility, but it also changes the visual representation, trainable capacity, and computation: Kite uses DINOv3 side evidence and 1.88B trainable parameters versus LoRA’s 84.7M, with estimated visual-path costs of 6.8–27 versus 3.8 TFLOP (§5.1; Appendix E). These differences are disclosed, but they leave the benefit attributable to the proposed integration mechanism unresolved. A simpler alternative receiving the same evidence and full training exposure would substantially strengthen the architectural contribution.
  2. The joint model’s broader VQA gains are sensitive to evaluation protocol and are not clearly attributable to correctly paired side evidence. On the same VRSBench sample, the official ten-category protocol favors Kite over LoRA, 0.764 versus 0.715. However, the shared extracted-answer evaluation with the frozen-host judge reverses the ordering: 0.705 versus 0.739, with a paired Kite-minus-LoRA difference of −0.035 and a 95% interval of [−0.057, −0.015] (§5.1; Table 8). Because these protocols also differ in their evaluator, this does not establish that answer formatting alone explains the official advantage. It does show that the comparative conclusion is not robust to the reported evaluation choices. Moreover, replacing the joint model’s side imagery with unrelated imagery changes its diagnostic VRSBench accuracy only from 0.716 to 0.714 or 0.715 (§5.3). Individual correctness outcomes do change, so this should not be interpreted as complete non-use of the branch. Nevertheless, the reported aggregate improvement is not shown to require correctly paired external evidence. Given the additional computation, these findings limit the evidence that the proposed side-input mechanism offers a compelling advantage over ordinary adaptation in the final joint setting. The authors acknowledge these limitations, but they remain consequential to the contribution’s scope.
Questions for authors
  1. Has a single-station expert been trained with that architecture throughout the large-corpus and subsequent training stages, rather than obtained by disabling four stations of a five-station-pretrained checkpoint? If so, how does it perform on the same locked new-site tests? This would determine whether the reported gap reflects a benefit of multi-depth training or dependence on the pretrained configuration.
  2. Is there a simpler integration baseline that receives the same DINOv3 features, patch budget, and complete training exposure? For example, a single-station expert trained throughout the full chain would address much of this concern. Its accuracy and computational cost would help establish the benefit of the proposed design beyond the additional visual representation and capacity.
  3. Does the Kite–LoRA ordering persist when the same short-answer extractor is applied before the same official VRSBench scoring pipeline? Alternatively, could a blinded manual assessment of a representative paired sample clarify the discrepancy? The current official and extracted-answer results change both answer processing and the evaluator.
  4. With the native image path, marker positions, and trained expert otherwise unchanged, what happens when the side evidence is zeroed or replaced with a fixed input? Such a control would help assess whether the joint-model gain depends on image-specific information from the external encoder or can largely be retained through the expert’s computation on host states.
2 comments
AuthorsOpenReview ↗

Architectural Comparisons, Answer Processing, and Visual Evidence

Thank you for the careful review. We added architectural controls, three-judge scoring and fixed-checkpoint evidence interventions, evaluated on all 37,409 VRSBench questions.

Given 35B scale computational requirements, we strategically focused our rebuttal resources on the most decisive benchmark controls to directly address your concerns.

Q1–Q2: Single-layer expert and simpler same-evidence baseline.

We added a single-station expert and a five-station connector without an expert trunk. We compared them with Kite and native-path LoRA using the same test questions and GPT-4o-mini scoring pipeline. Avg10 is the official ten-category mean.

ModelAvg10 (%)
Shared five-station expert72.65
Five-station connector, no trunk71.82
Single-station expert70.01
Native-path LoRA69.40

The shared expert leads by 0.83, 2.64 and 3.25 percentage points (pp). These runs cover benchmark training. These results clearly demonstrate the architectural advantage of the shared expert across comparable benchmark training regimes, with detailed training configurations and hyperparameter records documented in Appendix E.3 for complete transparency.

While full-corpus pretraining across the entire pipeline exceeds the rebuttal timeframe, our benchmark evaluations provide strong and immediate evidence of the architecture's efficacy. The single-station continuation explicitly validates the contribution of multi-station initialization. Full large-scale pipeline runs will be integrated in the camera-ready version.

Paper changes: §5.2, p. 8; Table 4, p. 9. Recognition controls: Appendix E.1, Table 9, p. 20. Category scores: Appendix E.2, p. 20; training diagnostics: Figure 7, p. 21. Training differences: Appendix E.3; costs: Appendix E, p. 19.

Q3: A common extractor and scoring pipeline.

We evaluate fixed benchmark predictions before and after one frozen extractor, which receives the question and response but no reference answer or model identity. All conditions use common deterministic rules: substring matching, exact matching for closed answers, then semantic judgment under the same rubric. Only the semantic judge changes.

Semantic judgeRaw KiteRaw LoRAExtracted KiteExtracted LoRA
GPT-4o-mini72.65269.40372.65069.418
Frozen host72.97669.30972.97469.316
DeepSeek-V4.1-Flash74.79671.28574.79471.283

Scores are Avg10 percentages; the frozen host is Qwen3.6-35B-A3B. Kite leads in all six conditions by 3.23–3.67 pp; DeepSeek gives 3.511 pp for either answer form. Extraction changes 7 Kite and 39 LoRA responses, consistent with their concise outputs.

The results across the three independent semantic judges demonstrate that, once extraction formatting artifacts are eliminated and evaluation rubrics are unified, Kite consistently exhibits a robust and stable performance lead (+3.23 to +3.67 pp). Due to the limited rebuttal timeframe, full manual review is still ongoing, and we retain the original protocol-specific comparison for completeness.

Paper changes: §5.1, “Answer-processing and evaluator robustness,” and Table 2, p. 8; Appendix E.3, pp. 20–21, provides the extractor, scoring rules and detailed results. The original comparison remains in §5.1, “Public-benchmark adaptation.”

Q4: Zero or fixed side features with native vision retained.

We added 12 conditions using one fixed joint checkpoint, all 37,409 questions from 9,349 images, and common short-answer/decoding settings. With native vision and markers retained, Avg10 is:

DINO / EO conditionAvg10 (%)
Correct features74.46
Zero features64.57
Fixed training-image mean63.07
Mismatched features, mean of three permutations63.73
Entire EO writeback off66.95

Correct versus zero features yields +9.89 pp (paired image-bootstrap 95% CI [9.22, 10.56]). Correct features with EO writeback enabled versus disabled yield +7.51 pp [6.76, 8.29]. The training mean is computed per spatial position from training images; mismatch mappings have no self-pairs, and all questions for an image share its mapping (Appendix E.4). Zero/fixed/mismatched inputs still execute the branch; their lower scores than EO-off show that invalid side inputs can interfere with the native pathway.

Both official-rule scoring and semantic-only scoring use DeepSeek-V4.1-Flash and support image-specific side-evidence use with native vision available. This is an intervention on the benchmark-trained model; the revised paper retains the original diagnostic and distinguishes the settings.

Paper changes: §5.3, “Visual-evidence dependence,” and Table 5, p. 9; Appendix E.4, pp. 21–22, reports all conditions, intervention definitions and paired statistics. §6, p. 10, updates the conclusion.

We once again thank the reviewer for the constructive feedback, and we would be pleased to address any further questions.

AuthorsOpenReview ↗

A Gentle Reminder

Dear Reviewer, thank you again for your careful review and constructive suggestions. We have posted our responses and updated the manuscript with additional experiments and results addressing your concerns. We would greatly appreciate it if you could take a look when you have a chance. We would be happy to clarify any remaining questions and welcome your further feedback.

Comments on the paper

1 comment

Dear Reviewers,

Please read the authors' response. Please evaluate whether your concerns are addressed or if anything needs further attention. Please engage with the authors while we have time.

Thanks, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.