acceptodds
Under review as a conference paper at ICLR 2027

ReCast: Contract-Preserving Protection for Fixed-Interface Multimodal Reasoning

Abstract

Remote multimodal models offer strong numerical reasoning capabilities over charts and speech, but sending private inputs risks exposing sensitive content. Text-only sanitization cannot directly satisfy fixed media interfaces, while identity anonymization leaves the underlying task content exposed. We introduce Recast, an agentic plug-in framework that replaces source-specific content while preserving task-relevant relations and the required input modality. Recast locally converts inputs into a shared textual evidence–query record, jointly rewrites entities and topics with a distilled 4B model, and substitutes values through a locally invertible, role-aware numerical map. A reconstruction agent generates and validates the required media from the protected record. The remote solver returns a program whose protected operands are restored locally before execution. On 4,000 held-out ChartQA and NMSQA examples, Recast achieves 75.10% accuracy, retaining 92.43% of unprotected remote accuracy, while a model-based audit flags source-content retention in 7.95% of solver-bound requests. It outperforms all evaluated local baselines, preserving the benefit of remote reasoning while reducing source-content exposure under existing media interfaces.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Privacy-Preserving Inference, Multimodal Reasoning, Arithmetic Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hfyzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RECAST, a contract-preserving framework for reducing source-content exposure in multimodal numerical reasoning through fixed remote interfaces. The method locally extracts a textual evidence-query record, jointly rewrites source-specific topics and entities, remaps numerical values with a locally invertible map, reconstructs protected chart or speech inputs, and executes a returned program after restoring the original operands locally. The engineering design is reasonable, and the assessment scope covers both graphical and audio tasks.

Critical strengths
  1. The paper focuses on the "source-content exposure" issue in fixed interface remote multimodal services. For services that require uploading images, charts or voice inputs, merely generating sanitized text usually fails to meet the interface requirements; RECAST has designed a deployable processing flow to address this practical limitation, with the task setting having a relatively clear application background.
  2. The shared evidence-query record and local recovery of original operands give the system a clear utility objective: the remote model identifies operations over protected inputs, while the client restores the original operands before final execution. This separation is well matched to numerical-reasoning tasks that require preserving relational structure.
Critical weaknesses
  1. RECAST primarily integrates existing text rewriting, reversible numerical substitution, media reconstruction, and local program execution. The paper does not provide sufficient component-level baselines to show that the shared record, joint rewriting, and reconstruction agent yield gains beyond a simple composition of existing components.
  2. The retention audit covers only solver-bound requests and excludes external reconstruction or review services that may process protected records or derived artifacts. Therefore, the reported retention rate does not characterize the source-exposure risk of the complete pipeline.
  3. The judge-based retention metric measures whether a fixed judge flags source information as retained, but it does not directly assess attack success. In particular, the evaluation does not test source retrieval, linkage through preserved relational or numerical patterns, or attribute inference.
Questions for authors
  1. Could the authors add simple-composition baselines, for example by combining an existing numerical-protection method with an off-the-shelf chart renderer or TTS system? Could they also ablate the independent contributions of the shared record, joint rewriting, and reconstruction agent?
  2. Could the authors evaluate source retrieval, linkage, or re-identification under an end-to-end threat model that includes all external reconstruction and review services, rather than relying only on judge-based retention?
Reviewer_89JTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses a practical challenge in working with private multimodal data: remote models offer strong reasoning capabilities, but uploading raw inputs risks exposing sensitive information, while local models may struggle to achieve sufficient accuracy. The authors propose ReCast, which first converts charts or speech into a local text record containing the evidence and question. A distilled 4B model jointly rewrites entities and topics across this record, and a reversible mapping replaces the numerical values. The system then reconstructs the protected record in the media format required by the remote service and checks its consistency. The remote model generates a program to solve the rewritten problem; before executing it, the client restores the original numerical operands to recover the answer to the original question. On ChartQA and NMSQA, the paper reports an overall accuracy of 75.10%, retaining 92.43% of the accuracy achieved by the remote model on unprotected inputs. A model-based audit flags source-content retention in 7.95% of requests sent to the remote solver.

Critical strengths

S1. The problem is practically important, and the framework is well motivated.

The paper addresses a realistic challenge: users want to benefit from strong remote multimodal models without exposing private inputs, while local models may lack sufficient reasoning capabilities. ReCast considers source-content protection, evidence–query consistency, and the input modality required by fixed media interfaces. The shared textual evidence–query record makes intermediate content inspectable and supports replaceable components. Local operand recovery provides a practical way to reuse remote-generated programs for the original task. Overall, the framework is clearly structured and closely aligned with the requirements of the setting.

S2. The reported results are promising, with clear benefits from distillation.

On ChartQA and NMSQA, ReCast retains 92.43% of Remote-native accuracy, while the model-based audit flags source-content retention in 7.95% of solver-bound requests. Distillation provides substantial gains: compared with the untuned Qwen3-4B rewriter, the distilled local rewriter improves overall accuracy by 6.40 percentage points and reduces source-content retention from 39.75% to 7.95%, with the other pipeline components held fixed. The authors also calibrate the retention judge against human annotations on 400 stratified pairs, reporting Cohen’s κ\kappa = 0.83 against the human majority, which provides empirical support for the audit metric.

Critical weaknesses

W1. Several inconsistencies in the experimental reporting require clarification.

W1.1. Inconsistent evaluation sizes. The paper reports 2,000 examples each from ChartQA and NMSQA (L334–336), with all methods using the same 4,000 sample IDs (L360–362). However, Appendix A.1 and Table 6 specify 1,000 examples per dataset (L697–705). Please clarify the final evaluation sizes and the denominators used for the reported results.

W1.2. A contribution statement contradicts the results. The contribution list claims that text release improves ChartQA accuracy by 10.60 percentage points over chart re-rendering (L98–99). However, Table 5 reports 69.80% for protected text and 70.20% for protected media (L455–458), and the accompanying discussion explicitly states that text is 0.40 points lower (L460–464). Please explain the source of the 10.60-point claim.

W1.3. Inconsistent dataset count. The paper states that five datasets are used but identifies only four: ChartQA, NMSQA, ChartX, and Spoken-MQA (L333–337).

W2. Some aspects of the experimental validation remain insufficient.

W2.1. Protection effectiveness is assessed primarily through a model-based audit. Human calibration provides useful support for the audit, but independent attack evaluations are still needed. Examples include sensitive-value recovery and joint inference across multiple queries about the same source data. Reporting attack success rates would provide stronger evidence of the framework’s protection effectiveness.

W2.2. Decisions under numerical transformation need targeted validation. The mapping does not preserve general differences, ratios, or cross-class comparisons. For tasks such as identifying the largest increase or comparing ratios, the remote solver could select the wrong operands based on transformed values, yielding an incorrect answer even after operand recovery. The paper acknowledges that these decisions should be deferred until after restoration, but how this is enforced remains unclear. Please clarify the supported task types and provide targeted evaluations.

W2.3. End-to-end costs are not evaluated. The pipeline involves multiple local models, remote services, and potentially several repair rounds. Although timeout and retry settings are provided, measured latency, remote-call counts, and resource consumption are not reported. These measurements would help assess the framework’s deployment feasibility.

Questions for authors

Please refer to the Critical Weaknesses section.

Reviewer_6M73OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RECAST, a pipeline for sending modified multimodal inputs to remote models while keeping original content local. It converts inputs to text, rewrites entities and topics, remaps numbers, and reconstructs the required modality. The remote model returns a program whose numerical operands are restored locally before execution.

Experiments on ChartQA and NMSQA report lower source-content retention while retaining much of the unprotected accuracy. The paper also includes component ablations and tests on two unseen datasets.

Critical strengths
  1. The method converts charts and speech into a common text format, so the same rewriting and number-protection steps can handle both. The code shows substantial engineering work in connecting these steps with media reconstruction and checks.

  2. Training the local 4B rewriter improves answer accuracy from 68.70% to 75.10% and reduces measured original-content retention from 39.75% to 7.95%. This shows that the training helps change the content while keeping information useful for answering, though it does not by itself establish privacy protection.

Critical weaknesses
  1. The privacy claim is not well supported. The paper measures how much original content remains, but does not clearly define what must be protected or what an attacker could recover. Changing names, topics, and numbers may still leave sensitive clues in the preserved relationships. Low content retention alone does not establish privacy. There can be implicit content thats more private. A more detailed analysis of whether privacy has been accurately maintained seems crucial to the very motivation of this work.

  2. The technical novelty and comparisons are limited. Much of the method breaks the task into familiar LLM steps: extraction, rewriting, reconstruction, and checking. The paper does not clearly establish what is new beyond combining them. It also lacks comparisons with simpler complete protection pipelines. Related approaches such as Hide and Seek and PAPILLON could provide useful adapted baselines after the same text-extraction step. The comparisons with other methods are incomplete. They do not report answer accuracy alongside privacy scores under the same conditions, so it is unclear whether RECAST offers a better balance between privacy and accuracy. Without these comparisons, the benefit of the added complexity is unclear.

  3. Answer recovery has an important limitation. Restoring original numbers cannot fix a wrong operation or category selected using the modified values. Errors in the initial text extraction can also affect every later step. The evaluation does not sufficiently show when these failures occur or whether the checks catch them.

  4. The release does not support full reproduction. The code shows substantial engineering work, but data and several core implementations are missing. In my inspection, the checks also allowed invalid rewrites into training data. It would also be nice if the authors can consider releasing the training (distillation) data to help the larger community.

Questions for authors
  1. How often do text extraction and rewriting introduce errors, and how reliably do the checks catch them? Are withheld requests counted as incorrect in the reported accuracy?

  2. Do you have comparisons with other protection methods that measure both answer accuracy and privacy under the same conditions? This would help establish RECAST’s benefit over simpler approaches.

  3. Why is the current evaluation enough to support the privacy claim? Could sensitive information still be inferred from the relationships that RECAST preserves, even after names, topics, and numbers are changed?

  4. Could changing the numbers cause the solver to choose the wrong calculation or category? Restoring the original numbers would not fix that choice. How does RECAST handle this, and have you tested such cases?

  5. Can you clarify the missing core implementations and whether the 15,795 distillation pairs will be shared? My initial inspection also found invalid rewrites that passed the training-data checks. How were such cases handled in the reported training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.