acceptodds
Under review as a conference paper at ICLR 2027

Certifying Culturally Faithful Generation: Feasibility, Ordering, and Release Gates Beyond Calibration

Abstract

Text-to-image systems deployed in cultural heritage settings can produce images that are visually plausible yet culturally unfaithful, substituting generic or neighboring traditions for the requested one. A common mitigation is selective generation: sample several candidates, score them with a verifier, and emit the highestscoring one. But what evidence justifies releasing such a system? Risk-control methods can calibrate a frozen score, yet release has two logically prior failure modes: the candidate pool may contain no culturally acceptable output, or an acceptable output may exist but the verifier ranks another candidate above it. We formalize this distinction with request-level estimands—infeasibility I, mixedpool mass M, and within-mixed-pool selection success γ—which obey the exact decomposition Rsel = I +M(1−γ). The identity is elementary; its value is attributional: the two actionable sources of risk become separately measurable. On top of this attribution layer we define a frozen-policy certification protocol: the operating point is chosen on development data only, the policy is frozen, and release is accepted if and only if an exact one-shot upper bound on the total selected-failure count x = f + o meets the target; Bonferroni component lower bounds only attribute rejections. Common release metrics do not identify the certified quantity: with four candidates, 80.5% pairwise accuracy is compatible with a top-one hit rate from 0% to 100%, and flattening into per-request risk control cannot recover the decomposition. On CulturalFrames, at the prespecified endpoint, the frozen SigLIP verifier’s ordering-error component has a one-sided 95% lower bound of 15.5%—above the full 10% risk budget—and a bounded Qwen2.5-VL repair fails its one-shot held-out gate. LiveCodeBench fails feasibility (I ˆ = 0.292 > 0.10); three benchmark rows pass, including a RewardBench certificate with exact upper risk bound 2.83%. Synthetic validation confirms the gates open on non-degenerate K > 2 pools, and a risk–coverage frontier view certifies coverage ≈ 0.90 at target 0.10. Our cultural audit is a negative, endpoint-conditional diagnosis; we do not claim a benchmark-native positive certificate for non-degenerate cultural pools.

Primary area:
generative models
Keywords:
selective generation, risk certification, cultural T2I, conformal risk control, release gates

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_onAiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper decomposes selective-generation risk into infeasible candidate pools and selection errors within mixed pools, using this distinction for diagnosis while certifying total risk with an exact binomial bound. CulturalFrames and LiveCodeBench illustrate different failure sources; three simpler benchmarks pass certification, and synthetic experiments examine nondegenerate pools and risk–coverage tradeoffs.

Critical strengths
  1. Acceptance uses total selected failures, while component bounds explain rejection, avoiding invalid combinations of uncertain component estimates.
  2. The paper provides useful identification results. Sharp bounds show why pairwise accuracy cannot reliably establish top-one success and why selected-only labels cannot establish improvement over random selection.
  3. In cultural failure diagnosis, SigLIP makes 21 avoidable selection errors among 94 emitted requests, with an adjusted lower bound of 14.4% exceeding the entire risk budget under the stated independence assumption.
Critical weaknesses
  1. In Table 2, there are multiple 'tbc's, which I suspected is some unfinished results?
  2. The usefulness of attribution for choosing repairs is insufficiently demonstrated. All positive benchmark rows are K=2, all-mixed cases; comparing generator and verifier interventions on a common cohort at matched coverage would better establish whether the diagnosis predicts effective improvements.
  3. The paper sweeps 60 thresholds without specifying simultaneous correction or independent confirmation of the selected threshold, so pointwise 95% bounds alone do not justify the claimed maximum certifiable coverage.
Questions for authors
  1. Can you provide Qwen’s cohort sizes and evaluation stages? The development interval matches mixed-pool successes, which appears inconsistent with a nominal 20% development split of 203 pools; please explain the cohort and distinguish the reported evaluation from the unscored sealed test.
  2. How are candidate-level cultural labels aggregated, and why was 0.75 selected?
Reviewer_87VTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Proposing to differentiate between when a model ranks plausible outputs vs. when no plausible output exists.

Critical strengths

None

Critical weaknesses

There are multiple issues with this work.

First, I suspect this entire ‘paper’ was entirely LLM generated with little human supervision, as quite frankly it doesn’t make much sense. Consider for example table 2 which contains “TBC” in multiple cells of the table, with the caption stating “tbc” marks counts to be completed from audit logs. Despite this, I will still engage with the paper and give it a review although the submitters lack of effort and waste of our time and resources should not be ignored.

Related work is missing so much literature on reward models, existing related methods for bias probing, verification of T2I outputs, and the models and evaluation metrics being used. Moreover, no citations throughout most of the paper to relevant work.

Considering this is about verifying outputs of text-to-image models, I’m surprised that there isn’t a single qualitative example. Moreover, the work is motivated by a specific setting termed cultural heritage, yet, there is no clear definition or examples of this. The first sentence states “Generative systems are increasingly used to depict cultural heritage: traditional attire, rituals, architecture, crafts, and everyday practices” without any citations or examples of how T2I models are used for such cases. As I mentioned, I suggest you use this as one of many use cases, rather than the only one.

What does this sentence mean? “A risk certificate is only as meaningful as the endpoint it controls: trustworthy cultural generation must be grounded in inheritor- or community-authorized cultural rules—the people whose tradition is depicted are entitled to specify what counts as faithful.”

Each sentence should be clearly understood and stand alone. The writing in this paper is incredibly verbose and unclear. Similarly, the contributions section is poorly written and does not stand alone.

“Threshold sweeps that reuse held-out outcomes are diagnostic frontier traces, not certificates. A certified operating point must be selected on development data and certified once on untouched evaluation data. Reporting multiple thresholds requires either a pre-registered grid with simultaneous exact bounds, or an additional sample-splitting step that separates threshold selection from final certification” -> This finding isn’t new.

“An emitted pool is infeasible if all candidates fail, mixed if it contains both passing and failing candidates, and safe” -> I’m not sure your usage of “safe” here is the correct term. Some suggestions to improve this. I encourage the others to take the underlying principles of their motivating example and broaden the category. Surely, the need to differentiated between a model scoring/ranking an image as last vs. no solution exists translates across image generation, even domain tasks overall.

Questions for authors

None.

Reviewer_HohFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper focuses on the issue that text-to-image generation process does not guarantee the cultural alignment. The authors tried to make generation selection as a risk certification problem. It separates failure into infeasible pools and selection errors within mixed pools, and uses a frozen policy certification to accept/reject the candidate.

Critical strengths

The problem of a global ranking selection emthod is not enough is valid.

Critical weaknesses
  • The logic flow of the paper is hard to follow.
  • The work is missing quite some contexts: e.g. no citations for the benchmarks they are using.
  • Not consistent story: from the beginning, they said they are studying culturally acceptable output, but the benchmark they are using, e.g., LiveCodeBench, HumanEvalPack are coding tasks. I don't know how they related to cultural issue. But I need to point it out, they do not provide the citations to these works.
  • The paper is incomplete — for example Table 2, it said clearly there is placeholder ‘tbc’ marks ‘counts to be completed from audit logs’.
  • Tables 1 and 4 extend into the right margin
Questions for authors

n/a

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.