- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2(+1)
- Soundness
- 1
Read review
Paper summary
The paper links demographic cues in Gemma-generated summaries to human valuation. Across 839 review pairs, Black-cue sentiment is 1.061 points lower. DPO uses 800 training and 200 development pairs. Among 775 focal survey completions, the reference-cell WTP gap changes across baseline and DPO summary conditions.
Critical strengths
-
The demographic-cue–text–valuation pathway is a meaningful contribution.
-
Paired generation holds products and reviews fixed; within-condition randomization separates summary effects from displayed-photo effects.
-
Broader face-panel checks and limitations of the generator comparison are disclosed.
Critical weaknesses
-
Gemma's 32.9-point gap on 12 faces shrinks to 1.3 points on the 400-image panel (q = .623). This limits generalization of the focal diagnostic, not the separate summary and valuation findings.
-
The approximately-zero diagnostic gap lacks precise uncertainty. Baseline group means are reported, but post-DPO absolute scores, source faithfulness, and capability retention are needed to distinguish reduced disparity from degraded summaries. Teacher identity and training hyperparameters also require clarification.
-
As Appendix G acknowledges, generator conditions do not isolate weight changes from cohort and generation-run differences. The additional, non-preregistered interaction supports attenuation for these fixed summary sets, not a general causal effect of DPO.
-
Only 16 products are used. Product selection and marginal effects across disclosure conditions should be reported; broader product-level claims require stimulus-level uncertainty.
Questions for authors
-
What are the post-DPO gap intervals, absolute scores, and faithfulness results?
-
How were generator cohorts and the 16 products selected?
-
Do marginal effects and product-level sensitivity analyses support the reference-cell result?
What the face benchmarks and summary comparisons show, and how buyer valuation gaps change with tuned summaries
Thank you for your time and detailed comments. We will upload an updated pdf soon.
1. The 12-face and UTKFace results. The focal diagnostic uses standardized, neutral-expression faces from Project Implicit’s Race-IAT materials, with gender balanced across groups and the same prompt repeated 50 times per face. Gemma’s Black-White gap was 32.9 percentage points on those 12 faces (95% CI [15.7, 48.4]). As you pointed out, on UTKFace, the gap was 1.3 points in the same direction. We looked further into this and it is because UTKFace has additional variance in age, expression, pose, and lighting, and its age and gender labels are estimated (Zhang et al., 2017). We will describe the UTKFace finding more narrowly in the revision. FairFace served a separate purpose. We matched the FairFace images before constructing the fine-tuning pairs to reduce noise in age, gender, background, and expression.
2. The near-zero gap result. In the 12-face run on tuned Gemma, every recorded descriptor was negative for both groups. The Black–White gap was zero and the between-group test was undefined. In a separate 12-face follow-up, ORPO fine-tuning yielded a gap of about -1 percentage point. Our main contribution comes from the buyer experiment. Randomly assigning summaries showed that demographic differences in generated text can change what buyers will pay for the same product. The smaller valuation gap for the tuned summary set shows how changing the text buyers receive can reduce that disparity in the tested setting. We used rank-8 LoRA. Two independent undergraduate research assistants manually reviewed separate batches of 500 preference pairs each.
3. Review-summary quality and faithfulness. For the texts buyers saw, mean sentiment on the 1-10 scale was as follows.
In a blind assessment of 16 summary pairs, two independent undergraduate raters unanimously agreed that every Black-cue version was more negative and verified that both versions drew evidence from the original reviews, just picked different words.
4. Survey groups and buyer valuation. English-speaking U.S. adults recruited on Prolific evaluated the same 16 products using either the untuned summary set (532 completions) or the DPO-tuned-model summary set (243 completions). Within each survey, we randomized the summary version and the displayed seller photo independently. With a human source label, no representativeness statement, and a White seller photo displayed, the Black-minus-White willingness-to-pay effect was −USD 0.688 for untuned summaries and −USD 0.022 for the DPO-tuned-model summary set. The gap narrowed by USD 0.666 (95% CI [0.162, 1.171], p<.01). This additional analysis compares the two fixed summary sets across survey groups. Within-survey randomization establishes that the generated summary can change what buyers will pay for the same product. The valuation gap was smaller for the DPO summary set in a marketplace task with real payoffs.
5. Product selection and marginal effects. Before the buyer study, three independent raters selected everyday items where possible and prioritized review summary pairs with visibly different summary wording. A significant average sentiment difference does not guarantee a noticeable difference for each product. The budget limited us to 16 products and the USD 6-15 product price range for the purchase lottery. The products were glass jars, cord organizers, a kitchen towel set, reusable hooks, push hooks, labels with a marker, magnets, measuring cups and spoons, a meat thermometer, wall décor, a microwave cover and silicone mat, motion-sensor lights, a table runner, a second kitchen towel set, trash bags, and an electric wine opener.
Averaging over the same disclosure and displayed-photo conditions in both groups, the Black-minus-White willingness-to-pay effect was −USD 1.247 for untuned summaries and −USD 0.056 for the DPO summary set. That is a USD 1.191 narrowing of the gap (95% CI [0.985, 1.397]). The gap also narrowed in each of the six disclosure conditions. We refit the regression after dropping each possible pair of products (120 combinations) for cross-validation. Comparing the untuned summary results to DPO summaries shifted the Black-minus-White willingness-to-pay gap by +USD 0.957 to +USD 1.331 (all p<.01).
Zhang, Zhifei, Yang Song, and Hairong Qi. 2017. “Age Progression/Regression by Conditional Adversarial Autoencoder.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.