- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies a meta-measurement problem: the single "demographic effect" number reported by audits of LLM hiring discrimination — a figure that carries legal force under New York City's Local Law 144 — and asks how much of it depends on design choices the audit leaves unreported, rather than on the model itself. The authors accordingly propose a minimum reporting set and a screening rule with a closed-form calibrated false-positive rate, and release all artifacts; every number in the text is interpolated from those artifacts at build time.
Critical strengths
-
The paper cleanly separates instrument sensitivity from repeat-measurement noise, and provides actionable engineering controls.
-
The reproducibility infrastructure is itself a contribution.
Critical weaknesses
W1. The paper does not answer: would a different set of wordings yield a similar magnitude? The twelve wordings were chosen by the authors, and the choice of wording set is itself an unreported researcher degree of freedom. Adding more names improves the precision of the estimate under each wording, but does not address how the wording set was selected. The paper convincingly shows that this set of wordings moves the conclusion; whether the 25%–64% magnitude generalizes to other equally defensible wording sets remains untested. A wording-set sensitivity analysis (split-half, leave-one-out, or an independently generated new set) would address this.
W2. The headline number is not easy to audit. The six model-by-posting cells behind the "25%–64%" range, their per-cell uncertainty, and whether the replicate noise floor was subtracted are scattered across the text and tables, with ambiguous accounting (§4 and §5 describe the floor correction inconsistently). A single per-cell summary table — effect, between-wording SD, ratio with interval, and the floor-correction convention — would resolve this.
W3. There is an inferential gap between the screening rule and the field-level conclusion. Moving from "no checkpoint in this panel passes the rule" to "no wording-robust disparity has yet been measured by anyone on these checkpoints" overreaches what this study's name list, postings, wording set, and limited power can support. Moreover, the rule's false-positive rate is derived under an equicorrelated null that conflicts with the paper's own finding that ~70% of between-wording variance is model-specific with near-zero cross-family correlation; and the degree to which true disparities are missed under this conservative rule is never quantified. The conclusion should be scoped to this panel, and a power analysis under the measured correlation structure should be added.
Questions for authors
-
How were the twelve wordings constructed, and how sensitive is the between-wording SD to the composition of this set? Would a split-half or leave-one-out analysis over the wording set (or an independently generated second set) leave the 25%–64% headline range materially unchanged?
-
Is the 25%–64% range for the open-weight panel reported before or after subtracting the byte-identical replicate floor? Could you provide a single per-cell table listing, for each of the 6 identified model-by-posting cells: the effect, the between-wording SD, the ratio with its bootstrap interval, and the floor-correction convention applied?
-
The headline range is computed only over cells whose effect is separable from zero. How should readers interpret the excluded cells — does conditioning on an identified denominator bias the reported ratios upward, and if so, can you bound that bias?
-
Under the measured between-wording correlation structure (rather than the equicorrelated null), what is the power of the screening rule — i.e., the probability that a true disparity of a given size passes the "significant under every wording" requirement? How does the closed-form false-positive rate change under block-structured or near-zero correlation?
-
Would you agree to scope the conclusion to this panel's design constants — i.e., "on these checkpoints, with this name list, postings, and wording set, no wording-robust disparity has been measured" — rather than the current broader phrasing?
-
The recommendation to build token-matched name lists from the outset is well taken, but how should auditors preserve construct validity (e.g., perceived-race ratings à la Gaddis 2017) when assembling such lists? Do you see a risk that token-matching constraints systematically select weaker racial signals?
-
Local Law 144 audits are typically expressed as impact ratios on selection rates. How does the log-odds probability-of-superiority metric used here map to that statutory quantity, and what do the 2.0×–484× conversion errors imply for an impact-ratio-based audit conclusion?
6 comments
Response to Reviewer W4hz (2/4): W2, Q2, Q3
W2 / Q2. Is the range reported before or after subtracting the replicate floor? A single per-cell table.
The single per-cell table is Table 1. For each of the six identified cells, four on open weights (one of them, Mistral-7B v0.3, registered nurse, is identified only under the percentile interval; §10) and two on the API, it gives the effect, the between-wording SD, the ratio with its name-pair bootstrap interval and the replicate floor, and the ratio under three conventions side by side: nothing subtracted (the published range, 25% to 64%), the floor subtracted in quadrature (22% to 55%), and the name-pair sampling variance removed as well (15% to 44%). The headline range is before any subtraction, and §4 and the caption of Table 1 now state that convention. To see the ratios with no floor to subtract, we re-ran the business-analyst grid on all four open-weight checkpoints with one request in flight and the prompt cache off, where the byte-identical pair agreed on every call. There, the two identified checkpoints give 22% and 29%, against 25% and 33% in the original run. That floor of zero holds within one machine state. It compares the pair within one run, so it cannot see the variation between machine states that App. F reports. Table 4 lists every cell, identified or not.
Q3. Does conditioning on an identified denominator bias the ratios upward, and can the bias be bounded?
We simulated it at each identified cell's measured covariance. The filter itself pulls the ratio down, not up, because it discards the draws whose small denominators would inflate it. What pushes the published ratio up is sampling noise in the numerator, which is there with or without the filter. With the true ratio set to each cell's R_τ, the median published ratio at the observed effect sizes is 0.98 to 3.80 times each draw's true ratio, largest on GPT-4o. R_τ, the ratio net of name-pair sampling noise, is the 15% to 44% range above; the top of the published range, GPT-4o's 64%, becomes 15%. Because the simulation fixes the true ratio at R_τ, it shows the direction and size of the published ratio's inflation, not the error of R_τ itself, which rests on the moment identity in App. B. Of the cells left out, GPT-4.1 cannot be measured, and the other nine have effects inseparable from zero. Their between-wording SDs on the PS scale, 0.038 to 0.131, span the identified cells' 0.057 to 0.108, so the dispersion is still there where the ratio is undefined (§4, App. B).
(continued in 3/4)
Response to Reviewer W4hz (3/4): W3, Q4, Q5
W3 / Q5. Scope the conclusion to this panel's design constants.
We agree and have scoped the conclusion to this panel's design constants. Part 3/3 of the general response quotes the new text.
W3 / Q4. The false-positive rate assumed an equicorrelated null, against the finding that most between-wording variance is model-specific; power under the measured structure; block and near-zero correlation.
The original conflated two different correlations, and §9 now separates them. The 70% figure comes from correlating wording effects across models. The rule depends on the correlation between two wordings' estimates on one model, which the shared name pairs induce (0.18 to 0.71 on the identified cells). Because the rule passes only when every single-wording test rejects on the same side, its false-positive rate is at most one test's level, whatever that correlation is. At each cell's measured covariance, it is at most 1.03×10⁻³; under the equicorrelated stand-in, 1.65×10⁻³; under a two-block structure separating paraphrases from null edits, 1.46×10⁻³; under near-zero correlation, no simulated draw passes.
Power at a given effect under the measured structure is the new Table 7. Against a true effect of 0.10 on the PS scale at 96 name pairs, the rule passes 54% to 89% of the time under each cell's measured covariance with no between-wording heterogeneity, and 16% to 69% with each cell's R_τ held (τ = R_τ × 0.10, below every cell's measured τ), against 75% to 94% on the equicorrelated grid of Table 6, which overstates power once heterogeneity is counted. At the panel's own twelve name pairs, with or without heterogeneity, the rule passes a 0.10 effect at most 5.1% of the time and a 0.20 effect at most 45.2% (Table 7), and a cell at its own measured effect passes 0% to 79% (Table 5, as in G2). The grid of Table 6 varies sample size and correlation at twelve wordings; the number of wordings varies only in Table 8.
(continued in 4/4)
Response to Reviewer W4hz (1/4): W1, Q1
Thank you for a review that proposed concrete tests; we ran all of them, including an independently written set. W and Q numbers follow your review.
The twelve were fixed in the study's first version and have not changed: six paraphrases of the screening question under one system text (advance to a first-round interview, recommend advancing, a good enough match, move forward to a phone screen, proceed, invite to interview), and six edits of the first paraphrase that change no word (one byte-identical, the others adding a trailing newline or space, doubling the space after each colon, putting a space in each blank line, or swapping the two system sentences). We wrote all twelve, as the Limitations now say. They are not a sample of wordings. The tests below ask how much the result depends on which ones we wrote.
The range is not carried by any single wording, and each arm alone gives a range of similar order, though single cells move a lot between arms and one set of six can land well away from the full set's value. R divides by each subset's own pooled effect, so it moves with the effect as well as the spread. The SD divides by nothing. Over all twelve wordings, it is 0.057 to 0.108 on the PS scale. Dropping any one wording changes a cell's SD by at most 33.1% (Mistral-7B v0.3, registered nurse; at most 16.2% on every other cell). Each cell's median over the split-halves is 0.91 to 1.02 times its full-set value, with the middle nine-tenths of halves within 0.44 to 1.37 times. The paraphrases alone give 0.051 to 0.118, and the null edits alone 0.034 to 0.121 (App. C).
For an independently written set, rather than write a second set ourselves, we used wordings other teams wrote before this paper: the six screening instructions that five of the surveyed audits print, each edited by one stated rule to ask for yes or no (App. C.1 prints each with every edit). They ran on the business-analyst posting of the four open-weight checkpoints, where two of the six headline cells sit, with the same résumés and names, under the deterministic configuration. Their between-instruction SD is 0.067 to 0.116, against 0.045 to 0.106 for our six paraphrases. Theirs is larger on every checkpoint, but on none is the difference distinguishable from zero; the intervals exclude theirs being more than 0.14 larger and allow it to be up to 0.073 smaller (Table 10). On these checkpoints, then, the spread is no smaller in point estimate on another set.
Where the effect sits does change. On Llama-3.1-8B, the published instructions' pooled PS is 0.46, not separable from 0.5, against 0.17 for ours; on Mistral-7B v0.3 only the published set's is separable (0.38 against 0.45). In a comparison added after the first results, and labelled post hoc in the paper, the two sets' pooled effects differ on two of the four checkpoints, Llama-3.1-8B among them and Mistral-7B v0.3 not. Because R divides by the effect, R on another set can be larger, or undefined. On Llama-2-7B, the one checkpoint where both sets identify the effect, it is 66% for the published set, above our range, against 37% for ours. So on this posting another set gives no sign of a smaller SD, though it cannot exclude one up to 0.073 smaller; the ratio does not transfer, because it depends on the set through its denominator. The Limitations now say that the published set spreads the effect no less widely in point estimate but moves where it sits, and that six edited instructions are not a sample of every wording an auditor might use.
(continued in 2/4)