acceptodds
Under review as a conference paper at ICLR 2027

The Instrument Is Not the Model: Measuring How Much of an LLM Hiring Disparity Comes from Unreported Design Choices

Abstract

Audits of large language models for hiring discrimination carry legal weight: New York City's Local Law 144 makes it unlawful to use an automated employment decision tool that has not been audited for bias. Such an audit reports a demographic effect, meaning how differently a model treats two resumes differing only in the name, and whatever the two share should cancel. It does not, by enough to change what an audit concludes. Holding the model fixed and varying only choices published audits do not report, over 31,468 matched pairs from 64,397 calls on 6 open-weight and 5 frontier checkpoints, the instruction wording moves the effect by 25% to 64% of itself across the 6 cells where the effect is separable from zero, including under edits that change no word; where the sign flips, the resume template outweighs the wording. Only 25% to 33% of name pairs from a standard validated list are token-matched. Choices made after the data exists matter as much. Resampling rows rather than name pairs narrows intervals 3.0 to 4.8x, and a fixed operating point misstates the percentage-point conversion by 2.0x to 484x. The measurement itself reproduces bitwise only once request batching and cache residency are controlled. Of 13 LLM hiring audits read in full text, none reports enough to reconstruct its number. We give the minimum an audit must report, a screening rule with a calibrated false-positive rate, and the full pipeline.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
algorithmic audit, fairness, hiring, large language models, measurement validity, reproducibility, nondeterminism, reporting standards

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_W4hzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies a meta-measurement problem: the single "demographic effect" number reported by audits of LLM hiring discrimination — a figure that carries legal force under New York City's Local Law 144 — and asks how much of it depends on design choices the audit leaves unreported, rather than on the model itself. The authors accordingly propose a minimum reporting set and a screening rule with a closed-form calibrated false-positive rate, and release all artifacts; every number in the text is interpolated from those artifacts at build time.

Critical strengths
  1. The paper cleanly separates instrument sensitivity from repeat-measurement noise, and provides actionable engineering controls.

  2. The reproducibility infrastructure is itself a contribution.

Critical weaknesses

W1. The paper does not answer: would a different set of wordings yield a similar magnitude? The twelve wordings were chosen by the authors, and the choice of wording set is itself an unreported researcher degree of freedom. Adding more names improves the precision of the estimate under each wording, but does not address how the wording set was selected. The paper convincingly shows that this set of wordings moves the conclusion; whether the 25%–64% magnitude generalizes to other equally defensible wording sets remains untested. A wording-set sensitivity analysis (split-half, leave-one-out, or an independently generated new set) would address this.

W2. The headline number is not easy to audit. The six model-by-posting cells behind the "25%–64%" range, their per-cell uncertainty, and whether the replicate noise floor was subtracted are scattered across the text and tables, with ambiguous accounting (§4 and §5 describe the floor correction inconsistently). A single per-cell summary table — effect, between-wording SD, ratio with interval, and the floor-correction convention — would resolve this.

W3. There is an inferential gap between the screening rule and the field-level conclusion. Moving from "no checkpoint in this panel passes the rule" to "no wording-robust disparity has yet been measured by anyone on these checkpoints" overreaches what this study's name list, postings, wording set, and limited power can support. Moreover, the rule's false-positive rate is derived under an equicorrelated null that conflicts with the paper's own finding that ~70% of between-wording variance is model-specific with near-zero cross-family correlation; and the degree to which true disparities are missed under this conservative rule is never quantified. The conclusion should be scoped to this panel, and a power analysis under the measured correlation structure should be added.

Questions for authors
  1. How were the twelve wordings constructed, and how sensitive is the between-wording SD to the composition of this set? Would a split-half or leave-one-out analysis over the wording set (or an independently generated second set) leave the 25%–64% headline range materially unchanged?

  2. Is the 25%–64% range for the open-weight panel reported before or after subtracting the byte-identical replicate floor? Could you provide a single per-cell table listing, for each of the 6 identified model-by-posting cells: the effect, the between-wording SD, the ratio with its bootstrap interval, and the floor-correction convention applied?

  3. The headline range is computed only over cells whose effect is separable from zero. How should readers interpret the excluded cells — does conditioning on an identified denominator bias the reported ratios upward, and if so, can you bound that bias?

  4. Under the measured between-wording correlation structure (rather than the equicorrelated null), what is the power of the screening rule — i.e., the probability that a true disparity of a given size passes the "significant under every wording" requirement? How does the closed-form false-positive rate change under block-structured or near-zero correlation?

  5. Would you agree to scope the conclusion to this panel's design constants — i.e., "on these checkpoints, with this name list, postings, and wording set, no wording-robust disparity has been measured" — rather than the current broader phrasing?

  6. The recommendation to build token-matched name lists from the outset is well taken, but how should auditors preserve construct validity (e.g., perceived-race ratings à la Gaddis 2017) when assembling such lists? Do you see a risk that token-matching constraints systematically select weaker racial signals?

  7. Local Law 144 audits are typically expressed as impact ratios on selection rates. How does the log-odds probability-of-superiority metric used here map to that statutory quantity, and what do the 2.0×–484× conversion errors imply for an impact-ratio-based audit conclusion?

6 comments
AuthorsOpenReview ↗

Response to Reviewer W4hz (1/4): W1, Q1

Thank you for a review that proposed concrete tests; we ran all of them, including an independently written set. W and Q numbers follow your review.

W1 / Q1. How were the twelve wordings constructed, how sensitive is the between-wording SD to the set's composition, and would split-half, leave-one-out or an independently generated set leave the 25% to 64% range materially unchanged?

The twelve were fixed in the study's first version and have not changed: six paraphrases of the screening question under one system text (advance to a first-round interview, recommend advancing, a good enough match, move forward to a phone screen, proceed, invite to interview), and six edits of the first paraphrase that change no word (one byte-identical, the others adding a trailing newline or space, doubling the space after each colon, putting a space in each blank line, or swapping the two system sentences). We wrote all twelve, as the Limitations now say. They are not a sample of wordings. The tests below ask how much the result depends on which ones we wrote.

Wording setRange of R across the identified cells
All twelve (as published)25% to 64%
Any one dropped20% to 68% (the largest relative change to one cell is 40%, on Mistral-7B v0.3, registered nurse)
All 462 split-halves, each cell's median25% to 63%
One half of six, 5th to 95th percentile15% to 83%
The six paraphrases alone22% to 75%
The six null edits alone27% to 69%

The range is not carried by any single wording, and each arm alone gives a range of similar order, though single cells move a lot between arms and one set of six can land well away from the full set's value. R divides by each subset's own pooled effect, so it moves with the effect as well as the spread. The SD divides by nothing. Over all twelve wordings, it is 0.057 to 0.108 on the PS scale. Dropping any one wording changes a cell's SD by at most 33.1% (Mistral-7B v0.3, registered nurse; at most 16.2% on every other cell). Each cell's median over the split-halves is 0.91 to 1.02 times its full-set value, with the middle nine-tenths of halves within 0.44 to 1.37 times. The paraphrases alone give 0.051 to 0.118, and the null edits alone 0.034 to 0.121 (App. C).

For an independently written set, rather than write a second set ourselves, we used wordings other teams wrote before this paper: the six screening instructions that five of the surveyed audits print, each edited by one stated rule to ask for yes or no (App. C.1 prints each with every edit). They ran on the business-analyst posting of the four open-weight checkpoints, where two of the six headline cells sit, with the same résumés and names, under the deterministic configuration. Their between-instruction SD is 0.067 to 0.116, against 0.045 to 0.106 for our six paraphrases. Theirs is larger on every checkpoint, but on none is the difference distinguishable from zero; the intervals exclude theirs being more than 0.14 larger and allow it to be up to 0.073 smaller (Table 10). On these checkpoints, then, the spread is no smaller in point estimate on another set.

Where the effect sits does change. On Llama-3.1-8B, the published instructions' pooled PS is 0.46, not separable from 0.5, against 0.17 for ours; on Mistral-7B v0.3 only the published set's is separable (0.38 against 0.45). In a comparison added after the first results, and labelled post hoc in the paper, the two sets' pooled effects differ on two of the four checkpoints, Llama-3.1-8B among them and Mistral-7B v0.3 not. Because R divides by the effect, R on another set can be larger, or undefined. On Llama-2-7B, the one checkpoint where both sets identify the effect, it is 66% for the published set, above our range, against 37% for ours. So on this posting another set gives no sign of a smaller SD, though it cannot exclude one up to 0.073 smaller; the ratio does not transfer, because it depends on the set through its denominator. The Limitations now say that the published set spreads the effect no less widely in point estimate but moves where it sits, and that six edited instructions are not a sample of every wording an auditor might use.

(continued in 2/4)

AuthorsOpenReview ↗

Response to Reviewer W4hz (2/4): W2, Q2, Q3

W2 / Q2. Is the range reported before or after subtracting the replicate floor? A single per-cell table.

The single per-cell table is Table 1. For each of the six identified cells, four on open weights (one of them, Mistral-7B v0.3, registered nurse, is identified only under the percentile interval; §10) and two on the API, it gives the effect, the between-wording SD, the ratio with its name-pair bootstrap interval and the replicate floor, and the ratio under three conventions side by side: nothing subtracted (the published range, 25% to 64%), the floor subtracted in quadrature (22% to 55%), and the name-pair sampling variance removed as well (15% to 44%). The headline range is before any subtraction, and §4 and the caption of Table 1 now state that convention. To see the ratios with no floor to subtract, we re-ran the business-analyst grid on all four open-weight checkpoints with one request in flight and the prompt cache off, where the byte-identical pair agreed on every call. There, the two identified checkpoints give 22% and 29%, against 25% and 33% in the original run. That floor of zero holds within one machine state. It compares the pair within one run, so it cannot see the variation between machine states that App. F reports. Table 4 lists every cell, identified or not.

Q3. Does conditioning on an identified denominator bias the ratios upward, and can the bias be bounded?

We simulated it at each identified cell's measured covariance. The filter itself pulls the ratio down, not up, because it discards the draws whose small denominators would inflate it. What pushes the published ratio up is sampling noise in the numerator, which is there with or without the filter. With the true ratio set to each cell's R_τ, the median published ratio at the observed effect sizes is 0.98 to 3.80 times each draw's true ratio, largest on GPT-4o. R_τ, the ratio net of name-pair sampling noise, is the 15% to 44% range above; the top of the published range, GPT-4o's 64%, becomes 15%. Because the simulation fixes the true ratio at R_τ, it shows the direction and size of the published ratio's inflation, not the error of R_τ itself, which rests on the moment identity in App. B. Of the cells left out, GPT-4.1 cannot be measured, and the other nine have effects inseparable from zero. Their between-wording SDs on the PS scale, 0.038 to 0.131, span the identified cells' 0.057 to 0.108, so the dispersion is still there where the ratio is undefined (§4, App. B).

(continued in 3/4)

AuthorsOpenReview ↗

Response to Reviewer W4hz (3/4): W3, Q4, Q5

W3 / Q5. Scope the conclusion to this panel's design constants.

We agree and have scoped the conclusion to this panel's design constants. Part 3/3 of the general response quotes the new text.

W3 / Q4. The false-positive rate assumed an equicorrelated null, against the finding that most between-wording variance is model-specific; power under the measured structure; block and near-zero correlation.

The original conflated two different correlations, and §9 now separates them. The 70% figure comes from correlating wording effects across models. The rule depends on the correlation between two wordings' estimates on one model, which the shared name pairs induce (0.18 to 0.71 on the identified cells). Because the rule passes only when every single-wording test rejects on the same side, its false-positive rate is at most one test's level, whatever that correlation is. At each cell's measured covariance, it is at most 1.03×10⁻³; under the equicorrelated stand-in, 1.65×10⁻³; under a two-block structure separating paraphrases from null edits, 1.46×10⁻³; under near-zero correlation, no simulated draw passes.

Power at a given effect under the measured structure is the new Table 7. Against a true effect of 0.10 on the PS scale at 96 name pairs, the rule passes 54% to 89% of the time under each cell's measured covariance with no between-wording heterogeneity, and 16% to 69% with each cell's R_τ held (τ = R_τ × 0.10, below every cell's measured τ), against 75% to 94% on the equicorrelated grid of Table 6, which overstates power once heterogeneity is counted. At the panel's own twelve name pairs, with or without heterogeneity, the rule passes a 0.10 effect at most 5.1% of the time and a 0.20 effect at most 45.2% (Table 7), and a cell at its own measured effect passes 0% to 79% (Table 5, as in G2). The grid of Table 6 varies sample size and correlation at twelve wordings; the number of wordings varies only in Table 8.

(continued in 4/4)

Reviewer_UAjeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how analysis choices (e.g., names, prompt wording, resume templates etc) can affect the demographic disparity reported by LLMs in hiring scenario. They have tested 13 LLM audits and showed none of them can fully report the information needed to reconstruct the disparity. They then propose a reporting checklist and screening criterion for disparity claims.

Critical strengths
  • Controlled measurement to examine the multiple sources of variation in hiring-audit results.
  • Their analyses over resampling, batching and cache setting is interesting.
  • They provide a guidance for documenting an audit.
Critical weaknesses

My biggest concern is the contribution over related work: As author has mentioned some of these observations have been documented in exsiting work. I have a hard time to find substantive insights from this work. It reads to me like an intergration analysis over known sensittives. In addition while the reporting survey identifies documentation gaps, but does not show whether the additional controls significantly change conclusions of existing audit protocal.

Questions for authors

Please refer to the concern above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer UAje: contribution and evidence

Thank you for naming the resampling, batching, and cache analyses as a strength. First, one correction to your summary: We did not run the 13 audits; we read and coded them in full text (§9). The experiment below runs six of their instructions on our checkpoints.

W1. Contribution over related work; the paper reads as an integration over known sensitivities.

Prompt sensitivity is documented, and §2 cites that work. The paper adds three things, each checkable in the revision.

First, it separates how much of a spread across prompts is instrument and how much is noise. A spread across prompts scored on a dozen name pairs grows with the sampling noise of those pairs even when every prompt measures the same thing. We read it against a byte-identical replicate floor (on the identified cells the between-wording SD is 2.0 to 5.1× that floor) and remove the name-pair sampling variance, after which the spread is still 15% to 44% of the effect (Table 1). The closest prior check, Gaebler et al. (2024), rewords one prompt four times on one model and judges from its intervals that the effect moves only slightly beyond estimation error. Neither it nor the other works we cite estimate how much of a spread remains once that error is removed (§2).

Second, it tests whether the spread reaches conclusions. Scored the way the city's rules prescribe for a score, single-wording audits of the same cell can land on both sides of four-fifths (G3), and the surveyed audits' own instructions reach different conclusions on the same checkpoint (below).

Third, it says what an auditor should do, with error rates: a rule whose false-positive rate is bounded under any correlation, with its power, compared with single- and multi-prompt designs at equal budget (G1, G2).

W2. The survey identifies documentation gaps but does not show whether the additional controls significantly change the conclusions of an existing audit protocol.

Within our design, we ran the audits' own instructions, though not their full protocols; that is the closest test the design allows (§9, App. C.1). Under an inclusion rule fixed before any call, five surveyed audits supply six screening instructions that judge one candidate for a job and that one stated edit rule turns into a question answered by yes or no alone; the appendix gives the reason each other printed instruction is out. Edited that way, they ran on our four open-weight checkpoints with the same résumés and names, so the instruction was the only thing that varied.

Tested one at a time without correction, as an audit using one would test it, one instruction finds a significant disparity on Llama-2-7B, two find one on Mistral-7B v0.3, where none of our paraphrases do, and none finds one on Llama-3.1-8B, where all six of ours do. Corrected within the six, only the first survives, and no two instructions reach opposite conclusions. Which conclusion a single-instruction audit reports on these checkpoints depends on whose instruction it ran, and a reader can see that only if the instruction and the spread across instructions are reported, the first item of the reporting set. The rule establishes none of the single-instruction findings, because the published six pass it on no checkpoint. Its verdict depends on the set as well: our six paraphrases pass it on Llama-3.1-8B only, and in the deterministic re-run our twelve miss there by one wording. That is why the set itself has to be reported.

Not every control changes conclusions, and the paper says which do not. Controlling batching and the cache changed no per-wording PS conclusion and no cell's identification (§5), and at each model's own threshold the name rarely changes a decision (G3). Four kinds of conclusion do change. Whether a single-instruction audit finds a disparity depends on whose instruction it ran (above). Which side of four-fifths a single-wording scoring audit lands on varies with the wording, partly through name-sample noise (G3). Whether the effect is identified at all depends on the response format: none of the three checkpoints identified when answering directly (one on the marginal résumé alone) stays identified when the model explains first, and the move in PS is separable from zero on each (§6, App. E). And significance verdicts flip with the resampling unit, since resampling rows instead of name pairs narrows intervals 3.0× to 4.8× (Table 2).

We would be glad to answer anything still open before the discussion closes.

Thanks. I've read the author rebuttal and lean to my original score.

Reviewer_68PhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper measures how unreported design choices affect demographic effects reported by LLM hiring audits. Holding the model fixed, it varies instruction wording, resume templates, and analysis-time choices to examine effects on the resulting estimates. The authors review existing audits and propose a minimum reporting set and a screening rule with a calibrated false-positive rate. They suggest that audits should report these choices alongside the results.

Critical strengths
  1. The paper quantifies how unreported design choices affect the reported demographic effects. The reporting set provides a practical checklist for LLM hiring audits, which helps readers understand how the results were obtained and which settings are needed for evaluation reproduction.

  2. The byte-identical repeats establish a useful noise floor and strengthen the evidence that wording sensitivity exceeds numerical noise. The null edits further show that changes to spaces or line breaks can affect the measured disparity without changing the words.

Critical weaknesses
  1. It is unclear what the proposed protocol adds compared to existing multi-prompt evaluations. The paper indeed compares individual design choices, but does not clearly show whether the full protocol leads to more reliable conclusions. The screening rule is evaluated for false positives, while its ability to detect real disparities remains insufficiently tested.

  2. The current results mainly rely on scores from constrained Yes/No responses. It is unclear whether the findings hold when models first explain their assessment and then make a decision. Changes in these scores also do not necessarily change the final screening outcome, so their practical impact needs further validation.

Questions for authors
  1. It would be better to involve the existing evaluation here as the baseline to show your add-ons. How does the proposed protocol improve on a single-prompt audit or a conventional multi-prompt evaluation under comparable budgets? Could the authors also report the screening rules's power across different sample sizes and wording correlations?

  2. Do the findings hold when models explain their assessment and then provide a Yes/NO decision? A comparison with direct constrained decoding would benefit the experimental analysis.

  3. How about reference labels about resumes meeting the job requirement? Could you please report baseline acceptance rates and whether log-odds changes affect final decisions across qual levels. When most resumes are consistently accepted or rejected?

  4. Could you please clarify how the evaluations differ between open-weight models and commercial APIs, including how scores are obtained and verdict-only output are handled?

  5. How often do changes in batching or cache reuse alter the final Yes/No decisions or the statistical significance of the demographic effect?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 68Ph (1/3): W1, Q1

Thank you for pressing on what the protocol adds and on whether a margin matters if the decision does not move. Both led to new analyses. W and Q numbers follow your review; G1 to G3 refer to the general response.

W1 / Q1. What does the protocol add over single-prompt and multi-prompt evaluation at comparable budgets, does it give more reliable conclusions, and what is the rule's power across sample sizes and wording correlations?

The equal-budget comparison is G1 and Table 8. That table separates the two designs that control the false-claim rate. The variance-carrying interval, which §9 reports beside the rule, makes a false claim 0% to 3% of the time, against 22% to 59% for a single prompt and 5% to 26% for the pooled multi-prompt interval at the heterogeneity G1 states. It pays in power: 76%, against 95% for the single prompt in G1's example. The every-wording rule has power 4% in the same example, below that interval at six and twelve wordings but above it at three, where the interval has only two degrees of freedom (Table 8). Empirically, the published instructions of the surveyed audits reach different conclusions from ours on the same checkpoint (G1). The rule's false-positive rate under the measured, block and near-zero correlation structures is in G2. Power over sample sizes and correlations at twelve wordings is in Table 6; the number of wordings varies only in Table 8. Power under each cell's measured covariance is in Table 5 at the cell's own effect and in Table 7 at a given effect. At twelve name pairs, the power at a cell's own effect is at most 10% on four cells, and 41% and 79% on the two cells that miss the rule by one wording (Table 5). That is why the paper now reads "no disparity passes the rule" as a statement about what this design can establish.

(continued in 2/3)

AuthorsOpenReview ↗

Response to Reviewer 68Ph (2/3): W2, Q2

W2 / Q2. Do the findings hold when the model first explains its assessment and then decides, compared with direct constrained decoding?

We ran it on the business-analyst grid of the deterministic re-run (twelve wordings, three résumé levels, and twelve name pairs on the four open-weight checkpoints), with each wording's closing "Answer yes or no." replaced by "Explain your assessment in two or three sentences, then give your decision as 'Decision: yes' or 'Decision: no'." A grammar fixes that format, and the margin is read at the decision token from the top hundred tokens of the unconstrained distribution, as for a direct answer, so the assessment and the decision come from one call. We recorded the design, the estimators, and the comparisons before the run. An interval on the difference between the two modes, and the rule that decisions are compared on every level, were added after it started and before any estimate was computed (App. E).

The between-wording SD on the PS scale is 0.059 to 0.188 when the model explains first, against 0.056 to 0.084 when it answers directly. There are no checkpoints where it is detectably smaller. The largest fall the intervals allow is 0.076, on Llama-2-7B, whose direct SD is 0.084, and on Mistral-7B v0.3 it is detectably larger. These SDs include the name pairs' sampling noise, which is larger explaining first on every checkpoint. Net of that noise, the point estimate of τ, computed after the estimates had been seen, falls on both Llama checkpoints, to zero on Llama-3.1-8B, and rises on both Mistral ones.

The effect itself moves. On the three checkpoints identified when answering directly (one read on the marginal résumé alone), explaining first moves the PS by between 0.29 and 0.44, to within 0.03 of 0.5. Each move is separable from zero, and none of the three stays identified, so R is undefined for them in that mode. Mistral-7B v0.1 is identified only when explaining first, though its two modes are not separable.

Decisions move as well. Verdicts agree with the direct ones on 46% to 95% of résumés. The two names of a pair get different verdicts in 4% to 16% of pairs, against 0% to 6% when answering directly; more pairs split explaining first on every checkpoint but Mistral-7B v0.1 (4% against 6% there). Pooled over wordings and levels, the decision gap explaining first has an interval that excludes zero on two checkpoints, both against the White-named résumés. Black-named résumés advance 4.2 points more often on Llama-3.1-8B (interval 1.2 to 6.9) and 3.9 points more often on Mistral-7B v0.3 (0.2 to 8.1). An exact swap test over name pairs, which the plan did not include, gives p = 0.035 and 0.125, and 0.14 and 0.25 after Benjamini-Hochberg correction within the four checkpoints.

So explaining first changes the effect it measures, which makes the response format one more choice an audit has to report (§6). Whether it lowers the sensitivity to wording, these intervals cannot say. App. E states three limits. On Mistral-7B v0.3, the margin is censored on 284 of 288 strong-résumé calls and on one of the 288 middle ones, so that checkpoint's margins are compared on the marginal level alone. 83% of Llama-3.1-8B's assessments reach the 600-character cap. And the byte-identical pair disagreed on all twenty-four of Llama-3.1-8B's marginal-résumé calls when explaining first. On its strong and middle résumés alone, where the pair agreed (a check added after the estimates had been seen), that checkpoint's effect explaining first is still not identified, and its move from the direct estimate still excludes zero.

That run held one server process throughout, so its numerics changed during the run without a restart. Probing further, we found a limit the revision now states (§5, App. F). On Llama-3.1-8B, the deterministic configuration reproduced margins bit for bit across processes minutes apart. After a restart with the same build, configuration, and weights (the server's GPU layer placement was not recorded), all twenty-four re-measured direct margins differed, by up to 0.25 log-odds, and none changed its verdict. None of the eighteen re-measured explain-first margins matched either, and two of those verdicts changed, on the marginal résumé where the pair had already disagreed.

(continued in 3/3)

AuthorsOpenReview ↗

Response to Reviewer 68Ph (3/3): Q3 to Q5

Q3. Reference labels for résumés meeting the job requirements; baseline acceptance rates; whether log-odds changes affect final decisions across qualification levels when most résumés are accepted or rejected consistently.

Each posting lists five requirements, and each résumé template is coded against them as written (Table 12). The strong résumé meets all five, the middle one three or four, and the marginal one at most one, so the strong one should advance and the marginal one should not. Table 13 gives acceptance at each level for every cell. You are right that most résumés are accepted or rejected consistently; 57% to 100% get the same verdict under every wording. That saturation is why the paper measures the margin: where qualification fixes the decision, only the margin still carries the comparison. The printed answer agrees with the margin's sign on 69% to 100% of calls.

Across levels, no identified cell has a pooled decision gap separable from zero. The largest, on Mistral-7B v0.3, registered nurse, sits entirely at the middle level, the one qualification leaves open. There, middle-level résumés advance 35% of the time with the White name and 26% with the Black name. Of the 144 pair-by-wording comparisons at that level, 19 advance only the White name and 5 only the Black one, and no comparison at the strong or marginal level is discordant. Pooled over the three levels, the cell's gap is 3.2 points (interval 0.0 to 6.2; exact swap test p = 0.11) and its impact ratio 0.93 (interval 0.87 to 1.00), above four-fifths. At the middle level alone, the ratio is 0.73 (interval 0.46 to 1.00), below four-fifths as an estimate but not separable from it. The pooled ratio is higher because the strong résumés, advanced under both names, count in both rates (App. D).

Q4. How do the open-weight and API evaluations differ, and how are verdict-only outputs handled?

Both arms send the same system text, wordings, résumés, and name pairs, and score the first answer token as log P(yes) minus log P(no), each summed over the same case, spacing, and one-letter variants. Locally, the margin comes from the top hundred tokens of the unconstrained distribution while a grammar restricts the printed answer to yes or no. Early wording-study rows, read from a window of twenty, were kept only where both answers fell inside it, so they can differ from a hundred-token reading only through variants ranked below twenty. The OpenAI API returns at most twenty alternatives, and we sent it no grammar, logit bias, or structured output to constrain the answer. Yes and no still carry at least 99.98% of first-token probability in every call. Where the losing answer falls outside the twenty, the margin is missing. That happens only on saturated calls, for 0, 6, 144 and 431 of the 432 matched pairs on GPT-4o-mini, GPT-4o, GPT-4.1-mini and GPT-4.1; such pairs are dropped, and the effect is averaged within name pairs first, so each name pair keeps equal weight. GPT-4.1 is therefore unmeasurable; GPT-4.1-mini loses a whole résumé level, and its effect is not separable from zero, so two API checkpoints enter the headline. On the API surface we tested, the fifth frontier endpoint returned only verdicts and no probabilities, so it cannot measure a margin effect. It is analysed as a verdict and never mixed with margins. Its verdict was a function of the résumé template alone, and the two names of a pair never received different verdicts (none of 220 pairs). We called the API checkpoints by alias and did not store the responses' model and fingerprint fields, so the served snapshot is unrecorded, contrary to our own reporting list; batching and caching cannot be set or observed from outside (App. G, Table 18).

Q5. How often do batching or cache reuse change the yes/no decision or the significance of the effect?

Rarely. Across 52 repeated runs of one matched-pair design in up to four serving configurations, no verdict changed in any of 23,760 comparisons between repeats or between configurations, and no checkpoint's identification depended on the configuration. Between the original grid and its re-run under the deterministic configuration, 0 to 8 of 864 verdicts changed per checkpoint, and no per-wording PS conclusion or cell identification changed on any of the four checkpoints (§5, Table 17). The nondeterminism moves margins and rarely decisions.

Changes made for this review: the equal-budget designs and the power tables (§9, App. B); decisions, reference labels, and impact ratios (§6, App. D); the explain-then-decide experiment (App. E); serving and decisions (App. F); how the API arm is scored (App. G). We would be glad to answer anything still open before the discussion closes.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response (1/3): Changes and a new protocol

We thank the three reviewers. 68Ph and W4hz asked for specific analyses, and UAje asked whether the controls we propose change what an existing audit concludes, so we ran all of them, as outlined in the revised PDF. Beyond the changes below, the abstract and introduction now report several new results, and we edited the paper's prose for readability. Every result below is printed in the revised paper, and the supplement holds the derived artifact behind each. Section numbers refer to the revision.

Concern (reviewers)What we didWhere
Protocol vs single- and multi-prompt audits at equal budget (68Ph W1, Q1)simulation at the panel's median correlation and a measured level of between-wording heterogeneity§9, Table 8
Do the controls change an existing audit's conclusion? (UAje; 68Ph W1)new experiment with the surveyed audits' own instructions on our checkpoints§4, §9, App. C.1
Rule's error rates and power (68Ph Q1; W4hz W3, Q4)false-positive rates under measured, equicorrelated, two-block and near-zero structures; power over n and ρ, and at a given effect under each cell's measured covariance§9, App. B
Explain, then decide (68Ph W2, Q2; UAje W2)new experiment on the business-analyst grid of the four open-weight checkpoints§6, App. E
Decisions, reference labels, impact ratios (68Ph W2, Q3; W4hz Q7)decisions at each model's threshold with their selection-rate impact ratio; the scoring-rate impact ratio with exact tests§6, App. D
Wording-set sensitivity (W4hz W1, Q1)leave-one-out, all 462 split-halves, each arm, and an independent set, for the ratio and the SD§4, App. C
Per-cell table, floor, selection (W4hz W2, Q2, Q3)one table, three conventions; a deterministic re-run; the selection bias simulatedTable 1, App. B
Scope of the conclusion (W4hz W3, Q5)rescoped to this panel's design constants§9, §11
Token matching and perceived race (W4hz Q6)per-name perception rates against token-match statusApp. H
API scoring; serving and decisions (68Ph Q4, Q5)how each arm is scored and censored; verdict changes across 52 runsApp. G, F
Presentation (all three)the headline under one stated convention, with every cell in one table; the new analyses in the appendices, each cited where the text uses itTable 1, appendices

G1. What the protocol adds, and whether it changes a conclusion. Table 8 compares three designs at equal budget. Between-wording heterogeneity is set at the panel's largest R_τ, a size this panel measures, which puts τ at 0.044; four of the six identified cells exceed that, so the false-claim rates below are not the panel's worst case. A single-prompt audit claims a disparity the cross-wording mean does not have 22% to 59% of the time. Pooling the same budget over several wordings into one interval, the usual multi-prompt report, still does so 5% to 26% of the time, most often with few wordings, because the interval treats the wordings drawn as the only ones. At the panel's smallest R_τ the two rates fall to 7% to 20% and 5% to 8%. The interval for the cross-wording mean that carries the between-wording variance, which §9 reports beside the rule, holds the rate to 0% to 3%. Table 8 now also gives the every-wording rule, which is stricter than that interval at six and twelve wordings and less strict at three, where the interval has only two degrees of freedom. The cost is power. Against a true cross-wording effect of 0.10 at 384 pairs over twelve wordings, the single prompt is significant 95% of the time, the pooled interval 86%, the variance-carrying interval 76% and the every-wording rule 4%.

For UAje's question, we ran six screening instructions that five of the surveyed audits print, each edited by one stated rule to ask for yes or no, with our résumés and names. Tested one at a time without correction, as an audit using one would test it, one instruction finds a significant disparity on Llama-2-7B, where five of our six paraphrases do; two find one on Mistral-7B v0.3, where none of ours does; and none finds one on Llama-3.1-8B, where all six of ours do. Corrected within the six, only the first survives. A single-instruction audit's conclusion on these checkpoints depends on whose instruction it ran. The published instructions spread the effect no less widely than ours in point estimate, but place it differently; our reply to W4hz (1/4) gives the numbers. Not every control changes conclusions: controlling batching and the cache changed no per-wording PS conclusion (§5).

AuthorsOpenReview ↗

General response (2/3): the rule's power, decisions and the city's rules

G2. The rule's error rates and power under the measured correlation. The rule passes only when every single-wording test rejects on the same side, so its false-positive rate cannot exceed one test's level, whatever the correlation between wordings. The original derived the rate under equicorrelation, but the bound does not need it. The correlation that matters is the one between two wordings' estimates on one model, which the shared name pairs induce (0.18 to 0.71 on the identified cells), not the near-zero correlation of wording effects across model families. At each cell's measured covariance, the rate is at most 1.03×10⁻³, under a two-block structure at most 1.46×10⁻³, and under near-zero correlation no simulated draw passes. Simulated as measured, at its own effect, a cell passes with probability 0% to 79% at twelve name pairs and 40% to 100% at 96 (Table 5). Against a fixed effect of 0.10 at 96 pairs under each cell's measured covariance, the rule passes 54% to 89% of the time without between-wording heterogeneity, and 16% to 69% with each cell's own R_τ held at that effect (τ = R_τ × 0.10, below every cell's measured τ; Table 7), against 75% to 94% on the equicorrelated grid (Table 6).

G3. Decisions and the city's rules (68Ph W2, Q3; W4hz Q7). We carried every margin through to decisions and to both impact ratios the city's rules define (§6). At each model's own yes/no threshold, decisions are set mostly by the résumé's qualification. The strong résumé advances 39% to 100% of the time, and the marginal one never does, except on Llama-2-7B. Between 57% and 100% of named résumés get the same verdict under all twelve wordings, the two names of a pair get different verdicts in 0% to 10% of pairs, and no decision-level disparity on an identified cell is separable from zero. Under one wording, the exact swap test can reach significance only if at least six of the twelve name pairs split the same way, a gap of 16.7 points. The largest single-wording gap on an identified cell is 8.3 points, and on four of the six cells no pair is discordant under any wording. Pooled over wordings, the largest gap, 3.2 points on Mistral-7B v0.3, registered nurse (interval 0.0 to 6.2), is not separable from zero either (exact swap test over name pairs, p = 0.11). 68Ph is right that a margin can move without a decision moving, and at the model's own threshold the decision mostly does not move.

For a tool that selects candidates, the city's rules divide each category's selection rate by the highest category's (6 RCNY § 5-301(b)), and the model's yes/no verdict is that selection. Pooled over the wordings and the three levels in equal shares, that ratio is 0.93 to 1.00 on the identified cells. No single wording takes it below four-fifths. One wording, on Mistral-7B v0.3, registered nurse, sits exactly at it (0.80), which the four-fifths rule, as Gaebler et al. (2024) describe it, would flag; the paper counts only ratios strictly below. The pooled ratio depends on the pool: middle-level résumés alone would put that cell at 0.73 (interval 0.46 to 1.00; App. D). For a tool that outputs a score, the rules use a scoring rate instead: the share of a category scoring above the sample median (§ 5-301(c)). On the identified cells, this gives pooled impact ratios of 0.73 to 0.92, and the exact swap test over name pairs confirms a disparity on four of them, each against the White-named résumés. Under a single wording, the test is significant on none to nine of the twelve wordings per cell, and the ratio falls strictly below four-fifths on none to nine of them, against 0.0 to 0.8 expected by chance under the exact null. On three of the four cells with a significant disparity, single-wording audits land on both sides of four-fifths; on GPT-4o-mini, seven fall below it and the other five sit exactly at it. Part of that spread is the noise of twelve name pairs. The between-wording SD of the scoring-rate disparity is 33% to 86% of the pooled disparity, and 0% to 43% once that noise is removed.

AuthorsOpenReview ↗

General response (3/3): the threshold and scope

The threshold, and the 2.0× to 484× figure (W4hz Q7). That figure is the error of converting log-odds to percentage points at one fixed operating point. The checkpoints do not share an operating point, so no fixed factor is right. In the impact ratio, the same problem takes the form of the threshold, which Gaebler et al. (2024) also vary on a rating scale. As the shortlist moves from 5% to 95% of candidates, the pooled impact ratio on the same margins spans 0.00 to 1.00, and every identified cell falls below four-fifths somewhere on that range. On GPT-4o, it does so only where the Black-named résumés are selected less, against its margin effect. An audit that reports an impact ratio should report its threshold; that is now item (5) of the reporting set.

Scope (W4hz W3, Q5). We agree and adopted the proposed scoping. §9 now reads: "On this panel no disparity passes the rule. [...] With this name list, these postings and this wording set, and at the power above, that says what this design can establish on these checkpoints, not that the disparities are absent." The Conclusion reads: "[...] call a disparity established only where it survives every wording. On these checkpoints, with this name list, these postings and these wordings, none does."

One additional change: the AI use statement now discloses that an AI agent assisted with proposing parts of the research design and assisted with running parts of the model experiments for the submitted version.

Point-by-point replies follow under each review, with the reviewers' W and Q numbers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.