The Deadline Effect: Identifying and Correcting Temporal Bias in Human Evaluation

Submission timing in human evaluation systems is universally treated as an irrelevant detail. We identify a systematic temporal leniency bias: across 129,023 reviews spanning three consecutive years of a large-scale ML venue, evaluators submitting closer to the deadline assign higher within-paper scores and produce less thorough assessments. The pattern holds in all 15 robustness specifications across all three years, and early evaluators achieve measurably lower Brier scores ($\Delta \approx 0.005$–$0.006$, stable across years). We propose two complementary corrections. Optimal Timing-Calibrated Aggregation (OTCA) derives a linear weight function $w(t)=a^{\star }t+1$ by minimising paper-level decision loss, significantly outperforming heuristic alternatives (McNemar ${\chi }^2=46.3$, $p<0.001$). Adversarial Temporal Debiasing (ATD) learns timing-invariant representations via Gradient Reversal, reducing timing discriminability by $37.4$ while preserving predictive utility. Combined in a joint scheme, the two methods improve borderline decision accuracy by up to $11.25$ over uniform averaging (McNemar $p<0.001$), consistently across all three years. Both methods require only submission timestamps, already logged by every major evaluation platform, making deployment cost-free.