acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Mean Accuracy Comparisons in Pedestrian Attribute Recognition

Abstract

Pedestrian attribute recognition (PAR) methods are commonly compared using mean accuracy (mA), the macro-average of per-attribute balanced accuracy. Our paper-and-code audit finds heterogeneous label-imbalance interventions paired with a small set of numerical inference thresholds. Because these interventions can change score meaning, native mA comparisons can involve different operating policies. We develop a decision-coordinate framework that derives objective-implied coordinates for identifiable loss and sampling components and estimates trained predictors' score-space coordinates through validation-set isotonic calibration. The framework complements native mA with objective and realized-policy normalization toward the common balanced-accuracy decision target. We evaluate controlled methods with ResNet-50 and ViT-B/16 backbones and published PAR methods, holding each predictor fixed across evaluation views and using disjoint test data. On RAP, the gap between inverse-prior reweighting and unweighted binary cross-entropy decreases from to percentage points with ResNet-50 and from to with ViT-B/16 under realized-policy normalization. Comparisons among published methods using the same backbone also exhibit reversals in mean ordering, while some gaps persist or widen. These results show that the magnitude and direction of method advantages can depend on operating policy, supporting joint reporting of native and normalized mA.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Pedestrian Attribute Recognition, Evaluation Protocols, Mean Accuracy

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 5 reviews, above 85% of the others.
Reviewer_xLTaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new evaluation framework to address the issue in PAR where heterogeneous label-imbalance interventions (e.g., reweighting) paired with a small set of numerical inference thresholds cause the score meaning of the commonly used metric mA to shift under different operating policies, making cross-method comparisons unfair. The authors argue that directly comparing native mA has fairness flaws. To address this, the authors develop a decision-coordinate framework that derives objective-implied coordinates (κobj) and estimates trained predictors' score-space policy coordinates (κest) through validation-set isotonic calibration, to perform objective normalization and realized-policy normalization, respectively. This framework aligns the decision boundaries of different methods to the common balanced-accuracy decision target (κ=0), thereby eliminating the interference caused by operating-policy differences and achieving a fair comparison at the evaluation level.

Critical strengths
  1. This paper identifies a practically relevant and often overlooked evaluation issue in PAR: heterogeneous label-imbalance interventions can alter the meaning of prediction scores, causing native mA comparisons to involve different operating policies. To address this issue, the authors develop a decision-coordinate framework with objective normalization and realized-policy normalization, providing a more comparable evaluation of PAR methods under a common balanced-accuracy decision target.
  2. This paper provides a systematic theoretical analysis supported by extensive empirical evaluations. The authors establish the relationship between training objectives, inference thresholds, and operating-policy coordinates, and conduct experiments on three widely used PAR benchmarks (PETA, PA100K, and RAP) using ResNet-50 and ViT-B/16 backbones. The evaluation covers controlled label-imbalance interventions, published PAR methods, comparisons across evaluation views, and sensitivity analyses of calibration choices and validation data, providing empirical support for the main findings.
Critical weaknesses

Experiments show that normalized method rankings may change with the calibration method. For the highly class-imbalanced PAR task, rare attributes have few positive samples, which further increases the uncertainty in calibration and threshold estimation. The authors need to quantify this uncertainty more explicitly, especially when the performance gaps between methods are very small.

Questions for authors
  1. The paper mainly reports changes in mA gaps after normalization, but their implications remain unclear. What do changes in mA after normalization reveal about the sources of a method's performance improvements?
  2. The authors are encouraged to analyze how normalization affects different label-imbalance interventions, such as resampling and loss reweighting [1], to further demonstrate the framework's interpretive value.
  3. The authors could select a representative PAR study (e.g., [2]) and re-evaluate four or five methods from its original comparisons, reporting both native and normalized mA.
  4. Further analysis of whether the original performance advantages and method rankings persist after normalization would help demonstrate the framework's practical value in reassessing existing PAR studies.

[1] Pedestrian Attribute Recognition as Label-balanced Multi-label Learning [2]Pedestrian Attribute Recognition via CLIP-based Prompt Vision-Language Fusion

5 comments
AuthorsOpenReview ↗

Response to Reviewer xLTa (1/3)

We appreciate your suggestions to quantify threshold uncertainty for rare attributes and to revisit a published PAR comparison to see which performance advantages persist under normalization. Below, we address the concern raised under Critical Weaknesses and then respond to your four questions in order.

[W1] Uncertainty in Est. thresholds and per-attribute mA for rare attributes.

We quantified the effect of validation sampling on two quantities: the Est. threshold for each attribute and its per-attribute test mA (BAj\mathrm{BA}_j in Section 3.1). For each quantity, we measured the standard deviation (SD) across validation refits. To compare thresholds across attributes and predictors, we expressed each Est. threshold as its percentile among the corresponding scores on the full validation set.

We chose ERM and RWinvRW_{\mathrm{inv}} on ResNet-50 because their Est. (Isotonic) mA gap on RAP is only 0.12 pp and its paired interval includes zero (Table 13). Across all three datasets, we used the three fixed predictors per method from Table 3 and refitted both calibrators on the same 1,000 validation bootstrap samples per dataset, keeping the training priors and test images fixed. For each attribute, we computed both SDs over supported refits and averaged them across the three predictors. We have added this analysis to the revised paper (Appendix E.6, Table 17). The table below reports the group medians as SD of the Est. threshold percentile / SD of per-attribute test mA, both in pp.

In all 12 dataset–method–calibrator combinations, the group with a validation positive rate below 5% had the highest median SD for both the Est. threshold and per-attribute test mA. The median SD of per-attribute test mA decreased from [0%, 5%) to [5%, 25%) and again to [25%, 50%) in every combination. For example, on PETA, the median threshold SD for ERM under Est. (Isotonic) was 3.75, 1.97, 1.92, and 2.15 pp across the four groups in order of increasing validation positive rate. The corresponding median SDs of per-attribute test mA were 1.24, 0.36, 0.22, and 0.32 pp. One possible explanation is that, for rare attributes, the calibration fit relies on relatively few positive validation images. Which of those images are repeated or omitted in a bootstrap sample can shift where the fitted curve crosses the fixed training prior, changing the Est. threshold and, in turn, the per-attribute test mA.

DatasetValidation positive rateAttributesPositive images per attributeERM Est. (Isotonic)ERM Est. (Sigmoid)RWinvRW_{\mathrm{inv}} Est. (Isotonic)RWinvRW_{\mathrm{inv}} Est. (Sigmoid)
PETA[0%, 5%)825–773.75 / 1.241.80 / 0.353.29 / 1.411.41 / 0.41
PETA[5%, 25%)14100–4651.97 / 0.360.70 / 0.152.30 / 0.350.83 / 0.13
PETA[25%, 50%)8514–9391.92 / 0.220.64 / 0.092.37 / 0.320.86 / 0.09
PETA[50%, 100%]5973–1,6252.15 / 0.320.66 / 0.092.49 / 0.310.76 / 0.09
RAP[0%, 5%)2314–1813.37 / 0.811.46 / 0.373.44 / 0.801.32 / 0.34
RAP[5%, 25%)16213–8402.50 / 0.300.67 / 0.132.40 / 0.280.80 / 0.12
RAP[25%, 50%)7988–1,5882.11 / 0.170.62 / 0.081.71 / 0.200.72 / 0.07
RAP[50%, 100%]52,071–3,5062.05 / 0.250.87 / 0.112.00 / 0.220.86 / 0.10
PA100K[0%, 5%)821–4032.95 / 0.711.13 / 0.532.54 / 0.801.13 / 0.47
PA100K[5%, 25%)9661–2,0561.74 / 0.210.58 / 0.101.78 / 0.250.66 / 0.11
PA100K[25%, 50%)62,602–4,9480.89 / 0.120.25 / 0.041.16 / 0.160.26 / 0.05
PA100K[50%, 100%]35,628–8,4730.81 / 0.200.20 / 0.070.92 / 0.340.23 / 0.08
AuthorsOpenReview ↗

Response to Reviewer xLTa (2/3)

[Q1] What do changes in mA gaps reveal about the sources of performance gains?

A higher Native mA shows an advantage for the complete PAR system under its documented inference rule. Normalization lets us examine how much of that advantage persists when the predictors stay fixed and both methods use decision rules that target balanced accuracy.

  • A change in the gap measures how much the comparison depends on the decision rule. We hold each predictor’s scores fixed and construct Obj. and Est. thresholds for the common balanced-accuracy target (κ=0\kappa=0). Obj. derives thresholds from identifiable loss or sampling components, while Est. uses the score-to-posterior relation estimated from validation scores and labels. Each predictor’s ranking of images for each attribute stays the same across views.

  • Operating-policy misalignment can contribute to a Native lead. Table 2 gives a concrete reason why the Native gap can shrink. At the same threshold of 0.5, inverse-prior reweighting places the objective-implied boundary at the mA target (κobj=0\kappa^{\mathrm{obj}}=0), whereas ordinary BCE requires stronger positive evidence for attributes with πj<0.5\pi_j<0.5 (κobj=−logit⁡πj>0\kappa^{\mathrm{obj}}=-\operatorname{logit}\pi_j>0). On RAP with ResNet-50, moving from Native to Est. raises ERM’s mA by about 9.6 pp and that of RWinvRW_{\mathrm{inv}} by only 0.8 pp (Table 3). The RWinv−ERMRW_{\mathrm{inv}}-\mathrm{ERM} gap falls from +8.99 to +0.12 pp (Table 13). ERM therefore closes much of the gap using scores it had already learned, supporting operating-policy misalignment as a contributor to the Native advantage.

  • A method’s advantage can also persist or grow under the common target. Normalization also helps interpret comparisons between methods that use the same imbalance intervention. L2L and PIL both use RWexp⁡RW_{\exp}-weighted BCE in their classification components and a Native threshold of 0.5 (Table 10). Yet L2L’s lead on RAP grows from 0.37 pp under Native to 2.28 pp under Obj. and 2.69 pp under Est. (Table 3). In this case, the advantage persists under both rules targeting balanced accuracy, even though its size depends on the decision rule. Attributing its advantage to a particular training component would require a method-specific ablation.

[Q2] How does normalization affect resampling and loss reweighting?

For the resampling and loss-reweighting methods in Table 3, Obj. changes the Native threshold when the decision boundary implied by the training rule differs from the mA target. Est. uses the score-to-posterior relation estimated for each trained predictor, so it can change the threshold even when the sampling or weighting rule already aligns the objective-implied boundary with that target.

  • Under Obj., balanced feature resampling and inverse-prior reweighting behave alike. Both give κobj(0.5)=0\kappa^{\mathrm{obj}}(0.5)=0, so FRDL and RWinvRW_{\mathrm{inv}} retain their Native thresholds on all three datasets and both backbones. For exponential reweighting (RWexp⁡RW_{\exp}), the Obj. threshold depends on the attribute’s training prior. Moving from Native to Obj. raises its ResNet-50 mA by 4.2 pp on PETA, 4.8 pp on PA100K, and 6.7 pp on RAP.

  • Under Est., FRDL overtakes RWinvRW_{\mathrm{inv}} on both PETA and PA100K with ResNet-50. On PETA, for example, RWinvRW_{\mathrm{inv}} leads FRDL under both Native and Obj. (85.97% vs. 85.83%). Under Est., FRDL reaches 87.53% and RWinvRW_{\mathrm{inv}} reaches 87.33%. For RWexp⁡RW_{\exp}, the ranking changes one step earlier: on both datasets, it moves from the lowest Native mA of the three methods to the highest mean mA under Obj. and retains that lead under Est.

AuthorsOpenReview ↗

Response to Reviewer xLTa (3/3)

[Q3] Re-evaluation of methods compared in the PromptPAR study [2].

Following your suggestion, we revisited the comparisons in the PromptPAR study, cited as [2] in your review.

[2] Xiao Wang, Jiandong Jin, Chenglong Li, Jin Tang, Cheng Zhang, and Wei Wang. "Pedestrian Attribute Recognition via CLIP-Based Prompt Vision-Language Fusion." IEEE Transactions on Circuits and Systems for Video Technology, 35(1), 148–161, 2025.

We selected four method families represented in Tables I–II of [2]: VAC, SSC, PARFormer, and PromptPAR. We report five configurations because we evaluate both the released PromptPAR checkpoint (*PromptPAR) and our retrained PromptPAR. For the retrained models, only the training split contributed to supervised parameter updates, and the validation split was used for calibration. The official PromptPAR code defaults to trainval, which includes our validation images, although the exact training and selection history of the released weights is not fully documented (Section 4.1 and Appendices D.1 and D.2.2). This difference from the default training protocol can affect how closely our retrained PromptPAR matches the Native mA reported in [2].

For both VAC and SSC, we used a Native probability threshold of 0.5 and derived the Obj. thresholds from each method's weighted BCE classification component. Let π^jtr\widehat{\pi}_j^{\mathrm{tr}} denote the positive rate of attribute jj in the training split. Both methods weight positive examples by exp⁡(1−π^jtr)\exp(1-\widehat{\pi}_j^{\mathrm{tr}}). VAC weights negative examples by exp⁡(π^jtr)\exp(\widehat{\pi}_j^{\mathrm{tr}}), while SSC gives them unit weight. The table below reports test mA (%) for all five configurations under Native, Obj., and Est. (Isotonic).

On PA100K, our Native results reproduce the direction of the SSC–VAC comparison in [2], while both normalized rules reverse it. Table I of [2] reports higher mA for SSCsoft than for VAC (81.87% vs. 79.16%). In our Native evaluation, SSC leads VAC by 1.94 pp. Under Obj., VAC leads SSC by 1.42 pp, and under Est. (Isotonic) it leads by 1.67 pp. The same reversal occurs on PETA and RAP in our evaluation. On PA100K, the released *PromptPAR retains the highest mA among the five configurations under all three rules.

MethodPETA NativePETA Obj.PETA Est.PA100K NativePA100K Obj.PA100K Est.RAP NativeRAP Obj.RAP Est.
VAC83.57 ± 0.4288.32 ± 0.2288.33 ± 0.2378.92 ± 0.2085.02 ± 0.2586.71 ± 0.4078.58 ± 0.2585.85 ± 0.0586.27 ± 0.25
SSC85.79 ± 0.2986.63 ± 0.3686.68 ± 0.1180.86 ± 0.1583.60 ± 1.1285.04 ± 0.3981.76 ± 0.3784.61 ± 0.2185.32 ± 0.21
PARFormer-B87.99 ± 0.1588.82 ± 0.2689.24 ± 0.2381.85 ± 0.3485.06 ± 0.7586.83 ± 0.5782.73 ± 0.3386.01 ± 0.8086.49 ± 0.53
*PromptPAR (released)87.7988.9988.0287.2389.8990.1085.4289.0688.83
PromptPAR (retrained)86.85 ± 0.1288.99 ± 0.1288.92 ± 0.1385.35 ± 0.3488.32 ± 0.5888.88 ± 0.2684.22 ± 0.0987.81 ± 0.0887.77 ± 0.15

[Q4] Further analysis of whether Native performance advantages and method rankings persist after normalization.

We analyzed the Native leader and all six pairwise orderings by reported mA within each architecture group on PETA, PA100K, and RAP. We used the four CNN methods in Table 3. For the Transformer group, we used HAP, PARFormer-B, ReSAGE-PAR, and the released *PromptPAR checkpoint, which represents PromptPAR in our original comparison.

  • The Native leader changes in five of the six settings under Est. (Isotonic). In the CNN group, FRDL+GOAT moves from first on all three datasets to second on PETA and PA100K and third on RAP. In the Transformer group, HAP overtakes PARFormer-B on PETA and released *PromptPAR on PA100K. Released *PromptPAR retains its lead on RAP.

  • Most pairwise orderings persist even though the top rank often changes. Of the 36 within-group comparisons, 20 retain their Native ordering under Est. (Isotonic) and 16 reverse. On PA100K, for example, FRDL+GOAT has two nearly equal Native leads: 3.05 pp over ReSAGE-PAR and 3.14 pp over L2L. Under Est. (Isotonic), it trails ReSAGE-PAR by 0.69 pp but still leads L2L by 1.69 pp. Obj. gives the same two orderings. Table 13 reports paired intervals for ReSAGE-PAR minus FRDL+GOAT on all three datasets. On PA100K, the 95% interval is [−3.41, −2.54] pp under Native and [+0.28, +1.30] pp under Est. (Isotonic). We provide the individual run results for PETA and RAP in our response to Reviewer@CX2L [W3/Q3].

These comparisons concern rankings by mA. We also examined which orderings persist for instance-level metrics (response to Reviewer@Po7R [Q1]) and how Est. mA rankings compare with macro-AUROC rankings (Appendix E.5, Table 16; response to Reviewer@JJ8Q [Q3]).

Reviewer_Po7ROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper determines whether comparisons of mean accuracy (mA) in the pedestrian attribute recognition (PAR) task are biased due to various operating thresholds created by different training objectives and inference approaches. The proposed framework complements native mA with objective- and realized-policy normalization to align with the balanced-accuracy decision target. The results demonstrate improvements, showing that the gap between inverse-prior reweighting and unweighted binary cross-entropy decreases on the RPA dataset.

Critical strengths
  • The motivation is clear and the paper presents a compelling empirical argument that the same numerical thresholds do not necessarily imply the same decision-making criterion when used by different classifier algorithms
  • The experiment is comprehensive and demonstrates the effect of the proposed decision-coordinate framework
Critical weaknesses
  • My concern is about the novelty. The decision theory and validation-based thresholds are already well-established. The main contribution is the use of objective-implied and estimated coordinates. However, the ablation does not clearly show the benefit of this contribution beyond established validation-threshold selection.
  • The concept of a “policy-normalized” comparison needs clarification. Native mA evaluates models based on their original inference rules, while Est. evaluates them after the validation-based threshold tunning. Therefore, the reduced performance gap does not necessarily mean that the original advantage was solely due to operating-policy misalignment or the models have equivalent discriminative ability. In addition, aligning likelihood-ratio thresholds does not ensure identical decision boundaries in input space.
Questions for authors
  • Beside mA, there are many other metrics (e.g Acc, Prediction, Precision, F1, etc). Could the authors discuss how policy normalization affects these metrics and whether the observed improvements remain consistent?
  • How does the proposed framework compare with recent PAR methods, such as [1] and [2]?
  • Could authors please provide the code?

Reference

[1] Joint Implicit and Explicit Language Learning for Pedestrian Attribute Recognition, AAAI 2026

[2] PFM-VEPAR: Prompting foundation models for RGB-Event camera based pedestrian attribute recognition, Pattern Recognition 2026, https://doi.org/10.1016/j.patcog.2026.113720

5 comments
AuthorsOpenReview ↗

Response to Reviewer Po7R (1/3)

We sincerely thank the reviewer for the careful reading and constructive feedback. We appreciate your recognition of the clear motivation and the empirical evidence that shared numerical thresholds can imply different decision conditions. We address your concerns and questions below and provide the additional metric analysis and requested code.

[W1] Novelty of the decision-coordinate framework for mA comparisons in PAR.

We appreciate the request for clearer evidence of our work’s novelty. Our work contributes a systematic audit of PAR evaluation practices, a decision-coordinate framework for mA comparisons, and paired evaluations of how decision policies affect mA gaps and method orderings. Within the framework, we construct normalization rules targeting the common mA reference and evaluate their consequences. We examine whether a Native mA advantage persists under a training-derived rule whose threshold is not fitted on validation labels, and how the trained predictor's estimated decision condition relates to its objective-implied reference.

To address the concern about the ablation, we examine what the training-derived Obj. control adds beyond validation-based threshold selection. At the shared Native probability threshold of 0.5, ordinary BCE and inverse-prior reweighting can imply different population decision coordinates (Table 2). The corresponding Obj. rules follow from these documented objectives and the common balanced-accuracy target under the stated assumptions. For the same fixed predictors across evaluation views, Table 13 reports the following test mA gaps (RWinv−ERMRW_{\mathrm{inv}}-\mathrm{ERM}, pp) on RAP:

BackboneNativeObj.Est. (Isotonic)Val-opt.
ResNet-50+8.99−0.48+0.12+0.12
ViT-B/16+8.16+0.31+0.28+0.29

Val-opt. shows how the two methods compare after their thresholds are selected to maximize validation mA. Obj. shows that the large Native gap on RAP already contracts on both backbones when thresholds are derived from the training objectives, training priors, and balanced-accuracy target alone.

The estimated coordinates then help distinguish two different situations. For ERM on RAP with ResNet-50, the isotonic estimates at Native thresholds give a median absolute discrepancy of 0.336 from its objective-implied coordinate, but a median absolute distance of 2.681 from the mA target (Table 11). The analyzed FRDL and LA components already have zero objective-implied coordinates at their Native thresholds, yet their estimated target distances are 0.906 and 0.364. Across all six dataset–backbone settings, both calibrators give the same result: by median absolute distance, ERM is closer than FRDL to its own objective-implied Native coordinate, yet farther from the common mA target (Appendix E.1, Table 11; see also our response to Reviewer@C1Ke [W1/Q1]). Val-opt. mA alone cannot distinguish these two aspects of a predictor's decision policy. These remain local estimates subject to the calibration considerations in Appendix B.5. In the RAP comparisons shown here, Est. and Val-opt. yield similar test mA gaps, consistent with their connection in Appendix C.4. We retain Est. to connect coordinate estimation with normalized evaluation: the same validation-fitted calibration maps used to estimate κest\kappa^{\mathrm{est}} are inverted at the training priors to construct the Est. decision rules (Appendix C.3).

[W2] Policy normalization targets a common mA reference for fixed predictors.

Thank you for highlighting these distinctions and the need to clarify the interpretation of the comparison. As specified in Section 3.4, policy-normalized comparisons use decision rules constructed to target the common balanced-accuracy reference κ=0\kappa=0, while keeping each predictor and its scores fixed. Native retains the documented inference rule. Obj. derives its rule from an identifiable training component, its parameters and training prior, whereas Est. constructs its rule from validation-fitted score-to-posterior maps. The maps and thresholds are frozen before evaluation on disjoint test data.

AuthorsOpenReview ↗

Response to Reviewer Po7R (2/3)

Our framework quantifies changes in method gaps and orderings under decision rules targeting the common balanced-accuracy reference. In Section 4.2, we observe gap contractions and ordering reversals while keeping each predictor and its scores fixed. A contraction means that the full Native advantage is not retained under the examined normalized rules. This is consistent with operating-policy misalignment contributing to that advantage. Joint reporting shows the Native advantage alongside the advantage retained under normalization. We do not claim that operating-policy misalignment is the sole source of the Native advantage or that the predictors have equivalent discriminative ability.

In Section 3.3 and Appendix B.5, we distinguish score-conditioned posteriors from the posterior conditioned on the full input. Score-conditioned posteriors can differ across predictors. Their decision rules can therefore target the same mA reference while producing different boundaries in input space. We have clarified the interpretation of policy-normalized comparisons in Sections 3.4 and 4.2 and the distinction between score space and input space in Appendix B.5. These revisions are highlighted in blue.

[Q1] Effects of policy normalization on other evaluation metrics.

As noted in our Introduction, mA, an attribute-level metric, averages balanced accuracy across attributes and is a principal criterion for PAR comparisons, having "greater practical implication" [3, Section V-B]. It is also used for PAR evaluation in human-centric benchmarks and generalist models, such as HumanBench [4] and Hulk [5]. For image ii, let TPiTP_i, FPiFP_i, and FNiFN_i denote the numbers of true positives, false positives, and false negatives across attributes, respectively. Instance-level Acc, Precision, Recall, and F1 are commonly reported as complementary measures. For NN images, these metrics are defined as: Acc=1N∑i=1NTPiTPi+FPi+FNi,\mathrm{Acc}=\frac{1}{N}\sum_{i=1}^{N}\frac{TP_i}{TP_i+FP_i+FN_i},

\mathrm{Rec}=\frac{1}{N}\sum_{i=1}^{N}\frac{TP_i}{TP_i+FN_i},\qquad \mathrm{F1}=\frac{2\cdot\mathrm{Prec}\cdot\mathrm{Rec}}{\mathrm{Prec}+\mathrm{Rec}}.$$ We report all five metrics for four controlled ResNet-50 methods on PETA, with each predictor held fixed across evaluation views. - **Within methods**, moving from Native to Est. raises mean mA and Recall but lowers mean Acc and Precision. Obj. shows similar trends for ERM and $RW_{\exp}$, while coinciding with Native for $RW_{\mathrm{inv}}$ and FRDL. One possible reason is that, for a rare attribute, recovering one missed positive improves balanced accuracy more than one additional false positive reduces it. When the Native threshold is conservative, Est. can therefore admit more positive predictions. This can raise Recall while lowering Precision if the additional predictions are less often correct. These opposing changes partly offset each other in F1, the harmonic mean of Precision and Recall. Across the four methods, the change in mean F1 ranges from −0.60 to +0.53 pp. - **Across methods**, $RW_{\mathrm{inv}}$ has higher mean mA than ERM under Native, whereas ERM has higher mean mA under Est. ERM retains higher mean F1 in both views. Thus, the mA ordering reverses for this pair, while the F1 ordering is preserved. Zhou et al. [3, Section V-B2] also report PAR comparisons with higher mA but lower F1. | Method | View | mA (%) | Acc (%) | Prec (%) | Rec (%) | F1 (%) | |:--|:--|--:|--:|--:|--:|--:| | ERM | Native | 82.10 ± 0.42 | 77.65 ± 0.21 | 87.82 ± 0.08 | 83.64 ± 0.34 | 85.68 ± 0.15 | | ERM | Obj. | 87.78 ± 0.18 | 76.91 ± 0.14 | 83.58 ± 0.13 | 86.81 ± 0.18 | 85.16 ± 0.10 | | ERM | Est. | 87.93 ± 0.09 | 76.08 ± 0.60 | 82.81 ± 0.62 | 88.27 ± 0.16 | 85.45 ± 0.40 | | $RW_{\exp}$ | Native | 83.23 ± 0.17 | 77.77 ± 0.11 | 86.75 ± 0.12 | 84.70 ± 0.02 | 85.71 ± 0.07 | | $RW_{\exp}$ | Obj. | 87.43 ± 0.20 | 75.27 ± 0.23 | 83.42 ± 0.30 | 86.78 ± 0.05 | 85.07 ± 0.15 | | $RW_{\exp}$ | Est. | 87.98 ± 0.16 | 75.18 ± 0.26 | 82.82 ± 0.32 | 87.52 ± 0.25 | 85.11 ± 0.20 | | $RW_{\mathrm{inv}}$ | Native/Obj. | 85.97 ± 0.12 | 77.34 ± 3.19 | 84.93 ± 3.72 | 85.92 ± 0.41 | 85.40 ± 2.08 | | $RW_{\mathrm{inv}}$ | Est. | 87.33 ± 0.71 | 75.92 ± 1.88 | 83.65 ± 2.04 | 87.13 ± 0.21 | 85.35 ± 1.16 | | FRDL | Native/Obj. | 85.83 ± 0.06 | 76.58 ± 0.07 | 84.38 ± 0.10 | 85.39 ± 0.09 | 84.88 ± 0.04 | | FRDL | Est. | 87.53 ± 0.03 | 75.63 ± 0.35 | 82.83 ± 0.41 | 88.16 ± 0.14 | 85.41 ± 0.29 | *F1 is computed for each predictor before aggregation.*
AuthorsOpenReview ↗

Response to Reviewer Po7R (3/3)

[Q2] Relation to recent PAR methods.

We appreciate the suggestion to relate our framework to these recent methods. JGEL [1] combines image-specific implicit language representations with explicit attribute guidance for RGB PAR, while PFM-VEPAR [2] uses event prompting and associative memory for RGB–event PAR. Our framework derives objective-implied decision references from documented training specifications and estimates trained predictors' score-space policies, enabling paired comparisons that assess how reported method advantages depend on evaluation rules.

Both methods provide concrete connections to our component-level analysis in Appendix B.2. Under the stated population-response assumptions, and writing the reference prior as π\pi:

  • JGEL: The classification losses reported for its ILGL and EAEL attribute heads are unweighted BCE (Eqs. (3) and (5) in [1]). For these components, our derivation gives κobj(t)=logit⁡t−logit⁡π\kappa^{\mathrm{obj}}(t)=\operatorname{logit}t-\operatorname{logit}\pi, where tt is a probability threshold. This provides a classification-derived reference within its joint language-guided learning objective.
  • PFM-VEPAR: Its official EventPAR implementation uses fixed class-weighted BCE and a native probability threshold of 0.50.5. For the training frequency ρ\rho used in its weights, a=e1−ρa=e^{1-\rho} and b=eρb=e^\rho give κobj(0.5)=2ρ−1−logit⁡π\kappa^{\mathrm{obj}}(0.5)=2\rho-1-\operatorname{logit}\pi. With ρ=π\rho=\pi, this recovers the RWexp⁡RW_{\exp} component covered by our framework.

This comparison concerns documented training and evaluation conditions. We have added both references and clarified their relation to our work in Section 2, with the revised text highlighted in blue.

References

[1] Joint Implicit and Explicit Language Learning for Pedestrian Attribute Recognition, AAAI 2026

[2] PFM-VEPAR: Prompting foundation models for RGB-Event camera based pedestrian attribute recognition, Pattern Recognition 2026, https://doi.org/10.1016/j.patcog.2026.113720

[3] Yibo Zhou et al. "Heterogeneous Feature Re-Sampling for Balanced Pedestrian Attribute Recognition." IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(4), 2706–2722, 2025.

[4] Shixiang Tang et al. "HumanBench: Towards General Human-Centric Perception with Projector Assisted Pretraining." CVPR, 2023.

[5] Yizhou Wang et al. "Hulk: A Universal Knowledge Translator for Human-Centric Tasks." IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(7), 5672–5689, 2025.

[Q3] Code availability.

Thank you for requesting the code. We provide the implementation of Est. threshold construction and the loss functions used in our controlled experiments in the supplementary material and in our anonymous code repository (https://anonymous.4open.science/r/par-evaluation-code-2603). est.py implements the isotonic and monotone-sigmoid variants, while controlled_losses/ contains the loss implementations adapted to the shared PAR baseline interface. The accompanying README provides usage instructions, links to the official model repositories, and references to the experimental settings in the paper.

We hope the additional comparisons clarify what the decision coordinates reveal beyond validation-based threshold selection, and we would be grateful if you would consider these results in your final assessment.

Reviewer_JJ8QOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper argues that PAR comparisons by mean accuracy (mA) not directly comparable: heterogeneous label-imbalance interventions change score meaning, so identical thresholds (0.5/0.45) encode different decision policies. The authors audit 66 papers and 28 implementations and propose a decision-coordinate framework measuring thresholds by their induced log-likelihood-ratio boundary κ. On PETA/PA100K/RAP, the mA gap between inverse-prior reweighting and BCE contracts from 8.99 to 0.12 pp.

Critical strengths

For a long time, the PAR field has taken it as a given that [email protected] values can be directly compared across different studies. This paper first shows through its audit that this assumption does not hold. It then uses κ, the log-likelihood ratio boundary implicit in decision thresholds, to reframe "thresholds" into a unified coordinate system, so that various distinct interventions can all be reduced to a simple shift of this same quantity. The derivation further covers prediction-related losses like focal and asymmetric loss.

Critical weaknesses

None

Questions for authors
  1. On the one hand, the paper defines κ = 0 as the sole method-independent objective (§3.1); on the other hand, it asserts the independence of native mA through statements such as "the native lead is conditional on the decision conditions used for evaluation" (§4.2) and "supporting joint reporting of native and normalized mA" (Abstract). Please clarify: is native mA merely a "record layer of the state of the literature," or is it a valid evaluation metric on a par with normalized mA? If the former, we suggest revising the wording in the Abstract and §4.2 to convey that meaning; if the latter, please explain in what sense decisions that deviate from κ = 0 remain justified.
  2. The definition κ*=0 is grounded in balanced mA; however, only 38 of the 66 audited papers can be verified as balanced. We recommend that the authors discuss the applicability of current methods under unbalanced mA and, where necessary, provide the derivation of the objective coordinates for the unbalanced case.
  3. Once Est. is used to align thresholds to κ = 0, inter-method differences ought to be governed by ranking quality. AUC offers a natural lens for assessing such differences. We therefore recommend that the authors clearly articulate the advantages of mA_Est over alternative threshold-independent ranking metrics such as AUC
3 comments
AuthorsOpenReview ↗

Response to Reviewer JJ8Q (1/2)

We sincerely thank the reviewer for the careful reading, thoughtful questions, and positive assessment of our decision-coordinate framework. These questions have helped us clarify the roles of Native and normalized mA, the scope of the framework under alternative class weightings, and the distinction between Est. and AUC.

[Q1] Native mA remains valid alongside normalized mA (Obj. and Est.).

We take the latter view: native mA is a valid evaluation of the complete method under its documented operating policy, not merely a "record layer of the state of the literature". As Sections 3.1 and 3.3 show, training interventions can make the same numerical threshold represent different policies, which we characterize through the κ\kappa coordinates. Obj. uses the policies implied by identifiable training components, whereas Est. uses score-space policies estimated from validation data for each fixed predictor. Both normalized views target the common balanced-accuracy reference κmA,j⋆=0\kappa_{\mathrm{mA},j}^{\star}=0 through their respective constructions (Section 3.4). We retain Native to evaluate the documented system and to provide the baseline for our paired comparisons. This does not assume that its original policy is optimal for mA. Obj. and Est. then test whether the same predictors' relative advantages persist under decision rules targeting the common balanced-accuracy reference.

We have clarified these complementary roles in Section 3.4 and adjusted the corresponding wording in the Abstract and Section 4.2, with the revisions highlighted in blue.

[Q2] Extending the framework to "unbalanced mA".

Here, "verified" refers to verification of the reported mA definition or evaluation protocol: the audited publication either explicitly defines standard balanced mA or explicitly adopts a traceable standard evaluation protocol (Appendix A.2). The 38/66 count records the papers meeting this criterion. For the remaining 28 papers, standard balanced mA was not verified; this does not mean that they used an unbalanced metric. Table 6 distinguishes these two evidence categories. All mA results in our experiments use the standard balanced definition.

To make the extension precise, we consider one possible interpretation of "unbalanced mA": the mean of ordinary per-attribute accuracies. For a single attribute under the common reference distribution of Appendix B.1, let Uα=α⋅TPR+(1−α)⋅TNRU_\alpha=\alpha\cdot\mathrm{TPR}+(1-\alpha)\cdot\mathrm{TNR}, where 0<α<10<\alpha<1. Writing fy(x)=p(x∣Y=y)f_y(x)=p(x\mid Y=y) for the class-conditional density, the optimal rule predicts positive when αf1(x)>(1−α)f0(x)\alpha f_1(x)>(1-\alpha)f_0(x), giving the decision target κα⋆=log⁡[(1−α)/α]\kappa_\alpha^\star=\log[(1-\alpha)/\alpha]. Balanced mA uses α=1/2\alpha=1/2 and recovers the zero target. For ordinary per-attribute accuracy, αj\alpha_j equals the attribute's positive rate πj\pi_j under the same reference distribution, giving κj⋆=−logit⁡πj\kappa_j^\star=-\operatorname{logit}\pi_j.

The training specification and inference threshold determine the objective-implied coordinate, while the evaluation metric determines the target. For weighted BCE with positive- and negative-class weights a,b>0a,b>0, the coordinate remains κobj(t)=logit⁡t+log⁡(b/a)−logit⁡π\kappa^{\mathrm{obj}}(t)=\operatorname{logit}t+\log(b/a)-\operatorname{logit}\pi. The normalized threshold solves κobj(tObj,α)=κα⋆\kappa^{\mathrm{obj}}(t_{\mathrm{Obj},\alpha})=\kappa_\alpha^\star. For ordinary accuracy (α=π\alpha=\pi), this gives tObj,α=a/(a+b)t_{\mathrm{Obj},\alpha}=a/(a+b), which reduces to 0.50.5 for ordinary BCE.

The new Appendix B.6 provides the derivation and applicability conditions, with a reference from Section 3.1. Section 3.2 and Appendix A.2 now explain the verification criterion and the meaning of the 38/66 count more explicitly. All these updates are marked in blue.

AuthorsOpenReview ↗

Response to Reviewer JJ8Q (2/2)

[Q3] Est. evaluates mA under a common decision target.

We agree that AUC provides a natural view of ranking quality. The benefit of Est. for our study is that it retains mA as the evaluation criterion while targeting a common balanced-accuracy decision reference. As formulated in Section 3.4 and evaluated in Section 4.2, we use Native and Est. to examine whether a method's Native mA advantage persists under the normalized decision rules, holding each predictor fixed across the two views. This directly addresses the dependence of the reported mA comparison on operating policy. AUC provides complementary information about ranking performance across operating points.

For an individual attribute, balanced accuracy at a score threshold is BA(τ)=12[1+TPR(τ)−FPR(τ)]\mathrm{BA}(\tau)=\frac{1}{2}\left[1+\mathrm{TPR}(\tau)-\mathrm{FPR}(\tau)\right], whereas ROC-AUC integrates the ROC curve over the false-positive rate. For a fixed predictor's score, even the maximum BA over thresholds depends on the maximum of TPR−FPR\mathrm{TPR}-\mathrm{FPR}, rather than the area under the curve. The two criteria therefore need not rank predictors identically. Our actual Est. rule uses validation-fitted calibration maps and training priors to construct thresholds targeting κ=0\kappa=0, and freezes them before test evaluation (Appendix C.3). Est. reports test mA under these validation-constructed decision rules; it does not maximize mA on the test set.

The full comparison of Est. mA, macro-AUROC, and macro-AP is reported in Appendix E.5, Table 16 and reproduced in our response to Reviewer@C1Ke [W1/Q1]. For the controlled ResNet-50 predictors in Table 3, we computed ROC-AUC separately for each attribute from the same test logits and labels used to evaluate Est., then averaged equally across attributes. On PETA, ASL has higher Est. mA than RWinvRW_{\mathrm{inv}} (87.91 vs. 87.33), while RWinvRW_{\mathrm{inv}} has higher macro-AUROC (94.49 vs. 94.36). Thus, the AUROC ordering alone does not tell us which method achieves higher mA at the common balanced-accuracy target.

Reviewer_JJ8QOpenReview ↗

Comment to authors' response

Thank you for the author's detailed response and revisions. The three concerns I mentioned last time have all received corresponding responses.

Regarding Q1, the author now explicitly states that Native mA evaluates deployment methods based on its recorded operational strategies, while Obj./Est. Evaluate the same fixed predictor under a common balance accuracy objective. Combining the revised abstract with Sections 4.2 and 3.4, the division of labor between the three viewpoints has been clarified. Actually, what I am concerned about is the ambiguity of the wording, not the framework itself. The author's existing modifications have further improved the article and enhanced the rigor of its logic

Q2. The author's description that "38/66 counting refers to the validation of the reported mA definition or evaluation scheme, and does not imply the use of imbalanced metrics in other papers" is acceptable. In addition, the new Appendix B.6 provides the derivatives I requested, which enhances the content of the article.

Q3. Regarding the difference between the two indicators, the author also made a reasonable comparison and analysis in their response, that is, the optimal BA for a threshold only depends on the maximum TPR-FPR difference, while the average value of ROC-AUC across all thresholds. Table 16 and PETA examples also provide evidence that AUROC and mA can be used to rank methods differently.

I am pleased to suggest acceptance.

Reviewer_CX2LOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how training objectives and inference thresholds affect mean accuracy (mA) comparisons in pedestrian attribute recognition. It audits 66 papers and 28 official implementations, derives objective-implied decision coordinates, and estimates predictor-specific coordinates through validation calibration. Using fixed predictors on PETA, PA100K, and RAP, it shows that threshold normalization can shrink method gaps and reverse mean orderings, while some advantages remain.

Critical strengths
  1. On RAP, the RWinv–ERM gap falls from over 8 to below 0.3 percentage points under Est. on both backbones, with the checkpoints fixed (Section 4.2, Table 3). This isolates a substantial threshold effect in the reported mA advantage.

  2. Table 1 documents a real reporting problem: 61 of 66 papers do not state the final decision rule, and the thresholded implementations use only three numerical cutoffs despite different objectives. The derivations in Section 3.3 explain why matching those numbers is insufficient for comparing decision policies.

  3. The large RAP contraction survives changes in calibrator and validation sample (Section 4.4). The authors also report advantages that persist or grow, such as L2L over PIL on RAP (Section 4.2).

Critical weaknesses
  1. Section 4.3 uses residual coordinates to diagnose departures from the mA target, but ASL's median residual on PA100K has opposite signs under isotonic and sigmoid (Table 12). Appendix E.1 acknowledges this dependence on the calibrator without quantifying variation across validation refits. Please report per-attribute sign agreement between calibrators and the frequency of sign changes relative to each full-validation fit when the validation data are resampled and the calibrators refitted.

  2. Section 5 recommends isotonic for its flexibility, but Section 4.4 reports slightly higher mA with sigmoid in most tested settings and lower variability of method gaps in every examined validation-subsampling comparison. PromptPAR's PETA drop from Obj. 88.99 to Est. 88.02 is not covered by these comparisons (Table 3). Please explain why isotonic is the preferred reference given these results, and report sigmoid and Val-opt. results for PromptPAR on PETA.

  3. In the CNN group, ReSAGE-PAR's Est. lead over FRDL+GOAT is about 0.35 points on PETA and 0.32 on RAP (Table 3). Per-method seed variation is reported, but paired intervals for these gaps are absent from Table 13. Please provide the individual seed results and paired intervals using Appendix E.2's procedure.

  4. PromptPAR uses a released checkpoint (Section 4.1), but RAP's calibration images come from the original training pool (Appendix D.1). Their overlap with the checkpoint's supervised training and selection data is unspecified. Please identify those splits and report any overlap with the calibration images.

Questions for authors
  1. For ASL on PA100K, how often do isotonic and sigmoid agree on the sign of each attribute's residual coordinate, and how often does each sign change relative to its full-validation fit after validation resampling and refitting?
  2. What supports choosing isotonic as the reference over sigmoid, and does PromptPAR's loss relative to Obj. on PETA persist under sigmoid and Val-opt.?
  3. What are the individual seed results and paired intervals for the Est. gaps between CNN-based FRDL+GOAT and ReSAGE-PAR on PETA and RAP?
  4. Which data were used for supervised training and selection of the released PromptPAR checkpoint, and do any of the RAP calibration images overlap with either split?
3 comments
AuthorsOpenReview ↗

Response to Reviewer CX2L (1/3)

Thank you for your careful review and for your constructive feedback. We appreciate your positive assessment of our audit of decision-rule reporting and of the comparisons we made with checkpoints held fixed. Below, we address each concern together with the corresponding question.

[W1/Q1] Per-attribute sign agreement and validation-refit stability for ASL on PA100K.

For ASL on PA100K, the median residual has opposite signs under isotonic and sigmoid calibration on both backbones. At the same fixed Obj. thresholds, however, the two full-validation fits agree on the residual sign for 57 of 78 attribute–predictor pairs (73.08%) on ResNet-50 and 68 of 78 (87.18%) on ViT-B/16.

To assess sign stability, we drew 1,000 bootstrap samples of 10,000 validation images with replacement and refitted both calibrators on each shared sample. For each attribute and score set, we first recorded each calibrator’s residual sign on the full validation set. We counted a flip whenever the same calibrator gave the opposite sign after refitting. The Obj. threshold and training prior were held fixed throughout. The following table summarizes 78,000 comparisons per calibrator and backbone (26 attributes × 3 score sets × 1,000 refits). It reports lower overall flip rates for isotonic than for sigmoid on both ResNet-50 (3.53% versus 11.41%) and ViT-B/16 (1.77% versus 7.08%).

BackboneFull-validation sign agreementIsotonic sign flipsSigmoid sign flips
ResNet-5057/78 (73.08%)2,756/78,000 (3.53%)8,903/78,000 (11.41%)
ViT-B/1668/78 (87.18%)1,383/78,000 (1.77%)5,520/78,000 (7.08%)

The table below reports per-attribute results in the order ResNet-50 / ViT-B/16. The agreement count is the number of score sets (out of three) for which the two calibrators gave the same sign on the full validation set. Each flip rate is based on 3,000 comparisons per attribute and backbone (three score sets × 1,000 refits). On ResNet-50, the two calibrators disagreed in all three score sets for four attributes (UpperSplice, Female, Front, and Side). These four attributes accounted for 12 of the 21 full-validation sign disagreements on ResNet-50, compared with just 3 disagreements in the 12 corresponding comparisons on ViT-B/16.

AttributeFull-fit agreements (out of 3 each)Isotonic flip rate (%)Sigmoid flip rate (%)
Hat3 / 30.00 / 0.000.00 / 8.93
Glasses3 / 33.67 / 0.0014.77 / 0.73
ShortSleeve3 / 21.50 / 0.0715.50 / 4.30
LongSleeve3 / 30.00 / 0.031.20 / 8.97
UpperStride2 / 232.80 / 3.301.13 / 13.10
UpperLogo3 / 30.00 / 0.0012.13 / 1.23
UpperPlaid2 / 32.57 / 0.0013.40 / 0.87
UpperSplice0 / 22.57 / 2.9312.47 / 21.13
LowerStripe1 / 110.30 / 13.5719.63 / 21.17
LowerPattern1 / 25.33 / 3.8339.40 / 15.60
LongCoat3 / 30.00 / 0.000.40 / 1.37
Trousers3 / 30.00 / 0.003.43 / 4.47
Shorts1 / 315.33 / 13.2312.53 / 0.07
Skirt&Dress3 / 20.00 / 1.434.67 / 9.23
boots3 / 316.97 / 6.3315.67 / 8.83
HandBag3 / 30.00 / 0.000.00 / 0.00
ShoulderBag3 / 30.00 / 0.000.00 / 8.17
Backpack3 / 30.00 / 0.000.00 / 0.00
HoldObjectsInFront3 / 30.00 / 0.002.07 / 8.50
AgeOver603 / 30.00 / 0.073.27 / 2.40
Age18-603 / 30.00 / 0.000.00 / 0.00
AgeLess183 / 30.00 / 0.000.00 / 0.00
Female0 / 20.00 / 1.3019.73 / 5.57
Front0 / 20.00 / 0.0017.57 / 16.00
Side0 / 30.83 / 0.0036.90 / 8.03
Back2 / 20.00 / 0.0050.90 / 15.33
AuthorsOpenReview ↗

Response to Reviewer CX2L (2/3)

[W2/Q2] Choice of the isotonic reference and PromptPAR calibration sensitivity.

As explained in Section 5, we use isotonic as the reference because we want to compare decision coordinates across predictors and attributes without assuming a sigmoid shape for every score-to-posterior curve. Isotonic lets the validation data determine each curve’s shape subject to monotonicity. We use the fitted curve to estimate the coordinate at the predictor’s native threshold and invert it at the training prior to construct the Est. threshold.

Our framework also accommodates a monotone sigmoid, which we use to check whether the method comparisons hold under a more constrained, two-parameter curve. Sigmoid gives slightly higher overall test mA in most settings (Figure 4). It also gives smaller standard deviations for the method gaps across validation subsets in every comparison examined (Figure 5). With either calibrator, the large Native gap between RWinvRW_{\mathrm{inv}} and ERM contracts under Est. on both RAP backbones. On PETA with ViT-B/16, HAP ranks below ReSAGE-PAR under Obj. and above it under either Est. rule. Some closer rankings do change with the calibrator (Table 4), which is why we report sigmoid results alongside the isotonic reference.

The PromptPAR entry in the original manuscript is now labeled *PromptPAR, denoting the released CLIP ViT-L/14 checkpoint. The official code defaults to training on trainval, which includes our validation images. However, the available release records do not fully establish the evaluated checkpoint’s training and selection history. We therefore additionally trained PromptPAR with CLIP ViT-L/14 using only the training set, with validation data used for calibration, to provide a comparison with explicit separation between supervised training and calibration.

Predictor on PETA (mA, %)NativeObj.Isotonic Est.Sigmoid Est.Val-opt.
*PromptPAR (released)87.7988.9988.0289.0287.93
PromptPAR (three runs)86.85 ± 0.1288.99 ± 0.1288.92 ± 0.1389.10 ± 0.1988.92 ± 0.10

For the released *PromptPAR checkpoint, the decrease relative to Obj. persists under Val-opt., but not under sigmoid. For our trained PromptPAR, isotonic Est. and Val-opt. remain close to Obj., while sigmoid gives slightly higher mA. Together, these results show that the observed change relative to Obj. depends on the evaluated checkpoint and threshold-construction rule. A possible explanation is that training–calibration overlap, if present in the released checkpoint, affected the transfer of validation-derived thresholds to the test set.

Table 3 now distinguishes the released *PromptPAR and retrained PromptPAR checkpoints and reports their Native, Obj., and isotonic Est. results. Their configurations and data protocols are clarified in Section 4.1 and Appendices D.1 and D.2.2.

AuthorsOpenReview ↗

Response to Reviewer CX2L (3/3)

[W3/Q3] Individual results and paired intervals for the CNN comparison.

The table below reports individual test mA values (%) for FRDL+GOAT and ReSAGE-PAR on PETA and RAP. For each method and dataset, we used the same three ResNet-50 models and their saved test logits as in Table 3. We computed the summary statistics and method gaps before rounding the individual values to two decimal places for display.

MethodseedPETA NativePETA Obj.PETA Est.RAP NativeRAP Obj.RAP Est.
FRDL+GOAT60586.7786.7787.9384.1484.1486.02
FRDL+GOAT60686.9386.9388.2184.0084.0086.14
FRDL+GOAT60786.9286.9288.1684.3084.3085.86
ReSAGE-PAR60584.9988.3488.5780.5786.0686.23
ReSAGE-PAR60684.7188.1188.1680.6286.3286.61
ReSAGE-PAR60784.9788.3688.6180.2185.5486.16

Table 13 now reports the ResNet-50 comparison between ReSAGE-PAR and FRDL+GOAT on all three datasets (Appendix E.2, Table 13). The table below reproduces the PETA and RAP results requested here. Each entry gives the signed test mA gap (ReSAGE-PAR minus FRDL+GOAT, in pp) and its 95% paired bootstrap interval.

DatasetNativeObj.Est. (isotonic)Est. (sigmoid)Val-opt.
PETA−1.98 [−2.13, −1.59]+1.40 [+1.23, +1.62]+0.35 [+0.07, +0.67]+0.56 [+0.35, +0.75]+0.36 [+0.08, +0.66]
RAP−3.69 [−4.43, −2.84]+1.82 [+0.98, +2.38]+0.32 [+0.21, +0.99]+0.31 [+0.14, +0.53]+0.32 [−0.11, +0.67]

[W4/Q4] PromptPAR checkpoint provenance and calibration overlap.

For the released checkpoint, denoted *PromptPAR in our response to [W2/Q2], the official code specifies trainval as the default supervised-training split. However, the available release records do not fully establish the actual training configuration or the data used to select each released checkpoint. We therefore cannot determine whether any of our RAP calibration images were used to train or select the released RAP checkpoint. The overlap implied by the official default training protocol is as follows:

DatasetDefault supervised-training poolOur calibration setCalibration images included in the default training pool
PETA11,4001,9001,900/1,900
RAP33,2683,6963,696/3,696
PA100K90,00010,00010,000/10,000

Thus, if the released checkpoints followed this default protocol, all calibration images would have participated in supervised training. For all methods retrained in our study, calibration images are disjoint from the supervised-training subsets, including the additional PromptPAR runs introduced in [W2/Q2]. We have clarified the data partitions and available checkpoint provenance in Appendices D.1 and D.2.2, and added the train-only PromptPAR results in Table 3.

Reviewer_C1KeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper argues that mean accuracy (mA), which is the macroscopic average of attribute-specific balance accuracy used as a key Probability-Related Assessment (PAR) metric, is incomparable across methodologies at the same probability threshold. This is because handling label imbalance (weighting, resampling, focus/asymmetric loss) alters the likelihood ratio boundary implemented by a fixed threshold, such as 0.5. An analysis of 66 PAR papers and 28 formula implementations revealed that final thresholds were rarely reported (2/66), and 0.3, 0.45, or 0.5 were used in 27/28 codebases. The authors derive objective function-based boundary coordinates κ^obj for identifiable loss and sampling components, and estimate score-space coordinates for the trained predictor through rank correction on the validation data. Nine controlled interventions and seven disclosed methods (formal implementations, native backbones) on ResNet-50 and ViT-B/16 backbones on the PETA, PA100K, and RAP datasets were evaluated from three perspectives (native, target value normalization, and realization policy normalization), with each checkpoint fixed. In RAP, the RWinv-ERM gap narrowed from 8.99/8.16 pp (native) to 0.12/0.28 pp (estimated), and the mean order among the disclosed methods was reversed for some, while others remained unchanged or widened. Sensitivity analysis of bootstrap intervals, calibration models, validation subsampling, and reference prior probabilities supports these patterns.

Critical strengths

This paper links specific gaps in evaluation practices with actual comparison results. Of the 66 papers reviewed, only 2 specified the final threshold numerically, and 27 out of 28 formal implementations reused 0.3, 0.45, or 0.5 across heterogeneous, disproportionate interventions. Since significant gap reduction and mean rank reversals occur in both control and existing method comparisons, these observations influence how a significant portion of reported PAR results are interpreted and reproduced, rather than being merely theoretical possibilities. The fact that some benefits persist or expand even after normalization also supports a balanced interpretation that not all benefits are reduced to threshold effects. This is the primary reason I am inclined to accept this paper.

Critical weaknesses

Contributions beyond standard validation-based threshold selection have not been clearly established. The authors link isotonic/PAV inversion to validation optimal threshold selection (§4.4, Appendix C.4), and in the three comparisons in Appendix E.2, the difference between the estimate and the validation optimal is only up to 0.035 pp for mean test mA and up to 0.013 pp for methodological differences. In Appendix E.3, pre-validation inversion shows a difference of less than 0.007 pp from the validation optimal across 50 settings. Therefore, the estimate is equivalent to attribute-specific threshold adjustment for the validation data, which is a long-standing practice. Furthermore, this paper does not compare the method of reporting attribute-specific AUROC/AP without thresholds alongside base mA, which is the most direct alternative for addressing the same operational policy problem without correction devices. Therefore, additional value can be seen in auditing, κ diagnostics, and target/estimate contrast, and the target is... This methodology itself is limited to identifiable components (not applicable for DBs; component level for disclosed methods). If it does not demonstrate how this methodology can derive conclusions that cannot be obtained from mA (or AUROC) reports validated and adjusted from existing methodologies, its methodological contribution in this field is minimal, and this is the main reason I did not give it the highest score.

Questions for authors

[Contribution - Explanation] Compared to a simple Native + Validation Tuned mA (Val-opt.) report, what additional diagnostic information does the κ coordinate provide? Please provide specific examples of conclusions that cannot be derived from Native and Val-opt. figures but require κ^obj/κ^est. In this regard, Appendix B.5 states that κ^est − κ^obj represents a mixture of the deviation between the learned score response and the response implied by the objective, as well as the corrected estimation error; to what extent can these two be separated in the current analysis? Finally, if attribute-specific AUROC/AP (no threshold) is reported for the same checkpoint, is the rank inversion in Table 3 maintained? If so, why is normalized mA recommended over such metrics?

[Rationale for Rank Inversion - Explanation] If the pairwise uncertainty analysis in Table 13 is applied to the FRDL+GOAT vs. ReSAGE-PAR comparison for all three datasets, is it possible to maintain the inversion in intervals excluding Native→Est. 0? This significantly impacts the importance of reinterpreting comparisons with existing methods.

[Protocol Reliability - Explanation and Proposal] How many attribute-predictor pairs in the entire validation set do not meet the support criteria in Appendix C.3? (D.3 reports this only in partial sampling.) As a proposal for the final version, associating the attribute-specific min(n_pos, n_neg) values ​​with the bootstrap variability of the estimated threshold and the test-balanced accuracy will allow readers to determine when the protocol is applicable to rare attributes.

3 comments
AuthorsOpenReview ↗

Response to Reviewer C1Ke (1/3)

We appreciate your recognition of the practical value of our audit and your thoughtful questions about what the decision coordinates reveal beyond validation-tuned mA and how reliably we can interpret changes in method rankings. We address these points in turn:

[W1/Q1] What do the decision coordinates add beyond Native and Val-opt. mA?

We agree that Est. (Isotonic) and Val-opt. can give very similar test mA. For the two RWinv−ERMRW_{\mathrm{inv}}-\mathrm{ERM} comparisons on RAP and the HAP–ReSAGE-PAR comparison on PETA, the differences are at most 0.035 pp in mean test mA and 0.013 pp in the method gap (Appendix E.2). Appendix C.4 explains this agreement: at the validation positive rate, thresholding the isotonic fit yields a validation-optimal score cut when an entire block tied at the target is assigned to one class. The difference of less than 0.007 pp reported in Appendix E.3 is for inversion at this validation positive rate. Our main Est. rule uses the training positive rate, so the two rules can give similar mA without selecting exactly the same threshold. We use isotonic as a monotone reference without imposing a sigmoid shape and examine sensitivity to this choice in Figure 4.

Even when Est. and Val-opt. give the same test mA for a fixed predictor, the fitted calibration map also lets us estimate the decision coordinate at its Native threshold. For an identifiable training component, we can compare that estimate with the coordinate implied by the training rule. Below, we address the three parts of your question: what the coordinates reveal beyond Native and Val-opt. mA, how far we can separate predictor deviation from calibration error, and whether macro-AUROC and macro-AP give the same method ordering as Est. mA.

  • The coordinates distinguish agreement with the training objective from alignment with the mA target. On RAP with ResNet-50, ERM has a smaller median absolute discrepancy between its estimated and objective-implied Native coordinates than FRDL (0.336 vs. 0.906), yet its estimated Native coordinate is farther from the mA target (2.681 vs. 0.906 in median absolute distance; Table 11, isotonic estimates). The two distances coincide for FRDL because its analyzed classification component has κobj=0\kappa^{\mathrm{obj}}=0 at the Native threshold. Native and Val-opt. mA report the performance of two decisions, but those values alone cannot tell us which predictor more closely follows its objective-implied policy or which operates closer to the common target.

  • We assess which coordinate diagnostics persist across calibrators. In all six dataset–backbone settings, both isotonic and sigmoid calibration give ERM a smaller discrepancy from its objective-implied Native coordinate than FRDL, yet a larger distance from the mA target (Appendix E.1). For ASL on PA100K, the median residual at the Obj. thresholds changes sign between calibrators on both backbones (Table 12), so its direction is calibrator-dependent. These comparisons quantify sensitivity to the fitted map. As explained in Appendix B.5, separating predictor deviation from calibration error would require a reliable reference for the true score-to-posterior relation at the threshold. In the current analysis, we therefore interpret κest−κobj\kappa^{\mathrm{est}}-\kappa^{\mathrm{obj}} as a combined discrepancy.

  • Macro-AUROC and macro-AP do not consistently give the same method ordering as Est. mA. For the FRDL+GOAT and ReSAGE-PAR comparison in Table 3, Est. mA ranks ReSAGE-PAR higher on all three datasets. Macro-AP gives the same ordering on all three, while macro-AUROC ranks ReSAGE-PAR higher only on PA100K and ranks FRDL+GOAT higher on PETA and RAP. Macro-AP can also disagree with Est. mA. Among the nine controlled methods on RAP, DB ranks first by macro-AP but ninth by Est. mA. We report Est. mA because it directly measures performance at decision thresholds targeting balanced accuracy. Macro-AUROC and macro-AP provide complementary information about score rankings across thresholds.

The two tables below reproduce the comparison in Appendix E.5, Table 16. AUROC/AP gives macro-AUROC and macro-AP, respectively. Rank is shown as Est. mA → (AUROC/AP) and is computed from unrounded values within each dataset and method group (1 = highest). F.+G. and R.-PAR denote FRDL+GOAT and ReSAGE-PAR, respectively.

AuthorsOpenReview ↗

Response to Reviewer C1Ke (2/3)

Controlled methods (ResNet-50)

MethodPETA Est.PETA AUROC/APPETA RankPA100K Est.PA100K AUROC/APPA100K RankRAP Est.RAP AUROC/APRAP Rank
ERM87.9394.54/80.764 → (5/6)85.9792.89/71.504 → (4/4)85.3991.58/68.135 → (5/5)
RWexp⁡RW_{\exp}87.9894.63/80.763 → (3/5)86.1992.98/71.321 → (2/5)85.5891.86/67.703 → (1/7)
RWinvRW_{\mathrm{inv}}87.3394.49/79.859 → (6/8)85.2692.34/67.719 → (8/9)85.5291.65/66.614 → (4/8)
FRDL87.5394.39/79.867 → (7/7)86.0593.02/71.783 → (1/2)85.2091.48/68.058 → (7/6)
MLFL87.9994.63/81.022 → (2/3)85.5992.61/69.977 → (7/8)85.6091.69/68.732 → (3/3)
ASL87.9194.36/81.295 → (8/2)85.9092.65/71.815 → (6/1)85.2991.45/68.926 → (8/2)
CB87.8194.61/78.986 → (4/9)86.1792.97/70.132 → (3/7)85.2391.58/64.597 → (6/9)
LA88.0294.68/81.021 → (1/4)85.8992.74/70.926 → (5/6)85.6891.81/68.281 → (2/4)
DB87.4994.18/82.098 → (9/1)85.3392.09/71.748 → (9/3)84.8190.89/69.789 → (9/1)

Published CNN methods

MethodPETA Est.PETA AUROC/APPETA RankPA100K Est.PA100K AUROC/APPA100K RankRAP Est.RAP AUROC/APRAP Rank
L2L87.6294.43/80.663 → (1/1)85.5992.00/70.364 → (3/3)86.3492.24/69.181 → (1/2)
PIL87.0893.39/77.104 → (4/4)86.0291.93/68.063 → (4/4)83.6589.68/60.474 → (4/4)
F.+G.88.1094.28/79.722 → (2/3)87.2892.87/70.522 → (2/2)86.0191.27/68.773 → (2/3)
R.-PAR88.4594.21/80.371 → (3/2)87.9794.34/74.361 → (1/1)86.3391.05/70.662 → (3/1)
AuthorsOpenReview ↗

Response to Reviewer C1Ke (3/3)

[Q2] Does the FRDL+GOAT versus ReSAGE-PAR ordering reverse on all three datasets after normalization?

Yes. FRDL+GOAT leads under Native and ReSAGE-PAR leads under Est. (Isotonic) on all three datasets. The 95% paired interval for the Native-to-Est. change in the gap lies entirely above zero on each dataset. The table below gives the signed mA gaps (ReSAGE-PAR minus FRDL+GOAT, in pp) and their 95% paired bootstrap intervals. On PETA, PA100K, and RAP, the Native intervals lie entirely below zero and the Est. intervals lie entirely above zero (Appendix E.2, Table 13). For the final column, we subtracted the Native gap from the Est. gap within each shared, supported bootstrap replicate and took the 2.5th and 97.5th percentiles of those changes.

DatasetNative gap [95% interval]Est. gap [95% interval]Native-to-Est. change [95% paired interval]
PETA−1.98 [−2.13, −1.59]+0.35 [+0.07, +0.67]+2.33 [+1.43, +2.57]
PA100K−3.05 [−3.41, −2.54]+0.69 [+0.28, +1.30]+3.74 [+3.15, +4.37]
RAP−3.69 [−4.43, −2.84]+0.32 [+0.21, +0.99]+4.01 [+3.15, +4.97]

[Q3] Full-validation support and threshold uncertainty for rare attributes.

Your question raises two aspects of reliability: whether a threshold can be constructed on the full validation set and how stable that threshold is when the validation images are resampled.

(1) All 8,848 attribute–predictor pairs in Table 3 supported isotonic Est. threshold construction on the full validation sets. Under the criteria in Appendix C.3, the unsupported counts were 0/2,765 on PETA, 0/2,054 on PA100K, and 0/4,029 on RAP. Each dataset contributed 79 fixed predictors across the controlled and published methods, with 35, 26, and 51 attributes, respectively. For the three comparisons examined in Figure 5 and Appendix D.3, we also checked both calibrators on the full validation sets: all 822 pairs supported threshold construction under both isotonic and sigmoid calibration. The unsupported comparison replicates reported in D.3 occurred only under sigmoid calibration with 25% validation subsets. On the full validation sets, both classes were present for every attribute, and each fitted isotonic map reached the training prior within the observed score range. Smaller subsets can lose coverage of sparsely sampled score regions. For example, the rarest attribute on RAP has 14 positive validation images, so a random 25% subset contains only 3.5 positives on average.

(2) Smaller minority-class counts were associated with greater variability in Est. thresholds and per-attribute test mA. Following your suggestion, we used min⁡(npos,nneg)\min(n_{\mathrm{pos}}, n_{\mathrm{neg}}) for each attribute, counting positive and negative images on the full validation set. We reused the 1,000 validation bootstrap refits for ERM and RWinvRW_{\mathrm{inv}} with ResNet-50 (Appendix E.6, Table 17; see also Reviewer@xLTa [W1]). For each attribute, we averaged the three fixed predictors' SDs as in Table 17. We then computed Spearman correlations across attributes, separately for each dataset, method, and calibrator. We have added these correlations to the revised paper (Appendix E.6, Table 18).

Both correlations were negative in all 12 settings. Thus, even when a threshold is supported on the full validation set, a small minority-class count can indicate greater sensitivity to validation sampling. The strength of the association varies across settings, so these results do not establish a universal minimum count. Each cell below gives the correlation with Est. threshold percentile SD / per-attribute test mA SD, with test images held fixed throughout.

DatasetAttributesERM Est. (Isotonic)ERM Est. (Sigmoid)RWinvRW_{\mathrm{inv}} Est. (Isotonic)RWinvRW_{\mathrm{inv}} Est. (Sigmoid)
PETA35−0.34 / −0.76−0.62 / −0.83−0.29 / −0.76−0.39 / −0.74
RAP51−0.39 / −0.91−0.57 / −0.88−0.34 / −0.83−0.46 / −0.86
PA100K26−0.68 / −0.90−0.69 / −0.90−0.57 / −0.80−0.74 / −0.90

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Comment

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all five reviewers for the time and care you have devoted to our paper. Your thoughtful questions and constructive suggestions helped us clarify what changes in mA can tell us about a method's reported advantage and strengthen the evidence for our conclusions. We are also grateful for the reviewers' encouraging comments on the following aspects of our work.

  • The audit changes how published gains are interpreted. Some advantages shrink when thresholds are normalized, while others persist or grow. (Reviewer@C1Ke)
  • The RAP gap contracts across calibration choices. The contraction holds on both backbones and across validation subsets with the predictors held fixed. (Reviewer@CX2L)
  • A common coordinate connects different training objectives. The framework expresses threshold policies on the same scale for reweighted, focal, and asymmetric losses. (Reviewer@JJ8Q)
  • The same threshold can imply different decisions. The reviewer appreciated the clear motivation and the empirical evidence that a shared numerical threshold can represent different operating policies. (Reviewer@Po7R)
  • The issue extends across PAR benchmarks. The theoretical analysis and experiments on three datasets and two backbones address an often-overlooked consequence of label imbalance for evaluation. (Reviewer@xLTa)

We are grateful for your specific suggestions on where the paper would benefit from clearer explanations and additional evidence. Below, we summarize how we addressed each reviewer's questions, with pointers to the relevant sections of the revised paper.

  • Decision coordinates add information beyond tuned mA. We distinguish agreement with the training objective from proximity to the mA target. We also report paired intervals for the CNN ranking reversal, AUROC/AP comparisons, full-validation support counts, and uncertainty by minority-class count (Appendices D.3, E.2, E.5, and E.6). (Reviewer@C1Ke)
  • Calibration sensitivity is visible at the attribute level. We report ASL sign agreement and stability across refits, explain the isotonic reference, and give the individual CNN runs. For PromptPAR, we compare calibration rules on PETA and clarify the provenance of the released and retrained checkpoints (Section 5, Table 3, and Appendix D.2.2). (Reviewer@CX2L)
  • The evaluation metric determines the decision target. We clarify why Native and normalized mA are both informative, derive the target for class-weighted accuracy, and show why AUROC alone cannot determine the Est. mA ordering (Section 3.4 and Appendices B.6 and E.5). (Reviewer@JJ8Q)
  • The RAP gap also contracts under objective-derived thresholds. This comparison shows what we can learn before fitting thresholds to validation labels (Sections 3.4 and 4.2). We also examine instance-level metrics, discuss JGEL and PFM-VEPAR, and share our anonymous code repository (https://anonymous.4open.science/r/par-evaluation-code-2603). (Reviewer@Po7R)
  • Many pairwise advantages survive a change in the top rank. Across the published-method groups, 20 of 36 pairwise orderings persist, while the leader changes in five of six settings (Table 3). We also quantify uncertainty for rare attributes (Appendix E.6), compare resampling with reweighting, and revisit four method families from the PromptPAR study. (Reviewer@xLTa)

The revisions are highlighted in blue. Below, we respond to each concern and question in detail in the corresponding reviewer's thread. Thank you again for your time and thoughtful suggestions.

Sincerely,
The Authors

AuthorsOpenReview ↗

Thank you for your thoughtful review

Dear Reviewer JJ8Q,

We are grateful for the time and care you devoted to reviewing our manuscript and considering our revisions. Your thoughtful questions helped us clarify why we report Native and normalized mA together and what each measure tells readers about a method’s performance. We are pleased to hear that our responses and revisions have addressed all three of your concerns, and we sincerely appreciate your recommendation for acceptance.

Yours sincerely, The Authors

AuthorsOpenReview ↗

Thank you for your careful review and positive assessment

Dear Reviewer CX2L,

We sincerely appreciate your careful review and constructive feedback. We are pleased that our clarifications have addressed your concerns, and we thank you for your continued positive assessment of our work.

Yours sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.