acceptodds
Under review as a conference paper at ICLR 2027

The Selection Rule Decides the Winner: A Pre-Registered Audit of Open-Set Graph Anomaly Detection

46 views

Abstract

Open-set graph anomaly detection trains on a few labeled anomalies from one class and must also find anomaly classes that were never labeled. Published results share three conventions: the test score is read at the best epoch on the test set, baseline numbers are copied from earlier papers, and most anomalies are minority classes relabeled as anomalous. We ask how much of the reported ranking these conventions decide. We re-run two recent methods, DEMO and NSReg, together with OUTPOST, a small first-order detector built for this study. All three use one protocol with identical seeds and splits on eight graphs (seven for the baselines, which cannot run on ogbn-mag), ten seeds each, and every run is scored under both the best-epoch rule and a deployable validation rule. Before the runs that test them, we registered 40 predictions. Three findings hold. First, the rule changes the leader: under the best-epoch rule, OUTPOST and NSReg each lead three of seven graphs, while under the validation rule, NSReg leads five. Second, the best-epoch bonus depends on how the benchmark was built: 0.045–0.080 AUC-ROC on the three small relabeled-class graphs and 0.002–0.014 on the three real fraud graphs. Third, pseudo-labeling in OUTPOST is worth 0.038–0.065 AUC-ROC on the same three graphs but gives no benefit on any real fraud graph. We also show that a 0.002 tie band for hyperparameter selection lies below the paired standard error on all six graphs tested, even at ten seeds. Twelve of our 40 predictions were falsified, and we report them. We close with a short reporting checklist. Our code is available at https://anonymous.4open.science/r/OUTPOST.

Primary area:
datasets and benchmarks
Keywords:
graph anomaly detection, open-set anomaly detection, evaluation protocol, model selection, reproducibility, benchmark validity, pseudo-labeling, pre-registration, graph neural networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ppLtOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the benchmarking setting on OpenSet GAD. It reveals that existing works use an oracle rule instead of a validation rule that practitioners can apply. Moreover, it proposes OUTPOST, a heavily engineered method whose performance is tied to baselines.

Critical strengths
  1. The paper investigates an interesting research problem: the reliability and robustness of benchmarking in open-set graph anomaly detection. From a real-world deployment perspective, this is a useful yet critical problem to research.

  2. This paper identifies the gap between research and deployed detectors - the oracle rule.

Critical weaknesses
  1. The scope of this work is overly narrow, limiting its potential impact and relevance. There are only two representative studies in this field for now. Conducting an empirical analysis of such a limited body of research is unlikely to generate broad interest within the research community.

  2. The author's opinion on the second convention is misleading and irresponsible. The authors claim in the Introduction that "Baseline numbers are copied from earlier papers ..." However, this claim does not appear to be supported by the existing open-set GAD literature. In particular, a comparison of the baseline results reported in NSReg and DEMO reveals differences in the reported values, as shown below.

PaperMethodPhotoComputersCS
NSRegTAM0.621±0.0040.435±0.0020.638±0.009
NSReg0.836±0.0310.752±0.0190.953±0.008
DEMOTAM0.6045±0.0150.4432±0.0140.6398±0.011
NSReg0.8360±0.0120.7403±0.0210.9032±0.035

These discrepancies indicate that DEMO did not simply copy the published baseline results from NSReg. Despite being an easy-to-spot issue, this incorrect statement still appears in the paper and lies at the center of the motivation, making me doubt the quality and technical solidity of the paper.

  1. OUTPOST is badly designed. a) It is not well-motivated and introduced properly. The authors rigidly combine this work with other experimental analyses. b) It is just a stacking of engineered components, rather than a well-designed innovation. There is no insight or novelty behind these modules. c) Despite claiming to be a competitive detector, as shown in Table 1, it performs worse than NSReg on 5 of 7 datasets when evaluated under the validation rule favored by the authors. d) It cannot even address the problem highlighted by the authors. As the results in Table 1 show, removing the oracle rule causes OUTPOST's performance to drop significantly.

  2. This paper is poorly written and organized. From an organizational perspective, despite being submitted to datasets and benchmarks, the method OUTPOST is still a major part of the contribution, which confuses people about what it aims to address.

Questions for authors
  1. Why do you include the method OUTPOST in your paper, despite it performing worse than NSReg on the majority of datasets?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ppLt (1/2): W2 and ethics concern

We thank the reviewer for checking our description of prior work against the sources. This comment addresses W2 and the ethics concern; the remaining points follow in a second comment.

W2 and the ethics concern

"The authors claim in the Introduction that 'Baseline numbers are copied from earlier papers ...' However, this claim does not appear to be supported by the existing open-set GAD literature."

The reviewer is correct that "copied" does not describe the two open-set papers. NSReg runs its own baselines, and DEMO trains its baselines on its own split (its Appendix F.1), which accounts for the differences in the reviewer's table. We have corrected the abstract, the introduction, and Table 5:

  • Introduction: "Each paper evaluates its baselines under its own splits, seed counts, and training budgets, so the same method receives different values in different papers, and values from different papers cannot enter a paired comparison."
  • Abstract: "baselines are evaluated under each paper's own splits, seeds, and budgets".
  • Table 5: the column of reported values now reads "Reported by Yu et al. (2026)", since its NSReg values are those DEMO reports for NSReg on its own split.

The point we intended is in the second half of the original sentence: values produced under different splits, budgets, and seed counts cannot be paired. The reviewer's table illustrates it. NSReg on CS is 0.953 in the NSReg paper and 0.9032 in DEMO, a gap of 0.050 that exceeds the 0.042 by which DEMO reports leading NSReg on that graph.

The first convention. We verified it in the same way. In DEMO's released trainer.py, the function eval keeps the maximum test value of each metric over all epochs and returns it as the result. NSReg's released runner (runners/train_runner.py) evaluates the test set during training and defines no separate validation set. Section 3 now states this.

Effect on the results. The correction changes no result. Every paired comparison and test in the paper uses our own re-runs on identical splits and seeds. Transcribed values appear only in a gray reference column, in Appendix D, and in one comparison on ogbn-mag, where neither DEMO nor NSReg can run; each is marked as unpaired.

We hope the correction resolves the concern, and we would be grateful if the reviewer would reconsider the ethics flag in light of it.

AuthorsOpenReview ↗

Response to Reviewer ppLt (2/2)

W1. Scope

"The scope of this work is overly narrow ... There are only two representative studies in this field for now."

The field is small, which is one reason to examine its conventions before later work adopts them; DEMO alone compares against 15 methods. Several findings also apply beyond the two papers. Validation saturation can arise in any benchmark built from relabeled classes with a small seen-class validation set, and the tie band below seed noise and the ablation gains collected only by the oracle rule concern semi-supervised GAD in general. The reporting checklist in Appendix L applies to any detector, not only the three studied here.

W3(a, b). Motivation and design of OUTPOST

"It is not well-motivated ... just a stacking of engineered components, rather than a well-designed innovation."

We built OUTPOST for the audit, and the revision states this role (contribution (iii) and Section 4). Measuring the contribution of a component requires a method in which every component can be removed, and a comparison on all eight graphs requires a method that runs on ogbn-mag, where neither baseline does. OUTPOST is built on established techniques, and we claim the findings it enables, not the components themselves. For example, pseudo-labeling carries the gains on the semi-synthetic graphs and gives none on real fraud, and the atlas gate has no measurable effect, which we report.

W3(c). Worse than NSReg under the validation rule

"... it performs worse than NSReg on 5 of 7 datasets when evaluated under the validation rule favored by the authors."

This is correct and is stated in the abstract. OUTPOST still has to be competitive for its ablations to be informative. Under the validation rule, OUTPOST is significantly ahead of NSReg on CS and Yelp, behind on Amazon, T-Finance, and ogbn-arxiv, and not significantly different on Photo and Computers (Table 10); in AUC-PR it leads on Computers, CS, and Yelp (Table 8). Which detector is better on most graphs therefore depends on the rule and the metric. We do not claim that OUTPOST is the best detector.

W3(d). Drop without the oracle rule

"... removing the oracle rule causes OUTPOST's performance to drop significantly."

The baselines show the same drop. On Photo, the oracle bonus is 0.114 for DEMO, 0.083 for NSReg, and 0.079 for OUTPOST; on Computers, OUTPOST's is the largest (0.080, against 0.051 and 0.052). Across methods it ranges from 0.041 to 0.114 on the semi-synthetic graphs and from 0.002 to 0.014 on the real ones, so it depends on the benchmark more than on the method (general response, points 3 and 4).

W4. Organization

"... the method OUTPOST is still a major part of the contribution, which confuses people about what it aims to address."

We agree. In the revision, contribution (iii) states the finding first and describes OUTPOST by its role in the ablations, Section 4 opens with this role and is shorter (details moved to Appendix B), and the SimSample analysis is in Appendix H.

Q1. Why include OUTPOST?

"Why do you include the method OUTPOST in your paper, despite it performing worse than NSReg on the majority of datasets?"

We use OUTPOST for the ablations, and it is the only one of the three methods that runs on all eight graphs. It is also competitive: it leads on CS and Yelp under all three rules, and under the validation rule it is significantly ahead of NSReg on both (p = 0.027 each).

Thank you to the authors for their responses. The authors' responses have addressed my concerns about ethical review. Despite that, my concerns about the quality of the work, the narrowed scope, and marginal innovation remain. I therefore keep my score unchanged.

Reviewer_Jcg1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This is a benchmark paper with a proposed baseline method specifically designed for open-set GAD. It aims to analyse whether the rankings of existing methods in the area are determined by the methods themselves or by the evaluation scheme. It tests three different evaluation conventions. The authors rerun two recent methods and one lightweight baseline implemented by themselves. All baselines are run under the same conditions. They find that rankings in open-set graph anomaly detection depend on the choice of evaluation scheme. These choices can even affect the rankings more than the methods themselves.

Critical strengths
  1. The evaluation design is interesting and sensible.
  2. The paper is general clear and easy to follow.
Critical weaknesses
  1. Only three baseline methods are evaluated. This is rather limited for a baseline and benchmark paper.
  2. The paper proposes an evaluation protocol towards fair evaluation. While the proposed protocol is deployable and reproducible, the authors do not show why evaluation using this protocol is a good or optimal standard.
  3. Even though the paper demonstrates the instability of existing open-set GAD methods, findings such as different data splits and epoch selection leading to performance variations are fairly well-known.
  4. The provenance claims made by the authors are based on only three graphs with relatively few attributes but high node degrees. This makes the claims lesssupported.
Questions for authors

Please see my weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Jcg1

We thank the reviewer. W2 and W4 led us to measure why seen-class validation fails on the semi-synthetic graphs (Section 5.2; general response, point 4).

W1. Number of methods

"Only three baseline methods are evaluated. This is rather limited for a baseline and benchmark paper."

Our central claim is that the selection rule can change the ranking of methods. One reversal under paired conditions shows that it can, and we observe two (Photo and Computers); more methods would show how often it happens, which is a broader question. The paper is not intended as a leaderboard. DEMO and NSReg are the two methods designed for open-set GAD, as Reviewer ppLt also notes; the other entries in published open-set tables are unsupervised detectors or detectors from other settings. Of the 9 method pairs on the three semi-synthetic graphs, the oracle and validation rules order 3 differently; of the 12 pairs on the other four graphs, none. The reorderings occur where the oracle bonus is large. Adding methods through their published numbers would reintroduce the unpaired comparison that the study examines.

W2. Why this protocol is a good standard

"... the authors do not show why evaluation using this protocol is a good or optimal standard."

We do not claim that the protocol is optimal. The revision measures where seen-class validation, the deployable default, falls short: on the semi-synthetic graphs it saturates, and for OUTPOST the label-free last epoch scores 0.043 to 0.054 above the validation-selected epoch (general response, point 4). What we claim is that two properties are necessary for a comparison to estimate deployed performance and to support a paired test: selection that reads no test label, and identical splits, seeds, and budgets. Selection on validation data is also the standard in GADBench (Tang et al., 2023). We therefore recommend reporting several rules, together with the validation ceiling and the tie-break (Appendix L), instead of a single rule.

W3. Novelty of the findings

"... findings such as different data splits and epoch selection leading to performance variations are fairly well-known."

We agree that sensitivity to splits and checkpoint choice is known in general; Section 2 cites Shchur et al., Errica et al., Gulrajani and Lopez-Paz, and Bouthillier et al. Re-evaluations of this kind have appeared at ICLR for related settings (Errica et al., 2020; Gulrajani and Lopez-Paz, 2021; Platonov et al., 2023), but none addresses open-set GAD. What was not known for this setting is the size of the effects and their consequences:

  1. On the semi-synthetic graphs, the oracle bonus (0.041 to 0.114 across methods) exceeds 13 of the 14 paired differences between methods.
  2. Of the apparent benefit of pseudo-labeling and the gate, 43 to 62% is collected only by the oracle rule, so ablation conclusions depend on the rule.
  3. The 0.002 tie band used for tuning lies below the paired standard error at three and at ten seeds, and two of the three selections we re-tested at ten seeds did not hold.
  4. For OUTPOST, the bonus comes mainly from the selection term, which concentrates in runs where seen-class validation saturates (Section 5.2).

We also registered 40 predictions before the runs that test them; 12 were falsified, and the paper reports them (Appendix M).

W4. Provenance on three graphs

"The provenance claims ... are based on only three graphs with relatively few attributes but high node degrees."

We agree that three graphs cannot separate provenance from the properties they share, and Section 6 now lists these factors. The saturation analysis addresses part of the concern without additional graphs. Within each semi-synthetic graph, where features and degree are fixed, runs whose validation reached 1.0 have a larger selection term in all nine method-graph pairs (Table R2). Across graphs, low degree alone does not produce a large bonus (ogbn-arxiv: median degree 1, relabeled anomalies, bonus 0.017), while feature dimension cannot be ruled out. Section 5.2 now presents the provenance effect as an association, not a cause.

Thank you to the authors for your response. My evaluation regarding the technical contribution of the paper remains the same. I therefore keep my score unchanged but am open to adjusting it after discussion with the AC and the other reviewers.

Reviewer_pZmSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript addresses an evaluation-auditing problem of broad importance across research fields. It identifies three common practices on which reported gains in open-set graph anomaly detection (GAD) may depend: selecting the checkpoint with the best test-set score (the oracle rule), directly reusing baseline results reported in the original papers, and constructing anomaly labels by relabeling minority classes. The manuscript investigates how these evaluation practices affect empirical performance rankings. The experiments yield several noteworthy findings: the model-selection rule changes the leading method on 2 of the 7 graphs; the oracle bonus varies monotonically with benchmark provenance, and on commonly used semi-synthetic benchmarks, the score shifts caused by the selection rule are larger than most pairwise differences between methods; and the gains from pseudo-labeling likewise depend on the source of the anomalies. These findings lead to concrete and readily actionable recommendations for improving evaluation practices in open-set GAD and potentially in the broader machine-learning community.

Critical strengths

The paper is a rigorous and practically important audit of evaluation practices. It examines an evaluation problem with implications beyond open-set graph anomaly detection: whether apparent methodological progress is partly determined by checkpoint selection, reused baseline results, and benchmark construction. By re-running DEMO, NSReg, and OUTPOST using identical splits, seeds, training budgets, and paired statistical comparisons, the study provides substantially more controlled evidence than comparisons based on numbers copied from prior papers. The pre-registration of 40 predictions, including the transparent reporting of falsified predictions, further strengthens the credibility and reproducibility of the audit.

Critical weaknesses

The paper’s central claims may be broader than the evidence supports. It compares only the oracle and validation rules, while validation AUC is computed using seen anomalies and may not reliably select models that detect unseen anomalies. Thus, the observed ranking reversals may be specific to these two criteria rather than reflecting general sensitivity to model-selection protocols. Moreover, the association between benchmark provenance and oracle bonus remains correlational: differences in training instability, anomaly prevalence, graph structure, or test-set overfitting are not disentangled, and data-split variance is not separated from initialization or epoch-selection variance. These limitations weaken the causal interpretation of the results.

Questions for authors
  1. The manuscript compares only two checkpoint-selection rules: Oracle, which selects the epoch with the highest test AUC, and Validation, which selects the epoch with the highest validation AUC. Is a comparison between only these two rules sufficient to support the strong claim in the title, “The Selection Rule Decides the Winner”? Could the reported effect simply reflect a ranking discrepancy between these two particular selection rules, rather than a more general sensitivity to model-selection protocols?
  2. In the open-set setting, the validation set contains no unseen anomalies. The authors select checkpoints based on validation AUC, which measures performance on seen anomalies. It is therefore unclear whether this metric is well aligned with the central objective of detecting unseen anomalies. Could this validation-based selection criterion itself introduce an inherent bias?
  3. The experimental results show a large oracle bonus on semi-synthetic datasets but a much smaller bonus on real-world fraud datasets. The manuscript does not disentangle the possible causes of this difference. For example, does it arise from greater training instability, differences in anomaly prevalence or signal-to-noise ratio, or a greater tendency to overfit the test set on semi-synthetic datasets? The current evidence is primarily correlational, and controlled experiments would be needed to substantiate any causal explanation.
  4. The effect of random data splits also warrants further investigation. The particular partition of anomaly classes into seen and unseen categories may itself introduce substantial variance. Are ten random seeds sufficient to characterize this variability? It would be helpful to separate the variance induced by the seen/unseen class split from that induced by epoch selection, so that one can determine whether the observed ranking reversals are attributable to random class partitioning or to the checkpoint-selection rule itself.
  5. Synthetic anomalies are intended to “expand coverage of unseen classes” and would therefore be expected to help on semi-synthetic graphs. However, Table 21 shows that removing this component improves performance on Photo by 0.018. The manuscript describes the results only as showing “mixed effects,” without explaining why this component makes a negative contribution on Photo. This counterintuitive finding requires further analysis or discussion.
2 comments
AuthorsOpenReview ↗

Response to Reviewer pZmS

We thank the reviewer. Questions 2 and 3 led us to the saturation analysis now in Section 5.2 (general response, point 4).

Q1. Two selection rules

"Is a comparison between only these two rules sufficient ... rather than a more general sensitivity to model-selection protocols?"

We added a third, label-free rule, the last epoch of the fixed budget (Table R1). It selects the oracle leader on all seven graphs, so the change of leader is tied to the rule that reads validation labels, which is the only label-based rule available in deployment:

GraphOracleValidationLast epoch
PhotoDEMO 0.888NSReg 0.793DEMO 0.870
ComputersOUTPOST 0.851NSReg 0.785OUTPOST 0.825

Across the three rules, the order of the three methods changes on three of seven graphs, and 43 to 62% of the apparent benefit of pseudo-labeling and the gate is collected only by the oracle rule (Table 15). The abstract now states the scope of the claim in the title, as contribution (i) did: the leader changes on two of seven graphs. On both, the two leading methods are within 0.02 of each other, which is the size of most differences in this comparison: 12 of the 14 oracle-rule differences between OUTPOST and the baselines are at or below 0.020.

Q2. Bias of validation-based selection

"Could this validation-based selection criterion itself introduce an inherent bias?"

Yes. On the semi-synthetic graphs, seen-class validation nears its ceiling early in training (it reaches 1.0 in 20 to 49% of runs), and the rule then often retains an early epoch that is poor for unseen classes. OUTPOST's unseen-class AUC-ROC at the three epochs:

GraphOracle epochLast epochValidation epoch
Photo0.8170.7370.640
Computers0.8240.7900.727
CS0.9840.9820.933

The bias lowers the reported score; it does not inflate it. Selecting on data that differ from the target is a known problem in domain generalization (Gulrajani and Lopez-Paz, 2021); its size and cause in open-set GAD had not been measured.

Q3. Causes of the provenance effect

"... does it arise from greater training instability, differences in anomaly prevalence or signal-to-noise ratio, or a greater tendency to overfit the test set on semi-synthetic datasets?"

Within a graph, a cause can be tested. Training instability or test-set overfitting would appear in the movement term, which for OUTPOST is similar across graph kinds. The selection term instead follows validation saturation: on the same graph it is larger in runs whose validation reached 1.0, in all nine method-graph pairs (Table R2). Across graphs the evidence is correlational, and Section 5.2 now describes it as an association. Graph size does not account for the effect (Amazon: 11,944 nodes, bonus 0.011), nor does homophily (T-Finance: same-class fraction 0.63, bonus 0.014) or prevalence (T-Finance has 4.6% fraud nodes, per Tang et al., 2023; each relabeled class holds at most 5% of nodes). Feature dimension, degree, and the absence of unseen classes remain confounded across graphs (Section 6).

Q4. Split variance

"Are ten random seeds sufficient to characterize this variability? ... separate the variance induced by the seen/unseen class split from that induced by epoch selection ..."

The seen class is enumerated, not sampled: each anomaly class serves once as the seen class (Section 3). The class partition is the largest source of variance (OUTPOST on Computers: 0.636 to 0.830 across classes, against a seed standard deviation of 0.017), but no single class produces the reversals. On Photo, OUTPOST minus DEMO changes sign between the rules on both rotations; on Computers, OUTPOST minus NSReg turns from positive to negative on three of five rotations and the other way on none (Section 5.1). The Photo sign change also holds in the runs where neither method's validation score reached 1.0, so ties at the ceiling do not produce it (Appendix F). A seed fixes the labeled nodes and the initialization together, so these two sources are not separated (Section 6). With ten seeds, the smallest attainable Wilcoxon p is 0.002, and paired differences near 0.01 can reach significance (OUTPOST minus NSReg on CS under the validation rule: +0.0095, p = 0.027; Table 10).

Q5. Synthetic anomalies on Photo

"Table 21 shows that removing this component improves performance on Photo by 0.018."

Synthesis does not help on Photo under any rule: removing it raises AUC-ROC by 0.0183, 0.0125, and 0.0129 under the oracle, validation, and last-epoch rules. On Computers and CS (three seeds), synthesis changes the oracle score by at most 0.0012 but adds 0.016 and 0.026 under the validation rule, so its effect operates through the epoch that validation selects. We have no verified explanation for Photo. Appendix G now reports these results in place of "mixed effects".

Thanks for the author's response, which addressed some of my concerns. I will keep my score as is.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their comments. The reviews found the problem important (pZmS, ppLt) and the evaluation design sensible (Jcg1). We have uploaded a revised PDF. Substantive changes are marked in blue and listed by location at the end of this note. New tables, figures, and equations are numbered R1 to R3, so all other numbers match the submission.

1. Description of prior work. Reviewer ppLt correctly notes that "copied" does not apply to NSReg or DEMO, both of which run their own baselines. The abstract and introduction now state the point we intended: each paper evaluates its baselines under its own splits, seeds, and budgets, so values cannot be paired across papers. The description of the first convention now follows the released code (DEMO reports the per-metric maximum over test epochs; NSReg defines no validation set). No result depends on either sentence.

2. Last-epoch rule. Every run stored the test metric at its last epoch, so we added a third, label-free rule without new training (Tables R1 and R3). It selects the oracle leader on every graph, and the order of the three methods differs between rules on three of seven graphs.

3. Decomposition of the oracle bonus. The last epoch splits the bonus into a movement term (oracle minus last epoch) and a selection term (last epoch minus validation epoch). AUC-ROC ranges over graphs:

Movement, semi-syntheticMovement, other graphsSelection, semi-syntheticSelection, other graphs
OUTPOST (five other graphs)0.002 to 0.0260.003 to 0.0270.043 to 0.054-0.006 to 0.001
DEMO, NSReg (four other graphs)0.017 to 0.0650.005 to 0.0240.015 to 0.096-0.010 to 0.014

The selection term separates the semi-synthetic graphs from the others for all three methods, with one exception: NSReg on CS (-0.024), whose test score declines late in training. The decomposition also led us to remove a sentence of the submitted Section 5.2 that attributed the bonus to late movement of the test curve.

4. Validation saturation. On Photo, Computers, and CS, the best validation AUC-ROC of a run averages 0.993 to 0.997 and equals 1.0 in 20 to 49% of runs; on the other five graphs it averages 0.73 to 0.97 and never reaches 1.0. Once validation is at its maximum it no longer distinguishes epochs, and the rule, as we apply it to all three methods, keeps the first epoch that reached it. Within each graph, runs that reach 1.0 have a larger selection term, for every method (OUTPOST: 0.089 to 0.103, against 0.014 to 0.033; Table R2). For OUTPOST, whose logs record every epoch, breaking ties toward the latest tied epoch raises the validation-rule AUC-ROC by 0.031 to 0.062 (118 of 150 runs); the stored baseline runs record only the selected epoch. The Photo sign change between OUTPOST and DEMO does not rest on ties at the ceiling: in the 13 of 20 runs where neither validation score reached 1.0, OUTPOST minus DEMO is -0.026 under the oracle rule and +0.025 under the validation rule. On Computers the reversal lies in the runs at the ceiling (Appendix F). Our reporting checklist (Appendix L) now recommends stating the best validation score and how the selection rule breaks ties.

Changes, by location. Prior work: abstract, Section 1, Section 3, Table 5. Third rule and decomposition: Section 3, Tables R1 and R3. Saturation: Section 5.2, Table R2, Appendix F (diagnostic details moved to Appendix I). Per-rotation check of the reversals: Section 5.1. Role of OUTPOST: contribution (iii), Section 4 (shortened, details in Appendix B), Appendix H. Limitations, synthesis results, and checklist: Section 6, Appendices G and L.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.