- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies online reputation estimation and platform decisions when feedback can be manipulated and validation arrives late or is corrected. FARR-SC separates bounded use of current reports from persistent source credit earned through subsequent validation. It combines source and cluster caps, frequency-aware aging, four specialists, delayed selection, calibration, and correction replay. The authors provide a capped-contamination analysis and component-level learning bounds. Experiments include controlled trace replays, a recruited-human pilot, and public trust networks. The reported benefits are concentrated in audit-blindspot settings, while clean settings and several external comparisons favor simpler alternatives or the variant without source learning.
Critical strengths
-
The problem is practically relevant: a platform needs to use informative reports before their reliability is fully known. Separating immediate influence from validation-earned credit gives a coherent way to address this tension.
-
The capped strategic-mass analysis directly addresses the robustness question. The upper bound and matching lower-bound construction clarify the dependence on the strategic weight that remains after capping. This provides a useful theoretical interpretation of the caps under the stated assumptions.
-
The paper includes meaningful component ablations rather than only comparisons of complete systems.
Critical weaknesses
-
The main new contribution is not clear enough. TRAVOS already uses later outcomes to assess sources, and several other components use standard methods. A clearer comparison would help show what this paper adds.
-
The ablations show that some components help, but the need for the full design is still unclear. Removing source learning gives lower reported loss in the human pilot and all four Bitcoin comparisons (Tables 6 and 9), though these differences may not be statistically significant.
-
Theorem 6 needs a clearer explanation. The proof has no explicit delay term, and it is unclear whether the bound applies to the original predictions or only to replay after labels arrive.
-
The real-world evidence is still limited. Bitcoin failures are defined by later user ratings, not independently checked outcomes. The human pilot has only 204 evaluated events and no delayed corrections, so it provides limited evidence for the full method in practice.
-
The text in Figure 1 is too small. Larger labels and a simpler layout would make the method easier to follow.
Questions for authors
-
Does Theorem 6 cover predictions made before labels arrive, or only the replay results? How is the delay handled?
-
Can the authors compare with a simple weighted average using the same caps, calibration, and decision rule? This would help show whether four specialists are needed.
-
Why does removing source learning reduce the reported loss on the external datasets? Under what conditions should this component help?