- Rating
- 1
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper considers the setting of a sequential process where marginal contributions of contributors to a jointly achieved predictive ensemble score (based on delayed labels and a proper scoring rule) are used 1) to attribute the achieved score to the contributors (settlement) and 2) to adjust contributor weights for ensemble predictions.
Under the assumption that the label probabilities additively decompose into the contributor predictions (so called types), this work analyzes how distortions in a contributor’s prediction affect future payments and weights via marginal contributions.
In particular, it is shown that the future weights of an agent, and thus its utility (defined as the sum of discounted payments and weights), can under certain conditions be enhanced by distorting its predictions (referred to as farming), such that truthful reporting is not optimal over time.
Within a so called cushion, such farming agents cannot even be detected as harmful (which is defined by negative expected contributions), while other agents might be flagged as harmful due to the farming agent’s distortions (referred to as framing), which may lead to reallocation of their weights to the farming agent.
Critical strengths
- Under the assumption of additive label probability decomposition, this paper presents a detailed analysis of when distortions are profitable and whether farming is detectable.
Critical weaknesses
- Practical relevance: The majority of contributions is derived by assuming that the labels follow a categorical distribution with the averages of the contributor types as probability parameters. This model lacks motivation and justification, and it is totally unclear why it would be practically relevant in the presented context of prediction ensembles. Even in their own experiments, this assumption does not hold under (real) data. Yet, the contributions heavily rely on this assumption and the paper does not analyze which of the contributions still hold without this assumption.
- Empirical support: While experiments on real data (where the labels were not changed) are presented, in the majority of experiments the real targets of the dataset are replaced by synthetic ones according to the assumed model (average of the contributor types as probability parameter). While this is a substantial intervention into the experimental setup, in my perception it is mentioned only briefly in the main part of the paper (only mentions "semi-synthetic replay" in the abstract). This is especially relevant, as the theoretical claims of the paper are only empirically supported in the case of synthetic labels. Nevertheless, some parts of the main paper convey the impression that the theory would have general empirical support.
- Presentation: The paper is written in a way that is too dense and overall it is very difficult to follow. Despite its density, it does not even properly motivate the overall setup and does not justify key assumptions.
Questions for authors
- Can you better motivate or justify the assumption that the labels follow a categorical distribution with the averages of the contributor types as probability parameters?
- Can you clarify whether any / which of the contributions still hold without this assumption but under weaker target models (without additive decomposition of the probability parameter)?
- Can you confirm that none of the theoretical claims of Theorems 1-3 were empirically supported on real data (where targets have not been modified)?