acceptodds
Withdrawn from ICLR 2027

B4I: Blackbox Platform Interpretation for Item-Level Assessment with Behavior Feedback

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Many creator-facing and analytics-driven online platforms operate as black-box systems for external stakeholders: ranking and exposure mechanisms, internal models, and the ways in which these components interact with user behavior are inaccessible, while only final feedback such as clicks or conversions is observable. Under this constraint, stakeholders must infer item-level outcomes without access to platform internals or direct model outputs. Existing approaches often rely on opaque neural predictors or aggregate observed statistics, offering limited insight into how user-conditioned interaction patterns shape item-level outcomes. To address this problem, we introduce feature-based neural–logical components into an interpretable behavioral surrogate that models the observable mapping from user and item features to platform-mediated feedback. Building on this idea, we propose B4I, a two-stage framework. Stage I learns the surrogate from historical user–item interactions, producing user-conditioned interaction scores and neural–logical explanatory signals. Stage II evaluates the surrogate over a fixed historical user-group panel and aggregates the resulting outputs to predict item-level popularity labels. B4I thereby transfers behavioral knowledge learned from historical user–item interactions to item-level assessment without requiring observed user–item interaction feedback for the target item, enabling interpretable item-level assessment under black-box platform constraints. The code is anonymously available at https://anonymous.4open.science/r/B4I-2B2C/.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Item-Level Assessment, Neural–Logical Explanation, Black-Box Platform Interpretation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4wbNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes B4I, a two-stage framework for predicting item popularity without platform internals or feedback on the target item.

  • Stage I learns a neural-logical surrogate from historical user-group–item interactions.
  • Stage II applies it to a fixed user-group panel and aggregates the resulting scores and rule representations to classify each item into one of four ordered popularity levels.

Experiments compare B4I with direct prediction baselines, along with ablations and a qualitative analysis of the learned rules.

Critical strengths

S1. Creators often need to estimate how content will perform and how audiences may differ before any feedback is available. The paper addresses this need directly.

S2. Using historical user–item data to inform item-level prediction is well motivated compared with training an item classifier alone. The fixed user panel also gives a clear and consistent basis for comparison.

Critical weaknesses

W1. Appendix D.2 cites FENCR, which already learns neural-logical rules over user/item feature embeddings, uses data-driven rule selection, and produces rule-level explanations with learned aggregation. So the capability itself is not new.

Is Stage I built on FENCR? If so, what was changed (e.g., ID pathway, rule operators, training objective, contribution computation)? If not, please give a full specification of RθR_\theta, SθS_\theta, and RuleEnc rather than the current abstract description.

W2. RθR_\theta, SθS_\theta, RuleEnc, and the Stage II modules (ϕ\phi, gψg_\psi, AA, qωq_\omega) appear only as abstract mappings. It is unclear how rules select features and form conditions, how logical operators are implemented, how rule number and length are set, what RuleEnc actually encodes, and which pooling is used in the experiments. Dimensions and training configuration are also missing.

Since the neural-logical surrogate is a claimed contribution, this goes beyond missing hyperparameters: the method cannot be reproduced, and its difference from FENCR cannot be assessed.

W3. Reported MAE values are inconsistent with the stated definition (§3.4, Table 2). §3.4 defines MAE as the error between predicted and true class indices. If predictions are obtained by argmax, then with 72 test items and 3 seeds, the mean MAE must be an integer multiple of 1/2161/216. The values in Table 2 for LR (0.7484) and B4I (0.5814) do not satisfy this constraint, and four-decimal rounding cannot account for the gap (e.g., 161/216≈0.7454161/216 \approx 0.7454, 162/216=0.7500162/216 = 0.7500; 125/216≈0.5787125/216 \approx 0.5787, 126/216≈0.5833126/216 \approx 0.5833).

Please clarify whether MAE is computed from the expected class, uses sample weights, or is aggregated differently, and state whether MSE and MAE use the same prediction transformation. Releasing the evaluation code would help resolve this.

W4. The appendix claims that requiring ≥10 impressions with zero clicks avoids mislabeling under-exposed items. But most iPinYou campaigns have CTR below 0.1%, and even at a true CTR of 0.1%, P(0 clicks in 10)=(1−0.001)10≈99%P(\text{0 clicks in 10}) = (1-0.001)^{10} \approx 99\%. I note that the selected items are those with the most impressions, so the actual exposure may be much higher. Still, please report the impression distribution of the 177 items to show that the labels are reliable.

W5. In the main table, B4I uses interaction-level supervision while direct classifiers use item-level labels, so the gains may reflect the extra supervision rather than the neural-logical + aggregation design.

Missing comparisons include

  • vanilla FENCR + the same Stage II
  • simple score statistics (mean/quantile/histogram) + LR/MLP
  • an interaction-pretrained model with the same aggregator
  • DeepSets/MIL on raw features
  • rule-only vs. scores-only ablations.

W6. Each item requires 155 or 2,048 surrogate calls, but

  • latency
  • memory
  • parameter count
  • the panel-size trade-off are not reported.

The data pipeline is also incomplete:

  • raw sample sizes
  • filtering criteria
  • class distribution
  • vocabulary fitting scope
  • semantic attribute extraction
  • time windows

should be described.

W7. The paper states that cui,ℓ>0c_{ui,\ell} > 0 supports a higher interaction score and cui,ℓ<0c_{ui,\ell} < 0 opposes it, but also says that ScoreθScore_\theta is a learned combination rather than an unweighted sum of contributions. If ScoreθScore_\theta is a general nonlinear function, a positive cui,ℓc_{ui,\ell} does not necessarily raise the final output. The manuscript neither gives the formula for cui,ℓc_{ui,\ell} nor validates its sign.

An exact logit decomposition, a monotonicity constraint, an interventional definition, or rule-deletion experiments would be needed to support the claim that signed contributions faithfully explain predictions.

Questions for authors

See Weakness

Reviewer_12yKOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes B4I, a two-stage framework for item-level assessment under "blackbox" platform constraints where internal mechanisms are inaccessible. Stage I learns a neural-logical behavioral surrogate from historical user-item interactions, predicting interaction scores and outputting rule encodings; Stage II evaluates each target item over a fixed historical user-group panel and aggregates the predicted user-conditioned signals into an item-level popularity prediction. Evaluation is conducted on two advertising datasets (CommAds and iPinYou), including ablation studies and case studies.

I am open to raising my score if the authors can address the following concerns in rebuttal: (a) provide concrete examples and systematic evaluation of the neural–logical rules' interpretability, including comparisons with established interpretable ML methods, and (b) strengthen the experimental evaluation with statistical significance testing on iPinYou and ideally results on a more modern, larger-scale dataset. The blackbox platform assessment problem is practically relevant, and with stronger empirical validation the paper could make a more compelling case.

Critical strengths

Well-motivated problem setting with practical relevance: The "blackbox platform" scenario—where external stakeholders can only observe user-item interactions without access to platform internals—is a genuine practical challenge for creators, advertisers, and analysts. The two-stage framework provides a clean separation: Stage I models platform-mediated behavior, Stage II transfers this behavioral knowledge to item-level assessment. The inclusion of both a proprietary large-scale dataset (CommAds, ~2M interactions) and a public benchmark (iPinYou) is a reasonable starting point for empirical evaluation.

Critical weaknesses
  1. Limited methodological novelty and transparency: B4I's two-stage design—learning an interaction model followed by aggregation into item representations—follows the standard stacking/meta-learning paradigm. The "neural–logical rules" concept, while branded as a key contribution, needs to be distinguished from an auxiliary feature encoding of the kind used in wide&deep or DeepFM. The generation mechanism, encoding scheme, and actual semantics of these rules are not adequately explained in the main text—most details are deferred to Appendix A—making it difficult to assess the substance of the interpretability claim. No concrete rule examples are provided in the form one would need to judge them, and the paper does not say what syntax a rule takes, how rules are induced during training, or how one would verify that an induced rule holds.

  2. Experimental design does not fully support the claims: The dataset selection is problematic: iPinYou dates from 2013 (714 items, ~20K interactions) and, as an open benchmark with a known bidding mechanism, does not align well with the "blackbox platform" premise. On iPinYou, B4I does win clearly on MSE (0.6806 vs. 0.7500 for DeepFM), MAE (0.5814 vs. 0.6411 for FinalMLP) and AUC macro (0.8118 vs. 0.7689), so the method is not at parity overall. The F1 comparison is where the support runs out: 0.5105 ± 0.0021 against DeepFM's 0.5081 ± 0.0073, a mean gap of 0.0024 that is about a third of DeepFM's own standard deviation, with no significance testing reported for that dataset. On 714 items and ~20K interactions there is no plausible route to significance, so the honest description is a tie on F1. The CommAds improvement (0.3165 to 0.4041, surviving Holm correction) is real but not stratified by item popularity, so it cannot be excluded that the gain concentrates in head items—which matters for the stated use in filtering and targeting. The "interpretability" evaluation is limited to a qualitative case study of 4 correctly predicted items—no user study, no domain expert validation, and no comparison with established interpretable ML methods.

Questions for authors
  1. Could you present 3–5 concrete neural–logical rules with their meanings? This would help assess the actual content of the "interpretability" claim.

  2. Is the F1 difference between B4I and DeepFM on iPinYou (0.0024) statistically significant given the reported variances? And on CommAds, how is the gain distributed when items are stratified by popularity?

  3. Have you considered applying interpretable ML methods to direct item-label baselines, and then systematically comparing their explanation quality against B4I's rule-based explanations?

Reviewer_3G34OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes B4I, an interpretable item-level assessment framework for external facing, blackbox online platforms using neural-logical models. The framework involves two stages. In the first stage, it uses user group, item and outcome pairs, and learns user-item activation rules that are used to compute interaction scores. In the second stage, given an item, it runs the frozen learned rules for each user group in a selected user group panel and then aggregate across per user scores and rules to make a prediction. Experiments were performed on CommAds and iPinYou, and showed improvements upon item-level baselines across different architectures.

Critical strengths
  • The proposed method B4I achieves consistent performance gain upon existing item-level methods, including more complex, deep learning methods, showcasing the effectiveness of B4I.
  • Ablation studies on the stages validate the effectiveness of neural-logical models as the choice and including rules is helpful. Case studies showcase scenarios that the proposed framework can provide more fine-grained, interpretable insights.
Critical weaknesses
  • To some extent the interpretability benefit is an inherited property of the neural-logical model itself. Additionally, there is limited zoomed-out discussion/analysis on the general usefulness and human intuitiveness of the learned rules or potential failure cases, making the actual interpretability of B4I a bit ambiguous.
  • Documentation of technical details is limited in some places, e.g. regarding the construction of user group panels, and training and hyperparameter details of the proposed B4I and baselines. Even though anonymized code was provided, it could still be useful to document these details in the paper for better clarity and understanding.
Questions for authors

See Weaknesses above -

  • Regarding interpretability, it would be helpful to have more insights on how many learned rules are generated, how they look like in general and how interpretable they are from a human perspective, as well as having some discussions on potential failure cases.
  • It would be helpful to have more detailed documentation on implementation details of B4I and the baselines, e.g. model size / number of rules, hyperparameter search protocol and final selected combinations, etc.
  • It would be helpful to have more documentation on the how the user panel is constructed, and ablation on how sensitive B4I is to the size or composition of this panel.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.