acceptodds
Under review as a conference paper at ICLR 2027

Counted Conditionals: Modeling Discrete Distributions From Pairwise Counts

Abstract

Generative modeling requires estimating high-dimensional distributions from finite data. Counting complete configurations is impractical, but relationships between informative parts recur across examples. We ask how far discrete distributions can be modeled from empirical token-pair counts. We introduce Counted Conditionals, which separates the statistics we store from the way we use them. Rather than pooling observations as separate neighbor votes, we fit tractable joint models on narrow spatial strips from first- and second-order counts. Conditioning sums over unknown tokens within each strip, and a calibrated pool combines the component predictions. Higher-order context counts provide an optional extension. On visual- token and land-cover grids, the strip construction improves completion accuracy and log loss over direct pairwise pooling across three mask patterns. Small-data and increasing-evidence experiments examine how it uses limited training examples and partial observations. We also evaluate the normalized joint induced by applying the pooled predictor in a fixed order, separately from its marginal forecasts. Because records contribute additive statistics, updating the counts and repeating the same deterministic construction reproduces a fresh build under fixed preprocessing and reference information. These results establish a count-based approach to discrete probability modeling and clarify its predictive and computational trade-offs.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
pairwise statistics, discrete probabilistic modeling, tractable inference, graphical models, conditional generation, sufficient statistics

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ZSJ5OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper considers the problem in the following setting:

  1. Each image is encoded by a frozen DINOv2 model, and its patch features are clustered into 16 categories, giving a 16×16 grid of integers per image. The paper also uses NLCD land-cover maps, 32×32 grids of 16 native land-cover labels.
  2. Thus, each integer carries some "pattern" / "semantics" or the patch.
  3. The paper studies the question of whether one can predict the probability of the value of some masked/hidden grid cell value, given all other available cell values.
  4. The paper proposed a "strip-based" method and it shows this improves over a pairwise pool that combines separate neighbor votes. It also claims that, because the statistics are additive, deleted training records can be removed with a result that exactly matches a fresh rebuild.
Critical strengths

The method improves marginally over the non Neural Network baseline methods.

Critical weaknesses
  1. The proposed, "strip-based" method lacks proper motivation and justification. E.g., it is unclear to me why a far-away cell in the nearby strip is considered, yet some other cells with closer distance are ignored.

  2. The contribution seems thin: the core idea "COUNT-FITTED STRIP MODELS" seems to be a simple trick.

  3. On NLCD, the naive baseline, i.e., 5×5 majority vote, is within about 1.5 points of the best neural model and beats the pairwise baseline. That the proposed method “being close to neural models” in this setting is weak evidence.

Questions for authors

The main evaluations use 16 categories on 16×16 (images) or 32×32 (maps) grids. Would the trade-offs still hold with richer vocabularies or larger grids?

Reviewer_z3mSOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces Counted Conditionals, a method for predicting missing categorical tokens on image and land-cover grids from explicit counting statistics. Rather than counting entire configurations, which is infeasible, it exploits recurring relationships between nearby tokens. The method collects and smooths pairwise co-occurrence counts, then fits tractable probabilistic models on narrow horizontal and vertical strips using L-BFGS. Exact conditioning within each strip sums over missing intermediate tokens, and calibrated pooling combines the predictions. An optional extension adds higher-order context counts with backoff smoothing. Additive statistics also support data updates and deterministic reconstruction. Experiments seem to show improvements over direct pairwise pooling, while neural models generally achieve better predictive performance. The method offers a compact, revisable alternative for modeling token distributions, although its image experiments rely on pretrained DINOv2 representations.

Critical strengths
  • Consistently outperforms pairwise and local methods.
  • Evaluated across several datasets and masking patterns.
  • Supports straightforward updates when training data change.
Critical weaknesses
  • Unclear presentation. The writing makes it difficult to identify the central problem and understand how counting, strip fitting, pooling, and optional contexts fit together. A clearer explanation of the complete pipeline would help establish the contribution.

  • Missing explanation of the notations. The notations seem not to be clearly introduced, making it hard to understand what is the logical thought of the paper. For example, the statement ``Tables are shared across positions with the same offset'' introduces spatial sharing without clearly explaining what an offset represents or how the tables are constructed. The subsequent description of R1,R2,R3R_1,R_2,R_3 would benefit from a diagram or worked example showing the neighborhoods and their 8, 24, and 48 offsets.

  • Insufficiently clear positioning. The related-work section names relevant approaches but does not sufficiently explain their underlying ideas or identify what is new beyond combining established probabilistic modeling and inference techniques.

  • Uncertain computational advantage. The method involves iterative L-BFGS fitting and calibration, rather than counting alone. The paper should more clearly explain when its accuracy, fitting cost, inference cost, and memory requirements offer an advantage over small neural networks.

Questions for authors
  • Computational benefit: Including counting, L-BFGS fitting, calibration, and inference, in which settings does the method offer a favorable accuracy–time–memory trade-off compared with CNN-S or MaskGIT-tiny?
  • The role of context are insufficiently explained. It is unclear how higher-order context is incorporated, what information it adds beyond the strip models, and why the proposed combination should improve predictions. On ImageNet, the gains are modest and vary across masking patterns despite a substantial increase in memory. The paper should explain this limited benefit and clarify when the context extension is useful.
  • Data revision: Which stages must be repeated after deleting data, and what retained records or auxiliary statistics are required? How do these requirements affect the claimed advantage over retraining a small neural model?
Reviewer_RaFPOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces a formulation of a type of MRF for count data in spatial grids. The proposal concern the use of standard smoothed count statistics, pairwise MRF for the strips, product of experts aggregation, count statistics updating and pooling. This model is fitted to unary and pairwise count statistics from vertical and horizontal strips of a 2D spatial grid. Higher order interactions are included via a count-based variable-length contexts (similar to techniques from language modeling). The authors present empirical studies of fitting with limited data, increasing observed evidence, ordered joint likelihood, and reconstruction after edits. All considered discrete distributions here are count data on a 2d spatial grid, and empirical datasets considered are images.

Critical strengths
  1. The idea of separating count stored statistics from inference is interesting
  2. The evaluation is broad enough in terms of baselines and support the narrow claim of count fitted strips vs pairwise pooling.
Critical weaknesses
  1. The conceptual novelty is small. The main method combines established components. Theory is direct application of standard identities and characterise only pooling, not the proposed strip predictor (either directly or via approximation bounds). Theory doesn't give insight why strips could work better.
  2. The empirical gain is limited. The evidence support only the narrow claim of improvement of strips over pairwise pooling.
  3. The empirical evidence does not isolate the gains and is fragmented. It is hard to follow the empirical narrative, there are different things being compared but it is not evident despite a careful read, what are the clear gains under which circumstances and such.
  4. The language is somewhat "bloated" and overclaims. For example, the title of the paper suggest a novel generic method for discrete distribution modeling, but instead we get a specific combination of existing "modules" designed for spatial grid count data.
Questions for authors
  • What are inference time, training time, and memory as functions of V , grid size, and strip width?
  • How do strips compare with Chow–Liu trees, mixtures of trees, and same-potential loopy inference under identical data and calibration budgets?
  • How do the methods compare using macro-F1, per-class recall, Brier score, and calibration error on the main test sets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.