- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces DPCs and DePCs with the goal of modeling defeasible and deontic (probabilistic) logic within a probabilistic circuit framework.
Critical strengths
The introduction of defeasibility and deontic logic to the PC literature is a needed one. There has been little work on belief revision in circuits. This work makes a step in this direction.
Critical weaknesses
The theoretical results of the paper are hard to follow. I believe that most (if not all of it) is AI generated, reflected by the confusing writing at times. There is an excessive use of new variables. It is best to show a few important results in a clear human-readable way (and defer the rest to the appendix) than to instead cram all results in the paper and very briefly and hurriedly speed through them.
My main concern is with the description of how to construct the defeasible and deontic gates. The paper teases an almost-answer to this in Section 4.2, but just like in the defeasible case, it does not concretely show an algorithm to identify the gates to edit, from what I understand.
Questions for authors
- Line 282-285: it is unclear what the exact statement of intractability is being made here. What exactly (in terms of probabilistic or logic queries) do the authors mean by auditing? Could one not have an sd-DNNF representing the policy and (assuming a compatible PC representing the learned distribution and deontic/defeasible logic) the model count of the disjunction of the two representing a measure of compliance to the policy?
- There is a lack of description on how GPT and Qwen were prompted and how questions were queried. How were autoregressive models evaluated?
- There is no discussion on how one selects which sum nodes to turn into defeasible or deontic gates (and why that is compliant with the policy or how this successfully represents an exception). This is arguably the most important contribution, yet there is no discussion of it. Could the authors explain how this is done concretely?
Minor issues (no need to answer during rebuttal):
- Typesetting of DePCs. Sometimes as DePCs and sometimes as \textsc{DePC}s.
- It takes a while to get used to the naming. I would suggest DefPCs and DeoPCs instead. This way the reader immediately understands which framework the term refers to.
- Line 250: Notation is undefined.
- Theorem 8 states a certificate, however the statement never says what this certificate is. One has to infer that is the certificate. It would be better to state it clearly.
- Line 258: What is ? It is left undefined. Is it when is the root node?
- There is very little elaboration on the intuition behind the theoretical results. How is one supposed to interpret the (very many) different variables defined and the equations that are derived henceforth? The statement of results are very technical and without a proper explanation of how they link to the overarching story. This makes the math hard to follow. There are too many variables being defined. It is better to repeat one-self rather than define many variables and rely on the memory of the reader.
- Line 372: What is CSPN? Missing citation to Shao et al. 2019?
- Line 374: What is Vanilla PC? Missing citation or just the same structure as CSPN but where weights are not conditional?
- Are the datasets evaluated tabular? In which case it is expected for PCs trained on these datasets to perform better than LLMs, which were only prompted with no fine-tuning.
3 comments
W1: clarity of the theory.
We agree that the theory was harder to follow than it should be. As our AI-use statement says, we used AI tools to help draft the text and to help develop and check the proofs. We reviewed every proof ourselves, and we take responsibility for them. In the revision, the main text points to the proofs and to each appendix, the symbols you could not find (S^c and K) are defined where they first appear, and the explanation after each main result is rewritten in plainer language. In plain terms, the main results say the following.
- Proposition 5. Any trained gate can be rewritten as an ordered list of rules: use exception 1 if it applies, otherwise exception 2, and so on, and otherwise the default. The applicability a_i is the probability that exception i fires given that no higher-priority exception did. Nothing about the model changes.
- Theorem 6. A new exception inserted at position t takes its mass, delta, only from the rules below position t.
- Theorem 8. Under evidence e, write r_e for the probability mass of e that the edit leaves untouched, v_e for the mass it removes, and u_e for the mass it adds. Then no conditional prediction can move by more than beta_s(e), the larger of v_e/(r_e+v_e) and u_e/(r_e+u_e). In words, a prediction can move only as much as the edit changes the mass consistent with the evidence.
- Theorem 12. The divergence between behavior under the policy and behavior under the model is at most a sum over gates. Each term is the probability that the gate is reached under the policy, times how differently the policy and the model route at that gate.
Minor issues.
- Typesetting of DePC. Fixed. DPC and DePC are now typeset the same way everywhere.
- Naming (DefPC, DeoPC). We see the point. The abstract on OpenReview already uses DPC and DePC, so we kept these names during the discussion period to keep the record consistent.
- Line 250. S^c is the set of variables outside the scope S of the edited node, that is, X \ S. It is now defined before it is used.
- Theorem 8. The certificate is beta_s(e). The theorem now says so.
- Line 258. Your reading is correct. K is the part of the root distribution that the edit leaves unchanged, K = A_s K_s + B_s, where A_s and B_s come from the affine form at line 244. When s is the root, A_s = 1 and B_s = 0, and thus K = K_s. The definition is now in the text.
- Intuition and number of variables. Please see our answer to W1. Each main result is now followed by a plain explanation.
- Line 372. CSPN is the conditional sum-product network of Shao et al. (2020). The citation is now given at that line.
- Line 374. It is the latter. The Vanilla PC has the same structure with sum weights that do not depend on the update. It models the joint distribution over (X, U) and is conditioned on u at query time.
- Tabular data. Yes, all variables are categorical. We agree that a zero-shot LLM is not a like-for-like competitor for a model trained on the environment, and we present the LLMs as reference points.
Thank you for responding to my concerns. Having read through the author response, I will maintain my score.
My main concerns were
- The presentation and writing, which reads awkwardly and sometimes confusingly. It appears to be heavily AI generated. I believe the revised PDF addresses some of the clarity concerns, but the presentation and writing continue to be confusing and clearly AI generated. There is hardly any intuition behind the explanation, and the text overflows with unnecessary technical jargon that is common in AI generated text. The rebuttal is also seemingly heavily AI generated. I recommend a thorough rewrite of the paper with human words.
- Clarity on the algorithms. I asked for a clear algorithm on the construction, learning and belief revision. No algorithm is given in the revised PDF. The rebuttal answer to this question is confusing and I believe completely AI generated.
- How the autoregressive models were queried. The rebuttal does not address this concern.
I believe none of these concerns have been sufficiently addressed.
Thank you for the careful review, and for the point that belief revision in circuits has received little attention. We answer every point below, starting with your main concern. All changes in the revised PDF are marked in blue.
There are three separate questions here, and we answer each one.
Gate placement is fixed when the circuit is built. We build four balanced region trees over the 32 variables and place an update-dependent sum node at every internal region (31 per tree). Every one of these nodes is a designated node, so no selection step is needed. Appendix E.2 of the revision describes this.
The first child of every gate is the designated branch: the default in a DPC and the compliant branch in a DePC. During training, its logit starts with a positive bias, and a regularizer encourages the gate to route to it under the neutral input. Thus, the designated branch carries the behavior under the neutral input, and the other branches receive mass when an update moves the gate away from it. This is the sense in which they are exceptions. No label tells the model which observations are exceptions or violations. Note that the role itself is a specification and is not learned: Proposition 4 shows that which behavior is compliant cannot be recovered from behavior alone. In a DePC, the normative reading routes every gate to its compliant branch, and the descriptive reading uses the learned weights.
The results in the paper give a selection rule, which we now state explicitly. Suppose the new rule concerns a set of variables V.
The candidate gates are the designated gates whose scope contains V. In a region tree, these are the smallest region that contains V and its ancestors. By Corollary 7, an edit at a gate with scope S cannot change the distribution of the variables outside S. Thus, the candidate with the smallest scope leaves the largest set of variables provably untouched, and it is the default choice. Before any edit is applied, one upward and one backward pass yield the certificate of Theorem 8 for every candidate gate at once (Appendix B). The user can therefore compare the candidates on the contexts that must be protected, and choose the gate with the smallest certificate. Theorem 6 then inserts the rule at the chosen priority position. The new branch takes mass only from lower-priority rules. With binary applicabilities, the gate selects the highest-priority applicable exception, which is the behavior that prioritized default reasoning asks for. For an audit, the question is which gates to inspect, and Theorem 12 answers it. Each term of the audit budget belongs to one gate, and the gates with the largest terms are where behavior departs from the policy under the given evidence.
Thank you for this suggestion. By auditing we mean two queries: how much the expected severity under the learned model exceeds the expected severity under the policy (Lemma 9), and which gates account for the difference (Theorem 12).
When a policy is a logical constraint on X, compiled into an sd-DNNF that is compatible with the PC, the probability that the constraint holds is tractable, and it is a useful measure of compliance. Our setting differs in two ways. First, a DePC labels branches of sum nodes and not assignments to X. A compliant branch and a violating branch can produce the same outcome. Thus, compliance is an event on traces, and it cannot be written as a constraint over X unless the latent routing is made explicit. Second, the audit gap measures how much worse the outcomes are, and not only whether the policy holds.
The statement at lines 282-285 was too strong. We meant that exact distances and divergences between two circuits are tractable only for particular measures and under structural conditions such as determinism (Vergari et al., 2021), which we do not assume. We did not mean that policy-satisfaction queries are intractable. The revision rewrites the sentence and adds your construction as a remark.
Each model receives the observed context and the eight candidate descriptions, as in the examples of Appendix G. In the physical task, it is asked to rank the updates by their effect on the hypothesis. In the clinical task, it is asked to rank the actions by their deviation from the applicable protocol. The model returns a complete ordering of the eight candidates in JSON, and we score it with the same metrics as the circuits. No gold scores or rankings appear in the prompt. Appendix E.2 gives the decoding settings.