- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper analyses the stability of two operators: aggregation and optimization under perturbations. It finds that aggregation is provably Lipschitz continous under belief perturbations, but optimization is not. Which is not suprising since argmax-optimization under belief perturbations is not a continous functions with the change being essentially only bounded by the hypothesis space.
The paper, being theory-heavy, is very hard to understand and confusingly written. Concepts are not properly defined, arguments to not support each other properly and contents of sections do not support the title. In its current form, it is hard to parse and not possible to review.
Critical strengths
None in it's current form
Critical weaknesses
The paper is extremely hard to parse. keywords are constantly introduced without being explained, sentences seemingly refer to each other without any obvious connection. For example, even the abstract is hard to understand: first off, the connection to neuro-symbolic systems is not clear: one can study the stability of a weighted sum versus computing the argmax without being embedded into a neuro-symbolic system. It is not clear how it connects to the title. Then, after establishing that the optimization is unstable, the abstract pivots to "We further show optimization-side intractability is not inevitable: (...)", but there is no tractability discussion to refer to!
Furthermore, section 3 is very crucial for the paper. It introduces the whole setting but is just reduced to a list of arbitrary keywords without explanation, definition or motivation. E.g. why is definition 1 now the object of interest for the optimization, whereas in the introduction we were concerned with the argmax? Every symbol in section 3 must be properly defined, explained and motivated, right now it is not possible to discern what is going on. Examples of unclear symbols/definitions/sections in section 3 are:
- Remark 1: is in unclear how it fits into the into the preliminaries, as message passing is neither mentioned not defined the computational object. Even more so, it is completely unclear how it would fit into the general architecture
- "a neural scorer fθ (assumed L-Lipschitz) produces soft confidences p∈[0,1]m over m candidate predicates;" what are these candidates predicates? What is the setting of the neural scorer?
- "a discretization step τ produces a predicate set P" What is the discretization step? A Thresholding?
- "a decision operator Ψ maps P to a solution ∆" So apparently this decision operator is both for aggregation and optimization, but for some reason the thresholding is not analyzed in the aggregation context?
- "Definition 1 (Optimization-mode inference)" unclear how it connects to the previous argmax-based explanation of the optimization
- "Given interpretations Ω, a satisfaction predicate µ :L×Ω →{0,1}over a predicate family L, and a target family K⊆L(derived from P above via τ), optimization-mode inference computes" What are the interpretations Omega? of what? What is L? "target family K⊆L(derived from P above via τ)" how is K derived from P via τ? What is the connection?
- " ω inclusion-minimal with this property " what does it mean to be inclusion-minimal with this property (probably refers to satisfaction predicate µ?)
- Motivation, intuition and interpretation of definition 1 is completely omitted
- "dP(P1,P2) is the symmetric-difference cardinality between two predicate families" unclear what the symmetric-difference cardinality is
- Definition 2: "|P1⋄P2|" unclear if this is now another symbol for dP(P1,P2)
- Decision sensitivity is entirely unmotivated, e.g. what does it measure? Why is this a useful concept to analyze?
This continues to section 2, the 6 hypothesis, which are central to the paper:
- "as the relevant uncertainty problem size/number of required waypoints (K) grows." waypoint were never defined, unclear what they should refer to
- "with the size of the discontinuity governed by margin, not by perturbation size" not clear what margin is referring to
- H4 (Recoverable tractability): unclear which intractability this refers to, as previously we were discussing stability. Also, these approaches restrict the function class (of message-passing), are well known, and message passing is never so far introduced. it is unclear how it embeds into the architecture
- "H5 (Margin CDF Predicts Optimization Instability). The empirical instability frequency is upper-bounded by the empirical margin CDF." it is unclear, and never mentioned what a margin cdf is, "empirical instability frequency" is also unclear, despite these being the main hypothesis of the paper
- "H6 (Impact of Temperature Scaling on Calibration and Instability)" calibration is also unclear how it fits into the picture, e.g. there is not uncertainty covered. also, we never defined the temperature inside the pipeline. Therefore, the main hypothesis section is hard to read and not understandable
This theme continues in section 5 (Theory), leading to sections like 5.1.2 ("WHAT ACTUALLY MAKES OPTIMIZATION UNSTABLE") which is not answered in this section, a seemingly unrelated definition is given instead. Or 5.2 ("INSTANTIATIONS"), with example like this:
"Instantiation 1: model-based diagnosis. Ω = 2^U for component universe U, µ = “hits every conflict,” K= the reported conflict family. Reiter diagnosis, structural diagnosis, MUS-based repair, and geometric MBD are recovered as instances by the same argument."
It is unclear what a component universe U is, µ = “hits every conflict,” is not a proper mathematical definition and unclear from the context what it should be. "K= the reported conflict family" is not a proper instantiation, it is unspecific prose. What Reiter diagnosis, structural diagnosis, MUS-based repair, and geometric MBD refer to is not defined. Also, which argument is referenced here?
Questions for authors
- What is the connection to neuro-symbolic learning?
- What are all the definitions/unclear prose mentioned above in "Critical Weaknesses"
3 comments
I want to thank the reviewers with engaging with my points. While explaining some points to me helps me understand the paper, other points were not addressed. I also do not see a revision, which I think is possible in the ICLR-reviewing process? The critique was less me personally understanding the paper but the poor quality of the submission. The points raised were also examples that illustrate the poor readability and not a "checklist" to explain in the rebuttal, as I was not satisfied with the quality of the writing and struggled to understand the paper (despite being familiar with neuro-symbolic approaches). I think the paper would immensely benefit from a careful reading and a major revision, while looking out for the underlying issues of the paper:
- is every symbol, concept or name sufficiently defined and explained? This was a major issue during reading, a paper should be self-contained except background that can be assumed from the typical reader
- is the underlying problem properly explained and motivated? What is the setting and why do we care about it? Here, an example would help to understand what the paper is talking about, instead of just starting with the formalization (which was unreadable, see the above point)
- What is the role of each section, and does the content support its title?
I will not increase my score based on this response and i encourage the authors to contemplate improving the readability of the submission with a careful rewrite.
rewrite now uploaded
FYI, we have an updated draft uploaded, so we hope that this improves readability.
response
Response to Reviewer yHLm
Thank you for the detailed list. We accept that Section 3 as submitted was not reviewable, and we have rewritten it.
Connection to neuro-symbolic learning. The objects studied are pipelines of Kautz's Neuro;Symbolic type: a neural network outputs beliefs over a fixed vocabulary of symbolic facts, and a symbolic engine reasons over the facts the network asserts. Examples include diagnosis engines fed by learned fault detectors, explanation engines over learned classifiers, and planners over learned hazard maps. The question is specific to this setting. When the neural component is trained and certified at the level of facts, does that guarantee transfer to the symbolic engine's answer? Decision sensitivity (Section 3) is exactly the exchange rate between fact-level and answer-level error. Theorem 1 shows it is bounded for aggregation and unbounded for optimization.
Why the result is not "argmax is discontinuous." See General response, item 1. Both query families now read the same thresholded input and are both discontinuous in x. They differ in whether one flipped fact has a bounded or unbounded effect.
Itemized definitions (all now in Section 3):