- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies how to select an answer from a fixed candidate pool using constraints extracted from the input. GLaCS converts these constraints into task specific states and factors, computes candidate scores through numerical repair, assignment search, or deterministic execution, and selects the candidate with the lowest energy. Experiments on arithmetic chains, logic grids, and BIG-Bench Hard report improvements over prompting and LLM selection baselines. Additional analyses examine candidate coverage, extraction quality, and the effects of constraint interventions.
Critical strengths
-
The paper addresses the important challenge of improving the reliability of language model reasoning.
-
The experiments evaluate the method across several reasoning tasks and report improvements over the baselines, demonstrating its effectiveness on the evaluated tasks.
Critical weaknesses
-
GLaCS relies on predefined adapters for the evaluated task families, including their representations, relations, and executable operations. However, the prompting and LLM selection baselines generate or select answers through model inference, without the predefined executable procedures used by GLaCS (Appendix A, “Prompting and decoding”; Appendix B.3, Table 22; Appendix B.5, Table 28). Therefore, GLaCS makes a stronger assumption about the availability of solution structures and operations. This limits its generality, as extending it to new task families requires designing adapters that define both problem representations and solving procedures, which can be difficult.
-
In Section “Grounded Constraint Construction”, the authors state that Normalize resolves entity references and task specific relations into a shared outer interface. However, the normalization rules and implementation are insufficiently specified, including how references are matched, relations are converted, and records are rejected. This makes the conversion from extracted records to executable constraints difficult to assess.
Questions for authors
-
How is Normalize implemented to resolve entity references and convert extracted relations into the shared interface?
-
How can GLaCS be extended to new task families, and what adapter design is required?
2 comments
Thanks to the authors for their response. However, supporting a new relation vocabulary requires an adapter that defines its executable semantics. The need to design such adapters for new task families limits the method’s generality. For this reason, I will keep my score.
Normalize implementation and extending GLaCS to new task families
Thank you for the review. We answer both questions below.
Q1. How Normalize is implemented. Normalize is deterministic. It maps each family's extracted records onto one shared interface of entities, typed relations and a query.
Entity references are matched in three stages. We first strip articles, lowercase the text and normalize punctuation and whitespace. A reference then resolves by exact match on the normalized string, by substring match, or by the highest string similarity above a threshold of 0.78, tried in that order. If two entities still tie, declaration order decides.
Relations are converted per family. Logical ordering compiles to fixed-position or relative-order predicates, and tracking to initial assignments followed by an ordered list of swaps. Temporal turns times into minutes and busy intervals, navigation becomes ordered moves and turns, Web of Lies becomes ordered truth links, and Boolean inputs become expression trees.
Records are rejected at two levels. A record that cannot be coerced is skipped on its own. A graph that lacks a required field or a valid query is a parser failure and goes to the same fixed fallback rule that the matched solver and execution controls use.
We logged every raw record, its normalized form, the matching path and any rejection reason for all 640 headline BBH graphs. Of 3,715 records, 3,630 are accepted and 85, or 2.3%, are rejected: 82 in logical ordering and 3 in navigation. No graph is lost.
The trace also shows that the headline numbers are conservative. The position parser rejects bare ordinals such as "third" or "last". We fixed a compatibility rule in advance that maps them to positions; it changes 44 of the 192 logical graphs. Accuracy rises from 149/192 to 163/192, a gain of 7.29 points with a paired 95% CI of [3.13, 11.98] and p = .00258. Z3 and enumeration still agree on every candidate energy. The headline tables keep the original protocol, so any gain from better normalization comes on top of the reported results.
Q2 and W1. Adapter design and new task families. Making the task semantics explicit is the design choice the paper evaluates. GLaCS authorizes a candidate by executing the declared relations, and the other selection methods instead generate an answer. Two controls make this a fair test. The graph-listwise selector receives exactly the same extracted graph and candidate pool as GLaCS, so it has the same access to structure. The shuffled-graph controls keep the executor fixed and break the alignment between graph and question, which tests whether the gain comes from the correct grounded state.
The language-to-solver work the paper builds on makes the same assumption: SatLM and Logic-LM also rely on a predefined solver language with fixed semantics.
A new family needs a state domain with its extraction schema, relation normalization, executable factors, a map from query to answer, a candidate compatibility check and a failure and tie policy, plus tests for valid, invalid and contradictory inputs. Candidate scoring, selection, the interventions and the coverage and extraction audits are shared and reused without changes. For relations we already support, the generic constraint DSL reproduces the task-specific translators' decisions on 448 examples across Temporal, Boolean, Tracking and logic grids, as reported in Table 31.
An adapter does not have to implement its own search either. The GLaCS energy is a standard minimum-violation objective, so an off-the-shelf solver can compute it. Independent Z3 encodings on identical graphs, candidates, weights and tie rules reproduce every GLaCS candidate energy and decision on 192 of 192 BBH logical examples and on 512 of 512 generated-candidate logic grids. On structured logic grids, Z3 solves all 96 instances at 4, 5, 6 and 8 entities. The 8-entity grids have 65,548,320,768,000 assignments and take a median of 0.3267 s per example. Most of the per-family effort therefore goes into declaring semantics rather than writing search code.
We will release the code together with the adapters, the extraction and selection prompts, raw predictions, normalization traces and frozen configurations, with source snapshots and input hashes.