acceptodds
Under review as a conference paper at ICLR 2027

TAIWO: Structured Test-Time Inference for Open-Ended Semantic Discovery Across Large Language Models

Abstract

Open-ended semantic discovery with large language models (LLMs) is increasingly performed through multi-step prompting, yet comparatively little is known about how reliably a fixed structured workflow transfers across model families or where it fails as concepts become less prevalent. We introduce TAIWO, a six-stage test-time inference pipeline that transforms source documents through familiarization, coding, theme construction, review, definition, and synthesis while retaining source identifiers. We operationalize TAIWO as a frozen benchmark on a deterministic 100-document corpus with nine latent concepts in three prevalence strata. Across 39 model runs, the task-consistent mean score is 0.670 (median 0.737; range 0–0.883); 36 runs complete all structured stages, while three return task-declared zeroes after malformed output. Within the 36 detailed outputs, dominant-concept recall averages 0.907 but rare-concept recall averages 0.481. A provenance audit recovers the exact task source and validates its hash, but also finds two consequential measurement limitations: an exporter treated the three zeroes as missing, and the score's “grounding” term verifies identifier existence rather than evidential support. Exact concept labels paired with deliberately wrong but valid source identifiers can therefore score 1.0. These results provide a transparent cross-model stress test of structured open-ended inference, identify rare concept recovery and schema compliance as concrete failure modes, and show why benchmark conclusions must include scorer-level adversarial audits. In an exploratory three-model compute-matched pilot, TAIWO's mean paired advantage over generic six-call decomposition is 0.0068 (95% model-clustered bootstrap CI ), providing no evidence of a pipeline advantage. A separate transfer pilot on 100 human-labelled Reddit mental-health posts likewise provides no evidence of an advantage: source-grounded category recall is 0.289 for TAIWO and 0.413 for generic decomposition (paired difference , 95% CI ).

Primary area:
datasets and benchmarks
Keywords:
structured test-time inference, large language models, open-ended semantic discovery, multi-stage inference, test-time computation, LLM benchmarking, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_y1azOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TAIWO, a six-stage LLM workflow for thematic discovery with source-linked evidence, and audits its semantic, interface, and measurement reliability. The study reports a historical 39-model evaluation on a controlled synthetic corpus and two exploratory paired comparisons against generic six-call decomposition, including transfer to human-labelled Reddit data. The audit identifies omitted failure scores and a source-ID validity term that does not establish evidential support, and introduces a support-aware revised metric. The paired experiments do not establish a consistent quality advantage for TAIWO.

Critical strengths

The paper identifies concrete evaluation failure modes. Restoring omitted model-output failures changes the reported mean, while executable scorer attacks demonstrate that valid source identifiers need not support the associated themes. These findings provide useful examples of how reporting and metric implementation can overstate reliability. The supplied code and synthetic paired-pilot records make important parts of the audit inspectable.

Critical weaknesses
  1. Insufficient comparative evidence for the workflow’s added value. Although the paper discusses related LLM-assisted thematic-analysis workflows such as GATOS [1], the completed experiments compare TAIWO only with generic six-call decomposition. No direct single-call baseline is reported for generating themes and supporting evidence from the same documents. The small paired pilots also do not establish a consistent quality advantage. These results leave unclear what TAIWO adds over simpler approaches and existing domain workflows. A comparison with an appropriately adapted related workflow, or a clear explanation of why such a comparison would be unsuitable, would help substantiate its added value.

  2. The necessity of the six stages and the contribution of the final synthesis remain unverified. No controlled ablations establish the contributions of individual stages or the chosen intermediate representations. This is particularly relevant because some stages operate only on compressed intermediate outputs, introducing a potential information-loss trade-off that is not isolated experimentally. Furthermore, although all six calls are executed, the semantic quality metrics evaluate the theme set available after stage five; the synthesis produced by stage six is not scored. The current evaluation therefore cannot establish the value of every stage or assess the quality contribution of the complete six-stage workflow.

[1] Katz, A., Fleming, G. C., & Main, J. B. (2026). Thematic analysis with open-source generative AI and machine learning: a new method for inductive qualitative codebook development. Humanities and Social Sciences Communications, 13, 209.

Questions for authors
  1. Comparative evidence: Can you provide results for a direct single-call baseline using the same model, documents, and final output requirements, together with its realized resource use? Could you also compare TAIWO with an appropriately adapted related workflow, such as GATOS [1], or explain why such a comparison would be inappropriate?

  2. Component evidence and evaluation coverage: Is there controlled evidence isolating the contributions of the individual stages and intermediate representations? Since stage six does not change the scored theme set, what is its intended contribution, and is there evidence assessing the quality of its synthesis?

Reviewer_RaGuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper audits structured LLM inference for open-ended semantic discovery and introduces TAIWO, a typed six-stage structured inference workflow. It builds a controlled synthetic corpus of 100 documents with nine latent concepts and runs benchmarks across 39 distinct model configurations, alongside a transfer pilot on the public Reddit mental health dataset. Experiments reveal substantially lower recall for rare latent concepts. The original v1 scoring metric has measurement flaws: it only validates whether source identifiers exist instead of verifying evidential support for themes. Two equal-call exploratory pilots find no empirical evidence that TAIWO outperforms generic six-call decomposition. The core contribution is an auditable evaluation framework with explicit failure semantics, not a validated superior workflow; the practical benefits of TAIWO’s typed design remain inconclusive. Preregistered follow-up confirmatory experiments are proposed.

Critical strengths

This paper decomposes reliability into semantic retention, interface contract compliance and measurement validity. It avoids LLM as judge, uses synthetic corpus with ground-truth latent concepts, and builds adversarial probes to expose scorer bugs, demonstrating reproducibility does not equal construct validity.

Critical weaknesses
  1. The central workflow consists of familiar operations, summarization, coding, theme construction, review, refinement, and synthesis, connected through typed intermediate artifacts. The paper itself acknowledges that these operations are not novel and that their typed composition has not been shown to provide additional value. More importantly, neither pilot isolates the proposed design choices from generic decomposition.
  2. The principal comparison provides essentially no evidence that TAIWO is better than generic decomposition. On the synthetic task, the reported paired difference is only 0.0068, with a 95% confidence interval of [−0.0845, 0.0908] and p=0.859. On the human-labelled transfer task, the difference is −0.1233, with a 95% confidence interval of [−0.3540, 0.2241]. Neither result establishes an advantage, and the transfer-task point estimate favors the baseline. Although the authors appropriately describe these experiments as exploratory, this leaves the proposed workflow's principal value proposition unsupported. A null result can be scientifically informative, but only when it answers a meaningful research question with sufficient rigor; here, the paper has neither established the workflow's superiority nor convincingly demonstrated a new failure mechanism attributable to its design.
  3. The primary benchmark contains only 100 synthetic documents and nine predefined concepts, each generated from a small set of templates. This setup facilitates deterministic scoring but risks rewarding lexical recovery rather than genuine open-ended semantic discovery. The reported recall gap between dominant and rare concepts is also confounded by concept identity: the same concepts remain assigned to the same prevalence strata, so prevalence cannot be separated from semantic difficulty. The authors acknowledge this limitation, but acknowledging a confound does not make the resulting observation strong evidence for the broader claim. The Reddit transfer experiment is similarly narrow, covering only 100 posts and four mutually exclusive root-cause categories. These experiments do not establish robust performance on realistic qualitative analysis, where themes may overlap, emerge inductively, and admit multiple defensible interpretations.
  4. The two pilots involve only three model routes and three seeds, yielding nine paired comparisons per task. Such a small number of model clusters makes estimates of cross-model variability and cluster-level uncertainty highly fragile. The wide confidence intervals do not establish equivalence between TAIWO and the baseline; they indicate that the experiments cannot reliably distinguish a meaningful benefit from a meaningful disadvantage. Consequently, the paper cannot support either a superiority claim or a strong claim that the two approaches are practically equivalent.
  5. A central motivation of this work is that benchmark scores can be misleading when their measurement contracts are flawed. Yet the paper's own historical evaluation relied on a score whose grounding component did not measure evidential support, and whose exported results omitted declared failures. The subsequent audit is useful, but it also substantially changes how the original benchmark results should be interpreted. Moreover, the corrected metric cannot be applied retrospectively to the historical breadth cohort because the necessary artifacts are unavailable. This creates an uncomfortable gap between the paper's methodological message and its evidential foundation: the authors demonstrate that their original measurement procedure was inadequate, but cannot fully reconstruct the corrected results needed to establish what the benchmark actually shows.
  6. Despite emphasizing reproducibility, the paper acknowledges that the provenance bundle omits raw metric files referenced by the canonical results and lacks some historical run artifacts. The missing artifacts prevent retrospective rescoring and stage-wise analysis of the 39-run cohort. In addition, the historical cohort mixes package versions, with the exact older source unavailable, confounding implementation version with model route. These issues do not automatically invalidate the reported findings, but they limit independent verification and make the reproducibility contribution less convincing than the paper's framing suggests.
Questions for authors

As shown above.

Reviewer_mEAyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits structured test-time inference for open-ended semantic discovery across document collections. It introduces TAIWO, a six-stage sequential prompting pipeline inspired by thematic analysis (familiarization →\to initial_coding →\to theme_identification →\to theme_review →\to theme_refinement →\to synthesis, Section 3.1 Eq. (1), Appendix B Table 4), and evaluates it in three parts:

  1. Frozen Breadth Study (Sections 4–5, Table 1, Appendix C Table 5): Evaluates 38 models (36 completed, 2 failed) on a 28-document synthetic corpus (6 common concepts, 3 single-document rare concepts, 1 noise document) under composite score Sv1=0.60Rbal+0.25PT+0.15VIS _ {\mathrm{v1}} = 0.60 R _ {\mathrm{bal}} + 0.25 P _ T + 0.15 V _ I (Eq. (3)).
  2. Construct-Validity Audit & Sv2S _ {\mathrm{v2}} Re-evaluation (Section 6, Appendices D–E Table 6): Demonstrates via adversarial probes that VIV _ I in Sv1S _ {\mathrm{v1}} rewards valid document IDs even when cited documents do not support the matched concept, and proposes a support-aware revision Sv2S _ {\mathrm{v2}}.
  3. Controlled Paired Pilots (Section 7, Appendices F–I Tables 7–9): Compares single_pass, generic_decomposition, taiwo_no_review_refine, and taiwo_full across 3 models ×\times 3 seeds on both the synthetic benchmark and a 90-post Reddit Mental Health Dataset (RMHD) slice.
Critical strengths
  1. Honest negative results and constructive measurement-audit framing. Rather than overclaiming superiority for a six-stage pipeline, the authors transparently report that taiwo_full does not consistently outperform single_pass (Section 7, Table 7) and dedicate Section 6 to auditing construct-validity flaws in their initial benchmark scorer Sv1S _ {\mathrm{v1}}.

  2. Exemplary artifact completeness enabling deep verification. The anonymous supplement (TAIWO_ICLR2027_Anonymous_Supplement) includes the full pipeline source, scorer implementation, unit tests, raw run JSONs/CSVs, and table-reproduction scripts (scripts/reproduce_iclr_tables.py), allowing every number in Tables 1–9 to be audited directly against code.

Critical weaknesses
  1. Broken construct validity in both Sv1S _ {\mathrm{v1}} and the proposed Sv2S _ {\mathrm{v2}}, plus a macro-vs-micro VIV _ I reporting mismatch in Table 1.

    • Proposed Sv2S _ {\mathrm{v2}} still awards 0.850 to ungrounded/fabricated-ID outputs (kbench_score.py:176-229): While Section 6.2 claims Sv2S _ {\mathrm{v2}} fixes Sv1S _ {\mathrm{v1}}'s grounding loophole, evaluate_kbench_controlled_discovery_v2 only replaces the 0.15VI0.15 V _ I term while leaving Eq. (4) (M(t,c)=0.65L(t,c)+0.35E(t,c)≥0.45M(t,c) = 0.65 L(t,c) + 0.35 E(t,c) \ge 0.45) unchanged for both RbalR _ {\mathrm{bal}} (weight 0.60) and PTP _ T (weight 0.25). Because lexical anchor matching carries weight 0.65, any theme matching 3 of 4 anchor words (L(t,c)≥0.75  ⟹  0.65×0.75=0.4875≥0.45L(t,c) \ge 0.75 \implies 0.65 \times 0.75 = 0.4875 \ge 0.45) achieves full recall and precision even when E(t,c)=0E(t,c) = 0 (no IDs, fabricated IDs D999, or wrong valid IDs). Consequently, all three adversarial probes in the Appendix D table (p. 13; tests/test_kbench_score.py:81-116) still score Sv2=0.850S _ {\mathrm{v2}} = 0.850—which is higher than every real model run in Table 6 (max 0.7368)! Moreover, _theme_text (kbench_score.py:38-40) only concatenates name and description and completely ignores the theme.evidence verbatim quotes required by the JSON schema (test_kbench_score.py:151-167).
    • Macro grounding vs. micro valid_source_rate in Table 1: Section 4.3 (p. 4, lines 203–208) defines VIV _ I as the per-theme mean fraction of valid IDs (grounding in kbench_score.py:85, 145-151, which scores 0.0 for themes with empty source_ids; mean 0.967, SD 0.168, min 0.007 across the 36 runs). However, Table 1 (p. 5) and Section 5.3 (p. 6, line 290) report micro-averaged valid_source_rate (which ignores empty-ID themes; mean 0.973, SD 0.165, min 0.011) in the VIV _ I column, making Sv1=0.60Rbal+0.25PT+0.15VIS _ {\mathrm{v1}} = 0.60 R _ {\mathrm{bal}} + 0.25 P _ T + 0.15 V _ I irreproducible from the printed columns (e.g., qwen/qwen3-next-80b-a3b-instruct has macro grounding = 0.800 vs. micro valid_source_rate = 1.000).
  2. Unenforced 3,000-token per-call cap in the breadth study, an unscored theme_count = 0 failure in RMHD v3 (Table 7), and Eq. (1) vs. prompt mismatches in core.py.

    • Disabled token cap (taiwo_iclr_structured_discovery.py:186,189): Section 4.1 (p. 4, line 172) and Table 3 (p. 12) state that every stage has a strict 3,000 completion-token cap (18,000 total across 6 calls). In code, enforce_output_token_cap=False sets ctx.budget = None (kbench_provider.py:129-136, 249 logs "output_token_cap_enforced": False). In kbench_breadth_runs.csv, 25 of 36 completed runs exceeded 18,000 completion tokens (median 21,796; up to 41,165 tokens / 6,861 per call for google/gemini-3.5-flash, an 11.5×11.5\times gap vs. 3,573 for claude-4-sonnet), confounding Section 5.5's token-efficiency claims.
    • Silent fallback in core.py and theme_count = 0 run in Table 7: In src/taiwo_iclr/pipeline/core.py:193-195, 201-203, if reviewed: and if refined: silently retain the previous stage's themes when Stage 4 or 5 outputs {"themes": []}. Worse, in artifacts/rmhd_transfer_v3_live_sanitized/run_scores.csv (row 13, qwen/qwen3-coder-480b-a35b-instruct, seed 2029, taiwo_full), Stage 3 emitted 0 themes, yielding final theme_count = 0 and score = 0.0000, which was averaged into Table 7 (dragging Qwen's paired delta from +0.3958 and +0.5000 down to -0.1889) instead of being marked status = failed per Section 4.3 (lines 210–213).
    • Omission of corpus DD in Stages 3, 5, 6 (core.py:145-215) and Stage-3 survival collapse: Contrary to Eq. (1) (zs=fs(D,z<s;m,τ,b,r)z _ s = f _ s(D, z _ {\lt s}; m, \tau, b, r)), Stages 3, 5, and 6 in core.py do not receive docs_block (DD), Stage 5 ignores Stage 4's reviews, and Stage 6's synthesis is never used by any scorer. In fact, taiwo_full generated more completion_tokens than generic_decomposition in both v3 pilots (7,744 vs. 7,467 on Synthetic; 10,977 vs. 9,443 on RMHD); its lower total_tokens in Table 8 comes solely from passing docs_block in 3 calls instead of all 6 calls (causal.py:166-184). Moreover, scoring the saved stage outputs in artifacts/paired_pilot_v3_live/run_store/results/*.json via Eqs. (5)–(6) reveals that rare-concept recall is 0.8519 (23/27) at Stage 2 (initial_coding) and collapses to 0.4074 (11/27, hazard 0.5217) at Stage 3 (theme_identification) when codes are clustered without DD (and stage_survival.py:39-62 has a bug returning [] for generic_1..5 because it overlooks raw["analysis"]).
Questions for authors
  1. Fixing the 0.850 loophole in Sv2S _ {\mathrm{v2}} and reconciling grounding vs. valid_source_rate in Table 1: In kbench_score.py:176-229, why does Sv2S _ {\mathrm{v2}} still allow a theme with zero document-ID support (E(t,c)=0E(t,c) = 0) to satisfy M(t,c)=0.65L(t,c)+0.35E(t,c)≥0.45M(t,c) = 0.65 L(t,c) + 0.35 E(t,c) \ge 0.45 via lexical overlap alone (Sv2=0.850S _ {\mathrm{v2}} = 0.850), and why are theme.evidence quotes ignored by _theme_text? How do Table 6 and Table 7 change if matching a concept requires multiplicative or conjunctive document support (E(t,c)>0E(t,c) \gt 0 and verified substring presence of theme.evidence in the cited document)? Please also correct Table 1 so that VIV _ I reports macro grounding matching Eq. (3).

  2. Unenforced 3,000-token cap (enforce_output_token_cap=False) and the theme_count = 0 run in RMHD v3 (Table 7): Why was enforce_output_token_cap=False used in benchmarks/taiwo_iclr_structured_discovery.py:186,189 while Section 4.1 and Table 3 state a strict 3,000-token per-stage cap? Furthermore, in RMHD v3 (Table 7), why was qwen3-coder-480b seed 2029 (taiwo_full, theme_count = 0, score = 0.0000) included as a completed run rather than marked failed under the non-empty output contract of Section 4.3, and why do Stages 4–5 silently fall back to Stage 3 themes when they output {"themes": []} (core.py:193-203)?

  3. Passing corpus DD to Stage 3 and computing stage-wise survival Sg(s)S _ g(s) (Eqs. (5)–(6)) on the saved v3 artifacts: Since the 18 JSON files in artifacts/paired_pilot_v3_live/run_store/results/ already store all 6 stage outputs—showing that rare-concept recall drops from 0.8519 at Stage 2 to 0.4074 at Stage 3 (where docs_block DD is omitted from the prompt in core.py:165-175)—why did Section 7 and Appendix J state that stage-wise survival could not be computed? What happens to rare-concept recall and Table 8 token counts when Stage 3 is given access to DD as specified in Eq. (1)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.