- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
TableLoop converts unstructured inputs into numeric features using learned rubrics, then combines these features with structured measurements for a tabular predictor. Rubrics are revised using validation performance and instance-level errors. The paper also introduces SH-merge, a successive-halving method that merges complementary rubrics. It is evaluated on nine tasks across text and image inputs, including classification and regression.
Critical strengths
Broad and consistent empirical validation across nine tasks, multiple modalities, and different evolution methods.
The learned rubrics produce inspectable, reusable feature tables, which is a practical advantage over end-to-end LLM prediction.
Thorough ablations covering initialization, evolution method, instance-level feedback, metadata integration, predictor choice, and training subsampling.
Critical weaknesses
The direct LLM/VLM baseline is zero-shot with affine calibration, not a fine-tuned or otherwise stronger baseline. This limits how much of the gain is attributable to TableLoop versus the choice of baseline.
Rubric evolution is computationally expensive. The paper acknowledges this and shows training subsampling helps, but the full search budget remains heavy.
The retrospective truncation analysis evaluates the test set at every iteration. The main experiments use a fixed budget and evaluate the test set once, so the concern is limited, but the analysis should be clarified.
Questions for authors
How does TableLoop compare to fine-tuned LLMs or strong multi-modal AutoML baselines?
What is the total computational cost of rubric search across the nine tasks?
2 comments
Response to Reviewer 875d (2/2)
Weakness 2 and Question 2: What is the total computational cost of rubric search across the nine tasks?
Response: Thank you for raising this point. We added a runtime and computational cost analysis in Appendix L (Tables 23–25), reporting per-example GPU time, scoring throughput, and GPU-hour breakdowns for an offline candidate evaluation and a held-out evaluation with one selected rubric. Importantly, Appendix L reports measured GPU-hours for complete training- and validation-set scoring and TabPFN evaluation per candidate, while Appendix I.2 specifies the search budgets for all three evolution methods. Together, these measurements quantify the computational scale of rubric search rather than relying solely on theoretical complexity or per-token estimates. We agree that offline search is relatively costly because it requires repeatedly scoring training and validation examples. This upfront cost is part of learning an interpretable representation in which each feature is defined by an explicit scoring criterion that can be inspected and reused. In the online stage, TableLoop applies the selected rubrics to new inputs and reuses the fitted tabular predictors, which is much less costly than running the full rubric search.
Several strategies can further reduce the cost of both search and prediction. The number of scoring calls during search can be reduced by evaluating fewer candidates through early stopping or by scoring fewer training examples per candidate. On ELLIPSE, using just 10% of the training examples per candidate while retaining the full validation set yielded comparable test MAE to full-data search (0.3323 vs. 0.3334, Appendix F). Fewer scoring repetitions can further reduce the number of calls, while parallel scoring can shorten wall-clock time in both stages. Most of the Qwen scorers’ GPU time goes to autoregressive decoding, and our benchmark shows that JEV System 1 (9B) achieves a roughly 5× speedup over Qwen3-8B in per-example GPU time on the same rubrics (Table 23). A promising direction for future work is to test whether JEV can reduce TableLoop’s end-to-end runtime without sacrificing predictive performance.
Weakness 3: The retrospective truncation analysis evaluates the test set at every iteration. The main experiments use a fixed budget and evaluate the test set once, so the concern is limited, but the analysis should be clarified.
Response: We thank the reviewer for this question. As described in Section 5.1, the main OpenEvolve experiments use a fixed budget of 50 iterations. Validation feedback guides rubric revisions and selection, and we evaluate the final ensemble on the test set only once. For Figure 3, after the runs were complete, we reconstructed the top-three ensemble at each checkpoint using validation performance alone and evaluated it on the test set. These retrospective evaluations show how held-out performance changes with the search budget and did not affect how we revised the rubrics, which rubrics we selected for the ensemble, or when we stopped the search.
Response to Reviewer 875d (1/2)
We thank the reviewer for acknowledging the practical value of TableLoop’s inspectable, reusable feature tables, the breadth of our evaluation across tasks and modalities, and the thoroughness of our ablation studies. We reply to each of the points raised in the review below. We have also uploaded a revised version of the manuscript highlighting the main changes in blue color.
Response: We thank the reviewer for the valuable question. We added a fine-tuned direct predictor and a supervised multimodal AutoML baseline in Appendix J.
LoRA fine-tuning. On Persuasion, we fine-tuned Qwen3-32B to predict donations using the same dialogues, metadata, and data splits as TableLoop. The fine-tuned model achieved a test ROC-AUC of 0.7930, while TableLoop (OpenEvolve) achieved 0.8393 with a frozen scorer (Appendix J.2).
Supervised multimodal AutoML. We evaluated AutoGluon-Multimodal (AutoMM) on seven tasks using the same source inputs, metadata, splits, and metrics as TableLoop (Appendix J.1). We used the best-quality preset with augmentation and also evaluated a long-context ModernBERT variant for text. Both configurations are reported below where applicable.
The results vary by task. TableLoop outperforms both AutoMM configurations on Persuasion and both ICLR tasks, with the largest gains over the stronger AutoMM configuration on Persuasion and ICLR (RL), at 0.1279 and 0.1152 ROC-AUC, respectively. The methods perform similarly on MIMIC-MV and PAD-UFES-20, while AutoMM performs better on ELLIPSE and Danish Fungi. This pattern is consistent with our hypothesis that rubric-based scoring is particularly useful when prediction involves complex reasoning over text.