Tailoring the Clustering Objective to the Data: Utility-Guided Validity-Index Selection and Candidate Reranking

Choosing a clustering configuration without labels requires a criterion, and no single cluster-validity index (CVI) is reliable across geometries. Automated clustering systems adapt the criterion to the dataset, but coarsely: one predicted index, a fixed set, or one global quality surrogate. We present CLUSTOPT, which composes the criterion per dataset for two-dimensional data. Label-free meta-features are mapped to a predicted utility profile: for each of 60 indices, how well it would rank candidate partitions of the data at hand. The search objective is a weighted combination of the indices predicted to be informative, and a learned reranker makes the final choice among the candidates the search visited. To give the profile something to select among, we contribute 46 structural indices. On 1,055 controlled datasets held out from all learned components, weighting the vocabulary by the predicted profile is worth +0.227 ARI over weighting it uniformly. The profile also beats every alternative criterion we tested, including random search, the best single index, a static three-index core and a dataset-independent profile over the same vocabulary. The comparison shows why: all criteria visit candidates of near-identical quality, and differ by up to 0.26 ARI in which candidate they return. Selection is what matters, and CLUSTOPT improves it twice. Reranking the slate adds a further +0.058 ARI and recovers 61% of the distance to the best candidate visited, at under 1% of the search cost. On an independent 50-dataset benchmark CLUSTOPT has the highest mean of any system, a significant paired advantage over ML2DAC, and a lead over AutoClust on non-convex, connectivity-defined structure. Three generalist indices transfer across benchmarks; the remaining new indices are specialists whose value appears only under selection