Leveraging Neural Network Layers as an Annealing Path

Sampling and optimization with trained neural networks can be difficult even when their gradients are available: high-order interactions can obscure useful search directions. We introduce layerwise annealing, which uses the functions decoded from intermediate network layers as an annealing trajectory. The motivation is that earlier-layer functions can approximate smoothed versions of the final objective, with high-degree Fourier components attenuated. Their gradients can reveal promising directions that the final objective obscures, while progressively deeper layers restore the detailed structure needed to refine the solution. We exploit this by applying prediction heads at successive depths of a frozen network and carrying samples or optimization states through the resulting objectives, ending with the original function. Controlled Boolean and continuous experiments demonstrate the proposed mechanism: high-degree interactions and narrow density modes emerge progressively across layers. We demonstrate practical utility in LLM prompt optimization. On AdvBench Harmful Strings with Llama-2-7B-Chat, layerwise Greedy Coordinate Gradient achieves 70% target-string success, compared with the published GCG reference of 57%, with mean target cross-entropy of 0.195 versus 0.30.