← Back to Index

Evolution Strategies at Scale

First successful application of ES to full-parameter LLM fine-tuning at billion-parameter scale without dimensionality reduction Organization: Cognizant AI Labs / University of Texas at Austin Published: September 2025 (v1), February 2026 (v2) Type: Research Paper (ICML 2026) Report Type: PhD-Level Technical Analysis Report Date: April 2026

Table of Contents

• Full Title and Attribution
• Authors and Team
• Core Contribution
• Supported Solutions
• LLM Integration
• Key Results
• Reproducibility
• Compute and API Costs
• Architecture Solution
• Component Breakdown
• Core Mechanisms (Detailed)
• Programming Language
• Memory Management
• Continued Learning
• Applications

1 Full Title and Attribution

Full Title: Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning

arXiv: 2509.24372 (cs.LG, cs.AI, cs.NE)

Repository: github.com/VsonicV/es-fine-tuning-paper (340+ stars)

Venue: ICML 2026

Submission History: - v1: September 29, 2025 (original submission) - v2: February 6, 2026 (revised version with additional experiments)

DOI: 10.48550/arXiv.2509.24372

License: CC BY-NC-SA 4.0

Lineage: Builds on the intellectual foundation of OpenAI ES (Salimans et al., 2017) and Natural Evolution Strategies (Wierstra et al., 2008, 2014). Positioned as the first work to scale ES from millions of parameters (prior art) to billions of parameters (LLMs). The paper explicitly challenges the assumption — widely held since Vemula et al. (2019) — that parameter-space exploration is inherently unscalable.

Key Claim: ES is not merely a viable alternative to RL for LLM fine-tuning, but a fundamentally different and powerful backpropagation-free post-training paradigm that opens a new direction for LLM fine-tuning.

2 Authors and Team

Author List

Team Context

This paper comes from Cognizant AI Labs, the research arm of Cognizant Technology Solutions, which maintains one of the most established evolutionary computation research groups in industry. The team is led by Babak Hodjat (co-founder of Sentient Technologies, one of the largest AI startups focused on evolutionary computation) and Risto Miikkulainen (UT Austin professor, inventor of NEAT — NeuroEvolution of Augmenting Topologies, one of the most cited neuroevolution algorithms).

Elliot Meyerson is notable for his work on evolutionary search with LLMs as mutation operators — his 2024 paper on "Language Model Crossover" is a key precursor to the LLM-as-evolutionary-operator paradigm used in systems like AlphaEvolve.

The team's collective expertise in neuroevolution gives this paper particular authority. When the group that invented NEAT and scaled evolutionary optimization to production systems at Sentient Technologies claims that ES scales to billion-parameter LLMs, the community pays attention.

Institutional Significance

Cognizant AI Labs occupies a unique position in the evolutionary AI landscape:

While DeepMind uses LLMs to evolve code (programs, algorithms), and Sakana uses evolution to merge LLMs, Cognizant's contribution is fundamentally different: they use evolution to directly optimize the parameters of LLMs. This is the most classical form of neuroevolution, applied at unprecedented scale.

3 Core Contribution

Key Novelty: For the first time, Evolution Strategies (ES) is successfully scaled to direct, full-parameter fine-tuning of LLMs with billions of parameters — without any dimensionality reduction (no LoRA, no final-layer-only, no action-space surrogates). The paper demonstrates that ES outperforms established RL methods (PPO, GRPO, Dr.GRPO) across multiple axes, overturning the widespread assumption that ES cannot scale to modern model sizes.

The Assumption Overturned

The conventional wisdom in the field, established by Vemula et al. (2019), held that parameter-space exploration complexity scales quadratically with parameter count ((O(d^2)) where (d) is dimensionality), making it intractable for models with billions of parameters. Prior ES applications had been limited to:

This paper:

The jump is dramatic: from millions to billions of parameters, and from populations of 10,000+ to just 30. Both changes were assumed to be individually fatal to ES performance — together, they should have been catastrophic. Instead, ES outperforms RL.

Six Advantages of ES Over RL for LLM Fine-Tuning

The paper identifies six systematic advantages:

1. Long-horizon reward tolerance. ES needs only response-level (outcome) rewards, not token-level credit assignment. For reasoning tasks where only the final answer is graded, ES avoids the credit assignment problem entirely.

2. Small populations in high-dimensional spaces. A population of just 30 is sufficient to search in multi-billion-parameter spaces. Previous work assumed populations must be proportional to dimensionality.

3. Cross-model robustness. ES consistently fine-tunes all tested LLMs (Qwen-2.5, Llama-3.x families, 0.5B–8B). RL methods fail on some models, particularly smaller ones.

4. Reward hacking resistance. ES optimizes a solution distribution (the Gaussian perturbation cloud), which is harder to hack than RL's single-solution optimization. RL tends to exploit reward function loopholes.

5. Cross-run stability. ES produces consistent results across multiple runs with the same hyperparameters. RL is often unstable, requiring expensive hyperparameter sweeps per model.

6. Memory efficiency. ES requires only inference (no backpropagation), eliminating the need for gradient storage, optimizer states, and activation caching. Significant GPU memory savings.

Paradigm Positioning

The paper positions ES not as a niche technique but as a new post-training paradigm:

Pre-training ─────────────────────────────────────────►
    │                    │                    │
    ▼                    ▼                    ▼
 SFT (Supervised      RLHF / RL            ES Fine-Tuning
 Fine-Tuning)         (PPO, GRPO,           (This Paper)
                       DPO, etc.)
                                            • No gradients
 • Gradient-based     • Gradient-based      • No backprop
 • Requires labels    • Requires reward     • Requires reward
 • Deterministic        model or rule         function only
   training           • Token-level or      • Response-level
                        outcome rewards       rewards only
                      • Action-space        • Parameter-space
                        exploration           exploration

4 Supported Solutions

Fine-Tuning Tasks Evaluated

The paper evaluates ES on three categories of tasks:

1. Symbolic Reasoning: Countdown Task

The Countdown task (Gandhi et al., 2024; Pan et al., 2025) requires the model to combine given numbers using arithmetic operations to reach a target number.

2. Behavioral Fine-Tuning: Conciseness

Fine-tuning LLMs to produce shorter, more concise responses to knowledge questions.

3. Math Reasoning: GSM8K, MATH500, Minerva Math, OlympiadBench

Extended comparisons with SOTA RL methods:

4. Puzzle Problem Solving

ES is applied to solve two puzzle problems that base LLMs struggle with:

Solution Space Characterization

5 LLM Integration

LLMs as Optimization Targets (Not Operators)

This paper uses LLMs fundamentally differently from systems like AlphaEvolve, FunSearch, or EvoPrompting. In those systems, the LLM is a tool that generates candidate solutions (code, programs). In this paper, the LLM is the optimization target — its parameters are the search space, and ES directly manipulates billions of floating-point values to improve task performance.

AlphaEvolve / FunSearch approach:
┌───────────────┐     ┌──────────────┐     ┌──────────────┐
│ LLM (frozen)  │────►│ Generated    │────►│ Evaluator    │
│ generates code│     │ code/program │     │ scores code  │
└───────────────┘     └──────────────┘     └──────────────┘
        ▲                                          │
        └──────────── fitness feedback ────────────┘

This paper's approach (ES at Scale):
┌─────────────────────────────────────────────────────┐
│              LLM Parameters (θ)                     │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐            │
│  │ Layer 1  │ │ Layer 2  │ │ Layer N  │  ...        │
│  │ weights  │ │ weights  │ │ weights  │  (billions  │
│  │          │ │          │ │          │   of params) │
│  └──────────┘ └──────────┘ └──────────┘            │
│       ↑              ↑              ↑               │
│  ε₁ ~ N(0,I)   ε₂ ~ N(0,I)   εₙ ~ N(0,I)         │
│  (perturbation)                                     │
└─────────────────────────────────────────────────────┘
         │
         ▼
┌──────────────┐     ┌──────────────┐
│ Perturbed LLM│────►│ Reward R(θ+σε)│
│ generates    │     │ (binary or   │
│ responses    │     │  composite)  │
└──────────────┘     └──────────────┘
         │
         ▼
θ ← θ + α · (1/N) · Σ Rₙ · εₙ   (parameter update)

Models Evaluated

Inference-Only Operation

A critical property of ES fine-tuning is that it requires only forward passes through the model:

This has profound implications for GPU memory:

GPU Memory Layout — RL Fine-Tuning:
┌────────────────────────────────────────────────────┐
│ Model weights (bf16)               │  ~16 GB (8B)  │
│ Gradient buffers                   │  ~16 GB       │
│ Optimizer states (Adam: m, v)      │  ~32 GB       │
│ Activation cache (for backprop)    │  ~8–32 GB     │
│ Reference model (KL penalty)       │  ~16 GB       │
├────────────────────────────────────┼───────────────┤
│ TOTAL                              │  ~88–112 GB   │
└────────────────────────────────────┴───────────────┘

GPU Memory Layout — ES Fine-Tuning:
┌────────────────────────────────────────────────────┐
│ Model weights (bf16)               │  ~16 GB (8B)  │
│ Layer-sized noise tensor (temp)    │  ~0.1–2 GB    │
│ Random seeds (N integers)          │  negligible   │
├────────────────────────────────────┼───────────────┤
│ TOTAL                              │  ~16–18 GB    │
└────────────────────────────────────┴───────────────┘

This ~5–6× memory reduction is a substantial practical advantage, enabling fine-tuning of larger models on smaller GPU clusters.

6 Key Results

Headline Result: Countdown Task (Table 1)

ES outperforms all RL baselines across all 7 tested models:

Key observations:

1. ES wins on every model. Not a single RL variant beats ES on any model size or family.
2. Largest gains on mid-sized models. Qwen-2.5-3B shows the most dramatic improvement: 60.5% vs 43.8%, a +16.7 absolute improvement.
3. Single hyperparameter set. ES uses the same hyperparameters (N=30, σ=0.001, α=5×10⁻⁴) for ALL models. RL requires per-model hyperparameter sweeps.
4. Cross-family robustness. ES works on both Qwen and Llama families. RL performance varies significantly across families.

Reward Hacking Analysis (Conciseness Task)

The conciseness task reveals a qualitative difference between ES and RL:

The paper explains: "ES optimizes a solution distribution (the perturbation cloud), which is more difficult to hack, while RL optimizes a single solution."

This is a fundamental insight. RL fine-tuning produces a single policy that can find and exploit reward function loopholes. ES produces a distribution of nearby policies — for any reward hack to persist, it must be robust to Gaussian perturbation of all parameters, which is much harder for degenerate behaviors to achieve.

Cross-Run Stability

ES shows dramatically lower variance across independent runs:

This stability has practical cost implications. If RL requires 3–5 runs to find a good hyperparameter configuration plus 2–3 runs for reliability, and ES requires a single run, the total compute cost may be lower for ES despite its per-iteration expense.

Math Reasoning Benchmarks (Extended Results)

ES is compared against additional SOTA RL baselines on math reasoning:

On these standard math benchmarks, ES performs comparably to the best RL methods, demonstrating that the advantages of ES (robustness, stability, no reward hacking) do not come at the cost of reduced performance on well-studied tasks.

Hyperparameter Sensitivity

The paper reports that a single ES configuration works across all 7 models, while RL requires separate hyperparameter sweeps for each model — a significant practical advantage.

7 Reproducibility

Code Availability

The complete source code is available at github.com/VsonicV/es-fine-tuning-paper:

Setup and Execution

# Environment setup
python -m venv es
source es/bin/activate
pip install -r requirement.txt

# Conciseness fine-tuning (2 GPUs)
accelerate launch \
    --num_processes 2 \
    --num_machines 1 \
    --machine_rank 0 \
    es_fine-tuning_conciseness.py \
    --gpu_threads=1 \
    --model_name=Qwen/Qwen2.5-7B-Instruct

# Countdown fine-tuning (4 GPUs)
accelerate launch \
    --num_processes 4 \
    --num_machines 1 \
    --machine_rank 0 \
    countdown/es_fine-tuning_countdown.py \
    --data_sample 200 \
    --model_name Qwen/Qwen2.5-3B-Instruct \
    --gpu_threads 1

# Accelerated version with vLLM (4 GPUs)
python es_fine-tuning_countdown_accl.py \
    --model_name Qwen/Qwen2.5-3B-Instruct \
    --cuda_devices 0,1,2,3 \
    --num_engines 4 \
    --population_size 30 \
    --num_iterations 1000

Reproducibility Assessment

Noise Variants

The repository provides two noise implementations:

The discussion at github.com/VsonicV/es-fine-tuning-paper/discussions/7 provides additional details on the difference. Both variants achieve similar results.

8 Compute and API Costs

Hardware Requirements

Per-Iteration Cost Analysis

Each ES iteration involves N=30 perturbed model evaluations:

Per-iteration compute:
┌────────────────────────────────────────────────────────┐
│ 1. Generate N=30 noise seeds                │ ~0 sec  │
│ 2. For each of N=30 perturbations:          │         │
│    a. Perturb model (layer-by-layer)        │ ~2 sec  │
│    b. Run inference on training batch       │ ~10 sec │
│    c. Compute reward                        │ ~1 sec  │
│    d. Restore model                         │ ~2 sec  │
│ 3. Normalize rewards (z-score)              │ ~0 sec  │
│ 4. Aggregate update (layer × seed)          │ ~5 sec  │
├────────────────────────────────────────────────────────┤
│ Total per iteration (serial, 4 GPUs):       │ ~2 min  │
│ Total per iteration (accelerated, vLLM):    │ ~12 sec │
└────────────────────────────────────────────────────────┘

Total Training Cost Estimation

For a typical Countdown experiment (1000 iterations, 4 × H100):

Cost Comparison with RL

The accelerated ES version is dramatically cheaper than RL alternatives, primarily because: 1. No hyperparameter sweeps needed (one config works for all models) 2. Lower memory footprint enables smaller/fewer GPUs 3. vLLM-based inference is highly optimized 4. No backpropagation overhead

Scaling Properties

9 Architecture Solution

Algorithm Architecture

The ES fine-tuning system has a clean, modular architecture:

┌─────────────────────────────────────────────────────────────────┐
│                    ES FINE-TUNING ARCHITECTURE                  │
│                                                                 │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │                    OUTER LOOP (T iterations)             │   │
│  │                                                          │   │
│  │  ┌──────────────────────────────────────────────────┐    │   │
│  │  │           INNER LOOP (N perturbations)           │    │   │
│  │  │                                                  │    │   │
│  │  │  For n = 1 to N (parallelizable):                │    │   │
│  │  │  ┌─────────────────────────────────────────────┐ │    │   │
│  │  │  │  1. Sample seed sₙ                          │ │    │   │
│  │  │  │  2. Perturb θ in-place (layer by layer):    │ │    │   │
│  │  │  │     For each layer ℓ:                       │ │    │   │
│  │  │  │       εₗ = generate_noise(sₙ, ℓ)           │ │    │   │
│  │  │  │       θₗ += σ · εₗ                          │ │    │   │
│  │  │  │  3. Generate responses (greedy decoding)    │ │    │   │
│  │  │  │  4. Compute reward Rₙ = R(responses)        │ │    │   │
│  │  │  │  5. Restore θ in-place (layer by layer):    │ │    │   │
│  │  │  │     For each layer ℓ:                       │ │    │   │
│  │  │  │       εₗ = generate_noise(sₙ, ℓ)           │ │    │   │
│  │  │  │       θₗ -= σ · εₗ                          │ │    │   │
│  │  │  └─────────────────────────────────────────────┘ │    │   │
│  │  └──────────────────────────────────────────────────┘    │   │
│  │                                                          │   │
│  │  Normalize: R̃ₙ = (Rₙ - mean(R)) / std(R)               │   │
│  │                                                          │   │
│  │  Update (decomposed, layer by layer, seed by seed):      │   │
│  │  For each layer ℓ:                                       │   │
│  │    For each seed sₙ:                                     │   │
│  │      εₗ = generate_noise(sₙ, ℓ)                         │   │
│  │      θₗ += α · (1/N) · R̃ₙ · εₗ                          │   │
│  └──────────────────────────────────────────────────────────┘   │
│                                                                 │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │                    EVALUATION                            │   │
│  │  • Periodic evaluation on held-out test set              │   │
│  │  • Checkpoint best-performing model                      │   │
│  └──────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Parallelization Strategy

The inner loop (N perturbations) is embarrassingly parallel — each perturbation evaluation is independent:

GPU 0                  GPU 1                  GPU 2                  GPU 3
┌──────────────┐      ┌──────────────┐      ┌──────────────┐      ┌──────────────┐
│ Model copy   │      │ Model copy   │      │ Model copy   │      │ Model copy   │
│              │      │              │      │              │      │              │
│ Perturb s₁   │      │ Perturb s₂   │      │ Perturb s₃   │      │ Perturb s₄   │
│ Evaluate R₁  │      │ Evaluate R₂  │      │ Evaluate R₃  │      │ Evaluate R₄  │
│ Restore      │      │ Restore      │      │ Restore      │      │ Restore      │
│              │      │              │      │              │      │              │
│ Perturb s₅   │      │ Perturb s₆   │      │ Perturb s₇   │      │ Perturb s₈   │
│ Evaluate R₅  │      │ Evaluate R₆  │      │ Evaluate R₇  │      │ Evaluate R₈  │
│ Restore      │      │ Restore      │      │ Restore      │      │ Restore      │
│    ...       │      │    ...       │      │    ...       │      │    ...       │
└──────┬───────┘      └──────┬───────┘      └──────┬───────┘      └──────┬───────┘
       │                     │                     │                     │
       └─────────────┬───────┴─────────────┬───────┘                     │
                     │                     │                             │
                     └─────── All-gather rewards ────────────────────────┘
                                    │
                                    ▼
                          Normalize + Aggregate Update
                          (decomposed across layers × seeds)

With gpu_threads > 1, each GPU can evaluate multiple perturbations concurrently using separate CUDA streams, further increasing parallelism.

Accelerated Architecture (vLLM)

The accelerated version replaces the standard HuggingFace inference with vLLM engines:

┌─────────────────────────────────────────────────────────────────┐
│                 ACCELERATED ES ARCHITECTURE                     │
│                                                                 │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │              vLLM Inference Engines (per GPU)            │   │
│  │                                                          │   │
│  │  Engine 0 (GPU 0)    Engine 1 (GPU 1)    ...  Engine K   │   │
│  │  ┌──────────────┐    ┌──────────────┐    ┌────────────┐  │   │
│  │  │ Continuous    │    │ Continuous    │    │ Continuous │  │   │
│  │  │ batching     │    │ batching     │    │ batching   │  │   │
│  │  │ PagedAttention│    │ PagedAttention│    │ PagedAttn  │  │   │
│  │  │ KV cache     │    │ KV cache     │    │ KV cache   │  │   │
│  │  └──────────────┘    └──────────────┘    └────────────┘  │   │
│  └──────────────────────────────────────────────────────────┘   │
│                                                                 │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │              Weight Perturbation Manager                 │   │
│  │                                                          │   │
│  │  • Perturbs vLLM model weights in-place                 │   │
│  │  • Regenerates KV cache after perturbation              │   │
│  │  • Coordinates across engines                           │   │
│  └──────────────────────────────────────────────────────────┘   │
│                                                                 │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │              TensorBoard Logging                         │   │
│  │                                                          │   │
│  │  • Training curves (reward, accuracy)                   │   │
│  │  • Per-iteration statistics                             │   │
│  └──────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

The key insight: vLLM's continuous batching and PagedAttention make inference dramatically faster than standard HuggingFace generation, yielding the reported 10× speed-up.

10 Component Breakdown

Core Algorithm Components

What Is Deliberately Excluded

The paper is notable for what it removes from the standard ES toolbox:

"This design choice isolates the core ES algorithm and demonstrates that strong performance can be achieved without auxiliary enhancements. In future work, each individual enhancement can be explored to further improve performance."

This minimalism is a strength of the paper — it demonstrates that raw ES (without engineering tricks) outperforms heavily tuned RL (with tricks and per-model hyperparameter sweeps).

Seven Implementation Innovations

The paper introduces seven implementation modifications that enable billion-parameter ES:

11 Core Mechanisms (Detailed)

Mechanism 1: Natural Evolution Strategies (Simplified)

The theoretical foundation is Natural Evolution Strategies (NES), which views the optimization as operating on the search distribution rather than individual solutions.

Standard NES formulation:

The objective is to maximize the expected reward under a parameterized search distribution (\pi_\psi):

[J(\psi) = \mathbb{E}{\theta \sim \pi\psi}[R(\theta)]]

For a Gaussian search distribution (\pi_\psi = \mathcal{N}(\mu, \Sigma)), the natural gradient update on (\mu) (with fixed (\Sigma = \sigma^2 I)) reduces to:

[\mu \leftarrow \mu + \alpha \cdot \frac{1}{N} \sum_{n=1}^{N} R_n \cdot \varepsilon_n]

where (\varepsilon_n \sim \mathcal{N}(0, I)) and (R_n = R(\mu + \sigma \cdot \varepsilon_n)).

Simplifications in this paper:

Mechanism 2: Memory-Efficient Perturbation via Seed-Based Noise

The most critical engineering innovation is the use of random seeds to represent perturbation noise implicitly:

Traditional approach (infeasible):
  Store N noise vectors, each of dimension d
  Memory: N × d × 4 bytes (float32)
  For d = 8B, N = 30: 30 × 8×10⁹ × 4 = 960 GB  ← IMPOSSIBLE

Seed-based approach (this paper):
  Store N random seeds (integers)
  Memory: N × 8 bytes (int64) = 240 bytes  ← TRIVIAL

  To apply perturbation for seed sₙ:
    rng = torch.Generator().manual_seed(sₙ)
    for each layer ℓ:
      εₗ = torch.randn(layer_ℓ.shape, generator=rng)
      layer_ℓ.data += σ · εₗ
      del εₗ  # only one layer-sized tensor at a time

  To restore:
    rng = torch.Generator().manual_seed(sₙ)  # SAME seed!
    for each layer ℓ:
      εₗ = torch.randn(layer_ℓ.shape, generator=rng)
      layer_ℓ.data -= σ · εₗ  # subtract to restore
      del εₗ

The key insight is that torch.randn with a fixed seed produces identical noise each time. By storing only the seed, the system can regenerate the exact noise for: 1. Applying the perturbation (add σ·ε) 2. Restoring the original parameters (subtract σ·ε) 3. Computing the parameter update (weight by R̃ₙ)

Mechanism 3: Decomposed Parameter Update

The standard update equation requires materializing the full update vector:

Standard: Δθ = α · (1/N) · Σₙ R̃ₙ · εₙ

This requires materializing Σₙ R̃ₙ · εₙ, which is d-dimensional.
For d = 8B: 32 GB in float32.

Decomposed (this paper):
  For each layer ℓ:
    For each seed sₙ:
      εₗ = generate_noise(sₙ, ℓ)
      θₗ += α · (1/N) · R̃ₙ · εₗ
      del εₗ

Peak memory: one layer-sized tensor (max ~2 GB for largest layers)

This decomposition exploits the linearity of addition: the order of summation doesn't matter, so we can accumulate the update layer-by-layer and seed-by-seed, never materializing the full d-dimensional update vector.

Mechanism 4: Greedy Decoding for Deterministic Evaluation

The use of greedy decoding during evaluation is a deliberate methodological choice:

With stochastic decoding (temperature > 0):
  Same model parameters → different responses each time
  Source of variation: parameter perturbation + sampling randomness
  Cannot attribute performance difference to parameter change alone

With greedy decoding (temperature = 0):
  Same model parameters → identical response every time
  Source of variation: parameter perturbation only
  Clean attribution: any performance difference is due to the perturbation

This is analogous to controlled experiments in science — by eliminating one source of variation (decoding randomness), the system can attribute performance differences purely to the parameter perturbation, enabling a cleaner gradient estimate.

Mechanism 5: Z-Score Reward Normalization

Reward normalization is critical for stable training:

Raw rewards (iteration t): [0.0, 0.0, 1.0, 0.0, ..., 1.0]  (binary)
Mean: 0.2, Std: 0.4

Normalized: [-0.5, -0.5, 2.0, -0.5, ..., 2.0]

Effect on update:
  Perturbations that led to correct answers (R̃ > 0) are reinforced
  Perturbations that led to incorrect answers (R̃ < 0) are suppressed
  The magnitude of reinforcement/suppression is proportional to
  how far above/below average the reward is

Without normalization, the update magnitude would depend on the absolute reward scale, which varies across tasks and training stages. Z-score normalization ensures consistent gradient magnitudes, removing a potential source of training instability.

Mechanism 6: Why Small Populations Work

The most surprising finding is that N=30 suffices for billion-parameter spaces. The paper does not provide a full theoretical explanation, but the results suggest several hypotheses:

1. Effective dimensionality is much lower than nominal dimensionality. LLM parameter spaces are highly structured — most directions in parameter space have minimal impact on output. The 30 perturbations may explore the effective subspace efficiently.

2. The pre-trained model provides a strong initialization. ES is not searching from scratch — it is fine-tuning from a pre-trained model that already has the right structure. Small perturbations around this initialization are sufficient to discover improved parameters.

3. Response-level reward provides a strong signal. For binary rewards (correct/incorrect), even a small population can identify which perturbations are beneficial because the reward signal is clear and unambiguous.

4. Greedy decoding amplifies perturbation effects. Without sampling noise, even small parameter changes produce detectable output differences, making each perturbation informative.

12 Programming Language

Implementation Stack

Code Structure

es-fine-tuning-paper/
├── es_fine-tuning_conciseness.py         # Conciseness task (correlated noise)
├── es_fine-tuning_conciseness_iid.py     # Conciseness task (i.i.d. noise)
├── es_fine-tuning_countdown_accl.py      # Countdown (accelerated, vLLM)
├── countdown/
│   ├── es_fine-tuning_countdown.py       # Countdown task (correlated noise)
│   └── es_fine-tuning_countdown_iid.py   # Countdown task (i.i.d. noise)
└── requirement.txt                        # Dependencies

Dependency Analysis

Key dependencies include:

Code Complexity

The implementation is notably compact:

This compactness is a strength — the entire ES fine-tuning pipeline fits in a single readable file, making the algorithm transparent and easy to modify.

Why Python?

Python is the obvious choice for several reasons: 1. PyTorch native. All tensor operations, GPU management, and model manipulation use PyTorch 2. HuggingFace ecosystem. Model loading, tokenization, and the Accelerate library are Python-native 3. vLLM. The accelerated version uses vLLM, which is a Python library 4. Research community. Python is the lingua franca of ML research

The per-iteration overhead of Python is negligible compared to the GPU computation time for inference and noise generation.

13 Memory Management

GPU Memory Analysis

The paper's most significant practical contribution is its memory efficiency. The analysis below compares ES and RL memory requirements for an 8B-parameter model in bf16 precision:

RL Fine-Tuning (PPO) Memory Breakdown:

ES Fine-Tuning Memory Breakdown:

Memory reduction: ~7–8×

Memory-Efficient Operations

Three operations dominate the memory management strategy:

1. In-Place Perturbation:

# Perturb model in-place (layer by layer)
for layer in model.layers:
    noise = torch.randn(layer.shape, generator=rng, device=device)
    layer.data.add_(noise, alpha=sigma)  # in-place!
    del noise  # free immediately

2. In-Place Restoration:

# Restore model in-place (regenerate same noise)
rng.manual_seed(seed)  # reset to same seed
for layer in model.layers:
    noise = torch.randn(layer.shape, generator=rng, device=device)
    layer.data.sub_(noise, alpha=sigma)  # subtract = restore
    del noise

3. Decomposed Update:

# Update in-place (layer by layer, seed by seed)
for layer_idx, layer in enumerate(model.layers):
    for seed_idx, (seed, reward) in enumerate(zip(seeds, normalized_rewards)):
        rng.manual_seed(seed)
        # skip to correct layer
        for skip in range(layer_idx):
            torch.randn(model.layers[skip].shape, generator=rng)
        noise = torch.randn(layer.shape, generator=rng, device=device)
        layer.data.add_(noise, alpha=lr * reward / N)  # in-place!
        del noise

Memory Scaling with Model Size

The memory savings scale better for larger models because RL's gradient and optimizer state overhead grows with parameter count, while ES's overhead remains constant (one layer-sized tensor).

Comparison with Memory-Efficient RL Alternatives

ES achieves the lowest memory footprint while maintaining the highest fine-tuning quality — a combination that no prior method achieved.

14 Continued Learning

Within-Run Learning Dynamics

The ES optimization trajectory exhibits distinctive learning dynamics compared to RL:

RL learning curve (typical):

Accuracy
│        ┌─── plateau / oscillation ───┐
│       /                               \____
│      /                                     \── reward hacking begins
│     /
│    /
│   /  rapid initial improvement
│  /
│ /
└─────────────────────────────────────── Iterations

ES learning curve (typical):

Accuracy
│                         ┌──── continued gradual improvement
│                        /
│                       /
│                      /
│                    /
│                  /
│               /
│            /
│         /
│      /    steady, monotonic improvement
│   /
│  /
│ /
└─────────────────────────────────────── Iterations

ES exhibits slower but more steady improvement without the plateaus, oscillations, or reward hacking that characterize RL fine-tuning. The learning curve is more predictable, enabling better estimation of required compute budgets.

Population Dynamics

Unlike population-based evolutionary systems (AlphaEvolve, OpenEvolve), ES in this paper maintains a single model with a distribution around it:

Iteration t:                  Iteration t+1:
                              (center shifted toward high-reward direction)

  ·   ·                         ·   ·
 · · · ·                       · · · ·
· · θₜ · ·    ────►           · · θₜ₊₁ · ·
 · · · ·                       · · · ·
  ·   ·                         ·   ·

The distribution (cloud) moves through parameter space.
Individual perturbations are transient — only the center persists.

This is fundamentally different from population-based approaches where multiple diverse solutions coexist. The trade-off:

Cross-Task Transfer

The paper does not investigate cross-task transfer — each experiment starts from a pre-trained model and fine-tunes for a single task. However, the results suggest several transfer possibilities:

1. Sequential fine-tuning. An ES-tuned model for one task could serve as the starting point for ES fine-tuning on a second task.
2. Multi-task reward. A composite reward function combining multiple task metrics could enable simultaneous multi-task fine-tuning.
3. Curriculum learning. Starting with an easy task (high reward signal) and progressively adding harder tasks could improve sample efficiency.

None of these are explored in the paper, representing opportunities for future work.

Continued Improvement Beyond Reported Results

The paper's accelerated version (10× speed-up) suggests that the original results may not represent the limit of ES performance. With 10× faster iterations, significantly more iterations become practical, potentially yielding better final performance.

The paper also notes that standard ES enhancements (mirrored sampling, rank transformation, Adam optimizer) were deliberately excluded. Adding these could further improve performance, and the paper explicitly invites this future work.

15 Applications

Primary Application: LLM Post-Training

The paper positions ES as a general-purpose post-training paradigm for LLMs. Current applications demonstrated:

Future Applications Suggested by the Paper

1. RLHF replacement. ES could replace PPO/GRPO in the RLHF pipeline, using human preference reward models but optimizing via ES rather than RL. The reward hacking resistance is particularly valuable for alignment.

2. Instruction following. Fine-tuning models to follow instructions more precisely, where correctness is binary (followed instruction or didn't).

3. Code generation. Fine-tuning for code generation tasks where reward is based on test case passage — a naturally long-horizon, binary-outcome task.

4. Safety alignment. ES's resistance to reward hacking makes it a candidate for safety-critical alignment tasks, where exploiting loopholes in the reward function is a major concern.

5. Distributed fine-tuning. ES's embarrassingly parallel nature makes it ideal for distributed fine-tuning across multiple machines or even multiple data centers. Only scalar rewards need to be communicated, not gradients.

Implications for the Evolutionary AI Field

This paper has several implications for the broader evolutionary AI landscape:

1. Neuroevolution is back. The paper revives direct parameter-space optimization of neural networks — a field that had been dormant since the early 2010s when gradient-based methods became dominant. By showing that ES works at billion-parameter scale, it reopens research directions that were considered closed.

2. Zeroth-order optimization is viable. The success of ES (a zeroth-order method) at scale challenges the assumption that gradient information is necessary for efficient optimization of LLMs. This opens the door to other zeroth-order methods (CMA-ES, random search, simulated annealing) being applied to LLMs.

3. Backpropagation is not always necessary. For post-training (as opposed to pre-training), backpropagation may not be the optimal optimization strategy. ES's ability to avoid backprop has practical benefits (memory, simplicity) without sacrificing quality.

4. Small populations suffice. The N=30 finding challenges the conventional wisdom in evolutionary computation that population size must scale with problem dimensionality. This has implications for all population-based optimization methods applied to high-dimensional problems.

Relevance to OmniEvolve

This paper is highly relevant to the OmniEvolve project from multiple angles:

Limitations

The paper acknowledges several limitations:

1. Scale ceiling unknown. The largest model tested is 8B parameters. Whether ES remains effective at 70B or 405B is an open question.
2. Convergence speed. ES is slower per-iteration than RL for some tasks. The total compute may be higher, even if the per-run cost is lower (due to hyperparameter stability).
3. Exploration radius. With fixed σ, the exploration radius is limited. Adaptive σ (as in CMA-ES) could improve performance but adds complexity.
4. No theoretical guarantees. The paper provides empirical evidence but no convergence guarantees for ES in billion-parameter spaces. The theoretical analysis of Vemula et al. (2019) would predict poor performance, which is empirically falsified but not theoretically explained.
5. Limited to post-training. ES is applied to fine-tuning from pre-trained models, not to pre-training from scratch. Whether ES could scale to pre-training is an open question.

Impact Assessment

Position in the Evolutionary AI Landscape

Parameter Space ─────────────────────────────────────► Action/Code Space
    │                                                        │
    │  ES at Scale              AlphaEvolve / FunSearch       │
    │  (this paper)             (Google DeepMind)             │
    │  ┌───────────┐            ┌─────────────────────┐      │
    │  │ Perturb   │            │ LLM generates code  │      │
    │  │ model     │            │ mutations; evaluator │      │
    │  │ weights   │            │ scores code quality  │      │
    │  │ directly  │            │                     │      │
    │  └───────────┘            └─────────────────────┘      │
    │                                                        │
    │  Sakana Model Merging     EvoPrompting                  │
    │  ┌───────────┐            ┌─────────────────────┐      │
    │  │ Evolve    │            │ Evolve prompts /    │      │
    │  │ merging   │            │ prompt templates    │      │
    │  │ weights   │            │                     │      │
    │  └───────────┘            └─────────────────────┘      │
    │                                                        │
Low-level ◄─────────────────────────────────────── High-level
(continuous)                                      (discrete/symbolic)

ES at Scale occupies the lowest-level, most direct position in this landscape — it evolves the raw parameters of the model, without any abstraction layer (code, prompts, merging weights). This directness is both its strength (no information loss through abstraction) and its limitation (exploration is local in parameter space).