Historical Development: From Genetic Programming to LLM Evolution

3.1 Overview and Motivation

The idea that computational processes might mimic biological evolution to discover novel solutions is older than modern computing itself. From Alan Turing's speculations on machine learning through evolutionary processes (Turing, 1950) to the sophisticated LLM-powered discovery engines of 2025, the field has undergone a series of paradigm shifts — each expanding the space of programs that evolution can realistically explore. This chapter traces that arc, identifying the key ideas, breakthroughs, and bottlenecks at each stage.

Understanding this history is not merely academic. The design choices embedded in modern systems like FunSearch (Romera-Paredes et al., 2024), AlphaEvolve (Google DeepMind, 2025), and OpenEvolve are direct responses to limitations discovered decades earlier. Bloat in tree-based genetic programming motivated parsimony-aware fitness and alternative representations. The brittleness of syntax-tree crossover motivated the move to LLM-based variation. The cost of fitness evaluation motivated cascade architectures and early stopping. Each generation of systems inherits solutions to the previous generation's failures.

A terminological clarification is essential to the argument of this chapter. We distinguish between two aspects of evolutionary program search that are sometimes conflated: representation — the data structure used to encode candidate programs (binary strings, syntax trees, source-code text) — and variation — the operators that generate new candidates from existing ones (bit-flip mutation, subtree crossover, LLM-based generation). These are logically independent: one can apply syntax-blind random mutations to source-code strings, or apply semantically informed operators to syntax trees. The historical development of the field, however, shows that advances in variation mechanisms often co-occurred with shifts in representation, because new variation operators enabled or demanded different encodings.

3.2 The Foundations: Evolutionary Computation (1950s–1980s)

3.2.1 Origins of Evolutionary Optimization

Three independent research threads converged in the mid-twentieth century to establish evolutionary computation as a field. In the United States, Lawrence Fogel introduced evolutionary programming (EP) in the early 1960s, evolving finite-state automata to predict time series (Fogel et al., 1966). In Germany, Ingo Rechenberg and Hans-Paul Schwefel developed evolution strategies (ES) in the mid-1960s, applying mutation and selection to continuous optimization problems in engineering (Rechenberg, 1973; Schwefel, 1977). In the United States, John Holland formalized genetic algorithms (GAs) in the 1970s, introducing the schema theorem and a principled framework for understanding how selection and crossover propagate beneficial building blocks through a population (Holland, 1975).

These early systems shared a common architecture that remains recognizable in modern evolutionary systems:

where $P_t$ is the population at generation $t$, $\text{Vary}$ applies mutation and/or recombination operators, $\text{Evaluate}$ assigns fitness scores, and $\text{Select}$ determines which individuals survive into the next generation. This loop — vary, evaluate, select — is the invariant skeleton of every system discussed in this book.

Holland's schema theorem (Holland, 1975) provided the first theoretical justification for why genetic algorithms work. It showed that short, low-order schemata (partial solution templates) with above-average fitness receive exponentially increasing representation in the population over time. Formally, for a schema $H$ with order $o(H)$ (number of fixed positions) and defining length $\delta(H)$ (distance between outermost fixed positions):

where $m(H, t)$ is the number of instances of schema $H$ in generation $t$, $f(H)$ is the average fitness of individuals matching $H$, $\bar{f}$ is the population average fitness, $p_c$ is the crossover probability, $p_m$ is the per-bit mutation probability, and $l$ is the chromosome length. The key insight: schemas that are short ($\delta(H)$ small), low-order ($o(H)$ small), and above-average fitness ($f(H) > \bar{f}$) are preferentially propagated. The schema theorem was later critiqued for its limited predictive power (Whitley, 1994; Fogel and Ghozeil, 1997), but it remains an influential foundational result that shaped how the community reasoned about evolutionary search.

3.2.2 Representation: The First Bottleneck

Early evolutionary systems operated on fixed-length binary strings or real-valued vectors. This representation was natural for parameter optimization — tuning the weights of a controller, the dimensions of an airfoil — but fundamentally limited the structural complexity of solutions that evolution could discover. A binary GA could optimize the coefficients of a polynomial, but could not discover that a different functional form entirely might be more appropriate.

This limitation was well understood by the 1980s and motivated the central question of the next era: can evolution operate directly on programs?

3.3 The Genetic Programming Era (1990s–2010s)

3.3.1 Koza's Genetic Programming

John Koza's foundational work on genetic programming (GP), published in his 1992 monograph Genetic Programming: On the Programming of Computers by Means of Natural Selection (Koza, 1992), answered the representation question by evolving tree-structured programs in a Lisp-like language. Programs were represented as abstract syntax trees (ASTs), where internal nodes were functions (e.g., +, *, IF) drawn from a predefined function set, and leaf nodes were terminals (constants, variables) from a terminal set.

Crossover in GP was subtree crossover: select a random node in each parent tree, then swap the subtrees rooted at those nodes. Mutation was subtree mutation: replace a randomly selected subtree with a new randomly generated subtree. This meant that variation operators acted on structural, syntactic units rather than on raw bits — though, as we discuss in §3.3.4, these operators remained syntactically motivated rather than semantically informed.

# Illustrative pseudocode: Tree-based genetic programming (Koza-style)
import random
from dataclasses import dataclass

@dataclass
class TreeNode:
    """AST node for a GP individual."""
    symbol: str              # Function or terminal name
    children: list           # Child nodes (empty for terminals)
    is_terminal: bool

def subtree_crossover(parent_a: TreeNode, parent_b: TreeNode) -> TreeNode:
    """Standard GP subtree crossover (Koza, 1992).
    
    Select a random node in each parent; swap the subtrees rooted 
    at those nodes. Returns a single offspring.
    """
    # Deep copy parent_a to create offspring
    offspring = deep_copy(parent_a)
    
    # Select random crossover points
    point_a = random_node(offspring)    # node in offspring tree
    point_b = random_node(parent_b)    # node in donor tree
    
    # Replace subtree at point_a with copy of subtree at point_b
    replace_subtree(offspring, point_a, deep_copy(point_b))
    return offspring

def subtree_mutation(individual: TreeNode, max_depth: int = 4) -> TreeNode:
    """Replace a random subtree with a freshly generated one."""
    mutant = deep_copy(individual)
    point = random_node(mutant)
    new_subtree = generate_random_tree(max_depth=max_depth)
    replace_subtree(mutant, point, new_subtree)
    return mutant

def gp_generation(population: list[TreeNode], fitness_fn) -> list[TreeNode]:
    """One generation of tree-based GP."""
    scored = [(ind, fitness_fn(ind)) for ind in population]
    scored.sort(key=lambda x: x[1], reverse=True)
    
    next_gen = []
    while len(next_gen) < len(population):
        # Tournament selection (size 7, standard Koza setting)
        tournament = random.sample(scored, k=7)
        parent_a = max(tournament, key=lambda x: x[1])[0]
        tournament = random.sample(scored, k=7)
        parent_b = max(tournament, key=lambda x: x[1])[0]
        
        if random.random() < 0.9:  # crossover probability
            child = subtree_crossover(parent_a, parent_b)
        else:
            child = subtree_mutation(parent_a)
        next_gen.append(child)
    
    return next_gen

Koza demonstrated GP on a range of symbolic regression, classification, and control problems, and later extended the approach to evolving electronic circuits, controllers, and game-playing strategies (Koza, 1994; Koza et al., 1999). The approach was genuinely powerful: evolution could discover the structure of solutions, not just their parameters.

3.3.2 The Bloat Problem

However, GP suffered from a persistent pathology: bloat. Trees grew progressively larger over generations without corresponding fitness improvement. Subtree crossover was widely identified as a primary contributor — swapping random subtrees tended to insert non-functional code (introns) that did not affect output but inflated tree size (Langdon and Poli, 1997; Banzhaf et al., 1998). Multiple theories have been proposed to explain bloat, including the defense against crossover hypothesis (Nordin et al., 1995), the removal bias theory (Soule and Foster, 1998), and the fitness-causes-bloat theory (Langdon and Poli, 2002). The phenomenon remains an active area of study, and no single theory fully accounts for all observed behavior. As a heuristic characterization (not a formal theorem), if $s_t$ denotes average program size at generation $t$:

Note: This inequality is a commonly observed empirical tendency, not a formally proven result. The conditions under which bloat occurs and its precise growth rate depend on the fitness landscape, selection pressure, and operator design. See Poli et al. (2008) for a comprehensive survey of bloat theories.

Multiple mitigation strategies were proposed: parsimony pressure (penalizing large trees in the fitness function), depth limits, size-fair crossover (Langdon, 2000), and Occam's razor-inspired multi-objective approaches trading fitness against complexity (Luke and Panait, 2006). These reduced bloat's severity but did not eliminate it. Bloat remained a tax on GP's scalability, limiting practical application to relatively small programs — typically dozens to hundreds of AST nodes.

3.3.3 Grammar-Guided and Strongly Typed GP

A second major limitation was closure: standard GP required all functions to accept and return the same type, which prevented evolving programs in real-world languages with type systems, control flow, and side effects. Responses included:

• Strongly typed GP (Montana, 1995): associating types with function arguments and return values, constraining crossover to type-compatible nodes.
• Grammar-guided GP (Whigham, 1995; McKay et al., 2010): defining a context-free grammar (CFG) and restricting evolution to derivation trees of that grammar, guaranteeing syntactic validity.
• Grammatical evolution (O'Neill and Ryan, 2001): encoding programs as integer vectors that index grammar production rules, enabling standard GA operators on a linear genome while producing structured programs.

These approaches improved the syntactic validity of evolved programs but introduced their own constraints. Grammar-guided GP required manually specifying a grammar that was expressive enough to contain good solutions but restrictive enough to make search tractable — a design tension that foreshadowed the prompt engineering challenges of the LLM era.

3.3.4 Alternative GP Representations

The tree-based representation introduced by Koza was not the only approach to evolving programs. Several important alternative representations emerged during this era, each addressing different aspects of the representation challenge:

Linear Genetic Programming. Linear GP represented programs as sequences of register-machine instructions rather than syntax trees (Brameier and Banzhaf, 2007). This representation avoided some bloat pathologies of tree-based GP and produced programs that mapped more naturally to imperative machine code. Linear GP was particularly successful in classification and symbolic regression tasks, and its instruction-level granularity offered a different locality profile than tree-based operators.

Cartesian Genetic Programming. Cartesian GP (CGP), introduced by Miller (1999) and systematically developed by Miller and Thomson (2000), represented programs as directed acyclic graphs indexed by Cartesian coordinates. CGP used a fixed-size genome with many potentially inactive (non-coding) genes, which served as a natural form of genetic drift and neutral evolution. CGP proved particularly effective for evolving digital circuits and image processing pipelines.

Stack-Based GP and PushGP. The Push language and PushGP system (Spector, 2001; Spector and Robinson, 2002) took a fundamentally different approach by evolving programs in a stack-based language with multiple typed stacks. Push's permissive execution model — where type mismatches resulted in no-ops rather than errors — eliminated the closure problem entirely and enabled evolution of programs with complex control flow including recursion and iteration. PushGP demonstrated that language design and variation operator design were deeply intertwined concerns.

ADATE and Higher-Order Evolution. Olsson's ADATE system (Olsson, 1995) evolved programs in a strongly-typed functional language using semantically motivated transformations including function composition, abstraction, and case distinction. ADATE's operators were more structured than random subtree swaps, representing an early attempt at semantically informed variation — a theme that would become central in the LLM era.

These diverse representations illustrate that the field's development was not a single linear progression from trees to source code, but rather a branching exploration of the design space. Each approach offered different tradeoffs between expressiveness, search efficiency, and the quality of variation operators — the same fundamental tensions that persist in LLM-powered systems today.

3.3.5 Limits of Syntax-Level Variation

By the 2010s, the GP community had accumulated two decades of experience with the fundamental tradeoffs of symbolic program evolution. The core limitation was clear: variation operators that act on syntax rarely produce semantically meaningful changes. Swapping a subtree that computes a loop counter with one that computes a conditional guard produces syntactically valid but semantically nonsensical code. The search space was vast, the fraction of meaningful mutations was small, and scaling to programs longer than a few hundred lines remained impractical.

It is important to be precise about this limitation. The bottleneck was not primarily about representation — tree, graph, and linear representations could all encode complex programs. The bottleneck was about variation quality: the operators available to generate new candidate programs from existing ones were syntax-level transformations with no model of what the code meant. This distinction matters for understanding the LLM-era breakthrough (§3.5), which primarily improved the variation mechanism rather than the representation.

3.4 Neural Program Synthesis and Learned Heuristics (2010s–2020s)

3.4.1 Neural Architecture Search

The deep learning revolution of the 2010s introduced a different approach to automated program and algorithm discovery: neural program synthesis. Rather than evolving symbolic programs directly, these methods used neural networks to learn to generate programs or to parameterize search over architectural spaces.

Neural Architecture Search (NAS), introduced by Zoph and Le (2017), used a recurrent neural network (the controller) trained with reinforcement learning to generate descriptions of neural network architectures. The controller was rewarded based on the validation accuracy of the generated architecture after training. This was, in a meaningful sense, a hybrid evolutionary–neural approach: the controller learned a distribution over architectures, and REINFORCE-style updates played the role of selection pressure.

NAS and its successors — including ENAS (Pham et al., 2018), DARTS (Liu et al., 2019), and AmoebaNet (Real et al., 2019) — demonstrated that automated search could discover architectures competitive with or superior to human-designed ones. AmoebaNet explicitly used evolutionary methods — tournament selection with aging — to search the architecture space, achieving state-of-the-art results on ImageNet. This confirmed that evolutionary approaches remained competitive with gradient-based alternatives even in the neural era.

3.4.2 Learning to Optimize

A parallel thread explored using neural networks to learn heuristics for combinatorial optimization. Vinyals et al. (2015) introduced the Pointer Network, which learned to output permutations for problems like the Traveling Salesman Problem (TSP). Bello et al. (2017) trained Pointer Networks using reinforcement learning, producing heuristics that outperformed some classical approximation algorithms on small instances.

This "learning to optimize" paradigm was powerful but limited in a crucial way: the learned heuristics were implicit — encoded in neural network weights rather than expressed as interpretable algorithms. A neural TSP solver could produce good tours but could not explain its strategy or transfer its approach to a human researcher. The programs were opaque.

3.4.3 Program Synthesis with Neural Guidance

A third thread attempted to bridge the gap between neural and symbolic approaches. DeepCoder (Balog et al., 2017) used a neural network to predict which domain-specific language (DSL) functions were likely needed for a given input-output specification, then used this prediction to guide an enumerative search over programs. FlashFill (Gulwani, 2011), while not neural, demonstrated the power of program synthesis in constrained domains and influenced later neural-guided approaches.

These systems worked well within their DSLs but could not generate programs in general-purpose languages. The domain-specific language was both the enabler (constraining search to manageable spaces) and the limitation (preventing discovery of solutions outside the DSL's expressiveness).

3.4.4 Evolutionary Program Repair and Neural-Symbolic Bridges

Several important research threads bridged the gap between the GP era and the LLM era but are sometimes overlooked in teleological accounts of the field's development:

Automated program repair. GenProg (Le Goues et al., 2012) applied GP-style operators (statement-level insertion, deletion, and replacement) to repair bugs in real-world C programs, guided by test suites as fitness functions. GenProg demonstrated that evolutionary methods could operate on real programs in general-purpose languages — a significant advance beyond the toy domains of classical GP. However, GenProg's operators remained syntax-level: they copied and transplanted code fragments without understanding their semantics, which limited repair quality and led to patches that passed tests but were sometimes semantically questionable (Qi et al., 2015). The automated program repair community (including later systems such as Prophet (Long and Rinard, 2016) and SapFix (Marginean et al., 2019)) explored progressively more informed repair operators, creating a lineage that connects directly to the LLM-as-repair-agent paradigm of the 2020s.

Neural-symbolic program induction. DreamCoder (Ellis et al., 2021) combined neural-guided search with library learning: a wake-sleep algorithm iteratively solved synthesis tasks in a DSL, then abstracted reusable program fragments into a growing library. DreamCoder's library learning mechanism can be seen as a precursor to the knowledge accumulation and cross-run learning that remains an open challenge in LLM-evolution systems (§3.11). The system demonstrated that building up reusable abstractions over evolutionary time was feasible and beneficial, though it remained limited to small, domain-specific languages.

Quality-diversity and novelty search. The MAP-Elites algorithm (Mouret and Clune, 2015) and novelty search (Lehman and Stanley, 2011) introduced a paradigm shift within evolutionary computation itself: rather than optimizing a single fitness function, these approaches maintained archives of diverse, high-performing solutions across multiple behavioral dimensions. Quality-diversity methods proved central to the LLM-evolution paradigm — FunSearch's island model and AlphaEvolve's MAP-Elites-style archive are direct descendants of this thread.

3.4.5 The Gap That Remained

By the early 2020s, the landscape of automated program discovery contained a clear gap:

Approach	Representation	Variation Method	Interpretable?	General-Purpose?	Scalable?
Genetic Programming	AST / grammar	Syntax-blind operators	Yes	Limited	No (bloat)
NAS / AmoebaNet	Architecture tokens	Evolutionary / RL	Partially	No (architectures only)	Yes
Neural optimization	NN weights	Gradient / RL	No	No (task-specific)	Yes
DSL synthesis	DSL programs	Enumeration + neural	Yes	No (DSL-bound)	Limited
Program repair	Source code	Statement-level transplant	Yes	Yes (C, Java)	Limited (patch quality)
Desired	General code	Semantically informed	Yes	Yes	Yes

No existing approach combined interpretable, general-purpose programs with semantically informed variation at scale. The missing ingredient was a variation operator that could manipulate code in a semantically coherent way — understanding what code means, not just how it is structured syntactically. Large language models would provide exactly this.

3.5 The LLM Breakthrough: Language Models as Variation Operators

3.5.1 The Key Insight

The emergence of large language models capable of generating and understanding code — beginning with Codex (Chen et al., 2021) and accelerating through GPT-4 (OpenAI, 2023), Claude (Anthropic, 2024), Gemini (Google DeepMind, 2024), and their successors — provided the missing variation operator that had constrained evolutionary program search for three decades. The fundamental insight can be stated precisely:

An LLM trained on a large code corpus serves as a powerful variation operator — an implicit model of the distribution over semantically plausible program transformations. When used within an evolutionary loop, it replaces syntax-blind subtree swaps with semantically informed edits: refactoring a loop into a vectorized operation, replacing a greedy heuristic with a dynamic programming approach, or restructuring a data structure for better cache locality. The LLM "understands" — in an operational sense — what the code does, and can propose modifications that are far more likely to preserve or extend functionality than random syntactic transformations.

It is important to distinguish this from a claim about representation. Source code in general-purpose languages (Python, C++) was always a possible representation for evolutionary search — automated program repair systems like GenProg (Le Goues et al., 2012) had used it. The problem was that no variation operator could manipulate source code effectively at scale. LLMs solved the variation problem, and this secondarily made source-code representations practical for evolutionary search over complex programs. The representation change (from ASTs to text) was a consequence of the variation breakthrough, not the other way around.

This reframes the role of the LLM within the evolutionary loop. Rather than treating the language model as a standalone problem solver (which it is, but imperfectly), the LLM-evolution paradigm treats it as a generator of variation — one component in a larger system where selection pressure, population diversity, and fitness evaluation provide the corrective feedback that the LLM alone lacks.

where $x$ is a candidate program, $\text{prompt}$ includes task description and mutation instructions, and $\text{context}$ may include fitness history, related programs from the population, and error traces from previous evaluations.

3.5.2 Why LLMs Succeed Where Syntax Operators Fail

To understand why this substitution is so effective, consider the probability that a random variation produces a fitness improvement. In classical GP with subtree crossover, this probability is governed by the locality of the representation: small genotypic changes should produce small phenotypic changes. AST representations have notoriously poor locality — a single node swap can transform a functioning sort algorithm into nonsense (Galván-López et al., 2011).

LLMs, by contrast, have learned a distribution over meaningful code transformations from millions of examples. When asked to "improve the efficiency of this function," an LLM is far more likely to propose a semantically coherent edit — changing an $O(n^2)$ loop to an $O(n \log n)$ algorithm, for instance — than a random subtree swap. We can express this intuition heuristically as:

Note: This inequality is a heuristic characterization of empirically observed behavior, not a formal result. Making it precise would require specifying the fitness landscape, the LLM's training distribution, the GP operator design, and the program space. The claim is supported empirically by the dramatically higher search efficiency observed in LLM-evolution systems compared to classical GP on comparable tasks (e.g., Romera-Paredes et al., 2024; Lehman et al., 2023), but a formal treatment remains an open problem (see §3.11).

where $\Delta f = f(x') - f(x)$ is the fitness change from parent $x$ to child $x'$. This higher rate of beneficial mutations dramatically improves search efficiency, enabling evolution to discover complex programs that would require astronomical numbers of random syntactic mutations.

# Illustrative pseudocode: LLM as mutation operator
# Contrast with the GP subtree_mutation shown earlier

def llm_mutate(
    program: str,
    fitness_history: list[float],
    error_trace: str | None,
    population_sample: list[str],
    task_description: str,
    llm_client,  # API client for the language model
) -> str:
    """Use an LLM to generate a semantically informed mutation.
    
    Unlike subtree_mutation which randomly replaces code fragments,
    this operator provides the LLM with rich context about the
    program's behavior, enabling targeted, meaningful edits.
    """
    prompt = f"""You are improving a program for the following task:
{task_description}

Current program:
```python
{program}
```

Fitness history (recent scores): {fitness_history[-5:]}
{"Error from last evaluation: " + error_trace if error_trace else ""}

For inspiration, here are other programs from the population:
{format_samples(population_sample[:3])}

Produce an improved version of the program. Focus on algorithmic 
improvements, not cosmetic changes. Return only the code.
"""
    response = llm_client.generate(prompt, temperature=0.8)
    return extract_code(response)

3.5.3 Early Explorations (2022–2023)

Before FunSearch formalized the paradigm, several research groups independently explored using LLMs within evolutionary or iterative improvement loops:

• ELM — Evolution through Large Models (Lehman et al., 2023): one of the first papers to explicitly frame LLMs as mutation operators within an evolutionary algorithm, demonstrating evolution of Sodarace agents with LLM-generated code variations. ELM introduced the concept of a "MAP-Elites + LLM" combination that would become central to later systems. The paper was posted as a preprint in June 2023.
• LLM-guided program repair: multiple groups showed that LLMs could fix bugs in programs when given failing test cases (Xia et al., 2023; Jiang et al., 2023), effectively performing a fitness-guided mutation — the simplest case of the evolutionary pattern.
• Self-refine and Reflexion (Madaan et al., 2023; Shinn et al., 2023): iterative self-improvement loops where an LLM critiques and revises its own output. While not framed as evolutionary, these systems implemented the core vary-evaluate-select loop with a population size of one.

These explorations confirmed the viability of the approach but lacked the systematic population management, diversity mechanisms, and rigorous evaluation infrastructure that would characterize the paradigm's mature systems.

3.6 FunSearch: The Paradigm Crystallizes (December 2023)

3.6.1 Architecture and Design

FunSearch, published by Romera-Paredes et al. at Google DeepMind (preprint December 2023; Nature, January 2024), was the first system to convincingly demonstrate that LLM-powered evolution could discover genuinely novel mathematical and algorithmic results — solutions not previously known to human researchers. Its significance extends beyond its specific results: FunSearch crystallized the LLM-evolution paradigm into a clear, reproducible architecture that all subsequent systems would build upon.

The core architecture of FunSearch, as described in the paper, consisted of four components:

1. Programs database: a structured collection of programs organized into separate clusters (termed "islands" in the paper) to maintain diversity. Within each island, programs were stored with their associated scores.
2. Sampler: selected programs from the database to construct prompts for the LLM. The paper describes sampling "best" programs — programs with high scores — and combining them into few-shot prompts. The exact selection distribution was described as biasing toward higher-scoring programs.
3. LLM backbone: a pre-trained code-generating model (Codey, based on PaLM 2; Anil et al., 2023) used as the variation operator — generating new candidate functions conditioned on selected programs and the task specification.
4. Evaluator: an automated assessment pipeline that executed generated programs and scored them against the task's objective function, with programs that crashed or produced invalid output receiving the lowest possible score.

3.6.2 Key Design Decisions

Several design decisions in FunSearch, as reported in the paper, proved foundational for the paradigm:

Evolving functions, not full programs. FunSearch constrained the search space by evolving a single function (the "priority function" or "heuristic") within a fixed program skeleton. The skeleton handled input parsing, data structures, and the evaluation loop; only the core algorithmic logic was mutable. This dramatically reduced the search space while preserving the ability to discover novel algorithms. This "skeleton + evolve-block" pattern has been adopted by nearly every subsequent system (Romera-Paredes et al., 2024, §Methods).

Island-based population structure. The programs database was partitioned into independent clusters called "islands." The paper describes a mechanism to maintain diversity: the lowest-scoring island is periodically reset with copies of programs from the highest-scoring island, preventing the entire population from converging prematurely. This is related to, but distinct from, the classical island-model migration in parallel evolutionary computation (Whitley et al., 1999); the FunSearch mechanism emphasizes periodic resets of under-performing subpopulations rather than gradual individual migration between subpopulations.

Score-biased parent selection. The sampler selected programs with a bias toward higher-scoring individuals when constructing prompts for the LLM. The paper describes this as sampling "best" programs to provide as few-shot examples, with some form of score-proportional selection to balance exploitation of known good solutions against exploration through the stochastic variation of LLM generation.

Automated, sandboxed evaluation. Every generated program was executed in a controlled environment with resource limits (time, memory). Programs that crashed, timed out, or produced invalid output received minimal scores. This automated evaluation was critical for running evolution at scale without human intervention.

Provenance note: The descriptions above are based on the published paper (Romera-Paredes et al., Nature, 2024) and its supplementary materials. FunSearch's source code has not been publicly released. Some implementation details — such as exact island counts, selection temperature parameters, and the precise reset schedule — are not fully specified in the paper. Descriptions of these details in secondary sources or open-source reimplementations (e.g., OpenEvolve) may differ from the original system.

3.6.3 Results and Significance

FunSearch's flagship results, as reported in the paper, demonstrated the paradigm's potential:

• Cap sets: FunSearch discovered new constructions for the cap set problem in additive combinatorics, finding large cap sets in $\mathbb{F}_3^n$ that exceeded previously known constructions for specific dimensions ($n = 8$). The paper reports that these constructions were verified by domain experts and constituted genuinely new mathematical knowledge (Romera-Paredes et al., 2024, Extended Data Table 1).
• Online bin packing: FunSearch evolved heuristics for online bin packing that outperformed established baselines including Best Fit and First Fit on standard benchmark distributions (Romera-Paredes et al., 2024, Figure 4).

The cap set result was particularly significant because it crossed the threshold from "automated optimization" to "automated discovery." The evolved program did not merely tune parameters of a known algorithm — it discovered a structurally novel construction strategy that human mathematicians had not previously identified.

3.6.4 What FunSearch Did Not Solve

FunSearch also revealed the paradigm's open challenges, as noted by the authors and subsequent commentary:

• Computational cost: FunSearch required millions of LLM calls over extended computation periods, leveraging substantial compute infrastructure at Google DeepMind. The paper reports experiments running for multiple days with large numbers of parallel samplers.
• Narrow evaluation scope: the system required a well-defined, automatically computable fitness function — limiting application to domains with clear quantitative objectives.
• Single-function evolution: evolving only a single function within a fixed skeleton limited the structural complexity of discoverable solutions.
• No learning across runs: each FunSearch run started from scratch, with no transfer of knowledge from previous evolutionary runs.

These limitations defined the research agenda for the systems that followed.

3.7 The Paradigm Matures: From Proof of Concept to Research Platform (2024–2026)

3.7.1 Rapid Proliferation

FunSearch's publication triggered a rapid proliferation of LLM-evolution systems. Within eighteen months, the field saw the emergence of multiple independent systems, each addressing different subsets of FunSearch's limitations. A survey conducted for this book identified at least 17 distinct systems or significant extensions published or released between late 2023 and early 2026 (see the system catalog in Chapter 2, Table 2.X for the full list and classification).

This proliferation was enabled by three converging factors:

1. Commodity LLM APIs: the widespread availability of capable code-generation models through commercial APIs (OpenAI, Anthropic, Google) and open-weight models (Code Llama (Rozière et al., 2023), DeepSeek Coder (Guo et al., 2024), Qwen-Coder (Bai et al., 2023)) lowered the barrier to entry.
2. Established evaluation infrastructure: benchmark suites from competitive programming, mathematics, and algorithm design provided ready-made fitness functions.
3. Clear architectural template: FunSearch established a modular architecture — population management, LLM variation, sandboxed evaluation, selection — that researchers could implement and extend independently.

3.7.2 Major Systems and Their Contributions

The systems that emerged during Era V (2024–2026) can be organized by their primary contribution to the paradigm. The following table lists representative systems with citations; Chapters 4–9 examine each in detail:

System	Year	Primary Contribution	Origin	Source
FunSearch	2023	Paradigm definition; mathematical discovery	Google DeepMind	Romera-Paredes et al., 2024
ELM	2023	MAP-Elites + LLM; open-ended evolution	Lehman et al.	Lehman et al., 2023
ReEvo	2024	Reflective evolution; self-critique mutation	Research	Ye et al., 2024
EoH	2024	Evolution of Heuristics; combinatorial optimization	Research	Liu et al., 2024a
AlphaEvolve	2025	Industrial scale; multi-domain; ensemble evaluation	Google DeepMind	Novikov et al., 2025
OpenEvolve	2025	Open-source FunSearch/AlphaEvolve-style platform	Community	GitHub repository
GEPA	2025	Multi-objective; ASI scoring; generalization	Research	Unpublished / preprint

Each of these systems is examined in detail in subsequent chapters. The trajectory is clear: from FunSearch's proof of concept, through academic extensions adding diversity mechanisms, multi-objective optimization, and reflective operators, to industrial-scale systems (AlphaEvolve) and community platforms (OpenEvolve) that made the paradigm accessible to practitioners.

3.7.3 AlphaEvolve: Industrial Scale

Google DeepMind's AlphaEvolve (Novikov et al., 2025), announced in May 2025, represented the paradigm's maturation to industrial scale. AlphaEvolve introduced several advances over FunSearch, as described in its technical report: ensemble model pools combining multiple LLMs of different sizes and capabilities; a cascade evaluation pipeline that filters candidates through progressively more expensive stages; application across diverse domains including mathematics, algorithm optimization, and hardware design; and integration into production systems at Google (with reported improvements to data center scheduling and chip design workflows).

AlphaEvolve demonstrated that LLM-powered evolution was not merely a research curiosity but a practical engineering methodology capable of discovering improvements to real-world systems at scale. Chapter 4 examines AlphaEvolve's architecture and results in detail.

3.7.4 The Open-Source Response

AlphaEvolve's publication as a system without public code release catalyzed the open-source community. OpenEvolve (Chapter 5) provided an implementation of the FunSearch/AlphaEvolve-style architecture with support for multiple LLM providers. GEPA (Chapter 7) added multi-objective optimization and advanced scoring. Several lighter-weight frameworks emerged for specific domains — LLM4AD (Chapter 8) provided a unified platform integrating multiple methods, and domain-specific tools appeared for combinatorial optimization, competitive programming, and scientific computing.

This open-source response was critical for the field's development: it enabled independent reproduction, ablation studies, and extensions that would not have been possible with closed systems alone.

3.8 Anatomy of the Paradigm Shift

3.8.1 What Changed and What Stayed

The LLM-evolution paradigm preserved the fundamental evolutionary loop — vary, evaluate, select — while primarily replacing the variation operator. This is worth stating precisely, because it clarifies both the novelty and the continuity of the approach:

# Illustrative pseudocode: The invariant evolutionary skeleton
# This structure is shared across ALL systems in this survey

def evolutionary_loop(
    population: Population,
    vary: Callable,          # GP: subtree ops  →  LLM era: language model
    evaluate: Callable,      # Domain-specific fitness function (unchanged)
    select: Callable,        # Tournament, MAP-Elites, Pareto (refinements, not revolution)
    max_generations: int,
) -> Program:
    """The core loop is identical across eras.
    
    What changed between GP and LLM evolution:
    - vary():    random syntax transforms  →  semantically informed LLM edits
    - evaluate(): simple scoring           →  cascade pipelines, sandboxes
    - select():  tournament/roulette       →  MAP-Elites, islands, bandits
    - population: ASTs                     →  source code strings
    
    The primary shift is in vary(). The representation change from ASTs
    to source-code strings is a consequence: LLMs natively operate on 
    text, making source code the natural representation. But it was the 
    variation quality, not the encoding, that was the bottleneck.
    """
    for generation in range(max_generations):
        parents = select(population)
        children = [vary(parent) for parent in parents]  # THE key change
        scored_children = [(c, evaluate(c)) for c in children]
        population = update(population, scored_children)
    
    return population.best()

The continuity is important: LLM-powered evolution is not a new paradigm in the Kuhnian sense of replacing the previous framework. It is an augmentation — replacing the weakest link (syntax-blind variation) with a dramatically more capable component (semantically informed variation via LLM), while preserving the population-based search, selection pressure, and diversity mechanisms that evolutionary computation developed over decades. The representation changed too — from ASTs to source-code text — but this was a downstream consequence of the variation operator's requirements: LLMs natively produce and consume text, making source-code strings the natural encoding. Previous systems like GenProg had already demonstrated that source-code representations were viable; what was missing was a variation operator capable of exploiting them effectively.

3.8.2 Five Dimensions of the Shift

The transition from classical GP to LLM-powered evolution can be characterized along five dimensions:

Dimension	Classical GP (1990s–2010s)	LLM-Powered Evolution (2023–present)
Variation mechanism (primary change)	Syntax-blind operators (subtree crossover/mutation)	Semantically informed LLM-based generation
Representation (secondary change)	Abstract syntax trees; domain-specific grammars	Source code in general-purpose languages (Python, C++)
Scale	Small programs (tens to hundreds of AST nodes)	Full functions and modules (hundreds of lines)
Cost model	CPU time for fitness evaluation	API cost per LLM call + CPU for evaluation
Knowledge source	Random initialization; domain-engineered primitives	Pre-trained on billions of lines of human code

We deliberately list variation mechanism first and representation second to reflect the causal structure of the transition: the LLM provided a new variation capability, and the representation shifted to accommodate it. The last dimension — knowledge source — is perhaps the most transformative in practice. Classical GP started from near-zero domain knowledge (a function set and terminal set chosen by the researcher). LLM-powered evolution starts from the accumulated programming knowledge of humanity, encoded in the language model's weights. This changes the nature of the search from exploration of a vast, mostly barren landscape to refinement within a much richer subspace of plausible programs.

3.8.3 New Challenges Introduced

The paradigm shift also introduced challenges that did not exist (or were less severe) in classical GP:

• Cost management: each LLM call incurs monetary cost (for API models) or compute cost (for self-hosted models), making the cost-per-variation orders of magnitude higher than classical GP. This motivated cascade evaluation architectures that filter cheap-to-evaluate failures before expensive full evaluation (Novikov et al., 2025).
• Model dependency: the quality of evolution depends on the capability of the underlying LLM, which is controlled by external providers and may change over time. Reproducibility becomes challenging when the variation operator itself is a moving target (see Chapter 10 for a detailed discussion of reproducibility concerns).
• Prompt sensitivity: the variation operator's behavior is controlled by natural-language prompts, introducing a new design surface (prompt engineering) that lacks the formal guarantees of mathematical operators.
• Evaluation bottleneck: with LLMs generating plausible-looking code at high rates, the evaluation pipeline — not the variation operator — becomes the throughput bottleneck. Sandboxing, caching, and early stopping become critical infrastructure.

3.9 A Consolidated Timeline

The following timeline synthesizes the major milestones discussed in this chapter, organized by the five-era taxonomy used throughout. Dates refer to publication or first public availability; internal development often preceded publication by months or years. Citations are provided for each entry to support attribution.

3.10 Theoretical Perspective: Why the Combination Works

3.10.1 Complementary Strengths

The effectiveness of the LLM-evolution combination can be understood through a complementarity argument. LLMs and evolutionary algorithms have complementary strengths and weaknesses:

Capability	LLM Alone	Evolution Alone (GP)	LLM + Evolution
Semantic code understanding	Strong	None	Strong (via LLM)
Systematic exploration	Weak (mode collapse)	Strong	Strong (via population)
Novelty beyond training data	Weak (interpolation)	Strong (random search)	Strong (guided extrapolation)
Correctness verification	Weak (hallucination)	Strong (fitness testing)	Strong (via evaluation)
Efficiency on small budgets	Strong (few-shot)	Weak (many evaluations)	Moderate
Scaling to long horizons	Weak (context limits)	Strong	Strong (via population memory)

The LLM provides informed variation — mutations that are likely to be syntactically valid and semantically meaningful. The evolutionary framework provides directed search — selection pressure, population diversity, and systematic exploration that the LLM alone cannot achieve. Together, they overcome the limitations of each approach in isolation.

3.10.2 A Formal View

More formally, we can sketch a framework — presented here as an intuitive formalization rather than a rigorous theory — for understanding why the combination is effective. Consider the search process as sampling from a sequence of improving distributions over the program space $\mathcal{P}$. At generation $t$, the population implicitly defines a distribution $\pi_t$ over programs. The variation operator transforms this into a proposal distribution $q_t$ from which children are drawn, and selection sharpens $q_t$ into $\pi_{t+1}$.

In classical GP, the proposal distribution $q_t^{\text{GP}}$ is determined by syntax-level operators and tends to have high entropy (many proposals are far from the parents in semantic space) and low probability mass on improvements. In LLM-powered evolution:

where $\text{context}_t$ includes the task description, fitness history, and potentially other population members. Because the LLM has learned a distribution over meaningful code transformations, $q_t^{\text{LLM}}$ concentrates more mass on semantically coherent programs, improving the hit rate of beneficial mutations.

Note: This distributional framing is an intuitive formalization, not a proven theorem. Making it rigorous would require specifying the program space $\mathcal{P}$, the topology under which $q_t$ concentrates mass, and convergence conditions for the population sequence $\{\pi_t\}$. The framework connects to theoretical work on estimation-of-distribution algorithms (EDAs) (Larrañaga and Lozano, 2001) and distribution-guided evolutionary search (Hauschild and Pelikan, 2011), but a formal treatment specific to LLM-powered evolution remains an open problem. See §3.11 for further discussion of theoretical gaps.

The evolutionary framework then ensures that even when the LLM proposes suboptimal variations (as it frequently does), selection discards them, and the population retains the best discoveries across all generations. The population serves as an external memory that the LLM lacks — accumulating improvements over timescales far longer than the LLM's context window.

3.11 Open Questions and Future Directions

The historical arc traced in this chapter points to several open questions that define the current frontier:

Theoretical foundations. While the empirical success of LLM-powered evolution is clear, theoretical understanding remains limited. Under what conditions does the LLM-evolution loop converge? What are the sample complexity bounds? How does the quality of the LLM (as measured, e.g., by pass@k on code generation benchmarks) affect the convergence rate of the evolutionary search? Classical GP had the schema theorem (Holland, 1975) and subsequent theoretical work on GP convergence (Poli et al., 2008). The LLM-evolution paradigm lacks comparable theoretical grounding — the distributional framework sketched in §3.10.2 provides intuition but not formal guarantees.

Representation evolution. Current systems evolve code within a fixed skeleton or function signature. Can the representation itself — the skeleton, the interfaces, the data structures — co-evolve with the solution? This would reconnect with the original ambition of GP: evolving not just parameters or functions, but entire program architectures. The PushGP tradition (Spector, 2001) and DreamCoder's library learning (Ellis et al., 2021) provide partial precedents.

Cross-run learning. Most current systems start each evolutionary run from scratch. Can knowledge from previous runs — successful mutation patterns, promising solution structures, dead-end avoidance — be transferred to accelerate future runs? This connects to meta-learning and curriculum learning research. GEPA's knowledge mechanisms (Chapter 7) and the "skills database" concept in some systems represent early steps in this direction.

Multi-agent and co-evolutionary approaches. Current systems use a single LLM (or a pool of interchangeable LLMs) as the variation operator. Co-evolutionary approaches — where separate populations of programs, test cases, and even prompts evolve simultaneously — are a natural extension that remains largely unexplored. The prompt co-evolution mechanisms in some systems (Chapter 6) represent initial explorations of this idea.

Scaling laws. Do LLM-evolution systems exhibit predictable scaling behavior as compute budget, population size, or LLM capability increases? Understanding these relationships would enable principled resource allocation and system design. Early empirical evidence from AlphaEvolve (Novikov et al., 2025) suggests that larger model ensembles and longer runs consistently improve outcomes, but systematic scaling studies remain to be conducted.

3.12 Summary