DiscoGen

27.1 Overview & Motivation

Automated Algorithm Discovery (AAD)—the use of AI systems to discover novel machine learning algorithms—has emerged as a rapidly growing subfield at the intersection of program synthesis, evolutionary computation, and large language model (LLM) reasoning. Systems such as FunSearch (Romera-Paredes et al., 2024), AlphaEvolve (Google DeepMind, 2025), and OpenELM (Lehman et al., 2024) have demonstrated that LLMs can discover novel optimizers, loss functions, and training procedures through iterative code generation and evaluation. However, the evaluation infrastructure for comparing and optimizing these Algorithm Discovery Agents (ADAs) has not kept pace with the systems themselves.

DiscoGen, introduced by Goldie et al. (March 2026), addresses this infrastructure gap by reframing algorithm discovery evaluation as a procedural generation problem [PAPER §1]. Rather than providing a fixed suite of benchmark tasks, DiscoGen generates parameterized algorithm discovery tasks spanning 14 machine learning domains. The authors report that the combinatorial task space exceeds 99.3 billion unique configurations [PAPER §3, Table 1], though the practical utility of this number depends on how meaningfully distinct these configurations are (discussed in §27.7).

The work identifies five specific deficiencies in existing evaluation practice for ADAs [PAPER §1]:

1. Tiny evaluation suites — existing benchmarks contain tens of tasks, enabling overfitting and unreliable comparisons.
2. No meta-train/meta-test separation — most suites evaluate discovered algorithms on the same datasets used during discovery.
3. Data contamination risk — static task sets may appear in LLM training corpora.
4. Saturated problems — many tasks are solved or nearly solved, providing insufficient signal.
5. Narrow domain coverage — most benchmarks target a single ML subfield.

DiscoGen operates at a distinct level from other systems surveyed in this volume. Where FunSearch, AlphaEvolve, and OpenELM are ADAs—systems that discover algorithms—DiscoGen is infrastructure that generates the problems those ADAs attempt. This complementary positioning makes DiscoGen a meta-level contribution: it does not compete with ADAs but rather provides the evaluation substrate on which they can be principally compared and optimized.

Attribution. DiscoGen was developed by a 20-author team spanning the University of Oxford, UC Santa Barbara, University College London (UCL), and collaborating institutions [PAPER §2]. The work is led by Alexander D. Goldie under the equal supervision of Roberta Raileanu, Shimon Whiteson, and Jakob N. Foerster. The paper appeared as arXiv:2603.17863 on March 18, 2026 [PAPER].

27.2 Architecture

27.2.1 Repository Audit

The following top-level structure is reported in the paper and documentation [PAPER §12, README]:

The reported domain directory structure per domain [PAPER §10] follows this layout:

# Pseudocode — reconstructed from paper §10 and documentation
# Not verified against actual repository files
discogen/domains/{DomainName}/
├── base/              # Complete baseline implementations (frozen)
│   ├── loss.py
│   ├── networks.py
│   ├── optim.py
│   └── ...
├── edit/              # Editable templates (function signatures only)
│   ├── loss.py
│   └── ...
├── utils/
│   ├── _reference.txt # Reference documentation for the domain
│   ├── environments.py
│   └── evaluation.py
├── datasets/          # Dataset configurations
├── config.yaml        # Domain-level defaults
└── install.sh         # Domain-specific dependency installer

27.2.2 Architecture Diagram

27.2.3 Execution Trace

The paper and documentation describe the following CLI-based workflow [PAPER §10, README]:

# Pseudocode — reconstructed from paper §7 and documentation
# CLI commands as documented; exact --flag names from README

# Step 1: List available domains
discogen get-domains

# Step 2: Sample a random task configuration
discogen sample-task-config --config-dest random_task.yaml

# Step 3: Create a task from configuration
discogen create-task \
  --task-domain OnPolicyRL \
  --config-path discobench_configs/task_42.yaml

# Step 4: Install domain-specific dependencies
cd task_src/OnPolicyRL
bash install.sh

# Step 5: Run the generated task (meta-train evaluation)
python run_main.py

# Step 6: Create and run meta-test evaluation
discogen create-task \
  --task-domain OnPolicyRL \
  --config-path discobench_configs/task_42.yaml \
  --test
cd task_src
python run_main.py

The --example flag is documented as generating tasks with editable (incomplete) modules, while the --test flag generates the meta-test evaluation version [README]. The expected output is a self-contained task_src/ directory.

Configuration fields reported in the paper [PAPER §10]:

27.2.4 Design Principles

The paper articulates four design principles [PAPER §9]:

1. Modularity over monolith. Algorithms are decomposed into independently editable modules rather than requiring wholesale implementation. This enables difficulty control (1 vs. 6 modules), attribution of improvements, and composability across tasks.
2. Separation of concerns. Strict separation between discovery phase (meta-train) and evaluation phase (meta-test), with no modification allowed during evaluation.
3. Configuration-driven generation. Every task aspect is determined by a YAML configuration, enabling deterministic reproduction, systematic difficulty sweeps, and automated curriculum construction.
4. Domain independence. The generator framework is domain-agnostic; adding a new domain requires implementing module base/edit versions, dataset adapters, evaluation metrics, and an install.sh script [PAPER §9].

27.3 Core Algorithms

27.3.1 Verification Matrix

27.3.2 Procedural Task Generation

The central mechanism of DiscoGen is the procedural generation of algorithm discovery tasks from a parameterized configuration space. The task count per domain is given by the following formula [PAPER §10]:

27.3.3 Modular Algorithm Decomposition

Each ML algorithm is decomposed into semantically meaningful, independently editable modules [PAPER §4, §11]. The decomposition varies by domain:

Each module has two versions [PAPER §10, §11]:

• Base version (base/*.py): a complete, working reference implementation that serves as both the frozen baseline and the starting point in baseline initialization mode.
• Edit version (edit/*.py): function signatures with defined input/output specifications but no implementation body, used in empty initialization mode.

# Pseudocode — reconstructed from paper §11
# Illustrative example of module interface structure; not verified against actual files

# edit/loss.py (empty initialization mode)
def compute_loss(
    log_probs: Tensor,        # shape: (batch, timesteps)
    advantages: Tensor,       # shape: (batch, timesteps)
    old_log_probs: Tensor,    # shape: (batch, timesteps)
    values: Tensor,           # shape: (batch, timesteps)
    returns: Tensor,          # shape: (batch, timesteps)
    clip_eps: float = 0.2
) -> Tensor:
    """Compute the policy optimization loss.
    
    Returns: scalar loss tensor for gradient descent.
    """
    # YOUR IMPLEMENTATION HERE
    raise NotImplementedError

The difficulty gradient created by module count is a key design feature. The paper demonstrates empirically that success rates decrease precipitously as more modules become editable [PAPER §6, Table]:

27.3.4 Meta-Train/Meta-Test Separation

Every generated task enforces a strict split between discovery and evaluation datasets [PAPER §3, §11]. During the discovery phase (meta-train), the ADA has access to a subset of datasets and can iteratively modify editable modules, train, and evaluate. During the evaluation phase (meta-test), the discovered algorithm is evaluated on held-out datasets with no further modification permitted.

# Pseudocode — reconstructed from paper §11
# Illustrative discovery-evaluation workflow

# Discovery phase (meta-train)
for iteration in range(max_iterations):
    modified_code = ada.edit(editable_modules, task_description)
    train_score = evaluate(modified_code, meta_train_datasets)
    ada.receive_feedback(train_score)

# Evaluation phase (meta-test) — no further edits
final_code = ada.get_best_solution()
test_score = evaluate(final_code, meta_test_datasets)  # Held-out

The paper provides empirical validation that this split matters: rank correlation between algorithms' meta-train and meta-test performance is reported to be weak, meaning algorithms that perform well during discovery frequently fail to generalize [PAPER §6]. This finding directly supports the methodological necessity of the split.

27.3.5 Meta-Meta-Learning Loop

The paper demonstrates a prompt optimization loop where an outer LLM optimizes the system prompt of an inner ADA [PAPER §5, §6, §11]. Over 30 optimization steps, the outer LLM proposes new ADA prompts based on performance traces from sampled DiscoGen tasks.

# Pseudocode — reconstructed from paper §5, §11
# Meta-meta-learning prompt optimization loop

best_prompt = initial_prompt
for step in range(30):
    task_config = discogen.sample_task()          # Fresh task
    score = run_ada(best_prompt, task_config)      # ADA attempts task
    new_prompt = optimizer_llm.propose(            # Outer LLM proposes
        history=prompt_score_history,              #   improved prompt
        latest_score=score
    )
    prompt_score_history.append((new_prompt, score))
    if score > best_score:
        best_prompt = new_prompt

The key variable in this loop is the number of distinct DiscoGen tasks seen during optimization. The paper reports that using a single task leads to overfitting, while 30 unique tasks yields the best generalization on DiscoBench [PAPER §6].

27.4 Key Results

27.4.1 Evaluation Caveats

27.4.2 DiscoBench Single (One Module, One Attempt)

Critical finding: No evaluated model consistently outperforms the baseline implementations when editing a single module [PAPER §6]. Even Deepseek-v3.2, the best-performing model, achieves a meta-test score of 1053 versus the baseline's 1177 — a deficit of 124 points (approximately 10.5% below baseline). This is a striking negative result: current LLMs, when modifying individual algorithm components in a single attempt, frequently produce algorithms that are worse than standard implementations.

27.4.3 DiscoBench All (All Modules, One Attempt)

Success rates collapse dramatically when all modules are editable simultaneously. GPT-OSS 120B drops from 68.2% to 11.4%, and notably achieves a meta-train score with a confidence interval crossing zero ([−183, 700]), indicating that the model frequently generates algorithms that fail to train at all [PAPER §6]. The wide confidence intervals in the "All" setting suggest high variance across tasks, domains, or runs.

27.4.4 Meta-Meta-Learning Results

Meta-test performance improves monotonically from 957 ($K=1$) to 1071 ($K=30$), a gain of 114 points (approximately 11.9%) [PAPER §6]. This is the paper's most important empirical validation: task diversity during ADA optimization directly improves generalization. The result supports DiscoGen's core hypothesis that its scale enables genuine learning rather than memorization of specific tasks.

27.4.5 Interpreting the Negative LLM Results

The finding that LLMs consistently underperform baselines in single-attempt algorithm discovery warrants careful interpretation. Several factors may contribute to this result, and the paper does not fully disentangle them:

1. Single-shot protocol. The DiscoBench Single evaluation gives models one attempt. Iterative systems like FunSearch or AlphaEvolve use hundreds or thousands of evaluation cycles. The "Until Success" column shows all three models eventually reach 100% success rate, suggesting the issue is partly protocol sensitivity rather than fundamental incapability.
2. Baseline strength. The reference implementations are described as standard, well-tuned algorithms (e.g., PPO for RL). Beating a well-implemented PPO on Atari by modifying only the loss function is genuinely difficult — the baselines may be closer to practical optima than they appear.
3. Domain heterogeneity. Aggregated scores across 14 diverse domains may mask domain-specific competence. A model might excel at RL loss design but fail at Bayesian optimization acquisition functions, and the aggregated score would not reveal this.
4. Module-level difficulty variation. Not all modules are equally editable. Modifying networks.py requires architectural knowledge different from modifying loss.py. The paper reports aggregate module counts but does not break down success rates by module type across domains.
5. Initialization mode. The DiscoBench evaluation does not report results separately for baseline vs. empty initialization, though the difficulty difference between them is likely substantial.

27.5 Implementation & Cost

27.5.1 Cost Analysis

Task generation cost: The procedural generation itself (assembling files from templates) is computationally negligible — it involves file copying and YAML parsing, not training or inference.

Task evaluation cost: This is the dominant cost and varies enormously by domain. Based on the domain descriptions [PAPER §8]:

LLM API cost for ADA evaluation: Each ADA attempt involves sending task descriptions and module templates as context, generating modified code, and potentially iterating. Per-task costs depend heavily on model pricing, context length, and number of iterations. For the meta-meta-learning experiment (30 optimization steps), the dominant cost is task evaluation compute (GPU time), not LLM API calls [PAPER §8].

Domain dependency isolation. Each domain has its own install.sh for dependencies [PAPER §10, §12], addressing the practical challenge that 14 ML domains may have conflicting requirements (e.g., JAX for RL environments vs. PyTorch for CV). This design implies that running tasks across all domains requires managing multiple Python environments or careful dependency resolution.

27.6 Reproducibility

27.6.1 Step-by-Step Verification Path

Based on the paper and documentation [PAPER §7, README], a reproduction attempt would proceed as follows:

1. Clone repository: git clone https://github.com/AlexGoldie/discogen.git
2. Install: make install (sets up uv environment + pre-commit hooks) [PAPER §12]
3. Alternatively: pip install discogen [README]
4. List domains: discogen get-domains — expect list of 14 domains
5. Create a DiscoBench task: discogen create-task --task-domain OnPolicyRL --config-path discobench_configs/task_42.yaml
6. Install domain deps: cd task_src/OnPolicyRL && bash install.sh
7. Run baseline: python run_main.py — expect a numeric score
8. Run meta-test: discogen create-task --task-domain OnPolicyRL --config-path discobench_configs/task_42.yaml --test
9. Verify scores match baseline range: Compare to reported DiscoBench baseline scores [PAPER §6]

What constitutes successful reproduction: (a) The generator produces a runnable task directory; (b) run_main.py completes without error; (c) baseline scores fall within the confidence intervals reported in the paper.

27.6.2 Reproducibility Assessment

Task generation is described as fully deterministic given configuration parameters — the same YAML config should produce the same task directory [PAPER §7]. This is a strong reproducibility property for the generator itself. However, the evaluation of generated tasks depends on domain-specific training processes (neural network training with stochastic gradient descent), which introduces the usual ML reproducibility challenges around hardware, library versions, and floating-point non-determinism.

27.7 Threats to Validity

Task count interpretability. The headline figure of ~99.3 billion unique tasks warrants scrutiny. While the combinatorial formula is mathematically correct, many of these configurations may not be meaningfully distinct. For example, changing only the meta-train/meta-test split while keeping all other parameters identical produces "different" tasks that test the same algorithmic challenge against different evaluation data. The effective task diversity — how many truly independent challenges the generator can produce — is likely substantially smaller than the combinatorial count, though still orders of magnitude larger than existing static benchmarks.

Baseline quality dependence. All results are relative to reference implementations provided with DiscoGen. The strength of the negative LLM results (LLMs underperforming baselines) depends critically on baseline quality. If the baselines are unusually strong or well-tuned, the gap may reflect implementation quality rather than fundamental LLM limitations. Conversely, if baselines are weak, beating them is less impressive. The paper does not provide evidence of baseline quality calibration against external implementations [PAPER §7].

Score aggregation opacity. Cross-domain score comparison requires normalization across metrics with fundamentally different scales (RL returns vs. classification accuracy vs. optimization regret). The paper reports aggregate scores but does not document the normalization procedure in sufficient detail to assess whether it introduces domain-weighting biases [PAPER §6].

Domain coverage gaps. While 14 domains is broader than any prior algorithm discovery benchmark, significant ML subfields are absent: graph neural networks, generative models (diffusion, flow matching), speech recognition, recommendation systems, and federated learning. The 14 domains also vary dramatically in scale: On-Policy MARL contributes 97.4 billion of the 99.3 billion total tasks, meaning the combinatorial space is heavily concentrated in a few domains.

No independent reproduction. All reported results are from the original authors. No independent group has, to this survey's knowledge, reproduced the DiscoBench evaluations or validated the meta-meta-learning findings.

Evaluation protocol single-shot bias. The DiscoBench Single evaluation gives models one attempt per task. This is methodologically clean but does not reflect how ADAs typically operate (iteratively, with many attempts). The strong difference between "Single" (68.2% for GPT-OSS 120B) and "Until Success" (100%) suggests the single-shot protocol may significantly understate capability while accurately measuring reliability.

Domain dependency conflicts. The per-domain install.sh approach creates practical challenges: running tasks across all 14 domains likely requires multiple isolated environments. Whether this fragmentation introduces evaluation inconsistencies (e.g., different PyTorch/JAX versions across domains affecting baseline scores) is not discussed [PAPER §7].

Temporal stability of task specifications. DiscoGen domains depend on external packages (Gymnax, MinAtar, BoTorch, etc.) that evolve independently. Whether task evaluations remain reproducible as these dependencies update is an open concern not addressed in the paper.

27.8 Limitations & Open Questions

No iterative ADA evaluation. DiscoBench currently evaluates ADAs in a single-shot or "until success" protocol. Real ADAs like FunSearch and AlphaEvolve operate iteratively, generating many candidates and refining solutions across hundreds or thousands of evaluations. DiscoGen's infrastructure supports iterative evaluation (the meta-meta-learning experiments demonstrate this), but DiscoBench as currently defined does not provide standardized iterative evaluation protocols [PAPER §6].

Module interface rigidity. The decomposition into fixed modules assumes that algorithm improvements can be localized to specific components. Some algorithmic innovations — such as new training paradigms that change the relationship between loss functions and update rules — may not fit cleanly into a single module. The paper acknowledges that multi-module editing is harder but does not discuss whether the module boundaries themselves might need to evolve [PAPER §11].

Evaluation type coverage. The energy and time evaluation types are described [PAPER §4] but evaluation results in the paper focus exclusively on the performance type. Whether the energy and time objectives produce meaningfully different algorithm rankings is not demonstrated.

27.9 Survey Positioning

DiscoGen occupies a unique niche in the landscape of LLM-powered evolutionary AI systems surveyed in this volume. It is not an Algorithm Discovery Agent — it is infrastructure for evaluating and optimizing ADAs. This complementary positioning means DiscoGen does not compete with systems like FunSearch, AlphaEvolve, or OpenELM, but rather provides the evaluation substrate on which they can be principally compared.

27.9.1 Comparison with Related Systems

Relationship to FunSearch and AlphaEvolve. These systems are ADAs — they discover algorithms through evolutionary LLM-based search. DiscoGen provides a standardized evaluation substrate for measuring and comparing such systems. A FunSearch or AlphaEvolve agent could be pointed at DiscoGen-generated tasks, and its performance measured on DiscoBench. This creates a potential standard evaluation layer that the field currently lacks [PAPER §15].

Relationship to ALE-Bench. ALE-Bench (Chapter 20) evaluates LLM agents on competitive programming problems from AtCoder. Both systems aim to benchmark AI capabilities on algorithmic tasks, but at different levels: ALE-Bench tests competitive programming skill, while DiscoGen tests algorithm design skill — the ability to create novel ML algorithms that generalize.

Relationship to UED literature. DiscoGen explicitly draws from the Unsupervised Environment Design paradigm [PAPER §3], applying procedural generation principles from RL training to the algorithm discovery setting. This is a novel application of a well-established idea — one that creates an intellectually satisfying recursion, since UED is itself one of DiscoGen's 14 domains.

27.9.2 Evolutionary Analogy

DiscoGen can be situated within an evolutionary framework, though the analogy is imperfect:

Where the analogy breaks down. DiscoGen's "organisms" (algorithms) do not reproduce with variation in a population — each is independently generated by an ADA. There is no population dynamics, no selection pressure within a generation, and no heredity in the biological sense. The meta-meta-learning loop optimizes a single prompt (not a population), making it closer to gradient-free optimization than to evolution proper. The evolutionary framing is most accurate when describing the task environment — DiscoGen does generate varying "fitness landscapes" in a manner directly analogous to UED.

27.10 Summary