Bilevel Autoresearch

37.1 Overview and Motivation

Every autoresearch system surveyed in this book—from Karpathy's single-track loop to AlphaEvolve's MAP-Elites archipelago—shares a common property: when those systems were improved, the improvement was designed by a human. A researcher read the code, identified a bottleneck, and wrote new logic to address it. Bilevel Autoresearch, introduced by Qu and Lu in March 2026, asks whether an LLM can perform that same design step autonomously.

The system formalizes this question as a bilevel optimization problem: an outer loop generates new search mechanisms as Python code and injects them at runtime into an inner autoresearch loop. The inner loop optimizes task performance (hyperparameter configurations for GPT pretraining); the outer loop optimizes the inner loop's search strategy. Both levels use the same LLM—DeepSeek's deepseek-chat—which the authors argue means improvements must arise from structural advantage rather than from deploying a more capable model at the meta level.

The paper reports a 5× improvement over the baseline inner loop (mean Δval_bpb of −0.045 vs. −0.009) on a GPT pretraining benchmark, with three distinct search mechanisms—Tabu Search, Multi-Scale Bandit, and Orthogonal Exploration—discovered autonomously from combinatorial optimization, online learning, and design of experiments, respectively. These mechanisms were generated as runnable Python code and injected at runtime without human specification of which algorithmic domains to explore. However, these results are based on only 3 repeats per group with high variance (Group C standard deviation ±0.029, yielding approximate 95% confidence intervals that include zero), and should be treated as exploratory evidence rather than statistically confirmed findings (see Section 37.5.4).

37.1.1 The Human Bottleneck in Autoresearch

The paper motivates its contribution by cataloguing human-designed improvements to prior autoresearch systems:

System	Human-Designed Improvement	Year
Karpathy autoresearch	Single-track propose-evaluate-accept loop	2026
AutoResearchClaw	Multi-batch parallel search (increased branching factor)	2026
EvoScientist	Persistent experience memory across runs	2026

Source: arXiv:2603.23420, Section 1.

Each of these improvements required a human to read the system's code, identify a specific bottleneck, and write new code to address it. The bilevel framework automates this meta-design step by having the same LLM that drives the inner loop also analyze the inner loop's behavior and generate new search logic.

37.1.2 From Parameter Tuning to Mechanism Design

The paper's sharpest empirical finding is the contrast between two forms of meta-optimization:

Adjustment Type	Level	What Changes	Mean Δval_bpb	$n$
Parameter-level	1.5	Which parameters are explored, in what order	−0.006 (no reliable gain)	3
Mechanism-level	2	How proposals are generated and filtered	−0.045 (5× gain, paper-reported)	3

Source: arXiv:2603.23420, Table 2. Based on 3 repeats per group; see Section 37.5.4 for uncertainty analysis showing that both groups' 95% confidence intervals include zero.

This result—that adjusting what the inner loop searches appears insufficient while changing how it searches yields substantial gains—has potential implications beyond this specific system, though the authors acknowledge that generalization beyond the single benchmark is unproven. If the finding holds more broadly, it aligns with decades of optimization theory: algorithm selection and configuration typically dominate parameter tuning in determining optimization performance.

37.2 Architecture

Bilevel Autoresearch employs a three-level nested architecture. Level 1 is a standard autoresearch inner loop (propose-evaluate-accept). Level 1.5 adjusts search parameters (freezing/unfreezing hyperparameters, injecting guidance). Level 2 generates entirely new search mechanisms as Python code and injects them at runtime.

37.2.1 Separation of Concerns

A critical design property described in the paper is the strict separation of modification scope across levels. Level 1 cannot change its own structure—it executes the propose-evaluate-accept cycle as written. Level 1.5 can redirect search by freezing/unfreezing parameters and injecting guidance strings, but it cannot alter proposal logic, acceptance criteria, or loop structure. Only Level 2 can perform structural modifications to the search mechanism by generating and injecting new Python code.

This separation ensures that the system's behavior can be attributed to specific levels during ablation. It also provides a partial safety boundary: Level 1.5 cannot introduce code-level changes that might break the inner loop, while Level 2's code injection is protected by a validate-and-revert pipeline (Section 37.3.5), though this validation has important limitations discussed therein.

37.2.2 Same-Model Recursion

All three levels use the same DeepSeek deepseek-chat model. The paper states this explicitly: "Both loops use the same LLM—no stronger model is needed at the meta level." The authors argue that the observed improvement therefore arises from the bilevel architecture—the structural arrangement of nested loops with code injection—rather than from deploying superior reasoning at the outer level. This interpretation is plausible but not conclusively established by the experiment: it is also possible that the LLM's code-generation capabilities (exercised at Level 2 but not at Level 1) contribute independently of the bilevel structure. Distinguishing architectural advantage from capability utilization would require additional ablations not present in the paper.

37.3 Core Algorithms

37.3.1 Bilevel Optimization Formulation

The paper formalizes autoresearch as a bilevel optimization problem. We present this formulation with explicit annotation of the approximations involved:

where:

• $\Phi$ is the space of search mechanisms—syntactically valid Python programs that implement the runner logic. This is a discrete, combinatorial space with no natural gradient structure.
• $\phi \in \Phi$ is a specific search mechanism (runner code).
• $\theta \in \Theta$ is the task configuration—a vector of hyperparameters (learning rate, weight decay, batch size, etc.) for the GPT pretraining task.
• $\mathcal{A}(\phi, T, \omega)$ is the approximate inner optimizer: the output of running mechanism $\phi$ for $T = 30$ iterations with stochastic seed $\omega$ (encompassing data ordering, weight initialization, and LLM sampling randomness). This is not an exact $\arg\min$—it is a fixed-budget, stochastic, greedy search that yields a different $\hat{\theta}$ for each realization of $\omega$.
• $f(\theta, \phi)$ is the inner objective: validation bits-per-byte (val_bpb) achieved after training for 300 seconds with configuration $\theta$ under search mechanism $\phi$.
• $F(\phi, \hat{\theta}(\phi)) = f(\hat{\theta}(\phi), \phi)$ is the outer objective: the best val_bpb achieved by the inner loop when running with mechanism $\phi$.

Several important distinctions from classical bilevel optimization deserve emphasis:

1. $\phi$ is a program, not a parameter vector. The outer level searches over a space of Python programs rather than adjusting continuous parameters. No gradient information is available; the LLM serves as the sole search operator for this space.
2. The inner problem is solved approximately. Classical bilevel optimization assumes the inner problem is solved to optimality ($\theta^*$). Here, the inner loop runs for a fixed budget of 30 iterations with greedy acceptance, yielding an approximate solution $\hat{\theta}$ that depends on stochastic factors $\omega$. The outer objective $F$ therefore inherits this stochasticity: the same $\phi$ may yield different $F$ values across runs.
3. The outer problem is also solved approximately. Level 2 generates at most a few candidate mechanisms per run via LLM dialogue. There is no systematic exploration of $\Phi$, no population of mechanisms maintained, and no selection pressure beyond "did this mechanism improve val_bpb relative to the default runner?"

The bilevel formulation is therefore a conceptual framework that clarifies what the system is trying to do (optimize the optimizer), not a guarantee of nested optimality or convergence. The practical system is better described as "LLM-guided stochastic search over programs that each execute LLM-guided stochastic search over hyperparameters."

37.3.2 Inner Loop Algorithm (Level 1)

Level 1 implements the canonical Karpathy autoresearch cycle: propose a configuration change, train for a fixed budget, evaluate, and accept if improved. According to the paper (Section 3.1), the LLM receives the current best training configuration, a list of editable parameters, a list of frozen parameters, and a strategic guidance string from Level 1.5. It returns a set of parameter changes and a one-sentence hypothesis.

The acceptance criterion is greedy: a proposal is accepted if and only if its val_bpb is strictly less than the current best. Formally, at iteration $t$:

where $\theta_t'$ is the proposed configuration and $\hat{\theta}_t^*$ is the current best. Note that $f(\theta_t', \phi)$ is itself stochastic—the same configuration may yield slightly different val_bpb across runs due to training randomness. The greedy acceptance criterion therefore makes the search path path-dependent on training noise, contributing to the variance observed across repeats.

The following is illustrative pseudocode reflecting the Level 1 algorithm as described in arXiv:2603.23420, Section 3.1. It is not derived from the repository source code. The function name run_inner_loop, parameter names, and control flow are the chapter author's reconstruction for pedagogical clarity; the actual implementation's naming and structure may differ entirely.

# ILLUSTRATIVE PSEUDOCODE — NOT FROM REPOSITORY
# Reconstructed from: arXiv:2603.23420, Section 3.1

def inner_loop(train_config: dict, editable_params: list[str],
               frozen_params: dict[str, Any], guidance: str,
               iterations: int = 30, train_budget_sec: int = 300) -> tuple[float, list]:
    """Illustrative: standard propose-evaluate-accept autoresearch cycle."""
    best_config = dict(train_config)
    best_val_bpb = train_and_evaluate(best_config, budget=train_budget_sec)
    trace = []

    for t in range(iterations):
        # LLM proposes parameter changes + one-sentence hypothesis
        proposal, hypothesis = llm_propose(
            current_config=best_config,
            editable=editable_params,
            frozen=frozen_params,
            guidance=guidance
        )

        # Train for fixed budget, measure val_bpb
        val_bpb = train_and_evaluate(proposal, budget=train_budget_sec)

        # Greedy acceptance: strictly less than current best
        accepted = val_bpb < best_val_bpb
        if accepted:
            best_config = proposal
            best_val_bpb = val_bpb

        trace.append({
            "iter": t, "params": proposal, "hypothesis": hypothesis,
            "val_bpb": val_bpb, "accepted": accepted
        })

    return best_val_bpb, trace

37.3.3 Search Strategy Adjustment (Level 1.5)

According to the paper, every $K = 5$ inner iterations, Level 1.5 analyzes the full search trace and generates a search-configuration update. The paper describes three components:

1. Freeze parameters proposed $\geq k$ times (the paper uses $k = 3$) with zero net improvement. This prevents the inner loop from wasting iterations on exhausted search directions.
2. Unfreeze parameters that were frozen but haven't been explored since the search moved to a new region of the configuration space.
3. Inject a guidance string instructing the inner loop to prioritize under-explored parameters.

Critically, the paper specifies that Level 1.5 cannot change the proposal generation logic, acceptance criterion, or loop structure. These constraints limit Level 1.5 to redirecting search within the existing mechanism, without altering the mechanism itself. This is the core distinction from Level 2.

The empirical result for Level 1.5 is notable for its failure: Group B (Level 1 + 1.5) achieved a mean improvement of only −0.006, compared to Group A (Level 1 alone) at −0.009. In one repeat (B-R1), Level 1.5 achieved zero improvement—the worst single outcome across all groups. The paper interprets this as evidence that parameter-level search redirection is insufficient to overcome the inner loop's fundamental limitations, though with only three repeats this interpretation remains tentative.

37.3.4 Mechanism Research (Level 2)

Level 2 is the system's primary contribution as described in the paper. According to Section 3.3 of arXiv:2603.23420, it conducts a 4-round structured dialogue, each round a single LLM call, to generate new search mechanisms as Python code:

Round	Name	Input (Paper-Described)	Output (Paper-Described)
1	EXPLORE	Runner source code + search trace	Survey of candidate mechanisms from adjacent fields (combinatorial optimization, online learning, DOE, Bayesian optimization)
2	CRITIQUE	Candidate mechanisms from Round 1	Evaluation against observed failure modes; selection of most promising mechanism
3	SPECIFY	Selected mechanism from Round 2	Interface specification: class name, constructor arguments, method signatures, integration points
4	GENERATE	Specification from Round 3	Complete, runnable Python code + modifications to runner

Source: arXiv:2603.23420, Section 3.3. The exact prompt templates, context formatting, and response parsing are not described in the paper.

The paper reports that each Level 2 session takes approximately 3 minutes of wall time (four sequential LLM calls). Level 2 triggers every $M = 2$ outer cycles—equivalently, every 10 inner iterations in Group C's configuration.

The following is illustrative pseudocode reflecting the Level 2 research session as described in arXiv:2603.23420, Section 3.3. It is not derived from the repository source code. The function names (research_session, build_explore_prompt, etc.) are invented by the chapter author.

# ILLUSTRATIVE PSEUDOCODE — NOT FROM REPOSITORY
# Reconstructed from: arXiv:2603.23420, Section 3.3

def research_session(runner_source: str, search_trace: list[dict]) -> str:
    """Illustrative: 4-round LLM dialogue to generate a new search mechanism."""

    # Round 1: EXPLORE — survey adjacent fields for candidate mechanisms
    candidates = llm_call(
        context=runner_source + format_trace(search_trace),
        instruction="Survey mechanisms from adjacent fields..."
    )

    # Round 2: CRITIQUE — evaluate candidates against observed failures
    selected = llm_call(
        context=candidates + format_trace(search_trace),
        instruction="Evaluate each candidate against the failure modes..."
    )

    # Round 3: SPECIFY — precise interface specification
    specification = llm_call(
        context=selected + runner_source,
        instruction="Write an interface specification: class name, "
                    "constructor args, method signatures, integration points..."
    )

    # Round 4: GENERATE — complete runnable Python code
    mechanism_code = llm_call(
        context=specification + runner_source,
        instruction="Implement the specified mechanism as runnable Python..."
    )

    return mechanism_code  # Full patched runner with mechanism integrated

37.3.5 Validate-and-Revert Pipeline

According to the paper, generated code is validated via Python's importlib dynamic loading before activation. If the import fails for any reason—syntax error, missing dependency, interface mismatch—the original runner is restored from backup. This provides a non-destructive safety net against certain classes of code injection failure.

The following is illustrative pseudocode reflecting the validation pipeline as described in arXiv:2603.23420. It is not derived from the repository source code. The actual implementation may use different error handling, file management, or validation strategies.

# ILLUSTRATIVE PSEUDOCODE — NOT FROM REPOSITORY
# Reconstructed from: arXiv:2603.23420, Section 3.4

import importlib
import sys

def inject_mechanism(mechanism_code: str, runner_path: str,
                     expected_attr: str) -> bool:
    """Illustrative: inject generated code with automatic rollback."""
    backup = open(runner_path).read()

    try:
        # Write patched runner
        with open(runner_path, "w") as f:
            f.write(mechanism_code)

        # Clear module cache to force reimport
        # (Paper reports a preliminary run was invalidated by
        #  omitting this step — see Section 37.7.2)
        module_name = runner_path.replace(".py", "")
        if module_name in sys.modules:
            del sys.modules[module_name]

        # Validate via import — catches syntax errors, missing imports
        module = importlib.import_module(module_name)
        assert hasattr(module, expected_attr)
        return True  # SUCCESS: mechanism activated

    except Exception as e:
        # Revert to backup on any failure
        with open(runner_path, "w") as f:
            f.write(backup)
        return False  # REVERTED to original

The sys.modules cleanup is essential. Without it, Python's module caching returns the old module on reimport, causing a silent fallback—the mechanism appears injected but the original code actually executes. The paper reports that a preliminary run was invalidated by exactly this bug before it was fixed for the reported results.

Scope and Limitations of Import-Time Validation

Import-time validation via importlib provides a specific, limited class of safety guarantees. It catches:

• Syntax errors in the generated Python code.
• Import-time failures from missing dependencies (e.g., the GP Regressor's missing sklearn).
• Missing entry points: the hasattr check confirms the expected function or class exists.

However, import-time validation does not catch:

• Semantic errors: a mechanism that imports correctly but implements logically incorrect logic (e.g., a tabu list that never rejects proposals) will pass validation.
• Runtime errors: type mismatches, division by zero, or out-of-bounds access that only manifest during execution.
• Unsafe side effects: generated code that performs file I/O, network access, or resource-intensive computation is not restricted by import validation alone.
• Subtle behavioral regressions: a mechanism that runs but performs worse than the original. The system relies on val_bpb comparisons during subsequent iterations to detect this, not on the validation step.
• Sandbox escapes: the paper does not describe any process-level isolation (containers, restricted namespaces). Generated code runs with full Python permissions in the host environment.

The validate-and-revert pipeline is therefore a syntactic and interface-level check, not a comprehensive safety guarantee.

37.3.6 Complete Algorithm

The full bilevel algorithm (Group C configuration, with all three levels active) integrates the components described above. The critical implementation detail noted in the paper is that the acceptance flag must be computed before updating the best val_bpb, so that the trace correctly records whether each proposal was accepted.

The following is illustrative pseudocode synthesizing the complete algorithm from arXiv:2603.23420, Sections 3.1–3.4. It is not derived from the repository source code. All variable names (e.g., theta_best, phi) correspond to the bilevel formulation in Section 37.3.1; the actual implementation's naming conventions are unknown.

# ILLUSTRATIVE PSEUDOCODE — NOT FROM REPOSITORY
# Synthesized from: arXiv:2603.23420, Sections 3.1–3.4

def bilevel_autoresearch(train_config: dict, runner_source: str,
                         T: int = 30, K: int = 5, M: int = 2):
    """
    Illustrative: Complete Bilevel Autoresearch (Group C configuration).
    T: inner iteration budget
    K: Level 1.5 trigger period (every K inner iterations)
    M: Level 2 trigger period (every M outer cycles)
    """
    theta_best = dict(train_config)
    best_val_bpb = train_and_evaluate(theta_best, budget=300)
    phi = runner_source       # current search mechanism
    editable = [...]          # editable parameter names
    frozen = {"DEPTH": 8, "ASPECT_RATIO": 64}
    guidance = ""
    trace = []
    outer_cycle = 0

    for t in range(T):
        # === Level 1: Inner autoresearch iteration ===
        proposal, hypothesis = llm_propose(
            current_config=theta_best, editable=editable,
            frozen=frozen, guidance=guidance
        )
        val_bpb = train_and_evaluate(proposal, budget=300)

        # Acceptance computed BEFORE updating best
        accepted = val_bpb < best_val_bpb
        if accepted:
            theta_best = proposal
            best_val_bpb = val_bpb

        trace.append({"iter": t, "proposal": proposal,
                       "hypothesis": hypothesis,
                       "val_bpb": val_bpb, "accepted": accepted})

        # === Level 1.5: Search strategy adjustment (every K iters) ===
        if (t + 1) % K == 0:
            config_update = llm_analyze_trace(trace)
            # Update frozen/editable params and guidance string
            update_search_config(frozen, editable, guidance, config_update)
            outer_cycle += 1

            # === Level 2: Mechanism research (every M outer cycles) ===
            if outer_cycle % M == 0:
                mechanism_code = research_session(phi, trace)
                if inject_mechanism(mechanism_code, "runner.py", "..."):
                    phi = mechanism_code  # Mechanism activated
                # else: reverted to backup, phi unchanged

    return theta_best, best_val_bpb

37.4 Autonomously Discovered Mechanisms

The most striking result of Bilevel Autoresearch is the content of the mechanisms generated by Level 2. The paper reports that, without human specification of which algorithmic domains to explore, the LLM independently discovered and implemented techniques from three distinct fields. The descriptions below are based on the paper's characterization of these mechanisms in Section 5 of arXiv:2603.23420. The actual generated code is reported to be available in the repository's experiment logs, but the specific implementations—including distance metrics, threshold values, and perturbation strategies—have not been verified from the repository for this chapter.

37.4.1 Tabu Search Manager

Source domain: Combinatorial optimization. Discovered in: Group C, Repeat 1. Reported result: −0.065 Δval_bpb (best single outcome across all 12 runs). Source: arXiv:2603.23420, Table 2.

According to the paper, the Tabu Search Manager maintains a list of recently visited configurations. When the inner-loop LLM proposes a change, the manager checks whether the proposed configuration is too close to any recently visited one. If so, it either rejects the proposal or perturbs it to force movement away from the tabu region.

The paper argues that this mechanism addresses a specific failure mode of the inner loop: the LLM exhibits strong recency bias, tending to propose changes similar to recent successful ones, leading to rapid convergence to local optima. Tabu search explicitly breaks this pattern by forbidding revisits.

The following is an illustrative reconstruction based on the mechanism description in arXiv:2603.23420, Section 5. The distance metric, threshold value, perturbation logic, and tabu tenure are the chapter author's plausible reconstruction; the actual generated code (reported to be in the repository's experiment logs) may use entirely different implementation strategies.

# ILLUSTRATIVE RECONSTRUCTION — NOT VERIFIED AGAINST REPOSITORY
# Based on mechanism description in: arXiv:2603.23420, Section 5
# Distance metric, threshold, and perturbation logic are author inference

from collections import deque

class TabuSearchManager:
    """Illustrative: prevents revisiting recently explored configurations."""

    def __init__(self, tabu_tenure: int = 5, distance_threshold: float = 0.1):
        # tabu_tenure and distance_threshold are author-chosen values;
        # the actual generated code's parameters are not reported
        self.tabu_list: deque = deque(maxlen=tabu_tenure)
        self.threshold = distance_threshold

    def is_tabu(self, proposed_config: dict) -> bool:
        """Check if proposal is too close to any recent configuration."""
        for tabu_config in self.tabu_list:
            if self._distance(proposed_config, tabu_config) < self.threshold:
                return True
        return False

    def filter_proposal(self, proposal: dict) -> dict:
        """Reject or perturb tabu proposals to force exploration."""
        if self.is_tabu(proposal):
            return self._perturb(proposal)
        return proposal

    def update(self, accepted_config: dict) -> None:
        """Add accepted configuration to tabu memory."""
        self.tabu_list.append(accepted_config)

    def _distance(self, a: dict, b: dict) -> float:
        """Author-inferred: normalized distance between configurations.
        The actual generated code may use a different distance metric."""
        shared = set(a.keys()) & set(b.keys())
        if not shared:
            return float('inf')
        diffs = [abs(a[k] - b[k]) / max(abs(b[k]), 1e-8) for k in shared
                 if isinstance(a[k], (int, float))]
        return sum(diffs) / len(diffs) if diffs else 0.0

    def _perturb(self, proposal: dict) -> dict:
        """Author-inferred: perturb proposal to escape tabu region."""
        import random
        perturbed = dict(proposal)
        numeric_keys = [k for k in proposal if isinstance(proposal[k], (int, float))]
        if numeric_keys:
            key = random.choice(numeric_keys)
            perturbed[key] = proposal[key] * random.uniform(0.5, 2.0)
        return perturbed

37.4.2 Multi-Scale Bandit Proposer

Source domain: Online learning / multi-armed bandits. Discovered in: Group C, Repeat 2. Reported result: −0.011 Δval_bpb (modest, comparable to the Group A baseline). Source: arXiv:2603.23420, Table 2.

According to the paper, this mechanism treats each editable parameter as a bandit arm. It maintains per-parameter statistics (proposal count, cumulative improvement) and uses an Upper Confidence Bound (UCB) exploration-exploitation tradeoff to select which parameters to modify at each iteration. The standard UCB1 score for parameter $p$ at total step count $n$ is:

where $R_p$ is the cumulative reward (improvement) from proposals involving parameter $p$, $N_p$ is the number of times $p$ has been proposed, and $n$ is the total number of proposals. Parameters with $N_p = 0$ receive infinite UCB score, ensuring all parameters are tried at least once. This is the standard UCB1 formula from Auer et al. (2002); the paper describes the mechanism as using this approach but does not specify whether the generated code modified or tuned the exploration coefficient.

To make this concrete, consider a worked example at iteration $t = 10$ with four editable parameters. The following values are hypothetical for illustration:

Parameter $p$	$N_p$	$R_p$	Exploit $\frac{R_p}{N_p}$	Explore $\sqrt{\frac{2 \ln 10}{N_p}}$	UCB$_1(p)$
LR	4	0.003	0.00075	$\sqrt{4.605/4} = 1.073$	1.074
BATCH_SIZE	2	0.005	0.00250	$\sqrt{4.605/2} = 1.518$	1.520
WEIGHT_DECAY	3	0.001	0.00033	$\sqrt{4.605/3} = 1.239$	1.239
HEAD_DIM	1	0.002	0.00200	$\sqrt{4.605/1} = 2.146$	2.148

Hypothetical values for pedagogical illustration. Selected: HEAD_DIM (highest UCB due to least exploration). Note that with small $N_p$, the exploration bonus dominates the exploitation term by orders of magnitude—the bandit effectively round-robins through under-explored parameters before rewards become decisive.

While theoretically sound, the bandit was less effective than the Tabu Search or Orthogonal Exploration mechanisms. The paper suggests this may be because the UCB exploration bonus was not calibrated aggressively enough for the small iteration budget ($T = 30$), though this explanation is speculative.

37.4.3 Orthogonal Exploration

Source domain: Design of Experiments (DOE). Discovered in: Group C, Repeat 3. Reported result: −0.058 Δval_bpb (second-best overall). Source: arXiv:2603.23420, Table 2.

According to the paper, this mechanism generates orthogonal arrays of parameter combinations, ensuring that each pair of parameters is explored independently. This prevents the confounding that occurs when the LLM changes multiple correlated parameters simultaneously.

The paper identifies a specific LLM bias this mechanism counters: LLMs tend to propose "package deals"—changing multiple parameters together based on training folklore (e.g., "if you increase batch size, decrease learning rate"). Orthogonal exploration breaks these correlations, enabling the system to discover that some conventional parameter couplings are suboptimal for the specific task.

37.4.4 GP Regressor (Failed Injection)

Source domain: Bayesian optimization. Discovered in: Group D, Repeat 2. Result: Reverted due to missing sklearn dependency. Source: arXiv:2603.23420, Section 5.

This case illustrates that Level 2 imposes no constraint on external dependencies. The generated code attempted to import sklearn.gaussian_process, which was not installed in the runtime environment. The validate-and-revert mechanism correctly handled the failure (this is exactly the class of error that import-time validation reliably catches), but the exposure to arbitrary import requirements is a reliability risk in production settings. More concerning, if the dependency had been installed, the code would have been activated without any verification of its correctness or safety—import-time validation only confirms that the code loads, not that it behaves correctly.

37.4.5 Why Generated Mechanisms Work: A Unifying Explanation

The paper identifies a unifying pattern across all successful mechanisms: they succeed by breaking the inner loop's deterministic search patterns, forcing exploration of directions the LLM's priors systematically avoid. The authors argue that the LLM's implicit prior—trained on ML literature—creates four specific biases:

LLM Bias (Paper's Analysis)	Observed Effect	Mechanism That Counters It
Parameter fixation	Repeatedly proposes changes to same parameters	UCB Bandit (forces under-explored parameters)
Correlation assumptions	Changes parameters in conventional bundles	Orthogonal Exploration (decorrelates changes)
Narrow exploration radius	Conservative changes near known-good values	Tabu Search (forces distance from visited configs)
Mode collapse	Converges on single search direction	Tabu Search (memory prevents revisiting)

Source: arXiv:2603.23420, Section 5 (search trace analysis). The causal attribution of biases to specific LLM behaviors is the authors' interpretation based on their analysis of the search traces. Whether these biases are inherent to the LLM or artifacts of the specific prompt construction is not established.

37.5 Key Results

37.5.1 Experimental Design

The experiment uses a controlled four-group ablation with 3 repeats per group (12 total runs). Each repeat runs the inner loop for 30 iterations with a 300-second training budget per evaluation on a single RTX 5090:

Group	Levels Active	Description	Repeats
A (Baseline)	Level 1 only	Standard inner autoresearch loop	3
B	Level 1 + 1.5	Inner loop + parameter-level adjustment	3
C	Level 1 + 1.5 + 2	Full bilevel system	3
D	Level 1 + 2	Inner loop + mechanism research (no Level 1.5)	3

37.5.2 Primary Results

Group	Mean Δval_bpb	Std Dev	Best Single	Worst Single
A (L1 only)	−0.009	±0.002	−0.011	−0.007
B (L1 + L1.5)	−0.006	±0.006	−0.012	−0.000
C (L1 + L1.5 + L2)	−0.045	±0.029	−0.065	−0.011
D (L1 + L2)	−0.034	±0.032	−0.065	−0.001

Source: arXiv:2603.23420, Table 2. All values are improvements in validation bits-per-byte relative to each repeat's baseline. Standard deviations computed from 3 repeats.

Group C achieves a 5× improvement in mean Δval_bpb over Group A. However, this result must be interpreted in the context of substantial statistical limitations detailed in Sections 37.5.4 and 37.5.6. The improvement is driven entirely by Level 2 mechanism generation. Level 1.5 provides no reliable gain—Group B's mean (−0.006) is actually slightly worse than Group A's (−0.009), though this difference is well within noise for $n = 3$.

37.5.3 Per-Repeat Breakdown

The per-repeat breakdown reveals the high variance characteristic of mechanism-level meta-optimization. Groups C and D show bimodal behavior: dramatic improvements when the generated mechanism is well-suited (C-R1: −0.065 with Tabu Search, C-R3: −0.058 with Orthogonal Exploration) and modest results when it is less effective (C-R2: −0.011 with Multi-Scale Bandit). Group A, by contrast, is highly consistent (−0.007 to −0.011) but limited in magnitude.

37.5.4 Uncertainty Analysis

With only $n = 3$ repeats per group, uncertainty quantification is critical for interpreting these results. The following table reports the complete per-run data alongside standard descriptive statistics and approximate confidence intervals. No inferential claims (hypothesis tests, $p$-values, or significance statements) are justified at this sample size.

Group	R1	R2	R3	$\bar{x}$	$s$	$\text{SE} = \frac{s}{\sqrt{3}}$	Approx. 95% CI
A (L1)	−0.011	−0.009	−0.007	−0.009	0.002	0.001	[−0.014, −0.005]
B (L1+L1.5)	−0.000	−0.012	−0.005	−0.006	0.006	0.003	[−0.019, +0.008]
C (L1+L1.5+L2)	−0.065	−0.011	−0.058	−0.045	0.029	0.017	[−0.118, +0.028]
D (L1+L2)	−0.065	−0.001	−0.036	−0.034	0.032	0.018	[−0.112, +0.044]

Δval_bpb values from arXiv:2603.23420, Table 2. Statistics computed by the chapter author. Approximate 95% CI uses $\bar{x} \pm t_{2, 0.025} \times \text{SE}$ with $t_{2, 0.025} = 4.303$ (Student's $t$-distribution, $\text{df} = n - 1 = 2$). These intervals are extremely wide due to the small sample size and should not be used for group comparisons.

To contextualize the coefficient of variation (CV): Group C's CV = $|s / \bar{x}| = 0.029 / 0.045 = 64\%$, and Group D's CV = $0.032 / 0.034 = 94\%$. By contrast, Group A's CV = $0.002 / 0.009 = 22\%$. This quantifies the tradeoff at the heart of the paper: mechanism-level meta-optimization produces higher mean improvements but with dramatically higher variance—a form of lottery effect where the outcome depends heavily on which mechanism Level 2 generates.

37.5.5 Hypothesis Assessment

The paper frames its results in terms of four hypotheses. Below, we assess each against the reported data, noting that with only $n = 3$ repeats per group, all assessments describe patterns in the observed data rather than statistically confirmed effects.

Hypothesis	Assessment	Per-Repeat Evidence
H1: Group B > Group A	Not supported by observed data	B mean (−0.006) slightly worse than A (−0.009). B-R1 achieved zero improvement. The difference is well within noise.
H2: Group C ≫ Group B	Consistent with observed data	C mean (−0.045) is 7.5× B's (−0.006). The separation is large in absolute terms, though C's CI includes zero (Section 37.5.4).
H3: L2 discovers novel mechanisms	Confirmed by inspection	Three distinct algorithmic domains discovered autonomously. This is a qualitative finding verifiable from the generated code.
H4: Group D ≈ Group C	Partially consistent	D mean (−0.034) reasonably close to C (−0.045). D-R2's near-zero result (GP Regressor revert) suggests D may be less robust, but this is a single observation.

Assessments are the chapter author's characterization of the reported data patterns, not statistical conclusions.

37.5.6 Statistical Limitations

The paper is transparent about significant statistical limitations. These are sufficiently important that they must be considered whenever interpreting the headline results:

• Small sample size ($n = 3$): Far below the threshold for rigorous statistical comparison. As shown in the uncertainty analysis (Section 37.5.4), the approximate 95% CIs for Groups B, C, and D all include zero. The authors state that "reliable estimates would require $n \geq 10$ repeats per group."
• Baseline variance: Baseline val_bpb varies across repeats (1.094–1.114) due to training randomness from data ordering and weight initialization. Relative improvement (Δval_bpb) mitigates this, but baseline-dependent effects cannot be ruled out.
• Single benchmark: All results are on one task (GPT-50M pretraining, 300s budget, RTX 5090). Generalization to other model sizes, architectures, or optimization domains is unproven.
• LLM nondeterminism: The DeepSeek API returns stochastic outputs. The same Level 2 dialogue may produce different mechanisms across runs, making exact reproduction impossible.
• No formal significance testing: The paper does not report $p$-values, confidence intervals, or effect sizes with uncertainty bounds. Given $n = 3$, any such tests would have very low statistical power (a two-sample $t$-test comparing Groups A and C with these sample sizes and variances would have power well below 0.50 for realistic effect sizes).

The 5× headline result, while dramatic in magnitude and intriguing in implication, should be regarded as a promising preliminary finding that motivates further experimentation with larger sample sizes and multiple benchmarks.

37.6 Implementation Details

37.6.1 Hardware and Compute

Component	Value	Source
GPU	Single NVIDIA RTX 5090 (32 GB)	arXiv:2603.23420, §4.1
Training budget per evaluation	300 seconds	arXiv:2603.23420, §4.1
Inner iterations per repeat	30	arXiv:2603.23420, §4.1
Total training time per repeat	~2.5 hours (30 × 300s + overhead)	Paper-derived estimate
Total experiment time (12 repeats)	~30 hours GPU time	Paper-derived estimate
Model under optimization	GPT, 50M parameters	arXiv:2603.23420, §4.1
GPU memory usage (model)	~200 MB (BF16)	Paper-derived estimate

The entire experiment runs on consumer-grade hardware. The 50M-parameter model occupies a small fraction of the RTX 5090's memory, meaning GPU memory is not a bottleneck. This accessibility is a notable feature: the authors demonstrate that exploring fundamental questions about autoresearch methodology does not require institutional infrastructure.

37.6.2 LLM API Costs

Level	Calls per Repeat	Call Type	Estimated Cost
Level 1	30	Short proposal generation	Low
Level 1.5	6	Search trace analysis	Moderate
Level 2	8 (2 sessions × 4 rounds)	Long code generation	Higher
Total per repeat	~44	Mixed	~$1–5 (author est.)
Total experiment	~528	Mixed	~$12–60 (author est.)

Source: arXiv:2603.23420, Section 4.2. Cost estimates are the paper authors' own calculations based on DeepSeek API pricing at time of publication.

The use of DeepSeek's API—among the cheapest frontier LLM APIs available in early 2026—keeps costs remarkably low. The authors estimate the entire 12-run experiment likely cost less than $100 in API fees. For context, systems like AlphaEvolve and FunSearch run on proprietary Google infrastructure with undisclosed but substantially larger compute budgets.

37.6.3 Software Stack

The entire system is pure Python with no multi-language complexity. The code injection mechanism relies on Python's dynamic nature (importlib, sys.modules, runtime patching). The task uses PyTorch for GPT training.

Paper-Described Component	Function	Verification Status
Inner loop runner	Level 1 propose-evaluate-accept cycle; target of Level 2 code injection	Paper-described: the paper refers to a "runner" that is dynamically replaced. Exact file name not independently verified.
GPT training script	PyTorch training logic; target of Level 1 hyperparameter optimization	Paper-described: the paper refers to a train.py-like script containing editable hyperparameters.
Level 1.5 strategy logic	Search trace analysis, freeze/unfreeze decisions, guidance injection	Paper-described: Section 3.2 describes this functionality. Module boundary not specified.
Level 2 research session	4-round LLM dialogue generating mechanism code	Paper-described: Section 3.3 describes the dialogue structure. Implementation details (prompt templates, parsing) not specified.
Code injection pipeline	importlib-based validation and revert	Paper-described: Section 3.4 describes the validate-and-revert pattern, including the sys.modules bug.
Experiment logs	Complete traces for all 12 runs	Paper-claimed release: the paper states these are released on GitHub. Not verified.
Generated mechanism code	Python modules generated by Level 2 (Tabu Search, Bandit, Orthogonal Exploration)	Paper-claimed release: the paper states generated code is included. Not verified.

All verification statuses as of the chapter writing date. "Paper-described" means the component is described in arXiv:2603.23420 but the corresponding repository artifact has not been independently located. "Paper-claimed release" means the paper states the artifact is available on GitHub.

37.7 Reproducibility

37.7.1 Released Artifacts (Paper-Claimed)

The paper states that the following artifacts are released at github.com/EdwardOptimization/Bilevel-Autoresearch:

• Full source code for all three levels
• Complete experiment logs for all 12 runs (4 groups × 3 repeats)
• Generated mechanism code (Tabu Search, Multi-Scale Bandit, Orthogonal Exploration)

This level of claimed transparency is commendable. However, the availability and completeness of these artifacts have not been independently verified for this chapter. Readers intending to reproduce or extend the work should verify the repository contents directly, including checking whether the experiment logs contain sufficient detail (raw val_bpb values per iteration, LLM prompts and responses, exact mechanism code as generated) to support independent analysis.

37.7.2 Reproducibility Challenges

The paper is candid about five significant reproducibility limitations:

1. Small sample size ($n = 3$): Insufficient for rigorous statistical comparison. See Section 37.5.4 for a detailed uncertainty analysis.
2. Baseline variance: Training randomness (data ordering, weight initialization) causes baseline val_bpb to vary between 1.094 and 1.114 across repeats.
3. Single benchmark: Only GPT-50M pretraining with a 300-second budget on RTX 5090 was tested.
4. Dynamic loading fragility: A preliminary run was invalidated because a sys.modules registration bug caused silent fallback to the original runner—mechanisms appeared injected but the original code actually executed. This failure mode is particularly dangerous because the system appears to function correctly while the generated code never actually runs.
5. LLM nondeterminism: The DeepSeek API returns stochastic outputs even at fixed temperature settings, making exact reproduction impossible.

37.7.3 What Full Reproducibility Would Require

Based on the paper's own analysis: fixed random seeds for training, $n \geq 10$ repeats per group, multiple benchmarks (different model sizes and tasks), and deterministic LLM outputs. The last requirement is fundamentally difficult with API-served models, which may not guarantee determinism even at temperature=0 due to implementation details such as floating-point non-associativity in parallel inference.

37.8 Limitations and Discussion

37.8.1 Structural Limitations

No cross-run learning. Unlike EvoScientist, which maintains persistent experience memory, Bilevel Autoresearch starts fresh each repeat. This means each run must rediscover effective mechanisms from scratch—a significant limitation for iterative research workflows.

No dependency management or sandbox isolation. Generated mechanisms can import arbitrary Python libraries. The GP Regressor failure (Group D, R2) demonstrates the reliability dimension: the code required sklearn, which was not installed. More broadly, generated code executes with the full permissions of the host Python process. The paper does not describe any sandboxing beyond import-time validation, which (as discussed in Section 37.3.5) does not prevent unsafe side effects, arbitrary file access, or resource exhaustion.

Validation is necessary but not sufficient. The validate-and-revert pipeline protects against syntactic failures and missing dependencies, but not against semantically incorrect or subtly harmful mechanisms. A mechanism that imports correctly and exposes the expected interface but implements a flawed algorithm will pass validation. The only defense is the inner loop's val_bpb metric, which detects performance degradation but not other forms of harm, and only after several iterations.

Fixed 4-round dialogue structure. The Level 2 research session always follows the EXPLORE → CRITIQUE → SPECIFY → GENERATE sequence. This structure was human-designed and not itself subject to meta-optimization.

37.8.2 Empirical Limitations

High variance. Group C's coefficient of variation (64%) means the system's performance depends heavily on which mechanism Level 2 generates—a form of lottery effect.

Time overhead. Each Level 2 session consumes approximately 3 minutes of wall time. With two sessions per repeat in Group C, this represents ~6 minutes of search time diverted from inner-loop iterations.

Single task, single scale. The GPT-50M pretraining task is small by modern standards. Whether the bilevel framework provides similar gains for larger models (1B+), different architectures, or non-ML optimization tasks remains unproven. The paper's generality claims should be weighted against this single-benchmark, $n = 3$ evidence base.

37.8.3 Comparison with Related Systems

System	Inner Loop	Outer Loop	Meta Target	Code Injection	Same LLM
Karpathy autoresearch	Propose-evaluate-accept	None	—	No	—
AutoResearchClaw	Multi-batch parallel	None (human)	—	No	—
EvoScientist	With experience memory	None (human)	—	No	—
FunSearch	LLM program generation	Evolutionary selection	Task programs	No	Yes
AlphaEvolve	LLM mutation + eval	MAP-Elites database	Task programs	No	Ensemble
Bilevel Autoresearch	Standard autoresearch	LLM-driven	Search mechanism	Yes (runtime)	Yes

Sources: arXiv:2603.23420, Section 6, and the respective publications for each system. System descriptions are the chapter author's summaries.

The critical distinction claimed by the paper is the target of meta-optimization. FunSearch and AlphaEvolve evolve task-level programs—heuristics for bin packing, matrix multiplication kernels. Bilevel Autoresearch evolves the search mechanism itself—the code that governs how proposals are generated, filtered, and accepted. Whether this constitutes a genuinely novel contribution depends on the precise boundaries drawn. Automated algorithm configuration (e.g., SMAC, irace) and meta-learning systems have a long history of optimizing search behavior, though typically via parameter adjustment rather than program synthesis and runtime injection. The authors' novelty claim is strongest when restricted to the specific combination of LLM-driven code generation and runtime injection of search mechanisms into an autoresearch loop. This characterization is the authors' own (arXiv:2603.23420, Section 1); this survey has not independently verified it against all concurrent work.

37.8.4 Potential Extensions

The paper and its analysis suggest several natural extensions, none of which have been implemented:

Mechanism library. Maintain a library of successfully generated mechanisms across runs. Level 2 could then select from the library (exploitation), generate new mechanisms (exploration), or combine features from multiple mechanisms (crossover).

Mechanism evolution. Apply evolutionary strategies to the mechanism library: mutation (variants of successful mechanisms), crossover (combining Tabu + DOE features), selection (retaining mechanisms that work across tasks).

Multi-task transfer. Test whether generated mechanisms generalize across tasks. Mechanisms that work across tasks would represent "meta-transferable" search strategies.

Recursive meta-optimization. A Level 3 that generates improvements to Level 2's 4-round research dialogue. The practical limit is likely context window exhaustion.

37.8.5 Evidence Categories

Throughout this chapter, four categories of evidence have been employed. The following table enables readers to assess the epistemic status of any specific claim:

Category	Examples in This Chapter	Epistemic Status
Paper-reported results	Δval_bpb values (Table in §37.5.2), group means, per-repeat outcomes, hardware specifications, LLM call counts	Directly from arXiv:2603.23420. Verifiable against the paper's tables and text.
Paper-claimed interpretations	"First system" novelty claim (§37.1), 5× improvement attribution (§37.1.2), mechanism-vs-parameter distinction (§37.1.2), LLM bias analysis (§37.4.5), release of code and logs (§37.7.1)	The authors' interpretations and claims. Plausible but not independently verified. Marked with "the authors argue/claim/report."
Chapter author reconstruction	All pseudocode listings (§§37.3.2–37.3.6), Tabu Search implementation details (§37.4.1), UCB worked example (§37.4.2), software component map (§37.6.3)	Reconstructed from paper descriptions. Not verified against the repository. Labeled as illustrative pseudocode throughout.
Chapter author analysis	Uncertainty analysis (§37.5.4), CV calculations (§37.5.4), validation scope analysis (§37.3.5), confidence interval computations, exploratory-vs-confirmatory framing	Independent analysis by the chapter author, informed by the paper's data but not restating it.

37.9 Broader Significance

The authors argue that Bilevel Autoresearch opens a new axis of variation for autoresearch and evolutionary systems. Prior systems operate along two axes: task parameters (hyperparameters, architecture choices) and search parameters (temperature, exploration rate). The paper proposes a third axis: search mechanisms—the algorithms that govern proposal generation, acceptance criteria, and exploration memory.

Axis	Examples	Level of Innovation (Paper's Framing)
Task parameters	Hyperparameters, architecture choices	Low (standard optimization)
Search parameters	Temperature, exploration rate, batch size	Medium (meta-parameter tuning)
Search mechanisms	Proposal logic, acceptance criteria, exploration memory	High (algorithm design)

Source: arXiv:2603.23420, Section 6. The "level of innovation" characterization is the authors' framing, not an independently established taxonomy.

The reported finding that mechanism-level optimization yields a 5× gain over parameter-level optimization—while using the same LLM at both levels—is intriguing as an observed pattern. However, as demonstrated in Section 37.5.4, this result is based on a single benchmark with $n = 3$ repeats per group, and the 95% confidence interval for Group C's mean improvement includes zero. If the finding generalizes beyond the specific experimental setting—a question that remains entirely open—it would have direct implications for the design of all LLM-guided optimization systems surveyed in this book: it would suggest that the search mechanism, not the search parameters, is the primary bottleneck. Readers should not treat the 5× figure as a reliable quantitative estimate of the mechanism-vs-parameter effect size.

The work also has methodological significance as a contribution from independent researchers operating outside major academic-industrial labs, using consumer hardware (single RTX 5090) and a budget LLM API (DeepSeek). This demonstrates that exploring fundamental questions about autoresearch methodology is accessible without institutional infrastructure—an important property for the health of the research ecosystem.

Finally, the autonomously discovered mechanisms—Tabu Search, UCB Bandit, Orthogonal Exploration—are independently interesting as a vocabulary of diversity-injection strategies for LLM-guided search. They provide concrete, empirically motivated answers to the question: how do you prevent an LLM from converging too quickly to local optima? The answer, across all three mechanisms reported in the paper, is the same: inject structured diversity that counteracts the LLM's implicit biases. Whether these specific mechanisms are optimal, or whether the bilevel framework reliably discovers effective mechanisms, are questions that await larger-scale experimentation.