SkyPilot Autoresearch

45.1 Overview and Motivation

A persistent bottleneck in autonomous research systems has been the assumption that a single GPU constitutes the execution substrate. Karpathy's autoresearch (March 2026) demonstrated that an LLM coding agent could autonomously optimize a GPT training script on one GPU, running sequential experiments under a strict five-minute compute budget. The system was elegant in its radical simplicity — three files, one metric, greedy hill-climbing — but the sequential execution model imposed a fundamental ceiling on the kind of research strategy the agent could pursue. With one experiment per decision cycle, the agent was restricted to one-at-a-time hypothesis testing, unable to detect interaction effects, map response surfaces, or exploit heterogeneous hardware.

SkyPilot Autoresearch, released by the Sky Computing Lab at UC Berkeley in March 2026, removes this infrastructure constraint. By replacing the single local GPU with a SkyPilot-managed cluster of 16 GPUs (13 H100s and 3 H200s on Kubernetes via CoreWeave), the extension preserves every design constraint from Karpathy's original — the five-minute budget, single-file modification, val_bpb as the sole metric — while enabling the agent to run 10–16 experiments per decision cycle. The central finding is that this quantitative change in compute produces a qualitative change in agent behavior: the same Claude Code agent that performs greedy hill-climbing on one GPU spontaneously develops factorial experimental design, discovers and exploits hardware heterogeneity, and constructs a two-tier screening/validation workflow — none of which were programmed, prompted, or anticipated.

45.1.1 System Lineage

SkyPilot Autoresearch is not a standalone system but an infrastructure extension layered on top of Karpathy's autoresearch. The lineage is direct and traceable:

System	Date	Execution Model	Search Strategy
nanochat (Karpathy)	2026	Manual single-GPU	Human-directed
Autoresearch (Karpathy)	March 2026	Sequential single-GPU	Greedy hill-climbing
SkyPilot Autoresearch	March 2026	Parallel 16-GPU cluster	Factorial grid search (emergent)

The SkyPilot team's contribution is primarily systems engineering rather than ML research: they identified that the sequential bottleneck in autoresearch was an infrastructure limitation, not an algorithmic one, and showed that removing it unlocks qualitatively different agent behaviors. The agent itself (Claude Code), the research rules (program.md), the training code (train.py), and the optimization metric (val_bpb) are all inherited unchanged from the parent system.

45.2 Architecture

The system architecture consists of three layers: the LLM agent (Claude Code), the infrastructure abstraction (SkyPilot), and the compute fleet (Kubernetes-managed GPU clusters). The agent interacts with the infrastructure exclusively through shell commands learned from a natural-language "skill" document — there is no SDK, no Python bindings, and no programmatic API. The agent reads documentation and constructs CLI invocations autonomously.

45.2.1 Dual-Layer Instruction System

The agent operates under two layers of natural-language instructions, composed at startup:

Layer	Document	Source	Content
1	program.md	Karpathy's autoresearch	Research rules, keep/discard logic, simplicity criterion, "never stop" directive
2	instructions.md	SkyPilot extension	SkyPilot CLI usage, cluster naming, workdir isolation, parallel result tracking

The agent fetches both at startup, reads the autoresearch codebase, then begins autonomous operation. The SkyPilot "skill" is a natural-language document that teaches the agent to use infrastructure commands — a form of procedural knowledge injection that requires zero code changes to the agent itself. The same Claude Code instance that runs sequential autoresearch can run the parallel version simply by reading different instructions.

45.2.2 Infrastructure Components

Five components constitute the system. Each is defined in the SkyPilot examples directory (skypilot/examples/autoresearch/ in the SkyPilot repository).

experiment.yaml — a declarative SkyPilot task specification that defines a single GPU experiment. It declares acceptable GPU types ({H100:1, H200:1}), the Docker image (nvcr.io/nvidia/pytorch:24.07-py3), setup commands, and a run block that captures structured output (experiment status, val_bpb, peak VRAM).

instructions.md — the parallelism-specific agent instructions layered on top of program.md. Covers cluster naming conventions (gpu-01 through gpu-16), workdir isolation for parallel code variants, detached execution (-d flag), and the extended results.tsv format with experiment IDs.

SkyPilot Skill — a natural-language instruction document (fetched from SkyPilot docs) that teaches the agent infrastructure commands: sky launch, sky exec, sky logs, sky status, sky queue, sky down. The agent acquires infrastructure management capabilities by reading this document — no code integration required.

results.tsv — an extended results log tracking all experiments across clusters, with columns for experiment ID, status (keep/discard/crash), val_bpb, memory usage, and description. Unlike sequential autoresearch's git-commit-based tracking, parallel execution uses agent-assigned experiment IDs to handle concurrency.

Cluster Fleet — the physical compute layer managed by SkyPilot on Kubernetes (CoreWeave in the reported experiment). Sixteen single-GPU clusters, each running independently with no inter-GPU communication.

45.2.3 Workdir Isolation and Job Pipelining

Two infrastructure mechanisms are critical for enabling true parallelism:

Workdir snapshotting solves the problem of running multiple code variants simultaneously. For each experiment, the agent copies the codebase to an isolated directory, modifies train.py in that copy, and launches with --workdir pointing to the isolated path. SkyPilot snapshots the directory at submission time, so subsequent modifications do not affect running experiments.

# From skypilot/examples/autoresearch — workdir isolation pattern
# The agent constructs these commands autonomously from instructions.md

# Step 1: Create isolated experiment directory
# mkdir -p /tmp/autoresearch/exp-03
# cp train.py prepare.py pyproject.toml experiment.yaml /tmp/autoresearch/exp-03/

# Step 2: Modify the copy (agent edits train.py in the isolated dir)
# e.g., change ASPECT_RATIO = 64 -> ASPECT_RATIO = 96

# Step 3: Launch with isolated workdir
# sky launch gpu-03 experiment.yaml \
#   --workdir /tmp/autoresearch/exp-03 \
#   --env EXPERIMENT_ID=exp-03 \
#   --env EXPERIMENT_DESC="AR=96 wider model" -d -y

# SkyPilot snapshots /tmp/autoresearch/exp-03/ at submission time
# Subsequent modifications to the directory do not affect the running job

Job pipelining eliminates GPU idle time between experiments on the same cluster. The sky exec command queues a new job on an existing cluster, which starts automatically when the previous job completes. Combined with the -d (detached) flag, the agent can submit experiments and immediately return to planning the next wave.

# Job pipelining: zero-idle-time queuing
# sky launch gpu-01 experiment.yaml -d -y --env EXPERIMENT_ID=exp-01
# (returns immediately; agent plans next experiment)

# Queue next experiment on same cluster (starts when exp-01 finishes):
# sky exec gpu-01 experiment.yaml -d --env EXPERIMENT_ID=exp-02

# Asynchronous result checking:
# sky logs gpu-01        # latest job
# sky logs gpu-01 2      # specific job ID by number

45.3 Core Algorithms and Mechanisms

45.3.1 Wave-Based Factorial Search

The central algorithmic shift is from greedy hill-climbing (one experiment per decision cycle) to wave-based factorial search (10–16 experiments per decision cycle). This is not a programmed algorithm — it is an emergent behavior of the agent when given parallel resources. However, the resulting pattern can be formalized:

Let $\mathbf{x} = (x_1, x_2, \ldots, x_d)$ denote a configuration vector over $d$ hyperparameters. In sequential mode, the agent tests one $\mathbf{x}$ per cycle. In parallel mode with $N$ GPUs, the agent constructs a factorial grid $\mathcal{G}_w$ for wave $w$:

where $\{x_{i_1}\}, \ldots, \{x_{i_k}\}$ are the sets of values tested for $k$ selected parameters, and $x_j^*$ is the current best value for each remaining parameter. The grid size $|\mathcal{G}_w| \leq N$ is bounded by the number of available GPUs. The agent selects which parameters to vary and which values to test based on its analysis of prior waves.

The information gain per wave scales with the grid structure. A one-factor-at-a-time (OFAT) sweep over $k$ parameters with $m$ levels each requires $k(m-1) + 1$ experiments to test all main effects. A full factorial over $k$ factors at $m$ levels requires $m^k$ experiments but reveals all interaction effects up to order $k$. The agent naturally balances these designs based on available GPUs:

where $T_{\text{design}} \approx 60\text{s}$ is agent planning time, $T_{\text{train}} = 300\text{s}$ is the fixed training budget, $T_{\text{collect}} \approx 60\text{s}$ is result collection, and $T_{\text{analyze}} \approx 60\text{s}$ is trend analysis. For a typical wave of $|\mathcal{G}_w| = 12$ experiments, this yields approximately 90 experiments per hour — a 9× throughput increase over the sequential rate of roughly 10 experiments per hour.

# Pseudocode: Wave-based factorial search (emergent agent behavior)
# Reconstructed from the blog post's description of agent actions

def parallel_autoresearch(clusters: list[str], program_rules: str):
    """
    The agent's emergent research loop, formalized as pseudocode.
    In practice, the agent reasons in natural language and constructs
    shell commands; this captures the logical structure.
    """
    history = ResultsLog("results.tsv")
    best_config = load_baseline_config("train.py")

    while True:  # "NEVER STOP" directive from program.md
        # Phase 1: Design factorial grid based on history
        factors = select_factors_to_vary(history, best_config)
        grid = build_factorial_grid(factors, max_size=len(clusters))

        # Phase 2: Submit all experiments in parallel (detached)
        for i, config in enumerate(grid):
            workdir = create_isolated_workdir(config, experiment_id=f"exp-{i}")
            # sky launch clusters[i] experiment.yaml --workdir workdir -d -y
            submit_async(clusters[i], workdir, config)

        # Phase 3: Wait for training to complete (~5 minutes)
        wait_for_completion(timeout=360)

        # Phase 4: Collect results from all clusters
        results = []
        for cluster in clusters:
            result = parse_logs(cluster)  # sky logs cluster
            results.append(result)

        # Phase 5: Analyze trends and interaction effects
        trends = identify_monotonic_trends(results)
        interactions = detect_interaction_effects(results)
        hw_effects = check_hardware_confounds(results, clusters)

        # Phase 6: Update best config and log results
        for r in results:
            if r.val_bpb < best_config.val_bpb:
                best_config = r.config
            history.append(r)

45.3.2 Emergent Hardware-Aware Strategy

The most scientifically significant finding is the agent's autonomous discovery and exploitation of hardware heterogeneity. The blog post captures the agent's reasoning in real time. The discovery unfolds in four stages:

Stage 1 — Anomaly detection. The agent observes that identical configurations produce systematically different val_bpb values on different clusters. It notes this inconsistency and investigates.

Stage 2 — Root cause identification. The agent checks sky status and discovers that three clusters (gpu-03, gpu-04, gpu-08) are H200s while the remaining thirteen are H100s. It hypothesizes that H200s are faster — completing more training steps in the fixed five-minute budget — and verifies: H200 runs approximately 9% more steps than H100.

Stage 3 — Strategy development. The agent reasons (quoting from the blog post): "Since H200 gets ~9% more steps than H100 in the same 5-minute budget, and I have only 3 H200 clusters, I should focus experiments on H200 clusters." It develops a two-tier workflow: screen hypotheses cheaply on the 13 H100 clusters, validate the top 2–3 on the 3 H200 clusters.

Stage 4 — Refinement. The agent discovers that parameter rankings sometimes differ across hardware types — a configuration that is best on H100 may not be best on H200. It adapts by treating H200 results as canonical and H100 results as preliminary screening.

The hardware performance difference can be modeled as follows. Let $S(h, t)$ denote the number of training steps completed by hardware type $h$ in time budget $t$. For the fixed $t = 300\text{s}$ budget:

Since val_bpb is a decreasing function of training steps (more steps $\rightarrow$ lower loss), a configuration measured on H200 will produce a systematically lower val_bpb than the same configuration on H100. Let $f(\mathbf{x}, s)$ be the validation loss for configuration $\mathbf{x}$ after $s$ steps. The hardware confound introduces a bias:

This bias is configuration-dependent — wider models have different step-count sensitivities than narrow ones — which is why the agent correctly identifies that rankings can invert across hardware types. This is a classical confounding variable in experimental design, and the agent's autonomous detection and handling of it mirrors sophisticated statistical reasoning.

45.3.3 Scaling Efficiency Analysis

The parallel system achieves 56% scaling efficiency (9× speedup on 16 GPUs rather than the ideal 16×). The overhead sources are identifiable:

Overhead Source	Estimated Contribution	Explanation
Wave design time	~12%	Agent spends longer planning factorial grids than single experiments
Result collection	~12%	Sequentially checking 16 cluster logs
Cluster provisioning	~5%	Initial setup and data download on new clusters
Inter-wave GPU idle	~10%	GPUs wait while agent analyzes results and plans next wave
Crashed experiments	~5%	Failed runs (OOM, numerical instability) waste GPU cycles

The primary bottleneck is the agent's reasoning time. With 16 GPUs, the agent is the serialization point — it must design, submit, collect, and analyze each wave sequentially. Scaling beyond 16 GPUs would require either faster agent reasoning, overlapping wave execution (submitting wave $w+1$ while wave $w$ is still running), or multi-agent coordination.

45.3.4 Experimental Design Progression

The agent's experimental methodology visibly matures over the eight-hour run. This progression mirrors the behavior of an experienced human researcher and can be characterized in five stages:

Wave Range	Strategy	Example	Sophistication
1–5	One-variable sweeps	10 values of weight_decay	Low (identical to sequential)
6–15	Multi-variable factorials	6 aspect ratios in one wave	Medium (exploiting parallelism)
16–25	Hardware-aware allocation	Screen on H100, validate on H200	High (emergent capability)
26–40	Two-tier screening/validation	Promote top 2–3 to H200 tier	High (research methodology)
40–75	Focused combinatorial tuning	2³ factorial over final parameters	High (diminishing-returns aware)

The transition from Phase 1 to Phase 2 is particularly informative. In the aspect ratio sweep, the agent tested AR $\in \{48, 64, 72, 80, 90, 96, 112\}$ in a single five-minute wave. The response surface is non-monotonic — performance improves with width up to AR=96, then degrades because the larger model completes fewer training steps in the fixed budget. Sequentially, the agent might have tested AR=64 (no improvement over baseline), possibly AR=72 (marginal), and potentially abandoned the direction. The parallel factorial revealed the complete curve, the optimal point, and the mechanistic explanation for degradation at AR=112.

45.4 Key Results

45.4.1 Headline Metrics

The following results are reported in the SkyPilot blog post (March 2026). All numbers are from a single eight-hour run and are therefore subject to the non-determinism inherent in LLM agent decision-making.

Metric	Value
Starting val_bpb	1.003 (baseline)
Final val_bpb	0.974
Total improvement	2.87% reduction
Total experiments submitted	~910
Experiments with valid results	~700
Runtime	~8 hours
GPUs used	16 (13 × H100 + 3 × H200)
Throughput	~90 experiments/hour
Estimated sequential equivalent	~72 hours
Wall-clock speedup	~9×

45.4.2 Five Phases of Discovery

The agent's research naturally organized into five phases. These were not pre-planned — each phase emerged from the results of the previous one. The return per experiment declined by approximately 20× from Phase 1 to Phase 5, following a pattern of diminishing returns characteristic of iterative optimization:

Phase	Experiments	Focus	Δ val_bpb	Key Finding
1. Hyperparameter Sweeps	~200	Batch size, betas, weight decay	0.022	batch_size 2¹⁸ > 2¹⁹ (more optimizer steps in fixed budget)
2. Architecture Discovery	~220	Model aspect ratio	0.004	ASPECT_RATIO=96 optimal (non-monotonic response surface)
3. Fine-Tuning	~140	Re-optimize for new architecture	0.002	Hyperparameters must be re-tuned after architectural changes
4. Optimizer Tuning	~140	Muon parameters	0.001	muon_beta2=0.98 smooths normalization for wider model
5. Diminishing Returns	~210	Remaining parameters	<0.001	Improvement curve flattened; further gains minimal

The return per experiment can be expressed as a power-law decline:

where $n$ is the experiment index, $c$ is a scaling constant, and $\alpha$ captures the rate of diminishing returns. From the blog post data: Phase 1 yields $r \approx 1.1 \times 10^{-4}$ per experiment, declining to $r < 4.8 \times 10^{-6}$ in Phase 5 — approximately a 23× reduction. This pattern suggests that a cost-aware stopping criterion (halt when expected improvement per dollar falls below a threshold) would be valuable, though the "NEVER STOP" directive from program.md overrides any such criterion.

45.4.3 Best Configuration Found

# Best configuration discovered by the parallel agent
# Source: SkyPilot blog post, reported from the 8-hour run

# Architecture
ASPECT_RATIO = 96          # model_dim = 8 * 96 = 768 (up from default 64)
DEPTH = 8                  # 8 transformer layers (unchanged)
WINDOW_PATTERN = "SL"      # alternating Sliding + Local attention

# Training
TOTAL_BATCH_SIZE = 2**18   # ~262K tokens/step (halved from default 2^19)

# Learning rates
MATRIX_LR = 0.05           # Muon LR for weight matrices (up from 0.04)
EMBEDDING_LR = 0.6         # AdamW LR for token embeddings
SCALAR_LR = 0.5            # AdamW LR for residual mixing scalars

# Optimizer
ADAM_BETAS = (0.70, 0.95)  # lower beta1 than default
WEIGHT_DECAY = 0.08        # reduced from 0.2
WARMDOWN_RATIO = 0.6       # increased from 0.5
FINAL_LR_FRAC = 0.05       # non-zero final LR
# Muon optimizer: momentum=0.95, ns_steps=5, beta2=0.98

45.4.4 Parallel vs. Sequential Comparison

Metric	Sequential (1 GPU)	Parallel (16 GPUs)	Ratio
Experiments/hour	~10	~90	9×
Information per decision	1 experiment	10–13 experiments	10–13×
Interaction effects found	Rarely	Routinely	Qualitative
Hardware-aware strategy	Impossible	Emergent	N/A
Estimated API cost	~$7	~$9	1.3×
Estimated GPU cost	~$144	~$263	1.8×
Estimated total cost	~$151	~$272	1.8×
Time to comparable result	~72 hours (estimated)	~8 hours	9× faster

Note: The sequential estimates are extrapolations from the blog post's analysis, not from a controlled sequential run. The 72-hour sequential estimate assumes the same agent would eventually discover the same improvements, which is not guaranteed given greedy hill-climbing's sensitivity to search order. The cost figures use reported GPU rates (~$2.00/hr for H100, ~$2.30/hr for H200) and estimated API usage.

45.5 Implementation Details

45.5.1 SkyPilot Task Definition

The infrastructure-as-code layer is defined in experiment.yaml (from skypilot/examples/autoresearch/). Several design choices in this file are worth examining:

# experiment.yaml — SkyPilot task definition
# Source: github.com/skypilot-org/skypilot/tree/master/examples/autoresearch

# Resources: accept either GPU type; SkyPilot picks based on availability
# resources:
#   accelerators: {H100:1, H200:1}
#   image_id: docker:nvcr.io/nvidia/pytorch:24.07-py3
#   infra: k8s

# Run block with structured output for reliable result parsing:
# run: |
#   uv run train.py 2>&1 | tee run.log
#   EXIT_CODE=${PIPESTATUS[0]}
#   if [ $EXIT_CODE -ne 0 ]; then
#     echo "EXPERIMENT_STATUS: crash"
#   else
#     VAL_BPB=$(grep "^val_bpb:" run.log | awk '{print $2}')
#     PEAK_VRAM=$(grep "^peak_vram_mb:" run.log | awk '{print $2}')
#     MEMORY_GB=$(echo "scale=1; ${PEAK_VRAM} / 1024" | bc)
#     echo "EXPERIMENT_STATUS: done"
#     echo "EXPERIMENT_RESULT: ${EXPERIMENT_ID} val_bpb=${VAL_BPB} memory_gb=${MEMORY_GB}"
#   fi

# Key design choices:
# - {H100:1, H200:1}: either type acceptable; creates heterogeneous fleet
# - PIPESTATUS[0]: captures train.py exit code (not tee's)
# - Structured output (EXPERIMENT_STATUS, EXPERIMENT_RESULT): enables parsing
# - setup block runs once per cluster; sky exec reuses it

The use of {H100:1, H200:1} as the resource specification is the direct cause of the heterogeneous fleet — SkyPilot allocates whichever GPU type is available from the Kubernetes cluster. The agent was never informed that it would receive a mixed fleet, making the subsequent discovery of hardware heterogeneity a genuinely emergent behavior.

45.5.2 SkyPilot CLI as Agent API

The SkyPilot CLI effectively serves as a natural-language-accessible API for the agent. The mapping between agent actions and CLI commands is learned from the skill document, not programmed:

Agent Action	CLI Command	Key Flags
Provision cluster + run job	sky launch <name> <yaml>	-d -y --workdir --env
Queue job on existing cluster	sky exec <name> <yaml>	-d --env
Check results	sky logs <name> [job_id]
Monitor fleet	sky status
Check job queue	sky queue <name>
Tear down cluster	sky down <name>	-y

SkyPilot supports over 20 infrastructure backends (AWS, GCP, Azure, Kubernetes, SLURM, Nebius, and others), meaning the agent's learned infrastructure skills transfer across cloud providers without modification. The infra: field in the YAML specification is the only change required to switch backends.

45.5.3 Cost Analysis

The blog post reports the following cost breakdown for the eight-hour parallel run:

Resource	Units	Hours	Estimated Rate	Estimated Cost
H100 GPUs	13	8	~$2.00/hr	~$208
H200 GPUs	3	8	~$2.30/hr	~$55
Claude Code API	~2.5M tokens	8	Anthropic rates	~$9
Total				~$272–300

The cost per experiment is approximately $0.30 ($0.29 GPU + $0.01 API), which is remarkably low. The API cost is negligible relative to GPU cost — the LLM agent adds roughly 3% to the total cost while providing full autonomous operation. For comparison, a human researcher performing the same 910 experiments manually would cost significantly more in salary alone, even before accounting for the cognitive overhead of managing 16 clusters.

45.5.4 Reproducibility

The system provides a one-line setup script (from the SkyPilot examples directory):

# One-line setup (from the blog post):
# curl -sL https://raw.githubusercontent.com/skypilot-org/skypilot/
#   master/examples/autoresearch/setup.sh | bash
# cd autoresearch
# claude "Read instructions.md and start running parallel experiments."

# Manual setup:
# 1. Clone both repositories
# git clone https://github.com/karpathy/autoresearch.git
# git clone https://github.com/skypilot-org/skypilot.git
# cd autoresearch
#
# 2. Copy SkyPilot experiment files
# cp ../skypilot/examples/autoresearch/experiment.yaml .
# cp ../skypilot/examples/autoresearch/instructions.md .
#
# 3. Prepare data locally
# pip install uv && uv sync && uv run prepare.py
#
# 4. Install SkyPilot skill, then launch agent

Because the LLM agent makes non-deterministic decisions, exact results will vary across runs. However, the blog post argues that qualitative findings should reproduce: batch size reduction is consistently discovered early, model width scaling provides the largest architectural gain, diminishing returns appear after ~500 experiments, and hardware heterogeneity is independently discovered when present. Quantitative values (exact val_bpb, experiment counts per phase) will differ across runs, agents, and hardware configurations.

Reproducibility Factor	Assessment
Code availability	Full source, open-source (SkyPilot repo + autoresearch repo)
Instructions	Complete instructions.md, self-contained
Data	Public dataset (climbmix-400b)
Compute requirements	GPU cluster (Kubernetes or any SkyPilot backend)
Agent determinism	Non-deterministic (LLM agent)
Infrastructure determinism	Variable (GPU availability, scheduling, hardware mix)

45.6 Context Management and Memory

The parallel execution model places substantially greater demands on the agent's context window compared to sequential operation. Each wave generates 15–25K tokens of context (designing experiments, submitting commands, collecting results, analyzing trends), compared to roughly 3K tokens per sequential cycle. Over 75 waves across eight hours, the cumulative context approaches 1.1 million tokens — approximately 3–4× more than the sequential agent's ~300K tokens for 100 experiments.

This context pressure is a potential scaling bottleneck. As the conversation history grows, the agent may lose access to early experimental results, potentially leading to redundant exploration or failure to recall previously discovered trends. The blog post does not report any explicit context management strategy (such as summarization or external memory), suggesting the agent relies on Claude Code's native context window and any built-in context compression.

The results.tsv file serves as a form of external memory — a structured log that the agent can re-read to recall past experiments without relying on its context window. This is functionally similar to the "learning log" pattern seen in more sophisticated evolutionary systems, though it lacks the semantic structure or embedding-based retrieval of dedicated knowledge management components.

45.7 Comparative Analysis

SkyPilot Autoresearch occupies a distinctive position in the landscape of parallel automated optimization systems. It is neither a classical hyperparameter optimization framework (like Optuna or Ray Tune) nor a purpose-built evolutionary search system (like AlphaEvolve). Instead, it is an infrastructure intervention that transforms agent behavior emergently.

System	Parallelism Model	Controller	Search Strategy	Hardware Awareness
Grid Search / Optuna	Embarrassingly parallel	Bayesian / RL	Principled optimization	Manual configuration
AlphaEvolve	100+ parallel evaluators	Evolutionary controller	MAP-Elites + islands	Managed internally
Autoresearch (1 GPU)	Sequential	LLM agent	Greedy hill-climbing	Single GPU
SkyPilot Autoresearch	16-GPU cluster	LLM agent	Factorial grids (emergent)	Emergent

The critical distinction is that parallelism in AlphaEvolve is engineered into the system design — the evolutionary controller, island topology, and evaluation pipeline are built for parallel operation. In SkyPilot Autoresearch, parallelism is an infrastructure capability that the agent discovers how to exploit. The agent was not programmed to run factorial grids or develop hardware-aware strategies; these behaviors emerged from the interaction between a capable LLM agent and a parallel execution environment.

This positions SkyPilot Autoresearch as a natural experiment in agent intelligence: give the same agent more resources and observe whether its behavior changes qualitatively. The answer — that it does, developing experimental methodology that mirrors classical design-of-experiments theory — has implications that extend beyond this specific system.

45.8 Limitations and Discussion

45.8.1 Structural Limitations

Limitation	Description	Impact
Single-agent bottleneck	One Claude Code instance manages all 16 GPUs	56% scaling efficiency; agent reasoning time is the serialization point
No inter-experiment communication	Clusters run independently with no shared state	Redundant computations possible; no early termination across clusters
Context window pressure	~1.1M cumulative tokens over 8 hours	Risk of losing early experimental context
No stopping criterion	"NEVER STOP" directive; runs until manually halted	Wastes compute during diminishing-returns phase
Non-deterministic results	Different LLM decisions each run	Exact quantitative results are not reproducible
Kubernetes dependency	Requires cluster infrastructure	Higher barrier than single-GPU autoresearch
Agent-specific	Tested only with Claude Code	May not generalize to other LLM agents

45.8.2 Methodological Concerns

The most significant methodological limitation is the absence of controlled replication. The reported results come from a single eight-hour run. Without multiple independent runs, it is impossible to distinguish between robust emergent behaviors and artifacts of a particular sequence of LLM decisions. The blog post argues that qualitative findings (batch size reduction, aspect ratio optimization, hardware awareness) should reproduce, but this claim has not been empirically verified across runs.

Additionally, the comparison between parallel and sequential performance relies on extrapolation. The 72-hour sequential estimate assumes the agent would eventually discover the same improvements through greedy hill-climbing, which is not guaranteed. Sequential search is more sensitive to search order — a suboptimal early decision could lead the agent into a different region of the configuration space. The factorial grid's ability to map complete response surfaces in one wave is not just faster but informationally superior.

The emergent hardware-aware strategy, while striking, benefits from a specific circumstance: the H100/H200 performance difference is large enough (~9%) to produce visible confounds in the results. With a more homogeneous cluster, this emergent behavior would not appear, and the system's scientific narrative would be correspondingly weaker. Whether agents would discover subtler confounds (e.g., 2–3% performance variation due to thermal throttling or memory bandwidth differences) remains an open question.

45.8.3 Scaling Projections

The blog post suggests several natural extensions that would address current limitations:

Multi-agent coordination. Replace the single agent with $k$ agents each managing $N/k$ GPUs, exploring different regions of the search space with periodic sharing of discoveries. This would remove the single-agent bottleneck but introduce coordination challenges (conflict resolution, communication overhead, credit assignment).

Adaptive compute allocation. Dynamically scale GPU count based on expected return per experiment — start with few GPUs for broad exploration, scale up when promising directions are found, scale down when diminishing returns are detected. This requires an explicit utility model and stopping criterion.

Cross-run persistence. Store results in a structured database across runs and use Bayesian surrogate models to guide subsequent exploration. The factorial data from parallel runs provides excellent training signal for response surface models.

45.9 Broader Significance

45.9.1 Compute as a Cognitive Amplifier

The most important finding of SkyPilot Autoresearch is that additional compute does not merely make the agent faster — it makes the agent smarter, by enabling strategies that are structurally impossible in sequential mode. A single data point per decision cycle restricts the agent to greedy local search. Ten to thirteen data points per cycle enable factorial design, trend detection, interaction analysis, and confound identification.

This has a precise formal interpretation. The agent's decision quality depends on the amount of information available per decision cycle. Let $I_w$ denote the information content of wave $w$, measured in bits. For a single experiment with binary outcome (keep/discard), $I_w \leq 1$ bit. For a factorial grid of $|\mathcal{G}_w|$ experiments, the information content scales as:

where the first term captures main effects and the second captures pairwise interaction effects, each contributing $I_{\text{interaction}}$ bits. With 12 experiments per wave, the agent can observe up to 12 main effects and 66 pairwise interactions — a qualitatively richer information landscape than a single binary signal.

45.9.2 Infrastructure as an Intelligence Bottleneck

The system demonstrates that infrastructure constraints can mask agent capabilities. The sequential autoresearch agent appears to use greedy hill-climbing because that is the only strategy available to it — not because it lacks the ability to reason about experimental design. When the infrastructure constraint is removed, sophisticated experimental methodology emerges without any change to the agent itself. This suggests that evaluations of LLM agent capabilities should account for the infrastructure substrate — an agent that appears limited on one GPU may be significantly more capable on sixteen.

45.9.3 Design of Experiments by AI

The agent's independent rediscovery of principles from classical experimental design theory — factorial grids (Fisher, 1935), blocking for confounding variables, response surface methodology, two-tier screening/validation — is a notable demonstration of LLM scientific reasoning. These principles took human statisticians decades to formalize. The agent rediscovered and applied them from first principles by observing its own results, suggesting that LLMs have internalized substantial experimental design knowledge from their training data.

45.10 Future Directions

Several extensions are suggested by the blog post and the experimental findings:

Multi-agent parallel autoresearch would replace the single agent managing 16 GPUs with multiple agents each managing a subset, exploring different regions of the search space with periodic sharing of discoveries. This would remove the single-agent serialization bottleneck and potentially enable even more sophisticated collective strategies.

Adaptive compute scaling would dynamically adjust the number of GPUs based on expected return per experiment. Start with a small fleet for broad exploration, scale up when promising directions are found, and scale down when diminishing returns are detected. This requires an explicit cost-utility model — halt when expected improvement per dollar drops below a threshold.

Cross-domain transfer is natural because the SkyPilot infrastructure layer is domain-agnostic. Only train.py and prepare.py would change to apply the parallel autoresearch methodology to reinforcement learning, compiler optimization, scientific simulation parameters, or other optimization domains.

Population-based training would replace greedy hill-climbing with a maintained population of configurations that evolve over time. Parallel resources make this natural — each GPU runs a different population member. This would bring autoresearch closer to systems like AlphaEvolve while preserving the simplicity of the instruction-based approach.