Arcgentica

18.1 Overview & Motivation

The Abstraction and Reasoning Corpus (ARC), introduced by François Chollet in 2019, remains one of the most demanding benchmarks for evaluating machine intelligence. Unlike language modeling or image classification, ARC tasks require discovering transformation rules from a handful of input–output grid pairs and generalizing those rules to unseen test inputs. ARC-AGI-2, the 2024 revision, raises the difficulty further with tasks demanding multi-step spatial reasoning, object manipulation, and compositional rule application. As of early 2026, state-of-the-art systems achieve only single-digit percentages on the hardest ARC-AGI-2 subsets, while average human performance exceeds 85% (Chollet, 2024).

Arcgentica, developed by Symbolica AI and released under the Apache 2.0 license, approaches ARC-AGI-2 through a paradigm the authors call runtime-as-context evolutionary program synthesis. Rather than treating an LLM as a direct solver that generates a solution in a single pass, Arcgentica uses the LLM as an informed mutation operator within a population-based evolutionary search. The central design choice is that each mutation is conditioned not only on the parent program's source code but on detailed runtime execution traces — intermediate variable states, produced outputs, exception information, and cell-level diffs against expected results. [Blog/README-reported]

This design addresses a well-documented weakness of large language models: their inability to accurately simulate program execution mentally. LLMs frequently err when asked to predict what a function produces for a given input, particularly for loops with complex index arithmetic, nested conditionals, and array operations. By providing actual execution results as context, Arcgentica shifts the LLM's task from "understand what this code does and fix it" to "given what this code actually does, decide what to change." [Blog/README-reported]

Arcgentica sits at the intersection of four research traditions. From genetic programming (Koza, 1992), it inherits the population-based search over programs with selection pressure favoring higher fitness. From program synthesis, it takes the problem formulation: induce a function from input–output specifications. From LLM-based code generation, it draws the mutation operator — frontier language models that generate syntactically valid, semantically meaningful code modifications. And from automated program repair (APR), it borrows the core technique of using execution traces and failing-test diffs to localize faults and guide repairs (Le Goues et al., 2012; Liu et al., 2019). Prior work in APR and trace-guided debugging has explored subsets of this combination; Arcgentica's contribution is best understood as the specific integration of all four applied to abstract reasoning tasks.

18.2 Architecture

The system follows an evolutionary loop architecture with seven principal components: a task parser, a population manager, an execution engine with trace capture, a fitness evaluator, a context builder, an LLM mutation engine, and a top-level orchestrator. Each ARC-AGI-2 task is treated independently: a fresh population is initialized, evolved for a configurable number of generations, and the best-performing individual is submitted as the solution. [Blog/README-reported]

18.2.1 Repository Structure and Component Mapping

The Arcgentica repository (github.com/symbolica-ai/arcgentica) is organized as a Python package with the following verified top-level structure [Repo-verified at commit d7e4b2f]:

# Repository excerpt — top-level structure (commit d7e4b2f, 2025-06-12)
arcgentica/
├── arcgentica/
│   ├── __init__.py
│   ├── main.py              # Entry point: orchestrates evolutionary loop
│   ├── evolve.py            # Core evolutionary search logic
│   ├── execute.py           # Sandboxed execution with trace capture
│   ├── evaluate.py          # Fitness evaluation (cell accuracy, composite scoring)
│   ├── mutate.py            # LLM mutation engine and prompt construction
│   ├── population.py        # Population management, selection, diversity
│   ├── task.py              # ARC task parsing and grid encoding
│   ├── models.py            # LLM provider abstraction (Anthropic, OpenAI, Google)
│   ├── config.py            # Configuration dataclass with defaults
│   ├── trace.py             # Runtime trace data structures
│   └── utils.py             # Grid formatting, diff computation, helpers
├── configs/
│   └── default.yaml         # Default hyperparameters
├── pyproject.toml            # Package metadata; entry point: arcgentica.main
├── requirements.txt
└── README.md

The following table maps each described architectural component to its verified implementation location:

Component	Module	Key Identifiers	Verification Status
Task Parser	arcgentica/task.py	Task dataclass, load_task()	[Repo-verified]
Execution Engine	arcgentica/execute.py	execute_program(), ExecutionResult	[Repo-verified]
Trace Capture	arcgentica/trace.py, execute.py	TraceData dataclass, output capture in execute_program()	[Repo-verified]
Fitness Evaluator	arcgentica/evaluate.py	evaluate_candidate(), compute_fitness()	[Repo-verified]
Context Builder	arcgentica/mutate.py	build_prompt(), prompt template strings	[Repo-verified]
LLM Mutation Engine	arcgentica/mutate.py, models.py	generate_mutation(), LLMClient	[Repo-verified]
Population Manager	arcgentica/population.py	Population class, select_parent(), cull()	[Repo-verified]
Orchestrator	arcgentica/evolve.py	evolve_task(), generation loop	[Repo-verified]
Configuration	arcgentica/config.py, configs/default.yaml	Config dataclass	[Repo-verified]

18.2.2 Verified Configuration Defaults

The Config dataclass in arcgentica/config.py and the configs/default.yaml file expose the following defaults [Repo-verified]:

Parameter	Default Value	Source
Population size	30	config.py: population_size = 30
Max generations	50	config.py: max_generations = 50
Execution timeout	5.0 seconds	config.py: timeout_seconds = 5.0
Elite fraction	0.1 (top 10%)	config.py: elite_fraction = 0.1
Tournament size	3	config.py: tournament_size = 3
Offspring per generation	10	config.py: offspring_per_gen = 10
Max concurrent LLM calls	5	config.py: max_concurrent = 5
Default model	claude-3-5-sonnet-20241022	default.yaml
Per-task budget cap	$5.00	config.py: per_task_budget = 5.0
Stagnation threshold	5 generations	config.py: stagnation_gens = 5

18.2.3 Data Flow

The data flow is cyclical within the evolutionary loop. Task I/O pairs flow into the execution engine, which produces runtime traces. These traces, together with parent source code, flow into the context builder, which assembles prompts for the LLM mutation engine. The LLM produces new candidate programs (offspring), which are added to the population via the population manager. The population manager feeds candidates back to the execution engine, closing the loop. Fitness scores from the evaluator inform both parent selection and population culling decisions. [Blog/README-reported; confirmed by code structure in evolve.py]

The orchestrator in evolve.py coordinates this loop asynchronously: multiple LLM mutations are dispatched concurrently via asyncio.gather(), with a semaphore (default: 5 concurrent calls) controlling parallelism. The orchestrator enforces per-task budget caps, triggers early termination when a perfect solution is found, and manages adaptive strategy selection based on population fitness statistics. [Blog/README-reported; async structure confirmed in evolve.py]

18.3 Core Algorithms

18.3.1 The Runtime-as-Context Mechanism

The defining algorithmic innovation is the systematic capture and use of runtime execution traces as conditioning context for LLM-guided mutation. When a candidate program is executed against a training example, the system captures four categories of information [Blog/README-reported; data structures confirmed in trace.py]:

Category	Contents	Implementation
Variable snapshots	Values of local variables at key program points	TraceData.variable_snapshots
Output comparison	Cell-by-cell diff between actual and expected output grids	utils.py: grid_diff()
Exception information	Error message, traceback, execution time	ExecutionResult.error, .traceback_text
Structural metadata	Output shape, loop iteration counts	TraceData.loop_iterations, output shape

The execution engine in execute.py runs candidate programs via Python's exec() with a restricted namespace and a 5-second default timeout. The following is a trimmed excerpt showing the core execution path [Repo-verified, arcgentica/execute.py]:

# Repository excerpt — arcgentica/execute.py (trimmed for length)
# Core execution path with trace capture

import numpy as np
import traceback as tb
from arcgentica.trace import TraceData

ALLOWED_NAMESPACE = {
    "np": np,
    "numpy": np,
    "__builtins__": {
        "range": range, "len": len, "int": int, "float": float,
        "list": list, "tuple": tuple, "dict": dict, "set": set,
        "min": min, "max": max, "sum": sum, "abs": abs,
        "enumerate": enumerate, "zip": zip, "map": map,
        "sorted": sorted, "reversed": reversed, "bool": bool,
        "isinstance": isinstance, "print": lambda *a, **k: None,
    },
}

def execute_program(source_code: str, input_grid: np.ndarray,
                    timeout_sec: float = 5.0) -> "ExecutionResult":
    """Execute candidate in restricted namespace with trace capture."""
    trace = TraceData()
    try:
        compiled = compile(source_code, "<candidate>", "exec")
        namespace = {**ALLOWED_NAMESPACE}
        with _timeout(timeout_sec):
            exec(compiled, namespace)
            transform_fn = namespace.get("transform")
            if transform_fn is None:
                return ExecutionResult(
                    success=False, output=None, trace=trace,
                    error="No 'transform' function defined")
            output = transform_fn(input_grid.copy())
        trace.final_output = np.asarray(output)
        return ExecutionResult(success=True, output=trace.final_output,
                               trace=trace)
    except TimeoutError:
        trace.error = "Execution timed out"
        return ExecutionResult(success=False, output=None, trace=trace,
                               error="Timeout")
    except Exception as e:
        trace.error = str(e)
        trace.traceback_text = tb.format_exc()
        return ExecutionResult(success=False, output=None, trace=trace,
                               error=str(e))

The grid diff computation in utils.py produces a human-readable representation where matching cells are shown normally and mismatches are highlighted as [actual→expected]. This allows the LLM to see precisely which spatial positions are incorrect [Repo-verified, arcgentica/utils.py: grid_diff()]:

# Repository excerpt — arcgentica/utils.py (trimmed)
# Cell-level diff between actual and expected grids

def grid_diff(actual: np.ndarray, expected: np.ndarray) -> str:
    """Produce human-readable diff. Mismatches highlighted as [a->e]."""
    if actual.shape != expected.shape:
        return f"Shape mismatch: got {actual.shape}, expected {expected.shape}"
    lines = []
    for i in range(expected.shape[0]):
        row = []
        for j in range(expected.shape[1]):
            if actual[i, j] == expected[i, j]:
                row.append(f" {actual[i,j]} ")
            else:
                row.append(f"[{actual[i,j]}->{expected[i,j]}]")
        lines.append(" ".join(row))
    return "\n".join(lines)

# Example output for a 5×5 grid:
#  0   0   0   0   0
#  0  [3->7]  0   0   0
#  0   0   2   0   0
#  0   0   0  [3->7]  0
#  0   0   0   0   0

Note on trace instrumentation granularity: the TraceData dataclass in trace.py includes fields for variable_snapshots and loop_iterations, but the mechanism by which these are populated at runtime (AST rewriting, sys.settrace, decorator injection, or manual instrumentation within generated code) varies. The primary trace information used in prompt construction consists of the final output, the grid diff, and exception data. Detailed variable-level snapshots appear to be populated opportunistically rather than via systematic instrumentation. [Repo-verified for data structure; instrumentation depth is partially documented]

18.3.2 Evolutionary Search Loop

Each ARC-AGI-2 task is solved by an independent evolutionary run. The core loop in evolve.py proceeds through discrete generations [Repo-verified]:

# Repository excerpt — arcgentica/evolve.py (trimmed, core loop)

async def evolve_task(task: Task, config: Config) -> Individual:
    """Evolve a population of candidate programs for a single task."""
    population = Population(max_size=config.population_size,
                            elite_fraction=config.elite_fraction)

    # Generation 0: seed via LLM zero-shot
    seeds = await generate_seeds(task, config)
    for seed in seeds:
        seed.fitness = evaluate_candidate(seed, task, config)
        population.add(seed)

    for gen in range(config.max_generations):
        # Check early termination
        best = population.best()
        if best and best.fitness >= 1.0:
            return best  # Perfect solution found

        # Select parents and generate offspring
        strategy = select_strategy(population, gen, config)
        parents = [population.select_parent(strategy)
                   for _ in range(config.offspring_per_gen)]
        offspring = await generate_mutations(parents, task, strategy, config)

        # Evaluate and add offspring
        for child in offspring:
            child.fitness = evaluate_candidate(child, task, config)
            population.add(child)

        population.cull()  # Enforce max size with diversity preservation

    return population.best()

The search is formalized as an optimization over the space of valid Python programs [Author-formalization]:

where $\mathcal{P}$ denotes the set of syntactically valid Python programs defining a function transform(grid) → grid that terminates within the timeout, $T = \{(x_i, y_i)\}_{i=1}^{N}$ is the set of $N$ training examples, and $F$ is the composite fitness function (§18.3.3). Each LLM mutation samples from a conditional distribution [Author-formalization]:

where $R(p_{\text{parent}}, T) = \{(r_i, d_i, e_i)\}_{i=1}^{N}$ is the collected runtime traces (with $r_i$ the variable snapshots, $d_i$ the output diff, and $e_i$ any exception information for training example $i$), and $M$ is the mutation instruction type (§18.3.4).

18.3.3 Composite Fitness Function

The fitness function in evaluate.py provides gradient signal for partial solutions, distinguishing "completely wrong" from "mostly wrong with a few cells correct." The implementation uses a composite of four components [Repo-verified, arcgentica/evaluate.py: compute_fitness()]:

# Repository excerpt — arcgentica/evaluate.py (trimmed)

def compute_fitness(cell_accuracies: list[float],
                    shapes_ok: list[bool],
                    exec_ok: list[bool]) -> float:
    """Composite fitness: worst-case + mean cell accuracy + shape + exec."""
    if not cell_accuracies:
        return 0.0

    # Perfect solution shortcut
    if all(ca == 1.0 for ca in cell_accuracies):
        return 1.0

    w_min, w_avg, w_shape, w_exec = 0.4, 0.3, 0.2, 0.1

    score = (
        w_min * min(cell_accuracies)
        + w_avg * (sum(cell_accuracies) / len(cell_accuracies))
        + w_shape * (1.0 if all(shapes_ok) else 0.0)
        + w_exec * (1.0 if all(exec_ok) else 0.0)
    )
    return score

The verified weight values are $w_1 = 0.4$, $w_2 = 0.3$, $w_3 = 0.2$, $w_4 = 0.1$ [Repo-verified]. Formally:

where the component functions are:

• $\text{cell\_acc}(p, x_i, y_i) = \frac{1}{|y_i|}\sum_{j,k} \mathbf{1}[p(x_i)_{j,k} = (y_i)_{j,k}]$ — fraction of correctly predicted cells for training example $i$. Defined as zero when output and expected shapes differ.
• $\overline{\text{cell\_acc}}(p) = \frac{1}{N}\sum_{i=1}^{N} \text{cell\_acc}(p, x_i, y_i)$ — mean cell accuracy across all $N$ training examples.
• $\text{shape\_match}(p) = \mathbf{1}[\forall i:\; p(x_i).\text{shape} = y_i.\text{shape}]$ — binary, task-level.
• $\text{no\_error}(p) = \mathbf{1}[\forall i:\; p(x_i) \text{ terminates without exception}]$ — binary, task-level.

The $\min$ term ensures generalization across all training examples: a program scoring perfectly on three examples but failing on the fourth will have $\min_i \text{cell\_acc} = 0$. The shape-match and no-error terms provide coarse gradient signal even when cell accuracy is zero, creating a fitness ladder: crash → run but wrong shape → right shape but wrong cells → partially correct → fully correct. [Blog/README-reported for design rationale; weights verified in code]

18.3.4 Mutation Strategies and Prompt Construction

Unlike classical genetic programming, Arcgentica's mutations are semantically informed LLM generations conditioned on runtime context. The mutate.py module implements multiple mutation types, each with a corresponding prompt template [Repo-verified]:

Mutation Type	Description	Trigger
Targeted Fix	Fix specific errors identified via runtime traces	Parent fitness > 0.7 with identifiable cell-level errors
Structural Rewrite	Rewrite a major section of the approach	Parent fitness 0.3–0.7 with structurally flawed approach
Strategy Shift	Completely change the algorithmic approach	Parent fitness < 0.3, or stagnation detected
Refinement	Minimal tweaks for edge cases, boundary conditions	Parent fitness > 0.9 (nearly solved)
Crossover	Combine elements from two parent programs	Population has diverse partial solutions

The prompt construction in mutate.py: build_prompt() assembles five sections. The following shows the verified structure [Repo-verified, arcgentica/mutate.py]:

# Repository excerpt — arcgentica/mutate.py (prompt construction, trimmed)

SYSTEM_PROMPT = (
    "You are an expert Python programmer specializing in grid transformation "
    "problems. You write clean, efficient numpy code.\n\n"
    "Rules:\n"
    "1. Write a single function: def transform(grid: np.ndarray) -> np.ndarray\n"
    "2. Use only numpy operations and Python standard library\n"
    "3. The function must be deterministic\n"
    "4. Return a numpy array with integer values 0-9\n"
    "5. Analyze the runtime traces carefully to understand what the current "
    "program does wrong, then fix it precisely"
)

MUTATION_INSTRUCTIONS = {
    "targeted_fix": (
        "Fix the specific errors shown in the runtime analysis. "
        "Focus on cells that are wrong. Make minimal changes."
    ),
    "structural_rewrite": (
        "The current approach has fundamental issues. Rewrite the core "
        "algorithm while keeping any parts that work correctly."
    ),
    "strategy_shift": (
        "Try a completely different approach. Analyze the input/output "
        "patterns from scratch and write a new program."
    ),
    "refinement": (
        "The program is nearly correct. Make precise, minimal changes "
        "to handle the remaining edge cases."
    ),
}

def build_prompt(parent, task, traces, mutation_type):
    """Assemble LLM prompt: system + task I/O + parent + traces + instruction."""
    parts = [format_task_pairs(task.train)]
    parts.append(f"\n## Current Program (fitness: {parent.fitness:.3f})\n")
    parts.append(f"```python\n{parent.source_code}\n```\n")

    # Runtime traces — the core innovation
    parts.append("\n## Runtime Analysis\n")
    for i, (trace, pair) in enumerate(zip(traces, task.train)):
        parts.append(f"\n### Training Example {i+1}:")
        if trace.error:
            parts.append(f"EXECUTION FAILED: {trace.error}")
            if trace.traceback_text:
                parts.append(trace.traceback_text[-500:])
        elif trace.final_output is not None:
            diff = grid_diff(trace.final_output, pair.output)
            parts.append(f"Output diff:\n{diff}")

    parts.append(f"\n## Instruction\n{MUTATION_INSTRUCTIONS[mutation_type]}")
    parts.append("\nProvide the complete improved function: def transform(grid):")
    return "\n".join(parts)

18.3.5 Adaptive Search Strategy

The evolutionary search dynamically adjusts its behavior based on population fitness statistics. The select_strategy() function in evolve.py implements the following logic [Repo-verified]:

Strategy	Condition	Behavior
Intensification	Best fitness > 0.95	All mutations target the near-solution with refinement
Exploitation	Best fitness > 0.7	Targeted fixes and refinements on top candidates
Balanced	Default (no special condition)	Mixed mutation types, standard selection
Exploration	Stagnation detected (no improvement for stagnation_gens = 5 generations)	Strategy shifts, increased diversity pressure
Restart	Extended stagnation (> 2× stagnation_gens) with best fitness < 0.1	Discard population, reseed from scratch

Stagnation is detected by comparing the best fitness at the current generation against the best fitness stagnation_gens generations ago. The improvement epsilon used for this comparison is 0.01 — if best_now - best_prev < 0.01, the generation counts as stagnant. [Repo-verified, evolve.py]

18.3.6 Parent Selection and Diversity Preservation

Parent selection in population.py uses a probabilistic mixture of strategies [Repo-verified]:

• Tournament selection (40% probability): Sample 3 individuals (default tournament_size), select the fittest.
• Fitness-proportionate selection (30%): Selection probability proportional to fitness + 0.01 epsilon.
• Diversity-weighted selection (20%): Combines fitness and behavioral uniqueness with equal weight (0.5/0.5).
• Uniform random selection (10%): Pure exploration.

Diversity is maintained at two levels. Code-level diversity uses token-set Jaccard similarity between program source texts. Behavioral diversity tracks whether programs produce different output fingerprints on the same inputs — two programs with different code but identical outputs are treated as duplicates. The cull() method in Population preserves elites (top 10%) and fills remaining slots preferring behaviorally distinct candidates. [Repo-verified, population.py]

18.4 Key Results

This section reports claims drawn exclusively from the Symbolica AI blog post "Runtime as Context." No independent reproduction was performed by the survey author. All numbers should be treated as source-reported claims, not independently verified measurements.

18.4.1 Source-Reported Performance Claims

Claim	Source	Missing Metadata
Direct LLM prompting achieves ~2–3% on ARC-AGI-2	Blog post	Task subset, model version, retry budget not specified
LLM + basic retry achieves ~3–5%	Blog post	Number of retries, budget matching not specified
Arcgentica achieves ~5–8% on ARC-AGI-2	Blog post	Task subset, run count, model, budget, stopping criteria all unspecified
~2–3× improvement over direct prompting	Blog post (derived)	Whether baselines used equivalent total LLM token budget: unspecified

The qualitative ordering — evolutionary search with traces outperforms retry without traces, which outperforms single-shot — is consistent with the described mechanism. The exact magnitudes cannot be independently assessed because the blog post does not report sufficient evaluation metadata.

18.4.2 Source-Reported Ablation Observations

The blog post describes directional observations about the contribution of individual components. These are qualitative observations, not controlled ablation results with statistical tests or confidence intervals. [Blog/README-reported]

Component Removed	Reported Direction	Unknowns
Runtime traces	Large performance drop (most impactful)	Task count, seeds, budget matching, model version
Population (single-program evolution)	Moderate performance drop	Whether total budget held constant
Diversity preservation	Smaller performance drop	Interaction with population size
Diff-based output representation	Minor performance drop	Isolation from other trace components

The reported ordering of component importance — traces > population > diversity > diff format — is the most defensible takeaway. Precise percentage drops sometimes cited (e.g., "40–60% for traces") appear in secondary material but lack documented provenance in the blog post itself; they should not be treated as measured effect sizes.

18.4.3 Missing Evaluation Metadata

The following metadata would be needed for independent assessment. All items are absent from the blog post and README.

Required Metadata	Impact
Benchmark subset (full ~400 tasks or subset?)	Cannot determine representativeness
Number of runs per task, random seeds	Cannot assess variance; LLM sampling is high-variance
LLM model version and API snapshot date	"Claude 3.5 Sonnet" may differ between months
Temperature, top_p, max_tokens	Significantly affect code generation quality
Per-task token/cost budget	Cannot compare fairly to baselines
Stopping criteria	Cannot replicate experimental conditions
Confidence intervals or variance	Cannot assess statistical reliability
Budget matching across conditions	Baselines may have used much less LLM compute

This places Arcgentica in the common position of open-source research systems: code is available, but the evidentiary bar is below peer-reviewed standards. The relative ordering of conditions is the most defensible takeaway; absolute percentages should be treated as rough indicators.

18.4.4 Worked Example: Runtime-as-Context in Action

The following illustrative example shows how the system processes a single evolutionary step for a simplified ARC-style task. [Author-constructed illustration of the verified mechanism; not from the repository.]

This example illustrates the core value proposition: without the runtime trace showing the IndexError at position (2,4) and the specific cell-level diffs, the LLM would need to mentally simulate array indexing across all examples to discover the boundary bug — a task that LLMs perform unreliably. With the trace, the fix is straightforward.

18.5 Implementation Details

18.5.1 LLM Model Support

The models.py module abstracts LLM provider access via an LLMClient class. The default.yaml configuration lists the default model as claude-3-5-sonnet-20241022. The README describes support for multiple providers [Repo-verified for default model; README-reported for full list]:

Model Family	Provider	Described Role
Claude 3.5 Sonnet / Claude 3 Opus	Anthropic	Primary mutation engine (default)
GPT-4o / GPT-4 Turbo	OpenAI	Alternative mutation engine
Gemini 1.5 Pro	Google	Long-context reasoning for large traces
Open-source models (e.g., DeepSeek-Coder)	Various	Cost-effective exploration

The blog discusses the possibility of routing different mutation types to different models (e.g., stronger models for strategy shifts, cheaper models for targeted fixes). The models.py module exposes a model parameter per call, making manual routing possible, but the evolve.py loop uses a single configured model by default. Automatic adaptive model routing does not appear to be implemented in the audited commit. [Repo-verified]

18.5.2 Context Window Management

The blog post describes five compression strategies for managing trace verbosity [Blog/README-reported]:

1. Selective snapshots: Include variable states only at loop boundaries and function entry/exit.
2. Diff-only mode: For high-fitness programs, show only differing cells rather than full grids.
3. Grid summarization: For large grids, display only the bounding box around changed regions.
4. Example prioritization: Full traces for failing examples; skip traces for passing examples.
5. Progressive detail: Start with summary traces; retry with more detail on failure.

The build_prompt() function in the audited commit implements strategies 2 (diff-only for high-fitness) and 4 (error-focused trace inclusion). The trace output is truncated to the last 500 characters for tracebacks. Full progressive-detail retry and grid summarization were not observed in the audited code path, though they may exist in auxiliary functions. [Repo-verified for observed strategies; blog-reported for full list]

18.5.3 Cost Analysis

The primary computational cost is LLM API calls; local Python execution is negligible. The following cost estimates are blog-reported claims, not independently verified:

Cost Component	Per Task (blog estimate)	Notes
LLM mutation calls	$0.50–$5.00	Depends on generations × population × model pricing
LLM ideation calls	$0.10–$0.50	Initial strategy generation
Local execution + trace capture	Negligible	CPU-only, sub-second per candidate
Total per task	$0.60–$5.50	Highly variable; blog estimates

For the full ARC-AGI-2 evaluation set (~400 tasks), blog-estimated total cost ranges from $200–$800 (standard) to $500–$2,000+ (exhaustive). The repository implements cost control via per-task budget caps (per_task_budget = 5.0 in config.py) and early termination on correct solutions. [Blog/README-reported for estimates; budget cap verified in code]

18.5.4 Reproduction Guide

The following reproduction protocol is based on verified repository artifacts [Repo-verified for install/entry point; user must supply API keys]:

# Step 1: Clone and pin commit
git clone https://github.com/symbolica-ai/arcgentica.git
cd arcgentica
git log -1 --format="Commit: %H  Date: %ci"

# Step 2: Install (verified from pyproject.toml)
pip install -e .
# or: pip install -r requirements.txt

# Step 3: Configure API key (required env var from config.py)
export ANTHROPIC_API_KEY="your-key-here"
# Optional: OPENAI_API_KEY, GOOGLE_API_KEY for multi-model

# Step 4: Run on a single task (entry point: arcgentica.main)
python -m arcgentica --task-id <task-id> --config configs/default.yaml

# Step 5: Inspect outputs
# Solutions written to: outputs/<task-id>/
# Logs: stdout (structured) with per-generation fitness

# Step 6: Record reproducibility metadata
python --version
git log -1 --format="%H"
# Record: model version from API response headers, total tokens, wall time

18.5.5 Execution Sandbox

Candidate programs are executed via Python's exec() with a restricted namespace (verified in execute.py, shown in §18.3.1). The ALLOWED_NAMESPACE dict explicitly whitelists NumPy, basic builtins (range, len, min, max, etc.), and stubs out print. This constitutes in-process namespace restriction, not a security sandbox [Repo-verified]:

• The restricted namespace prevents accidental access to open, os, subprocess, and other dangerous builtins, but Python introspection or attribute access on permitted objects (e.g., np.__builtins__) could potentially circumvent these restrictions.
• No container-level, VM-level, or subprocess-level isolation is implemented.
• Resource limits beyond the execution timeout are not enforced (no memory cap, no file-descriptor limit).

This is an acceptable trade-off for ARC-AGI-2, where generated code is constrained to grid transformations produced by trusted LLM providers. For adversarial contexts, additional sandboxing would be necessary.

18.6 Limitations & Discussion

18.6.1 Absolute Performance

Despite the reported relative improvement, Arcgentica's approximate ARC-AGI-2 score of 5–8% (per the blog post) remains far below human performance (~85%). This gap reflects fundamental limitations: the system can only discover solutions expressible as relatively short Python/NumPy functions that the LLM can plausibly generate given its training distribution. Tasks requiring deep compositional reasoning, novel mathematical insight, or long chains of spatial logic remain beyond reach.

18.6.2 No Cross-Task Learning

Each ARC-AGI-2 task is solved independently — the system does not transfer knowledge from solved tasks to unseen ones. If the same subroutine (e.g., "detect connected components" or "rotate object 90°") appears in multiple tasks, it must be rediscovered each time. The blog post identifies potential extensions — a solution library, retrieval-augmented generation, DSL induction — but none are implemented in the audited commit. [Blog/README-reported; absence confirmed in code]

18.6.3 Frozen LLM Weights

Arcgentica does not fine-tune the underlying language model. All "learning" occurs through evolutionary search and prompt content. While this preserves the LLM's generality and avoids fine-tuning cost, it means the system cannot internalize domain-specific code patterns. A fine-tuned mutation model trained on (parent, trace, improved_child) triples could potentially provide stronger mutations at lower cost, though this remains speculative.

18.6.4 Sensitivity to Trace Quality

The reported ablation finding that removing traces causes the largest single performance drop highlights a dependency: the system is highly reliant on trace informativeness. Programs that crash immediately produce minimal trace information — just an error message and stack trace — which may not provide enough diagnostic context. Additionally, more context is not always better in practice: trace verbosity, formatting, and compression strategy all affect whether additional information helps or hinders the LLM's reasoning.

18.6.5 Search Space Constraints

The implicit constraint to Python/NumPy programs biases the search toward solutions expressible in array-manipulation idioms. This may miss solutions more naturally expressed in terms of logical rules, constraint programming, or domain-specific abstractions. A hybrid approach — runtime-as-context mutations within a DSL — is a plausible future direction. [Blog/README-reported]

18.6.6 Evidence Quality

The system's quantitative claims rest on a blog post, with no peer-reviewed publication, no independent reproduction, and no controlled ablation study with reported statistical significance. The repository provides the implementation, but the evaluation methodology is not publicly documented in sufficient detail for independent verification (§18.4.3). This is common among open-source research systems but limits the strength of quantitative conclusions.

18.6.7 Comparison with Related Systems

System	Organization	Key Similarity	Key Difference
AlphaEvolve	Google DeepMind	Evolutionary LLM-guided code synthesis	Targets mathematical algorithm discovery; uses MAP-Elites archive, not simple populations
FunSearch	Google DeepMind	LLM + evolutionary search for programs	Targets combinatorial math; simpler fitness signal, no runtime traces
OpenEvolve	Open-source	Open-source evolutionary code generation	General-purpose; island-based topology; less emphasis on execution feedback
BARC	UC Berkeley	Program synthesis for ARC	DSL-based synthesis, not free-form Python
Greenblatt's approach	Independent	LLM-based ARC solving with iteration	Direct prompting with retry, not population-based evolution
GenProg / TBar	Academia	Execution traces and failing tests for fault localization and repair	Repair a single buggy program; do not use LLMs or population search

Arcgentica's most distinctive feature relative to FunSearch and AlphaEvolve is the explicit capture and utilization of runtime execution traces at mutation time. FunSearch and AlphaEvolve evaluate candidates and use fitness scores to guide selection, but they do not feed detailed execution dynamics back to the LLM during mutation. This makes Arcgentica closer in spirit to automated program repair (APR) systems — such as GenProg (Le Goues et al., 2012) and TBar (Liu et al., 2019) — but embedded within an evolutionary population-search framework.

The combination of evolutionary population search, LLM-based mutation, and runtime trace conditioning represents a distinctive integration point. While individual components have clear precedents, the specific combination applied to abstract reasoning tasks appears to be among the first open-source systems to fully integrate all three, based on available public documentation. BARC's DSL-based approach offers complementary strengths: constrained search space enabling more systematic enumeration but less flexibility for novel solutions.

18.7 Summary

References

1. Symbolica AI. Arcgentica: Runtime-as-Context Evolutionary Program Synthesis. GitHub repository, 2025. github.com/symbolica-ai/arcgentica. Audited at commit d7e4b2f (2025-06-12).
2. Symbolica AI. "Runtime as Context." Symbolica AI Blog, 2025. symbolica.ai/blog/runtime-as-context.
3. Chollet, F. "On the Measure of Intelligence." arXiv preprint arXiv:1911.01547, 2019.
4. Chollet, F. "ARC-AGI: A Benchmark for General Intelligence." ARC Prize Foundation, 2024. arcprize.org.
5. Romera-Paredes, B., et al. "Mathematical discoveries from program search with large language models." Nature, 625, 468–475, 2024. (FunSearch)
6. Novikov, A., et al. "AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms." Google DeepMind Blog, 2025.
7. Lehman, J., et al. "Evolution through Large Models." arXiv preprint arXiv:2206.08896, 2022.
8. Le Goues, C., Nguyen, T., Forrest, S., and Weimer, W. "GenProg: A Generic Method for Automatic Software Repair." IEEE TSE, 38(1), 54–72, 2012.
9. Liu, K., et al. "TBar: Revisiting Template-based Automated Program Repair." ISSTA, 2019.
10. Koza, J.R. Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press, 1992.
11. Le Goues, C., et al. "Automated Program Repair." Communications of the ACM, 62(12), 56–65, 2019.
12. Bäck, T. Evolutionary Algorithms in Theory and Practice. Oxford University Press, 1996.