AgentLaboratory: End-to-End Research Workflow

52.1 Overview & Motivation

The automation of scientific research represents one of the most ambitious applications of large language models. While individual components of the research pipeline — literature search, code generation, data analysis, report writing — have each seen LLM-assisted tooling, integrating these into a coherent, autonomous end-to-end workflow poses qualitatively different challenges. AgentLaboratory, introduced by Schmidgall et al. (2025), addresses this gap by orchestrating multiple specialized LLM agents through the full lifecycle of a computational research project: from reading relevant papers through designing and executing experiments to writing a complete scientific report.

The system's motivation stems from a concrete observation: conducting research involves a long sequence of cognitively demanding subtasks — surveying literature, formulating hypotheses, designing experiments, writing and debugging code, analyzing results, and composing manuscripts — each of which individually benefits from LLM assistance but which, taken together, require sustained coherence and iterative feedback that exceeds single-prompt capabilities. AgentLaboratory decomposes this pipeline into phases handled by role-specialized agents that share context and build on each other's outputs.

A distinctive element of the AgentLaboratory project is its introduction of AgentRxiv, a cumulative knowledge-sharing mechanism inspired by academic preprint servers. AgentRxiv allows outputs from completed research runs to be stored, indexed, and retrieved by subsequent runs, enabling a form of agent-to-agent collaboration where knowledge compounds across independent research episodes. This positions AgentLaboratory not merely as a single-run automation tool but as a research ecosystem with cross-run memory.

52.2 Architecture

52.2.1 System Design Overview

AgentLaboratory is organized as a phased pipeline of specialized agents, each responsible for a distinct stage of the research workflow. The system is implemented in Python and available as an open-source repository at github.com/SamuelSchmidgall/AgentLaboratory. The core orchestration follows a sequential-with-feedback architecture: agents execute in a defined order, but each phase can iterate internally (e.g., debugging code until experiments run successfully) and can incorporate human feedback at phase boundaries when operating in human-in-the-loop mode.

The repository is structured around several key modules [repo]:

File	Purpose	Key Identifiers	Tier
ai_lab_repo.py	Main orchestration entry point; defines the LaboratoryWorkflow class that manages phase transitions, state accumulation, and human-in-the-loop checkpoints	LaboratoryWorkflow, perform_research(), PHASES	[repo]
agents.py	Agent class definitions (single file, not a directory); agents for each research phase inherit from a common base class	BaseAgent, phase-specific agent classes	[repo]
inference.py	LLM backend wrappers; dispatches to OpenAI, DeepSeek, and other providers based on configuration	query_model()	[repo]
prompts.py	Role-specific prompt templates for each agent type (single file, not a directory)	Phase-specific prompt constants	[repo]
tools.py	External tool wrappers: Semantic Scholar API search, Python subprocess execution, file I/O	semantic_scholar_search(), code execution functions	[repo]
requirements.txt	Python dependency listing	—	[repo]

The high-level architecture comprises seven phases, executed sequentially [paper, §3]:

1. Literature Review — An agent searches for, retrieves, and summarizes relevant papers using the Semantic Scholar API.
2. Plan Formulation — Based on the literature review and user-provided research topic, a planning agent formulates a research plan including hypotheses and experimental design.
3. Data Preparation — An agent identifies, downloads, or generates appropriate datasets for the planned experiments.
4. Experiment Implementation — A coding agent writes the experimental code, iterating through code-execution-debug cycles.
5. Results Interpretation — An agent analyzes experimental outputs, generates figures, and interprets findings.
6. Report Writing — A writing agent composes a LaTeX research paper incorporating all prior outputs.
7. Report Refinement — A review agent critiques and improves the draft, mimicking peer review feedback.

52.2.2 Agent Specialization and Role Design

Each phase in the AgentLaboratory pipeline is handled by a specialized agent. The repository implements these as Python classes in agents.py [repo] that inherit from a base agent class. The key design choice is that agents are role-specialized rather than general-purpose: each agent receives phase-specific system prompts (defined in prompts.py [repo]), has access to phase-appropriate tools (from tools.py [repo]), and maintains phase-local state that is passed forward to subsequent agents.

The agent architecture follows a common pattern in multi-agent LLM systems: each agent operates as a ReAct-style (Reason + Act) loop [paper, §3], where the LLM generates both reasoning about what to do next and tool-call actions. For the coding agent, this manifests as a write-execute-debug cycle; for the literature agent, as a search-read-summarize cycle. The base agent class handles common functionality including LLM API calls (dispatched through inference.py via the query_model() function), conversation history management, and tool dispatch [repo].

The following code block shows the verified CLI interface and orchestration structure drawn from the repository's entry point and README:

# Verified CLI interface and orchestration structure from ai_lab_repo.py [repo]
#
# The repository entry point is ai_lab_repo.py, which defines the
# LaboratoryWorkflow class. Key CLI arguments (from README and argparse
# definitions in the file):

# CLI invocation pattern [repo, README]:
#   python ai_lab_repo.py \
#     --research-topic "Your research topic here" \
#     --llm-backend "o1-preview" \
#     --copilot-mode \
#     --compile-latex \
#     --num-papers-lit-review 8

# Verified CLI arguments [repo]:
#   --research-topic    : str  — the research question or topic
#   --llm-backend       : str  — model selection
#                         Known values: "gpt4o", "o1-preview", "o1-mini",
#                         "deepseek-chat" [paper, §5]
#   --copilot-mode      : flag — enables HITL feedback at phase boundaries
#   --compile-latex     : flag — compile final LaTeX report to PDF
#   --num-papers-lit-review : int — number of papers to retrieve

# Verified phase sequence [paper §3, repo]:
PHASES = [
    "literature_review",       # Semantic Scholar search + summarization
    "plan_formulation",        # Hypotheses, experimental design
    "data_preparation",        # Dataset identification/generation
    "running_experiments",     # Code generation + execute-debug loop
    "results_interpretation",  # Analysis, figure generation
    "report_writing",          # LaTeX paper composition
    "report_refinement",       # Self-review and revision
]

# Verified orchestration pattern [repo]:
# LaboratoryWorkflow.perform_research() iterates through PHASES:
#   1. Instantiates the phase-appropriate agent from agents.py
#   2. Passes accumulated state (literature notes, plan, code, results)
#   3. Agent executes via ReAct loop using query_model() from inference.py
#   4. Agent output is collected and appended to shared state
#   5. If --copilot-mode: pauses for human textual feedback [paper, §3.2]
#   6. Proceeds to next phase
#
# Verified from inference.py [repo]:
# query_model() dispatches LLM calls to the configured backend,
# handling API-specific formatting for OpenAI and DeepSeek endpoints.

Provenance: File names, the LaboratoryWorkflow class, query_model() function, CLI arguments, and the phase list are drawn from the repository's publicly visible code and README. The orchestration loop pattern (sequential phases with accumulated state and optional HITL pauses) is described in the paper §3 and reflected in the repository's perform_research() method. Readers should verify exact signatures against the current codebase, as the project may have been refactored.

52.2.3 Human-in-the-Loop Integration

A defining feature of AgentLaboratory is its human-in-the-loop (HITL) mode, activated via the --copilot-mode CLI flag [repo]. When enabled, the system pauses after each phase, presents the agent's output (literature notes, research plan, experimental code, results, or draft report), and solicits textual feedback that is injected into the context for subsequent phases. This is not merely a safety check — it is architecturally integrated as a mechanism for steering research direction [paper, §3.2].

The paper reports that human-in-the-loop operation substantially improves the quality of final research outputs compared to fully autonomous runs (see Section 52.4 for specific evaluation results). This design reflects a pragmatic acknowledgment that current LLM agents, while capable of executing individual research subtasks, benefit from human judgment at strategic decision points — particularly in plan formulation and results interpretation where domain expertise is most critical.

52.3 Core Algorithms and Mechanisms

52.3.1 Formal Pipeline Model

We formalize the AgentLaboratory pipeline as a state-transition system. This formalization is introduced by the chapter author to provide analytical precision; it is not presented in the original paper but is constructed to faithfully represent the pipeline semantics described in Schmidgall et al. (2025, §3) and observed in the repository.

Definition 52.1 (Pipeline State). The research state after phase $i$ is a tuple:

$S_i = (T,\; L_i,\; P_i,\; D_i,\; C_i,\; R_i,\; W_i)$

where $T$ is the research topic (fixed input), and $L_i, P_i, D_i, C_i, R_i, W_i$ denote the accumulated literature notes, research plan, data preparation artifacts, code and execution outputs, results analysis, and written report, respectively. Initially, $S_0 = (T, \emptyset, \emptyset, \emptyset, \emptyset, \emptyset, \emptyset)$.

Definition 52.2 (Phase Transition). Each phase $\phi_i$ for $i \in \{1, \ldots, 7\}$ is a transition function:

$\phi_i : \mathcal{S} \times \mathcal{H} \to \mathcal{S}$

$S_i = \phi_i(S_{i-1},\; h_i)$

where $h_i \in \mathcal{H}$ is optional human feedback (the empty string $\varepsilon$ in autonomous mode, free-text in --copilot-mode). Each $\phi_i$ updates exactly one primary component of the state tuple while reading from prior components:

• $\phi_1$: reads $T$, writes $L_1$ (literature review)
• $\phi_2$: reads $T, L_1$, writes $P_2$ (plan formulation)
• $\phi_3$: reads $T, L_2, P_2$, writes $D_3$ (data preparation)
• $\phi_4$: reads $T, L_3, P_3, D_3$, writes $C_4$ (experiment implementation — see Definition 52.3)
• $\phi_5$: reads $C_4$, writes $R_5$ (results interpretation)
• $\phi_6$: reads $L_5, P_5, D_5, C_5, R_5$, writes $W_6$ (report writing)
• $\phi_7$: reads $W_6$, writes $W_7$ (report refinement)

The pipeline is a function composition $S_7 = (\phi_7 \circ \phi_6 \circ \cdots \circ \phi_1)(S_0, h_1, \ldots, h_7)$, where the final output $W_7$ is the refined research paper.

Definition 52.3 (Code-Debug Loop Termination). Phase $\phi_4$ (experiment implementation) contains an internal iterative loop. Let $\texttt{code}_t$ denote the code state at iteration $t$, and let $\texttt{exec}(\texttt{code}_t) = (r_t, \sigma_t, \varepsilon_t)$ where $r_t \in \{0, 1\}$ is the return code (0 = success), $\sigma_t$ is stdout, and $\varepsilon_t$ is stderr. The loop is defined by:

$\texttt{code}_{t+1} = \begin{cases} \texttt{code}_t & \text{if } r_t = 0 \quad \text{(success: terminate)} \\ \texttt{LLM}(\texttt{code}_t,\; \varepsilon_t,\; P_3,\; D_3) & \text{if } r_t \neq 0 \text{ and } t < T_{\max} \\ \bot & \text{if } t = T_{\max} \quad \text{(timeout: return partial results)} \end{cases}$

The termination step is $t^* = \min\bigl(\{t \mid r_t = 0\} \cup \{T_{\max}\}\bigr)$. The output of the phase is $C_4 = (\texttt{code}_{t^*}, \sigma_{t^*}, \varepsilon_{t^*})$. The paper describes this bounded iteration cycle in §3.4 [paper]; the exact value of $T_{\max}$ is configurable in the implementation [repo] but the paper does not report the default.

Remark. The $\texttt{LLM}$ function in the debug step is not a random mutation — it receives the full error trace, the current code, and the research plan as context, making it a directed repair operator. This distinguishes the loop from stochastic local search: the "fitness evaluation" (code execution) is binary and implicit, while the "mutation" (LLM-guided repair) is informed and non-random.

52.3.2 Literature Review Automation

The literature review phase leverages the Semantic Scholar API [paper §3.1, repo] to discover and retrieve papers relevant to the user-specified research topic. The tools.py module provides wrapper functions for the Semantic Scholar API [repo]. The agent operates through an iterative search-and-summarize loop:

1. Query generation: The LLM generates search queries derived from the research topic, expanding initial terms into related concepts and technical keywords.
2. Paper retrieval: Queries are issued against the Semantic Scholar API. The --num-papers-lit-review CLI argument controls the number of papers retrieved [repo]. Retrieved papers include titles, abstracts, and metadata.
3. Relevance filtering: The agent assesses each paper's relevance to the research topic and retains a subset for detailed review.
4. Summarization: Retained papers are summarized, with key findings, methodologies, and results extracted into structured notes.
5. Synthesis: Individual summaries are integrated into a coherent literature review that identifies gaps, trends, and opportunities for the planned research.

The quality of the literature review directly affects downstream phases. The literature notes $L_1$ become part of the shared context that informs the planning agent's hypothesis generation ($\phi_2$) and the writing agent's related work section ($\phi_6$). This creates a beneficial dependency chain but also a vulnerability: if the literature review misses critical related work or mischaracterizes prior results, these errors propagate through the entire pipeline (a property formalized in the sequential composition above — there is no feedback from later phases to earlier ones).

52.3.3 Experiment Design and Code Generation

The experiment implementation phase ($\phi_4$) is the most technically complex component of the system. The coding agent must translate a research plan into executable Python code, handle data loading, implement the proposed methodology, run experiments, and produce structured outputs. This requires multi-step reasoning, awareness of library APIs, and the ability to recover from execution errors.

The implementation uses the iterative code-execute-debug cycle formalized in Definition 52.3 above [paper, §3.4]. The following code block shows the execution mechanism as observed in tools.py:

# Code execution mechanism from tools.py [repo]
#
# The repository uses Python's subprocess module to execute agent-generated
# code. The following reflects the execution pattern observed in tools.py.
# Exact function name and signature should be verified against current code.

import subprocess

def execute_code(code_file: str, timeout: int = 3600) -> dict:
    """
    Execute a Python file in a subprocess with timeout.

    This function is called by the experiment agent during the code-execute-
    debug cycle (Definition 52.3). The agent writes code to a file in the
    working directory, then calls this function to execute it.

    Args:
        code_file: Path to the Python file to execute.
        timeout: Maximum execution time in seconds. The default shown here
                 is illustrative; the actual default should be verified
                 against the repository. [repo: subprocess timeout is used;
                 exact default value unverified]

    Returns:
        Dict with 'success' (bool), 'stdout' (str), 'stderr' (str).
    """
    try:
        proc = subprocess.run(
            ["python", code_file],
            capture_output=True,
            text=True,
            timeout=timeout,
        )
        return {
            "success": proc.returncode == 0,
            "stdout": proc.stdout,
            "stderr": proc.stderr,
        }
    except subprocess.TimeoutExpired:
        return {
            "success": False,
            "stdout": "",
            "stderr": f"Execution timed out after {timeout} seconds",
        }

Provenance: The subprocess-based execution pattern is described in the paper (§3.4) and confirmed by the presence of subprocess usage in tools.py [repo]. The function interface above reflects the observed pattern; the exact function name, parameter defaults, and error handling should be verified against the current codebase. The file-based code management (writing generated code to files in a working directory rather than executing inline strings) is confirmed from the repository structure [repo].

The debug loop operates as follows, corresponding to the formal Definition 52.3:

# Debug loop pseudocode [pseudocode — conceptual reconstruction]
# This illustrates the experiment agent's core loop as formalized in
# Definition 52.3. It is NOT a repository excerpt.
#
# The actual loop is implemented inside the experiment agent class in
# agents.py, using query_model() from inference.py for LLM calls and
# the execution function from tools.py for subprocess calls.

def experiment_debug_loop(plan, data_notes, max_iterations):
    """
    Pseudocode for the code-execute-debug cycle (Definition 52.3).

    The experiment agent:
    1. Generates initial code from the research plan and data notes
    2. Executes via subprocess (tools.py)
    3. On failure: feeds error to LLM for repair, writes updated code
    4. On success or max iterations: returns results

    This is a conceptual reconstruction; see agents.py for implementation.
    """
    # Step 1: Initial code generation via LLM
    code = query_model(
        prompt=format_coding_prompt(plan, data_notes),
        # system prompt loaded from prompts.py
    )
    write_to_file("experiment.py", code)

    for t in range(max_iterations):  # t = 0, ..., T_max - 1
        # Step 2: Execute (corresponds to exec(code_t) in Def. 52.3)
        result = execute_code("experiment.py")

        if result["success"]:  # r_t = 0: terminate
            return collect_results(result["stdout"])

        # Step 3: LLM-guided repair (code_{t+1} = LLM(code_t, ε_t, P, D))
        code = query_model(
            prompt=format_debug_prompt(
                current_code=code,
                error_output=result["stderr"],
                plan=plan,
            )
        )
        write_to_file("experiment.py", code)

    # T_max reached: return partial results (⊥ case in Def. 52.3)
    return collect_partial_results()

Provenance: This block is explicitly labeled [pseudocode] — a conceptual reconstruction of the debug loop formalized in Definition 52.3 and described in the paper §3.4. The actual implementation is in agents.py and uses query_model() from inference.py and execution functions from tools.py [repo]. The reconstruction captures the described semantics but does not claim to reproduce exact method names, prompt formatting, or control flow.

52.3.4 Report Writing and LaTeX Generation

The report writing phase ($\phi_6$) synthesizes all prior outputs into a structured scientific paper in LaTeX format [paper, §3.6]. The writing agent receives the literature review, research plan, experimental code, results (including figures and tables), and any human feedback. It generates a multi-section paper following standard academic structure: abstract, introduction, related work, methodology, experiments, results, discussion, and conclusion.

The report refinement agent ($\phi_7$) subsequently acts as a reviewer, critiquing the draft for clarity, consistency, missing citations, and logical gaps. This two-stage write-then-review process mimics the academic peer review cycle and produces higher-quality outputs than single-pass generation [paper, §3.7]. The refinement agent generates specific suggestions that the writing agent incorporates in a revision pass. The --compile-latex flag [repo] controls whether the system invokes the LaTeX compiler to produce a PDF.

52.3.5 AgentRxiv: Cumulative Knowledge Sharing

AgentRxiv is the system's mechanism for inter-run knowledge accumulation [paper, §4]. Inspired by academic preprint servers (arXiv, bioRxiv), AgentRxiv stores completed research artifacts — including literature reviews, experimental code, results, and written reports — in an indexed repository that subsequent research runs can query. This enables agent-to-agent collaboration where insights and methods from past research episodes inform future ones.

The knowledge sharing operates at multiple granularities [paper, §4]:

• Paper-level: Complete research reports from prior runs serve as additional "literature" that new runs can reference, building on previous agent-generated findings.
• Method-level: Successful experimental approaches (code patterns, model architectures, hyperparameter configurations) are available for retrieval and adaptation.
• Insight-level: Lessons learned, negative results, and methodological notes from prior runs prevent repeated mistakes and guide exploration toward productive directions.

Definition 52.4 (AgentRxiv Archive — Author Formalization). We formalize the AgentRxiv mechanism as an append-only indexed archive. This formalization is constructed by the chapter author to capture the conceptual semantics described in the paper (§4); the implementation may differ. Let $\mathcal{A}_n = \{(k_i, S_i^{\mathrm{final}})\}_{i=1}^{n}$ be the archive after $n$ completed runs, where $k_i$ is a topic descriptor and $S_i^{\mathrm{final}} = S_7^{(i)}$ is the terminal pipeline state of run $i$.

Update rule: After each completed run $n+1$:

$\mathcal{A}_{n+1} = \mathcal{A}_n \cup \{(k_{n+1},\; S_{n+1}^{\mathrm{final}})\}$

Retrieval function: Given a new research topic $T'$, the literature review phase $\phi_1$ of a new run queries the archive:

$\texttt{retrieve}(T', \mathcal{A}_n) = \{S_i^{\mathrm{final}} \mid \texttt{relevance}(T', k_i) > \tau\}$

where $\texttt{relevance}(\cdot, \cdot)$ is a scoring function and $\tau$ is a relevance threshold. Retrieved artifacts augment the literature context:

$L_1^{\mathrm{augmented}} = L_1^{\mathrm{external}} \cup \texttt{retrieve}(T', \mathcal{A}_n)$

Key property: The archive grows monotonically — there is no deletion, pruning, or quality-based filtering. This distinguishes AgentRxiv from evolutionary knowledge bases where selection pressure removes low-fitness entries.

52.3.6 Cost Model

A practical concern for autonomous research systems is the cost of LLM API calls across the full pipeline. The paper reports that AgentLaboratory can produce research papers at costs of approximately $2–15 USD per research run depending on the LLM model used and task complexity [paper, §5, Table 2]. Specifically:

• Runs using GPT-4o as the backbone are at the lower end of the cost range [paper, §5].
• Runs using o1-preview (the most capable model tested) are at the higher end [paper, §5].
• The experiment implementation phase ($\phi_4$) typically dominates cost due to its iterative debug cycles, which generate many LLM calls [paper, §5].
• The report writing phase ($\phi_6$) incurs cost through large context windows (accumulated state from all prior phases) [paper, §5].

Total cost for a single run can be expressed as a standard LLM cost-accounting formula (not specific to AgentLaboratory):

$C = \sum_{p=1}^{7} \sum_{t=1}^{T_p} \bigl(c_{\mathrm{in}} \cdot n_{\mathrm{in}}^{(p,t)} + c_{\mathrm{out}} \cdot n_{\mathrm{out}}^{(p,t)}\bigr)$

where $T_p$ is the number of LLM calls in phase $p$, $n_{\mathrm{in}}^{(p,t)}$ and $n_{\mathrm{out}}^{(p,t)}$ are the input and output token counts for the $t$-th call, and $c_{\mathrm{in}}, c_{\mathrm{out}}$ are per-token costs for the chosen model.

Quantitative provenance: The $2–15 cost range is drawn from the AgentLaboratory paper (Schmidgall et al., 2025, §5, Table 2). These figures depend on the specific LLM pricing at time of publication (early 2025) and would vary with model selection, task complexity, number of debug iterations, and subsequent API pricing changes. The paper does not report a breakdown of cost by phase or a distribution across the 8 evaluated topics. No independent cost audit has been conducted for this survey.

52.4 Key Results

52.4.1 Evaluation Methodology

Evaluating an autonomous research system is inherently challenging. Unlike traditional benchmarks with well-defined metrics, the quality of a research paper is subjective, multi-dimensional, and context-dependent. The paper employs several evaluation strategies [paper, §5]:

• Human expert evaluation: Generated papers are assessed by human researchers on multiple quality dimensions using a structured scoring rubric [paper, §5].
• Peer review proxy: Papers were submitted to peer-reviewed workshop venues to test whether agent-generated research meets community acceptance thresholds [paper, §5].
• Multi-model comparison: The same research topics are run with different LLM backends to assess the effect of model capability on output quality [paper, §5].
• HITL ablation: Comparison between human-in-the-loop and fully autonomous operation [paper, §5].

52.4.2 Experimental Setup

The following table summarizes the evaluation protocol as reported in the paper. Missing fields are explicitly noted.

Parameter	Value	Source
Number of research topics	8 topics in machine learning	[paper, §5]
LLM backends tested	GPT-4o, o1-preview, o1-mini, DeepSeek	[paper, §5]
HITL conditions	Autonomous vs. human-in-the-loop (--copilot-mode)	[paper, §5]
Evaluation dimensions	Clarity, Correctness, Significance, Novelty, Overall Quality (on a numerical scale)	[paper, §5]
Number of human evaluators	Multiple researchers; the paper reports evaluation by human reviewers but does not specify the exact number of unique evaluators or inter-rater agreement statistics	[paper, §5] — exact count not reported
Scoring rubric	Each dimension scored on a numerical scale; the paper provides dimension-level scores per model and topic	[paper, §5]
Number of runs per topic	Not reported; it is unclear whether multiple runs per topic were conducted or whether variance across runs was measured	Not available
Seeds / reproducibility controls	Not reported	Not available
Cost per run	$2–15 USD depending on model	[paper, §5, Table 2]

52.4.3 Reported Performance

The following results are directly reported in the paper (Schmidgall et al., 2025, §5). They are separated from interpretive claims to allow independent assessment of the evidence base.

Finding	Evidence	Source & Caveats
End-to-end completion	System produces complete research papers (code, figures, LaTeX) across 8 ML research topics	Paper §5. Topics span multiple ML subfields including representation learning, optimization, and generalization [paper]
Workshop acceptance	Agent-generated papers accepted at peer-reviewed workshops. The paper specifically notes acceptance at an ICLR-affiliated workshop	Paper §5. Small sample size; HITL was enabled for accepted papers. The number of submissions vs. acceptances is not fully detailed [paper]
Model ranking	o1-preview consistently produces the highest-quality outputs across evaluation dimensions; GPT-4o is intermediate; o1-mini and DeepSeek trail	Paper §5, evaluation tables. Per-model scores are reported across the 8 topics, showing a consistent ranking [paper]
HITL improvement	Human-in-the-loop mode improves evaluation scores across all dimensions compared to fully autonomous operation. The improvement is described as substantial, particularly for plan quality and experimental design	Paper §5. Comparison methodology: same topics run with and without HITL. Not a large-scale controlled study; no confidence intervals reported [paper]
Human evaluation scores	Papers evaluated on Clarity, Correctness, Significance, Novelty, and Overall Quality. The paper reports per-model aggregate scores across these dimensions. With o1-preview + HITL, average scores approach (but do not match) the quality level of competent human-written workshop papers	Paper §5, evaluation tables. Specific numerical scores are provided in the paper's tables; readers should consult the original for exact values, as they vary by topic and model [paper]
Cost efficiency	$2–15 USD per run (GPT-4o ≈ low end; o1-preview ≈ high end)	Paper §5, Table 2. Based on early-2025 API pricing; does not include human time in HITL mode [paper]

52.4.4 Interpretive Assessment

The following observations represent the chapter author's interpretation of the reported evidence, rather than direct claims from the paper:

• Existence proof, not guaranteed performance: The workshop acceptances demonstrate that the system can produce research papers that pass peer review, but the evidence does not establish a reliable success rate. The sample is small (8 topics, with HITL enabled for the strongest results), and workshop acceptance thresholds vary across venues. Readers should not infer that the system routinely produces publishable research.
• Model capability matters significantly: The consistent quality ranking (o1-preview > GPT-4o > o1-mini) across the 8 topics suggests that the pipeline's output quality is substantially bounded by the underlying LLM's reasoning capability. This implies that AgentLaboratory's value proposition will improve as LLMs improve, but also that the orchestration framework itself adds less marginal value than it might appear — much of the quality difference is explained by the backbone model.
• HITL is currently necessary for high-quality output: The evaluation evidence strongly suggests that fully autonomous mode does not match HITL quality. This is an honest and important finding that tempers the "autonomous research" framing.
• Cost comparison to human research is incomplete: The $2–15 cost figure covers LLM API costs only. It does not account for human time in HITL mode, topic selection effort, result verification, researcher oversight, or the opportunity cost of the human expert's time. A fully loaded cost comparison has not been published.

52.4.5 Comparative Analysis

AgentLaboratory exists within a growing ecosystem of autonomous research systems. The following comparison uses a structured framework with evidence citations for each claim. All capability assessments reflect the chapter author's analysis of published descriptions as of early 2025.

Dimension	AgentLaboratory	AI Scientist (Lu et al., 2024)	MLAgentBench (Huang et al., 2024)
Pipeline scope	Full: lit review → code → paper Evidence: paper §3, 7 phases	Full: idea → code → paper → review Evidence: Lu et al. §3	Partial: code → execution only Evidence: Huang et al. §3
Execution isolation	subprocess.run() with timeout; no filesystem/network restriction Evidence: tools.py [repo]	Docker container with resource limits and network isolation Evidence: Lu et al. §3.4, repo docker/ directory	Docker sandbox with filesystem restrictions Evidence: Huang et al. §3.2, repo sandbox/
HITL support	Explicit per-phase checkpoints via --copilot-mode Evidence: paper §3.2, repo	Fully autonomous; no structured HITL mechanism Evidence: Lu et al. — paper does not describe HITL	Human provides initial setup; agent runs autonomously Evidence: Huang et al. §3
Cross-run knowledge	AgentRxiv archive for cumulative knowledge Evidence: paper §4	No documented cross-run memory Evidence: Lu et al. — not described	No documented cross-run memory Evidence: Huang et al. — not described
Automated review	Report refinement agent critiques draft Evidence: paper §3.7	Dedicated LLM reviewer with structured scoring; more extensively validated Evidence: Lu et al. §3.5	Not applicable (no paper output) —
Evaluation breadth	8 ML topics, human evaluation + workshop submission Evidence: paper §5	Multiple research domains (NanoGPT, diffusion, etc.), automated + human review, multiple conference-style evaluations Evidence: Lu et al. §4	15 Kaggle-style ML tasks with automated scoring Evidence: Huang et al. §4
Open source	Yes Evidence: github.com/SamuelSchmidgall/AgentLaboratory	Yes Evidence: github.com/SakanaAI/AI-Scientist	Yes Evidence: github.com/snap-stanford/MLAgentBench

Key comparative observations (chapter author analysis):

• Execution safety: The AI Scientist and MLAgentBench both implement container-based isolation (Docker), providing filesystem restriction, network isolation, and resource limits that AgentLaboratory's subprocess approach does not. This is documented in the respective papers and repositories (Lu et al., 2024, §3.4; Huang et al., 2024, §3.2). AgentLaboratory's subprocess model provides crash isolation and timeout enforcement but not adversarial-grade containment.
• Evaluation extensiveness: The AI Scientist has been evaluated across more diverse research domains and includes both automated and human review, making its evidence base broader than AgentLaboratory's 8-topic evaluation. However, budget-matched comparisons between these systems have not been conducted in any published work known to this survey.
• Unique contributions: AgentLaboratory's two distinctive features — structured HITL integration and AgentRxiv cumulative knowledge — are not replicated in comparable systems. Whether these features translate to meaningfully better outputs at scale remains to be established through controlled comparison.

52.5 Implementation Details

52.5.1 Repository Structure

The AgentLaboratory repository at github.com/SamuelSchmidgall/AgentLaboratory is organized as a single-package Python project [repo]. The following table lists key components with their verification status:

File / Module	Purpose	Tier
ai_lab_repo.py	Main orchestration: LaboratoryWorkflow class, perform_research(), phase sequencing, state management, argparse CLI definition	[repo]
agents.py	Agent class definitions for each research phase; base agent class with LLM call interface, tool dispatch, conversation history; single file, not a directory	[repo]
inference.py	LLM backend wrappers: query_model() function dispatching to OpenAI, DeepSeek, and other providers based on backend configuration	[repo]
prompts.py	Role-specific prompt templates for each agent type; single file, not a directory	[repo]
tools.py	External tool wrappers: Semantic Scholar API search, code execution via subprocess, file I/O operations	[repo]
requirements.txt	Python dependency listing	[repo]

52.5.2 LLM Backend Configuration

AgentLaboratory supports multiple LLM backends, selectable via the --llm-backend CLI flag [repo]. The paper documents experiments with the following models [paper, §5]:

• GPT-4o — OpenAI's multimodal model; positioned at the lower end of the cost range ($\approx$\$2–5 per run)
• o1-preview — OpenAI's reasoning-focused model; highest quality but highest cost ($\approx$\$10–15 per run)
• o1-mini — Smaller variant of the o1 series; lower quality and cost
• DeepSeek — Open-weight model alternative; accessed via "deepseek-chat" backend string

The inference.py module's query_model() function handles the dispatch to the appropriate API based on the backend string [repo]. The paper describes the possibility of configuring different models for different phases [paper, §3], recognizing that some phases (e.g., coding and debugging) may benefit from stronger models while others (e.g., literature summarization) may work adequately with less expensive options.

52.5.3 Execution Environment and Safety

The experiment execution phase involves running agent-generated Python code via subprocess.run() with timeout controls [repo]. This provides basic isolation — generated code runs in a separate process with output capture — but does not constitute sandboxing in the security sense. Specifically:

Property	AgentLaboratory	Evidence
Process isolation	Yes — crash in subprocess does not crash orchestrator	[repo] — subprocess.run() in tools.py
Timeout enforcement	Yes — configurable timeout parameter	[repo] — timeout argument to subprocess.run()
Output capture	Yes — stdout and stderr captured via capture_output=True	[repo]
Network isolation	No — generated code has full network access	[repo] — no network restriction observed
Filesystem restriction	No — generated code can read/write any path accessible to the user	[repo] — no chroot, no filesystem sandboxing
Memory/CPU limits	No — beyond timeout, no resource limits	[repo] — no cgroup or ulimit usage observed

Security note: Agent-generated code executes with the full permissions of the user's Python process. For research settings involving untrusted inputs or adversarial scenarios, additional containment (Docker containers, virtual machines, or dedicated sandbox environments) is necessary. This contrasts with The AI Scientist (Lu et al., 2024), which uses Docker containers with network isolation and resource limits (documented in their paper §3.4 and repository's docker/ directory), and MLAgentBench (Huang et al., 2024), which implements a sandbox environment with filesystem restrictions (documented in their paper §3.2 and repository's sandbox/ module).

52.5.4 Reproducibility Considerations

Reproducing exact outputs from AgentLaboratory is inherently challenging due to several sources of non-determinism:

• LLM non-determinism: Commercial LLM APIs do not guarantee deterministic outputs, even with temperature set to zero, particularly across API versions.
• External API variability: Literature search results from Semantic Scholar depend on the current index state, which evolves over time.
• AgentRxiv state: If using cumulative knowledge, the initial knowledge base state affects outputs. Fresh runs will differ from runs that leverage prior research artifacts.
• Execution environment: Library versions, hardware differences, and stochastic elements in experiments (random seeds in ML training) all introduce variation.

The repository does not currently include a formal reproducibility protocol (e.g., pinned model versions, fixed API snapshots, deterministic seeds) [repo]. This is common among research prototypes but should be considered when interpreting reported results.

52.6 Connection to Evolutionary AI

52.6.1 Research as an Evolutionary Process: Analogies and Limits

While AgentLaboratory does not employ evolutionary algorithms, it bears structural similarities to evolutionary processes that merit analysis in the context of this survey. The following analogies are interpretive frameworks offered by the chapter author, not claims made by the AgentLaboratory paper. Each analogy is presented alongside its limitations to prevent overclaiming.

Analogy 1 — Iterative refinement as local search: The code-execute-debug cycle (Definition 52.3) functions as a form of iterative optimization. Each debug attempt generates a code variant that is evaluated (does it execute without error?) and the successful variant is retained. Formally, this resembles a $(1+1)$ evolutionary strategy where a single candidate is modified and the better version survives. The fitness function is binary: $f(\texttt{code}_t) = \mathbb{1}[r_t = 0]$.

Limitation: The "mutation" is intelligent (guided by the LLM's understanding of the error and research context), not random. The LLM does not explore the code space stochastically; it applies directed reasoning. The fitness evaluation is binary (runs/doesn't run), not a nuanced scalar. There is no population, no selection pressure across competing candidates, and no crossover.

Analogy 2 — AgentRxiv as cultural evolution: The cumulative knowledge mechanism (Definition 52.4) creates an analogue to cultural evolution, where knowledge accumulates across generations (research runs) rather than being discovered independently each time. In biological terms, this loosely resembles Lamarckian inheritance — acquired knowledge is directly transmitted to successors.

Limitation: AgentRxiv grows monotonically; there is no selection, pruning, or competition among archived artifacts. Biological and cultural evolution involve differential fitness and selective retention; AgentRxiv is an append-only archive ($|\mathcal{A}_{n+1}| = |\mathcal{A}_n| + 1$). The analogy captures the "knowledge accumulation" aspect but not the "selective pressure" aspect of evolution.

Analogy 3 — Multi-agent specialization as division of labor: The role-specialized agent architecture mirrors the division of labor seen in social insect colonies and other evolved cooperative systems. Each agent occupies a distinct functional role, and overall output quality emerges from their coordinated interaction through the state-transition pipeline (Definitions 52.1–52.2).

Limitation: The agent roles are statically assigned by the system designer, not evolved or adapted. There is no competition between agents, no emergence of specialization through selection, and no dynamic role allocation.

52.6.2 Relationship to LLM-Powered Evolutionary Systems

Several systems surveyed elsewhere in this volume use LLMs as mutation operators within explicit evolutionary frameworks (FunSearch, OpenELM, EvoTorch with LLM guidance). AgentLaboratory takes a fundamentally different approach: rather than evolving programs or heuristics through population-based search, it uses LLMs to execute a structured research workflow. The evolutionary element, if any, emerges only weakly from:

• The iterative debug loop (local search / hill-climbing on code correctness — Definition 52.3)
• The AgentRxiv knowledge accumulation (append-only cross-run knowledge transfer — Definition 52.4)
• The potential for running multiple research episodes on related topics (manual exploration of a research landscape)

A natural extension would be to explicitly combine AgentLaboratory with evolutionary search: generating multiple competing research plans, evaluating them against fitness criteria (novelty, feasibility, expected impact), and selecting the most promising for full execution. This would bridge the gap between autonomous research systems and the evolutionary program synthesis systems covered in earlier chapters.

52.7 Limitations & Discussion

52.7.1 Quality Ceiling and Failure Modes

Despite its demonstrated ability to produce workshop-accepted papers (with HITL), AgentLaboratory faces several fundamental limitations:

• Novelty generation: LLMs are trained on existing literature and tend toward recombination of known ideas rather than genuinely novel insights. Agent-generated research may be technically competent but conceptually incremental.
• Experimental scope: The system is most effective for computational experiments expressible as self-contained Python scripts. Research requiring physical experiments, specialized hardware, proprietary datasets, or complex multi-system infrastructure is beyond its current scope [paper, §6].
• Error propagation: The sequential pipeline (Definition 52.2) means that errors in early phases compound through later phases. A poor literature review ($L_1$) leads to a poorly motivated plan ($P_2$), which leads to misguided experiments ($C_4$), which leads to a weak paper ($W_7$). There is no feedback from later phases to earlier ones.
• Evaluation limitations: The system lacks robust self-evaluation mechanisms beyond code executability ($r_t = 0$ in Definition 52.3). It cannot reliably assess whether results are statistically significant, whether conclusions follow from the data, or whether the experimental design has confounds.
• Hallucination risk: LLMs may generate plausible but incorrect claims, fabricated citations, or misleading descriptions of results. The code-execute-debug loop provides a partial check for the experimental phase (code must actually run), but the literature review and report writing phases are more vulnerable to factual errors.

52.7.2 Scalability and Scope Limitations

AgentLaboratory's sequential pipeline architecture imposes a scalability ceiling. Each phase must complete before the next begins, and the accumulated context $S_i$ grows throughout the pipeline, potentially exceeding LLM context windows for later phases (particularly $\phi_6$ and $\phi_7$, which must synthesize all prior outputs). The system's effectiveness is most clearly demonstrated for well-scoped computational ML research tasks with single datasets and relatively simple experimental setups [paper, §5–6].

52.7.3 Context Management

A critical implementation challenge is managing the growing research context as it flows through pipeline phases. By the report-writing phase ($\phi_6$), the agent must have access to — or at least summaries of — all prior outputs [paper, §3.6].

52.7.4 Ethical and Community Implications

The ability to generate research papers at low cost raises important questions for the scientific community:

• Review burden: If autonomous systems can produce papers at scale, the already strained peer review system could face an even greater submission volume.
• Attribution and credit: The appropriate attribution model for AI-generated research is an open question. Should AI systems be listed as authors? How should human-in-the-loop contributions be credited?
• Quality floor: Easy paper generation may lower the average quality of submissions, making it harder for reviewers to identify genuinely valuable contributions.
• Positive potential: Conversely, these tools could democratize research by enabling researchers with limited resources to explore ideas that would otherwise be too costly to investigate.

52.7.5 Open Research Questions

1. Scaling AgentRxiv: How does the quality of cumulative knowledge degrade or improve as the archive grows? Are there diminishing returns, or does the system exhibit compounding capabilities? Would introducing selection pressure (pruning low-quality entries) improve retrieval quality?
2. Multi-agent collaboration: Can the sequential pipeline be replaced or augmented with true multi-agent collaboration, where agents negotiate, critique, and refine each other's work in real time rather than in a fixed sequence?
3. Automated research evaluation: Can LLM-based reviewers reliably assess the quality of LLM-generated research? The risk of mode collapse (reviewer and author sharing the same biases) is significant.
4. Domain transfer: How well does the system transfer to domains beyond computational ML? Experimental science, theoretical research, and interdisciplinary work pose distinct challenges.
5. Integration with evolutionary search: Can AgentLaboratory's pipeline be embedded within an explicit evolutionary framework, where populations of research plans compete and evolve? This would connect the system to the evolutionary program synthesis paradigm discussed in earlier chapters.

52.8 Technical Deep Dive: Tool Integration and Agent Mechanics

52.8.1 Tool Integration Patterns

AgentLaboratory agents interact with external tools through interfaces defined in tools.py [repo]. The primary tool categories are:

Tool Category	Used By Phase	Mechanism	Tier
Academic search	$\phi_1$ (Literature review)	HTTP API calls to Semantic Scholar; wrapper functions in tools.py	[paper §3.1, repo]
Code execution	$\phi_4$ (Experiment implementation)	Python subprocess.run() with timeout and output capture	[paper §3.4, repo]
File management	All phases	Read/write to working directory for code, data, results, and report files	[repo]
LaTeX compilation	$\phi_6$, $\phi_7$ (Report writing/refinement)	LaTeX toolchain invocation (when --compile-latex is set)	[repo]

The tool integration follows function-calling conventions compatible with the underlying LLM API [paper, §3]. Each tool is described with a schema (name, description, parameters), and the LLM generates structured tool calls that the agent dispatches. This approach leverages the LLM's ability to select appropriate tools based on the task context but introduces failure modes when the LLM generates malformed tool calls or selects inappropriate tools.

52.8.2 Agent Communication via State

Agents in AgentLaboratory communicate exclusively through the shared pipeline state $S_i$ (Definition 52.1) — there is no direct agent-to-agent messaging. This design simplifies the architecture and makes each phase independently testable, but limits the system's ability to handle situations where later agents need to request clarification or additional work from earlier agents.

The state object is accumulated in a working directory [repo]. Each agent reads its predecessors' outputs from files in this directory and writes its own outputs for successors. This file-based communication pattern is observable in the repository's structure, where each run produces a directory of artifacts including literature notes, plans, code files, result data, figures, and LaTeX source.

52.9 Broader Context: Autonomous Research Systems Landscape

AgentLaboratory belongs to a rapidly growing family of systems that aim to automate significant portions of the research process. This section situates it within the broader landscape.

The AI Scientist (Sakana AI, 2024) is the closest point of comparison. Both systems implement the full research pipeline from idea generation through paper writing. Key architectural differences:

• Human integration: AgentLaboratory provides explicit HITL checkpoints (--copilot-mode); The AI Scientist operates as a fully autonomous loop without structured human checkpoints [Lu et al., 2024, §3].
• Knowledge accumulation: AgentLaboratory introduces AgentRxiv for cross-run knowledge sharing; The AI Scientist does not include a documented cumulative knowledge mechanism [Lu et al., 2024].
• Execution isolation: The AI Scientist uses Docker containerization with network isolation [Lu et al., 2024, §3.4; repo docker/ directory]; AgentLaboratory uses subprocess with timeout only [repo, tools.py].
• Automated review: The AI Scientist includes a more extensively developed automated reviewer with structured scoring criteria [Lu et al., 2024, §3.5]; AgentLaboratory's report refinement agent provides a lighter review mechanism [paper, §3.7].
• Evaluation breadth: The AI Scientist has been evaluated across more diverse research domains (NanoGPT language modeling, 2D diffusion, Grokking, etc.) with both automated and human evaluations [Lu et al., 2024, §4]; AgentLaboratory evaluates on 8 ML topics [paper, §5].

ResearchAgent (Baek et al., 2024) focuses on the idea generation and refinement portion of the pipeline, using a multi-agent debate framework to generate and critique research ideas. It does not include experiment execution, making it complementary to AgentLaboratory's full-pipeline approach.

SciMON (Wang et al., 2024) and related systems focus on scientific text generation with novelty-aware retrieval, emphasizing the literature grounding aspect. AgentLaboratory goes further by actually executing the proposed experiments.

Comparison methodology note: No published work has conducted a budget-matched, controlled comparison between these systems on identical research tasks. The comparisons above are based on the systems' respective papers and repositories, which use different evaluation protocols, topics, and success criteria. Feature presence (e.g., "supports literature review") does not imply comparable quality of that feature across systems.

The trend across these systems is toward increasing automation scope and sophistication. First-generation tools (2023) focused on individual research subtasks; second-generation systems (2024–2025) like AgentLaboratory and The AI Scientist integrate multiple subtasks into coherent pipelines; emerging third-generation approaches are beginning to incorporate evolutionary search, multi-agent negotiation, and long-term research programs that span multiple papers and topics.

52.10 Summary