Skip to main content

AlphaDiana

A System for Evaluating Reasoning Agents

Get StartedGitHub

Reasoning agents such as OpenClaw are model-harness systems. The model supplies reasoning capacity, while the harness realizes it through tool use, multi-turn control, execution feedback, memory, and runtime policy. This shifts the paradigm from model reasoning to agentic reasoning. Existing evaluations often test models without a harness or fix a single harness, which conflates base-model reasoning with harness orchestration, leaving it unclear whether gains reflect stronger models or specific harness components.

From model-centric to harness-aware evaluation
From model-centric to harness-aware evaluation. A foundation model is an engine evaluated by stateless prompt-in/answer-out scoring; an agent is the car that embeds the model in a runtime with control, tools, state, policies, and sandbox constraints. AlphaDiana acts as the tournament organizer: it fixes evaluation rules across agents, runs them on shared benchmark tracks, and logs canonical traces for analyzing correctness, efficiency, reliability, and failures.

System Architecture

One runner, explicit evaluation boundaries

AlphaDiana resolves this conflation by making the evaluation process explicit, modular, and auditable: an experiment run is represented as ℛ = 𝒰(ℬ, fθ, ℋ, ℰ, 𝒮, 𝒞), where ℛ denotes saved results, 𝒰 the shared runner, ℬ the benchmark, fθ the base model, ℋ the agent harness,ℰ the execution environment, 𝒮 outcome and process evaluation, and 𝒞 the run configuration. Together, the model and harness form the evaluated agent 𝒜θ,ℋ = ℋ ⊙ fθ; Direct is its no-tool control.

AlphaDiana architecture from experiment configuration through benchmark, agent, environment, scorer, result store, and analysis
Architecture of AlphaDiana. The shared runner connects experiment configuration, agent execution, scoring, result storage, and analysis in one auditable workflow.

What AlphaDiana standardizes

Harness-Aware Evaluation

Treat the evaluated unit as model plus harness, with Direct inference as the control.

Controlled Comparison

Match the model, task, scorer, and shared budgets while disclosing harness-specific runtime conditions.

Open Harness Interfaces

Run direct inference and reasoning agent harnesses through one shared result schema.

Verifiable Reasoning Tasks

Use deterministic or programmatic scorers for math, science, multimodal, and coding tasks.

Trajectory Attribution

Log actions, observations, resource use, termination states, and scoring metadata.

Sandboxed Execution

Run supported tool-using paths in task-scoped environments with explicit resource and reset policies.

Macro And Micro Studies

Compare whole harnesses, then ablate tools, skills, and memory to explain the score shifts.

Reproducible Reports

Regenerate summaries from saved records and inspect the evaluation loop in the dashboard.

Dashboard

Launch, monitor, and compare

AlphaDiana turns a declarative configuration into a run, exposes it in a shared result store, and lets you monitor and compare evaluations side by side. The screenshots below illustrate the dashboard workflow for an AIME 2026 DirectLLM evaluation with four samples per problem. See the documentation for details.

Creating an AIME 2026 Direct LLM evaluation
Create an AIME 2026 Direct LLM evaluation from explicit configuration fields.
AIME 2026 Direct LLM evaluation results
Inspect the completed run with Pass@4, Avg@4, token usage, and runtime statistics.

Tournament Results

Harness gains depend on the model, task, and harness

The reported results compare the Direct no-harness baseline with open harnesses under matched models, tasks, scorers, and shared budgets while recording harness-specific runtime conditions. Harnessing is not a uniform upgrade: it lifts Gemma, sinks Qwen3.5, and produces mixed behavior on Kimi-K2.6. Green means the harness beat Direct; red means it lost ground.

Model perspective. Harnessing is not a model-agnostic improvement. The evaluated harnesses never improve Qwen3.5-27B over its reference, help Gemma-4-31B-IT on several answer-based tasks, and yield mixed outcomes for Kimi-K2.6.
Task perspective. Harness benefits are task-dependent. Gains on answer-based reasoning do not transfer automatically to software-engineering or terminal tasks, where every evaluated harness either matches or trails the task-native reference.
Harness perspective. No harness is uniformly best. Each harness has configurations in which it is competitive and others in which it causes a substantial loss, so scores should be read as properties of model-harness-task triples.
Gemma-4-31B-ITHigher overall
HarnessIMO-AnswerBenchAvg@1HLE-VerifiableAvg@1GPQA-DiamondAvg@1AIME 26Pass@4AIME 26Avg@4MMMU-ProAvg@1
Directbaseline59.027.983.396.792.565.8
OpenClaw59.5+0.524.2-3.785.4+2.1100.0+3.397.5+5.056.8-9.0
ZeroClaw61.5+2.529.1+1.286.4+3.1100.0+3.396.7+4.266.4+0.6
OpenCode62.5+3.524.0-3.987.9+4.6100.0+3.396.7+4.267.4+1.6

Several harness rows improve on IMO, GPQA, and AIME in this selected comparison.

Values are percentages (higher is better); deltas are absolute differences vs. Direct. This view presents selected results from Table 1 of the AlphaDiana study. Consult the corresponding archived runs when citing exact values.

Quick Start

Run an AIME OpenClaw smoke evaluation

The command below uses configs/examples/openclaw_aime2024.yaml with the ROCK-managed OpenClaw runtime, downscaled to one problem and one sample. Connect it to a local model server or a hosted API in step 2. See the getting-started guide for setup details.

1. Install

bash
git clone https://github.com/tmlr-group/AlphaDianacd AlphaDiana
# Required by the security preflight in the current releaseexport OPENCLAW_GATEWAY_TOKEN="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
# Checkout-local conda env, dependencies, and servicesbash scripts/quickstart.sh

2. Run

bash
# Activate the environment once per terminalsource scripts/activate.sh
# Use either a local OpenAI-compatible model server:export OPENAI_MODEL_NAME="your-local-model"export OPENAI_BASE_URL="http://127.0.0.1:8000/v1"export OPENAI_API_KEY="sk-local"
# Or a hosted API (replace the three values above):# export OPENAI_MODEL_NAME="your-api-model"# export OPENAI_BASE_URL="https://your-provider.example/v1"# export OPENAI_API_KEY="your-api-key"
# Validate and run a one-task AIME OpenClaw smokealphadiana validate configs/examples/openclaw_aime2024.yaml \  -o benchmark.config.max_tasks=1alphadiana run configs/examples/openclaw_aime2024.yaml \  -o run_id=quickstart_aime_openclaw_t1_k1 \  -o benchmark.config.max_tasks=1 \  -o num_samples=1 \  -o max_concurrent=1
# Generate a report from the smoke runalphadiana report results

Citation

@inproceedings{
  zhou2026reasoning,
  title={Reasoning Is More Than the Model: Harness-Aware Evaluation of Agents on Verifiable Reasoning Tasks},
  author={Zhanke Zhou and Zongze Li and Weikai Huang and Xuan Li and Chentao Cao and Xiao Feng and Xiangyu Lu and Jinbo Hu and Menghan Lu and Yi Xie and Nico Pelleriti and Shiyang Liu and Max Zimmer and Brando Miranda and Jiangchao Yao and Bo Liu and Sanmi Koyejo and Sebastian Pokutta and Bo Han},
  booktitle={ICML 2026 Workshop: AI as a Tool for Mathematics, Computer Science, and Machine Learning},
  year={2026},
  url={https://openreview.net/forum?id=fnHhEf0cSE}
}

@inproceedings{
  zhou2026reasoning,
  title={Reasoning Is More Than the Model: Harness-Aware Evaluation of Agents on Verifiable Reasoning Tasks},
  author={Zhanke Zhou and Zongze Li and Weikai Huang and Xuan Li and Chentao Cao and Xiao Feng and Xiangyu Lu and Jinbo Hu and Menghan Lu and Yi Xie and Nico Pelleriti and Shiyang Liu and Max Zimmer and Brando Miranda and Jiangchao Yao and Bo Liu and Sanmi Koyejo and Sebastian Pokutta and Bo Han},
  booktitle={3rd AI for Math Workshop: Toward Self-Evolving Scientific Agents},
  year={2026},
  url={https://openreview.net/forum?id=4vARlk9o95}
}