Port pi-autoresearch to OpenClaw

Restructure monolithic pi-autoresearch extension into modular OpenClaw
plugin with proper tool cwd resolution via api.resolvePath and explicit
cwd overrides for nested repo targeting.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Gianfranco Piana
2026-03-13 17:47:17 -03:00
co-authored by Claude Opus 4.6
parent 2227029fa5
commit 9e651fcda8
42 changed files with 4597 additions and 5470 deletions
+2
View File
@@ -0,0 +1,2 @@
node_modules/
.tmp-*.md
+1
View File
@@ -1,6 +1,7 @@
MIT License
Copyright (c) 2026 Tobi Lutke, David Cortés
Copyright (c) 2026 Gianfranco Piana
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
+78 -109
View File
@@ -1,134 +1,103 @@
# pi-autoresearch — autonomous experiment loop for pi
# openclaw-autoresearch
**[Install](#install)** · **[Usage](#usage)** · **[How it works](#how-it-works)**
Autonomous experiment loop for any optimization target.
*Try an idea, measure it, keep what works, discard what doesn't, repeat forever.*
Faithful OpenClaw port of [`davebcn87/pi-autoresearch`](https://github.com/davebcn87/pi-autoresearch).
Inspired by [karpathy/autoresearch](https://github.com/karpathy/autoresearch). Works for any optimization target: test speed, bundle size, LLM training, build times, Lighthouse scores.
## How it works
---
The agent runs a loop: edit code, run a benchmark, measure the result, keep or discard. Each iteration is logged. The loop runs autonomously until interrupted.
![pi-autoresearch dashboard](pi-autoresearch.png)
Three tools drive the loop:
---
## What's included
| | |
| Tool | What it does |
|---|---|
| **Extension** | Tools + live widget + `/autoresearch` dashboard |
| **Skill** | Gathers what to optimize, writes session files, starts the loop |
| `init_experiment` | Configures the session: name, primary metric, unit, direction (lower/higher). Re-calling starts a new segment. |
| `run_experiment` | Executes a shell command, times it, captures stdout/stderr, returns pass/fail via exit code. |
| `log_experiment` | Records the result. `keep` auto-commits to git. `discard`/`crash` log without committing. Tracks secondary metrics alongside the primary. |
### Extension tools
Each tool also accepts an optional `cwd` so callers can target a nested repo explicitly instead of relying on the current session working directory.
| Tool | Description |
|------|-------------|
| `init_experiment` | One-time session config — name, metric, unit, direction |
| `run_experiment` | Runs any command, times wall-clock duration, captures output |
| `log_experiment` | Records result, auto-commits, updates widget and dashboard |
### UI
- **Status widget** — always visible above the editor: `🔬 autoresearch 12 runs 8 kept │ best: 42.3s`
- **`/autoresearch`** — full results dashboard (`Ctrl+X` to toggle, `Escape` to close)
### Skill
`autoresearch-create` asks a few questions (or infers from context) about your goal, command, metric, and files in scope — then writes two files and starts the loop immediately:
All state lives in four repo-root files:
| File | Purpose |
|------|---------|
| `autoresearch.md` | Session document — objective, metrics, files in scope, what's been tried. A fresh agent can resume from this alone. |
| `autoresearch.sh` | Benchmark script — pre-checks, runs the workload, outputs `METRIC name=number` lines. |
|---|---|
| `autoresearch.md` | Session doc: objective, metrics, files in scope, constraints, what's been tried. A fresh agent reads this to resume. |
| `autoresearch.sh` | Benchmark script. Outputs `METRIC name=number` lines. |
| `autoresearch.jsonl` | Structured log: config headers + experiment entries (metric, status, timestamp, segment, commit hash). |
| `autoresearch.ideas.md` | Backlog of promising ideas not yet tried. Optional. |
---
The design is file-first: any agent can pick up the repo-root files and continue the loop without prior context.
## Install
```bash
pi install https://github.com/davebcn87/pi-autoresearch
npm install
```
<details>
<summary>Manual install</summary>
Then load this repo path in OpenClaw plugin discovery and restart the gateway:
```yaml
plugins:
load:
paths:
- /absolute/path/to/openclaw-autoresearch
entries:
openclaw-autoresearch:
enabled: true
```
OpenClaw discovers `openclaw.plugin.json`, loads `extensions/openclaw-autoresearch/index.ts`, and exposes `autoresearch-create`.
Manual install is also possible: copy the plugin root, `extensions/openclaw-autoresearch/`, and `skills/autoresearch-create/` into your managed OpenClaw locations, then restart.
Verify:
- skill: `autoresearch-create`
- tools: `init_experiment`, `run_experiment`, `log_experiment`
- command: `/autoresearch` (recommended)
- direct skill fallback: `/skill autoresearch-create`
Prefer the explicit `/autoresearch` command surface in OpenClaw. The auto-generated native skill alias `/autoresearch_create` may not trigger reliably on some hosts, so use `/skill autoresearch-create` if you need to invoke the skill directly.
## Use
In the repo you want to optimize:
1. Load the plugin.
2. Run `/autoresearch` or `/autoresearch setup <goal>`.
3. Send a normal message with the goal, command, metric (+ direction), files in scope, and constraints.
4. If you need the raw skill invocation, use `/skill autoresearch-create`.
5. The agent writes `autoresearch.md` and `autoresearch.sh`, runs a baseline, then starts looping.
6. Use `/autoresearch` or `/autoresearch status` to re-prime context on a later turn.
To resume an existing session, a new agent reads the repo-root files and continues from where the last one stopped.
### User steers
Messages sent while an experiment is running are queued and surfaced after the next `log_experiment`. The agent finishes the current experiment before incorporating the steer.
### Ideas backlog
When the agent discovers promising but complex ideas mid-loop, it appends them to `autoresearch.ideas.md`. On resume, the agent reads the backlog, prunes stale entries, and uses the remaining ideas as experiment paths.
## Upstream reference
This port preserves upstream semantics, names, and file contracts while adapting presentation to OpenClaw. There is no Pi-style widget, dashboard, or editor shortcut layer. Remaining differences are tracked in [`docs/non-parity.md`](docs/non-parity.md).
- upstream repo: `https://github.com/davebcn87/pi-autoresearch`
- pinned upstream commit: `2227029fa5712944a36938b5fe59f709cb30ed22` (`2227029f`)
## Validation
```bash
cp -r extensions/pi-autoresearch ~/.pi/agent/extensions/
cp -r skills/autoresearch-create ~/.pi/agent/skills/
npm install --include=dev
npm run typecheck
npm test
npm run validate
```
Then `/reload` in pi.
</details>
---
## Usage
### 1. Start autoresearch
```
/skill:autoresearch-create
```
The agent asks about your goal, command, metric, and files in scope — or infers them from context. It then creates a branch, writes `autoresearch.md` and `autoresearch.sh`, runs the baseline, and starts looping immediately.
### 2. The loop
The agent runs autonomously: edit → commit → `run_experiment``log_experiment` → keep or revert → repeat. It never stops unless interrupted.
Every result is appended to `autoresearch.jsonl` in your project — one line per run. This means:
- **Survives restarts** — the agent can resume a session by reading the file
- **Survives context resets** — `autoresearch.md` captures what's been tried so a fresh agent has full context
- **Human readable** — open it anytime to see the full history
- **Branch-aware** — each branch has its own session
### 3. Monitor progress
- **Widget** — always visible above the editor
- **`/autoresearch`** — full dashboard with results table and best run
- **`Escape`** — interrupt anytime and ask for a summary
---
## Example domains
| Domain | Metric | Command |
|--------|--------|---------|
| Test speed | seconds ↓ | `pnpm test` |
| Bundle size | KB ↓ | `pnpm build && du -sb dist` |
| LLM training | val_bpb ↓ | `uv run train.py` |
| Build speed | seconds ↓ | `pnpm build` |
| Lighthouse | perf score ↑ | `lighthouse http://localhost:3000 --output=json` |
---
## How it works
The **extension** is domain-agnostic infrastructure. The **skill** encodes domain knowledge. This separation means one extension serves unlimited domains.
```
┌──────────────────────┐ ┌──────────────────────────┐
│ Extension (global) │ │ Skill (per-domain) │
│ │ │ │
│ run_experiment │◄────│ command: pnpm test │
│ log_experiment │ │ metric: seconds (lower) │
│ widget + dashboard │ │ scope: vitest configs │
│ │ │ ideas: pool, parallel… │
└──────────────────────┘ └──────────────────────────┘
```
Two files keep the session alive across restarts and context resets:
```
autoresearch.jsonl — append-only log of every run (metric, status, commit, description)
autoresearch.md — living document: objective, what's been tried, dead ends, key wins
```
A fresh agent with no memory can read these two files and continue exactly where the previous session left off.
---
The local test shim supports typechecking and tests without a full OpenClaw host checkout. Runtime behavior depends on a real OpenClaw host.
## License
+95
View File
@@ -0,0 +1,95 @@
# Non-Parity Notes for the OpenClaw Port
This repo aims to be a **faithful port** of `davebcn87/pi-autoresearch`, but it will not be a literal 1:1 port of Pi's UI/runtime.
Pinned upstream reference:
- Repo: `https://github.com/davebcn87/pi-autoresearch`
- Commit: `2227029fa5712944a36938b5fe59f709cb30ed22` (`2227029f`)
## Principle
Non-parity is acceptable only when it is forced by the host/runtime difference.
That means:
- preserve semantics first
- preserve names and file layout second
- adapt presentation/runtime integration last
## Expected non-parity in v1
### 1. No Pi widget parity
The Pi extension renders an always-visible status widget above the editor.
OpenClaw may provide a thinner status surface instead, such as:
- tool output
- command output
- lightweight summaries
### 2. No fullscreen dashboard / TUI parity
Pi provides an inline dashboard with keyboard interaction.
OpenClaw v1 should not try to fake this with a new UI system. If a status view exists, it should be thin and optional.
### 3. No keyboard shortcut parity
Pi has `Ctrl+X` and `Escape` affordances tied to its editor runtime.
These are considered host-specific and are not part of the core port contract.
### 4. Lifecycle hook names will differ
Pi uses hooks such as:
- `session_start`
- `session_switch`
- `session_fork`
- `session_tree`
- `before_agent_start`
- `agent_end`
- `input`
OpenClaw has a different hook model. We should preserve intent, not literal event names.
In practice, this port now uses documented OpenClaw lifecycle hooks such as:
- `before_prompt_build`
- `message_received`
- `agent_end`
- `session_end`
The result should preserve queued-steer handling and ideas-backlog continuation intent without pretending that Pi's hook names or UI affordances exist in OpenClaw.
### 5. `/autoresearch` command is thinner than Pi
The upstream repo includes a dedicated `/autoresearch` dashboard/entry surface.
OpenClaw v1 keeps the main UX command-first and implements `/autoresearch` as a mode and status helper that:
- provides a stable explicit entrypoint when host-native skill aliases vary
- detects canonical repo-root files
- enables or disables autoresearch mode for later agent turns
- offers terse status text
- points the agent back to `autoresearch.md`
It is intentionally not a dashboard replacement or a fullscreen UI.
## Non-parity that is **not** acceptable
The following would be design drift, not justified non-parity:
- moving canonical runtime files under `.autoresearch/` in v1
- renaming `init_experiment`, `run_experiment`, or `log_experiment`
- making a provider/runtime the product identity
- replacing the explicit command and direct-skill setup surface with a provider-specific worker-first UX
- changing keep/discard/crash behavior for convenience
- relying on hidden runtime state instead of file-first resumability
## Honest product statement for v1
The correct way to describe v1 is:
> A faithful OpenClaw port of `pi-autoresearch` that preserves upstream semantics, names, and file contracts, while explicitly not matching Pi's editor widget/dashboard UX.
## Future parity work
Possible later work, if OpenClaw surfaces support it cleanly:
- richer status presentation
- optional command polish
- optional provider adapters behind the plugin boundary
These should remain secondary to semantic fidelity.
@@ -0,0 +1,2 @@
- try a lighter parser
- remove redundant setup
@@ -0,0 +1,6 @@
{"type":"config","name":"Speed up tests","metricName":"seconds","metricUnit":"s","bestDirection":"lower"}
{"run":1,"commit":"aaaaaaa","metric":12.5,"metrics":{"compile_ms":40},"status":"keep","description":"baseline","timestamp":1,"segment":0}
{"run":2,"commit":"bbbbbbb","metric":11.8,"metrics":{"compile_ms":38},"status":"keep","description":"cache tweak","timestamp":2,"segment":0}
{"type":"config","name":"Speed up tests","metricName":"seconds","metricUnit":"s","bestDirection":"lower"}
{"run":1,"commit":"ccccccc","metric":10.9,"metrics":{"compile_ms":34,"bundle_kb":120},"status":"keep","description":"new baseline","timestamp":3,"segment":1}
{"run":2,"commit":"ddddddd","metric":11.4,"metrics":{"compile_ms":33,"bundle_kb":121},"status":"discard","description":"bad tradeoff","timestamp":4,"segment":1}
@@ -0,0 +1,8 @@
# Autoresearch: Speed up tests
## Objective
Make the benchmark faster.
## What's Been Tried
- cache tweak
- baseline reset after a strategy shift
+30
View File
@@ -0,0 +1,30 @@
import type { OpenClawPluginApi } from "openclaw/plugin-sdk/core";
import {
AUTORESEARCH_PLUGIN_DESCRIPTION,
AUTORESEARCH_PLUGIN_ID,
AUTORESEARCH_PLUGIN_NAME,
autoresearchPluginConfigSchema,
} from "./src/config.js";
import { createInitExperimentTool } from "./src/tools/init-experiment.js";
import { createRunExperimentTool } from "./src/tools/run-experiment.js";
import { createLogExperimentTool } from "./src/tools/log-experiment.js";
import { createAutoresearchStatusTool } from "./src/tools/autoresearch-status.js";
import { registerAutoresearchHooks } from "./src/hooks.js";
import { registerAutoresearchCommand } from "./src/commands/autoresearch.js";
const plugin = {
id: AUTORESEARCH_PLUGIN_ID,
name: AUTORESEARCH_PLUGIN_NAME,
description: AUTORESEARCH_PLUGIN_DESCRIPTION,
configSchema: autoresearchPluginConfigSchema,
register(api: OpenClawPluginApi) {
registerAutoresearchHooks(api);
registerAutoresearchCommand(api);
api.registerTool(createInitExperimentTool(api));
api.registerTool(createRunExperimentTool(api));
api.registerTool(createLogExperimentTool(api));
api.registerTool(createAutoresearchStatusTool(api));
},
};
export default plugin;
@@ -0,0 +1,182 @@
import * as fs from "node:fs";
import type { OpenClawPluginApi } from "openclaw/plugin-sdk/core";
import {
AUTORESEARCH_ROOT_FILES,
getAutoresearchRootFilePath,
type AutoresearchRootFileKey,
} from "../files.js";
import { reconstructStateFromJsonl } from "../state.js";
import { formatAutoresearchStatusText } from "../tools/autoresearch-status.js";
import {
clearAutoresearchSteers,
getAutoresearchRuntimeState,
setAutoresearchPendingCommand,
setAutoresearchRunInFlight,
setAutoresearchRuntimeMode,
} from "../runtime-state.js";
type CommandContext = {
args?: string;
channel?: string;
senderId?: string;
cwd?: string;
};
const COMMAND_USAGE = [
"Enable or inspect repo-root autoresearch mode.",
"",
"Usage:",
"/autoresearch",
"/autoresearch on",
"/autoresearch off",
"/autoresearch setup",
"/autoresearch status",
"/autoresearch help",
].join("\n");
export function registerAutoresearchCommand(api: OpenClawPluginApi): void {
api.registerCommand({
name: "autoresearch",
description: "Enable, disable, or inspect repo-root autoresearch mode.",
acceptsArgs: true,
handler: (ctx: CommandContext) => {
const cwd = resolveCommandCwd(api, ctx);
const rawArgs = (ctx.args ?? "").trim();
const [verb, ...rest] = rawArgs.split(/\s+/).filter(Boolean);
const action = (verb ?? "").toLowerCase();
const remainder = rest.join(" ").trim() || null;
if (!rawArgs || action === "resume" || action === "on") {
return {
text: enableAutoresearchMode(cwd, rawArgs && action !== "resume" && action !== "on" ? rawArgs : remainder),
};
}
if (action === "setup") {
return { text: primeAutoresearchSetup(cwd, remainder) };
}
if (action === "off") {
setAutoresearchRuntimeMode(cwd, "off");
setAutoresearchPendingCommand(cwd, null);
clearAutoresearchSteers(cwd);
setAutoresearchRunInFlight(cwd, false);
return {
text: [
"Autoresearch mode OFF.",
`Canonical files remain at repo root: ${Object.values(AUTORESEARCH_ROOT_FILES).join(", ")}`,
].join("\n"),
};
}
if (action === "status") {
return { text: buildAutoresearchCommandText(cwd, "status") };
}
if (action === "help") {
return { text: `${COMMAND_USAGE}\n\n${buildAutoresearchCommandText(cwd, "default")}` };
}
return {
text: enableAutoresearchMode(cwd, rawArgs),
};
},
});
}
export function buildAutoresearchCommandText(
cwd: string,
mode: "default" | "status",
): string {
const runtimeState = getAutoresearchRuntimeState(cwd);
const presentFiles = getPresentCanonicalFiles(cwd);
const hasSession = presentFiles.length > 0;
if (!hasSession) {
return [
"No repo-root autoresearch session detected.",
"",
`Expected canonical files: ${Object.values(AUTORESEARCH_ROOT_FILES).join(", ")}`,
"Recommended OpenClaw entrypoint: `/autoresearch` or `/autoresearch setup <goal>`.",
"Direct skill fallback: `/skill autoresearch-create`.",
].join("\n");
}
const state = reconstructStateFromJsonl(cwd);
const lines = [
`Autoresearch session detected at repo root: ${presentFiles.join(", ")}`,
`Read \`${AUTORESEARCH_ROOT_FILES.sessionDoc}\` before resuming or changing the loop.`,
];
if (mode === "status") {
lines.push("", formatAutoresearchStatusText(state, runtimeState));
} else if (state.mode === "active" || state.hasSessionDoc) {
lines.push(
"Use `/autoresearch` or `/autoresearch on` to enable mode for the next agent turn, then continue the upstream loop with `init_experiment`, `run_experiment`, and `log_experiment` as needed.",
);
} else {
lines.push(
`The canonical files exist, but the session brief looks incomplete. Open \`${AUTORESEARCH_ROOT_FILES.sessionDoc}\` and finish setup, or restart via \`/skill autoresearch-create\` if needed.`,
);
}
return lines.join("\n");
}
function enableAutoresearchMode(cwd: string, args: string | null): string {
setAutoresearchRuntimeMode(cwd, "on");
const presentFiles = getPresentCanonicalFiles(cwd);
const hasSession = presentFiles.length > 0;
if (!hasSession) {
setAutoresearchPendingCommand(cwd, {
kind: "setup",
args,
});
return [
"Autoresearch mode ON.",
"No repo-root session was detected, so the next agent turn will be primed for setup.",
"Next step: send a normal message so the next agent turn can gather setup details.",
"Direct skill fallback: `/skill autoresearch-create`.",
args ? `Captured setup instruction: ${args}` : "Run `/autoresearch setup <goal>` to attach a setup hint.",
].join("\n");
}
setAutoresearchPendingCommand(cwd, {
kind: "resume",
args,
});
return [
"Autoresearch mode ON.",
`Next agent turn will be primed from \`${AUTORESEARCH_ROOT_FILES.sessionDoc}\` and the canonical repo-root files.`,
args ? `Captured resume instruction: ${args}` : "Send a normal message to continue the loop, or use `/autoresearch status` for a snapshot first.",
].join("\n");
}
function primeAutoresearchSetup(cwd: string, args: string | null): string {
setAutoresearchRuntimeMode(cwd, "on");
setAutoresearchPendingCommand(cwd, {
kind: "setup",
args,
});
return [
"Autoresearch setup primed.",
"The next agent turn will be told to create the canonical repo-root files and start the loop.",
"Continue with a normal message on the next turn, or invoke the skill directly with `/skill autoresearch-create`.",
args ? `Captured setup instruction: ${args}` : "Add an argument to `/autoresearch setup` if you want a specific goal or constraint carried forward.",
].join("\n");
}
function resolveCommandCwd(api: OpenClawPluginApi, ctx: CommandContext): string {
if (typeof ctx.cwd === "string" && ctx.cwd.trim().length > 0) {
return ctx.cwd;
}
return api.resolvePath(".");
}
function getPresentCanonicalFiles(cwd: string): string[] {
const present: string[] = [];
for (const key of Object.keys(AUTORESEARCH_ROOT_FILES) as AutoresearchRootFileKey[]) {
if (fs.existsSync(getAutoresearchRootFilePath(cwd, key))) {
present.push(AUTORESEARCH_ROOT_FILES[key]);
}
}
return present;
}
@@ -0,0 +1,7 @@
import { emptyPluginConfigSchema } from "openclaw/plugin-sdk/core";
export const AUTORESEARCH_PLUGIN_ID = "openclaw-autoresearch";
export const AUTORESEARCH_PLUGIN_NAME = "Autoresearch";
export const AUTORESEARCH_PLUGIN_DESCRIPTION = "Faithful OpenClaw port of pi-autoresearch.";
export const autoresearchPluginConfigSchema = emptyPluginConfigSchema();
@@ -0,0 +1,122 @@
import { spawn } from "node:child_process";
const OUTPUT_TAIL_LINES = 80;
const DEFAULT_TIMEOUT_SECONDS = 600;
const FORCE_KILL_GRACE_MS = 1_000;
export type ExperimentExecutionResult = {
readonly command: string;
readonly exitCode: number | null;
readonly durationSeconds: number;
readonly passed: boolean;
readonly crashed: boolean;
readonly timedOut: boolean;
readonly tailOutput: string;
readonly stdout: string;
readonly stderr: string;
};
export async function executeExperimentCommand(options: {
command: string;
cwd: string;
timeoutSeconds?: number;
signal?: AbortSignal;
}): Promise<ExperimentExecutionResult> {
const timeoutSeconds = options.timeoutSeconds ?? DEFAULT_TIMEOUT_SECONDS;
const timeoutMs = Math.max(0, timeoutSeconds) * 1_000;
const startedAt = Date.now();
return await new Promise<ExperimentExecutionResult>((resolve) => {
const child = spawn("bash", ["-c", options.command], {
cwd: options.cwd,
stdio: ["ignore", "pipe", "pipe"],
});
let stdout = "";
let stderr = "";
let timedOut = false;
let forceKillTimer: NodeJS.Timeout | undefined;
const timeoutTimer =
timeoutMs > 0
? setTimeout(() => {
timedOut = true;
child.kill("SIGTERM");
forceKillTimer = setTimeout(() => {
if (!child.killed) {
child.kill("SIGKILL");
}
}, FORCE_KILL_GRACE_MS);
}, timeoutMs)
: undefined;
const abortHandler = () => {
child.kill("SIGTERM");
forceKillTimer = setTimeout(() => {
if (!child.killed) {
child.kill("SIGKILL");
}
}, FORCE_KILL_GRACE_MS);
};
if (options.signal) {
if (options.signal.aborted) {
abortHandler();
} else {
options.signal.addEventListener("abort", abortHandler, { once: true });
}
}
child.stdout.on("data", (chunk: string | Buffer) => {
stdout += chunk.toString();
});
child.stderr.on("data", (chunk: string | Buffer) => {
stderr += chunk.toString();
});
child.on("error", (error) => {
stderr += `${stderr ? "\n" : ""}${String(error.message || error)}`;
});
child.on("close", (code) => {
if (timeoutTimer) {
clearTimeout(timeoutTimer);
}
if (forceKillTimer) {
clearTimeout(forceKillTimer);
}
if (options.signal) {
options.signal.removeEventListener("abort", abortHandler);
}
const durationSeconds = (Date.now() - startedAt) / 1_000;
const passed = code === 0 && !timedOut;
resolve({
command: options.command,
exitCode: code,
durationSeconds,
passed,
crashed: !passed,
timedOut,
tailOutput: createOutputTail(stdout, stderr),
stdout,
stderr,
});
});
});
}
function createOutputTail(stdout: string, stderr: string): string {
const combined = [stdout, stderr]
.filter((value) => value.trim().length > 0)
.join("\n")
.trim();
if (!combined) {
return "";
}
return combined.split(/\r?\n/).slice(-OUTPUT_TAIL_LINES).join("\n");
}
@@ -0,0 +1,37 @@
import * as fs from "node:fs";
export const AUTORESEARCH_ROOT_FILES = {
sessionDoc: "autoresearch.md",
runnerScript: "autoresearch.sh",
resultsLog: "autoresearch.jsonl",
ideasBacklog: "autoresearch.ideas.md",
} as const;
export type AutoresearchRootFileKey = keyof typeof AUTORESEARCH_ROOT_FILES;
export function getAutoresearchRootFilePath(
cwd: string,
file: AutoresearchRootFileKey,
): string {
return `${cwd}/${AUTORESEARCH_ROOT_FILES[file]}`;
}
export function readAutoresearchRootFile(
cwd: string,
file: AutoresearchRootFileKey,
): string | null {
const filePath = getAutoresearchRootFilePath(cwd, file);
if (!fs.existsSync(filePath)) {
return null;
}
return fs.readFileSync(filePath, "utf8");
}
/**
* PR 2 skeleton only.
* This module will own canonical root-level file IO helpers in later PRs.
*/
export function describeCanonicalFiles(): typeof AUTORESEARCH_ROOT_FILES {
return AUTORESEARCH_ROOT_FILES;
}
+132
View File
@@ -0,0 +1,132 @@
import { spawnSync } from "node:child_process";
export type GitCommandResult = {
readonly code: number | null;
readonly stdout: string;
readonly stderr: string;
readonly combinedOutput: string;
};
export type GitKeepResult = {
readonly attempted: true;
readonly committed: boolean;
readonly commit: string;
readonly summary: string;
readonly command: GitCommandResult;
};
function runGitCommand(cwd: string, args: readonly string[]): GitCommandResult {
const result = spawnSync("git", [...args], {
cwd,
encoding: "utf8",
});
const stdout = result.stdout ?? "";
const stderr = result.stderr ?? "";
return {
code: result.status,
stdout,
stderr,
combinedOutput: `${stdout}${stderr}`.trim(),
};
}
export function commitKeptExperiment(options: {
cwd: string;
description: string;
metricName: string;
metric: number;
metrics: Record<string, number>;
commit: string;
status: "keep";
}): GitKeepResult {
const resultData: Record<string, unknown> = {
status: options.status,
[options.metricName || "metric"]: options.metric,
...options.metrics,
};
const commitMessage = `${options.description}\n\nResult: ${JSON.stringify(resultData)}`;
const repoRootResult = runGitCommand(options.cwd, ["rev-parse", "--show-toplevel"]);
if (repoRootResult.code !== 0 || repoRootResult.stdout.trim().length === 0) {
return {
attempted: true,
committed: false,
commit: options.commit,
summary: `Git repo check failed${formatExit(repoRootResult.code)}: ${truncateOutput(repoRootResult.combinedOutput)}`,
command: repoRootResult,
};
}
const repoRoot = repoRootResult.stdout.trim();
const addResult = runGitCommand(repoRoot, ["add", "-A"]);
if (addResult.code !== 0) {
return {
attempted: true,
committed: false,
commit: options.commit,
summary: `Git add failed${formatExit(addResult.code)}: ${truncateOutput(addResult.combinedOutput)}`,
command: addResult,
};
}
const diffResult = runGitCommand(repoRoot, ["diff", "--cached", "--quiet"]);
if (diffResult.code === 0) {
return {
attempted: true,
committed: false,
commit: options.commit,
summary: "Git: nothing to commit (working tree clean)",
command: diffResult,
};
}
if (diffResult.code !== 1) {
return {
attempted: true,
committed: false,
commit: options.commit,
summary: `Git diff check failed${formatExit(diffResult.code)}: ${truncateOutput(diffResult.combinedOutput)}`,
command: diffResult,
};
}
const commitResult = runGitCommand(repoRoot, ["commit", "-m", commitMessage]);
if (commitResult.code !== 0) {
return {
attempted: true,
committed: false,
commit: options.commit,
summary: `Git commit failed${formatExit(commitResult.code)}: ${truncateOutput(commitResult.combinedOutput)}`,
command: commitResult,
};
}
const revParseResult = runGitCommand(repoRoot, ["rev-parse", "--short=7", "HEAD"]);
const actualCommit =
revParseResult.code === 0 && revParseResult.stdout.trim().length >= 7
? revParseResult.stdout.trim().slice(0, 7)
: options.commit;
const firstLine = commitResult.combinedOutput.split("\n")[0]?.trim() || "commit created";
return {
attempted: true,
committed: true,
commit: actualCommit,
summary: `Git: committed - ${firstLine}`,
command: commitResult,
};
}
function truncateOutput(output: string): string {
const normalized = output.trim();
if (!normalized) {
return "no output";
}
return normalized.length > 200 ? `${normalized.slice(0, 200)}...` : normalized;
}
function formatExit(code: number | null): string {
return code === null ? "" : ` (exit ${code})`;
}
@@ -0,0 +1,234 @@
import type { OpenClawPluginApi } from "openclaw/plugin-sdk/core";
import { AUTORESEARCH_ROOT_FILES } from "./files.js";
import { reconstructStateFromJsonl } from "./state.js";
import {
clearAutoresearchRuntimeState,
consumeAutoresearchContinuationReminder,
consumeAutoresearchPendingCommand,
getAutoresearchRuntimeState,
queueAutoresearchSteer,
setAutoresearchContinuationReminder,
} from "./runtime-state.js";
type BeforeAgentStartEvent = {
systemPrompt?: string;
};
type HookContext = {
cwd?: string;
};
type HookCapablePluginApi = OpenClawPluginApi & {
on?: (hookName: string, handler: (event: unknown, ctx: HookContext) => unknown) => void;
registerHook?: (
hookName: string,
handler: (event: BeforeAgentStartEvent, ctx: HookContext) => BeforeAgentStartEvent | void,
) => void;
};
export function registerAutoresearchHooks(api: OpenClawPluginApi): void {
const hookApi = api as HookCapablePluginApi;
if (typeof hookApi.on === "function") {
hookApi.on("before_prompt_build", (_event, ctx) => {
const cwd = resolveHookCwd(api, ctx);
if (cwd === null) {
return;
}
const addition = buildBeforePromptBuildContext(cwd);
if (addition === null) {
return;
}
return {
appendSystemContext: addition,
};
});
hookApi.on("message_received", (event, ctx) => {
const cwd = resolveHookCwd(api, ctx);
if (cwd === null) {
return;
}
const state = getAutoresearchRuntimeState(cwd);
if (!state.runInFlight) {
return;
}
const messageText = extractMessageText(event);
if (messageText === null || isCommandLikeMessage(messageText)) {
return;
}
queueAutoresearchSteer(cwd, messageText);
});
hookApi.on("agent_end", (_event, ctx) => {
const cwd = resolveHookCwd(api, ctx);
if (cwd === null) {
return;
}
const state = reconstructStateFromJsonl(cwd);
if (state.mode === "active" && state.ideas.hasBacklog) {
setAutoresearchContinuationReminder(cwd, true);
}
});
hookApi.on("session_end", (_event, ctx) => {
const cwd = resolveHookCwd(api, ctx);
if (cwd === null) {
return;
}
clearAutoresearchRuntimeState(cwd);
});
return;
}
if (typeof hookApi.registerHook !== "function") {
return;
}
hookApi.registerHook("before_agent_start", (event, ctx) => {
const cwd = resolveHookCwd(api, ctx);
if (cwd === null) {
return;
}
const addition = buildBeforePromptBuildContext(cwd);
if (addition === null) {
return;
}
return {
...event,
systemPrompt: `${event.systemPrompt ?? ""}${addition}`,
};
});
}
export function buildBeforePromptBuildContext(cwd: string): string | null {
const state = reconstructStateFromJsonl(cwd);
const runtimeState = getAutoresearchRuntimeState(cwd);
const modeEnabled =
runtimeState.mode === "on" ||
(runtimeState.mode !== "off" && (state.mode === "active" || state.hasSessionDoc));
if (!modeEnabled) {
return null;
}
const canonicalFiles = [
AUTORESEARCH_ROOT_FILES.sessionDoc,
AUTORESEARCH_ROOT_FILES.runnerScript,
AUTORESEARCH_ROOT_FILES.resultsLog,
];
const pendingCommand = consumeAutoresearchPendingCommand(cwd);
const needsContinuationReminder = consumeAutoresearchContinuationReminder(cwd);
const lines = ["", "", "## Autoresearch Mode (ACTIVE)"];
if (pendingCommand?.kind === "setup" || !state.hasSessionDoc) {
lines.push(
`No ${AUTORESEARCH_ROOT_FILES.sessionDoc} was detected. Gather context and set up the experiment now with the canonical repo-root files.`,
`Create ${AUTORESEARCH_ROOT_FILES.sessionDoc} and ${AUTORESEARCH_ROOT_FILES.runnerScript}, then initialize the loop with init_experiment, run_experiment, and log_experiment.`,
);
if (pendingCommand?.args) {
lines.push(`Additional setup instruction from /autoresearch: ${pendingCommand.args}`);
}
} else {
lines.push(
`Autoresearch files live at repo root: ${canonicalFiles.join(", ")}.`,
`Read ${AUTORESEARCH_ROOT_FILES.sessionDoc} before resuming or changing the experiment loop, and re-read it after compaction.`,
"Resume the autonomous upstream loop: edit, run_experiment, log_experiment, keep/discard/crash, repeat.",
"Use init_experiment, run_experiment, and log_experiment for experiment state changes. Never stop unless the user explicitly interrupts the loop.",
);
if (pendingCommand?.args) {
lines.push(`Additional resume instruction from /autoresearch: ${pendingCommand.args}`);
}
}
lines.push(
`For discard or crash results, log_experiment records the outcome but does not revert your tree for you. Run \`git checkout -- .\` yourself after logging when you want to discard tracked changes.`,
);
if (state.ideas.hasBacklog) {
lines.push(
`${AUTORESEARCH_ROOT_FILES.ideasBacklog} exists with ${state.ideas.pendingCount} pending idea${state.ideas.pendingCount === 1 ? "" : "s"}; use it as continuation fuel for promising paths you have not exhausted.`,
);
}
if (needsContinuationReminder && state.ideas.hasBacklog) {
lines.push(
`The previous autoresearch run ended with pending ideas. Read ${AUTORESEARCH_ROOT_FILES.ideasBacklog}, prune stale items, and spin those ideas into the next experiments before declaring the work done.`,
);
}
if (runtimeState.queuedSteers.length > 0) {
lines.push(
`${runtimeState.queuedSteers.length} user steer${runtimeState.queuedSteers.length === 1 ? "" : "s"} arrived during the current experiment window. If the next followup turn repeats a steer already surfaced in log_experiment output, treat it as the same request rather than a new branch of work.`,
);
}
return lines.join("\n");
}
function resolveHookCwd(api: OpenClawPluginApi, ctx: HookContext | undefined): string | null {
if (ctx && typeof ctx.cwd === "string" && ctx.cwd.trim().length > 0) {
return ctx.cwd;
}
try {
const resolved = api.resolvePath(".");
return resolved.trim().length > 0 ? resolved : null;
} catch {
return null;
}
}
function extractMessageText(event: unknown): string | null {
if (!event || typeof event !== "object") {
return null;
}
const record = event as Record<string, unknown>;
const direct = firstString(record.text, record.content, record.body, record.prompt);
if (direct) {
return direct;
}
const message = record.message;
if (message && typeof message === "object") {
const nested = message as Record<string, unknown>;
const messageText = firstString(nested.text, nested.content, nested.body);
if (messageText) {
return messageText;
}
}
const context = record.context;
if (context && typeof context === "object") {
const nested = context as Record<string, unknown>;
const contextText = firstString(nested.content, nested.commandBody, nested.text);
if (contextText) {
return contextText;
}
}
return null;
}
function firstString(...values: unknown[]): string | null {
for (const value of values) {
if (typeof value === "string" && value.trim().length > 0) {
return value.trim();
}
}
return null;
}
function isCommandLikeMessage(text: string): boolean {
return /^[\/!]/.test(text.trim());
}
@@ -0,0 +1,55 @@
import * as fs from "node:fs";
import { getAutoresearchRootFilePath } from "./files.js";
export type AutoresearchConfigHeader = {
type: "config";
name: string;
metricName: string;
metricUnit: string;
bestDirection: "lower" | "higher";
};
export function createConfigHeader(config: {
name: string;
metricName: string;
metricUnit: string;
bestDirection: "lower" | "higher";
}): AutoresearchConfigHeader {
return {
type: "config",
name: config.name,
metricName: config.metricName,
metricUnit: config.metricUnit,
bestDirection: config.bestDirection,
};
}
export function writeConfigHeader(
cwd: string,
header: AutoresearchConfigHeader,
mode: "create" | "append",
): void {
const jsonlPath = getAutoresearchRootFilePath(cwd, "resultsLog");
const line = `${JSON.stringify(header)}\n`;
if (mode === "append") {
fs.appendFileSync(jsonlPath, line);
return;
}
fs.writeFileSync(jsonlPath, line);
}
export type AutoresearchResultEntry = {
readonly run: number;
readonly commit: string;
readonly metric: number;
readonly metrics: Record<string, number>;
readonly status: "keep" | "discard" | "crash";
readonly description: string;
readonly timestamp: number;
readonly segment: number;
};
export function appendResultEntry(cwd: string, entry: AutoresearchResultEntry): void {
const jsonlPath = getAutoresearchRootFilePath(cwd, "resultsLog");
fs.appendFileSync(jsonlPath, `${JSON.stringify(entry)}\n`);
}
@@ -0,0 +1,143 @@
export type AutoresearchRuntimeMode = "auto" | "on" | "off";
export type PendingAutoresearchCommand =
| {
readonly kind: "resume" | "setup";
readonly args: string | null;
}
| null;
export type AutoresearchRuntimeSnapshot = {
readonly mode: AutoresearchRuntimeMode;
readonly runInFlight: boolean;
readonly queuedSteers: readonly string[];
readonly needsContinuationReminder: boolean;
readonly pendingCommand: PendingAutoresearchCommand;
};
type MutableAutoresearchRuntimeState = {
mode: AutoresearchRuntimeMode;
runInFlight: boolean;
queuedSteers: string[];
needsContinuationReminder: boolean;
pendingCommand: PendingAutoresearchCommand;
};
const MAX_QUEUED_STEERS = 20;
const runtimeStates = new Map<string, MutableAutoresearchRuntimeState>();
function createDefaultRuntimeState(): MutableAutoresearchRuntimeState {
return {
mode: "auto",
runInFlight: false,
queuedSteers: [],
needsContinuationReminder: false,
pendingCommand: null,
};
}
function getMutableRuntimeState(cwd: string): MutableAutoresearchRuntimeState {
let state = runtimeStates.get(cwd);
if (!state) {
state = createDefaultRuntimeState();
runtimeStates.set(cwd, state);
}
return state;
}
export function getAutoresearchRuntimeState(cwd: string): AutoresearchRuntimeSnapshot {
const state = getMutableRuntimeState(cwd);
return {
mode: state.mode,
runInFlight: state.runInFlight,
queuedSteers: [...state.queuedSteers],
needsContinuationReminder: state.needsContinuationReminder,
pendingCommand: state.pendingCommand,
};
}
export function setAutoresearchRuntimeMode(
cwd: string,
mode: AutoresearchRuntimeMode,
): AutoresearchRuntimeSnapshot {
const state = getMutableRuntimeState(cwd);
state.mode = mode;
return getAutoresearchRuntimeState(cwd);
}
export function setAutoresearchRunInFlight(
cwd: string,
runInFlight: boolean,
): AutoresearchRuntimeSnapshot {
const state = getMutableRuntimeState(cwd);
state.runInFlight = runInFlight;
return getAutoresearchRuntimeState(cwd);
}
export function queueAutoresearchSteer(
cwd: string,
steer: string,
): AutoresearchRuntimeSnapshot {
const normalized = steer.trim();
if (!normalized) {
return getAutoresearchRuntimeState(cwd);
}
const state = getMutableRuntimeState(cwd);
state.queuedSteers.push(normalized);
if (state.queuedSteers.length > MAX_QUEUED_STEERS) {
state.queuedSteers = state.queuedSteers.slice(-MAX_QUEUED_STEERS);
}
return getAutoresearchRuntimeState(cwd);
}
export function consumeAutoresearchSteers(cwd: string): readonly string[] {
const state = getMutableRuntimeState(cwd);
const queued = [...state.queuedSteers];
state.queuedSteers = [];
return queued;
}
export function clearAutoresearchSteers(cwd: string): AutoresearchRuntimeSnapshot {
const state = getMutableRuntimeState(cwd);
state.queuedSteers = [];
return getAutoresearchRuntimeState(cwd);
}
export function setAutoresearchPendingCommand(
cwd: string,
pendingCommand: PendingAutoresearchCommand,
): AutoresearchRuntimeSnapshot {
const state = getMutableRuntimeState(cwd);
state.pendingCommand = pendingCommand;
return getAutoresearchRuntimeState(cwd);
}
export function consumeAutoresearchPendingCommand(
cwd: string,
): PendingAutoresearchCommand {
const state = getMutableRuntimeState(cwd);
const pending = state.pendingCommand;
state.pendingCommand = null;
return pending;
}
export function setAutoresearchContinuationReminder(
cwd: string,
needsReminder: boolean,
): AutoresearchRuntimeSnapshot {
const state = getMutableRuntimeState(cwd);
state.needsContinuationReminder = needsReminder;
return getAutoresearchRuntimeState(cwd);
}
export function consumeAutoresearchContinuationReminder(cwd: string): boolean {
const state = getMutableRuntimeState(cwd);
const needsReminder = state.needsContinuationReminder;
state.needsContinuationReminder = false;
return needsReminder;
}
export function clearAutoresearchRuntimeState(cwd: string): void {
runtimeStates.delete(cwd);
}
@@ -0,0 +1,290 @@
import { readAutoresearchRootFile } from "./files.js";
export type SecondaryMetricDef = {
readonly name: string;
readonly unit: string;
};
export type AutoresearchMode = "inactive" | "active";
export type AutoresearchIdeasSnapshot = {
readonly hasBacklog: boolean;
readonly pendingCount: number;
readonly preview: readonly string[];
};
export type AutoresearchRunSnapshot = {
readonly run: number;
readonly commit: string;
readonly metric: number;
readonly metrics: Record<string, number>;
readonly status: "keep" | "discard" | "crash";
readonly description: string;
readonly timestamp: number;
readonly segment: number;
};
export type AutoresearchStateSnapshot = {
readonly name: string | null;
readonly metricName: string;
readonly metricUnit: string;
readonly bestDirection: "lower" | "higher";
readonly secondaryMetrics: readonly SecondaryMetricDef[];
readonly currentSegment: number;
readonly currentRunCount: number;
readonly totalRunCount: number;
readonly currentBaselineMetric: number | null;
readonly currentBestMetric: number | null;
readonly lastRun: AutoresearchRunSnapshot | null;
readonly mode: AutoresearchMode;
readonly hasSessionDoc: boolean;
readonly ideas: AutoresearchIdeasSnapshot;
};
type MutableStateSnapshot = {
name: string | null;
metricName: string;
metricUnit: string;
bestDirection: "lower" | "higher";
currentSegment: number;
currentRunCount: number;
totalRunCount: number;
currentBaselineMetric: number | null;
currentBestMetric: number | null;
lastRun: AutoresearchRunSnapshot | null;
mode: AutoresearchMode;
hasSessionDoc: boolean;
ideas: AutoresearchIdeasSnapshot;
};
type JsonlEntry = {
readonly type?: string;
readonly name?: string;
readonly metricName?: string;
readonly metricUnit?: string;
readonly bestDirection?: "lower" | "higher";
readonly run?: number;
readonly commit?: string;
readonly metric?: number;
readonly metrics?: Record<string, number>;
readonly status?: "keep" | "discard" | "crash";
readonly description?: string;
readonly timestamp?: number;
readonly segment?: number;
};
export function createEmptyStateSnapshot(): AutoresearchStateSnapshot {
return {
name: null,
metricName: "metric",
metricUnit: "",
bestDirection: "lower",
secondaryMetrics: [],
currentSegment: 0,
currentRunCount: 0,
totalRunCount: 0,
currentBaselineMetric: null,
currentBestMetric: null,
lastRun: null,
mode: "inactive",
hasSessionDoc: false,
ideas: {
hasBacklog: false,
pendingCount: 0,
preview: [],
},
};
}
function createMutableEmptyStateSnapshot(): MutableStateSnapshot {
return {
...createEmptyStateSnapshot(),
};
}
export function reconstructStateFromJsonl(cwd: string): AutoresearchStateSnapshot {
const sessionDoc = readAutoresearchRootFile(cwd, "sessionDoc");
const ideasBacklog = readAutoresearchRootFile(cwd, "ideasBacklog");
const jsonl = readAutoresearchRootFile(cwd, "resultsLog");
const state = createMutableEmptyStateSnapshot();
state.hasSessionDoc = sessionDoc !== null;
state.mode = detectAutoresearchMode(sessionDoc);
state.ideas = summarizeIdeasBacklog(ideasBacklog);
if (jsonl === null) {
return {
...state,
secondaryMetrics: [],
};
}
const currentSecondaryMetrics = new Map<string, SecondaryMetricDef>();
let currentRunIndex = 0;
let hasSeenAnyRun = false;
const lines = jsonl
.split("\n")
.map((line) => line.trim())
.filter(Boolean);
for (const line of lines) {
let entry: JsonlEntry;
try {
entry = JSON.parse(line) as JsonlEntry;
} catch {
continue;
}
if (entry.type === "config") {
if (entry.name) {
state.name = entry.name;
}
if (entry.metricName) {
state.metricName = entry.metricName;
}
if (entry.metricUnit !== undefined) {
state.metricUnit = entry.metricUnit;
}
if (entry.bestDirection === "lower" || entry.bestDirection === "higher") {
state.bestDirection = entry.bestDirection;
}
if (hasSeenAnyRun) {
state.currentSegment += 1;
}
state.currentRunCount = 0;
state.currentBaselineMetric = null;
state.currentBestMetric = null;
currentRunIndex = 0;
currentSecondaryMetrics.clear();
continue;
}
if (typeof entry.metric !== "number") {
continue;
}
hasSeenAnyRun = true;
currentRunIndex += 1;
state.currentRunCount = currentRunIndex;
state.totalRunCount += 1;
const run: AutoresearchRunSnapshot = {
run: typeof entry.run === "number" ? entry.run : currentRunIndex,
commit: entry.commit ?? "",
metric: entry.metric,
metrics: normalizeMetrics(entry.metrics),
status: entry.status ?? "keep",
description: entry.description ?? "",
timestamp: typeof entry.timestamp === "number" ? entry.timestamp : 0,
segment: typeof entry.segment === "number" ? entry.segment : state.currentSegment,
};
if (state.currentBaselineMetric === null) {
state.currentBaselineMetric = run.metric;
}
if (run.status === "keep" && run.metric > 0) {
if (
state.currentBestMetric === null ||
isBetter(run.metric, state.currentBestMetric, state.bestDirection)
) {
state.currentBestMetric = run.metric;
}
}
for (const metricName of Object.keys(run.metrics)) {
if (!currentSecondaryMetrics.has(metricName)) {
currentSecondaryMetrics.set(metricName, {
name: metricName,
unit: inferMetricUnit(metricName),
});
}
}
state.lastRun = run;
}
return {
...state,
secondaryMetrics: [...currentSecondaryMetrics.values()],
};
}
function normalizeMetrics(metrics: Record<string, number> | undefined): Record<string, number> {
if (!metrics || typeof metrics !== "object") {
return {};
}
return Object.fromEntries(
Object.entries(metrics).filter(([, value]) => typeof value === "number"),
);
}
function detectAutoresearchMode(sessionDoc: string | null): AutoresearchMode {
if (sessionDoc === null) {
return "inactive";
}
const normalized = sessionDoc.toLowerCase();
if (
normalized.includes("# autoresearch") ||
normalized.includes("## objective") ||
normalized.includes("## what's been tried") ||
normalized.includes("## how to run")
) {
return "active";
}
return sessionDoc.trim().length > 0 ? "active" : "inactive";
}
function summarizeIdeasBacklog(ideasBacklog: string | null): AutoresearchIdeasSnapshot {
if (ideasBacklog === null) {
return {
hasBacklog: false,
pendingCount: 0,
preview: [],
};
}
const ideas = ideasBacklog
.split("\n")
.map((line) => line.trim())
.filter((line) => /^([-*+]|\d+\.)\s+/.test(line))
.map((line) => line.replace(/^([-*+]|\d+\.)\s+/, "").trim())
.filter(Boolean);
return {
hasBacklog: ideas.length > 0,
pendingCount: ideas.length,
preview: ideas.slice(0, 3),
};
}
function isBetter(
current: number,
best: number,
direction: "lower" | "higher",
): boolean {
return direction === "lower" ? current < best : current > best;
}
function inferMetricUnit(name: string): string {
if (name.endsWith("_µs") || name.includes("µs")) {
return "µs";
}
if (name.endsWith("_ms") || name.includes("ms")) {
return "ms";
}
if (name.endsWith("_s") || name.includes("sec")) {
return "s";
}
if (name.endsWith("_kb") || name.includes("kb")) {
return "kb";
}
if (name.endsWith("_mb") || name.includes("mb")) {
return "mb";
}
return "";
}
@@ -0,0 +1,99 @@
import type { OpenClawPluginApi } from "openclaw/plugin-sdk/core";
import { Type } from "@sinclair/typebox";
import { reconstructStateFromJsonl, type AutoresearchStateSnapshot } from "../state.js";
import { getAutoresearchRuntimeState, type AutoresearchRuntimeSnapshot } from "../runtime-state.js";
import { resolveToolCwd } from "./tool-cwd.js";
const AutoresearchStatusParams = Type.Object(
{
cwd: Type.Optional(
Type.String({
description:
"Optional working directory for repo-local autoresearch state. Use this when the tool call originates outside the target repo session cwd.",
}),
),
},
{ additionalProperties: false },
);
export function createAutoresearchStatusTool(api: OpenClawPluginApi) {
return {
name: "autoresearch_status",
label: "Autoresearch Status",
description:
"Read-only summary of autoresearch state reconstructed from root-level files.",
parameters: AutoresearchStatusParams,
async execute(
_toolCallId: string,
params: {
cwd?: string;
},
_signal: AbortSignal,
_onUpdate: unknown,
) {
const cwd = resolveToolCwd(api, params.cwd);
const state = reconstructStateFromJsonl(cwd);
const runtimeState = getAutoresearchRuntimeState(cwd);
return {
content: [{ type: "text" as const, text: formatAutoresearchStatusText(state, runtimeState) }],
details: {
status: "ok",
state,
runtime: runtimeState,
},
};
},
};
}
export function formatAutoresearchStatusText(
state: AutoresearchStateSnapshot,
runtimeState?: AutoresearchRuntimeSnapshot,
): string {
const lines = [
`Mode: ${state.mode}`,
`Session doc: ${state.hasSessionDoc ? "present" : "missing"}`,
`Ideas backlog: ${state.ideas.hasBacklog ? `${state.ideas.pendingCount} pending` : "empty"}`,
`Metric: ${state.metricName} (${state.metricUnit || "unitless"}, ${state.bestDirection} is better)`,
`Current segment: ${state.currentSegment}`,
`Runs: ${state.currentRunCount} current / ${state.totalRunCount} total`,
`Baseline: ${formatMetric(state.currentBaselineMetric, state.metricUnit)}`,
`Best kept: ${formatMetric(state.currentBestMetric, state.metricUnit)}`,
];
if (state.name) {
lines.splice(1, 0, `Session: ${state.name}`);
}
if (runtimeState) {
lines.splice(
state.name ? 2 : 1,
0,
`Runtime mode: ${runtimeState.mode}`,
`Experiment window: ${runtimeState.runInFlight ? "running" : "idle"}`,
`Queued steers: ${runtimeState.queuedSteers.length}`,
);
}
if (state.lastRun) {
lines.push(
`Last run: #${state.lastRun.run} ${state.lastRun.status} ${formatMetric(state.lastRun.metric, state.metricUnit)} ${state.lastRun.commit} ${state.lastRun.description}`.trim(),
);
}
if (state.ideas.preview.length > 0) {
lines.push(`Ideas preview: ${state.ideas.preview.join(" | ")}`);
}
return lines.join("\n");
}
function formatMetric(value: number | null, unit: string): string {
if (value === null) {
return "n/a";
}
const rendered = value === Math.round(value) ? `${Math.round(value)}` : value.toFixed(2);
return `${rendered}${unit}`;
}
@@ -0,0 +1,90 @@
import type { OpenClawPluginApi } from "openclaw/plugin-sdk/core";
import { InitExperimentParams } from "./schemas.js";
import { createConfigHeader, writeConfigHeader } from "../logging.js";
import {
createEmptyStateSnapshot,
reconstructStateFromJsonl,
type AutoresearchStateSnapshot,
} from "../state.js";
import { resolveToolCwd } from "./tool-cwd.js";
export function createInitExperimentTool(api: OpenClawPluginApi) {
return {
name: "init_experiment",
label: "Init Experiment",
description:
"Initialize the experiment session. Call once before the first run_experiment to set the name, primary metric, unit, and direction. Writes the config header to autoresearch.jsonl.",
parameters: InitExperimentParams,
async execute(
_toolCallId: string,
params: {
cwd?: string;
name: string;
metric_name: string;
metric_unit?: string;
direction?: "lower" | "higher";
},
_signal: AbortSignal,
_onUpdate: unknown,
) {
const cwd = resolveToolCwd(api, params.cwd);
const previousState = reconstructStateFromJsonl(cwd);
const isReinit = previousState.currentRunCount > 0;
const nextState: AutoresearchStateSnapshot = {
...createEmptyStateSnapshot(),
name: params.name,
metricName: params.metric_name,
metricUnit: params.metric_unit ?? "",
bestDirection: params.direction ?? "lower",
currentSegment: isReinit ? previousState.currentSegment + 1 : previousState.currentSegment,
};
try {
writeConfigHeader(
cwd,
createConfigHeader({
name: nextState.name ?? params.name,
metricName: nextState.metricName,
metricUnit: nextState.metricUnit,
bestDirection: nextState.bestDirection,
}),
isReinit ? "append" : "create",
);
} catch (error) {
return {
content: [
{
type: "text" as const,
text: `Failed to write autoresearch.jsonl: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
details: {
status: "error",
},
};
}
const reinitNote = isReinit
? " (re-initialized - previous results archived, new baseline needed)"
: "";
return {
content: [
{
type: "text" as const,
text:
`Experiment initialized: "${nextState.name}"${reinitNote}\n` +
`Metric: ${nextState.metricName} (${nextState.metricUnit || "unitless"}, ${nextState.bestDirection} is better)\n` +
"Config written to autoresearch.jsonl. Now run the baseline with run_experiment.",
},
],
details: {
status: "ok",
state: nextState,
},
};
},
};
}
@@ -0,0 +1,378 @@
import * as fs from "node:fs";
import type { OpenClawPluginApi } from "openclaw/plugin-sdk/core";
import { LogExperimentParams } from "./schemas.js";
import { commitKeptExperiment } from "../git.js";
import { appendResultEntry, type AutoresearchResultEntry } from "../logging.js";
import { getAutoresearchRootFilePath } from "../files.js";
import {
reconstructStateFromJsonl,
type AutoresearchStateSnapshot,
type SecondaryMetricDef,
} from "../state.js";
import {
consumeAutoresearchSteers,
setAutoresearchRunInFlight,
} from "../runtime-state.js";
import { resolveToolCwd } from "./tool-cwd.js";
export function createLogExperimentTool(api: OpenClawPluginApi) {
return {
name: "log_experiment",
label: "Log Experiment",
description:
"Record an experiment result. Tracks metrics and preserves keep/discard/crash semantics. Call after every run_experiment.",
parameters: LogExperimentParams,
async execute(
_toolCallId: string,
params: {
cwd?: string;
commit: string;
metric: number;
status: "keep" | "discard" | "crash";
description: string;
metrics?: Record<string, number>;
force?: boolean;
},
_signal: AbortSignal,
_onUpdate: unknown,
) {
const cwd = resolveToolCwd(api, params.cwd);
const state = reconstructStateFromJsonl(cwd);
const secondaryMetrics = params.metrics ?? {};
if (state.secondaryMetrics.length > 0) {
const validationError = validateSecondaryMetrics(
state.secondaryMetrics,
secondaryMetrics,
params.force ?? false,
);
if (validationError) {
return {
content: [{ type: "text" as const, text: validationError }],
details: {
status: "error",
phase: "validate",
},
};
}
}
const knownSecondaryMetrics = mergeSecondaryMetrics(state.secondaryMetrics, secondaryMetrics);
const currentResults = readCurrentSegmentResults(cwd, state.currentSegment);
const experiment: AutoresearchResultEntry = {
run: state.currentRunCount + 1,
commit: params.commit.slice(0, 7),
metric: params.metric,
metrics: secondaryMetrics,
status: params.status,
description: params.description,
timestamp: Date.now(),
segment: state.currentSegment,
};
let finalExperiment = experiment;
let gitSummary = "";
let gitAction: Record<string, unknown> = {
action: params.status === "keep" ? "commit" : "skip",
attempted: params.status === "keep",
};
if (params.status === "keep") {
const gitResult = commitKeptExperiment({
cwd: cwd,
description: params.description,
metricName: state.metricName,
metric: params.metric,
metrics: secondaryMetrics,
commit: experiment.commit,
status: "keep",
});
gitSummary = gitResult.summary;
gitAction = {
action: "commit",
attempted: true,
committed: gitResult.committed,
commit: gitResult.commit,
};
if (gitResult.committed) {
finalExperiment = {
...experiment,
commit: gitResult.commit,
};
}
} else {
gitSummary =
`Git: skipped commit (${params.status}) - ` +
"revert tracked changes yourself with git checkout -- . when you want to discard them.";
gitAction = {
action: "skip",
attempted: false,
};
}
try {
appendResultEntry(cwd, finalExperiment);
} catch (error) {
return {
content: [
{
type: "text" as const,
text: `Failed to append autoresearch.jsonl: ${
error instanceof Error ? error.message : String(error)
}`,
},
],
details: {
status: "error",
phase: "write",
},
};
}
const baselineMetric =
currentResults.length > 0 ? currentResults[0].metric : experiment.metric;
const baselineSecondaryMetrics = findBaselineSecondaryMetrics(
currentResults,
knownSecondaryMetrics,
);
const nextState: AutoresearchStateSnapshot = reconstructStateFromJsonl(cwd);
const queuedSteers = consumeAutoresearchSteers(cwd);
setAutoresearchRunInFlight(cwd, false);
return {
content: [
{
type: "text" as const,
text: buildResultText({
state,
experiment: finalExperiment,
baselineMetric,
baselineSecondaryMetrics,
totalRunCount: finalExperiment.run,
gitSummary,
knownSecondaryMetrics,
queuedSteers,
}),
},
],
details: {
status: "ok",
experiment: finalExperiment,
state: nextState,
git: gitAction,
},
};
},
};
}
type CurrentSegmentResult = {
readonly metric: number;
readonly metrics: Record<string, number>;
};
function validateSecondaryMetrics(
knownMetrics: readonly SecondaryMetricDef[],
providedMetrics: Record<string, number>,
force: boolean,
): string | null {
const knownNames = new Set(knownMetrics.map((metric) => metric.name));
const providedNames = new Set(Object.keys(providedMetrics));
const missing = [...knownNames].filter((name) => !providedNames.has(name));
if (missing.length > 0) {
return (
`Missing secondary metrics: ${missing.join(", ")}\n\n` +
`You must provide all previously tracked metrics. Expected: ${[...knownNames].join(", ")}\n` +
`Got: ${[...providedNames].join(", ") || "(none)"}\n\n` +
`Fix: include ${missing.map((name) => `"${name}": <value>`).join(", ")} in the metrics parameter.`
);
}
const added = [...providedNames].filter((name) => !knownNames.has(name));
if (added.length > 0 && !force) {
return (
`New secondary metric${added.length > 1 ? "s" : ""} not previously tracked: ${added.join(", ")}\n\n` +
`Existing metrics: ${[...knownNames].join(", ")}\n\n` +
"If this metric has proven very valuable to watch, call log_experiment again with force: true to add it. Otherwise, remove it from the metrics parameter."
);
}
return null;
}
function mergeSecondaryMetrics(
knownMetrics: readonly SecondaryMetricDef[],
providedMetrics: Record<string, number>,
): readonly SecondaryMetricDef[] {
const merged = [...knownMetrics];
for (const metricName of Object.keys(providedMetrics)) {
if (!merged.find((metric) => metric.name === metricName)) {
merged.push({
name: metricName,
unit: inferMetricUnit(metricName),
});
}
}
return merged;
}
function readCurrentSegmentResults(cwd: string, segment: number): CurrentSegmentResult[] {
const jsonlPath = getAutoresearchRootFilePath(cwd, "resultsLog");
if (!fs.existsSync(jsonlPath)) {
return [];
}
const results: CurrentSegmentResult[] = [];
let currentSegment = 0;
let hasSeenResult = false;
const lines = fs
.readFileSync(jsonlPath, "utf8")
.split("\n")
.map((line) => line.trim())
.filter(Boolean);
for (const line of lines) {
let entry: Record<string, unknown>;
try {
entry = JSON.parse(line) as Record<string, unknown>;
} catch {
continue;
}
if (entry.type === "config") {
if (hasSeenResult) {
currentSegment += 1;
}
continue;
}
hasSeenResult = true;
if (currentSegment !== segment || typeof entry.metric !== "number") {
continue;
}
results.push({
metric: entry.metric,
metrics:
entry.metrics && typeof entry.metrics === "object"
? (entry.metrics as Record<string, number>)
: {},
});
}
return results;
}
function findBaselineSecondaryMetrics(
currentResults: readonly CurrentSegmentResult[],
secondaryMetrics: readonly SecondaryMetricDef[],
): Record<string, number> {
const baseline =
currentResults.length > 0 ? { ...currentResults[0].metrics } : {};
for (const metric of secondaryMetrics) {
if (baseline[metric.name] !== undefined) {
continue;
}
for (const result of currentResults) {
const value = result.metrics[metric.name];
if (value !== undefined) {
baseline[metric.name] = value;
break;
}
}
}
return baseline;
}
function buildResultText(options: {
state: AutoresearchStateSnapshot;
experiment: {
run: number;
commit: string;
metric: number;
metrics: Record<string, number>;
status: "keep" | "discard" | "crash";
description: string;
};
baselineMetric: number;
baselineSecondaryMetrics: Record<string, number>;
totalRunCount: number;
gitSummary: string;
knownSecondaryMetrics: readonly SecondaryMetricDef[];
queuedSteers: readonly string[];
}): string {
let text = `Logged #${options.experiment.run}: ${options.experiment.status} - ${options.experiment.description}`;
text += `\nBaseline ${options.state.metricName}: ${formatMetric(options.baselineMetric, options.state.metricUnit)}`;
if (options.experiment.run > 1 && options.experiment.status === "keep" && options.experiment.metric > 0) {
const delta = options.experiment.metric - options.baselineMetric;
const pct = options.baselineMetric === 0 ? null : (delta / options.baselineMetric) * 100;
text += ` | this: ${formatMetric(options.experiment.metric, options.state.metricUnit)}`;
if (pct !== null) {
const sign = delta > 0 ? "+" : "";
text += ` (${sign}${pct.toFixed(1)}%)`;
}
}
if (Object.keys(options.experiment.metrics).length > 0) {
const parts = Object.entries(options.experiment.metrics).map(([name, value]) => {
const metricDef = options.knownSecondaryMetrics.find((metric) => metric.name === name);
let part = `${name}: ${formatMetric(value, metricDef?.unit ?? "")}`;
const baselineValue = options.baselineSecondaryMetrics[name];
if (
baselineValue !== undefined &&
options.experiment.run > 1 &&
baselineValue !== 0
) {
const delta = value - baselineValue;
const sign = delta > 0 ? "+" : "";
part += ` (${sign}${((delta / baselineValue) * 100).toFixed(1)}%)`;
}
return part;
});
text += `\nSecondary: ${parts.join(" ")}`;
}
text += `\n(${options.totalRunCount} experiments in current segment)`;
text += `\n${options.gitSummary}`;
if (options.queuedSteers.length > 0) {
const steerLabel = options.queuedSteers.length === 1 ? "steer" : "steers";
text += `\n\nQueued user ${steerLabel} captured during this experiment:`;
for (const steer of options.queuedSteers) {
text += `\n- ${steer}`;
}
text +=
"\nTreat any immediate followup turn that repeats the same steer as the normal OpenClaw queue/backlog delivery for these messages.";
}
return text;
}
function inferMetricUnit(name: string): string {
if (name.endsWith("_µs") || name.includes("µs")) {
return "µs";
}
if (name.endsWith("_ms") || name.includes("ms")) {
return "ms";
}
if (name.endsWith("_s") || name.includes("sec")) {
return "s";
}
return "";
}
function formatMetric(value: number, unit: string): string {
const rendered =
value === Math.round(value) ? `${Math.round(value)}` : value.toFixed(2);
return `${addCommas(rendered)}${unit}`;
}
function addCommas(value: string): string {
const [integerPart, fractionalPart] = value.split(".");
const normalizedInteger = integerPart.replace(/\B(?=(\d{3})+(?!\d))/g, ",");
return fractionalPart ? `${normalizedInteger}.${fractionalPart}` : normalizedInteger;
}
@@ -0,0 +1,64 @@
import type { OpenClawPluginApi } from "openclaw/plugin-sdk/core";
import { RunExperimentParams } from "./schemas.js";
import { executeExperimentCommand } from "../execute.js";
import { setAutoresearchRunInFlight } from "../runtime-state.js";
import { resolveToolCwd } from "./tool-cwd.js";
export function createRunExperimentTool(api: OpenClawPluginApi) {
return {
name: "run_experiment",
label: "Run Experiment",
description:
"Run a shell command as an experiment. Times wall-clock duration, captures output, detects pass/fail via exit code. Use for any autoresearch experiment.",
parameters: RunExperimentParams,
async execute(
_toolCallId: string,
params: {
cwd?: string;
command: string;
timeout_seconds?: number;
},
signal: AbortSignal,
onUpdate: ((update: unknown) => void | Promise<void>) | undefined,
) {
const cwd = resolveToolCwd(api, params.cwd);
setAutoresearchRunInFlight(cwd, true);
if (onUpdate) {
await onUpdate({
content: [{ type: "text" as const, text: `Running: ${params.command}` }],
details: { phase: "running" },
});
}
let details;
try {
details = await executeExperimentCommand({
command: params.command,
cwd,
timeoutSeconds: params.timeout_seconds,
signal,
});
} catch (error) {
setAutoresearchRunInFlight(cwd, false);
throw error;
}
let text = "";
if (details.timedOut) {
text += `TIMEOUT after ${details.durationSeconds.toFixed(1)}s\n`;
} else if (!details.passed) {
text += `FAILED (exit code ${details.exitCode ?? "null"}) in ${details.durationSeconds.toFixed(1)}s\n`;
} else {
text += `PASSED in ${details.durationSeconds.toFixed(1)}s\n`;
}
text += `\nLast 80 lines of output:\n${details.tailOutput || "(no output)"}`;
return {
content: [{ type: "text" as const, text }],
details,
};
},
};
}
@@ -0,0 +1,72 @@
import { Type } from "@sinclair/typebox";
const CwdParam = Type.Optional(
Type.String({
description:
"Optional working directory for repo-local autoresearch state. Use this when the tool call originates outside the target repo session cwd.",
}),
);
export const InitExperimentParams = Type.Object({
cwd: CwdParam,
name: Type.String({
description:
'Human-readable name for this experiment session (e.g. "Optimizing liquid for fastest execution and parsing")',
}),
metric_name: Type.String({
description:
'Display name for the primary metric (e.g. "total_µs", "bundle_kb", "val_bpb").',
}),
metric_unit: Type.Optional(
Type.String({
description:
'Unit for the primary metric. Use "µs", "ms", "s", "kb", "mb", or "" for unitless. Default: "".',
}),
),
direction: Type.Optional(
Type.String({
description: 'Whether "lower" or "higher" is better for the primary metric. Default: "lower".',
enum: ["lower", "higher"],
}),
),
});
export const RunExperimentParams = Type.Object({
cwd: CwdParam,
command: Type.String({
description: "Shell command to run (e.g. 'pnpm test:vitest', 'uv run train.py')",
}),
timeout_seconds: Type.Optional(
Type.Number({
description: "Kill after this many seconds (default: 600)",
}),
),
});
export const LogExperimentParams = Type.Object({
cwd: CwdParam,
commit: Type.String({ description: "Git commit hash (short, 7 chars)" }),
metric: Type.Number({
description:
"The primary optimization metric value (e.g. seconds, val_bpb). Use 0 for crashes.",
}),
status: Type.String({
description: "Result status for this experiment.",
enum: ["keep", "discard", "crash"],
}),
description: Type.String({
description: "Short description of what this experiment tried.",
}),
metrics: Type.Optional(
Type.Record(Type.String(), Type.Number(), {
description:
'Additional metrics to track as { name: value } pairs, e.g. { "compile_µs": 4200, "render_µs": 9800 }.',
}),
),
force: Type.Optional(
Type.Boolean({
description:
"Set to true to allow adding a new secondary metric that was not tracked before.",
}),
),
});
@@ -0,0 +1,19 @@
import path from "node:path";
import type { OpenClawPluginApi } from "openclaw/plugin-sdk/core";
export function resolveToolCwd(api: OpenClawPluginApi, requestedCwd?: unknown): string {
const normalizedRequestedCwd =
typeof requestedCwd === "string" ? requestedCwd.trim() : "";
const cwd = normalizedRequestedCwd
? path.isAbsolute(normalizedRequestedCwd)
? normalizedRequestedCwd
: api.resolvePath(normalizedRequestedCwd)
: api.resolvePath(".");
if (typeof cwd !== "string" || cwd.trim().length === 0) {
throw new Error("Could not resolve repo cwd for autoresearch tool execution.");
}
return cwd;
}
File diff suppressed because it is too large Load Diff
+14
View File
@@ -0,0 +1,14 @@
{
"id": "openclaw-autoresearch",
"name": "Autoresearch",
"description": "Faithful OpenClaw port of pi-autoresearch.",
"skills": [
"./skills"
],
"version": "1.0.0",
"configSchema": {
"type": "object",
"additionalProperties": false,
"properties": {}
}
}
+1293 -3883
View File
File diff suppressed because it is too large Load Diff
+24 -12
View File
@@ -1,17 +1,29 @@
{
"name": "pi-autoresearch",
"name": "openclaw-autoresearch",
"version": "1.0.0",
"description": "Autonomous experiment loop for pi — run, measure, keep or discard. Inspired by karpathy/autoresearch.",
"keywords": ["pi-package"],
"license": "MIT",
"pi": {
"extensions": ["./extensions"],
"skills": ["./skills"]
"description": "Faithful OpenClaw port of pi-autoresearch.",
"type": "module",
"scripts": {
"typecheck": "tsc -p tsconfig.json --noEmit",
"test": "vitest run",
"validate": "npm run typecheck && npm run test"
},
"peerDependencies": {
"@mariozechner/pi-ai": "*",
"@mariozechner/pi-coding-agent": "*",
"@mariozechner/pi-tui": "*",
"@sinclair/typebox": "*"
"keywords": [
"openclaw-plugin",
"autoresearch"
],
"license": "MIT",
"devDependencies": {
"@types/node": "24.5.2",
"typescript": "5.9.2",
"vitest": "3.2.4"
},
"dependencies": {
"@sinclair/typebox": "0.34.48"
},
"openclaw": {
"extensions": [
"./extensions/openclaw-autoresearch/index.ts"
]
}
}
Binary file not shown.

Before

Width:  |  Height:  |  Size: 874 KiB

+4 -2
View File
@@ -11,7 +11,7 @@ Autonomous experiment loop: try ideas, keep what works, discard what doesn't, ne
- **`init_experiment`** — configure session (name, metric, unit, direction). Call again to re-initialize with a new baseline when the optimization target changes.
- **`run_experiment`** — runs command, times it, captures output.
- **`log_experiment`** — records result. `keep` auto-commits. `discard`/`crash``git checkout -- .` to revert. Always include secondary `metrics` dict. Dashboard: ctrl+x.
- **`log_experiment`** — records result. `keep` auto-commits. `discard`/`crash``git checkout -- .` to revert. Always include secondary `metrics` dict.
## Setup
@@ -86,4 +86,6 @@ When there is no `autoresearch.ideas.md` file and the loop ends, the research is
## User Steers
User messages sent while an experiment is running are **automatically queued** and delivered to you after your next `log_experiment` call. Finish your current experiment first — don't stop or ask for confirmation. Incorporate the user's idea in the next experiment.
If the host exposes the OpenClaw message hooks, user steers that arrive while an experiment is running are captured and surfaced after your next `log_experiment` call. OpenClaw may also preserve the same steer in the normal followup backlog, so if the next turn repeats a steer you already saw in `log_experiment`, treat it as the same request rather than a brand new branch of work.
Finish the current experiment first, then incorporate the user's idea in the next experiment. Don't stop mid-experiment or ask for confirmation unless the user explicitly interrupts the loop.
+120
View File
@@ -0,0 +1,120 @@
import { describe, expect, it, vi } from "vitest";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { AUTORESEARCH_ROOT_FILES } from "../extensions/openclaw-autoresearch/src/files.js";
import {
buildAutoresearchCommandText,
registerAutoresearchCommand,
} from "../extensions/openclaw-autoresearch/src/commands/autoresearch.js";
import { getAutoresearchRuntimeState } from "../extensions/openclaw-autoresearch/src/runtime-state.js";
function createTempDir(): string {
return fs.mkdtempSync(path.join(os.tmpdir(), "autoresearch-command-test-"));
}
describe("buildAutoresearchCommandText", () => {
it("guides setup when no canonical files exist", () => {
const cwd = createTempDir();
expect(buildAutoresearchCommandText(cwd, "default")).toContain(
"Recommended OpenClaw entrypoint: `/autoresearch` or `/autoresearch setup <goal>`.",
);
expect(buildAutoresearchCommandText(cwd, "default")).toContain(
"Direct skill fallback: `/skill autoresearch-create`.",
);
});
it("routes active sessions back to the canonical files", () => {
const cwd = createTempDir();
fs.writeFileSync(
path.join(cwd, AUTORESEARCH_ROOT_FILES.sessionDoc),
"# Autoresearch\n\n## Objective\n\nReduce runtime.\n",
);
fs.writeFileSync(
path.join(cwd, AUTORESEARCH_ROOT_FILES.resultsLog),
[
JSON.stringify({
type: "config",
name: "Runtime optimization",
metricName: "total_ms",
metricUnit: "ms",
bestDirection: "lower",
}),
JSON.stringify({
run: 1,
commit: "abc1234",
metric: 101,
metrics: {},
status: "keep",
description: "baseline",
timestamp: 1700000000000,
segment: 0,
}),
].join("\n"),
);
const text = buildAutoresearchCommandText(cwd, "status");
expect(text).toContain("Autoresearch session detected at repo root:");
expect(text).toContain("Read `autoresearch.md` before resuming or changing the loop.");
expect(text).toContain("Mode: active");
expect(text).toContain("Runtime mode: auto");
expect(text).toContain("Baseline: 101ms");
});
it("registers a mode-aware /autoresearch command that primes resume instructions", () => {
const cwd = createTempDir();
fs.writeFileSync(
path.join(cwd, AUTORESEARCH_ROOT_FILES.sessionDoc),
"# Autoresearch\n\n## Objective\n\nReduce runtime.\n",
);
const api = {
resolvePath: vi.fn(() => cwd),
registerCommand: vi.fn(),
};
registerAutoresearchCommand(api as never);
const command = api.registerCommand.mock.calls[0]?.[0];
expect(command?.name).toBe("autoresearch");
const result = command.handler({ args: "resume focus parser cache", cwd });
expect(result.text).toContain("Autoresearch mode ON.");
expect(result.text).toContain("Captured resume instruction: focus parser cache");
expect(getAutoresearchRuntimeState(cwd)).toMatchObject({
mode: "on",
pendingCommand: {
kind: "resume",
args: "focus parser cache",
},
});
});
it("uses the explicit command flow for setup before falling back to the raw skill call", () => {
const cwd = createTempDir();
const api = {
resolvePath: vi.fn(() => cwd),
registerCommand: vi.fn(),
};
registerAutoresearchCommand(api as never);
const command = api.registerCommand.mock.calls[0]?.[0];
const result = command.handler({ args: "", cwd });
expect(result.text).toContain("Autoresearch mode ON.");
expect(result.text).toContain(
"Next step: send a normal message so the next agent turn can gather setup details.",
);
expect(result.text).toContain("Direct skill fallback: `/skill autoresearch-create`.");
expect(getAutoresearchRuntimeState(cwd)).toMatchObject({
mode: "on",
pendingCommand: {
kind: "setup",
args: null,
},
});
});
});
+3
View File
@@ -0,0 +1,3 @@
- Retry the parser change with a safer cache key
- Separate compile and runtime metrics
- Investigate benchmark startup overhead
+6
View File
@@ -0,0 +1,6 @@
{"type":"config","name":"Parser optimization","metricName":"total_ms","metricUnit":"ms","bestDirection":"lower"}
{"run":1,"commit":"abc1234","metric":125,"metrics":{"compile_ms":50},"status":"keep","description":"baseline","timestamp":1700000000000,"segment":0}
{"run":2,"commit":"def5678","metric":120,"metrics":{"compile_ms":48},"status":"discard","description":"attempt 1","timestamp":1700000001000,"segment":0}
{"type":"config","name":"Parser optimization","metricName":"total_ms","metricUnit":"ms","bestDirection":"lower"}
{"run":1,"commit":"9876543","metric":130,"metrics":{"compile_ms":55,"bundle_kb":10},"status":"keep","description":"new baseline","timestamp":1700000002000,"segment":1}
{"run":2,"commit":"7654321","metric":118,"metrics":{"compile_ms":44,"bundle_kb":9},"status":"keep","description":"keep winner","timestamp":1700000003000,"segment":1}
+5
View File
@@ -0,0 +1,5 @@
# Autoresearch
## Objective
Reduce benchmark runtime without changing behavior.
+106
View File
@@ -0,0 +1,106 @@
import { describe, expect, it, vi } from "vitest";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { AUTORESEARCH_ROOT_FILES } from "../extensions/openclaw-autoresearch/src/files.js";
import {
buildBeforePromptBuildContext,
registerAutoresearchHooks,
} from "../extensions/openclaw-autoresearch/src/hooks.js";
import {
getAutoresearchRuntimeState,
queueAutoresearchSteer,
setAutoresearchContinuationReminder,
setAutoresearchPendingCommand,
setAutoresearchRunInFlight,
setAutoresearchRuntimeMode,
} from "../extensions/openclaw-autoresearch/src/runtime-state.js";
function createTempDir(prefix: string): string {
return fs.mkdtempSync(path.join(os.tmpdir(), prefix));
}
function seedActiveSession(cwd: string): void {
fs.writeFileSync(
path.join(cwd, AUTORESEARCH_ROOT_FILES.sessionDoc),
"# Autoresearch\n\n## Objective\n\nReduce runtime.\n\n## What's Been Tried\n\n- baseline\n",
);
fs.writeFileSync(
path.join(cwd, AUTORESEARCH_ROOT_FILES.ideasBacklog),
"- retry the parser cache\n- measure compile_ms separately\n",
);
fs.writeFileSync(
path.join(cwd, AUTORESEARCH_ROOT_FILES.resultsLog),
[
JSON.stringify({
type: "config",
name: "Runtime optimization",
metricName: "total_ms",
metricUnit: "ms",
bestDirection: "lower",
}),
JSON.stringify({
run: 1,
commit: "abc1234",
metric: 101,
metrics: {},
status: "keep",
description: "baseline",
timestamp: 1700000000000,
segment: 0,
}),
].join("\n"),
);
}
describe("autoresearch hooks", () => {
it("builds a stronger before_prompt_build context for active sessions", () => {
const cwd = createTempDir("autoresearch-hooks-context-");
seedActiveSession(cwd);
setAutoresearchRuntimeMode(cwd, "on");
setAutoresearchPendingCommand(cwd, {
kind: "resume",
args: "focus parser cache",
});
queueAutoresearchSteer(cwd, "try branchless parsing");
setAutoresearchContinuationReminder(cwd, true);
const context = buildBeforePromptBuildContext(cwd);
expect(context).toContain("## Autoresearch Mode (ACTIVE)");
expect(context).toContain("Never stop unless the user explicitly interrupts the loop.");
expect(context).toContain("Additional resume instruction from /autoresearch: focus parser cache");
expect(context).toContain("The previous autoresearch run ended with pending ideas.");
expect(context).toContain("1 user steer arrived during the current experiment window");
const secondContext = buildBeforePromptBuildContext(cwd);
expect(secondContext).not.toContain("The previous autoresearch run ended with pending ideas.");
});
it("registers OpenClaw-compatible hooks and queues steer messages only while experiments are in flight", () => {
const cwd = createTempDir("autoresearch-hooks-runtime-");
const handlers = new Map<string, (event: unknown, ctx: { cwd?: string }) => unknown>();
const api = {
resolvePath: vi.fn(() => cwd),
on: vi.fn((hookName: string, handler: (event: unknown, ctx: { cwd?: string }) => unknown) => {
handlers.set(hookName, handler);
}),
};
registerAutoresearchHooks(api as never);
expect(handlers.has("message_received")).toBe(true);
setAutoresearchRunInFlight(cwd, false);
handlers.get("message_received")?.({ text: "ignore this" }, { cwd });
expect(getAutoresearchRuntimeState(cwd).queuedSteers).toEqual([]);
setAutoresearchRunInFlight(cwd, true);
handlers.get("message_received")?.({ text: "/help" }, { cwd });
handlers.get("message_received")?.({ text: "try branchless parsing" }, { cwd });
expect(getAutoresearchRuntimeState(cwd).queuedSteers).toEqual([
"try branchless parsing",
]);
});
});
+528
View File
@@ -0,0 +1,528 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { createInitExperimentTool } from "../extensions/openclaw-autoresearch/src/tools/init-experiment.js";
import { createRunExperimentTool } from "../extensions/openclaw-autoresearch/src/tools/run-experiment.js";
import { createLogExperimentTool } from "../extensions/openclaw-autoresearch/src/tools/log-experiment.js";
import { getAutoresearchRootFilePath } from "../extensions/openclaw-autoresearch/src/files.js";
import * as gitModule from "../extensions/openclaw-autoresearch/src/git.js";
import {
getAutoresearchRuntimeState,
queueAutoresearchSteer,
} from "../extensions/openclaw-autoresearch/src/runtime-state.js";
function createTempDir(prefix: string): string {
return fs.mkdtempSync(path.join(os.tmpdir(), prefix));
}
function createAbortSignal(): AbortSignal {
return new AbortController().signal;
}
function createApi(cwd: string) {
return {
resolvePath: vi.fn(() => cwd),
};
}
function readJsonl(cwd: string): Array<Record<string, unknown>> {
const jsonlPath = getAutoresearchRootFilePath(cwd, "resultsLog");
return fs
.readFileSync(jsonlPath, "utf8")
.split("\n")
.map((line) => line.trim())
.filter(Boolean)
.map((line) => JSON.parse(line) as Record<string, unknown>);
}
async function initExperiment(
cwd: string,
params: {
name: string;
metric_name: string;
metric_unit?: string;
direction?: "lower" | "higher";
},
) {
return await createInitExperimentTool(createApi(cwd) as never).execute(
"tool-call",
params,
createAbortSignal(),
undefined,
);
}
async function runExperiment(
cwd: string,
params: {
command: string;
timeout_seconds?: number;
},
onUpdate?: (update: unknown) => void | Promise<void>,
) {
return await createRunExperimentTool(createApi(cwd) as never).execute(
"tool-call",
params,
createAbortSignal(),
onUpdate,
);
}
async function logExperiment(
cwd: string,
params: {
commit: string;
metric: number;
status: "keep" | "discard" | "crash";
description: string;
metrics?: Record<string, number>;
force?: boolean;
},
) {
return await createLogExperimentTool(createApi(cwd) as never).execute(
"tool-call",
params,
createAbortSignal(),
undefined,
);
}
async function seedExperiment(cwd: string) {
await initExperiment(cwd, {
name: "Parser optimization",
metric_name: "total_ms",
metric_unit: "ms",
direction: "lower",
});
}
describe("experiment lifecycle tools", () => {
afterEach(() => {
vi.restoreAllMocks();
});
it("writes the first config header to the root autoresearch.jsonl file", async () => {
const cwd = createTempDir("autoresearch-init-");
const result = await initExperiment(cwd, {
name: "Parser optimization",
metric_name: "total_ms",
metric_unit: "ms",
direction: "lower",
});
expect(result.details).toMatchObject({
status: "ok",
state: {
name: "Parser optimization",
metricName: "total_ms",
metricUnit: "ms",
bestDirection: "lower",
currentSegment: 0,
},
});
expect(readJsonl(cwd)).toEqual([
{
type: "config",
name: "Parser optimization",
metricName: "total_ms",
metricUnit: "ms",
bestDirection: "lower",
},
]);
});
it("appends a new config header on re-init instead of overwriting prior history", async () => {
const cwd = createTempDir("autoresearch-reinit-");
await initExperiment(cwd, {
name: "Parser optimization",
metric_name: "total_ms",
metric_unit: "ms",
direction: "lower",
});
fs.appendFileSync(
getAutoresearchRootFilePath(cwd, "resultsLog"),
`${JSON.stringify({
run: 1,
commit: "abc1234",
metric: 120,
metrics: {},
status: "keep",
description: "baseline",
timestamp: 1700000000000,
segment: 0,
})}\n`,
);
const result = await initExperiment(cwd, {
name: "Parser optimization v2",
metric_name: "total_ms",
metric_unit: "ms",
direction: "lower",
});
expect(result.content[0]?.type).toBe("text");
expect((result.content[0] as { text: string }).text).toContain("re-initialized");
expect(readJsonl(cwd)).toEqual([
{
type: "config",
name: "Parser optimization",
metricName: "total_ms",
metricUnit: "ms",
bestDirection: "lower",
},
{
run: 1,
commit: "abc1234",
metric: 120,
metrics: {},
status: "keep",
description: "baseline",
timestamp: 1700000000000,
segment: 0,
},
{
type: "config",
name: "Parser optimization v2",
metricName: "total_ms",
metricUnit: "ms",
bestDirection: "lower",
},
]);
expect(result.details).toMatchObject({
status: "ok",
state: {
currentSegment: 1,
currentRunCount: 0,
},
});
});
it("reports a successful run and emits a running update", async () => {
const cwd = createTempDir("autoresearch-run-success-");
const updates: unknown[] = [];
const result = await runExperiment(
cwd,
{
command: "printf 'hello\\nworld\\n'",
},
(update) => {
updates.push(update);
},
);
expect(updates).toEqual([
{
content: [{ type: "text", text: "Running: printf 'hello\\nworld\\n'" }],
details: { phase: "running" },
},
]);
expect(result.details).toMatchObject({
command: "printf 'hello\\nworld\\n'",
exitCode: 0,
passed: true,
crashed: false,
timedOut: false,
stdout: "hello\nworld\n",
stderr: "",
tailOutput: "hello\nworld",
});
expect((result.content[0] as { text: string }).text).toContain("PASSED");
expect((result.content[0] as { text: string }).text).toContain(
"Last 80 lines of output:\nhello\nworld",
);
});
it("reports failed runs with exit code and stderr in the captured tail", async () => {
const cwd = createTempDir("autoresearch-run-failure-");
const result = await runExperiment(cwd, {
command: "echo 'bad stderr' >&2; exit 7",
});
expect(result.details).toMatchObject({
exitCode: 7,
passed: false,
crashed: true,
timedOut: false,
stdout: "",
stderr: "bad stderr\n",
tailOutput: "bad stderr",
});
expect((result.content[0] as { text: string }).text).toContain(
"FAILED (exit code 7)",
);
});
it("marks timed out runs and keeps the output-tail shape to the last 80 lines", async () => {
const cwd = createTempDir("autoresearch-run-timeout-");
const timeoutResult = await runExperiment(cwd, {
command: "echo start; sleep 0.2; echo done",
timeout_seconds: 0.05,
});
expect(timeoutResult.details).toMatchObject({
exitCode: null,
passed: false,
crashed: true,
timedOut: true,
});
expect((timeoutResult.content[0] as { text: string }).text).toContain("TIMEOUT");
expect(timeoutResult.details.stdout).toContain("start\n");
expect(timeoutResult.details.stdout).not.toContain("done\n");
const tailResult = await runExperiment(cwd, {
command:
"i=1; while [ $i -le 100 ]; do echo line-$i; i=$((i+1)); done",
});
const tailLines = tailResult.details.tailOutput.split("\n");
expect(tailLines).toHaveLength(80);
expect(tailLines[0]).toBe("line-21");
expect(tailLines[79]).toBe("line-100");
expect(tailResult.details.tailOutput).not.toContain("line-20");
});
it("appends a result row to autoresearch.jsonl and preserves secondary metrics in state", async () => {
const cwd = createTempDir("autoresearch-log-append-");
await seedExperiment(cwd);
const result = await logExperiment(cwd, {
commit: "abc1234",
metric: 120,
status: "discard",
description: "baseline",
metrics: {
compile_ms: 15,
},
});
const rows = readJsonl(cwd);
expect(rows).toHaveLength(2);
expect(rows[1]).toMatchObject({
run: 1,
commit: "abc1234",
metric: 120,
metrics: {
compile_ms: 15,
},
status: "discard",
description: "baseline",
segment: 0,
});
expect(result.details).toMatchObject({
status: "ok",
experiment: {
run: 1,
commit: "abc1234",
},
state: {
currentRunCount: 1,
totalRunCount: 1,
secondaryMetrics: [{ name: "compile_ms", unit: "ms" }],
},
git: {
action: "skip",
attempted: false,
},
});
expect((result.content[0] as { text: string }).text).toContain(
"Git: skipped commit (discard) - revert tracked changes yourself with git checkout -- .",
);
});
it("validates missing and newly added secondary metrics unless forced", async () => {
const cwd = createTempDir("autoresearch-log-metrics-");
await seedExperiment(cwd);
const baselineResult = await logExperiment(cwd, {
commit: "abc1234",
metric: 120,
status: "discard",
description: "baseline",
metrics: {
compile_ms: 15,
},
});
expect(baselineResult.details).toMatchObject({ status: "ok" });
const missingMetric = await logExperiment(cwd, {
commit: "def5678",
metric: 110,
status: "discard",
description: "missing secondary metric",
metrics: {},
});
expect(missingMetric.details).toMatchObject({
status: "error",
phase: "validate",
});
expect((missingMetric.content[0] as { text: string }).text).toContain(
"Missing secondary metrics: compile_ms",
);
const newMetricWithoutForce = await logExperiment(cwd, {
commit: "def5678",
metric: 110,
status: "discard",
description: "add bundle size",
metrics: {
compile_ms: 12,
bundle_kb: 7,
},
});
expect(newMetricWithoutForce.details).toMatchObject({
status: "error",
phase: "validate",
});
expect((newMetricWithoutForce.content[0] as { text: string }).text).toContain(
"New secondary metric not previously tracked: bundle_kb",
);
const forcedNewMetric = await logExperiment(cwd, {
commit: "def5678",
metric: 110,
status: "discard",
description: "add bundle size",
metrics: {
compile_ms: 12,
bundle_kb: 7,
},
force: true,
});
expect(forcedNewMetric.details).toMatchObject({
status: "ok",
state: {
secondaryMetrics: [
{ name: "compile_ms", unit: "ms" },
{ name: "bundle_kb", unit: "kb" },
],
},
});
});
it("routes keep to commit and leaves discard/crash as manual reverts while preserving logged statuses", async () => {
const cwd = createTempDir("autoresearch-log-status-");
await seedExperiment(cwd);
const commitSpy = vi.spyOn(gitModule, "commitKeptExperiment").mockReturnValue({
attempted: true,
committed: true,
commit: "def5678",
summary: "Git: committed - [main def5678] keep improved parser",
command: {
code: 0,
stdout: "[main def5678] keep improved parser\n",
stderr: "",
combinedOutput: "[main def5678] keep improved parser",
},
});
const keepResult = await logExperiment(cwd, {
commit: "abc1234",
metric: 95,
status: "keep",
description: "keep improved parser",
});
expect(keepResult.details).toMatchObject({
status: "ok",
git: {
action: "commit",
attempted: true,
committed: true,
commit: "def5678",
},
experiment: {
commit: "def5678",
status: "keep",
},
});
expect(commitSpy).toHaveBeenCalledTimes(1);
const discardResult = await logExperiment(cwd, {
commit: "def5678",
metric: 130,
status: "discard",
description: "discard regression",
});
expect(discardResult.details).toMatchObject({
status: "ok",
git: {
action: "skip",
attempted: false,
},
experiment: {
status: "discard",
},
});
expect((discardResult.content[0] as { text: string }).text).toContain(
"revert tracked changes yourself with git checkout -- .",
);
const crashResult = await logExperiment(cwd, {
commit: "def5678",
metric: 0,
status: "crash",
description: "crashed benchmark",
});
expect(crashResult.details).toMatchObject({
status: "ok",
git: {
action: "skip",
attempted: false,
},
experiment: {
status: "crash",
metric: 0,
},
});
expect((crashResult.content[0] as { text: string }).text).toContain(
"revert tracked changes yourself with git checkout -- .",
);
const statuses = readJsonl(cwd)
.filter((entry) => entry.type !== "config")
.map((entry) => entry.status);
expect(statuses).toEqual(["keep", "discard", "crash"]);
});
it("keeps the experiment window open across run_experiment and surfaces queued steers in log_experiment", async () => {
const cwd = createTempDir("autoresearch-queued-steers-");
await seedExperiment(cwd);
await runExperiment(cwd, {
command: "printf 'baseline\\n'",
});
expect(getAutoresearchRuntimeState(cwd)).toMatchObject({
runInFlight: true,
});
queueAutoresearchSteer(cwd, "try a parser cache");
queueAutoresearchSteer(cwd, "watch compile_ms too");
const result = await logExperiment(cwd, {
commit: "abc1234",
metric: 120,
status: "discard",
description: "baseline",
});
expect((result.content[0] as { text: string }).text).toContain(
"Queued user steers captured during this experiment:",
);
expect((result.content[0] as { text: string }).text).toContain("- try a parser cache");
expect((result.content[0] as { text: string }).text).toContain("- watch compile_ms too");
expect(getAutoresearchRuntimeState(cwd)).toMatchObject({
runInFlight: false,
queuedSteers: [],
});
});
});
+31
View File
@@ -0,0 +1,31 @@
import { describe, expect, it, vi } from "vitest";
import plugin from "../extensions/openclaw-autoresearch/index.js";
describe("plugin registration", () => {
it("registers the command, OpenClaw hooks, and all tool surfaces", () => {
const api = {
resolvePath: vi.fn(() => "/tmp/repo"),
registerTool: vi.fn(),
registerCommand: vi.fn(),
on: vi.fn(),
};
plugin.register(api);
expect(api.registerCommand).toHaveBeenCalledTimes(1);
expect(api.on).toHaveBeenCalledTimes(4);
expect(api.on.mock.calls.map(([hookName]) => hookName)).toEqual([
"before_prompt_build",
"message_received",
"agent_end",
"session_end",
]);
expect(api.registerTool).toHaveBeenCalledTimes(4);
expect(api.registerTool.mock.calls.map(([tool]) => tool.name)).toEqual([
"init_experiment",
"run_experiment",
"log_experiment",
"autoresearch_status",
]);
});
});
+25
View File
@@ -0,0 +1,25 @@
export type CommandRegistration = {
name: string;
description: string;
acceptsArgs?: boolean;
handler: (ctx: { args?: string; cwd?: string }) => { text: string };
};
export type ToolRegistration = {
name: string;
};
export type OpenClawPluginApi = {
resolvePath(path: string): string;
registerTool(tool: ToolRegistration): void;
registerCommand(command: CommandRegistration): void;
on?(hookName: string, handler: (event: unknown, ctx: { cwd?: string }) => unknown): void;
registerHook?(
hookName: string,
handler: (event: { systemPrompt?: string }, ctx: { cwd?: string }) => { systemPrompt?: string } | void,
): void;
};
export function emptyPluginConfigSchema(): Record<string, never> {
return {};
}
+41
View File
@@ -0,0 +1,41 @@
import { describe, expect, it } from "vitest";
import path from "node:path";
import { reconstructStateFromJsonl } from "../extensions/openclaw-autoresearch/src/state.js";
const activeSessionFixture = path.resolve("test/fixtures/active-session");
describe("reconstructStateFromJsonl", () => {
it("rebuilds the active session snapshot from canonical fixtures", () => {
const state = reconstructStateFromJsonl(activeSessionFixture);
expect(state.name).toBe("Parser optimization");
expect(state.mode).toBe("active");
expect(state.hasSessionDoc).toBe(true);
expect(state.currentSegment).toBe(1);
expect(state.currentRunCount).toBe(2);
expect(state.totalRunCount).toBe(4);
expect(state.currentBaselineMetric).toBe(130);
expect(state.currentBestMetric).toBe(118);
expect(state.lastRun).toMatchObject({
run: 2,
commit: "7654321",
metric: 118,
status: "keep",
description: "keep winner",
segment: 1,
});
expect(state.secondaryMetrics).toEqual([
{ name: "compile_ms", unit: "ms" },
{ name: "bundle_kb", unit: "kb" },
]);
expect(state.ideas).toEqual({
hasBacklog: true,
pendingCount: 3,
preview: [
"Retry the parser change with a safer cache key",
"Separate compile and runtime metrics",
"Investigate benchmark startup overhead",
],
});
});
});
+28
View File
@@ -0,0 +1,28 @@
import { describe, expect, it } from "vitest";
import { formatAutoresearchStatusText } from "../extensions/openclaw-autoresearch/src/tools/autoresearch-status.js";
import { reconstructStateFromJsonl } from "../extensions/openclaw-autoresearch/src/state.js";
import { getAutoresearchRuntimeState } from "../extensions/openclaw-autoresearch/src/runtime-state.js";
import path from "node:path";
const activeSessionFixture = path.resolve("test/fixtures/active-session");
describe("formatAutoresearchStatusText", () => {
it("renders a concise status summary from reconstructed state", () => {
const text = formatAutoresearchStatusText(
reconstructStateFromJsonl(activeSessionFixture),
getAutoresearchRuntimeState(activeSessionFixture),
);
expect(text).toContain("Session: Parser optimization");
expect(text).toContain("Runtime mode: auto");
expect(text).toContain("Experiment window: idle");
expect(text).toContain("Queued steers: 0");
expect(text).toContain("Metric: total_ms (ms, lower is better)");
expect(text).toContain("Runs: 2 current / 4 total");
expect(text).toContain("Best kept: 118ms");
expect(text).toContain("Last run: #2 keep 118ms 7654321 keep winner");
expect(text).toContain(
"Ideas preview: Retry the parser change with a safer cache key | Separate compile and runtime metrics | Investigate benchmark startup overhead",
);
});
});
+184
View File
@@ -0,0 +1,184 @@
import { describe, expect, it, vi } from "vitest";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { createInitExperimentTool } from "../extensions/openclaw-autoresearch/src/tools/init-experiment.js";
import { createAutoresearchStatusTool } from "../extensions/openclaw-autoresearch/src/tools/autoresearch-status.js";
import { createRunExperimentTool } from "../extensions/openclaw-autoresearch/src/tools/run-experiment.js";
import { createLogExperimentTool } from "../extensions/openclaw-autoresearch/src/tools/log-experiment.js";
import { AUTORESEARCH_ROOT_FILES } from "../extensions/openclaw-autoresearch/src/files.js";
function createTempDir(): string {
return fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), "autoresearch-tool-test-")));
}
function createApi(cwd: string) {
return {
resolvePath: vi.fn(() => cwd),
};
}
describe("autoresearch tools", () => {
it("init_experiment resolves cwd from api.resolvePath instead of expecting a fifth execute arg", async () => {
const cwd = createTempDir();
const api = createApi(cwd);
const tool = createInitExperimentTool(api as never);
const result = await tool.execute(
"call-1",
{
name: "Repo robustness",
metric_name: "escaped_mutations",
metric_unit: "",
direction: "lower",
},
new AbortController().signal,
undefined,
);
expect(api.resolvePath).toHaveBeenCalledWith(".");
expect(result.details).toMatchObject({ status: "ok" });
expect(fs.existsSync(path.join(cwd, AUTORESEARCH_ROOT_FILES.resultsLog))).toBe(true);
});
it("autoresearch_status resolves cwd from api.resolvePath", async () => {
const cwd = createTempDir();
const api = createApi(cwd);
await createInitExperimentTool(api as never).execute(
"call-1",
{
name: "Repo robustness",
metric_name: "escaped_mutations",
metric_unit: "",
direction: "lower",
},
new AbortController().signal,
undefined,
);
const result = await createAutoresearchStatusTool(api as never).execute(
"call-2",
{},
new AbortController().signal,
undefined,
);
expect(api.resolvePath).toHaveBeenCalledWith(".");
expect(result.content[0]?.text).toContain("Session: Repo robustness");
expect(result.content[0]?.text).toContain("Metric: escaped_mutations");
});
it("run_experiment resolves cwd from api.resolvePath", async () => {
const cwd = createTempDir();
const api = createApi(cwd);
const updates: unknown[] = [];
const result = await createRunExperimentTool(api as never).execute(
"call-3",
{ command: "node -e \"console.log('ok')\"", timeout_seconds: 10 },
new AbortController().signal,
async (update) => {
updates.push(update);
},
);
expect(api.resolvePath).toHaveBeenCalledWith(".");
expect(updates.length).toBe(1);
expect(result.details).toMatchObject({ passed: true, timedOut: false });
});
it("log_experiment resolves cwd from api.resolvePath", async () => {
const cwd = createTempDir();
const api = createApi(cwd);
await createInitExperimentTool(api as never).execute(
"call-1",
{
name: "Repo robustness",
metric_name: "escaped_mutations",
metric_unit: "",
direction: "lower",
},
new AbortController().signal,
undefined,
);
const result = await createLogExperimentTool(api as never).execute(
"call-4",
{
commit: "abc1234",
metric: 5,
status: "discard",
description: "baseline",
},
new AbortController().signal,
undefined,
);
expect(api.resolvePath).toHaveBeenCalledWith(".");
expect(result.details).toMatchObject({ status: "ok" });
expect(result.content[0]?.text).toContain("Logged #1: discard - baseline");
});
it("supports explicit cwd overrides for nested repo tool execution", async () => {
const workspaceCwd = createTempDir();
const repoCwd = createTempDir();
const api = createApi(workspaceCwd);
const initResult = await createInitExperimentTool(api as never).execute(
"call-1",
{
cwd: repoCwd,
name: "Nested repo robustness",
metric_name: "escaped_mutations",
metric_unit: "",
direction: "lower",
},
new AbortController().signal,
undefined,
);
expect(initResult.details).toMatchObject({ status: "ok" });
expect(api.resolvePath).not.toHaveBeenCalled();
expect(fs.existsSync(path.join(repoCwd, AUTORESEARCH_ROOT_FILES.resultsLog))).toBe(true);
expect(fs.existsSync(path.join(workspaceCwd, AUTORESEARCH_ROOT_FILES.resultsLog))).toBe(false);
const runResult = await createRunExperimentTool(api as never).execute(
"call-2",
{
cwd: repoCwd,
command: "node -e \"console.log(process.cwd())\"",
timeout_seconds: 10,
},
new AbortController().signal,
undefined,
);
expect(runResult.details).toMatchObject({ passed: true, timedOut: false });
expect(runResult.details.stdout.trim()).toBe(repoCwd);
const statusResult = await createAutoresearchStatusTool(api as never).execute(
"call-3",
{ cwd: repoCwd },
new AbortController().signal,
undefined,
);
expect(statusResult.content[0]?.text).toContain("Session: Nested repo robustness");
const logResult = await createLogExperimentTool(api as never).execute(
"call-4",
{
cwd: repoCwd,
commit: "abc1234",
metric: 5,
status: "discard",
description: "baseline",
},
new AbortController().signal,
undefined,
);
expect(logResult.details).toMatchObject({ status: "ok" });
expect(logResult.content[0]?.text).toContain("Logged #1: discard - baseline");
});
});
+19
View File
@@ -0,0 +1,19 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "NodeNext",
"moduleResolution": "NodeNext",
"strict": true,
"noEmit": true,
"skipLibCheck": true,
"types": ["node", "vitest/globals"],
"baseUrl": ".",
"paths": {
"openclaw/plugin-sdk/core": ["./test/shims/openclaw-plugin-sdk-core.ts"]
}
},
"include": [
"extensions/openclaw-autoresearch/**/*.ts",
"test/**/*.ts"
]
}
+20
View File
@@ -0,0 +1,20 @@
import { defineConfig } from "vitest/config";
import { fileURLToPath } from "node:url";
const rootDir = fileURLToPath(new URL(".", import.meta.url));
const shimPath = fileURLToPath(
new URL("./test/shims/openclaw-plugin-sdk-core.ts", import.meta.url),
);
export default defineConfig({
resolve: {
alias: {
"openclaw/plugin-sdk/core": shimPath,
},
},
test: {
environment: "node",
include: ["test/**/*.test.ts"],
root: rootDir,
},
});