mirror of
https://github.com/openclaw/lobster.git
synced 2026-08-14 00:48:09 +00:00
fix: workflow steps keep waiting for the model after timeout_ms expires (#132)
* fix(llm): honor step timeouts and cancellation during model calls llm.invoke never read ctx.signal, and both adapter helpers called fetch without one, so nothing could end a request once it was in flight. A workflow step with timeout_ms kept waiting for the adapter and, when the model eventually answered, reported the step as completed well past its budget. Shell steps already honor the same signal, so the two step kinds disagreed about what timeout_ms means. Thread ctx.signal into both adapter fetch calls, check it before each attempt so an already-cancelled run does not pay for another call, and rethrow abort errors unwrapped: the workflow runner recognizes timeouts and external cancellation by the error's identity, so wrapping them in "request failed" would misreport the outcome even once the request stops. * fix(llm): stop waiting for adapters that ignore the cancellation signal Threading the signal into fetch only covers the HTTP adapters. An injected ctx.llmAdapters adapter is awaited directly, so one that never observes ctx.signal still holds a timed-out step open for as long as it likes, which leaves SDK and tool-runtime users on the weaker timeout contract the rest of this change removes. Await the adapter call against the signal instead of the adapter alone. The adapter's own work cannot be killed from here, but the step stops waiting on it and the abort reason reaches the workflow unchanged. Cooperative adapters are unaffected: they still receive ctx and can end their own work. * fix(llm): do not finish a cancelled run from cached results Run-state and file-cache hits return before any adapter call, so a run that was already cancelled still completed successfully from cached data. That contradicts the rest of this change: the same invocation reports cancellation when it has to reach a model and success when it does not, purely on cache contents. Check the signal once at entry, before either lookup. The retry-loop check stays, since cancellation can also arrive between validation attempts. * docs(readme): state the cancellation contract for injected LLM adapters Racing an adapter promise against the step signal restores workflow liveness, but it cannot cancel work the adapter has already started: an injected adapter that ignores `ctx.signal` keeps its model request running after Lobster stops awaiting it, and a configured retry can then overlap it. That is a contract for the host to meet, so the host-facing docs have to say it rather than leaving it to be discovered from a duplicated charge. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014Bu6MvbjKGtfRiNesARMyZ * fix(llm): re-check cancellation after each awaited reuse boundary The entry check only caught a run that was already cancelled when the command started. Draining pipeline input waits on the upstream step, and the run-state and cache lookups are file I/O, so a step timeout could fire during any of them and the reuse branch would still return its stored answer as a successful result. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011gD5sJTbh2jn1uvgNCmkLq * fix(llm): do not report a step that finished after its deadline The cancellation checks stopped at the adapter call, but both return paths then await run-state and cache writes. A deadline crossed during those writes was never observed, so the step returned success and the workflow cleared its timer -- a model step could still complete late. The check goes after both writes rather than between them: the answer has already been paid for, and leaving it stored lets the retry replay it instead of calling the model a second time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011gD5sJTbh2jn1uvgNCmkLq * test(llm): let the persistence tests run where the state fix has not landed The two cancellation-during-write tests let the command create the cache namespace directory itself, which is where #125 throws on Windows: the recursive mkdir reports an extended-length path and the chain sync then opens `<dir>\C:`. Both tests failed there before reaching an assertion, so this branch added two Windows failures over main for a reason that is not its own. Creating the directory up front is what the seeded-cache tests in this file already do. ensureDirectory only syncs a chain it created, so an existing directory never reaches the broken path, and the tests still fail against the code they cover -- now on the missing rejection rather than on ENOENT. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwU1bvb9Pt3cR662P52CTp * fix(llm): let a host's own abort reason still read as cancellation The workflow runner classifies cancellation by name alone: only `AbortError` or `ABORT_ERR` bypasses the step's retry and `on_error` policy. Every abort path here rejected with `signal.reason` untouched, so a host that calls `controller.abort(new Error("stop"))` handed the runner an ordinary-looking error. A cancelled `llm.invoke` step under `on_error: continue` was recorded as a step failure and the run carried on -- returning `status: "ok"` for a run the host had cancelled. The gap is narrow, which is why it survived earlier review. An HTTP adapter is saved by fetch rejecting with its own `AbortError`, and the runner's own step timer is not affected. It opens only on the injected `ctx.llmAdapters` path, where this command's rejection is the only thing the runner ever sees. Normalize at the boundary instead of guessing upstream: a reason that already reads as an abort passes through untouched, anything else keeps its message and moves to `cause` under an abort-identified error. The host's reason is never discarded, and callers that already match on `AbortError` keep matching. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WHhk7xRwXzYcSXvw2kp5F5 * fix(llm): watch an adapter that rejects after aborting the run `abortable` guards against an adapter that ignores `ctx.signal`, but its already-aborted branch rejected the wrapper and returned before anything was attached to the adapter's own promise. An adapter that cancels the run from inside its own `invoke` -- an SDK client tearing itself down on a fatal error -- and then rejects leaves that rejection unobserved. The step still fails correctly, with the `AbortError` the runner needs. The damage lands afterwards: under Node's default unhandled-rejection handling the process exits on an error nothing is waiting for, seconds after the run was cancelled cleanly. Attach settlement handling first in both branches, so the promise is watched before an abort can win the race. Removing a listener that was never added is harmless, which keeps the two paths one shape. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011hFZxjivm9JTTvmRjQ4A8S --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
911f35c9b8
commit
096f5fedcb
@@ -271,6 +271,8 @@ Built-in providers today:
|
|||||||
- `pi` via `LOBSTER_PI_LLM_ADAPTER_URL` (typically supplied by the Pi extension)
|
- `pi` via `LOBSTER_PI_LLM_ADAPTER_URL` (typically supplied by the Pi extension)
|
||||||
- `http` via `LOBSTER_LLM_ADAPTER_URL`
|
- `http` via `LOBSTER_LLM_ADAPTER_URL`
|
||||||
|
|
||||||
|
A host embedding Lobster can supply its own adapters through `ctx.llmAdapters`. Step `timeout_ms` and workflow cancellation reach an adapter as `ctx.signal`: Lobster stops waiting as soon as that signal aborts, so the step fails or retries on time either way, but it cannot cancel work an adapter has already started. An injected adapter should observe `ctx.signal` and abort its own request — otherwise a timed-out step can leave a model call running, and billed, in the background.
|
||||||
|
|
||||||
Workflow `_meta.cost` and `cost_limit` use a static pricing table plus optional overrides from `LOBSTER_LLM_PRICING_JSON`, for example `{"my-model":{"input":1.0,"output":2.0}}` in USD per million tokens. Unknown or missing model IDs still record token counts with zero estimated cost, but Lobster warns on stderr so stale or missing pricing does not fail silently.
|
Workflow `_meta.cost` and `cost_limit` use a static pricing table plus optional overrides from `LOBSTER_LLM_PRICING_JSON`, for example `{"my-model":{"input":1.0,"output":2.0}}` in USD per million tokens. Unknown or missing model IDs still record token counts with zero estimated cost, but Lobster warns on stderr so stale or missing pricing does not fail silently.
|
||||||
|
|
||||||
A cached or replayed answer is not billed again: a model call is counted once, in the run that made it, however many later steps re-emit its answer. This holds while the answer stays inside Lobster: through pipelines, renderers, projections, run state, `workflow:` steps and a resume. It does not survive a stage that hands the items to an external process and reads them back — `exec --stdin json --json ...` — because what comes back is whatever that process printed, and Lobster cannot tell a faithful copy of a replay from a fresh claim to have made the call. Such a step is billed as a call, which is what earlier versions did everywhere. A `workflow:` step counts what its sub-workflow spent, and a replay the sub-workflow returned is not billed a second time by the run that composed it. Spend also survives a pause — a workflow that stops at an approval or `input` gate keeps what it has recorded, so `_meta.cost` covers the whole run after a resume and `cost_limit` applies to the whole run rather than to the steps after the last gate.
|
A cached or replayed answer is not billed again: a model call is counted once, in the run that made it, however many later steps re-emit its answer. This holds while the answer stays inside Lobster: through pipelines, renderers, projections, run state, `workflow:` steps and a resume. It does not survive a stage that hands the items to an external process and reads them back — `exec --stdin json --json ...` — because what comes back is whatever that process printed, and Lobster cannot tell a faithful copy of a replay from a fresh claim to have made the call. Such a step is billed as a call, which is what earlier versions did everywhere. A `workflow:` step counts what its sub-workflow spent, and a replay the sub-workflow returned is not billed a second time by the run that composed it. Spend also survives a pause — a workflow that stops at an approval or `input` gate keeps what it has recorded, so `_meta.cost` covers the whole run after a resume and `cost_limit` applies to the whole run rather than to the steps after the last gate.
|
||||||
|
|||||||
@@ -739,6 +739,10 @@ async function runLlmInvoke({
|
|||||||
config: CommandConfig;
|
config: CommandConfig;
|
||||||
}) {
|
}) {
|
||||||
const env = ctx.env ?? process.env;
|
const env = ctx.env ?? process.env;
|
||||||
|
const signal: AbortSignal | undefined = ctx?.signal;
|
||||||
|
// Run-state and cache hits return before any adapter call, so a cancelled run
|
||||||
|
// would otherwise still finish as a success.
|
||||||
|
throwIfCancelled(signal);
|
||||||
const provider = resolveProvider(args, env, config.defaultProvider, ctx);
|
const provider = resolveProvider(args, env, config.defaultProvider, ctx);
|
||||||
const adapter = resolveAdapter({ provider, env, args, config, ctx });
|
const adapter = resolveAdapter({ provider, env, args, config, ctx });
|
||||||
const prompt = extractPrompt(args);
|
const prompt = extractPrompt(args);
|
||||||
@@ -785,6 +789,9 @@ async function runLlmInvoke({
|
|||||||
|
|
||||||
const inputArtifacts: any[] = [];
|
const inputArtifacts: any[] = [];
|
||||||
for await (const item of input) inputArtifacts.push(item);
|
for await (const item of input) inputArtifacts.push(item);
|
||||||
|
// Draining pipeline input waits on the upstream step, so a timeout can fire
|
||||||
|
// here. Re-check before the reuse lookups below can answer with a success.
|
||||||
|
throwIfCancelled(signal);
|
||||||
|
|
||||||
const normalizedArtifacts = [...inputArtifacts, ...providedArtifacts].map(normalizeArtifact);
|
const normalizedArtifacts = [...inputArtifacts, ...providedArtifacts].map(normalizeArtifact);
|
||||||
const artifactHashes = normalizedArtifacts.map(hashArtifact);
|
const artifactHashes = normalizedArtifacts.map(hashArtifact);
|
||||||
@@ -801,6 +808,9 @@ async function runLlmInvoke({
|
|||||||
|
|
||||||
if (stateKey && !forceRefresh) {
|
if (stateKey && !forceRefresh) {
|
||||||
const stored = await readReusableLlmState(env, stateKey, ctx.signal);
|
const stored = await readReusableLlmState(env, stateKey, ctx.signal);
|
||||||
|
// Reading run state is I/O of unbounded duration; a signal that aborted
|
||||||
|
// during it must not be overtaken by the replay below.
|
||||||
|
throwIfCancelled(signal);
|
||||||
const reused = pickReusableState(stored, cacheKey, config.stateType);
|
const reused = pickReusableState(stored, cacheKey, config.stateType);
|
||||||
if (reused) {
|
if (reused) {
|
||||||
const replay: LlmProvenance = { cacheKey, replayed: true };
|
const replay: LlmProvenance = { cacheKey, replayed: true };
|
||||||
@@ -816,6 +826,7 @@ async function runLlmInvoke({
|
|||||||
|
|
||||||
if (!disableCache && !forceRefresh) {
|
if (!disableCache && !forceRefresh) {
|
||||||
const cache = await readCacheEntry(env, cacheKey, config.cacheNamespace, ctx.signal);
|
const cache = await readCacheEntry(env, cacheKey, config.cacheNamespace, ctx.signal);
|
||||||
|
throwIfCancelled(signal);
|
||||||
if (cache) {
|
if (cache) {
|
||||||
const replay: LlmProvenance = { cacheKey, replayed: true };
|
const replay: LlmProvenance = { cacheKey, replayed: true };
|
||||||
return {
|
return {
|
||||||
@@ -849,6 +860,7 @@ async function runLlmInvoke({
|
|||||||
let lastValidationErrors: string[] = [];
|
let lastValidationErrors: string[] = [];
|
||||||
|
|
||||||
while (true) {
|
while (true) {
|
||||||
|
throwIfCancelled(signal);
|
||||||
attempt += 1;
|
attempt += 1;
|
||||||
if (attempt > 1) {
|
if (attempt > 1) {
|
||||||
payload.retryContext = {
|
payload.retryContext = {
|
||||||
@@ -862,9 +874,16 @@ async function runLlmInvoke({
|
|||||||
let responseEnvelope: LlmResponseEnvelope;
|
let responseEnvelope: LlmResponseEnvelope;
|
||||||
try {
|
try {
|
||||||
ctx.signal?.throwIfAborted();
|
ctx.signal?.throwIfAborted();
|
||||||
responseEnvelope = await adapter.invoke({ env, args, payload, signal: ctx.signal });
|
responseEnvelope = await abortable(
|
||||||
|
adapter.invoke({ env, args, payload, signal: ctx.signal }),
|
||||||
|
signal,
|
||||||
|
);
|
||||||
ctx.signal?.throwIfAborted();
|
ctx.signal?.throwIfAborted();
|
||||||
} catch (err: any) {
|
} catch (err: any) {
|
||||||
|
// Cancellation is the caller's error, not an adapter failure: surface it
|
||||||
|
// as an abort so workflow timeout and abort handling still recognizes it,
|
||||||
|
// rather than wrapping it in a "request failed" adapter error.
|
||||||
|
if (signal?.aborted) throw asCancellation(err, signal);
|
||||||
throw new Error(`${config.name} request failed: ${err?.message ?? String(err)}`);
|
throw new Error(`${config.name} request failed: ${err?.message ?? String(err)}`);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -938,6 +957,66 @@ async function runLlmInvoke({
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Build the error a cancelled run rejects with.
|
||||||
|
*
|
||||||
|
* The workflow runner only treats an error named `AbortError` or coded
|
||||||
|
* `ABORT_ERR` as cancellation; everything else follows the step's retry and
|
||||||
|
* `on_error` policy. A host may abort with any reason it likes, so passing
|
||||||
|
* `signal.reason` straight through means `controller.abort(new Error("stop"))`
|
||||||
|
* leaves a cancelled step looking like an ordinary failure -- retried, or
|
||||||
|
* swallowed by `on_error: continue` and reported as a successful run. Keep the
|
||||||
|
* host's message and hang its reason off `cause`, but mark the rejection so the
|
||||||
|
* runner recognizes it.
|
||||||
|
*/
|
||||||
|
function cancellationError(signal: AbortSignal): unknown {
|
||||||
|
const reason: any = signal.reason;
|
||||||
|
if (reason === undefined || reason === null) {
|
||||||
|
return new DOMException("The operation was aborted.", "AbortError");
|
||||||
|
}
|
||||||
|
if (reason?.name === "AbortError" || reason?.code === "ABORT_ERR") return reason;
|
||||||
|
const message =
|
||||||
|
typeof reason?.message === "string" && reason.message ? reason.message : String(reason);
|
||||||
|
const error: any = new Error(message, { cause: reason });
|
||||||
|
error.name = "AbortError";
|
||||||
|
error.code = "ABORT_ERR";
|
||||||
|
return error;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Rethrow `err` when it already reads as cancellation, else the run's reason. */
|
||||||
|
function asCancellation(err: any, signal: AbortSignal): unknown {
|
||||||
|
if (err?.name === "AbortError" || err?.code === "ABORT_ERR") return err;
|
||||||
|
return cancellationError(signal);
|
||||||
|
}
|
||||||
|
|
||||||
|
function throwIfCancelled(signal?: AbortSignal): void {
|
||||||
|
if (signal?.aborted) throw cancellationError(signal);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Reject as soon as the run is cancelled instead of waiting for `promise`.
|
||||||
|
* HTTP adapters are cancelled at the socket, but an injected `ctx.llmAdapters`
|
||||||
|
* adapter may ignore `ctx.signal` entirely; without this, one of those keeps a
|
||||||
|
* timed-out step waiting for as long as it likes.
|
||||||
|
*/
|
||||||
|
function abortable<T>(promise: Promise<T>, signal?: AbortSignal): Promise<T> {
|
||||||
|
if (!signal) return promise;
|
||||||
|
return new Promise<T>((resolve, reject) => {
|
||||||
|
const onAbort = () => reject(cancellationError(signal));
|
||||||
|
// Observe `promise` before anything can settle the wrapper, including the
|
||||||
|
// already-aborted case below. An adapter can cancel the run from inside its own
|
||||||
|
// `invoke` and reject afterwards; rejecting here without watching that promise
|
||||||
|
// leaves the rejection unhandled, which ends the process under Node's default
|
||||||
|
// handling -- long after this step was cancelled cleanly.
|
||||||
|
promise.then(resolve, reject).finally(() => signal.removeEventListener("abort", onAbort));
|
||||||
|
if (signal.aborted) {
|
||||||
|
onAbort();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
signal.addEventListener("abort", onAbort, { once: true });
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
function resolveProvider(
|
function resolveProvider(
|
||||||
args: any,
|
args: any,
|
||||||
env: any,
|
env: any,
|
||||||
@@ -983,6 +1062,7 @@ function resolveAdapter({
|
|||||||
config: CommandConfig;
|
config: CommandConfig;
|
||||||
ctx: any;
|
ctx: any;
|
||||||
}): Adapter {
|
}): Adapter {
|
||||||
|
const signal: AbortSignal | undefined = ctx?.signal;
|
||||||
const direct = getDirectAdapter(ctx, provider);
|
const direct = getDirectAdapter(ctx, provider);
|
||||||
if (direct) {
|
if (direct) {
|
||||||
const invoke = typeof direct === "function" ? direct : direct.invoke;
|
const invoke = typeof direct === "function" ? direct : direct.invoke;
|
||||||
|
|||||||
@@ -422,6 +422,380 @@ test("llm.invoke uses Pi adapter over local HTTP bridge", async () => {
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test(
|
||||||
|
"llm.invoke aborts the in-flight adapter request when ctx.signal aborts",
|
||||||
|
{ timeout: 20_000 },
|
||||||
|
async () => {
|
||||||
|
const registry = createDefaultRegistry();
|
||||||
|
const cmd = registry.get("llm.invoke");
|
||||||
|
assert.ok(cmd);
|
||||||
|
|
||||||
|
const stalled = createStalledAdapter();
|
||||||
|
await stalled.listen();
|
||||||
|
|
||||||
|
const controller = new AbortController();
|
||||||
|
try {
|
||||||
|
const pending = cmd.run({
|
||||||
|
input: streamOf([]),
|
||||||
|
args: { _: [], provider: "http", prompt: "Summarize", "disable-cache": true },
|
||||||
|
ctx: {
|
||||||
|
...baseCtx(
|
||||||
|
{ LOBSTER_LLM_ADAPTER_URL: `http://127.0.0.1:${stalled.port}/invoke` },
|
||||||
|
registry,
|
||||||
|
),
|
||||||
|
signal: controller.signal,
|
||||||
|
},
|
||||||
|
} as any);
|
||||||
|
|
||||||
|
await stalled.requestReceived;
|
||||||
|
controller.abort();
|
||||||
|
|
||||||
|
const err = await pending.then(
|
||||||
|
() => null,
|
||||||
|
(e: any) => e,
|
||||||
|
);
|
||||||
|
assert.ok(err, "llm.invoke should reject once the run is aborted");
|
||||||
|
assert.ok(
|
||||||
|
err.name === "AbortError" || err.code === "ABORT_ERR",
|
||||||
|
`expected an abort error, got ${err.name}: ${err.message}`,
|
||||||
|
);
|
||||||
|
// The abort must stay recognizable; wrapping it hides the cancellation from
|
||||||
|
// workflow timeout and abort handling.
|
||||||
|
assert.doesNotMatch(String(err.message), /request failed/);
|
||||||
|
await stalled.requestClosed;
|
||||||
|
} finally {
|
||||||
|
controller.abort();
|
||||||
|
await stalled.close();
|
||||||
|
}
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
test(
|
||||||
|
"llm.invoke stops waiting for a direct adapter that ignores ctx.signal",
|
||||||
|
{ timeout: 20_000 },
|
||||||
|
async () => {
|
||||||
|
const registry = createDefaultRegistry();
|
||||||
|
const cmd = registry.get("llm.invoke");
|
||||||
|
assert.ok(cmd);
|
||||||
|
|
||||||
|
let invoked = () => {};
|
||||||
|
const adapterInvoked = new Promise<void>((resolve) => (invoked = resolve));
|
||||||
|
// A supported ctx.llmAdapters adapter that never resolves and never looks
|
||||||
|
// at ctx.signal.
|
||||||
|
const stubborn = {
|
||||||
|
invoke() {
|
||||||
|
invoked();
|
||||||
|
return new Promise<never>(() => {});
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
const controller = new AbortController();
|
||||||
|
const pending = cmd.run({
|
||||||
|
input: streamOf([]),
|
||||||
|
args: { _: [], provider: "stubborn", prompt: "Summarize", "disable-cache": true },
|
||||||
|
ctx: {
|
||||||
|
...baseCtx({}, registry),
|
||||||
|
llmAdapters: { stubborn },
|
||||||
|
signal: controller.signal,
|
||||||
|
},
|
||||||
|
} as any);
|
||||||
|
|
||||||
|
await adapterInvoked;
|
||||||
|
controller.abort();
|
||||||
|
|
||||||
|
const err = await pending.then(
|
||||||
|
() => null,
|
||||||
|
(e: any) => e,
|
||||||
|
);
|
||||||
|
assert.ok(err, "llm.invoke should reject once the run is aborted");
|
||||||
|
assert.ok(
|
||||||
|
err.name === "AbortError" || err.code === "ABORT_ERR",
|
||||||
|
`expected an abort error, got ${err.name}: ${err.message}`,
|
||||||
|
);
|
||||||
|
assert.doesNotMatch(String(err.message), /request failed/);
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
test("llm.invoke observes a direct adapter that rejects after aborting the run", async () => {
|
||||||
|
const registry = createDefaultRegistry();
|
||||||
|
const cmd = registry.get("llm.invoke");
|
||||||
|
assert.ok(cmd);
|
||||||
|
|
||||||
|
const controller = new AbortController();
|
||||||
|
// An adapter that cancels the run from inside its own invoke and then fails:
|
||||||
|
// the shape of an SDK client that tears itself down on a fatal error.
|
||||||
|
const selfaborting = {
|
||||||
|
invoke() {
|
||||||
|
controller.abort();
|
||||||
|
return Promise.reject(new Error("adapter tore down its client"));
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
const unhandled: any[] = [];
|
||||||
|
const onUnhandled = (reason: any) => unhandled.push(reason);
|
||||||
|
process.on("unhandledRejection", onUnhandled);
|
||||||
|
try {
|
||||||
|
const err = await cmd
|
||||||
|
.run({
|
||||||
|
input: streamOf([]),
|
||||||
|
args: { _: [], provider: "selfaborting", prompt: "Summarize", "disable-cache": true },
|
||||||
|
ctx: {
|
||||||
|
...baseCtx({}, registry),
|
||||||
|
llmAdapters: { selfaborting },
|
||||||
|
signal: controller.signal,
|
||||||
|
},
|
||||||
|
} as any)
|
||||||
|
.then(
|
||||||
|
() => null,
|
||||||
|
(e: any) => e,
|
||||||
|
);
|
||||||
|
assert.ok(
|
||||||
|
err?.name === "AbortError" || err?.code === "ABORT_ERR",
|
||||||
|
`expected an abort error, got ${err?.name}: ${err?.message}`,
|
||||||
|
);
|
||||||
|
// The adapter's own rejection must not outlive the cancelled step. Nothing is
|
||||||
|
// waiting on it once the abort has won the race, and an unobserved rejection
|
||||||
|
// ends the process under Node's default handling.
|
||||||
|
await new Promise((resolve) => setTimeout(resolve, 50));
|
||||||
|
assert.equal(
|
||||||
|
unhandled.length,
|
||||||
|
0,
|
||||||
|
`adapter rejection went unhandled: ${unhandled[0]?.message ?? unhandled[0]}`,
|
||||||
|
);
|
||||||
|
} finally {
|
||||||
|
process.off("unhandledRejection", onUnhandled);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test("llm.invoke does not complete from cache when ctx.signal is already aborted", async () => {
|
||||||
|
const registry = createDefaultRegistry();
|
||||||
|
const cmd = registry.get("llm.invoke");
|
||||||
|
assert.ok(cmd);
|
||||||
|
const cacheDir = await mkdtemp(path.join(tmpdir(), "lobster-cache-"));
|
||||||
|
await mkdir(path.join(cacheDir, "llm.invoke"), { recursive: true });
|
||||||
|
|
||||||
|
let requests = 0;
|
||||||
|
const server = http.createServer((req, res) => {
|
||||||
|
requests += 1;
|
||||||
|
req.resume();
|
||||||
|
req.on("end", () => {
|
||||||
|
res.writeHead(200, { "content-type": "application/json" });
|
||||||
|
res.end(
|
||||||
|
JSON.stringify({
|
||||||
|
ok: true,
|
||||||
|
result: { runId: "cached_1", output: { text: "hello", data: null } },
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
await new Promise<void>((resolve) => server.listen(0, resolve));
|
||||||
|
const addr = server.address();
|
||||||
|
const port = typeof addr === "object" && addr ? addr.port : 0;
|
||||||
|
const env = {
|
||||||
|
LOBSTER_LLM_ADAPTER_URL: `http://127.0.0.1:${port}/invoke`,
|
||||||
|
LOBSTER_CACHE_DIR: cacheDir,
|
||||||
|
};
|
||||||
|
const args = { _: [], provider: "http", prompt: "Summarize" };
|
||||||
|
|
||||||
|
try {
|
||||||
|
// Populate the cache, then repeat the same call on a cancelled run.
|
||||||
|
const warm = await cmd.run({
|
||||||
|
input: streamOf([]),
|
||||||
|
args,
|
||||||
|
ctx: baseCtx(env, registry),
|
||||||
|
} as any);
|
||||||
|
assert.equal((await collect(warm.output!))[0].cached, false);
|
||||||
|
assert.equal(requests, 1);
|
||||||
|
|
||||||
|
const controller = new AbortController();
|
||||||
|
controller.abort();
|
||||||
|
await assert.rejects(
|
||||||
|
() =>
|
||||||
|
cmd.run({
|
||||||
|
input: streamOf([]),
|
||||||
|
args,
|
||||||
|
ctx: { ...baseCtx(env, registry), signal: controller.signal },
|
||||||
|
} as any),
|
||||||
|
(err: any) => err?.name === "AbortError" || err?.code === "ABORT_ERR",
|
||||||
|
);
|
||||||
|
assert.equal(requests, 1, "the cached run must not reach the adapter either");
|
||||||
|
} finally {
|
||||||
|
await rm(cacheDir, { recursive: true, force: true });
|
||||||
|
await closeServer(server);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test("llm.invoke does not replay a cache hit when ctx.signal aborts while input is drained", async () => {
|
||||||
|
const registry = createDefaultRegistry();
|
||||||
|
const cmd = registry.get("llm.invoke");
|
||||||
|
assert.ok(cmd);
|
||||||
|
const cacheDir = await mkdtemp(path.join(tmpdir(), "lobster-cache-"));
|
||||||
|
await mkdir(path.join(cacheDir, "llm.invoke"), { recursive: true });
|
||||||
|
|
||||||
|
let requests = 0;
|
||||||
|
const server = http.createServer((req, res) => {
|
||||||
|
requests += 1;
|
||||||
|
req.resume();
|
||||||
|
req.on("end", () => {
|
||||||
|
res.writeHead(200, { "content-type": "application/json" });
|
||||||
|
res.end(
|
||||||
|
JSON.stringify({
|
||||||
|
ok: true,
|
||||||
|
result: { runId: "cached_1", output: { text: "hello", data: null } },
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
await new Promise<void>((resolve) => server.listen(0, resolve));
|
||||||
|
const addr = server.address();
|
||||||
|
const port = typeof addr === "object" && addr ? addr.port : 0;
|
||||||
|
const env = {
|
||||||
|
LOBSTER_LLM_ADAPTER_URL: `http://127.0.0.1:${port}/invoke`,
|
||||||
|
LOBSTER_CACHE_DIR: cacheDir,
|
||||||
|
};
|
||||||
|
const args = { _: [], provider: "http", prompt: "Summarize" };
|
||||||
|
|
||||||
|
try {
|
||||||
|
const warm = await cmd.run({
|
||||||
|
input: streamOf([]),
|
||||||
|
args,
|
||||||
|
ctx: baseCtx(env, registry),
|
||||||
|
} as any);
|
||||||
|
assert.equal((await collect(warm.output!))[0].cached, false);
|
||||||
|
assert.equal(requests, 1);
|
||||||
|
|
||||||
|
// The step timeout fires while the upstream step is still producing input,
|
||||||
|
// which is after the entry check and before the cache lookup.
|
||||||
|
const controller = new AbortController();
|
||||||
|
const abortingInput = (async function* () {
|
||||||
|
await new Promise((resolve) => setTimeout(resolve, 5));
|
||||||
|
controller.abort();
|
||||||
|
})();
|
||||||
|
|
||||||
|
await assert.rejects(
|
||||||
|
() =>
|
||||||
|
cmd.run({
|
||||||
|
input: abortingInput,
|
||||||
|
args,
|
||||||
|
ctx: { ...baseCtx(env, registry), signal: controller.signal },
|
||||||
|
} as any),
|
||||||
|
(err: any) => err?.name === "AbortError" || err?.code === "ABORT_ERR",
|
||||||
|
);
|
||||||
|
assert.equal(requests, 1, "a cancelled run must not reach the adapter either");
|
||||||
|
} finally {
|
||||||
|
await rm(cacheDir, { recursive: true, force: true });
|
||||||
|
await closeServer(server);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test("llm.invoke does not replay run state when ctx.signal aborts while input is drained", async () => {
|
||||||
|
const registry = createDefaultRegistry();
|
||||||
|
const cmd = registry.get("llm.invoke");
|
||||||
|
assert.ok(cmd);
|
||||||
|
const stateDir = await mkdtemp(path.join(tmpdir(), "lobster-state-"));
|
||||||
|
|
||||||
|
let requests = 0;
|
||||||
|
const server = http.createServer((req, res) => {
|
||||||
|
requests += 1;
|
||||||
|
req.resume();
|
||||||
|
req.on("end", () => {
|
||||||
|
res.writeHead(200, { "content-type": "application/json" });
|
||||||
|
res.end(
|
||||||
|
JSON.stringify({
|
||||||
|
ok: true,
|
||||||
|
result: { runId: "state_1", output: { text: "hello", data: null } },
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
await new Promise<void>((resolve) => server.listen(0, resolve));
|
||||||
|
const addr = server.address();
|
||||||
|
const port = typeof addr === "object" && addr ? addr.port : 0;
|
||||||
|
const env = {
|
||||||
|
LOBSTER_LLM_ADAPTER_URL: `http://127.0.0.1:${port}/invoke`,
|
||||||
|
LOBSTER_STATE_DIR: stateDir,
|
||||||
|
};
|
||||||
|
const args = {
|
||||||
|
_: [],
|
||||||
|
provider: "http",
|
||||||
|
prompt: "Summarize",
|
||||||
|
"state-key": "step-1",
|
||||||
|
"disable-cache": true,
|
||||||
|
};
|
||||||
|
|
||||||
|
try {
|
||||||
|
const warm = await cmd.run({
|
||||||
|
input: streamOf([]),
|
||||||
|
args,
|
||||||
|
ctx: baseCtx(env, registry),
|
||||||
|
} as any);
|
||||||
|
assert.equal((await collect(warm.output!))[0].cached, false);
|
||||||
|
assert.equal(requests, 1);
|
||||||
|
|
||||||
|
const controller = new AbortController();
|
||||||
|
const abortingInput = (async function* () {
|
||||||
|
await new Promise((resolve) => setTimeout(resolve, 5));
|
||||||
|
controller.abort();
|
||||||
|
})();
|
||||||
|
|
||||||
|
await assert.rejects(
|
||||||
|
() =>
|
||||||
|
cmd.run({
|
||||||
|
input: abortingInput,
|
||||||
|
args,
|
||||||
|
ctx: { ...baseCtx(env, registry), signal: controller.signal },
|
||||||
|
} as any),
|
||||||
|
(err: any) => err?.name === "AbortError" || err?.code === "ABORT_ERR",
|
||||||
|
);
|
||||||
|
assert.equal(requests, 1, "a cancelled run must not reach the adapter either");
|
||||||
|
} finally {
|
||||||
|
await rm(stateDir, { recursive: true, force: true });
|
||||||
|
await closeServer(server);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test("llm.invoke does not call the adapter when ctx.signal is already aborted", async () => {
|
||||||
|
const registry = createDefaultRegistry();
|
||||||
|
const cmd = registry.get("llm.invoke");
|
||||||
|
assert.ok(cmd);
|
||||||
|
|
||||||
|
let requests = 0;
|
||||||
|
const server = http.createServer((req, res) => {
|
||||||
|
requests += 1;
|
||||||
|
req.resume();
|
||||||
|
res.writeHead(200, { "content-type": "application/json" });
|
||||||
|
res.end(JSON.stringify({ ok: true, result: { runId: "r1", output: { data: {} } } }));
|
||||||
|
});
|
||||||
|
|
||||||
|
await new Promise<void>((resolve) => server.listen(0, resolve));
|
||||||
|
const addr = server.address();
|
||||||
|
const port = typeof addr === "object" && addr ? addr.port : 0;
|
||||||
|
|
||||||
|
try {
|
||||||
|
const controller = new AbortController();
|
||||||
|
controller.abort();
|
||||||
|
|
||||||
|
await assert.rejects(
|
||||||
|
() =>
|
||||||
|
cmd.run({
|
||||||
|
input: streamOf([]),
|
||||||
|
args: { _: [], provider: "http", prompt: "Summarize", "disable-cache": true },
|
||||||
|
ctx: {
|
||||||
|
...baseCtx({ LOBSTER_LLM_ADAPTER_URL: `http://127.0.0.1:${port}/invoke` }, registry),
|
||||||
|
signal: controller.signal,
|
||||||
|
},
|
||||||
|
} as any),
|
||||||
|
(err: any) => err?.name === "AbortError" || err?.code === "ABORT_ERR",
|
||||||
|
);
|
||||||
|
assert.equal(requests, 0, "an already-cancelled run must not reach the adapter");
|
||||||
|
} finally {
|
||||||
|
await closeServer(server);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
test("llm.invoke does not retry schema validation after adapter cancellation", async () => {
|
test("llm.invoke does not retry schema validation after adapter cancellation", async () => {
|
||||||
const registry = createDefaultRegistry();
|
const registry = createDefaultRegistry();
|
||||||
const cmd = registry.get("llm.invoke");
|
const cmd = registry.get("llm.invoke");
|
||||||
@@ -878,6 +1252,44 @@ function baseCtx(envOverrides: Record<string, string>, registry?: any) {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// An adapter that accepts the request and never answers, so the only thing that
|
||||||
|
// can end the call is the caller cancelling it.
|
||||||
|
function createStalledAdapter() {
|
||||||
|
const sockets = new Set<import("node:net").Socket>();
|
||||||
|
let markReceived = () => {};
|
||||||
|
let markClosed = () => {};
|
||||||
|
const requestReceived = new Promise<void>((resolve) => (markReceived = resolve));
|
||||||
|
const requestClosed = new Promise<void>((resolve) => (markClosed = resolve));
|
||||||
|
|
||||||
|
const server = http.createServer((req, res) => {
|
||||||
|
req.resume();
|
||||||
|
req.on("end", () => {
|
||||||
|
res.on("close", () => markClosed());
|
||||||
|
markReceived();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
server.on("connection", (socket) => {
|
||||||
|
sockets.add(socket);
|
||||||
|
socket.on("close", () => sockets.delete(socket));
|
||||||
|
});
|
||||||
|
|
||||||
|
return {
|
||||||
|
requestReceived,
|
||||||
|
requestClosed,
|
||||||
|
port: 0,
|
||||||
|
async listen() {
|
||||||
|
await new Promise<void>((resolve) => server.listen(0, resolve));
|
||||||
|
const addr = server.address();
|
||||||
|
this.port = typeof addr === "object" && addr ? addr.port : 0;
|
||||||
|
},
|
||||||
|
async close() {
|
||||||
|
for (const socket of sockets) socket.destroy();
|
||||||
|
if (!server.listening) return;
|
||||||
|
await new Promise<void>((resolve) => server.close(() => resolve()));
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
async function closeServer(server: http.Server) {
|
async function closeServer(server: http.Server) {
|
||||||
if (!server.listening) return;
|
if (!server.listening) return;
|
||||||
await new Promise<void>((resolve) => server.close(() => resolve()));
|
await new Promise<void>((resolve) => server.close(() => resolve()));
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
import test from "node:test";
|
import test from "node:test";
|
||||||
import assert from "node:assert/strict";
|
import assert from "node:assert/strict";
|
||||||
import { promises as fsp } from "node:fs";
|
import { promises as fsp } from "node:fs";
|
||||||
|
import http from "node:http";
|
||||||
import os from "node:os";
|
import os from "node:os";
|
||||||
import path from "node:path";
|
import path from "node:path";
|
||||||
import { PassThrough } from "node:stream";
|
import { PassThrough } from "node:stream";
|
||||||
@@ -13,6 +14,8 @@ async function runWorkflow(
|
|||||||
opts?: {
|
opts?: {
|
||||||
signal?: AbortSignal;
|
signal?: AbortSignal;
|
||||||
dryRun?: boolean;
|
dryRun?: boolean;
|
||||||
|
env?: Record<string, string>;
|
||||||
|
llmAdapters?: Record<string, unknown>;
|
||||||
},
|
},
|
||||||
) {
|
) {
|
||||||
const tmpDir = await fsp.mkdtemp(path.join(os.tmpdir(), "lobster-step-timeout-"));
|
const tmpDir = await fsp.mkdtemp(path.join(os.tmpdir(), "lobster-step-timeout-"));
|
||||||
@@ -30,10 +33,11 @@ async function runWorkflow(
|
|||||||
stdin: process.stdin,
|
stdin: process.stdin,
|
||||||
stdout: process.stdout,
|
stdout: process.stdout,
|
||||||
stderr,
|
stderr,
|
||||||
env: { ...process.env, LOBSTER_STATE_DIR: stateDir },
|
env: { ...process.env, LOBSTER_STATE_DIR: stateDir, ...opts?.env },
|
||||||
mode: "tool",
|
mode: "tool",
|
||||||
signal: opts?.signal,
|
signal: opts?.signal,
|
||||||
dryRun: opts?.dryRun,
|
dryRun: opts?.dryRun,
|
||||||
|
llmAdapters: opts?.llmAdapters,
|
||||||
registry: createDefaultRegistry(),
|
registry: createDefaultRegistry(),
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
@@ -188,6 +192,51 @@ test("external abort still propagates when timeout is configured", async () => {
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test("timed-out llm.invoke step stops waiting for the adapter", { timeout: 20_000 }, async () => {
|
||||||
|
const sockets = new Set<import("node:net").Socket>();
|
||||||
|
let markClosed = () => {};
|
||||||
|
const requestClosed = new Promise<void>((resolve) => (markClosed = resolve));
|
||||||
|
|
||||||
|
// An adapter that accepts the request and never answers.
|
||||||
|
const server = http.createServer((req, res) => {
|
||||||
|
req.resume();
|
||||||
|
req.on("end", () => res.on("close", () => markClosed()));
|
||||||
|
});
|
||||||
|
server.on("connection", (socket) => {
|
||||||
|
sockets.add(socket);
|
||||||
|
socket.on("close", () => sockets.delete(socket));
|
||||||
|
});
|
||||||
|
|
||||||
|
await new Promise<void>((resolve) => server.listen(0, resolve));
|
||||||
|
const addr = server.address();
|
||||||
|
const port = typeof addr === "object" && addr ? addr.port : 0;
|
||||||
|
|
||||||
|
try {
|
||||||
|
const started = Date.now();
|
||||||
|
await assert.rejects(
|
||||||
|
() =>
|
||||||
|
runWorkflow(
|
||||||
|
{
|
||||||
|
steps: [
|
||||||
|
{
|
||||||
|
id: "ask",
|
||||||
|
pipeline: 'llm.invoke --prompt "Summarize" --disable-cache',
|
||||||
|
timeout_ms: 300,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{ env: { LOBSTER_LLM_ADAPTER_URL: `http://127.0.0.1:${port}/invoke` } },
|
||||||
|
),
|
||||||
|
/timed out after 300ms/,
|
||||||
|
);
|
||||||
|
assert.ok(Date.now() - started < 10_000, "the step must not wait for the adapter");
|
||||||
|
await requestClosed;
|
||||||
|
} finally {
|
||||||
|
for (const socket of sockets) socket.destroy();
|
||||||
|
await new Promise<void>((resolve) => server.close(() => resolve()));
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
test("dry-run renders timeout and on_error details", async () => {
|
test("dry-run renders timeout and on_error details", async () => {
|
||||||
const { stderrOutput } = await runWorkflow(
|
const { stderrOutput } = await runWorkflow(
|
||||||
{
|
{
|
||||||
@@ -205,3 +254,46 @@ test("dry-run renders timeout and on_error details", async () => {
|
|||||||
assert.match(stderrOutput, /timeout: 5000ms/);
|
assert.match(stderrOutput, /timeout: 5000ms/);
|
||||||
assert.match(stderrOutput, /on_error: continue/);
|
assert.match(stderrOutput, /on_error: continue/);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test(
|
||||||
|
"external abort with a custom reason still stops the workflow",
|
||||||
|
{ timeout: 20_000 },
|
||||||
|
async () => {
|
||||||
|
let invoked = () => {};
|
||||||
|
const adapterInvoked = new Promise<void>((resolve) => (invoked = resolve));
|
||||||
|
// Ignores ctx.signal, so what the runner classifies is llm.invoke's own
|
||||||
|
// abort rejection rather than a cancelled socket.
|
||||||
|
const stubborn = {
|
||||||
|
invoke() {
|
||||||
|
invoked();
|
||||||
|
return new Promise<never>(() => {});
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
const controller = new AbortController();
|
||||||
|
// A host may abort with any reason. A plain Error must still read as
|
||||||
|
// cancellation, not as an ordinary step failure.
|
||||||
|
void adapterInvoked.then(() => controller.abort(new Error("cancelled by host")));
|
||||||
|
|
||||||
|
await assert.rejects(
|
||||||
|
() =>
|
||||||
|
runWorkflow(
|
||||||
|
{
|
||||||
|
steps: [
|
||||||
|
{
|
||||||
|
id: "ask",
|
||||||
|
// on_error: continue is what makes a misclassified abort
|
||||||
|
// visible -- the run would otherwise report success.
|
||||||
|
pipeline: 'llm.invoke --provider stubborn --prompt "Summarize" --disable-cache',
|
||||||
|
on_error: "continue",
|
||||||
|
},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{ signal: controller.signal, llmAdapters: { stubborn } },
|
||||||
|
),
|
||||||
|
(err: any) =>
|
||||||
|
(err?.name === "AbortError" || err?.code === "ABORT_ERR") &&
|
||||||
|
/cancelled by host/.test(String(err?.message)),
|
||||||
|
);
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|||||||
Reference in New Issue
Block a user