* fix(llm): honor step timeouts and cancellation during model calls
llm.invoke never read ctx.signal, and both adapter helpers called fetch
without one, so nothing could end a request once it was in flight. A
workflow step with timeout_ms kept waiting for the adapter and, when the
model eventually answered, reported the step as completed well past its
budget. Shell steps already honor the same signal, so the two step kinds
disagreed about what timeout_ms means.
Thread ctx.signal into both adapter fetch calls, check it before each
attempt so an already-cancelled run does not pay for another call, and
rethrow abort errors unwrapped: the workflow runner recognizes timeouts
and external cancellation by the error's identity, so wrapping them in
"request failed" would misreport the outcome even once the request stops.
* fix(llm): stop waiting for adapters that ignore the cancellation signal
Threading the signal into fetch only covers the HTTP adapters. An injected
ctx.llmAdapters adapter is awaited directly, so one that never observes
ctx.signal still holds a timed-out step open for as long as it likes, which
leaves SDK and tool-runtime users on the weaker timeout contract the rest of
this change removes.
Await the adapter call against the signal instead of the adapter alone. The
adapter's own work cannot be killed from here, but the step stops waiting on
it and the abort reason reaches the workflow unchanged. Cooperative adapters
are unaffected: they still receive ctx and can end their own work.
* fix(llm): do not finish a cancelled run from cached results
Run-state and file-cache hits return before any adapter call, so a run that
was already cancelled still completed successfully from cached data. That
contradicts the rest of this change: the same invocation reports cancellation
when it has to reach a model and success when it does not, purely on cache
contents.
Check the signal once at entry, before either lookup. The retry-loop check
stays, since cancellation can also arrive between validation attempts.
* docs(readme): state the cancellation contract for injected LLM adapters
Racing an adapter promise against the step signal restores workflow liveness,
but it cannot cancel work the adapter has already started: an injected adapter
that ignores `ctx.signal` keeps its model request running after Lobster stops
awaiting it, and a configured retry can then overlap it. That is a contract for
the host to meet, so the host-facing docs have to say it rather than leaving it
to be discovered from a duplicated charge.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Bu6MvbjKGtfRiNesARMyZ
* fix(llm): re-check cancellation after each awaited reuse boundary
The entry check only caught a run that was already cancelled when the
command started. Draining pipeline input waits on the upstream step, and
the run-state and cache lookups are file I/O, so a step timeout could
fire during any of them and the reuse branch would still return its
stored answer as a successful result.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gD5sJTbh2jn1uvgNCmkLq
* fix(llm): do not report a step that finished after its deadline
The cancellation checks stopped at the adapter call, but both return
paths then await run-state and cache writes. A deadline crossed during
those writes was never observed, so the step returned success and the
workflow cleared its timer -- a model step could still complete late.
The check goes after both writes rather than between them: the answer
has already been paid for, and leaving it stored lets the retry replay
it instead of calling the model a second time.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gD5sJTbh2jn1uvgNCmkLq
* test(llm): let the persistence tests run where the state fix has not landed
The two cancellation-during-write tests let the command create the cache
namespace directory itself, which is where #125 throws on Windows: the
recursive mkdir reports an extended-length path and the chain sync then
opens `<dir>\C:`. Both tests failed there before reaching an assertion,
so this branch added two Windows failures over main for a reason that is
not its own.
Creating the directory up front is what the seeded-cache tests in this
file already do. ensureDirectory only syncs a chain it created, so an
existing directory never reaches the broken path, and the tests still
fail against the code they cover -- now on the missing rejection rather
than on ENOENT.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwU1bvb9Pt3cR662P52CTp
* fix(llm): let a host's own abort reason still read as cancellation
The workflow runner classifies cancellation by name alone: only
`AbortError` or `ABORT_ERR` bypasses the step's retry and `on_error`
policy. Every abort path here rejected with `signal.reason` untouched,
so a host that calls `controller.abort(new Error("stop"))` handed the
runner an ordinary-looking error. A cancelled `llm.invoke` step under
`on_error: continue` was recorded as a step failure and the run carried
on -- returning `status: "ok"` for a run the host had cancelled.
The gap is narrow, which is why it survived earlier review. An HTTP
adapter is saved by fetch rejecting with its own `AbortError`, and the
runner's own step timer is not affected. It opens only on the injected
`ctx.llmAdapters` path, where this command's rejection is the only thing
the runner ever sees.
Normalize at the boundary instead of guessing upstream: a reason that
already reads as an abort passes through untouched, anything else keeps
its message and moves to `cause` under an abort-identified error. The
host's reason is never discarded, and callers that already match on
`AbortError` keep matching.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WHhk7xRwXzYcSXvw2kp5F5
* fix(llm): watch an adapter that rejects after aborting the run
`abortable` guards against an adapter that ignores `ctx.signal`, but its
already-aborted branch rejected the wrapper and returned before anything
was attached to the adapter's own promise. An adapter that cancels the
run from inside its own `invoke` -- an SDK client tearing itself down on
a fatal error -- and then rejects leaves that rejection unobserved.
The step still fails correctly, with the `AbortError` the runner needs.
The damage lands afterwards: under Node's default unhandled-rejection
handling the process exits on an error nothing is waiting for, seconds
after the run was cancelled cleanly.
Attach settlement handling first in both branches, so the promise is
watched before an abort can win the race. Removing a listener that was
never added is harmless, which keeps the two paths one shape.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011hFZxjivm9JTTvmRjQ4A8S
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(workflows): do not bill cached llm.invoke replays against cost_limit
A cache or run-state hit returns the stored result verbatim, including the
usage of the call that produced it. trackStepCost recorded that usage again,
so every replay was charged as if it had reached a provider. A workflow that
asks the same question in several steps therefore reports a multiple of what
it spent, and cost_limit with action: stop aborts a run that stayed well
inside its budget.
Skip items whose source is "cache" or "run_state". Those are the only two
paths that re-emit a stored result, and no tokens are consumed on either, so
the reported token totals stay truthful too.
* fix(llm): mark replayed results so live calls are never mistaken for them
The cost guard keyed on source being "cache" or "run_state", but source is not
a replay marker: for a direct adapter it defaults to the provider name
(llm_invoke.ts:549), so an embedder registering an adapter under either name
produces live results that the guard would silently drop from _meta.cost and
cost_limit. Any step emitting that source field had the same effect.
Set an explicit replayed flag at the two sites that re-emit a stored item, and
key the guard on it. The flag is added when the item is replayed, not when it
is stored, so entries already on disk are covered.
* fix(workflows): keep billing workflow output that only looks replayed
trackStepCost inspects every JSON result a workflow step produces, not just
LLM results, so keying the exemption on a bare replayed field let any command
that reports its own replay state beside a real usage object drop out of
_meta.cost and slip past cost_limit. Under-reporting spend is the mirror of
the bug this branch fixes.
Gate the exemption on the normalized item llm.invoke and llm_task.invoke
actually re-emit: the replay marker plus cached, a known item kind, and the
string cacheKey, status, createdAt, and source fields the command always
writes. A shell step or an unrelated tool that happens to carry replayed no
longer qualifies. Checking the shape alone would not be enough either, since
a live call through an adapter registered as "cache" carries the same kind.
* fix(workflows): key the replay exemption to in-process provenance
The replay marker was a set of JSON fields, and workflow cost accounting
reads the JSON of every step — including shell steps, whose stdout is
parsed straight into the same shape. A step that printed a replay-shaped
object beside a real `usage` object dropped out of `_meta.cost`, which
under-reports spend and lets a configured `cost_limit` be bypassed.
Mark replays with a symbol key instead. `JSON.parse` cannot produce one,
so only items this process built in `llm.invoke` are exempt; the public
`replayed` field stays for consumers but no longer grants the exemption.
Tests now drive the exemption through the real path (a pipeline step
running `llm.invoke` against a stub provider, for both the cache and
run-state replays) and pin the forged case: a shell step printing every
accepted replay field plus real usage is still billed and still trips
`cost_limit`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6UVfFPP39RkoYx5jZ3KKM
* fix(workflows): keep replay provenance through a rendered pipeline
`llm.invoke | json` is a supported step shape, and it defeated the
symbol-keyed replay exemption. The renderer prints its items and returns
an empty stream, so the step has no pipeline items and its JSON is parsed
back out of stdout — objects that no symbol can survive into. Cached and
run-state replays were billed again there, inflating `_meta.cost` and
stopping workflows below their real provider spend.
Keep the originals instead of trying to recover them. Each stage's
renderer now records the objects it was handed, the pipeline returns them
alongside its items, and a step whose pipeline produced no items carries
them on a non-enumerable symbol key. Cost accounting reads those in
preference to the re-parsed JSON, so a rendered replay is recognized for
what it is.
Nothing about the forged case changes: the side channel holds only items
this process built, so a step that prints the full replay shape — through
a renderer or not — is still billed and still trips `cost_limit`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016YJKpXm7WiSbyjQzrFS5Ps
* fix(llm): mark the usage record so a projection keeps replay provenance
`pick model,usage` builds a new object out of named fields, so a replayed
item's own mark never reaches workflow cost accounting and the replay is
billed again — the same defect the renderer had, one command over.
Mark the usage record as well as the item. A projection carries the usage
record across by reference, and the usage record is what gets billed, so
the exemption survives without accounting having to recognize every
command that can build a new object. `where`, `head`, `sort` and `dedupe`
yield the original item and were already covered; `group_by` and `map
--wrap` nest the usage where nothing bills it.
`JSON.parse` still cannot produce the key, so a step that prints a usage
object is billed whether or not it is projected or rendered first.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016YJKpXm7WiSbyjQzrFS5Ps
* fix(llm): bill a replay that stands in for a retried step's live call
A workflow records a step's cost only once the step succeeds, but `llm.invoke`
persists its answer to run state and the cache before it returns. So a step that
fails *after* its LLM call and is retried never bills the live item, and the
retry replays the stored answer -- which the replay exemption then dropped,
reporting $0 for a provider call that really happened.
Provenance now carries the cache key and whether the item was replayed, and a
live call opens an unbilled charge against that key. Accounting claims the
charge, so the live item and every replay of it settle it exactly once: a replay
is exempt only while the call behind it has already been paid for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Bu6MvbjKGtfRiNesARMyZ
* fix(llm): scope the unbilled-spend ledger to the run that opened it
The ledger of live calls awaiting a charge was process-global, so a call made
outside a cost-tracked run -- the SDK pipeline API, or a workflow whose step
never succeeded -- left a key behind that a later, unrelated workflow could
claim by replaying the same cache entry. That workflow called no provider, yet
its `_meta.cost` and `cost_limit` grew by someone else's spend.
The ledger is now created per workflow run and reaches commands through the
pipeline context, so only the run that paid for a call can recover its charge
from a replay. A live call with no ledger in context opens no charge at all,
which is what an SDK caller outside cost accounting should do.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Bu6MvbjKGtfRiNesARMyZ
* fix(llm): count outstanding charges per cache key, not just their presence
Two identical calls that race on a cold cache -- two parallel branches asking
the same question -- are two provider charges under one cache key. The ledger
held a set of keys, so both recorded the same entry and only one of the replays
that later stood in for them could settle it. A retried step therefore reported
one call's spend for two, in `_meta.cost` and against `cost_limit`.
Charges are now counted per key, so N live calls can be settled N times and
never fewer. A replay still settles at most one charge, so nothing is billed
twice.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Bu6MvbjKGtfRiNesARMyZ
* fix(workflows): carry recorded spend across an approval or input resume
A gate pauses a run; it does not reset what the run has spent. The resume is a
separate run with its own accounting, so the provider call made before the gate
lived only in the paused run's record: a later step repeating that prompt
replayed it and correctly billed nothing, leaving the call in no total at all.
The same reset let a `cost_limit` be walked past one gate at a time, on `main`
as much as here.
The paused run now stores its cost summary in the resume state, and the run that
resumes seeds its tracker from it, so `_meta.cost` covers the whole workflow and
the budget survives the pause. Stored entries are rebuilt through the same
normalization as live usage, and a resume state written before this change
simply seeds nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Bu6MvbjKGtfRiNesARMyZ
* fix(workflows): keep an unbilled call across a pipeline input resume
A pipeline can pause between paying for a call and billing it: `llm.invoke |
ask` suspends in tool mode after the model has answered, and `ask` consumes the
item that carried the usage, so the step never reports it. The charge existed
only as an outstanding entry in the paused run's ledger, which the resumed run
recreated empty -- so the replay a later step produced was exempt and the call
was billed nowhere.
Outstanding charges now travel in the same resume state as the cost summary and
reopen on resume. It is the same run continuing, so this cannot hand a charge to
an unrelated one, and a resume state written before this change reopens nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Bu6MvbjKGtfRiNesARMyZ
* fix(llm): open the charge when the provider answers, not after it is stored
`record()` ran after `persistOutputs` and the cache write. Either can fail --
an unwritable `LOBSTER_CACHE_DIR`, a full disk -- after run state already holds
a replayable copy of the answer, so the step failed, the retry succeeded from
that replay, and the replay had no charge to settle: a completed provider call
was billed nowhere.
The charge exists the moment the provider answers, so it is opened there. A
store that then fails leaves an entry nothing claims, which is bounded and
per-run; the reverse dropped real spend.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Bu6MvbjKGtfRiNesARMyZ
* docs(readme): state that spend is counted once and survives a pause
The two accounting properties this branch establishes are invisible from the
outside until a bill disagrees with a budget: a replayed answer is not charged
again, and a workflow that pauses at a gate keeps what it has spent, so a cost
limit covers the run rather than the steps after the last pause. The second one
changes what an existing resumed workflow does, which is a thing to read before
it happens.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Bu6MvbjKGtfRiNesARMyZ
* fix(workflows): carry replay provenance across a composed workflow
A `workflow:` step rebuilds its result from the child's serialized
output, so the in-process marker that says which model call an item
replays did not survive the boundary. A child ending in a rendered
pipeline such as `llm.invoke | json` therefore handed back an unmarked
replay, and the composing run billed it again and could stop on a
cost_limit it never spent.
The child now exposes the items it produced on its run result, and its
ledger is chained to the composing run's, so a live call is opened as a
charge in both: each of them counts it once at its own boundary, and a
replay neither of them paid for still settles nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gD5sJTbh2jn1uvgNCmkLq
* fix(workflows): settle live results before the replays billed with them
A step's parallel branches are accounted in declaration order. A cached
replay declared before an identical --refresh branch claimed the charge
that branch had just opened and was billed for it, and the live branch
was billed whatever its own claim returned -- one provider call charged
twice, enough to stop a run at a budget it had not spent.
Replays are now held back until every live result accounted alongside
them has settled its own charge, so a replay can only draw on a call
nothing else will bill.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gD5sJTbh2jn1uvgNCmkLq
* fix(workflows): bill a call no item ever carried to the accounting point
An item is the usual carrier of a call's usage, but three paths lose it
before anything is billed: a renderer consumes the pipeline and a later
stage emits items of its own, an `ask` gate swallows the only item and
the resume produces no replay, and a composed run ends on a step that
carries nothing back. In each the run had already paid the provider, and
the spend was missing from both `_meta.cost` and `cost_limit`.
Each charge now carries what the call cost, and a run settles whatever
is still open when it completes. A paused run never reaches that point,
so its charges keep travelling in the resume state instead.
The same rule reaches a call made by a step that then failed: nothing
billed its items, so the charge is the only record left that the
provider answered, and the run that made the call is where it belongs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gD5sJTbh2jn1uvgNCmkLq
* fix(workflows): settle a charge from the copy of an item that lost its mark
A stage that JSON round-trips the stream hands on an item that kept
`model` and `usage` and lost everything this process attached to it.
That copy is billed, as it always was, but nothing settled the charge
behind it, so the end-of-run pass billed the same provider call again.
The copy now settles it. The public cache key alone cannot -- any step
can print one -- so the ledger settles a charge only when the cost being
billed is the cost it recorded. An invented item can still add spend to
a run, which was always true, but it cannot hide any.
Comparing only the numbers matters: a live item's usage record carries
this module's provenance symbol, which a copy that went through JSON
never can, so comparing the objects whole never matches.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gD5sJTbh2jn1uvgNCmkLq
* fix(workflows): bill each call at its own cost, and before the next step
Two ways the ledger still reported the wrong number.
A replay was billed from the copy it carries. Identical calls that race
on a cold cache are separate charges under one key, only one of their
answers is stored, and every replay repeats that one -- so a retried
step reported both calls at the price of whichever won the cache write.
The charge records what its own call cost, so that is what is billed.
A charge no item carried waited for the end of the run, so `cost_limit`
saw no spend at the step that blew the budget and let every later step
run its side effects first. It is settled as each step finishes instead.
Settling per step means a copy of that item can surface a step later, so
the ledger now remembers what it has already accounted for: such a copy
either settles an open charge and is billed as its carrier, or stands
behind a call already billed and is not billed twice. Both answers need
the cost to be one the run recorded, so an invented item still cannot
hide spend.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gD5sJTbh2jn1uvgNCmkLq
* fix(workflows): enforce the budget a failed step already spent
A step that pays a provider and then fails takes its items with it, so
`on_error: continue` advanced the run with the charge still open and the
limit unchecked. A budget already gone let the next step run its side
effects, which is exactly what `action: stop` exists to prevent.
The charge is settled and the limit checked on that path too, before the
run is allowed to move on.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gD5sJTbh2jn1uvgNCmkLq
* fix(workflows): match a copied item on the tokens that are billed
Comparing every number in a usage record asked a stricter question than
the accounting does. A stage that rebuilds an item can keep the token
counts and drop or recompute a `totalTokens` nobody is charged for, and
the copy then settled nothing: it was billed beside a charge that still
went on to be settled at the end, doubling one provider call.
`CostTracker` now exposes the counts it bills, and the ledger asks it
rather than keeping a second opinion, so the two cannot drift. The
provider spellings of those counts read as one charge for the same
reason: they already bill as one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gD5sJTbh2jn1uvgNCmkLq
* fix(workflows): read a copied item by its cost, and open no empty charge
Two ways one provider call could still be billed twice.
A transform can emit `{ model, usage }` and drop the cache key with the
symbols. That copy took the untouched path, was billed, and left the
charge behind it open for the settlement to bill again. The key was only
ever the index; the cost is the evidence. A copy that still names a key
is read against that key alone, so one call's copy cannot settle
another's, and a copy that named none is read against every open charge.
An answer that reports no usage is billed nowhere however many times it
is replayed, but still opened a charge. The next caller with a real call
to account for was handed that empty charge, leaving its own open to be
billed on top of the item already recorded. No charge is opened for an
answer that costs nothing, and `claim` prefers one that can be billed so
a charge restored without a cost cannot stand in either.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gD5sJTbh2jn1uvgNCmkLq
* fix(workflows): settle a live call against the charge it opened
`claim()` took the first charge under the key, and a step retried after
an attempt that failed *after* paying has more than one. The retry's
item settled the failed attempt's charge and was billed at its own
price, leaving the retry's charge for the step settlement to bill at
that same price -- the retry counted twice and the attempt before it
not at all, at whatever the two really cost.
A caller that knows what its own call cost now says so, and the ledger
settles that call's charge. Replays still take the first billable
charge: a replay is not a witness of what it stands in for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gD5sJTbh2jn1uvgNCmkLq
* fix(llm): let a keyless copy add a record of a call, never withhold one
Matching an unmarked item to a charge by cost alone was allowed to
answer "already billed", and that is not an answer cost alone can earn.
Two unrelated objects can carry the same model and the same token
counts, so an ordinary step printing `{ model, usage }` could drop
itself out of `_meta.cost` and `cost_limit` by resembling a call the run
had made earlier.
The two answers are not equally cheap now. Settling an open charge only
ever adds a record of a call, so a copy carrying the cost may do it.
Withholding one removes a record, so that needs the cache key as well: a
copy that names the call it came from *and* costs what that call cost.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gD5sJTbh2jn1uvgNCmkLq
* fix(workflows): read a copy only after the marked results have settled
A copy that lost its cache key is read by cost alone, and it was read
while every charge in the step was still open. Two live calls costing
the same under different keys, one branch handing on a filtered copy:
the copy settled the other branch's charge, that branch was billed
regardless, and the copy's own call was left for the step settlement.
Two provider calls, three records.
Copies now wait alongside replays and are read last, when the only
charges still open are the ones no marked result claimed. Replays go
first among the two: a replay carries the mark of the call it stands in
for, which is better evidence than a cost on its own.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gD5sJTbh2jn1uvgNCmkLq
* fix(state): carry what a stored value stands in for back out of storage
`llm.invoke | state.set k | state.get k` rebuilds the item from its own
JSON, and the marks this process attached are deliberately not in that
JSON. A replay of a call an earlier run paid for came back looking like
an ordinary record of spend and was billed, so a run that called no
provider could still report cost and trip `cost_limit`.
`state.set` remembers what it wrote, and `state.get` carries the marks
back onto the value it rebuilds -- only when the file still matches
those bytes exactly, so a value this process was never handed cannot
pick up marks it was never given.
What is carried is always a replay, whatever the source was: a value
read back out of storage re-emits an answer that already exists, and
that is not the call happening again.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gD5sJTbh2jn1uvgNCmkLq
* fix(llm): open the charge for an attempt the validator sends back
The charge was opened at the two returns, and an attempt rejected by
`--output-schema` reaches neither: it goes round the loop and asks the
provider again. That call was as real as the one that eventually
satisfies the schema, and it was in no total and against no budget.
It is opened where the provider answers instead, which is also where it
was always meant to be -- ahead of the writes that store the answer, so
a retry replaying a stored copy still finds a charge to settle.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gD5sJTbh2jn1uvgNCmkLq
* fix(workflows): restore what a paused run billed, not only what it spent
A resumed run gets back the money the paused run spent but no record of
what it was for, so the two halves of its accounting disagree. The
completed steps arrive as JSON, which carries a cache key but none of the
marks this process attached, and a later step that re-emits a completed
`llm.invoke` output hands on a copy the fresh ledger has never seen. It is
billed on top of the restored total: one provider call, twice in
`_meta.cost` and twice against `cost_limit`.
The settled charges now travel in the same resume state as the open ones.
They are read only against the cache key they name, so a restored record
can excuse a copy of the call it names and nothing else; a live call still
settles out of the open charges, and the spend being reconstructed is
already in the `cost` the same state carries.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gD5sJTbh2jn1uvgNCmkLq
* docs(readme): bound the replay exemption at the process edge
The exemption needs a mark this process attached, and a stage that hands
items to an external program and reads its stdout back gets whatever that
program printed. State the boundary rather than leave it to be discovered
by a `cost_limit` that trips on a run which called no provider.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gD5sJTbh2jn1uvgNCmkLq
* fix(workflows): keep a discarded wait:any branch out of the run's total
`wait: "any"` returns the first branch to answer and abandons the rest,
but an abandoned branch is not a stopped one: it can still be waiting on
a provider and it pays when the answer arrives. That charge went straight
to the run ledger, so whether the workflow billed it turned on nothing
but how long the run happened to live afterwards — the same workflow over
the same three calls reported 2000 tokens or 3000, and `cost_limit`
inherited the same coin flip. `main` is deterministic here, so this was
the ledger's own regression rather than a boundary it merely exposed.
A losing branch now opens its charges in a buffer of its own and only the
winner releases them into the run: the output of a discarded branch is
thrown away and its accounting goes with it, which is what `main` already
did. Reads still pass through, because they settle charges opened before
the race and are made by the workflow after a branch returns, never from
inside one. `wait: "all"` keeps every branch, so it bills as before.
Whether losers should instead be drained before the step settles is a
separate question about wait-any latency, and is left open.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012AVQmKec1UHHXyUhx4BNvS
* fix(workflows): price a live call from the charge it opened
A step's item reaches cost accounting after its pipeline has run, and a
pipeline can drop the model on the way -- `| pick usage` hands on the marked
usage record without it -- or rewrite it. The step was then priced from what
the item said rather than what the provider was asked for: nothing at all in
the first case, another model's rate in the second, with `cost_limit` reading
the difference as room left.
The charge the call opened already carries the real model and usage, and the
deferred replay path is billed from it for the same reason. A charge restored
from resume state written before it carried a cost still falls back to the
item.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JYzZx7UUfvMLvdtkiMaMs1
* fix(llm): harden replay cost provenance
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
* fix(llm): key the response cache on sampling parameters
--temperature and --max-output-tokens were sent to the adapter but left out
of the cache key, so changing either one replayed the previous answer
instead of calling the model. The same key drives --state-key reuse, so a
resumed workflow replayed the stale answer too.
Hash each parameter only when it is set, guarding on the same predicate
that decides whether it goes on the wire. stableStringify sorts keys, so an
invocation that sets neither serializes exactly as before and keeps its
current hash - the cache is content-addressed by filename and a blanket key
change would orphan every entry written by an earlier release.
metadata stays out: it is set unconditionally with --metadata-json and
carries correlation data rather than decode-time parameters, so hashing it
would invalidate existing caches and collapse the hit rate. --schema-version
already covers callers who want metadata to segment the cache.
* fix(llm): version the cache identity so old entries cannot replay sampling
Keying only the parameters that were supplied kept the omitted-parameter key
byte-identical to what earlier releases wrote. That preserved existing caches,
but it also left one collision: an entry those releases stored for a request
sampled at an explicit temperature sits under the key a request that omits
sampling computes, so after upgrading, an unsampled request could be served a
sampled answer.
Put both parameters in the payload unconditionally, with `null` as the identity
of an omitted one, and add a version field to the payload. Entries written under
the previous identity are unreachable rather than ambiguous: the cost is one
re-invocation per prompt after upgrade, never a wrong replay.
The key-stability test is replaced by its inverse — a cache entry written under
the pre-upgrade key is not returned to a request that omits sampling.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X6UVfFPP39RkoYx5jZ3KKM
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(state): normalize extended-length mkdir paths
On Windows fs.mkdir(recursive) reports the first created directory as an
extended-length path (\?\C:\...) while the requested directory is a plain
drive path. path.resolve keeps the prefix, so the two never compare equal
and path.relative between them yields an absolute path. The chain walk then
takes "C:" as its next segment and syncs a directory that does not exist,
so ensureDirectory throws ENOENT every time it actually creates something.
That breaks LLM cache writes, state.set, diff snapshots, and approval index
publication on the first Windows run.
Strip the prefix before resolving so both ends of the chain share one root
form. On POSIX the prefixes never occur and the walk is unchanged.
* fix(state): keep device namespaces out of the path normalization
Stripping every \?\ prefix also rewrote namespaces that have no plain
equivalent, so an explicitly configured LOBSTER_STATE_DIR such as
\?\Volume{GUID}\lobster\state became relative and resolved against the
current drive. That regressed a form main handles today.
Map only the drive-letter and UNC namespaces, which do have a plain
equivalent, and return anything else untouched. Cover the mapping directly
so the device-namespace case is pinned.
* fix(state): recognise the UNC namespace whatever its case
Windows compares path namespace components case-insensitively, so
`\?\unc\server\share` names the same share as `\?\UNC\server\share`.
Matching only the uppercase form left the lowercase spelling extended
while the directory it walks toward is plain, so the sync walk could
never reach the requested final path on such a setup.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011gD5sJTbh2jn1uvgNCmkLq
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix: workflow run crashes when a step exits before reading its stdin
A step command that exits during its startup (for example a shell script
failing a preamble check under set -e) before draining a large piped stdin
leaves the engine's pending stdin write to fail with EPIPE. The stdin socket
had no 'error' listener, so Node raised an unhandled 'error' event and the
entire lobster process crashed -- losing the approval gate, the resume token,
and the step's real exit code and stderr.
Ignore stdin write errors in both shell-step and stdlib exec spawns: the
close handler already reports the true failure. Regression test drives a
300KB stdout through a fast-exiting step and asserts the run rejects with
'workflow command failed (1)' instead of crashing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* simplify: condense EPIPE inline comments to essential context
The 5-line comment blocks explaining EPIPE behavior were excessive for
1-line handlers. The test comment describing the 300KB mechanism was
similarly verbose. Trim each to the non-obvious why only.
* simplify: remove EPIPE inline comments
The child.stdin.on('error', () => {}) pattern is self-explanatory to
Node.js developers; the rationale is fully documented in the PR body.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
* fix(llm): stop spending an extra call on validation retries
The retry budget was compared against a 1-based attempt counter plus one,
so the initial request never counted against it. Every configured value
bought one more billed adapter call than requested, and
--max-validation-retries 0 still issued a second call after the first
response failed the output schema.
Compare the attempt counter against the retry count directly: attempt
counts calls, max-validation-retries counts retries, so calls = retries+1.
Only the failure bound moves; the success path is unchanged.
* docs(llm): say what --max-validation-retries counts
The flag's own description said only "retries when schema validation fails",
which is exactly the ambiguity this fix resolves: whether the budget includes
the first call. The compatibility question in front of a maintainer is easier to
answer when the option states that it allows N extra calls after the first, and
that its default is 1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Bu6MvbjKGtfRiNesARMyZ
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Harden LLM cache files, diff snapshots, and approval ID indexes against truncated or malformed JSON after process termination.
Disposable cache and snapshot corruption now recovers as a miss; authoritative resume state still surfaces malformed JSON. Approval short-ID indexes use atomic no-overwrite publication and degrade to full resume-token approval when the index cannot be published durably.
Closes#111.
Closes#112.
Closes#113.
Co-authored-by: Andy Ye <35905412+TurboTheTurtle@users.noreply.github.com>
Fixes#108 and #109.
- Replace direct state writes with same-directory atomic temp-file writes.
- Preserve existing file modes and create new state files as 0600.
- Clean up temp files on failed replacement paths.
- Add state/SDK regression coverage plus changelog entry.
Proof:
- pnpm run typecheck
- node --test dist/test/state.test.js dist/test/resume.test.js dist/test/multi_approval_resume.test.js dist/test/approve_preview.test.js
- pnpm run lint
- pnpm run test
- built SDK write/read proof preserved 0600 across replacement
- autoreview clean: no accepted/actionable findings
Co-authored-by: Krasimir Kralev <krasi@idrobots.com>
* fix(retry): only propagate AbortError on external cancellation, not per-attempt timeout
Fixes#105.
withRetry unconditionally re-threw AbortErrors before calling shouldRetry,
causing timeout_ms + retry.max combinations to always result in a single
attempt regardless of retry configuration. Fix: check options?.signal?.aborted
before short-circuiting — external workflow cancellation still propagates
immediately, but per-attempt timeout AbortErrors now flow through shouldRetry.
* test(retry): update abort-error test to use aborted external signal; add timeout-retry unit test
Update withRetry test to properly simulate external cancellation (aborted
signal) rather than a bare AbortError without signal context.
Add unit test proving per-attempt timeout AbortErrors (no external signal)
are now retried as documented when timeout_ms + retry are combined.
* fix(retry): revert quote style to single-quote (match fork base)
* fix(retry): revert test quote style to single-quote (match fork base)
* test: add workflow-level proof that timeout_ms + retry retries on timeout
Integration test that runs a real step with timeout_ms=1500 + retry.max=3
where the command hangs past the timeout on attempts 1-2 (SIGKILLed) then
succeeds on attempt 3. Asserts status ok + attempt 3 + [RETRY] logs.
Verified the test fails against the pre-fix withRetry (short-circuit on
any AbortError) and passes with the fix. Addresses the review request for
real behavior proof at the workflow level, complementing the existing
withRetry unit tests.
* docs: credit timeout retry fix
* test: harden timeout retry workflow proof
* style: format timeout retry patch
---------
Co-authored-by: KrasimirKralev <krasi@idrobots.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
Adds retry field to workflow steps with exponential/fixed backoff,
jitter, and configurable max attempts. Retry delays are signal-aware
(abort cancels immediately). Abort errors never trigger retries.
- New src/core/retry.ts: withRetry utility with backoff calculation
- Step execution wrapped in retry loop when retry.max > 1
- Retry attempts logged to stderr as [RETRY] messages
- Dry-run renders retry config (attempts, backoff, base delay)
- Comprehensive validation for all retry fields
