StreamPilot prototype · CLI-first · sandbox-only
Error-triggered lesson recall
Lessons should fire when the failure happens — not at the start of the task.
recall pipes a failing command's stderr through your project's
ContextStream memory and hands back the fix you already captured, at the exact
moment you need it. If nothing matches, the failure is packaged for one-tap
capture so it's caught next time. Nothing to configure — it runs in the sandbox
with a single command.
Demo harness
Five failures, one seed command
The harness seeds a small sandbox lesson set (idempotent — re-running never duplicates), then pushes five realistic failures through the full pipeline. Every sample prints the same anatomy: raw excerpt, stable signature, scoped search, result, confidence, and either the captured fix or a capture payload.
Interactive matcher
Pipe in an error of your own
Paste any stderr below — or load one of the fixtures — and run it through the same normalizer and matcher the harness uses. Toggle the degraded mode to see how the CLI behaves when ContextStream search is unreachable: it falls back to local keyword-only matching over the seeds and says so, instead of pretending the lesson doesn't exist. Degraded results are capped at the tentative tier — keyword overlap alone never claims high confidence.
streampilot-demo-app
Component anatomy
Four small parts, one honest pipeline
Everything ships as a standalone CLI in the sandbox. ContextStream search is called through a thin client with a local mock fallback, so the demo runs offline and the production MCP server is never touched.
Error normalizer
An ordered regex pipeline turns volatile noise into inspectable placeholders — deterministic, and idempotent, so normalizing a signature twice changes nothing.
Kept intact:
Same failure family, different machine → identical signature. Different failure kinds never collapse into one.
Lesson matcher
One hybrid query against ContextStream lesson & incident search, always constrained to the active project. At most three candidates come back, ranked by a blended score:
A weak best-candidate never suppresses the capture fallback, and is never presented as an authoritative fix.
No-match fallback
When nothing clears the threshold, the failure is packaged for capture on the spot: a short neutral title built from the error class, the stable signature, a bounded raw excerpt, the project id, and source: error-triggered-recall.
The payload is printed and written to out/pending-lessons/, shaped for one-tap capture by the real save-lesson tool. It is never saved automatically — capture stays a deliberate act.
Demo harness & exit codes
Subcommands: seed · match · demo · replay. Human output stays readable at 80 columns; --json emits the same information machine-readably. Search failure is a distinct state — it is never mislabeled as “no match.”
exit 0useful match found
exit 2invalid or empty input
exit 3no match — payload emitted
exit 4search unavailable — degraded keyword-only
exit 5configuration failure (unknown project)
Sandbox seed set
Eight seeded entries, seven in scope
recall seed loads these synthetic fixtures into the sandbox index —
re-running skips existing ids. One entry deliberately lives in another project
to prove scope discipline: it would match sample five perfectly, and the
matcher must refuse to use it.
README · replay study
Measuring recall against 30 days of real sessions
The matcher is only worth shipping if it catches failures that actually repeat. The README documents a read-only replay over ~30 days of session transcripts that answers one question: when a failure recurred, did recall surface the lesson we had already captured?
- Export transcripts locallyReference ~30 days of session transcripts in a local directory. The replay only ever reads them.
- Detect error eventsAn error event is a non-zero exit paired with a stderr block; excerpts are bounded to 400 characters before processing.
- Walk chronologically, per projectEvents are processed in timestamp order within each project so history unfolds exactly as it did live.
- Time-fence every queryOnly lessons captured before the event's timestamp may participate — a lesson from the future can never inflate the score.
- Record outcomesFor each event: top-1 hit, top-3 hit, or no-match, with confidence and candidate ids.
- Compare against the observed fixThe suggested fix is checked against the fix actually applied later in the same session — that's the relevance judgment.
- Aggregate and discardThe report keeps counts and distributions only; raw transcript content is never persisted.
Future lessons cannot leak into earlier events, and every query carries the project id — replay measures what the matcher would really have known at that moment.
Primary metric
Repeat-failure hit rate
÷ repeats where a prior captured fix existed
- top-1 hit rate
- false-positive rate (confident but irrelevant)
- no-match rate
- confidence distribution per tier
- normalization collision rate
- project-scope leakage — must be zero
# read-only replay over local transcripts $ recall replay ./transcripts \ --project streampilot-demo-app --days 30 # machine-readable aggregate report $ recall replay ./transcripts --days 30 \ --json > out/replay-report.json
streampilot-demo-app/ ├─ bin/recall.js seed · match · demo · replay ├─ src/normalizer.js signature builder ├─ src/matcher.js hybrid client + blend scoring ├─ src/fallback.js save-lesson payload generator ├─ src/render.js 80-column terminal output ├─ seeds/lessons.json 8 sandbox lessons, fixes included ├─ samples/*.txt 5 fixture errors └─ out/pending-lessons/ payloads awaiting capture
{
"window_days": 30,
"error_events": —,
"repeat_failures": —,
"repeat_failure_hit_rate": —,
"top1_hit_rate": —,
"false_positive_rate": —,
"no_match_rate": —,
"collision_rate": —,
"scope_leakage": 0 expected
}