Introducing stromboli: an open-source AI SRE you host yourself
Nov 16, 2026 | by Thibault Le Ouay Ducasse | [engineering]

We have spent three years telling you when your service is down.
What happens next was always your problem: an alert lands in Slack, someone wakes up, opens five tabs (logs, metrics, recent deploys, the dependency graph in their head, the runbook nobody updated since 2024), and starts piecing together what broke.
We do the exact same thing when openstatus pages us. Our synthetic checkers run
on Fly.io, our background workers and queues run on Google Cloud, our dashboard
sits on Vercel, and our logs live in Axiom and Cloud Logging. Every incident
starts with the same ten-minute ritual of running fly status, diffing recent
commits, and checking whether a downstream queue backed up before anyone knows
what action to take.
Over the past year we built small, single-purpose agents around incident communication: a Slack agent to draft status reports in-thread and an agent webhook to translate monitor failures into human-readable updates. Useful, but they only helped after someone figured out what was going on.
So we started a side project to tackle those first ten minutes. It's called stromboli: an open-source AI SRE agent we run on our own production, and today we're releasing it under the MIT license on GitHub.
It is deliberately a standalone project, separate from the openstatus monorepo. Nobody really knows yet what an AI SRE that engineers actually trust looks like—us included—and stromboli is our lab for finding out on real incidents, in the open.
What happens when an alert fires
alert ──▶ 1. match ──▶ 2. brief ──▶ 3. mitigate ──▶ 4. investigate ──▶ 5. report
& dedupe (no LLM) (posted now) (read-only tools) (Slack + API)
When an alert hits a stromboli webhook (from openstatus, Alertmanager, Grafana, or a generic JSON payload), it runs a five-stage pipeline:
- Match and deduplicate. stromboli maps the alert to a service in its
catalog (by openstatus monitor ID or
service/app/joblabels). If Alertmanager or your monitor fires five times for the same outage, repeats merge into the running investigation instead of burning tokens five times. Once the alert resolves—or a mitigation action is applied—deduplication resets so a regression is treated as a fresh incident. - Build a deterministic brief (zero LLM tokens). Before any model is
called, Go code queries your stack in parallel and builds the exact snapshot
an on-call engineer pulls by hand:
- the service, its upstream and downstream dependencies, and its transitive blast radius
- log error patterns in the incident window diffed against a preceding
baseline window of the same length (so you see
"new error pattern: 340 vs 0 in baseline", not 340 raw JSON lines) - per-instance metrics (CPU, throttling, memory, OOM kills, 5xx rate, p95 latency, Pub/Sub or Cloud Tasks backlog) compared against the same baseline
- recent deploys and commits for both the failing service and its dependencies
- attached runbooks, similar past investigations, and—on Fly.io—live machine states with exact mitigation commands
- Propose a mitigation first. Within a handful of tool calls, the triage model commits to a concrete action to stop user impact and posts it to Slack immediately.
- Investigate the root cause. A second phase forms hypotheses and tests them against your logs, metrics, and commit diffs using read-only tools.
- Publish a structured report. Likely cause, confidence, supporting evidence, hypotheses that were ruled out, whether the initial mitigation still holds, and the long-term fix—edited straight into the Slack message with the full audit trail in the thread.
Because the brief in step 2 is pure Go code, it doesn't need an LLM to be
useful. You can run stromboli brief api --since 1h from your terminal at any
time and get the full cross-system diff in seconds, for free.
Mitigate first, investigate second
When an alert fires, two clocks start ticking: how long users are hurting, and how long until someone understands why.
Most AI SRE tools optimize for the second clock. They spend three minutes and twenty tool calls hunting for a root cause in a commit diff before outputting a single actionable sentence. That makes the on-call engineer wait for the slowest, least certain step while production is still bleeding.
Experienced incident responders work the other way around: stop the bleeding first, debug second. If 5xx errors spiked ninety seconds after a deploy, roll it back—whether the bug is an unindexed query or a bad env var can wait until traffic is healthy.
stromboli enforces this split in its state machine:
- A closed set of mitigations. Inspired by generic mitigations, Phase 1 (triage) ends as soon as the model
commits to one action from a fixed enum:
rollback,restart,scale_up,drain,disable_feature,block_traffic,failover,degrade, ornone. Every mitigation must include the target, the exact command to run, how to verify it worked, how to undo it, and a risk level (safe,reversible, ordestructive). noneis a first-class answer. When there is no clear user impact or no safe action to take, sayingnoneis much better than inventing a placebo command.- No wandering. After six tool calls (
STROMBOLI_MITIGATION_TOOL_CALLS) without a mitigation, stromboli injects a prompt reminding the model that users may still be affected and forces it to commit. Submitting a root-cause report without first recording a mitigation is rejected. - Self-correction in Phase 2. Early mitigations are based on symptoms, so they can be wrong. If the deeper investigation proves a restart won't help because a downstream database is saturated or a config push broke parsing, the model publishes a revised mitigation to Slack and automatically supersedes any pending action.
Because the two phases optimize for different things, you can point them at different models. A fast model handles triage in seconds; a frontier reasoning model takes over the exact same conversation history for root-cause analysis:
STROMBOLI_TRIAGE_MODEL=anthropic:claude-haiku-5-5
STROMBOLI_TRIAGE_EFFORT=low
STROMBOLI_INVESTIGATE_MODEL=openai:gpt-5
STROMBOLI_INVESTIGATE_EFFORT=high
Safety by construction, not prompt engineering
If you run infrastructure for a living, your first reaction to "point an LLM at production during an outage" is probably absolutely not. Between hallucinated flags and prompt injections hiding in user-controlled log lines, asking a model to "please be careful" in a system prompt is not a security model.
stromboli doesn't rely on the model behaving well. It removes the capability to misbehave at the architecture level:
- Every LLM tool is strictly read-only. The model has no tool to run shell commands, deploy, restart, or mutate state. Its only outputs are structured mitigation and report objects. Log lines and commit messages are treated as untrusted data—so even if an attacker crafts a prompt injection in an HTTP header that lands in Loki, the worst it can do is confuse a text summary.
- Connectors use read-only credentials. A
fly tokens create readonlyorg token, GCProles/logging.viewerandroles/monitoring.viewer, a read-only GitHub token. Even a bug in stromboli's own Go code cannot mutate resources with those keys. - stromboli is the MCP client, not the model. When a provider hosts an
official remote MCP server (Google Cloud Logging and Monitoring, GitHub,
Axiom, Vercel), stromboli connects to it from Go, enforces a strict
per-connector tool allow-list, and verifies that every allowed tool carries
the
readOnlyHintannotation before starting. Anything else (deploy,delete,exec) is refused at the client boundary. - No local CLI wrappers masquerading as MCP. Where a provider's MCP server
is just a local wrapper around a privileged CLI—for instance,
flyctl mcp serverexposesdestroy,exec, andsecretswithout read-only annotations—stromboli skips MCP entirely and talks directly to the provider's HTTP API and managed Prometheus. - Pre-aggregation and redaction before the context window. Raw logs and diffs pass through a size cap (24 KiB per call) and a regex redaction pipeline that strips Bearer tokens, JWTs, cloud keys, Slack tokens, URL credentials, and email addresses before a single byte leaves your server for the LLM API.
The one opt-in exception: human-approved Fly.io machine actions
By default, every mitigation is a command for a human to copy-paste. On Fly.io,
where we run our own checkers, we added an opt-in path
(STROMBOLI_ACTIONS_ENABLED=true) for the mitigations you run half-asleep at 3
AM: restart, clone, cordon, and uncordon.
Even with actions enabled, the model still has no execution tool. It can only attach a proposed action to its mitigation. Before anything touches the Fly Machines API:
- Actions require a separate
fly_machinesconnector and an explicit per-service allow-list in the catalog (stopandstartare excluded by default). - stromboli validates the target machine IDs against the Fly app twice—when proposed and when approved—caps batch size at 10, and refuses to clone machines with attached volumes (since a clone starts with an empty volume).
- A named human must click Apply in Slack (verified via Slack's
HMAC-SHA256 signature and restricted to
SLACK_APPROVERSuser IDs) or runstromboli actions approvebefore the 30-minute TTL expires. - Approval claims the action atomically so double-clicks never execute twice.
stromboli runs the action one machine at a time, waits up to 60 seconds
for each machine to reach
started, and halts immediately on the first machine that fails to come back—so a bad rolling restart can never take down an entire pool.
No data ingestion, no second observability tax
Many observability and AI incident tools start by asking you to ship your logs and metrics into their storage engine.
You already pay for a log store and a metrics backend. stromboli doesn't become a second one: there is no ingestion pipeline, no ClickHouse cluster to size, and no duplicate copy of PII-heavy production logs sitting in another database.
It queries your existing backends on demand for the incident window:
- Logs: Grafana Loki, Axiom, AWS CloudWatch Logs, Google Cloud Logging, Vercel
- Metrics: Prometheus (and anything speaking the Prometheus HTTP API: Mimir, Thanos, VictoriaMetrics, Grafana Cloud), Fly.io Managed Prometheus, Google Cloud Monitoring
- Deploys & commits: GitHub Actions / Deployments / commits, Vercel
All stromboli stores in Postgres is the service catalog, encrypted connector credentials (AES-256-GCM), and the aggregated, redacted evidence each investigation actually used—so your incident audit trail stays intact even after your log store's retention window expires.
A small, GitOps-friendly service catalog
An alert only says "api is returning 5xx". To investigate without guessing
Loki labels or PromQL selectors, stromboli keeps a lightweight service catalog.
You don't have to write it from scratch. Point catalog discover at your log
or metric connectors and stromboli inspects your active labels, Cloud Run
services, Pub/Sub subscriptions, or Fly apps to generate the initial YAML:
stromboli catalog discover loki > services.yaml
Then add what telemetry can't tell you—owners, dependencies, and runbooks:
- name: api
description: Public REST API
owners: ["@team-backend"]
tier: 1
depends_on: [db-proxy, redis]
logs:
source: loki
selector: '{service_name="api"}'
metrics:
source: fly
selectors: ['app="api"']
deploys:
source: github
repo: acme/api
workflow: deploy.yml
runbooks: [runbooks/api-5xx.md]
monitors: ["openstatus:1234"]
runtime: fly.io
Keep services.yaml and your markdown runbooks in git, review changes in PRs,
and sync them from CI with stromboli catalog import services.yaml --runbooks runbooks/ --prune --yes. Every write validates dependency cycles, connector
types, and missing runbooks before it lands.
Why an AI SRE has to be open source and self-hosted
Every AI SRE we looked at is closed-source SaaS. Datadog has Bits AI SRE, the other observability incumbents are bundling their own, and a wave of startups will happily take a webhook and your cloud credentials. You send an alert into a black box and get back a paragraph in Slack—with no way to read the system prompt, inspect the tool definitions, audit the redaction rules, or understand why the agent fixated on an unrelated warning.
We think that's backwards for software that sits next to production during an outage.
First, debuggability: when an agent misdiagnoses an incident, you should be
able to read the exact system prompt, check the tool schemas, inspect the
ordered step log, and open a PR to fix it. In stromboli, even our architecture
trade-offs live in docs/adr/.
Second, data sovereignty: an AI SRE needs read access to your production logs, metrics, commit diffs, and internal runbooks. Handing those credentials to a multi-tenant third party is a tough sell for any security-conscious team. stromboli is designed as one single-tenant instance per organization. Your credentials stay in your Postgres encrypted at rest, and nothing leaves your network except the redacted prompt sent to the LLM provider you choose—Anthropic, OpenAI, Gemini, DeepSeek, OpenRouter, Vercel AI Gateway, or Ollama if not a single token can leave your VPC.
Self-hosting in five minutes (really)
We'll be honest: self-hosting the full openstatus platform is a project. Turso, Tinybird, Go checkers, several apps, and background queues—we made it much smoother, but it's still a weekend commitment.
We designed stromboli to be the exact opposite: one static Go binary and a
Postgres database. The same binary runs both the ConnectRPC server and the
clig.dev-compliant CLI (--json output, TTY detection,
hidden secret prompts, grpcurl reflection out of the box), and database
migrations run automatically on startup.
cp .env.example .env # STROMBOLI_SECRET_KEY, STROMBOLI_API_TOKEN, and an LLM key
docker compose up -d # stromboli + Postgres on http://localhost:8080
Install the CLI via Homebrew (or grab a binary from GitHub Releases), wire up your connectors, and run your first investigation:
brew install openstatusHQ/stromboli/stromboli
stromboli connector add loki loki url=http://loki:3100
stromboli connector add github github token
stromboli catalog discover loki
stromboli brief api --since 1h
stromboli investigate api --description "5xx spike on /v1/monitors"
If you just want to watch it work without wiring up production first, the repo
ships with a local dev stack (docker compose -f docker-compose.dev.yml up -d)
that spins up Postgres, Loki, and a demo service that triggers a real failure
after three minutes. There are also one-command deploy scripts for Fly.io and
Railway.
Built to pair with openstatus, or run completely on its own
stromboli has its own repository, its own release cycle, and zero hard
dependency on openstatus. If your stack runs on Prometheus, Alertmanager, and
Grafana, point them at /webhooks/alertmanager or /webhooks/grafana and you
never need an openstatus account.
If you do use openstatus, the two fit together naturally: point a webhook
notification at POST /webhooks/openstatus, map your monitor ID in
services.yaml, and the next failed check automatically triggers a brief, a
mitigation in Slack, and a root-cause report—and resets deduplication as soon
as the monitor recovers.
stromboli also exposes an MCP server (stromboli mcp over stdio or /mcp
over HTTP) and a Claude Code skill. During an incident, you can ask Claude Code
or Cursor right from your editor what stromboli saw on api, pull the
deterministic brief, or inspect the evidence behind a mitigation before
shipping the code fix.
Try it and break it
stromboli is MIT licensed and available today at v0.1. It is early, and it is unapologetically shaped by the stack we run in production (Fly.io, Google Cloud, Vercel, Axiom, Loki, Prometheus, GitHub).
If you run a different log store, orchestrator, or alerting pipeline—or if you throw a gnarly incident at it and it gets the hypothesis wrong—we want to hear about it:
- Website: stromboli.dev
- GitHub: github.com/openstatusHQ/stromboli (and the ADRs behind the design)
- Community: Discord for feedback, connector requests, and war stories from production.