openstatus logoDocsPricingDashboard

Introducing stromboli: an open-source AI SRE you host yourself

Nov 16, 2026 | by Thibault Le Ouay Ducasse | [engineering]

Introducing stromboli: an open-source AI SRE you host yourself

We have spent three years telling you when your service is down.

What happens next was always your problem: an alert lands in Slack, someone wakes up, opens five tabs (logs, metrics, recent deploys, the dependency graph in their head, the runbook nobody updated since 2024), and starts piecing together what broke.

We do the exact same thing when openstatus pages us. Our synthetic checkers run on Fly.io, our background workers and queues run on Google Cloud, our dashboard sits on Vercel, and our logs live in Axiom and Cloud Logging. Every incident starts with the same ten-minute ritual of running fly status, diffing recent commits, and checking whether a downstream queue backed up before anyone knows what action to take.

Over the past year we built small, single-purpose agents around incident communication: a Slack agent to draft status reports in-thread and an agent webhook to translate monitor failures into human-readable updates. Useful, but they only helped after someone figured out what was going on.

So we started a side project to tackle those first ten minutes. It's called stromboli: an open-source AI SRE agent we run on our own production, and today we're releasing it under the MIT license on GitHub.

It is deliberately a standalone project, separate from the openstatus monorepo. Nobody really knows yet what an AI SRE that engineers actually trust looks like—us included—and stromboli is our lab for finding out on real incidents, in the open.

What happens when an alert fires

alert ──▶ 1. match ──▶ 2. brief ──▶ 3. mitigate ──▶ 4. investigate ──▶ 5. report
           & dedupe     (no LLM)     (posted now)    (read-only tools)   (Slack + API)

When an alert hits a stromboli webhook (from openstatus, Alertmanager, Grafana, or a generic JSON payload), it runs a five-stage pipeline:

  1. Match and deduplicate. stromboli maps the alert to a service in its catalog (by openstatus monitor ID or service / app / job labels). If Alertmanager or your monitor fires five times for the same outage, repeats merge into the running investigation instead of burning tokens five times. Once the alert resolves—or a mitigation action is applied—deduplication resets so a regression is treated as a fresh incident.
  2. Build a deterministic brief (zero LLM tokens). Before any model is called, Go code queries your stack in parallel and builds the exact snapshot an on-call engineer pulls by hand:
    • the service, its upstream and downstream dependencies, and its transitive blast radius
    • log error patterns in the incident window diffed against a preceding baseline window of the same length (so you see "new error pattern: 340 vs 0 in baseline", not 340 raw JSON lines)
    • per-instance metrics (CPU, throttling, memory, OOM kills, 5xx rate, p95 latency, Pub/Sub or Cloud Tasks backlog) compared against the same baseline
    • recent deploys and commits for both the failing service and its dependencies
    • attached runbooks, similar past investigations, and—on Fly.io—live machine states with exact mitigation commands
  3. Propose a mitigation first. Within a handful of tool calls, the triage model commits to a concrete action to stop user impact and posts it to Slack immediately.
  4. Investigate the root cause. A second phase forms hypotheses and tests them against your logs, metrics, and commit diffs using read-only tools.
  5. Publish a structured report. Likely cause, confidence, supporting evidence, hypotheses that were ruled out, whether the initial mitigation still holds, and the long-term fix—edited straight into the Slack message with the full audit trail in the thread.

Because the brief in step 2 is pure Go code, it doesn't need an LLM to be useful. You can run stromboli brief api --since 1h from your terminal at any time and get the full cross-system diff in seconds, for free.

Mitigate first, investigate second

When an alert fires, two clocks start ticking: how long users are hurting, and how long until someone understands why.

Most AI SRE tools optimize for the second clock. They spend three minutes and twenty tool calls hunting for a root cause in a commit diff before outputting a single actionable sentence. That makes the on-call engineer wait for the slowest, least certain step while production is still bleeding.

Experienced incident responders work the other way around: stop the bleeding first, debug second. If 5xx errors spiked ninety seconds after a deploy, roll it back—whether the bug is an unindexed query or a bad env var can wait until traffic is healthy.

stromboli enforces this split in its state machine:

  • A closed set of mitigations. Inspired by generic mitigations, Phase 1 (triage) ends as soon as the model commits to one action from a fixed enum: rollback, restart, scale_up, drain, disable_feature, block_traffic, failover, degrade, or none. Every mitigation must include the target, the exact command to run, how to verify it worked, how to undo it, and a risk level (safe, reversible, or destructive).
  • none is a first-class answer. When there is no clear user impact or no safe action to take, saying none is much better than inventing a placebo command.
  • No wandering. After six tool calls (STROMBOLI_MITIGATION_TOOL_CALLS) without a mitigation, stromboli injects a prompt reminding the model that users may still be affected and forces it to commit. Submitting a root-cause report without first recording a mitigation is rejected.
  • Self-correction in Phase 2. Early mitigations are based on symptoms, so they can be wrong. If the deeper investigation proves a restart won't help because a downstream database is saturated or a config push broke parsing, the model publishes a revised mitigation to Slack and automatically supersedes any pending action.

Because the two phases optimize for different things, you can point them at different models. A fast model handles triage in seconds; a frontier reasoning model takes over the exact same conversation history for root-cause analysis:

STROMBOLI_TRIAGE_MODEL=anthropic:claude-haiku-5-5
STROMBOLI_TRIAGE_EFFORT=low
STROMBOLI_INVESTIGATE_MODEL=openai:gpt-5
STROMBOLI_INVESTIGATE_EFFORT=high

Safety by construction, not prompt engineering

If you run infrastructure for a living, your first reaction to "point an LLM at production during an outage" is probably absolutely not. Between hallucinated flags and prompt injections hiding in user-controlled log lines, asking a model to "please be careful" in a system prompt is not a security model.

stromboli doesn't rely on the model behaving well. It removes the capability to misbehave at the architecture level:

  • Every LLM tool is strictly read-only. The model has no tool to run shell commands, deploy, restart, or mutate state. Its only outputs are structured mitigation and report objects. Log lines and commit messages are treated as untrusted data—so even if an attacker crafts a prompt injection in an HTTP header that lands in Loki, the worst it can do is confuse a text summary.
  • Connectors use read-only credentials. A fly tokens create readonly org token, GCP roles/logging.viewer and roles/monitoring.viewer, a read-only GitHub token. Even a bug in stromboli's own Go code cannot mutate resources with those keys.
  • stromboli is the MCP client, not the model. When a provider hosts an official remote MCP server (Google Cloud Logging and Monitoring, GitHub, Axiom, Vercel), stromboli connects to it from Go, enforces a strict per-connector tool allow-list, and verifies that every allowed tool carries the readOnlyHint annotation before starting. Anything else (deploy, delete, exec) is refused at the client boundary.
  • No local CLI wrappers masquerading as MCP. Where a provider's MCP server is just a local wrapper around a privileged CLI—for instance, flyctl mcp server exposes destroy, exec, and secrets without read-only annotations—stromboli skips MCP entirely and talks directly to the provider's HTTP API and managed Prometheus.
  • Pre-aggregation and redaction before the context window. Raw logs and diffs pass through a size cap (24 KiB per call) and a regex redaction pipeline that strips Bearer tokens, JWTs, cloud keys, Slack tokens, URL credentials, and email addresses before a single byte leaves your server for the LLM API.

The one opt-in exception: human-approved Fly.io machine actions

By default, every mitigation is a command for a human to copy-paste. On Fly.io, where we run our own checkers, we added an opt-in path (STROMBOLI_ACTIONS_ENABLED=true) for the mitigations you run half-asleep at 3 AM: restart, clone, cordon, and uncordon.

Even with actions enabled, the model still has no execution tool. It can only attach a proposed action to its mitigation. Before anything touches the Fly Machines API:

  1. Actions require a separate fly_machines connector and an explicit per-service allow-list in the catalog (stop and start are excluded by default).
  2. stromboli validates the target machine IDs against the Fly app twice—when proposed and when approved—caps batch size at 10, and refuses to clone machines with attached volumes (since a clone starts with an empty volume).
  3. A named human must click Apply in Slack (verified via Slack's HMAC-SHA256 signature and restricted to SLACK_APPROVERS user IDs) or run stromboli actions approve before the 30-minute TTL expires.
  4. Approval claims the action atomically so double-clicks never execute twice. stromboli runs the action one machine at a time, waits up to 60 seconds for each machine to reach started, and halts immediately on the first machine that fails to come back—so a bad rolling restart can never take down an entire pool.

No data ingestion, no second observability tax

Many observability and AI incident tools start by asking you to ship your logs and metrics into their storage engine.

You already pay for a log store and a metrics backend. stromboli doesn't become a second one: there is no ingestion pipeline, no ClickHouse cluster to size, and no duplicate copy of PII-heavy production logs sitting in another database.

It queries your existing backends on demand for the incident window:

  • Logs: Grafana Loki, Axiom, AWS CloudWatch Logs, Google Cloud Logging, Vercel
  • Metrics: Prometheus (and anything speaking the Prometheus HTTP API: Mimir, Thanos, VictoriaMetrics, Grafana Cloud), Fly.io Managed Prometheus, Google Cloud Monitoring
  • Deploys & commits: GitHub Actions / Deployments / commits, Vercel

All stromboli stores in Postgres is the service catalog, encrypted connector credentials (AES-256-GCM), and the aggregated, redacted evidence each investigation actually used—so your incident audit trail stays intact even after your log store's retention window expires.

A small, GitOps-friendly service catalog

An alert only says "api is returning 5xx". To investigate without guessing Loki labels or PromQL selectors, stromboli keeps a lightweight service catalog.

You don't have to write it from scratch. Point catalog discover at your log or metric connectors and stromboli inspects your active labels, Cloud Run services, Pub/Sub subscriptions, or Fly apps to generate the initial YAML:

stromboli catalog discover loki > services.yaml

Then add what telemetry can't tell you—owners, dependencies, and runbooks:

- name: api
  description: Public REST API
  owners: ["@team-backend"]
  tier: 1
  depends_on: [db-proxy, redis]
  logs:
    source: loki
    selector: '{service_name="api"}'
  metrics:
    source: fly
    selectors: ['app="api"']
  deploys:
    source: github
    repo: acme/api
    workflow: deploy.yml
  runbooks: [runbooks/api-5xx.md]
  monitors: ["openstatus:1234"]
  runtime: fly.io

Keep services.yaml and your markdown runbooks in git, review changes in PRs, and sync them from CI with stromboli catalog import services.yaml --runbooks runbooks/ --prune --yes. Every write validates dependency cycles, connector types, and missing runbooks before it lands.

Why an AI SRE has to be open source and self-hosted

Every AI SRE we looked at is closed-source SaaS. Datadog has Bits AI SRE, the other observability incumbents are bundling their own, and a wave of startups will happily take a webhook and your cloud credentials. You send an alert into a black box and get back a paragraph in Slack—with no way to read the system prompt, inspect the tool definitions, audit the redaction rules, or understand why the agent fixated on an unrelated warning.

We think that's backwards for software that sits next to production during an outage.

First, debuggability: when an agent misdiagnoses an incident, you should be able to read the exact system prompt, check the tool schemas, inspect the ordered step log, and open a PR to fix it. In stromboli, even our architecture trade-offs live in docs/adr/.

Second, data sovereignty: an AI SRE needs read access to your production logs, metrics, commit diffs, and internal runbooks. Handing those credentials to a multi-tenant third party is a tough sell for any security-conscious team. stromboli is designed as one single-tenant instance per organization. Your credentials stay in your Postgres encrypted at rest, and nothing leaves your network except the redacted prompt sent to the LLM provider you choose—Anthropic, OpenAI, Gemini, DeepSeek, OpenRouter, Vercel AI Gateway, or Ollama if not a single token can leave your VPC.

Self-hosting in five minutes (really)

We'll be honest: self-hosting the full openstatus platform is a project. Turso, Tinybird, Go checkers, several apps, and background queues—we made it much smoother, but it's still a weekend commitment.

We designed stromboli to be the exact opposite: one static Go binary and a Postgres database. The same binary runs both the ConnectRPC server and the clig.dev-compliant CLI (--json output, TTY detection, hidden secret prompts, grpcurl reflection out of the box), and database migrations run automatically on startup.

cp .env.example .env   # STROMBOLI_SECRET_KEY, STROMBOLI_API_TOKEN, and an LLM key
docker compose up -d   # stromboli + Postgres on http://localhost:8080

Install the CLI via Homebrew (or grab a binary from GitHub Releases), wire up your connectors, and run your first investigation:

brew install openstatusHQ/stromboli/stromboli

stromboli connector add loki loki url=http://loki:3100
stromboli connector add github github token
stromboli catalog discover loki
stromboli brief api --since 1h
stromboli investigate api --description "5xx spike on /v1/monitors"

If you just want to watch it work without wiring up production first, the repo ships with a local dev stack (docker compose -f docker-compose.dev.yml up -d) that spins up Postgres, Loki, and a demo service that triggers a real failure after three minutes. There are also one-command deploy scripts for Fly.io and Railway.

Built to pair with openstatus, or run completely on its own

stromboli has its own repository, its own release cycle, and zero hard dependency on openstatus. If your stack runs on Prometheus, Alertmanager, and Grafana, point them at /webhooks/alertmanager or /webhooks/grafana and you never need an openstatus account.

If you do use openstatus, the two fit together naturally: point a webhook notification at POST /webhooks/openstatus, map your monitor ID in services.yaml, and the next failed check automatically triggers a brief, a mitigation in Slack, and a root-cause report—and resets deduplication as soon as the monitor recovers.

stromboli also exposes an MCP server (stromboli mcp over stdio or /mcp over HTTP) and a Claude Code skill. During an incident, you can ask Claude Code or Cursor right from your editor what stromboli saw on api, pull the deterministic brief, or inspect the evidence behind a mitigation before shipping the code fix.

Try it and break it

stromboli is MIT licensed and available today at v0.1. It is early, and it is unapologetically shaped by the stack we run in production (Fly.io, Google Cloud, Vercel, Axiom, Loki, Prometheus, GitHub).

If you run a different log store, orchestrator, or alerting pipeline—or if you throw a gnarly incident at it and it gets the hypothesis wrong—we want to hear about it: