How Instacart's Blueberry AI Rewrites the On-Call Playbook

A New Kind of On-Call Sidekick

When a service goes down at 3 a.m., the on-call engineer doesn't just fix the problem—they first have to figure out what the problem is. That means checking which team owns the service, scanning recent deploys, digging through logs and metrics, and hoping the docs aren't outdated. By the time they've gathered the basics, precious minutes have slipped away.

Instacart has been living that reality for years, and they've built an AI assistant named Blueberry to change the opening act. Blueberry doesn't take over the incident. Instead, it does the boring, time-consuming homework that used to eat up an engineer's first half hour. The system watches the company's Slack-based incident flow, and when an alert fires, it spins up a handful of sub-agents that go gather context in parallel.

From 60% to 90%: The Accuracy Leap

The numbers are worth pausing on. Instacart says that in April alone, Blueberry ran about 25,000 diagnoses across more than 270 Slack channels. The system's accuracy in pinpointing root causes jumped from around 60% to above 90%, a leap the company credits to one thing: access to over 14 years of incident history.

That historical data is the secret sauce. Blueberry doesn't just rely on a general-purpose language model. It connects to internal resources—past incident write-ups, service ownership charts, logs, deployment records, and other debugging signals. The AI's guesses are grounded in what actually happened before, not just in pattern-matching from the internet.

How Blueberry Works: Agents, Threads, and Three Minutes

When an alert triggers, Blueberry launches about ten sub-agents at once. Each one tackles a slice of the puzzle: one looks at recent code changes, another scans metrics, a third pulls up related incidents from the archive. They report back into the same Slack thread where the engineer is already working, so nobody has to jump between tools.

According to Instacart's CTO, Anirban Kundu, the system typically produces an initial set of root-cause hypotheses in around three minutes. That's not a final answer—it's a well-informed starting point. The engineer still makes the call, but they're making it with context already laid out in front of them.

Building on a Tool-Aware Framework

Blueberry's architecture is more than a chatbot with a search bar. It's built on a persistent, tool-aware framework. The AI agents can call external tools via MCP (Model Context Protocol), pulling live data as they work. They also keep a persistent state of what they've checked and what they've ruled out, so the investigation doesn't lose the thread.

That persistence matters. If an agent checks a service's health and finds it clean, that fact stays in the running log. Later, when the engineer asks a follow-up, Blueberry doesn't re-litigate the same ground. It builds on what it already knows, which makes the whole process feel more like a competent junior colleague than a glorified search engine.

Human Oversight: The Guardrails That Matter

One of the most important design choices is what Blueberry doesn't do. It doesn't make changes to production systems. It doesn't auto-remediate. It gathers information, generates hypotheses, and assists with debugging—but the final decisions stay with the human engineer.

That's a deliberate line in the sand. For an AI system in production ops, the core challenge is trust. If the tool starts taking actions on its own and gets it wrong, you've got a bigger incident on your hands. By keeping Blueberry in an advisory role, Instacart gets the speed boost without losing accountability.

From Zero to Context: Changing the Starting Line

Alan Wong, Instacart's director of software engineering, points to a subtle shift in how on-call work feels now. Before, an engineer started from nothing. They had to build a mental model of the system from scratch, pulling logs and checking dashboards one by one. Now, they start with a pile of relevant context—logs, deployment info, related systems—already assembled.

That doesn't just save time. It changes the cognitive load. Instead of spending the first twenty minutes just trying to understand what's happening, an engineer can jump straight into evaluating hypotheses. The boring part is handled, and the human brain is freed up for the actual thinking.

More Than a Model: The Engineering Around It

Instacart's experience with Blueberry suggests that effective AI ops tools aren't just about the model's intelligence. They're about the framework around it: the context you feed it, the workflows you build, the tools you let it touch, and the feedback loops that keep it improving.

The company reports that the system handled about 25,000 diagnoses in a month with a workflow success rate of 99.9%. It made over 58,000 MCP tool calls and adapted to the working patterns of roughly 60 different teams. Those numbers speak to the operational maturity required to make AI actually useful in a chaotic production environment.

What's Next for AI in Incident Response

Blueberry is part of a broader push at Instacart to explore AI agents across the business. For the SRE and on-call community, it's a concrete example of what's possible when you combine large language models with an organization's own institutional memory.

The system also leaves a trace. Every incident Blueberry helps with adds to the knowledge base, so the next time something similar happens, the AI has an even richer set of historical incidents to draw from. It's a flywheel: more incidents lead to better context, which leads to better diagnoses, which leads to faster resolutions.

For anyone running a large-scale service, the takeaway is clear. The real value isn't in the AI itself—it's in the historical data and the careful integration with human workflows. Blueberry works because it's built on 14 years of Instacart's own scars, and because it knows when to stay out of the way.

The Bottom Line

Instacart's Blueberry isn't a futuristic robot taking over the on-call pager. It's a pragmatic tool that does the unglamorous legwork of incident response, and it does it well. By grounding AI in real operational history and keeping humans in the loop, they've managed to turn a common pain point into a competitive advantage.

If your team is still starting every incident from square one, Blueberry is proof that you don't have to. The tools exist, the patterns are emerging, and the data you've already got might be the most valuable asset you're not using.