Inside Instacart’s ‘Brownie Recipe Problem’: Why Real-Time AI Needs Fine-Grained Context, Not Just Better Reasoning

Large language models (LLMs) are getting dramatically better at abstract reasoning, planning and natural language interaction. Yet when those same models are dropped into real-world, real-time products, the gaps become obvious: they often don’t know what’s actually available, where, when, or for whom.

Instacart CTO Anirban Kundu calls this the “brownie recipe problem.” It’s a concrete illustration of why reasoning alone is not enough for production systems that must act on live data, honor user preferences, and return reliable results in under a second.

For AI engineers and product leaders, Instacart’s experience highlights a set of design patterns: splitting reasoning from state, offloading work to specialized smaller models, and orchestrating micro-agents over heterogeneous third-party systems. It also exposes the practical limits of today’s tooling and protocols when latency and reliability are non-negotiable.

The ‘Brownie Recipe Problem’: When Intent Isn’t Enough

On the surface, “I want to make brownies” sounds like a classic LLM use case. A general-purpose model can quickly generate an ingredient list, walk a user through the steps, and even suggest variations.

In a production grocery platform like Instacart, this is the easy part. The hard part begins once the user’s intent collides with reality:

• What’s actually in stock at nearby stores?
• Does the user prefer organic eggs or regular eggs?
• Are there dietary constraints that change which products are suitable?
• Will refrigerated or frozen items survive the delivery window without spoiling?

Kundu’s “brownie recipe problem” encapsulates this challenge. To be genuinely assistive, the system cannot stop at a generic recipe. It must synthesize:

• User preferences (e.g., organic vs. conventional products, brands, health goals)
• Local availability (inventory at specific retailers in the user’s geography)
• Fulfillment constraints (delivery times, temperature sensitivity, spoilage risk)

And it has to do this under tight latency constraints. According to Kundu, Instacart targets experiences under about one second. If reasoning or orchestration stretches to 15 seconds per interaction, users drop off. That user behavior constraint defines everything about the architecture.

Reasoning vs. Real-World State: Why LLMs Alone Aren’t Enough

Instacart’s domain exposes a fundamental split that many real-time systems face: there is a “world of reasoning” and a “world of state.”

The reasoning world is where LLMs excel today: interpreting free-form intent, understanding language like “healthy snacks for kids,” and making high-level decisions about what might satisfy that request.

The state world is everything the model doesn’t inherently know and can’t reliably memorize:

• Per-store, per-region product catalogs
• Real-time or near real-time inventory and substitutions
• Delivery zones, time windows, and logistics constraints
• User-specific histories and preferences

A naive approach would be to stuff as much of this state as possible into a single large model or prompt: purchase histories, catalogs, inventory, policies, and more. Kundu is blunt about the consequences: the model would “blow up into a size that will be unmanageable.”

For engineers, this reflects two hard limits:

• Context window and token limits: Even with large context windows, ingesting full catalogs and user histories is unrealistic and expensive.
• Update frequency: Real-world state—especially inventory and logistics—changes faster than large models can be retrained or even efficiently re-embedded.

The implication: production systems need to treat LLMs as one component in a larger architecture that manages state externally, surfaces only the most relevant slices of context, and keeps the heavy lifting off the critical path where possible.

Slicing the Problem: Foundational Models Plus Task-Specific SLMs

Instacart addresses the scale and latency constraints by splitting processing into multiple stages rather than relying on a single monolithic model.

Step 1: Intent and categorization with a foundational model. Incoming user requests—“plan a kid-friendly healthy lunch,” “get ingredients for brownies”—are first processed by a large, general-purpose LLM. This model interprets intent and helps categorize the request in terms that downstream systems can act on.

Step 2: Routing to small language models (SLMs). Once intent and high-level structure are clear, the system routes the problem to smaller, focused models. Kundu points to two main types:

• Catalog-context SLMs: These understand product relationships and multi-level details around an order—what items go together and what reasonable substitutions look like when a primary product is not available.
• Semantic-understanding SLMs: These specialize in interpreting requests like “healthy snacks for children,” understanding what “healthy” and “for an 8-year-old” imply in terms of product attributes and appeal.

For catalog context specifically, substitutes are critical. Kundu notes “over double digit cases” where a product is unavailable in a local market. Each of those cases requires the model to:

• Know which products are functionally compatible in the recipe or use case
• Respect user preferences (e.g., staying organic if the user leans that way)
• Surface options that are actually available in the user’s geography

In the “healthy snacks for children” example, the semantic SLM has to connect several layers:

• What nutritional profiles typically qualify as “healthy”
• Which products are appropriate for children of a given age
• What kinds of snacks are likely to appeal to an 8-year-old
• What related subsets exist when exact matches aren’t available

By pushing these tasks into specialized SLMs, Instacart narrows the context each model must handle, making the system more tractable to scale and tune. For practitioners, this suggests a pattern: use a large model for general understanding and delegation, then rely on small, tightly scoped models to reason with local context and domain-specific rules.

Logistics, Spoilage, and Latency: The Hidden Constraints

Even with good semantic understanding and product matching, logistically naive recommendations can fail the user. Instacart’s domain brings physical-world constraints directly into the AI loop.

Certain products—ice cream, frozen vegetables, other temperature-sensitive items—have narrow delivery tolerances. If the system recommends or assembles a basket that cannot be reliably delivered before spoilage, the user experience and the business both suffer.

As Kundu describes it, the model stack must not only interpret intent and categorize products, but also ask: “logistically, how do you do it?” That means integrating:

• Product-specific characteristics: Is the item frozen, refrigerated, or shelf-stable?
• Environmental context: Weather and ambient temperatures impact how long items survive in transit or on a driver’s route.
• Operational parameters: Available delivery windows, route lengths, and service-level commitments.

All of this has to be reconciled within the same sub-second target. From an engineering perspective, this becomes a classic orchestration and caching challenge: keep as much of the logistics reasoning precomputed and close to the decision boundary as possible, and ensure models are only ever working with the minimum context they need to make a safe, user-appropriate decision.

From One Big Agent to Many Microagents

Like many organizations, Instacart is exploring AI agents as a way to coordinate complex tasks—shopping, payment, fulfillment—across multiple systems. The company’s experience echoes a familiar lesson in systems design: a single “do everything” agent quickly becomes unwieldy.

Kundu compares Instacart’s approach to the Unix philosophy: instead of one monolithic agent that handles every possible workflow, the company leans on multiple smaller “microagents,” each focused on a specific capability or integration surface.

This modularity becomes especially important with payments and other critical back-end functions that have distinct failure modes. Different payment systems, for example, may:

• Expose different APIs and data formats
• Fail in different ways and at different rates
• Have different update intervals and reconciliation processes

Trying to accommodate all of those behaviors in a single agent environment creates tightly coupled, hard-to-evolve systems. By contrast, a microagent approach allows Instacart to encapsulate each integration and its quirks, then orchestrate them at a higher level.

Engineers building agentic systems can take away a few architectural guidelines highlighted by Instacart’s experience:

• Encapsulate integrations: Treat each third-party platform—POS, catalog, payment—as a separate capability with its own agent and error models.
• Prefer composition over expansion: Add new microagents rather than bloating existing ones when capabilities or services expand.
• Design for heterogeneous reliability: Assume different systems will have different uptime patterns, latencies, and failure semantics.

MCP, UCP, and the Reality of Tool Integration

To manage the growing web of tools, systems, and agents, Instacart has adopted emerging standards that aim to simplify how models talk to external resources.

OpenAI’s Model Context Protocol (MCP) is one of those. MCP provides a way to standardize how AI models connect to tools and data sources—defining a predictable interface for calling out to services and retrieving structured results. For Instacart, this supports the orchestration of multiple agents and tools behind the scenes.

Google’s Universal Commerce Protocol (UCP) is another. UCP is an open standard that lets AI agents directly interact with merchant systems in a more uniform way. In a landscape of diverse merchant platforms, this kind of standardization is appealing: it promises fewer bespoke integrations and more reusable tooling.

However, Kundu is clear that adopting MCP and UCP does not make the integration problem disappear. The question is less “can we connect?” and more:

• How reliably do those integrations behave over time?
• How well do users understand what’s happening behind the scenes?
• How easily can the system discover which services exist and which are appropriate for a given task?

Instacart has implemented MCP and UCP in “very different” use cases, and the biggest pain points mirror what many production teams encounter:

• Failure modes: Different services fail in different, often poorly documented ways, forcing the team to invest heavily in defensive handling and retries.
• Latency differences: Response times vary widely between services, complicating overall latency budgeting when end-user interactions must feel instantaneous.

Kundu estimates that “probably two thirds of the time” is spent fixing error cases stemming from these differences. For practitioners, this is a reminder that adopting a protocol is the start of the reliability journey, not the end.

Practical Lessons for Real-Time, Context-Aware AI Systems

Instacart’s “brownie recipe problem” surfaces a set of patterns and constraints that generalize beyond grocery delivery to any real-time, context-rich product:

• Reasoning is necessary but not sufficient. Strong LLM reasoning must be paired with a disciplined approach to live state—inventory, logistics, user preferences—that lives outside the model and is selectively surfaced.
• Architect for latency from day one. Treat user tolerance (e.g., sub-second responses) as a hard constraint. Design your model stack, caching, and orchestration around that budget.
• Use large models to understand and route, not do everything. Keep foundational models focused on intent understanding and high-level decision-making, then delegate to small, specialized models or agents.
• Embrace microagents over monolithic agents. Decompose by capability and integration boundary, especially when external systems have different APIs and failure semantics.
• Expect integration standards to reduce friction, not remove complexity. Protocols such as MCP and UCP can standardize interfaces, but you will still spend much of your time on error handling, latency management, and service discovery.

For AI engineers and product leaders, Instacart’s experience is a concrete signal: the frontier is no longer just “smarter” models, but architectures that can pull the right sliver of context from a messy, dynamic world and act on it reliably in real time. The brownie recipe, it turns out, is less about baking and more about designing systems that can reason, know their limits, and still deliver before the timer runs out.