AWS and Berkeley's Infinity runs 75,000 suspended agents in 8 GB — and delivers async events to models as synthetic tool calls

On 23 October 2026, AWS and UC Berkeley released Infinity, an Apache-2.0 agent runtime that suspends an agent the moment it calls a tool and resumes it later from its saved conversation. At ~100 KB per idle agent, the authors extrapolate to roughly 75,000 concurrent agents in the 8 GB of a Raspberry Pi 5 — and because an idle agent holds no process or connection, the same design runs on serverless functions that can exit mid-tool-call.

The figure grabs attention, but the mechanism matters more to platform builders: every event that can wake an agent — tool result, webhook, child-agent report, timer — is a message on a queue (durable in serverless/step mode, an in-memory channel when embedded), and asynchronous events reach ordinary, unmodified chat-completion models as synthetic tool calls.

What shipped

Infinity is part of the Hydro project, a collaboration between AWS and UC Berkeley, and ships as a Rust framework on GitHub under Apache 2.0. A companion Reactive Agent Protocol (RAP) defines the tool contract. It is explicitly an early research release, shared for feedback.

The runtime executes each agent as a sequence of bounded slices: load the conversation from a pluggable store, run exactly one model completion, persist the output, dispatch at most one tool call, then end. A slice never blocks on anything external — the tool call is the yield point, and the tool result re-enters through the input queue whether it takes 100 milliseconds or three days.

Everything that can resume an agent is one message type: user text, tool results, subscription events, OAuth challenges, timer wake-ups. Each carries a group_id identifying its conversation thread plus an optional synthetic tag. The runtime keeps no in-process state machine tracking "waiting for tool X" — whatever the agent is waiting for eventually shows up as a message, and the slice that processes it rebuilds everything from storage.

The concurrency claim comes from a benchmark of 80,000 agents running twenty tool-calling turns each — 3.2 million completions total — on a single thread of an AMD EPYC 9R14 with in-memory stores. Resident memory grew linearly at about 100 KB per idle agent; the 75,000-in-8 GB figure is the authors' extrapolation to a Raspberry Pi 5's memory, and most of that footprint is conversation history, since an idle agent has no task, stack, or open connection.

The serverless story follows from the same invariant. The reference AWS deployment uses an SQS FIFO queue as the input queue with one MessageGroupId per thread; each delivery invokes a Lambda that runs one step and exits. Conversation histories live in Aurora DSQL, deduplication state in DynamoDB, and the sleep tool uses SQS delivery delays and EventBridge Scheduler for timers, with CDK constructs in the repo.

Making that correct on at-least-once infrastructure takes real machinery, all documented in the architecture docs:

• Bounded execution time and no process affinity. A slice is one completion plus a few storage/dispatch calls; all state is keyed by thread ID, so any invocation can process any thread's next message.
• Idempotent redelivery. Every input message and completion carries a stable ID, and the state store tracks which IDs each thread has processed, so a redelivered message is recognized and skipped.
• Durable turns. Streamed model output is buffered and committed only when the turn completes; a partial stream is discarded and the completion retried, so half-streamed assistant messages never reach storage. When a tool call ends a turn, the buffer flushes before dispatch — the call is durable in history before its result can arrive.
• Ordering without locks. FIFO per thread (one slice at a time, since two concurrent slices on the same history would corrupt it), parallelism across threads.

Why it matters

The synthetic tool call is the part that generalizes beyond the demo. Because an ordinary provider API can't receive an unprompted event on its own, Infinity synthesizes one: every asynchronous event arrives as a fabricated tool-result turn that an unmodified model can consume.

• Agent-to-agent messaging in the same format. Spawning a child thread creates a new message group; children inherit the parent's history up to the spawn point and run their own slices in parallel, and when a child reports back, its report is tagged as a thread report that the parent sees as a synthetic tool result.
• Context hygiene for event streams. A subscription event does not get appended to the parent's history; instead the runtime spawns a temporary child thread seeded with the event, and the parent sees a report only if it mattered. Events marked associative — like log lines from a long-running command — skip the child thread and inject inline.
• Scale-to-zero as hibernation. Since slices never block, the platform's own scale-to-zero behavior does the suspending: the project claims to be the first agent runtime that fits serverless's execution model directly rather than holding invocations open while tools run.

There is also a tool-authoring contract change worth noting. A tool's execute method starts the work and returns immediately; it sends the result to the agent's input queue when done. The same tool code runs in a long-lived process or on Lambda — only the InputSender differs, an in-memory channel versus a durable queue.

The abstraction worth stealing for platform builders: the conversation is the state. Nothing else is needed to resume an agent on any process with access to the store, and both the memory profile and the serverless story fall out of that one design decision.

Caveats

All numbers are vendor-reported. The benchmark used a simulated model with in-memory stores — no model doing real tool work was tested — and the authors publish no throughput, latency, or cost numbers; AI Matter's coverage flags exactly this. The 75,000-in-8 GB figure is an extrapolation to Pi 5 memory, not a measurement on a Pi 5, and the "first agent runtime that fits the platform's execution model directly" claim is the project's own.

It is an early research release. The open questions the authors raise — how models handle interleaved events, scheduling priorities, compaction across agent trees, and, most pointedly for the core trick, event authenticity — are itemized in coverage of the post. No independent replication of the benchmark surfaced in the coverage we found, and the repo shows little traction so far: 22 stars, 4 forks, and 54 open issues at the time of writing.

Sources

• Introducing Infinity: a runtime for extreme agent concurrency — AWS Open Source Blog
• hydro-project/infinity — GitHub
• Infinity Runtime architecture — infinity.hydro.run docs
• Reactive Agent Protocol (RAP) — infinity.hydro.run docs
• AWS and UC Berkeley release Infinity for 75,000 agents in 8 GB — AI Matter
• Introducing Infinity: a runtime for extreme agent concurrency (AI overview) — devfeed