Teel: AI21's shadow-interpreter runtime parallelizes sequential Python agents — no async, 83% faster Tree of Thoughts
A C-extension shadow interpreter reads CPython's live frame stacks to run code before the program reaches it — but the magic comes with a purity contract and a hard Python 3.12.13 pin.
AI21 open-sourced Teel, a Python runtime that overlaps independent IO-bound calls in sequential code by reading ahead through your bytecode with a shadow interpreter — no async, no await, no gather. On Princeton's unmodified tree-of-thoughts-llm, it reports an 83% average runtime reduction (8.07x on the longest run), and it ships explicitly as a proof-of-concept pinned to exactly one CPython build.
What shipped
Teel ("Turbo-Efficient Execution Library") is a thin API: three decorators and one context manager (repo, announcement). Mark heavy, parallelizable steps with @cacheable, mark orchestration with @jit, optionally mark cheap helpers with @inline, and wrap the call in with Teel():. Outside that context, every decorated function runs as plain Python, untouched.
from teel import cacheable, jit, Teel
@cacheable
def fetch(url): ... # an LLM call, an HTTP request, a retrieval
@cacheable
def summarize(text): ...
@jit
def research(a, b):
pa = fetch(a) # no dependency between these two
pb = fetch(b) # Teel runs them at the same time
return summarize(pa + pb)
with Teel():
answer = research("a.txt", "b.txt")
The reported numbers are all AI21's own, and they are internally consistent between the blog and the README:
- Tree of Thoughts, unmodified. AI21 ran Princeton's
tree-of-thoughts-llmon Game of 24 samples with GPT-4, applying the decorators from outside the repo. Longest run went 307s → 38s (8.07x); the low end went 148s → 24s; average runtime reduction was 83%. - Deep Agents comparison. In a fork of LangChain's Deep Agents
dcodeCLI, they swapped the model-written snippet language from async JavaScript to synchronous Python under Teel. Across ten mixed web-search-and-LLM prompts: Teel averaged 58s vs 71s for async JS (19% faster) and 150s for the same Python run sequentially (2.58x). AI21 itself flags that the same prompt yields different generated code each run, "so this compares distributions of generated programs, not two implementations of one." - Code-review agent demo. Plain Python 7.1s, handwritten
asyncio3.0s, Teel 2.9s — the asyncio version needed three tasks, two cancellations and afinallyblock to match what Teel does by speculatively following both branches of anifbefore the verdict arrives.
Why it matters
The pitch targets agent orchestration. Most of an agent run's wall clock is a thread sitting on a socket — model calls, retrievals, tool calls, a shell exec — and overlapping those waits in Python means rewriting the code first. Make one function async and the change climbs the whole call chain, until orchestration logic and concurrency plumbing are the same lines of code. That is a real cost when a model writes the orchestration itself: AI21 argues that models are inconsistent at reaching for Promise.all correctly.
Teel's bet is that concurrency can live in the runtime, not the code. You write the sequential version; the runtime finds the parallelism. The mechanism is genuinely unusual. When a @jit function runs, Teel duplicates execution state into a limited shadow interpreter and steps through the bytecode the main thread hasn't reached yet. It grabs the frame stack with sys._getframe(), then a small C extension called frameinspect reads each frame's full value stack out of CPython and uses f_lasti to find the last executed instruction. The shadow interpreter walks forward on its own copy with copy-on-write, marks anything built from an in-flight call as "pending," and dispatches any @cacheable call whose arguments are already known to a ThreadPoolExecutor. It also speculatively follows conditional branches — cloning state and walking both jump targets, bounded by configured width/depth to avoid exponential blowup — which is how it starts fallback work before the decision that needs it.
Two design choices keep this from being a correctness hazard. First, the shadow interpreter never produces your answer: CPython still executes the function to completion, instruction for instruction, and the only difference is that some calls find their result already cached. "A wrong guess costs time and tokens, not correctness." Second, lookahead can run from multiple threads, so calls are memoized on a key built from the function plus json.dumps(args) — a second lookahead that discovers the same fetch(url) attaches to the first instead of firing another request.
Caveats
This is a proof-of-concept, and AI21 says so in the README. The headline benchmarks are vendor-reported — as of publication, no independent reproduction has been published — and the Deep Agents comparison is explicitly distributional.
The @cacheable contract is the sharp edge. Teel runs a function earlier than your code would, sometimes runs one your code never reaches, and hands back a saved result instead of running it again. Each of those is only legal if the function does not care when or how often it runs. AI21 is explicit: no database writes, no reading the clock, no hidden state, and JSON-serializable arguments (they form the memo key). A speculatively-executed branch that sends an email or charges a card is a real hazard — nothing unsends the email.
Other limits, all from AI21:
- GIL. Background calls ship to a
ThreadPoolExecutor, so parallelism only helps for I/O (requests.get) or GIL-free CPU work (numpy). CPU-bound, GIL-constrained functions get nothing;ProcessPoolExecutororInterpreterPoolExecutor(Python 3.14+) are listed as future work, not available now. - Version pin. Teel requires exactly Python 3.12.13, because
frameinspect.ccompiles against CPython's internal headers. Porting to other versions is future work. - Failure handling. Failed calls are dropped rather than cached, on the assumption the failure was transient; the authors themselves flag that exception-specific retry handling might be worth revisiting.
- Token cost. Speculative execution can run fallback branches that are later discarded — a wrong guess burns a background call plus its tokens.
