openTPU: AI agents designed an FPGA inference accelerator that runs Qwen3.5 and Gemma 4 at up to 85.8 tok/s
An agent tournament gated by hard verifiers produced a full RTL/ISA/compiler stack that saturates a Kintex-7's DDR3 — and every number is self-reported.
AI agents designed a complete open-source inference accelerator — SystemVerilog RTL, a custom ISA, a bit-exact simulator, a compiler and a profiler in one Apache-2.0 repo — that runs real LLM weights on a Kintex-7 FPGA card at up to 85.8 tok/s. The project, openTPU, hit the Hacker News front page this week (334 points, 387 comments). It is a working demonstration of the author's thesis: the agent loop is commodity, the verifier is the moat.
What shipped
openTPU is a monorepo holding everything needed to build and run the accelerator: SystemVerilog RTL, an ISA of 8×32-bit-word instructions, a bit-exact Python simulator that serves as the spec, a Python-embedded kernel language and compiler (@ol.jit), host tools (otpu-chat, otpu-smi), and the Lens profiler with roofline and per-cycle timelines. The design is deliberately simple: a sequencer issues one instruction per cycle to a DMA engine, an int8 systolic matrix unit, an fp32 vector unit and a quantizer. There is no cache and no hidden scheduling — every data movement is an instruction, so a trace shows exactly where cycles go.
The target is a decommissioned Inspur YPCB-00338 PCIe card (Xilinx Kintex-7 xc7k480t, two DDR3-1066 channels, 17.1 GB/s peak) running a single 133.33 MHz bitstream for all models. LiteDRAM controllers are calibrated by a small CPU inside the memory core. Measured device decode per the README (64 greedy tokens after a 512-token prompt, host argmax in the loop):
| Model | tok/s (device / wall) | DRAM bandwidth |
|---|---|---|
| LFM2.5-230M (4-bit, int8 head) | 85.8 / 82.1 | 14.1 GB/s (82%) |
| Qwen3-0.6B (4-bit) | 31.3 / 30.7 | 13.9 GB/s (82%) |
| Qwen3.5-2B (4-bit) | 12.09 / 12.03 | 15.8 GB/s (92%) |
| Gemma 4 E2B (4-bit, 4-bit head) | 12.14 / 12.09 | 15.5 GB/s (91%) |
| Phi-4-mini 3.8B (4-bit) | 6.56 / 6.55 | 15.8 GB/s (92%) |
Across all ten models and configs the card runs at 82–94% of its DRAM peak — memory-bound, exactly where a simple, cache-free design should land.
Two details are worth stealing. First, 4-bit quantization: FP4 (E2M1) elements with a two-level scale per 128 elements — a bf16 block scale times an unsigned 4-bit multiplier per 32 — at 4.25 bits per weight, keeping the LM head in int8. It cuts bytes per token by about a third and raises decode speed 40–45%; the per-model perplexity cost is documented (Qwen3-0.6B goes 23.50 → 27.84 on the book sample in the fp4/int8-head format). Second, MoE expert offload: the card keeps experts in per-layer slots in its own DRAM and the host streams only the missing ones over PCIe, gated by a single new instruction (WAITW). Measured on the card: LFM2.5-8B-A1B at 10.6 tok/s with 98.5% of expert uses hitting slots (5.2 MB streamed per token); Qwen3.5-35B-A3B at 3.95 tok/s with 62% hits and 153 MB per token at 1.41 GB/s over PCIe.
The claim that makes this more than a hobby FPGA project is bit-exactness: the card produces the same tokens as the simulator "bit for bit." The repo's tools/validate.py checks a device's greedy tokens and logits against a CPU golden, and an RTL-vs-simulator check runs in four minutes on a 16-core host. The author is candid about the limit: the golden cannot round exactly as the device does, so once one int8 value rounds the other way, the difference propagates through later layers.
Why it matters
openTPU is the second product of the author's auto-arch-tournament method, previously used to evolve RISC-V cores. The loop: a hypothesis agent proposes a change, an implementation agent applies it in an isolated git worktree, and hard eval gates decide whether it merges — riscv-formal checks, Verilator cosim against a Python ISS, three-seed nextpnr place-and-route, and CoreMark CRC re-validation. In that run, 73 hypotheses over 9h51m took the champion from 301 to 577 iter/s (+92% fitness, +26% over VexRiscv's published CoreMark/MHz, 40% fewer LUTs); 50 of the 73 were regressions. The orchestrator is hardcoded — the model never edits it.
The write-up of that method states the thesis openTPU now demonstrates: "The agent loop is a producer. The verifier is the only thing standing between you and a confidently-wrong number… The loop is commodity… The verifier is not commodity." On Hacker News the author said the accelerator started at "a few tokens per second" and reached 80+ tok/s through a recursive self-improvement loop; the next step is DDR4/HBM2 FPGAs — the obvious lever for a design that already saturates DDR3.
For engineers, the transferable lesson is not the tok/s numbers. It is that a full hardware toolchain — RTL, ISA, simulator, compiler, profiler — can be treated as an agent-evolvable artifact when every change is gated by a bit-exact verifier; the verifier, not the model, is what makes the result trustworthy. That is a workflow you can run this week: the repo builds with a Makefile, and everything except the card runs on a laptop via the simulator.
Caveats
Every performance number here is self-reported in the README. There is no independent reproduction and no like-for-like GPU comparison yet — AICoder's coverage flags exactly this. The author also admits on HN that the fp32 unit has "bugs/'non standard behavior'" for very small or very large values, which he says are "proven harmless for LLM inference." When a commenter asked for the proofs, the answer was that tests compare against a CPU Hugging Face implementation and the bugs are being fixed. Treat "proven harmless" as a claim, not a result.
There are internal inconsistencies too: docs/offload.md still opens with "Nothing here has run on the card yet," contradicting the README's measured MoE numbers. The doc is stale relative to the README, but it is the kind of thing that makes independent verification harder. And while the RISC-V tournament published its full hypothesis log, no equivalent accept/reject log for openTPU's own evolution is in the repo — the "recursive self-improvement" narrative is the author's account, not a reproducible artifact. Note also that this openTPU is unrelated to the older UCSB OpenTPU reimplementation of Google's TPU.
The honest read: an agent-driven design loop, gated by a strong verifier, produced a working, inspectable, memory-saturated inference accelerator on commodity FPGA hardware. The headline numbers still need a second pair of hands to confirm.