---
file: TRAIN.md
from: ChipTrain (https://chiptrain.ai)
what: training data and RL environments for AI that designs chips
---

# TRAIN.md

ChipTrain: training data and RL environments for AI that designs chips. We refine chip-design history (real bugs, real regressions, real agent failures) into graded environments with measured pass rates, across the whole flow from spec to signoff. We set the benchmark, publish the results, and license the data to the teams building the next generation of models.

What follows is what we believe about training AI to design chips, and what we sell.

## 0. Why chips

Coding agents are now good at writing software. They do much worse on chip design: on hardware repositories, the same agents score 37 to 58 points lower than on software (Phoenix-bench).

## 1. The tests passed

The agent's patch passes all 12 tests, and the design is still wrong. It has a bug that none of the tests look for. A test suite only checks the cases someone wrote down, so a model trained only to pass tests learns to satisfy the tests rather than the spec.

## 2. Grade with proof

Most grading today is someone's opinion: a reviewer, or another model, reads the patch and says it looks good. We grade with proof. A formal equivalence check compares the patched module with the original, finds they behave differently, and shows the exact cycle where it fails. In the illustrative example, a 2-deep FIFO gets full, read and write at cycle 5, and a stale word comes out at cycle 8.

## 3. Set the difficulty

If no model can solve a task, it teaches nothing: at a 0% pass rate there is no learning signal. If every model solves it, it teaches nothing either. So we set how hard each task is. Labs want tasks that models solve at least 2 to 3% of the time (Epoch AI), and evaluations tell models apart best around 20 to 40%. Every task comes with its measured pass rate. We run frontier models on every task before it ships, so you know what you're buying.

## End of file

Models improve quickly, and tasks that were hard stop being hard. Every public benchmark gets solved eventually. We keep making harder tasks as models catch up: when a batch is solved, we build the next one from the same sources and graders.

---

# The fine print

## Why chip design

- The labs are becoming chip companies: silicon is on the critical path of frontier AI, and labs now design their own accelerators. Coding agents struggle on hardware because the context is long, the tools are specialised, and passing tests doesn't mean the design is right.
- Verification is where models score worst: on the CVDP benchmark (arXiv 2506.14074), the best-scoring model (Claude 3.7 Sonnet, non-agentic pass@1, Table 2) passes 6 to 25% of verification tasks (testbenches and assertions), against 34 to 48% for writing RTL. It is the largest share of effort on a real chip project.
- The data that would fix it is locked inside chip companies, behind customer contracts, foundry NDAs, third-party IP and export rules.

## What we sell

- **Public benchmark**: we pick the tasks, run the models, publish the results. Open harness, reproducible runs; Pass@1, Pass@k and per-gate scores; refreshed as models saturate it.
- **Calibrated task sets**: held-out tasks across the whole flow. Pass rate measured on every task; exclusive or non-exclusive licences; new tasks after the last batch is solved.
- **RL environments**: executable worlds for training agents on real silicon work. Sim, lint, synth, formal, coverage, waveforms; MCP tool servers and pinned EDA toolchains; dense, hard-to-game reward; grader outside the sandbox; works with your own training harness.
- **Private evaluations**: under NDA, on your own RTL, a contamination-free measurement of how AI agents really perform on your design.

Three ways in: **non-exclusive licence** (labs and model builders; the fastest way to start), **exclusive build** (one lab; your harness, task mix and difficulty band), **private evaluation** (chip and EDA companies).

## Graders

Every task is scored by hardware's own ground truth, run outside the environment the agent acted in, and replayed to make sure the grader agrees with itself.

- **Formal equivalence**: is the patched design provably the same behaviour as the reference? If not, here is the counterexample.
- **Mutation kill rate**: does the testbench actually verify the spec? We plant hidden bugs and count how many it catches, so the score is a fraction rather than pass or fail.
- **Co-simulation**: every retired instruction checked against a reference model, not just the final register.
- **Coverage delta**: did the new tests reach the logic they were asked to reach, under a fixed simulation budget?
- **Measured PPA**: frequency, power and area from a real flow; there is no maximum score.

Bounded tasks are hard tasks with a measured pass rate. Unbounded tasks are speed, power and area, where there is always a better result to find.

## Coverage

Tasks cover the whole flow: spec → micro-arch → RTL → verification → synthesis → floorplan → place & route → timing/power → signoff.

- **Spec → RTL**: read a specification, build the module.
- **RTL development**: add a feature to a working design, and flag the change request that's subtly wrong.
- **Bug fix**: fix a defect from a symptom report, without breaking anything else.
- **Verification**: testbenches, assertions, coverage closure, debug and triage, functional safety.
- **Physical & PPA**: synthesis, floorplan, place and route, judged on the numbers the flow returns.
- **Long-horizon flows**: atomic tasks stitched along the real flow, from spec to measured silicon metrics.

## Where the tasks come from

Bug reports and regression logs aren't training data on their own. We turn everything we take in into tasks that replay the same way every time, are scored by hardware's own checks, and are calibrated on frontier models so there is still something to learn.

1. **Planted bugs**: defects injected into real designs, kept only if formally provable. We can make as many as needed, at the difficulty you want.
2. **Open history**: real bugs and real fixes from the public history of open hardware projects.
3. **Licensed history**: bug trackers, regressions, coverage and testbenches licensed from chip teams, including ones that have shut down.
4. **Agent failures**: where today's chip-design agents break, rebuilt on open designs so no customer IP ever ships.

## For data partners

- **Chip teams, shut-down startups, design-services firms**: your design history is valuable as training data. Bug trackers with fixes, regression and coverage history, testbenches and verification IP you own. Exclusive or non-exclusive, upfront or revenue share. This includes the archives of companies that have shut down.
- **AI-for-EDA companies**: we can turn your agent's failures into revenue. We rebuild each failure on open designs so your customers' RTL never leaves, share the proceeds, and hand you back the environment to train your own agent.

What we never take: customer designs from services work, foundry PDKs and cell libraries, licensed third-party IP cores, personal data of employees. Everything is export-screened and reviewed with you before anything leaves your hands.

## Contact

Ask us for a sample task. Tell us which part of the flow you care about and we'll send a sample task, its environment, and how it's graded.

hello@chiptrain.ai. Subjects: "ChipTrain: sample task request", "ChipTrain: license our chip design data", "ChipTrain: agent-failure partnership". Essays: https://chiptrain.ai/essays · Data layer for chip teams: https://chipdev.ai
