TRAIN.md●

TRAIN.md

ChipTrain makes training data and RL environments for AI that designs chips.

Scroll to see what we believe about training it, and what we sell.

 

Coding agents are now good at writing software.

They do much worse on chip design.

On hardware repositories, the same agents score 37 to 58 points lower than on software (Phoenix-bench).

The agent's patch passes all 12 tests.

The patch looks right, and the tests that came with the design agree.

The design is still wrong.

It has a bug that none of the tests look for.

A test suite only checks the cases someone wrote down.

A model trained only to pass tests learns to satisfy the tests rather than the spec.

01234567891011clkfullwr_enrd_endout (ref)dout (model)D0D1D2D0D1D0full+rd+wr≠ ref

Most grading today is someone's opinion.

A reviewer, or another model, reads the patch and says it looks good.

We grade with proof.

A formal equivalence check compares the patched module with the original and finds they behave differently.

The proof also shows the exact cycle where it fails.

In this example, full, read and write all happen at cycle 5, and a stale word comes out at cycle 8.

0%100%pass rate 0%

If no model can solve a task, it teaches nothing.

At a 0% pass rate there is no learning signal. It makes a good paper and useless training data.

If every model solves it, it teaches nothing either.

At 100% there is nothing left to learn.

So we set how hard each task is.

Labs want tasks that models solve at least 2 to 3% of the time (Epoch AI). Evaluations tell models apart best around 20 to 40%.

Every task comes with its measured pass rate.

We run frontier models on every task before it ships, so you know what you're buying.

Models improve quickly, and tasks that were hard stop being hard.

Every public benchmark gets solved eventually.

We keep making harder tasks as models catch up.

When a batch is solved, we build the next one from the same sources and graders.

 

## the fine print · data · benchmarks · RL environments

Training data for AI that designs chips.

We refine chip-design history (real bugs, real regressions, real agent failures) into graded environments with measured pass rates, across the whole flow from spec to signoff. We set the benchmark, publish the results, and license the data to the teams building the next generation of models.

## why chip design

Labs now design chips, and their models still can't verify one.

The labs are becoming chip companies.

Silicon is on the critical path of frontier AI, and labs now design their own accelerators. Coding agents struggle on hardware because the context is long, the tools are specialised, and passing tests doesn’t mean the design is right.

Verification is where models score worst.

On the CVDP benchmark, the best-scoring model passes 6 to 25% of verification tasks (testbenches and assertions), against 34 to 48% for writing RTL. It is the largest share of effort on a real chip project.

The data that would fix it is locked up.

It sits inside chip companies, behind customer contracts, foundry NDAs, third-party IP and export rules. Getting it out safely, and making it gradeable, is the job.

Source: CVDP (Comprehensive Verilog Design Problems), arXiv 2506.14074, Table 2: Claude 3.7 Sonnet, non-agentic pass@1.

## what we sell

We sell a public benchmark, task sets to train on, and environments to train in.

Public benchmark

We pick the tasks, run the models, publish the results. Automated graders do the scoring, so nobody has to judge by eye.

  • +Open harness, reproducible runs
  • +Pass@1, Pass@k and per-gate scores
  • +Refreshed as models saturate it

Calibrated task sets

Held-out tasks across the whole flow. Each task comes with its measured pass rate, so you don't have to calibrate it yourself.

  • +Pass rate measured on every task
  • +Exclusive or non-exclusive licences
  • +New tasks after the last batch is solved

RL environments

Executable worlds for training agents on real silicon work: pinned EDA toolchains, MCP tool servers, and a grader outside the sandbox.

  • +Sim, lint, synth, formal, coverage, waveforms
  • +Dense, hard-to-game reward
  • +Works with your own training harness

Private evaluations

Under NDA, on your own RTL: a contamination-free measurement of how AI agents really perform on your design.

  • +For chip and EDA companies
  • +For AI-for-EDA teams proving their agent
  • +Your design never leaves your control

THREE WAYS IN

Labs and model builders

Non-exclusive licence

Held-out task sets and environments, calibrated and refreshed. The fastest way to start.

One lab, built to spec

Exclusive build

Environments built around your harness, your task mix and your difficulty band. Only you get them.

Chip & EDA companies

Private evaluation

Your own RTL, under NDA, measured without contamination.

## graders

The grader is the hard part, so we use hardware's own checks.

Every task is scored by hardware's own ground truth (proofs, mutants, reference models, the flow's numbers), run outside the environment the agent acted in, and replayed to make sure the grader agrees with itself.

  • Formal equivalenceIs the patched design provably the same behaviour as the reference? If not, here is the counterexample.
  • Mutation kill rateDoes the testbench actually verify the spec? We plant hidden bugs and count how many it catches, so the score is a fraction rather than pass or fail.
  • Co-simulationEvery retired instruction checked against a reference model, not just the final register.
  • Coverage deltaDid the new tests reach the logic they were asked to reach, under a fixed simulation budget?
  • Measured PPAFrequency, power and area from a real flow. There is no maximum score, so it never runs out of room.

BOUNDED

Hard tasks with a measured pass rate. The pass rate is what you are paying for.

UNBOUNDED

Speed, power and area, where there is always a better result to find.

## coverage

Tasks cover the whole flow, from spec to signoff.

  1. Spec→
  2. Micro-arch→
  3. RTL→
  4. Verification→
  5. Synthesis→
  6. Floorplan→
  7. Place & route→
  8. Timing / power→
  9. Signoff

Every stage, including physical design.

Spec → RTL

Read a specification, build the module.

RTL development

Add a feature to a working design, and flag the change request that's subtly wrong.

Bug fix

Fix a defect from a symptom report, without breaking anything else.

Verification

Testbenches, assertions, coverage closure, debug and triage, functional safety.

Physical & PPA

Synthesis, floorplan, place and route, judged on the numbers the flow returns.

Long-horizon flows

Atomic tasks stitched along the real flow, from spec to measured silicon metrics.

## sources

Where the tasks come from

Bug reports and regression logs aren't training data on their own. We turn everything we take in into tasks that replay the same way every time, are scored by hardware's own checks, and are calibrated on frontier models so there is still something to learn. All four sources go through the same graders.

  1. 01Planted bugsDefects injected into real designs, kept only if formally provable. We can make as many as needed, at the difficulty you want.
  2. 02Open historyReal bugs and real fixes from the public history of open hardware projects.
  3. 03Licensed historyBug trackers, regressions, coverage and testbenches licensed from chip teams, including ones that have shut down.
  4. 04Agent failuresWhere today's chip-design agents break, rebuilt on open designs so no customer IP ever ships.

chip teams · shut-down startups · design-services firms

Your design history is valuable as training data.

AI gets better at chip design by learning how a design was made correct, more than from the final RTL. That record lives in bug trackers with their fixes, regression and coverage history, and the testbenches and verification IP you own. We license it exclusively or non-exclusively, for an upfront fee or a revenue share. This includes the archives of companies that have shut down.

License your data →

AI-for-EDA companies

We can turn your agent's failures into revenue.

The places where your agent breaks are what labs want to train on. We rebuild each failure on open designs, so your customers' RTL never leaves. We share the proceeds with you and give you the environment to train your own agent.

Partner with us →

WHAT WE NEVER TAKE

  • ×Customer designs from services work
  • ×Foundry PDKs and cell libraries
  • ×Licensed third-party IP cores
  • ×Personal data of employees

Everything is export-screened and reviewed with you before anything leaves your hands.

## essays

Essays on these ideas

Ask us for a sample task.

Tell us which part of the flow you care about. We'll send a task, its environment and how it's graded, so you can judge the quality before we talk.

Request a sample task

---

end of file · last edited Nov 2026 · read as TRAIN.md · llms.txt · hello@chiptrain.ai · data layer for chip teams? chipdev.ai

© 2026 ChipTrain