Benchmarks

Benchmarks

Every number we publish

BM-01

How our router, our placer and the whole pipeline compare with other tools, and with the people who drew the boards. Every chart says who judged it, when, and how to check it yourself.

At a glanceOurs against the next best, on each bench.

Routing · LightcycleOther people’s boards, routed again and judged by KiCad.

External · LG AI Research

PCBWorld, small boards

Clean pass on 99 boards, under LG’s own rules and scoring.

KiCad 9.0.8 · 17 Nov 2026LG: best of fiveOurs: one run
Table

External · LG AI Research

PCBWorld, medium boards

The 10 boards of LG’s medium test, 11 to 26 nets. Every LLM agent scores 0.

KiCad 9.0.8 · 17 Nov 2026Large set: 9 of 10, the authors 9LG publishes none
Table

External · LG AI Research

Clean pass against time

PCBWorld’s small boards. Up and to the left is better. Each method’s time is on its own hardware.

Ours: 1.3 s a board, an Apple M5 laptopLG’s times: LG’s hardware
Table
PCBWorld in fullHow it was run, where it missed, and what changes the number

PCBWorld takes open-source KiCad boards, strips their tracks, and asks a router to lay them again. KiCad’s own rule check scores the result. We rebuilt LG’s boards and LG’s scoring, and ran Lightcycle through them: one run a board, against LG’s best of five.

LG’s rows are Table 3 of the PCBWorld paper, the best of five rollouts a board, with Freerouting and the RL routers averaged over four seeds. Lightcycle’s are one deterministic run a board, with exact pad margins on, under each board’s own rules as LG prepared them. Those rules let a track pass inside another pad’s solder mask opening. soldermask keeps it out by default, and with that the small set is 0.99 (98 of 99).

How it was run

Boards
LG’s own preparation scripts, unmodified, on the public PCBench collection (dec3be7) with KiCad 9.0.8, the version LG used. They produced all 99 small boards and all 20 medium and large ones.
Rules
Each board’s own project file as LG prepared it: its net classes, clearances, via sizes, and the checks LG switches off.
Scoring
KiCad 9.0.8’s rule check at error severity under that project, plus KiCad’s count of unmade connections. That is LG’s clean-pass definition (eval/metrics.py). On the authors’ own copper it passes 99 of 99 small, 10 of 10 medium and 9 of 10 large boards.
Our router
Lightcycle 0.1.6 with exact pad margins, at each board’s default net class, on the author’s placement with every part fixed. It lays tracks and vias only, as LG’s routers do.
Runs
One deterministic run a board. LG keeps the best of five rollouts for each board.
Not tuned here
PCBench boards are kept out of the benches Lightcycle is developed on. Our first run, on 16 Nov, showed five ways soldermask misread a KiCad file, such as a round outline drawn off centre. We fixed those on 17 Nov and ran every board again. The router was not tuned on them.
Speed
1.3 s a board on the small set, 2.2 s on the medium and 8.6 s on the large (median 4.3 s, slowest 32 s): the whole build, read to written file, one core a board on an Apple M5 laptop. LG’s own times, on its hardware: PPO 1.8 s, Freerouting 7.1 s, KiCadRoutingTools 0.7 s on the small set.

Where it missed

The misses

Across all 119 boards, Lightcycle’s copper drew no rule error. Every small and medium board passes.

  • One large board with one connection open, of 52 nets
  • With soldermask’s own mask rule, one small board too: a microphone whose pins leave only through its ground pad’s mask opening

With a ground pour

soldermask also pours a ground plane on both layers by default. LG’s boards carry no pour and LG’s routers lay none, so the headline is without it. With it, the small set is 0.99 (98 of 99), and the medium and large sets do not move.

  • One small board: a header’s ground pin between two others, which the pour cannot reach

Every number

MethodKindSmall CPRoutabilityTime, sMedium CPRoutabilityTime, s
Lightcycleours1.001.001.331.001.002.15
PPO w/o finishRL0.940.993.050.420.8714.44
PPORL0.860.951.830.450.8510.77
GRPORL0.850.983.540.100.6911.20
PPO (terminal)RL0.820.952.580.380.8112.60
Freeroutingrouter0.800.917.090.781.009.94
KiCadRoutingToolsrouter0.740.940.650.200.863.27
GPT-5.4LLM agent0.650.91231.150.000.62865.83
Qwen3.5-397BLLM agent0.340.7581.820.000.51222.29
GPT-5.4-nanoLLM agent0.300.73100.760.000.59510.25
GPT-5.4-miniLLM agent0.280.7256.850.000.61422.06
OrthoRouterouter0.020.532.200.000.449.30

CP is clean pass. Lightcycle’s routability is KiCad’s count of connections made over those the bare board needs, which can differ slightly from LG’s pad-group count. Times are each method’s own hardware. Source: LG AI Research, PCBWorld: A Benchmark Environment for Engine-Grounded PCB Design Automation (KDD 2026 workshop), and its repository at 5773825. Our run: soldermask f555c96, Lightcycle 0.1.6, KiCad 9.0.8, 17 Nov 2026.

Public · soldermask-bench

Two layers, by board size

186 public tasks, each a published board with its copper removed. Pass rate by board size.

Lightcycle 0.1.5, 16 Nov 2026, KiCad 10.0.8Others 30 Sep, KiCad 10.0.6Re-run: soldermask-bench
Table

Public · soldermask-bench

Public and hidden sets

The 113 hidden tasks are scored on request, so no router can tune to them.

Referee 0.3Lightcycle 16 Nov 2026 · others 30 Sep

Held out · 249 boards

Held-out boards

Boards none of our routers was developed on. The test plan was written before the first route.

600 s a boardLightcycle 16 Nov 2026 · others 4 OctRunner: tools/bench_v2.py
Table

Public boards · PCBench

PCBench, 366 boards

PCBench boards with no rule errors before routing, stripped of copper and routed again. A different set and judge from PCBWorld.

346of 366 route clean, 94.5%
Referee 0.3 · KiCad 10.0.8Lightcycle 0.1.5 · 16 Nov 2026Rebuild: pcbench/build.py

Held out · 141 boards

Four layers

Boards their authors routed on four layers, the test half frozen before the run.

Referee 0.3 · 600 sLightcycle 16 Nov 2026 · others 5 OctPaired: 44 to 1
Table

Lightcycle’s rows are Lightcycle 0.1.5 on 16 Nov 2026, exact margins off as the product ships, judged by KiCad 10.0.8 because 10.0.6 can no longer be installed. With exact margins on: public 173, hidden 104, held out 219, four layers 98. The other routers’ rows are their September and October runs on KiCad 10.0.6.

Placing · QuorraWhere the parts go, judged by whether the board routes.

Against a published placer

Quorra against Cypress

Cypress, from NVIDIA and Cornell (ISPD 2025), in its shipped setup. 60 held-out boards, six layouts each, the best kept.

Both routed by Lightcycle · KiCad DRC 0.127 mm8 Nov 2026p = 1.5e-5
Table

Against people

Quorra against the boards’ designers

705 open-hardware two-layer boards: how often a placement routes clean.

5 Sep 2026The router of that day
Table

We lose here The layout gym

Boards of 4 to 10 layers, under their own rules

26 open-source KiCad boards, each laid out again under its own hole rules, net classes and custom rules. The development split: the boards our fixes were made against. The test split has not been run.

KiCad DRC under each board’s rules15 and 16 Nov 2026Exact margins are off in the product
Table

The whole pipelineA sentence in, a board that passes out.

60 circuits, four tiers

Claude’s designs, built end to end

The 60 circuits Claude wrote over MCP in September, rebuilt with no model in November. A pass: every part found, no ERC error, every net routed, KiCad clean.

The build’s verdictRunner: python -m soldermask.evaluateSaturated for these designs
Table

We lose here The hard set

Prompts that break text-to-board tools

24 prompts: four layers, USB and Ethernet pairs, dense copper. 17 designed by our own design stage. The prompts stay private, so the set stays held out.

Three judges12 and 13 Nov 2026
Table

Every design we have kept

305 saved designs, replayed

Every design this repository has kept, from September to November, built again with no model.

277of 305 pass
The build’s verdict · KiCad DRC16 Nov 2026The replay pool is not yet public

Head to head · siliXon’s own test

A small model wins siliXon’s test

siliXon’s three published prompts, run on soldermask the way they ran them: Claude Code at medium effort, one tool connected. Three runs a prompt, the median shown. Hover a bar for its cost and errors. Below the charts, Haiku’s board from each prompt’s median run, as KiCad renders it.

SD logger: the board Haiku 5.5 designed through soldermask, rendered in KiCad
SD logger“An ESP32 board that logs temperature to an SD card.” 0.4 min, 0.35¢, 27 parts, passes with 0 errors.
Soil sensor: the board Haiku 5.5 designed through soldermask, rendered in KiCad
Soil sensor“A garden sensor that texts me when the soil is dry.” 0.5 min, 0.35¢, 31 parts, passes with 0 errors.
Bike light: the board Haiku 5.5 designed through soldermask, rendered in KiCad
Bike light“A bike light that gets brighter as I brake.” 1.2 min, 1.9¢, 27 parts, passes with 0 errors.

Haiku 5.5 with soldermask built all nine boards in 0.4 to 1.9 minutes, for 0.4 to 2.7 cents of Claude usage each. Seven came from Slipspace, which builds a board from circuit blocks that are each checked before they are used. All nine pass soldermask’s verdict with 0 KiCad errors under JLCPCB’s published limits. Since 18 Nov that verdict also fails a circuit that would not work as wired, such as an LED driven past its rating. Three of the nine boards from our first run, on 17 Nov, had one, so these are from a new run. Fable 5.2 with siliXon took 5.5 to 8.2 minutes and $1.25 to $1.64, not counting siliXon’s own compute or plan. soldermask runs on your own machine, so Claude usage is the whole bill. siliXon tested Fable only. The boards are not the same designs, and none has been made yet.

Theirs: siliXon’s post, 10 Nov 2026, self-reportedOurs: soldermask df37fd6, 18 Nov 2026, on a laptopThree runs a prompt
Table

Simulation

Faults planted, faults found

144 September designs, one realistic fault at a time. The bars were set before the run. False alarms: 3 designs, 2.1%.

8 Nov 2026Runner: tools/sim_bench.py
Table

Other judgesA contest, and another group’s router.

Contest · comma.ai’s PCBGolf

Nine entries, 4th of 20

Score: the board’s bounding volume, plus 50 a via and 5,000 a copper layer. Lower is better. The last: five boards of one layer each, no vias.

Scored by comma.aiClosed 13 Oct 2026The entry
Table

Photonics · Mendicant

A 16 × 16 Clements mesh

Nets routed by LiDAR, an open-source photonic router from another group. Apollo’s row is its published table.

Judged by LiDARSep 2026Mendicant
Table

soldermask-bench is public now. The other runners are in the soldermask repository, which opens with the release.

Their rules.Our numbers.

01 / 02

Request access

The alpha is private. Leave an address and an invite is sent by hand.

The code is in a private repository. With a username, you are added to it.