AI CodeCompare

How we test

Every comparison on this site is built in three layers, and each page tells you which layers it is standing on. This page is the method in full — it is also our commitment: if we ever publish something that does not follow it, the method is the thing to hold us to.

Layer 1 — the data layer

Pricing tiers, free-tier terms, quotas, platform coverage and product form, read from the vendor's official pricing or documentation pages. Every profile links its source and shows the date it was read, and the data layer is re-checked monthly, because this category reprices constantly — in 2026 alone, Copilot changed its metering, Windsurf changed its name, and Cursor changed its entire plan line-up.

Where a vendor does not publish a number — Cursor's free request limits, Codex's free usage limits — the profile says “not published”. We do not estimate quotas, and we treat any third-party figure as third-party: labeled as such, or left out.

Layer 2 — the hands-on benchmark

Every tool with a standing free tier is run through the same frozen benchmark repository (a TypeScript/Node project with a Python module, fixed at a known commit), with a fresh install, default settings, the vendor's default or automatic model selection, and the free plan. Same repo, same task wording, same rules — the tool is the only variable.

The core task set:

  1. Code completion — 20 preset completion points (loops, conditionals, async code, type definitions); scored by how many completions are accepted unmodified. Editor-form tools only.
  2. Single-file implementation — implement an LRU cache with O(1) get/put against a provided interface; scored by hidden unit tests, with one repair round allowed.
  3. Bug fixing — the repo ships with three seeded bugs (an off-by-one, a race, a regex error); the task is to make the failing suite pass without changing the tests.
  4. Cross-file feature — add a GET /health endpoint returning JSON, plus a test for it, plus README documentation; scored on all three deliverables and the test run.
  5. Code Q&A — five fixed questions about the benchmark repo itself, scored against a frozen answer key.
  6. Refactoring — remove duplication from a given 60-line function without changing behavior or the public API; the original suite must stay green.

For each task we record wall-clock time, the number of interaction rounds, the quota consumed (usage-panel screenshots before and after), the unit-test pass count, and the size of the diff. Scores come from the test suite, not from impressions; short quality notes (readability, out-of-scope changes, error clarity) are labeled as observations, on a fixed 1–3 scale.

Replit is the exception that proves the setup: as a web app builder rather than a local editor, it runs an adapted set — build-and-publish tasks in the browser — and its profile says so, so its results are never mixed into the same scoring table as the local tools.

Screenshots and screen captures are our own, taken during our runs. We do not use vendor marketing images or other sites' captures as evidence.

Layer 3 — the verdict layer

Each head-to-head page ends in a written verdict that is written for that pairing — which tool suits which budget, form factor and way of working — citing the data layer today and the benchmark layer once its runs are complete. Verdict text is never assembled by pasting profile paragraphs together, and a verdict may only claim what the layers under it support: a pairing between two untested tools is labeled a data comparison, in the page's benchmark block, in plain sight.

Rules we hold ourselves to

Current status

As of October 2026: the data layer is live for all eight tools. The benchmark is scheduled for the six free-tier tools (Copilot, Cursor, Windsurf/Devin Desktop, Kiro, Codex, Replit); results will appear in each tool's profile and in the relevant comparisons as the runs complete. Claude Code and Tabnine are data-only, because neither offers a standing free tier for the protocol to run on.