How it works
Awesome Jev is a list that judges itself. Five stages run end to end; the only place a model is called is stage three, and the only place a human decides anything is an override with a written reason.
Pipeline
- 1
Discover
Candidates come from the GitHub API: code and repository search for Jev and TypeSafe System One, plus the repositories other Jev lists already link to. Forks are dropped. Everything is appended to a candidate file, so discovery is repeatable and auditable.
- 2
Enrich
For each candidate the pipeline pulls the material a judgment needs: README excerpt, package manifests, declared dependencies, and repository metadata. Results are cached per repository so a rerun does not re-fetch what has not changed.
- 3
Judge
One Jev call per repository answers the whole question set at once: is this genuinely about Jev, does it use Jev at runtime, is it a meta list, is it a reimplementation, which category and decision pattern, and how substantial, documented, and novel it is. Raw judgments are immutable and append-only, tagged with model id and question set.
- 4
Curate
Policy lives in code, not in the model. Thresholds turn probabilities into listed, review, or excluded; weights turn the 0-3 scales into a composite score. Changing a threshold regenerates the index without re-running inference.
- 5
Publish
The curated file is the single source for both the README and this site. The site is a static export with no server logic and no API keys: every number you see here was computed at build time.
Policy in force
These are the exact values applied to the current index, read straight from the curated data generated 2026-09-18 16:44 UTC.
| Key | Value |
|---|---|
| qset | v2 |
| gate.listed_min | 0.5 |
| gate.review_min | 0.3 |
| gate.category_conf_min | 0.5 |
| gate.substance_min | 0.5 |
| gate.meta_list_min | 0.7 |
| gate.reimpl_min | 0.7 |
| weights.substance | 0.45 |
| weights.docs | 0.25 |
| weights.novelty | 0.3 |
| star_bonus_max | 0.1 |
| readme_top_per_category | 15 |
Calibration
A hand-labelled gold set is the test suite for the judgments. The sweep below moves the genuine-probability gate and reports how the resulting listing decision compares to the human labels on 80 repositories, question set v2.
| Threshold | TP | FP | FN | TN | Precision | Recall | F1 |
|---|---|---|---|---|---|---|---|
| 0.20 | 65 | 3 | 5 | 7 | 0.956 | 0.929 | 0.942 |
| 0.30 | 64 | 2 | 6 | 8 | 0.970 | 0.914 | 0.941 |
| 0.40 | 63 | 1 | 7 | 9 | 0.984 | 0.900 | 0.940 |
| 0.50 | 59 | 0 | 11 | 10 | 1.000 | 0.843 | 0.915 |
| 0.60 | 55 | 0 | 15 | 10 | 1.000 | 0.786 | 0.880 |
| 0.70 | 49 | 0 | 21 | 10 | 1.000 | 0.700 | 0.824 |
| 0.80 | 41 | 0 | 29 | 10 | 1.000 | 0.586 | 0.739 |
| 0.90 | 35 | 0 | 35 | 10 | 1.000 | 0.500 | 0.667 |
- Category agreement
- 92.4%
- Gold set size
- 80
- Question set
- v2
- Computed
- 2026-09-18 03:58 UTC
Calibration report
The gold set is 80 repositories labelled by hand from README, manifests and code-search evidence: a first pass by an assistant, every label reviewed by the maintainer. It was sampled in strata so the interesting cases are over-represented: noise that keyword search drags in, repositories Jev scored near the gate, and genuine projects spread across category, stars and language. Two fields are labelled separately on purpose: genuine asks whether the repository is about Jev at all, substance how real it is. A one-line scaffold with a TypeSafe dependency is genuine and empty.
The full write-up, including five failure cases and what each one says about the evidence or the question set, is in docs/calibration.md.
Threshold
The listed gate is genuine ≥ 0.5. On the gold set no repository a human called noise scores 0.5 or higher, so lowering the gate from 0.6 buys recall without paying precision; 0.4 admits the first noise. A separate substance floor keeps empty scaffolds in review even when they pass the gate. The sweep above over-states the miss rate: repositories near the gate were deliberately over-sampled.
Category
Jev and the human label agree on 92% of 66 categorised repositories. Where they disagree:
| Human | Jev | Count |
|---|---|---|
| SDKs and Clients | Research and Evals | 1 |
| Agent and Dev Tooling | Research and Evals | 1 |
| Integrations | Agent and Dev Tooling | 2 |
| Integrations | SDKs and Clients | 1 |
Reported confidence tracks accuracy, which is what makes the category-confidence threshold in the policy meaningful:
| Confidence | n | Accuracy |
|---|---|---|
| 0.0 – 0.5 | 4 | 50% |
| 0.5 – 0.7 | 9 | 78% |
| 0.7 – 0.9 | 4 | 75% |
| 0.9 – 1.0 | 49 | 100% |
Substance
Jev's 0–3 substance score against the human one: Spearman 0.84, mean absolute error 0.27 (n=54). Good enough to rank within a category, not to be read as a grade.
Where Jev and the humans disagree on the gate
Every disagreement at the current gate. Most are repositories whose README says nothing about Jev while the code uses it; README-only evidence cannot see those.
| Repository | Jev | Human | Why |
|---|---|---|---|
| vercel/eve | 0.03 | genuine | ships Jev as default eval model in experimental evaluate path; README does not say so |
| danieljvdm/effect-agent | 0.08 | genuine | package.json depends on typesafe, README silent; same class as vercel/eve; rhc confirmed true. ★112 |
| monotykamary/pi-fabric | 0.10 | genuine | code-search hit only, README silent; rhc confirms related project. ★231 |
| BunsDev/typesafe-ui | 0.14 | genuine | shadcn UI kit for TypeSafe AI apps; no API calls in evidence; rhc confirmed true |
| cgarrot/pi-knowledge-fabric | 0.16 | genuine | README lists "Jev/TypeSafe (bounded judgments)" as an optional adapter, disabled by default; jev gate 0.16 — same class as vercel-labs/ai-cli |
| ndolinschi/cartshield | 0.26 | genuine | Next.js stub + SPEC.md; README one line |
| ASpooky/jev-kata | 0.39 | genuine | kata scaffold with src/ and typesafe dep; README one line |
| ndolinschi/spendbrake | 0.42 | genuine | stub dashboard; README 150 chars |
| tylorsaling-source/typesafe_use_cases | 0.43 | genuine | no README; demos + tests import typesafe_sdk (pyproject dep) |
| vercel-labs/ai-cli | 0.46 | genuine | CLI can run Jev as evaluation model; README mention is thin |
| AliceRoselia/Typesafe_chess_eval | 0.49 | genuine | single-file eval of Jev at chess; jev gate 0.49 |
Question set v1 → v2
Measured on the 33 gold repositories judged under both sets. v2 removed the "official" category (now an owner flag in code) and rewrote the gate question to name what counts: uses, wraps, evaluates or reimplements Jev.
| Metric | v1 | v2 |
|---|---|---|
| mean genuine, human true | 0.69 | 0.85 |
| mean genuine, human false | 0.03 | 0.03 |
| gate precision @0.5 | 1.00 | 1.00 |
| gate recall @0.5 | 0.83 | 0.93 |
| gate F1 @0.5 | 0.91 | 0.97 |
| category agreement | 0.86 | 0.90 |
| mean category confidence | 0.87 | 0.89 |
Cost and latency
- Calls stored
- 723
- Input tokens per call
- 3154
- Latency p50 / p90
- 256 / 498 ms
- Estimated spend to date
- $0.10 at $0.042/1M input
Corrections
Judgments are wrong sometimes, and the wrong ones are the interesting ones. Open an issue with the Jev got it wrong template, or read the pipeline yourself in the repository.