nextjob
Ingests job ads by paste or forwarded alert, scores each against a written brief, and gates every prompt change behind golden-set evals.
Problem Solved
A job search that scores itself. An AI ingestion pipeline that reads job advertisements, scores them against a written brief, and refuses to hide its own mistakes.
Search criteria are written as a document first, then applied mechanically to every advertisement received. The filters and scoring rubric live in exactly one module, as data. That module is interpolated into the model's prompt and consumed by the deterministic code paths, so the model's instructions and the program's checks cannot drift apart — they are the same text.
Key Features
Belt-and-Braces Dual Judging
Every ad is judged twice — the model catches nuance a regular expression can't, like a title that oversells seniority or an "AI Architect" posting that's really a licence rollout, while deterministic code catches the unambiguous rejects a bad generation might wave through. Findings are unioned rather than trusted from either side alone, and rejections are recorded with their reason and shown in a dimmed lane — never deleted.
Aggressive Input Cleaning
The first genuine forwarded alert extracted zero listings from an email that plainly contained five, because they were buried in zero-width padding, image alt markers, and tracking URLs hundreds of characters long. Cleaning before extraction cut that email by 83% and all five listings appeared; known job links are canonicalised rather than stripped, since the URL is the listing's identity.
Identity via Requisition Reference
Deduplication originally keyed on model-generated title plus company text, which needed two independent generations to agree word for word — and broke when a fallback rule told the model to substitute the agency name whenever the client was hidden. The fix keys identity on the employer's own requisition reference, quoted in the ad and previously discarded as title noise, and trusts it only when it looks like one: a URL identifies a posting, a reference identifies the job.
Three-Tier Golden-Set Evals
A golden-set harness scores every prompt change in three tiers: hard gates on rule-id set equality, listing counts, and null-where-silent; a banded tier that asserts score as a range rather than an exact integer, because scores flap between runs; and a reporting tier that surfaces per-rule precision and recall instead of a single pass/fail. The model-free suite alone once caught four literal backspace control characters standing in for word boundaries in two regular expressions that could never fire — the build, linter, and type checker were all green.
Fail-Closed Behaviour
The access gate fails closed — a missing secret returns 503 rather than falling open — and ingestion degrades rather than throws, with a blocked fetch returning a reason and the interface asking for a paste instead. Every inbound email is stored before parsing, notifications are marked sent only on success, and source text is retained so a re-score never needs the network.
Architecture
Architecture Overview
Job ads arrive by paste or forwarded email alert. A cleaning pass runs before extraction — canonicalising known job links rather than stripping them, since those URLs are part of a listing's identity — before the ad is scored twice: once by the model against the brief, once by deterministic code against the same rubric. Findings from both paths are unioned rather than reconciled by trust, and every filter failure carries an enumerated, machine-readable rule id rather than prose.
- Ingestion: paste or forwarded email alert, stored before parsing so a parse failure is recorded against the message rather than losing the mail
- Cleaning: strips zero-width padding, image alt markers, and long tracking URLs while canonicalising job links, cutting one real inbox digest by 83%
- Judging: model and deterministic code score every ad against the same rubric-as-data module; findings are unioned, never single-sourced
- Identity: dedup keyed on the employer's requisition reference quoted in the ad, not model-generated title-plus-company text
- Evals: a three-tier golden-set harness — hard gates, banded score ranges, and per-rule precision/recall — run against every prompt change
- Delivery: notifications marked sent only on success, so an outage leaves a role retryable rather than announced twice
What Is Not Finished
The eval suite is deliberately red: golden labels describe the correct answer, not current behaviour. Four of five filter rules have no real-traffic coverage yet — the compensation rule, the heaviest weight in the rubric, has never executed in production. Extraction is non-deterministic on degenerate input: a body-less email passed on one run and failed on the next. Scheduled board scanning is not built; it needs credential storage and anti-bot handling that hasn't been designed.
Built and operated by one person, in production, against a real job search. Screens and repository available on request — the running instance holds live personal data and stays private.
Key Decisions
Rubric as Data, Not Prose
The scoring rubric lives in exactly one module, as data, and that same module is interpolated into the model's prompt and consumed by the deterministic code paths — so the model's instructions and the program's checks are structurally the same text and cannot drift apart.
Unioned Findings, Never Single-Sourced
Neither the model's judgement nor the deterministic checks are trusted alone. Findings from both are unioned, and rejections are kept visible with their reason rather than silently discarded.
Enumerated Rule IDs at the Extraction Boundary
Filter failures carry a machine-readable rule id, not prose, and that id is an enumerated type enforced at the extraction boundary — so the model cannot invent a rule that doesn't exist.
Dedup on Requisition Reference, Not Generated Text
Collapsing two real roles into one is worse than carrying a visible duplicate, so identity is keyed on the employer's own requisition reference rather than model-generated title-and-company prose, and a reference is only trusted when it looks like one.
Screenshots
Interested in something similar?
We scope, design, and ship data and AI tools that solve the problems your team actually has.
Let's Talk