Projects

Site-safety systems fail in one of two ways. Pure rules are fast and predictable but cannot tell a genuine near-miss from a tracking glitch, so operators drown in false alarms. Pure LLM pipelines can reason about context but are too slow and costly to run on every frame, and will state a distance that sounds confident and is simply wrong - unacceptable when the question is how close the forklift got.

Two planes and a referee neither can override. The fast plane detects and tracks people and vehicles, projects each to floor coordinates through one of three calibration methods, and fails safe: a camera whose calibration error exceeds 2.0 px keeps zone rules but switches off distance and speed rules. The only thing that crosses planes is a Pydantic-validated TriggerEvent plus the tracking window around it; the agent never sees video. Prime Agent runs headless over JSON-lines RPC, one fresh process per incident, and must answer from code it executes in its REPL rather than from prose. A feasibility spike showed its own turn limit did not stop a runaway task, so the real limit is an external 210 s timeout and kill in the adapter. The Band-3 gate recomputes distance, speed and zone dwell independently; any disagreement, timeout or malformed output becomes a needs_review incident with the evidence kept, never a dropped one. Redis Streams, Postgres, FastAPI with WebSocket push and a React dashboard carry the result; Terraform for AWS, GCP and Azure is validated in CI and deliberately never applied.

Against the real Prime Agent CLI, 26/30 hand-labeled incidents passed first time (86.7%); all four misses were the same timeout and completed correctly on retest, so the limit was raised on that evidence. Classification agreement with the labels was 82.1%, at about $0.002 and 43-56K tokens per incident. The gate earned its place on the live demo: the agent called a real aisle intrusion a false positive, and after a zone check was added to the gate the agent repeated the mistake and the gate caught it, sending the incident to review. The recorded transcript showed the agent had misread a rule id, fixed by putting rule definitions in the prompt. YOLO11s runs 35 FPS single-stream at 1080p; for three streams both the rested (30.2 FPS) and thermally throttled (9.5-13.2 FPS) numbers are published. 191 Python tests plus 6 dashboard tests, green in CI. Still unproven: inputs are looped stock clips, the demo calibrations are estimates rather than surveys, the gate cannot catch a plausible wrong judgment between allowed classes, and speed verdicts are not gated.

Raw movement data is not intelligence: positions are noisy, misclassified, and easy to over-read, and LLM agents over maps routinely invent intent, causes, and completeness that the data does not support. Missing data gets silently treated as zero, and "all vessels at the port" claims smuggle in scope the source never had.

Deterministic analytics before any AI: six versioned component metrics (vessel count, moving fraction, dwell estimates, entry/exit) computed over a canonical observation model that rejects invalid data instead of clamping it, inside nine reviewed geofences with per-vertex provenance and a named human review owner. The agent is evaluator-gated end to end: a typed-intent parser abstains rather than guessing scope, tools read only manager snapshots, a claim evaluator rejects causal/threat/predictive language, requires provenance on every claim, and verifies every number against the metrics — and the generator renders accepted claims only, structurally. Every live source went through a primary-source terms admission gate (AISStream risk-accepted with no ToS, adsb.lol under ODbL, CBP keyless; OpenSky rejected on its own written terms; rail deferred on evidence), and facility-level signals (border wait times) enter as a separate FacilityMetric model so entity data is never polluted.

341 offline deterministic tests, CI-green with zero credentials and zero network. Groundedness evaluation: 25/25 labeled scenarios (fabricated numbers rejected, causal/threat/predictive language rejected, scope overreach rejected, out-of-domain questions abstain). Detector evaluation: precision 1.0, recall 0.75, zero false positives with known failures documented. Replay performance profiled against documented budgets (render mean ~3ms, zero idle renders, 0% heap growth) — the profiler caught a real cursor bug synthetic tests had missed. Live at pcschmidt.github.io/chokepoint_app with a PWA offline shell, plus a Docker Compose observability stack (Prometheus + Grafana, bounded-label metrics, redacted JSON logs). The CBP live layer was verified against the real feed (85 crossings) and the demo honestly shows Suez as coverage-UNKNOWN rather than zero.

AI coding agents fail predictably on multi-session projects: they declare features "done" before tests run, forget decisions across context resets, grade their own work generously, and drift from the original scope. None of these are prompt-quality problems — you cannot word-craft your way out of a structural failure.

Built infrastructure, not prompts. Gates are a project-defined YAML dependency graph enforced by shell hooks that exit 2 to block execution. A Gate Evaluator subagent runs in a fresh context, told "you did not produce this work," and blocks any gate scoring below 7.0. Three schema-validated JSONL memory types persist patterns, session events, and predicted-vs-actual calibration across resets. Every gate transition, block, and verdict is logged to queryable telemetry, and enforcement relocates to the git/CI boundary for non-Claude platforms.

Validated the core thesis by experiment: an agent scored its own work 5.5/10 while an independent evaluator scored the same work 2.5/10 — a 3.0-point gap the harness closes mechanically. 240 assertions across 18 suites, all three recipes verified end-to-end. Dogfooded on two production projects, where a gate caught stub citation code an agent had marked complete and forced it to be rebuilt as real infrastructure before it could ship as technical debt.

The data that moves the AI, defense, and energy build-outs — DoD awards, EDGAR filings, NRC licensing, hyperscaler capex, export-control actions — is public but fragmented, slow to read, and disconnected from its market meaning. Serious investors don’t lack data; they lack the time to aggregate it and the tooling to connect a raw event to the security it affects.

Built a full production stack: a Next.js reader on Vercel, a FastAPI intelligence engine, and Supabase (Postgres + pgvector). A per-source scheduler pulls four live adapters (USAspending, SEC EDGAR, arXiv, NRC) through a retry/backoff fetcher with DB-level dedup and a circuit breaker; records resolve into a bitemporal entity-resolution graph; change-detection plus graph-grounded RAG generate a cached BLUF brief per desk; an eval gate checks for fabrication and grades each item’s attribution before publish. A daily cron ingests once and publishes all three desks.

Live end-to-end and publishing daily from live data. The entity resolver holds precision 1.0 / recall 1.0 with zero false links on a golden set, rendering each item’s companies as clickable entity chips and flagging cross-desk convergence — a company appearing on two or more desks — as the core signal. An interactive Convergence Graph makes the entity network visible, with the cited stories behind each connection and a federal-funding overlay. Payments are built but not yet wired to a processor; turning on subscriptions is the next step to public launch.

Autonomous research agents now write papers that clear automated review bars, which leaves two gaps: almost nobody can afford to run them (the comparison system reports about $3,800 and 2 to 3 days per task), and nobody independent checks whether what they write is true. A fluent review with plausible citations is easy to produce and hard to trust.

Three layers: a cheap-by-default judge, an auditor, and a research agent that works under a hard budget. Each claim is one assertion with one quote; every other sentence is classed as synthesis (which must name the claims it follows from) or a transition (which must state no fact), and each is checked. A cheap judge answers first; a confident "no" on a claim or statement is put to a reference judge (Claude), an unsure reference makes the item "unsettled" rather than failed or passed, and a disagreement between judges is shown, not hidden. Every model call goes through a ledger and a budget that raises before a limit is crossed, the key is held in memory and written nowhere, and no component grades its own output. Test sets and code were frozen and hashed before the measurement they serve. Built with Claude Code under the Meridian harness: gated increments, human approvals, and an independent evaluator scoring every review.

A literature review costs $0.11 to $0.21 and a review paper $0.53 to $0.77 in model fees; the whole project spent $16.05. The audit caught 50 of 52 planted faults, and the sentence checks caught 15 of 21 real defects that earlier checks had missed (0 of 21), while flagging 14 of 69 sound items. No cheap judge met the 97% agreement bar against the reference (89% claims, 95% sentences, 89% statements), and the reference itself is unsure on about a fifth of verdicts. The headline negative: making the checks stricter made the light more honest but did not make the papers measurably better; independent readers still found claims that say a little more than their quotes. The six showcase papers are published as they are, three red and two amber, with their deficiencies listed. The research loop did not beat either parent baseline. Not shown: a new user completing a run, or labels from a person after Increment 6.

A raw difference between two satellite photos lights up everywhere: crops change colour with the season, clouds drift, sun angles differ. Sending every changed patch to a vision model is slow and costly, and a model that is allowed to approve changes will confidently approve the wrong ones. The open question is whether a vision model behind a deterministic screen actually helps, and whether the confidence it reports can be trusted at all.

The screen is fixed code and never sees a model. The gate is separate and deterministic: a reply that is not valid JSON, names a class outside the enum, or falls below the confidence floor is rejected, and a model can propose a label but never widen what is allowed through. The evaluation was written down first (PROTOCOL.md, with every deviation logged), tuned on a development split of 14 pairs and run once on 10 test pairs, with pair-level bootstrap intervals. Two local models (Gemma 4 12B and Qwen2.5-VL 7B) were compared against an RGB-difference baseline. The follow-up (PROTOCOL_JEV.md) was written before the first API call and is labelled as a second look at an already-seen test split.

The design screen (NDVI or SSIM) was not the best screen: development F1 0.227 against 0.297 for a plain RGB difference, and both screens find only about 39% of the 668 labelled change objects. Behind Gemma 4 12B the gate raised precision by +0.24 [+0.09, +0.33] for the design screen (0.42 to 0.67) while keeping about 70% of real candidates; Qwen2.5-VL 7B added nothing. An F1 gain and a win over a three-number regression are not established at 10 pairs. The models' verbalized confidence failed calibration. The calibrated judge matched Gemma's precision lift and recall, missed its own calibration-error target (0.166 and 0.124 against 0.10), did not beat a stacked logistic regression (AUROC 0.701 against 0.782), and cost about $0.06 in all: a calibration layer, not a better judge. 54 tests. Not shown: any effect with no labelled data, which is where the judge should help, and any result beyond one dataset of 24 pairs.

Linking the same company across federal contracting, registry and filing data fails when it relies on fuzzy strings alone: similar names merge different companies, renamed companies split, and one oversized alias list can dominate a metric. A system that always takes the best-scoring merge has no way to promise that two companies with different official IDs stay apart.

Blocking and a composite name and address score propose pairs; a deterministic constraint vetoes any pair with contradicting IDs, same-ID records are treated as evidence, and community detection (Louvain) cuts clusters on the vetoed edges so the guarantee holds for whole clusters, not just pairs. B-cubed F1 is the primary metric because pairwise F1 was dominated by one 256-alias entity. The plans (PROTOCOL_SAMPLE, PROTOCOL_GATE, PROTOCOL_SPLIT) were pushed before the data were downloaded or run, and compared against a tuned plain fuzzy matcher and Splink. Several of my own mistakes were caught and are documented, including a first sample that was only the first 10,000 awards in name order.

On the random DoD sample, the name-only pipeline scored B-cubed F1 0.911 against 0.885 for tuned fuzzy matching (+0.026, 95% interval +0.019 to +0.034) and 0.893 for Splink; better blocking keys raised completeness but not F1, and my guess that token-sort scoring would help was wrong. On the UK data the full gate beat the same pipeline without IDs by +0.027 [0.025, 0.029], all 30 held-out slices improving, with zero rule-breaking clusters against 49 to 77 without the veto. Nearly all of the F1 gain came from the same-ID evidence, not the veto, and the graph-splitting step moved F1 by 0.0001 at a tuned threshold while making the guarantee true (6 rule-breaking clusters without it) and being worth +0.20 at an untuned one. 43 tests. Not shown: the registry ID is a company number, not a tax ID, and its visibility is simulated; 17% of true pairs are renames nobody recovers; the 100k-plus record benchmark is not done.

Generated briefings and RAG answers invent facts and mis-cite real ones. An LLM used as the judge is non-deterministic, costs money per claim and shares the generator's biases. What is needed is a cheap, repeatable check that fails toward flagging, plus a measured answer to how often it flags the wrong thing.

The checker is deterministic around one small model: sentence-level claim extraction, a footnote-to-passage mapping, an NLI model (DeBERTa-v3 small or base) scoring entailment, contradiction and neutral, and a fixed rule that anything not entailed is flagged. A 2x2 ablation crossed model size with whether passages carry their page title, because missing context was the suspected cause of false flags. Thresholds were fixed on development data per condition and each condition ran once on the same held-out items. A model trained on FEVER was skipped on purpose to avoid contaminating the benchmark, and an invalid first run is kept in the results with the reason.

Small model plus page titles reached F1 0.920 for flagging unsupported claims (precision 0.914, recall 0.927) while keeping 82.5% of supported claims, at 46 ms median; the base model with titles reached 0.933, a gap within noise. Missing context, not model size, caused most of the false flags: adding titles moved retention by 22 to 26 points, while moving from small to base without titles moved it by 8. The raw entailment probability is overconfident (calibration error 0.08 to 0.17) and should not be shown as a probability. 10 tests, no model download needed. Not shown: long documents, tables, non-English text or adversarial claims; support for a claim is not truth if the source is wrong; HaluEval, GPU timings and a PyPI release are not done.

AI assistants write PySpark and Pandas that passes a toy test and misbehaves at scale: a whole table pulled to the driver, a join that explodes, a path that exists only on the author's laptop. A linter can catch these without executing anything, but only if its findings are worth a reviewer's time, and a rule that cries wolf gets switched off.

Three AST visitor rules, each deterministic and individually testable, with property-based tests and a strict type check. The evaluation separates three questions: is a finding faithful to the rule, is it actionable to a reviewer, and does the tool find defects planted in real code (mutation testing). Every version was frozen and tagged before the next sample was drawn, with the protocol pushed first (PROTOCOL through PROTOCOL_V4) and one rater, the author. A change that lowered the failure rate by changing a severity is recorded as that, not as an improvement.

On real code the rules were accurate and stable, but the one I had made critical was not useful: AG001 (unbounded collect) was faithful to its rule in 97% of findings and actionable in only 3 of 44 (about 1 in 15), so v2 demoted it to a warning, which cut the files failing the default gate from 21% to none with real findings without reducing the noise. AG002 (join without keys) has not found a true positive in real code; AG003 (hard-coded paths) went from 0.58 faithful to 0.94 on the first fix, then 0.83 on unseen files, and v4 reached 59 of 59 faithful on 300 fresh files with about 22% actionable and no new false-alarm kinds. Planted defects in their plain forms were found every time, which is a sanity check, not a recall estimate. 28 tests. Not shown: that it changes what engineers do, recall on forms I did not think of, or anything beyond one rater and a convenience sample of public repositories.

Tactical and offline deployments cannot use a cloud model, but small local models are unreliable at structured output, and an unchecked tool call is a way to run arbitrary code. Two questions need numbers: does constraining the output format actually help, and is a valid-looking call a correct, safe one?

A deterministic gate sits between the model and anything that executes: schema validation, then an AST check that the call is exactly one registered tool with the right argument form. The same weights run on both backends so CPU and GPU can be compared, and every result is scored three ways (schema valid, passes the gate, equals the expected call). A first pass that was invalid is kept with the reason. A post-hoc column shows what the strict keyword-argument rule costs, labelled as a diagnostic, not the headline.

Constrained decoding never lowered format adherence: Qwen2.5-Coder 7B went from 60 to 65% valid output to 100%, Llama 3B Q8 from 75% to 100%. A valid JSON wrapper was not a valid call: Llama 3B Q8 was 100% schema-valid but only 50 to 60% correct, because the grammar closed a string at an unescaped inner quote and truncated the call, which only the AST gate caught. One strict rule explains much of the Llama result (65% against 100% if keyword arguments are accepted). Across 320 first-pass outputs nothing outside the registered tools ran. GPU speeds (61 to 135 tokens per second) are throttled-laptop numbers, not hardware limits. 23 tests. Not shown: reasoning, long contexts or multi-turn tool use, other hardware, or any difference of one to three prompts out of 20 (about plus or minus 20 points).

One camera sees part of a site and loses people behind objects; several cameras can follow someone across views, but only if the system knows where each person is on the ground and whether two detections are the same person. A tracker that always attaches the nearest detection will teleport a track onto someone else, and a synthetic benchmark can make any such tracker look good.

Homography projection of box bottom-centres, chi-square gating that refuses surprising associations and counts them, a constant-velocity Kalman filter, and a speed-ceiling flag. Each real-data study was pre-registered: tuned on development frames, then run once on held-out sequences, with block-bootstrap intervals and ground-plane CLEAR-MOT and IDF1. The detector study compared a COCO Faster R-CNN, a COCO RetinaNet and a Faster R-CNN fine-tuned on earlier sessions of the same rooms, against the dataset's own boxes, with seven numeric hypotheses written first.

The synthetic score (MOTA 0.93 to 0.96) did not predict real footage: on Wildtrack the tracker scored 0.79 with perfect boxes and minus 0.17 with a COCO detector, mostly because that scene is crowded and sampled at 2 frames per second; on MMPTrack it scored 0.996 with clean boxes and 0.57 to 0.60 with a detector. In the detector study pooled MOTA was 0.524, 0.594 and 0.759 for the three detectors against 0.986 for clean boxes, so fine-tuning added +0.235 [0.164, 0.304] while a different COCO architecture added 0.07; five of seven hypotheses held. Two did not: with the best detector, fusion beat the best single camera in only one of three sequences, and camera consensus still helped (+0.021). 22 tests. Not shown: generalisation to a new site (the fine-tuned detector saw the same rooms), why about 8.8k false positives remain, YOLO with ByteTrack, an extended Kalman filter, or PETS 2009.

Generative models are often used to flag anomalies, but a score alone does not say how many alarms are false or how many real problems are missed. Conformal prediction promises a fixed alarm rate on normal data for any score, however poor, which is easy to state and easy to mistake for a promise that the detector works.

Plain split-conformal calibration (a deterministic, tested threshold) wrapped around small VAE, DDPM, energy-based and Gaussian-mixture models. The synthetic data are two half-moons with an exact density, so every score can be compared with the true likelihood. For the real tables, five hypotheses with numeric criteria, the grid of choices and the seeds were fixed in advance; choices were made on development seeds from a separate pool of rows and ten test seeds were run once, with paired comparisons.

Coverage held for every score on every dataset (0.895 to 0.902 synthetic, 0.897 to 0.907 on the real tables, nominal 0.90), as theory says it must, and said nothing about usefulness: novelty detection on Covertype was poor, with a best AUROC of 0.70 and 64 to 75% of novel rows still inside the 90% region. Under shift, the better a score noticed the shift, the faster its coverage fell. Two of five hypotheses were falsified: deep scores did beat a tuned mixture model by about 0.05 AUROC, and VAE reconstruction error was competitive rather than among the weakest. 43 tests. Not shown: other kinds of shift, adaptive conformal methods, larger models, image or text data, or a tight diffusion likelihood; the mixture baseline was capped at the edge of its grid, and UCI Sensorless Drive could not be downloaded.