In progressRail crossing safety · agentic AI engineering

AI engineering / competing risks / retrieval-augmented LLMs / agentic review

The model said removing the gates would cut risk by 70%.

It was wrong. Finding out why led to a four-layer retrieval system that predicts, explains, and prescribes for 3,310 highway–rail grade crossings — and refuses to prescribe when the data cannot support it.

AI engineeringCompeting risksRAGAgentic LLMsCausal guardrails

3,310crossing records

662held-out crossings

6LLMs compared

3competing severities

5counterintuitive findings

01 / DECISION PROBLEM

Which crossing do you fix, and what do you change?

A rail agency has thousands of crossings, a fixed budget, and a list of possible upgrades. The question is not only whether a crossing is risky. It is which change, at which crossing, buys the most safety.

That requires two systems in sequence: a model that estimates how risk evolves and why, then a decision layer that turns the estimates into an intervention an engineer can inspect and defend.

02 / RISK ENGINE

Estimating risk when almost nothing has happened.

Of 3,310 crossing records covering 28 years, 2,835 have no recorded crash. Survival analysis retains that history as evidence, while competing risks respects that property damage, injury, and fatality are distinct outcomes and only one occurs first.

The shared trunk learns general crossing risk; each head specializes by severity. Rare fatal outcomes can borrow structure learned from more common cases without collapsing the outcomes into one scale.

Single head

SH-MNN

One network and one outcome. Severity is ignored, creating an honest baseline every richer model must beat.

Main model

MH-MNN

A shared trunk with three heads. Rare fatal outcomes benefit from patterns learned across more common cases.

Cause specific

CS-MNN

Three separate networks. Maximum flexibility by severity, but each learns crossing risk with less data.

03 / INTELLIGENCE LAYER

Four retrieval layers on top of the model.

Three curves and forty contributing attributes are not a decision. Each layer retrieves real comparable crossings to calibrate risk, explain it, prescribe from observed analogs, and eventually ground the audit in published rail-safety evidence.

MH-MNN3 headsCIF + SHAP

RAG-1calibrationblend with neighbors

RAG-2explanationSHAP → plain language

RAG-3prescriptionmatched safer analogs

RAG-4literaturedesigned · not built

Agentic LLM workflow

1 · attributes only2 · + crash history3 · calibrate vs actual4 · SOTIF audit

Senior reviewreads all stagescritiques · decides

Stages run sequentially and in isolation to prevent crash-history leakage. A senior-review agent reads the evidence, critiques the junior stages, and produces a decision that remains traceable to retrieved records.

04 / THE DISCOVERY

Every device change looked like an improvement.

The first prescription design simply changed the device in the feature vector and reran the model. It produced a convincing, fluent, and causally invalid answer.

4-year property-damage risk after swapping the deviceInjury Δ

Installed devicegates + lights + audible

17.83%

baseline

Crossbucks onlymuch weaker

6.48%

+420% injury

No device at allnothing · n=39

5.30%

+462% injury

Gates + lightsslightly weaker · n=34

6.70%

+324% injury

Lights onlyweaker · n=7

4.60%

+316% injury

+ Traffic signalsstronger

5.98%

+446% injury

Ripping out all protection scored “better” than adding traffic signals. The model was reading the installed device as a marker of an already-dangerous crossing: confounding by indication, not a causal benefit.

05 / SAFE PRESCRIPTION

Prescribe from real crossings, not from the model.

The redesign retrieves exposure-matched crossings where a safer configuration already exists. Recommendations come from observed analogs, and the system returns no prescription when comparability fails.

Retrieval

Similar, but safer.

Find real crossings that resemble the target but carry less risk. Their actual values provide both direction and magnitude.

Guardrail

Match exposure first.

Safer analogs must match traffic, train counts, switching moves, and lane count. If no match exists, positivity fails and the system refuses to answer.

Ranking

One index, three outcomes.

A model-independent severity weighting resolves trade-offs without hiding the individual risk curves.

Fatal · .77Injury · .16.07

Deployment rule

If the model's explanation for a variable is paradoxical at a crossing, the system does not prescribe a change to that variable — under any circumstances.

06 / AGENTIC AUDIT

Teach the system to flag its own strange results.

The audit agent receives principles rather than answer keys. It tests sample size, confounding patterns, and prior engineering knowledge before committing to the most probable explanation.

Factor	What the model found	Most probable explanation
Daytime trains	More daytime trains lowers fatal risk	Higher visibility and driver compliance in daylight
Night trains	More night trains raises fatal risk	Reduced visibility compounds severity — the data pattern supports it
Truck share	More trucks lowers every crash type	Trucks approach crossings more slowly and cautiously
Distance to intersection	Greater distance raises fatal risk	Beyond about 1,400 m, sight-distance limits may outweigh vehicle-storage benefits
Crossing angle	A perpendicular crossing raises fatal risk	A 90° impact transfers force directly into the passenger compartment

Next build

The literature check is being replaced with retrieval over rail-safety publications so the agent cites passages rather than its own memory — and can return “no supporting work found” as a valid result.

07 / PERFORMANCE

Cheap enough to interrogate.

Attribution and analog search require repeated model runs. All three architectures were profiled across batch size, precision, and execution strategy.

Model	Training	Speedup	Inference	Speedup
MH-MNN · 3 heads	178.6 s → 3.9 s	45.8×	305.6 ms → 8.1 ms	37.9×
SH-MNN · 1 head	8.2 s → 2.6 s	3.1×	194.7 ms → 5.3 ms	36.7×
CS-MNN · 3 heads	11.4 s → 3.4 s	3.3×	282.3 ms → 8.0 ms	35.3×

11–12% slower

Vectorized parallel mapping across only three heads lost to sequential execution. The intuitive optimization was the wrong one.

08 / IN PROGRESS

What ships next.

The pipeline runs end to end on a single crossing across six models. What remains is evaluation at scale — listed here as commitments, not results.

Component	Purpose	Status
Survival engine	Cumulative incidence per severity, with feature attribution	Complete
Retrieval layers 1–3	Calibration, explanation, and prescription with the positivity guardrail	Complete
Junior → senior workflow	Four-stage analysis plus a decisive review layer	Complete
Literature retrieval	Ground the audit in retrieved sources rather than model recall	Designed
Full evaluation	6 models × 662 crossings × 5 stages	Running
Hallucination scoring	Check every numeric claim against the source documents supplied	Queued
Model comparison	Test which model reasons best about safety, and how cheaply	Queued

The metric that matters is refusal.

The comparison is not which LLM writes the best prose. It is which one refuses correctly, avoids invented numerical support, and says no change is warranted when the evidence does not support one.

09 / APPLICATION

Where this fits in a safety program.

Capital planning

Rank crossings on one severity-weighted index.

Keep the per-severity breakdown visible while turning three curves into a defensible priority list.

Intervention

Trace every recommendation to comparable sites.

An engineer can inspect the matched crossings and observed outcomes behind a prescription.

Model assurance

Surface counterintuitive results before review.

The audit raises strange findings itself and argues each one through rather than burying it.

Reporting

Produce reviewable analysis at portfolio scale.

Structured stages and numeric fact-checking make the reasoning inspectable instead of a black box.

10 / STACK

Built end to end.

Raw crossing inventory and 28 years of incident records through to model-generated engineering reports.

Competing-risk survivalMulti-head neural networksPyTorchSHAPFAISSRAGAgentic workflowsLLM evaluationCausal inferenceAHP weightingA100 profiling