Ops Triage AI

Problem

Operational ticket triage usually depends on manual reading or an opaque classifier: no structured decision, no audit trail, and no honest measurement of where automation is right and where it needs human review.

Ops Triage AI tackles this problem within clear bounds: it receives tickets and produces structured classification (category, priority, risk, suggested team) plus an auditable decision about what can proceed automatically and what requires human review. The repository is public, so every decision described here can be inspected in the code.

Architecture

The system’s main flow:

HTTP → validation → PersistedTriageService → TriageTicket → DeterministicTriageClassifier → OllamaTriageClassifier → HybridPolicy → TriageDecision → Prisma/PostgreSQL → Feedback/Audit Trail.

Each piece has an explicit responsibility:

• DeterministicTriageClassifier: a predictable, cheap, reproducible baseline with heuristic signals and confidence.
• OllamaTriageClassifier: semantic interpretation with a local LLM (qwen2.5:7b-instruct-q5_K_S) and validated structured output.
• HybridPolicy: decides human review, fallback, and decision source. It is a pure decision layer — no field-level merging between classifiers.
• PersistedTriageService: orchestrates the persisted lifecycle: IDs, runs, decisions, and feedback.
• Persistence: Ticket → TriageRun → TriageDecision → Feedback, with append-only feedback — the original decision is never overwritten.

Engineering decisions

• The deterministic classifier is the guardrail and comparison signal; the LLM is the semantic classifier. Both receive the same ticket and are not treated as statistically independent.
• Known LLM failures (timeout, unavailability, invalid response) produce a deterministic fallback decision, always with human review. Unknown failures keep propagating — graceful degradation without claiming the results are equivalent.
• Security and operations from the start: API key, body and concurrency limits, X-Request-Id, redacted JSON logs, /metrics, /health, /ready, graceful shutdown, and abandoned-run reconciliation.

Evaluation

The official evaluation is a frozen held-out benchmark: 70 synthetic tickets, a single run, no rule tuning against held-out errors. Frozen parameters — commit f36e8ef8fcc70be09ab5ba9883efc7c34f3591a2, prompt ollama-triage-v3, model qwen2.5:7b-instruct-q5_K_S, 120000ms timeout.

The official results table below belongs to the historical deterministic, Ollama, and HybridPolicy benchmark. A later, isolated evaluation added Jev 1.13 as a third paradigm on the same frozen held-out set; Jev was not integrated into HybridPolicy. Laya was evaluated afterward, first zero-shot and then with domain adaptation, also outside the runtime.

The harness compares the deterministic baseline, the standalone LLM, and the hybrid on quality, severity recall, disagreement, fallback, review reasons, and latency. It deliberately does not emit: human-review precision/recall (the held-out set has no independent ground truth for it), suggested-team correctness, standalone LLM latency, or separate persistence/HTTP latency.

Official results

Operationally on the held-out set: 0.5143 review rate (severity, disagreement, and confidence signals), 0.3286 baseline/LLM disagreement, 0 fallback, no LLM failures, 70/70 decisions with hybrid source. Hybrid latency: p50 6257.7ms, p95 7078.6ms, max 7556.9ms.

On a frozen 70-ticket held-out benchmark, the local LLM improved category accuracy from 82.9% to 95.7% and HIGH/CRITICAL priority recall from 78.6% to 100%, while the deterministic baseline retained higher overall risk accuracy (95.7% vs 91.4%). The hybrid policy preserved the LLM’s classification quality while routing 51.4% of cases to human review based on severity, disagreement, and confidence signals.

Trade-offs

• The hybrid preserves LLM quality and pays for it with human review on more than half the cases plus 6-to-7-second latency — fit for async triage, not for a latency-critical synchronous path.
• The baseline won on overall risk accuracy (95.71% vs 91.43%): the LLM was not better at everything, and the regression is published above.
• An adversarial DEV baseline (out of distribution) dropped the deterministic classifier to 0.5 category, 0.5 priority, and 0.75 risk accuracy — evidence that explicit rules are brittle outside known vocabulary, which justifies combining with the LLM instead of a straight swap.

Limits of the historical hybrid benchmark

Limits of this historical evaluation:

• Synthetic English-only dataset with 70 examples.
• A single local model evaluated.
• One official run — no LLM variance measured.
• Confidence is heuristic, not calibrated probability.
• The held-out set has no independent ground truth for human review.
• Review rate is not precision/recall.
• Suggested-team correctness was not emitted.
• Standalone LLM latency was not emitted.
• Separate persistence/HTTP latency was not emitted.
• The system was not validated under real production traffic.
• The baseline had higher overall risk accuracy.
• Hybrid latency landed at roughly 6 to 7 seconds.
• The adversarial DEV baseline showed 0.5 category accuracy, 0.5 priority accuracy, and 0.75 risk accuracy.

Jev 1.13 experiment

This experiment evaluated a third paradigm against the same taxonomy: deterministic rules, a generative LLM, and a typed probabilistic decision model. It was an isolated evaluation: Jev was not integrated into HybridPolicy and did not replace Ollama.

The TriageClassifier adapter used Bun native fetch and the OpenRouter Decisions API with typesafe/jev-1.13; all 73 real responses resolved to typesafe/jev-1.13-20260917. No dependency was added. Three DEV calls ran before the freeze (1b6d7ff); commit 4c41e0b froze the configuration and ran one held-out evaluation on the same 70 synthetic labels used in the historical comparison.

Standalone Jev got the category/priority/risk tuple exactly right in 66/70 cases (0.9429). The historical benchmark did not emit this metric for the standalone deterministic and Ollama classifiers; the 0.8286 exact tuple accuracy above belongs to the historical hybrid path, which has a different decision flow.

Across Jev’s 70 held-out calls, standalone latency averaged 569.4 ms, with a 545.5 ms p50, 712.2 ms p95, and 1,142.2 ms maximum. The API reported 90,229 input tokens and US$ 0.003789618 in cost; there were no API or schema failures. The historical Ollama benchmark did not emit standalone latency or cost. Its 6–7 second latency belongs to the hybrid path and is not a like-for-like comparison with this standalone Jev call.

Jev retained probability distributions, continuous confidence, and top-1 / top-2 margins. Still, one incorrectly classified HIGH risk had confidence 0.86: high confidence does not guarantee semantic correctness.

Selective automation and calibration analysis

The exploratory analysis below requires all three confidence values to meet the threshold. These are observed results on the same small dataset, not a production threshold choice:

Exploratory calibration analysis measured Brier scores of 0.0005 for category, 0.0386 for priority, and 0.0800 for risk; top-1 ECE was 0.0054, 0.0329, and 0.0226, respectively. These are descriptive estimates from 70 synthetic tickets with sparse bins; they do not establish calibration for this domain.

On the same frozen synthetic held-out labels, Jev produced the strongest measured category and priority results in this experiment, along with typed probability distributions and sub-second average standalone latency. The small synthetic dataset does not establish production superiority. The architectural decision remains benchmark/evaluation only; there was no HybridPolicy integration.

Laya: from zero-shot to domain adaptation

After Jev, I evaluated the general convaiinnovations/laya checkpoint, a non-autoregressive decision model with code and open weights under Apache 2.0. I used laya==0.3.23 and the protocol’s frozen revision, retaining the same held-out set. This base checkpoint differs from laya-typed-decisions, which has already been specialized on other workflows.

Zero-shot, Laya got 8/70 tuples right (11.43%). The clearest pattern was 40/40 expected LOW priorities predicted as MEDIUM. Auditing options, indices, mapping, normalization, and parsed outputs found no integration bug that invalidated the behavior. The next hypothesis was that the general model needed adaptation to our taxonomy and domain.

Adaptation protocol

I separated 1,120 TRAIN tickets for changing weights and 280 VALIDATION tickets for checkpoint selection. The 70 HELD-OUT tickets stayed outside data generation, training, validation, selection, tuning, thresholds, and calibration. TRAIN and VALIDATION are synthetic too, sharing scenario families with distinct wording.

Fine-tuning started from the same base model. VALIDATION selected epoch 4 by highest exact-tuple accuracy, using predeclared tie-breakers. Freeze 2 fixed data, weights, code, and configuration before the single official adapted evaluation. The sequence is baseline → Ollama → HybridPolicy → separate Jev evaluation → Laya zero-shot → adaptation → selection → Freeze 2 → adapted held-out.

Training infrastructure

The inference host, with four vCPUs and roughly 4 GB RAM, was unsuitable for this training. I used a temporary RunPod GPU: NVIDIA RTX A5000, 24 GB VRAM, Python 3.11, PyTorch 2.5.1, and CUDA 12.1. Training ran for four epochs with mixed precision (FP16 autocast / FP32 master weights), gradient accumulation, and gradient checkpointing to manage memory.

In the first run, the scheduler advanced even when GradScaler skipped an optimizer update. I preserved the artifacts, rejected the run on methodological validity, and corrected the coordination before repeating training and opening the adapted held-out evaluation. The official run recorded 420 attempts, 413 applied updates, seven skips, and 413 scheduler steps. It was not chosen for more favorable accuracy.

The .safetensors checkpoint was retained locally outside the Pod; SHA-256, size, selection, and configuration are documented in public artifacts. Adapted weights were not published. This cycle supplies my first hands-on evidence in ML Systems / AI Infrastructure, bounded to training, checkpoints, and experimental evaluation.

Final evaluation ran on CPU without the GPU, averaging 1,619 ms per ticket, versus 1,414 ms zero-shot. Jev had recorded 569 ms over API/network in a different context. Local inference did not automatically mean lower latency on this CPU; the serving paths do not support a like-for-like speed comparison.

Measured GPU tasks totaled 1,523.51 seconds. At the recorded rate of roughly US$ 0.28/hour including disk, US$ 0.1185 estimates those tasks, excluding setup and idle time; it is not the Pod’s total bill. Laya incurred US$ 0 in external per-token/request charges, while infrastructure and operations remained our responsibility. The report does not document teardown: the Pod had not been destroyed when evaluation was completed.

Results and severity regression

Accuracy by field and exact tuple, on the same held-out set:

Correct tuples rose from 8/70 to 60/70, and LOW → MEDIUM errors fell from 40/40 to 0/40. HIGH risk recall rose from 57.14% to 100% (4/7 → 7/7), but HIGH/CRITICAL priority recall regressed from 92.86% to 85.71% (13/14 → 12/14). Aggregate improvement did not eliminate severity errors.

Jev had reached 66/70 correct tuples (94.29%) without any of our TRAIN examples; adapted Laya received 1,120. The comparison is asymmetric: Jev showed strong out-of-the-box generalization on this benchmark, while Laya showed the control open weights offer for adapting behavior to a domain.

Confidence and generalization limits

Adapted Laya’s confidence stayed close to 1, yet 10/70 tuples were still wrong. Thresholds through 0.90 retained every ticket; 0.95 and 0.99 removed only one, leaving exact-tuple accuracy below 90%. High confidence did not become a reliable estimate of correctness, and its semantics differ from Jev’s reported confidence.

Exact-tuple accuracy was 100% on TRAIN → 92.50% on VALIDATION → 85.71% on HELD-OUT. Strong fit and saturated confidence are signs consistent with memorization and overconfidence; this small sample does not establish how severe the effect is.

These evaluations use English, a single domain, and synthetic data, with one official run per protocol and no validation on real traffic. Jev and Laya remain evaluation-only, outside HybridPolicy and the runtime. The experiment exercised evaluation and adaptation, without deploying an adapted production service.

The Jev article covers the initial evaluation; Laya vs Jev explains the protocols, calibration, and cost of control. The

collects the metrics and links the artifacts for auditing.

Evidence

• Repository: ops-triage-ai on GitHub
• Official held-out report: official-held-out-f36e8ef.md
• Machine-readable artifact: official-held-out-f36e8ef.json
• Jev 1.13 experiment report: jev-1.13-held-out-4c41e0b.md
• Jev held-out artifact: jev-1.13-held-out-4c41e0b.json