Decision Models Are Cheap, But Only as Good as the Evidence You Feed Them

A new class of AI model has arrived in security operations. Instead of generating text, these models take state and typed questions and return a choice, a score, or a yes/no with a probability. TypeSafe AI released Jev on September 15, OpenAI opened a public beta of its Decisions API on October 6, and Cribl published an open LoRA adapter called cribl-decision.

For alert triage, the appeal is obvious. A calibrated probability can be thresholded: close automatically above 0.97, escalate below 0.6, send the middle to an analyst. Jev costs $42 per billion input tokens with responses in 70 to 500 milliseconds, putting the bill for roughly 450,000 alerts a year near a hundred dollars. Every verdict is logged and replayable.

But a calibrated guess is still a guess. When vendors say these models cannot hallucinate, they mean the output matches the schema. A well-formed answer can still be wrong. Calibration is a promise about a distribution, and the model only knows what it is given.

The AI SOC Report 2026, covering 25 million alerts triaged in 2025, found that of 82,000 alerts that went through a live memory scan, about 2,600 machines had an active infection, and 51% of those had already been marked mitigated by the EDR. Nearly 1% of real incidents traced back to the lowest severity alerts. Cribl's own model card warns the model can be overconfident and is sensitive to wording.

The answer is to collect forensic evidence before deciding: process trees, detonation results, memory contents, and organizational context about what is normal. That context cannot be bought and decays quickly, so it has to be learned continuously and tied back to outcomes to check whether 90% still means 90% in your environment.