Excellent article and totally agree I have been writing about this topic especially when it comes to big problems with vector db and hallucinations and the very serious risk involved for compliance in EU. Customers/patient have a right be protected both in terms of security not to be hacked or automated descionmaking that is a black box
Good point, especially around compliance and black-box decisions. What I’ve seen is that many of these risks don’t come from hallucination directly, but from decisions being made outside the domain model, once that happens, you lose both explainability and control at the same time, which is exactly what makes it hard from a regulatory perspective.
You’re pointing at the right boundary, where determinism is required vs where ambiguity can be tolerated, and that’s exactly where most systems get it wrong, not because teams don’t understand the idea, but because in practice those boundaries blur under pressure, AI starts upstream as “just insight” and slowly moves into decision paths without anyone explicitly redesigning the system, and by the time it’s noticed, the model isn’t assisting anymore, it’s effectively replacing parts of the domain logic, that’s where things start to drift quietly, not because the model is bad, but because nothing is enforcing the rules at that boundary anymore.
It’s usually defined upfront, the issue is it doesn’t break, it drifts, under delivery pressure and “good enough” results, validation gets relaxed or pushed downstream, and over time the model quietly moves from assisting decisions to shaping them, without the same guarantees.
In my experience the rules are still encoded, the issue is they’re not always on the execution path, the deterministic step exists, but parts of the system start making decisions before reaching it, so the rules don’t disappear, they just stop being consistently applied.
Excellent article and totally agree I have been writing about this topic especially when it comes to big problems with vector db and hallucinations and the very serious risk involved for compliance in EU. Customers/patient have a right be protected both in terms of security not to be hacked or automated descionmaking that is a black box
Good point, especially around compliance and black-box decisions. What I’ve seen is that many of these risks don’t come from hallucination directly, but from decisions being made outside the domain model, once that happens, you lose both explainability and control at the same time, which is exactly what makes it hard from a regulatory perspective.
You’re pointing at the right boundary, where determinism is required vs where ambiguity can be tolerated, and that’s exactly where most systems get it wrong, not because teams don’t understand the idea, but because in practice those boundaries blur under pressure, AI starts upstream as “just insight” and slowly moves into decision paths without anyone explicitly redesigning the system, and by the time it’s noticed, the model isn’t assisting anymore, it’s effectively replacing parts of the domain logic, that’s where things start to drift quietly, not because the model is bad, but because nothing is enforcing the rules at that boundary anymore.
It’s usually defined upfront, the issue is it doesn’t break, it drifts, under delivery pressure and “good enough” results, validation gets relaxed or pushed downstream, and over time the model quietly moves from assisting decisions to shaping them, without the same guarantees.
Sure, happy to answer, the questions are great.
In my experience the rules are still encoded, the issue is they’re not always on the execution path, the deterministic step exists, but parts of the system start making decisions before reaching it, so the rules don’t disappear, they just stop being consistently applied.