AI products aren't being engineered for high variance — but maybe some should be?
When most people build with AI today, their first instinct is to make it more predictable. Clamp temperature, write stricter prompts, do more post-processing. Try to wrangle the model into behaving reliably, safely, deterministically.
But in many domains, consistency isn't and shouldn't be the goal.
Think about how we evaluate creative work. In marketing, sales, or design, you're not trying to get ten B+ answers. You're looking for the one thing that stands out. The idea that cuts through, surprises you, or reframes the problem entirely. Most of the outputs can be mediocre, as long as one of them hits.
The same goes for research. The first answer is often wrong or boring. But the third or fourth might make you pause, or send you in a direction you hadn’t considered. That doesn’t happen if you tune randomness out of the system.
Take Cluely's marketing. I dislike a lot about it w/ the theater & the intentional provocation, but I have to respect how it leaned fully into being out-of-distribution.
That same principle applies to how we use AI. The best outputs often aren’t the safest ones. They’re the odd ones that somehow land. And yet almost nobody is building tooling to catch those. There’s no QA loop for “actually good.” There’s no layer that says “give me 100 weird things and just flag the one that might change my mind.”