Cryptosmith

“Garbage in, garbage out.” In computing we’ve recited the GIGO motto since I was in diapers (Univac I and vacuum tubes were state of the art). Comfirmation bias is like GIGO: if people already know (think, guess, believe) something is true then they’re inclined to accept new evidence in its favor and reject evidence against it.

It’s everyone’s favorite type of careless thinking. We all do it by default.

Janelle Shane, the blogger of AI Weirdness, has posted a fine example and summary of how GPT-4, ChatGPT, etc are giant engines for confirmation bias. Here’s her insight:

Finding whatever you ask for, even if it doesn’t exist, isn’t ideal behavior for chatbots that people are using to retrieve and summarize information. It’s like weaponized confirmation bias. This is the phenomenon behind, for example, a lawyer citing nonexistent cases as legal precedent.

People call this “hallucination” but it’s really a sign of the fundamental disconnect between what we’re asking for (find information) versus what the language models are trained to do (predict probable text).

(boldface mine)

And that’s the classical software mismatch: if the software solves enough of a problem, we’ll pretend it solves the whole problem.

If a vendor is really, really lucky, the market redefines a task to match the software’s capabilities. If the software can’t do it, then it’s no longer that task. Typing becomes word processing, travel planning becomes online reservation, multilevel security becomes multilevel security (an inside joke).

I wrote a trivial text predictor/generator in college and fed it my friends’ online essays and papers. The results were nothing but weirdness, but entertaining weirdness. The output was occasionally grammatical, but the mashed-together phrases from different contexts were often hilarious, much like Shane’s AI Weirdness.

AI has always worked this way.

My rules of thumb:

• AI is always entertaining.
• Some AI techniques find practical though limited applications.
• It’s easy to believe a partial solution is a complete solution, and not just for AI.