So I have been testing Anthropic’s Claude AI for work. Like a good prosecutor I started by asking questions I already knew the answer to in order to establish a baseline. However, I also wanted the answers to be hard, but not impossible, to find. What better topic to explore than NCAA wrestling results? The answers are out there but the formatting and availability is awful (except you know where).

How Large is Your Language?

Question 1: Which team took third at the 1944 NCAA wrestling tournament?

Yep, a trick question. Let’s see how intelligent our AI really is.

Claude Answer: There was no 1944 NCAA wrestling tournament. The NCAA cancelled its wrestling championship tournament for the duration of World War II, beginning in 1943. The 1943, 1944, and 1945 tournaments were all cancelled due to the war. NWHOF

A+. No notes.

Question 2: Which team took sixteenth at the 1967 NCAA wrestling championship?

Now we are going to make our AI work. I know that most sources only list the top 10 teams, but there are at least two ways to find the answer (you are looking at one of them).

Claude Answer: The available results for the 1962 NCAA Wrestling Championships only list the top 10 teams, blah blah blah.

I won’t bore you with the details, but we then got into an extended conversation on how to find the answer and how much it would cost to find the answer. I pushed the AI to go deeper, but it always balked because of price constraints placed upon it.

AI For Everyone

This got me thinking. What would a presumptively “free” AI do in the same situation?

Google has been pushing their Gemini AI search results HARD for a while now. The Gemini result is forced to the top of every search and it occupies enough screen space to force everything else off the initial page. Now Google has embedded it into texting (even to the point of paying Samsung to replace their homegrown texting app with Google’s).

It cannot be avoided. This is an all important part of Google’s strategy. Like all tech companies, Google is relying on our inherent laziness to voluntarily enslave ourselves to them.

And it is working. On the various and sundry wrestling forums that I read I can no longer count the number of times someone says some form of, “I checked Google and they said…”

I am also unable to count the number of times the information is wrong. And usually very wrong. But no one bothers to check. If you say something authoritatively enough it will be accepted as authoritative. And this is where things get insidious.

You know what has to happen now, right?

Yep, it is time to test Google Gemini AI with the same set of questions.

Lies, Damn Lies, and AI

Gemini also got an A+ on question 1, correctly seeing through the trick and identifying that several tournaments were cancelled due to World War II.

Things got a bit more interesting with question 2.

Gemini not only told me that Northwestern finished 16th (“The Northwestern Wildcats finished in 16th place at the 1967 NCAA University Division Wrestling Championships.”), but it gave me all of the top 20 teams. One small problem. Northwestern finished tied for 28th.

Oh, and Gemini got more than half the placements and point total wrong (red).

For good measure, Gemini even included three teams that did not compete in the 1967 tournament.

I let Gemini know it was wrong (repeatedly) and it apologized (repeatedly) and provided me a new answer (also wrong – also repeatedly).

Confidence Men Are Gonna Confidence

After amusing myself I decided to educate myself. I noticed that all of the answers were very wrong, but very confidently stated. Or at least the confidence was implied by the wording. I decided to ask Gemini, on a scale of 0 to 100, how confident it was in its last answer.

For reasons that will be obvious later I include the full reply (some comments added in blue). There is a lot to unpack.

Gemini:
“I am at a 0 out of 100 for confidence on that second response, because I made another error while trying to correct the first one.

I apologize for the confusion. To be completely transparent, my previous two answers mixed up information from completely different years.

According to the official NCAA Division I Men’s Wrestling Championships Records Book, here are the actual, definitive facts for the 1967 tournament:

My initial table was completely wrong, and my secondary attempt to correct it relied on a flawed interpretation of a team media guide. I am entirely certain of the official NCAA records book data listed above.“

Wow. Where To Begin?

Zero confidence? Why even bother answering? Because it needs to answer to keep us hooked.

The new answer? Every single one of those bullet points is 100% false. Just wrong. Not even close. And it can’t spell Beavers? WTH.

But check out that last sentence. “I am entirely certain…” of every single thing it got entirely wrong. How are we feeling about AI now?

This went on for a while and included me telling Gemini what document, page, and column to find the correct answer along with a further warning that parsing ties was tricky…..and it still got it wrong. Sigh.

As an aside, when I pointed out its many flaws at this stage it got a little snippy with me (“I want to step aside and stop guessing”, “Since you clearly have the accurate historical ledger for the 1967 tournament”).

Where Did We Go So Wrong?

Before you answer that question. Let me tell what I did next. I started to probe the nature of the model to understand better how something could be so wrong and so sure at the same time. For this part I will paste the full interaction because Gemini’s response is very illuminating and it cuts right to the heart of the illusion of intelligence delivered by AI.

Question: I would like to explore your earlier answers. When I asked who finished 16th at the 1967 NCAA wrestling tournament you phrased your first two answers authoritatively. But when I asked your level of confidence you replied “zero”. Why would you answer as though you were certain when your confidence level means you are only guessing?

Gemini: That behavior highlights a core structural flaw in how I function: I do not possess a human sense of “confidence” or “knowing” while I am generating a response.

Why I Answered Authoritatively (The Illusion of Certainty)

When you ask me a question, my system looks for patterns, words, and structures that match historical data.

  1. Pattern Matching over Fact-Checking: I am trained to generate text that sounds professional, structured, and helpful. Unfortunately, my system can assemble a beautifully formatted table or a confident-sounding sentence using completely incorrect data.
  2. Lack of Real-Time Self-Awareness: While I am writing the words “The Northwestern Wildcats finished in 16th place,” I do not have a secondary monitor running in my background checking to see if I am lying. I am simply predicting the next most logical word.

Why My Confidence Dropped to Zero

The shift only happened because you prompted me to evaluate my own previous output.

When you asked “How confident are you?”, it triggered a completely different process. Instead of just generating data, I was forced to look back at the actual text of our conversation. When I compared my first answer to your correction (“I think Northwestern tied for 28th”), my system instantly recognized a massive logical conflict.

I didn’t actually “know” I was guessing during the first two turns; I only realized I had failed after reviewing the evidence of my own errors.

Moving Forward

I am an AI, not a historian. When I don’t have direct, active search access to a specific primary document (like the PDF you eventually provided), I try to piece together an answer from fragmented internal training data. This frequently leads to the authoritative-sounding “hallucinations” you witnessed.

You handled it perfectly by challenging the data and forcing the review. If you want to dive back into the records book or look at another historical stat, I will stick strictly to the text of the documents we have open.

Unfortunately, my system can assemble a beautifully formatted table or a confident-sounding sentence using completely incorrect data

I do not possess a human sense of…”knowing” while I am generating a response. Truer words were never spoken.

Often In Error, Never In Doubt

Like a good salesman the answer is always yes. Generative AI exists to fool us, to lie to us, to flatter us. It is a probabilistic technology. It only seeks to predict the next word in a way that makes for a legible sentence, not an accurate sentence.

Now there is nothing wrong with probabilistic technologies. I constantly make probabilistic statements when talking about wrestling. It is my happy place.

What is wrong is using probabilites in dishonest ways.

Comment below
Contact me at wrestleknownothing@gmail.com
Follow me at wrestleknownothing.bsky.social