False AI Accusations in College: The Flaw in AI Detectors

By C. Rich

I stumbled into this subject for a reason I wish I never had. I have two family members in college who were accused of using artificial intelligence to write their work. There was only one problem: they did not use AI. I know that for a fact, and I cannot think of two people who dislike AI more than these two. They were not secretly asking ChatGPT to write their papers and then denying it when they got caught. They wrote their own work, turned it in, and a piece of software decided there was something suspicious about the way they wrote. That got my attention and made me wonder what exactly was happening here. I started wondering how often this happens and how many college students have been accused of cheating because a computer program said their writing looked like AI. I wanted to know how many students had received a zero on an assignment, failed a class, been dragged before an academic integrity board, lost a scholarship, had graduation delayed, or had something placed in their academic record because an algorithm produced a percentage on a screen. I went looking for that number, and what I found bothered me more than the original question because nobody seems to have it.

There is no national database keeping track of students falsely accused of using AI. There is no central accounting of how many detector-based accusations were later overturned, and there is no reliable national number telling us how many innocent students have received failing grades or academic misconduct findings because software got it wrong. We have individual cases, university statistics, research studies, lawsuits, and scattered reports, but we do not have the one number I wanted most: how many innocent students have already been hurt by this? That is astonishing when you consider what colleges are actually doing with these systems. AI detectors are not plagiarism detectors in the traditional sense. A plagiarism program can find a sentence in a student’s paper and show you another document containing the same sentence. There is evidence that can be examined because you can put the two passages next to each other and determine whether copying occurred. An AI detector generally cannot do anything like that.

An AI detector does not open some secret ChatGPT database and discover the student’s paper sitting there. It does not reconstruct the conversation in which the paper was supposedly generated, and it does not possess a forensic stamp saying that ChatGPT wrote something. Instead, it examines characteristics of language and makes a statistical judgment about whether those characteristics resemble machine-generated writing. That distinction becomes enormously important when we are talking about accusing someone of academic dishonesty. Human writing and AI writing overlap. A human being can write in a predictable, clean, and grammatically consistent manner. A human being can use ordinary vocabulary and write sentences of similar length. A careful student following an academic format may produce exactly the kind of orderly prose that an AI detector considers suspicious. Non-native English speakers have presented an especially troubling problem in research because their writing can be more statistically predictable, which is precisely the characteristic some detectors have historically associated with AI.

The numbers I found ranged all over the place depending on the detector and the study. Some newer systems perform considerably better than older ones, and some companies claim extremely low false-positive rates under particular testing conditions. Other independent studies have produced much uglier results. One well-known Stanford study found extraordinarily high false-positive rates when detectors evaluated essays written by non-native English speakers, and OpenAI itself abandoned its own AI-text classifier in 2023 after acknowledging that its accuracy was inadequate. The more I looked, the harder it became to understand how anyone could responsibly treat one of these scores as proof of misconduct. Suppose, however, that we give the detector companies every possible benefit of the doubt and assume a detector has only a 1 percent false-positive rate. That sounds wonderful until you remember what 1 percent means when tens of thousands or millions of papers are being processed. Vanderbilt University made exactly this calculation. It had processed approximately 75,000 student papers through Turnitin in a year. At a 1 percent false-positive rate, approximately 750 human-written papers could potentially be incorrectly flagged. Vanderbilt ultimately disabled Turnitin’s AI detector. Seven hundred and fifty may look like a small statistical error when buried inside a spreadsheet containing 75,000 papers, but it does not look small if you are one of those students.

This is something I think gets lost whenever people talk about percentages. A university administrator can look at 99 percent and think that sounds excellent, and a professor can look at 99 percent and assume the machine is almost always right. If you are the student standing in front of an academic integrity board trying to explain that you actually wrote your own paper, however, the relevant statistic is not how impressive the software looks across thousands of documents. The relevant question is whether it is right about you. There is another statistical trap here that deserves far more attention. If a detector has a 1 percent false-positive rate, that does not mean a student whose paper has been flagged has only a 1 percent chance of being innocent. Those are completely different probabilities. Once a paper has been flagged, the probability that the accusation is correct also depends upon how common AI cheating actually is in the population being examined, how sensitive the detector is, what kind of writing is being tested, how long the document is, and several other factors. This is a classic base-rate problem, but I doubt many frightened college students are receiving a statistics lesson before somebody tells them that a machine thinks they cheated.

Now imagine being nineteen years old, or in your fifties because you went back to college, and knowing you wrote the paper yourself; how disheartening this must be. My family is dumbing down their good grammar to they to find a safe place with this crooked shame meter. I’ve got a grammar Nazi in my family who corrects everyone when they speak or write substandardly. Now the grammar Nazi is dumbing down her writing so as not to be caught up in a scandal that would raise more suspicion, because this has happened to her already.

Now, the part that gets me steaming is that I am paying that college to do this to my family! I pay for this! I pay for them to hide their excellence; it costs me money to hide how smart they are! Excuse my overuse of exclamation points, because my grammar Nazi in my family would not stand for that, but I’m pretty angry. I can thunder down my grandiloquent omnipotence and make it arduous for you to apperceive the axiomatic reality of my lexicon as someone who gets paid to write for twenty years, but there is no need; I could get flagged as suspicious or something even worse, dishonest. Don’t use those big words, Charlie. Could get you in trouble, Charlie.

College in America, on so many levels, is flat-out disappointing and underwhelming. How exactly are you supposed to prove that you did not do something? That question bothers me enormously because it reverses the normal burden of proof. The machine makes an accusation, and suddenly the human being is expected to establish innocence. The student is placed in the absurd position of trying to prove a negative against an algorithm whose internal reasoning the student cannot inspect. There are many better ways to investigate suspected cheating. A professor can examine Google Docs version history and watch a paper develop over hours or days. Word documents can contain editing histories and timestamps. Students can produce outlines, notes, research materials, and earlier drafts. Professors can compare the paper with previous writing, or they can sit across from the student and ask questions about the argument, sources, unusual word choices, and reasoning behind particular passages before a derogatory accusation is made. Those methods actually investigate authorship because they attempt to reconstruct how the work came into existence rather than merely assigning a probability to the finished prose.

That does not mean AI detectors are completely useless. They might have a legitimate role as a signal telling a professor that something deserves a closer look. There is an enormous difference, however, between saying that a result gives someone a reason to investigate and saying that the result proves a student cheated. Even Turnitin has warned against treating its AI indicator as definitive proof of misconduct, and some universities have disabled or restricted these systems precisely because of false-positive concerns. Yet students continue to encounter professors and institutions willing to place enormous weight on the score. This is where the subject became much bigger to me than what happened to two people in my family. We are entering a world in which algorithms increasingly make judgments about human beings. AI decides what looks suspicious, what looks fraudulent, what looks abnormal, and what looks machine-generated, and there is a natural temptation to assume that because a computer produced a number, the number must represent something objective. Sometimes that number may be useful, but sometimes it is little more than an educated statistical guess presented with the authority of scientific precision.

There is also an irony here that would almost be funny if the consequences were not so serious. We have spent years warning students not to blindly trust artificial intelligence because AI can hallucinate, make mistakes, and confidently present false information as fact. Students are told to check their work, verify their claims, and never assume that something is true simply because a computer generated it. Then an AI-related detection system points at a student and says that the student cheated, and suddenly some of the same institutions forget their own advice. I am not arguing that students are not using AI to cheat because obviously some of them are. Students have always cheated, and every new technology creates new ways of doing it. Colleges have every right to enforce their academic standards and punish students when there is sufficient evidence that those standards have been violated. Catching guilty people, however, does not justify treating an unreliable accusation as proof, particularly when the consequences can follow a young person, or an older person, long after one assignment, or one semester, has ended. There is an old principle buried underneath all of this that predates artificial intelligence by centuries: the seriousness of an accusation should determine the quality of evidence required to sustain it. If a student’s reputation, grade, scholarship, degree, or academic record is at stake, the fact that a computer thinks the student cheated should NEVER be enough by itself. A detector can raise a question, but a question is not evidence of guilt.

Somewhere tonight, there may be a student staring at a paper they know they wrote while looking at a screen that says artificial intelligence wrote it for them. Their professor may believe the machine, and their university may believe the professor. The student may have drafts, notes, memories of writing the sentences, and an entire human creative process behind the document, yet somehow the burden has shifted to that student to prove that they exist behind their own words. I started looking into this because it happened to two people I know, but I came away wondering how many people I do not know have had exactly the same thing happen to them. Until colleges can answer that question, they should be extremely careful about allowing a machine that cannot establish authorship to become an authority on who is telling the truth. One more thing, I know this sounds outrageous and unthinkable, but we could put this to rest forever if colleges demanded that all work be done by hand. You know, handwriting, like the old days. Even if they are copying AI when writing by hand, they will learn anyway. End the rant, Charlie, end the rant…

Charles Richard Walker