How AI text detectors work?

I’ve been recently copy-pasting a few good looking and quite interesting articles in one of those free AI detectors that you can find at a Google Search distance. To my surprise, some of them have achieved the perfect score of 100% being AI.

Even if it is hard to believe, humans love shortcuts. Why waste a couple of hours trying to write a good article, when you can just feed a prompt to Chat GPT (to be read GiPiTi) and he (or she?, or it?)’ll do all the magic.

AI-generated content may not live up to the quality, relevance, and nuance of human writing. However, As AI has become a popular content generation tool, organizations, including schools, marketing agencies, and brands, have begun to rely on AI content detectors as a way to tell if the work they receive was written by a human or generated by software.

These can also be used to check for quality or plagiarism issues and to spot-check if you’re likely getting human writing (not AI drafts) from employees, freelancers, and agency partners.

I’m not in any way affiliated to any of them, but I’ll list some if you are curious to learn more: QuillBot AI, Cramly.ai, Content at Scale, and Copyleaks.

But, how do they even work? AI detectors are given a set of training data, which typically contains both human and AI-generated text. It analyzes those articles to figure out which characteristics best identify the AI-generated pieces.

Two of the major characteristics AI detectors analyze are:

• Perplexity: The unpredictability of the content. AI-generated text tends to have low perplexity, while human writing has higher perplexity.
• Burstiness: Variation in the length and structure of the sentences. AI content tends to be more steady and have lower burstiness than human writing.

That said, AI detectors can’t guarantee anywhere close to 100% accuracy because they are based in large part on probabilities. Also, each of the detectors use different datasets of content to train them. So, they can often provide different results from one another.

It seems there are a couple of signals that pop-up when an article has been AI generated: repetition of words, repetitive sentence structure, weird & unnatural word usage, generic and impersonal tone, contradictory statements, inconsistent verb tense, or stiff, formal writing style.

Based on this signals, and how often they appear in the article, the AI detector gives you its best guess on how the content was produced. Some will even highlight the parts that are most likely to not be written by a human being.

They can sometimes trow false positives though. According to AI expert, Soheil Feizi: “Current detectors of AI aren’t reliable in practical scenarios,” an assistant professor of computer science at UMD. “There are a lot of shortcomings that limit how effective they are at detecting. For example, we can use a paraphraser and the accuracy of even the best detector we have drops from 100% to the randomness of a coin flip. If we simply paraphrase something that was generated by an LLM, we can often outwit a range of detecting techniques.”

In particular, the use of AI-based paraphrasing tools like Quillbot or Wordtune can make AI content nearly undetectable in many cases.

On the other hand, human writing can also be incorrectly flagged as AI text. In these cases, it can damage relationships if you wrongly accuse a student, employee, or independent contractor of using AI instead of writing the copy themselves.

We live some interesting times. As we witness, further evolution within artificial intelligence comes with an increased level of sophistication within machine-generated content, which presents a distinctive challenge towards detecting such content effectively.

Therefore all involved in its development process need to work on creating even more cutting-edge and accurate tools that can keep up.

Detecting false information created by AI accurately is vital for maintaining online information credibility, which will be the only way to counter these threats effectively.

Moreover, it is important that we pay close attention to ethical considerations surrounding privacy encroachment, consent violations, and the potential for misuse of this powerful technology.