BOT or NOT! The ChatGPT detector bandwagon

Tech loves a bit of hype, and it doesn’t any more hyperventilating than ChatGPT. Blockchain? NFT? Who cares anymore.

AI jackpot

AI has hit the jackpot. ChatGPT’s creator, OpenAI, is valued at $29,000,000,000. Not bad for a company that expects to make one-twenty-ninth of that in revenue … in 2024. While Google’s Bard knocked a cool $160bn off the value of the company with just one small ‘hallucination’, as invented ‘facts’ are called in the AI chatbot world.

Don’t get us wrong, ChatGPT and its kin are remarkable achievements. They’re great fun to play with too and starting to prove very useful, if a bit immature as products.

Text-to-image AI generated “Torpedo Bomber” … amusingly literal, epically inaccurate.

But as for stealing our jobs, putting a generation of knowledge workers out on the street? Or helping cheats pass degrees without all the bothersome ’learning’ bit? We should probably take an in-depth look at that.

People are rightly concerned that AI can be used to “fake” human effort. It seems even easier to cheat than the old method of lifting content off webpages and pasting it into essays.

To counter this cheating, an arms race has started. Universities now habitually employ plagiarism detection software to stop old style cheating. At the same time, a new breed of AI writing detectors has started to spawn.

We have created our own: Find the real writer: Bot or Not

It began as an experiment and blossomed into a fascinating product. We reasoned, possibly unjustifiably, it must be possible to detect the difference between a soulless machine writing and a real flesh and blood human. And you can, to a reasonable degree of confidence, but it was quite challenging.

After all, the AI has been taught with mind-bending quantities of actual human writing on which to base its own writing. Not to mention, writing is really really really really complex.

We had to consider what makes ChatGPT and other language-processing AI models “good” and how that “good” is measured. There must be some metric for evaluation?

After digging around, we discovered several measures that usually get mentioned:

• perplexity
• cross-entropy
• bits-per-character

We will go into what these measures in depth in another blog, possibly by asking an AI tool to write it for us. We’ll also blog about how we built the app (spoiler alert: iteratively). And we’ll write about how we tested it (and retested and tested it again), thinking about all the ways humans can try and write robotically. 🧐

Right now, please go and play with our AI detector, or you can watch our live stream, where we discuss the tool and ChatGPT generally. We’d love to hear your thoughts.