How accurate are AI detectors? You have probably used one, but can you explain how it works? Do you know what particular parts of your writing it can or cannot scan? We talked about AI detection accuracy with our experts, Victoria Davis, TOP-writer at Customwritings.com, and Jaime Clark, a professional writer at the Computer Science Department. Are you ready to dive right into the results of this productive discussion? Here you are, enjoy, and learn more about how you can use AI text detectors more effectively.
Jamie Clark on a Brief History of AI Detectors. Why Did We Hear Nothing About Them Before?
Steven: Jaime, as a Bachelor of Engineering, you probably know a little bit more about AI writing detection tools. Can you please start our discussion and tell us how AI writing detectors appeared? As far as I know, AI-written content has been known for a while in the IT communities, but AI detection software only became popular recently. How did that happen? How do you perceive the modern popularity of AI?
Jaime: Yes, artificial intelligence is a bit broader than it is commonly considered to be. Basically, any machine that is capable of adjusting its old code to the new lines of code is AI written content. AI-written machines are the ones that can “learn” from previous experience, i.e., previous code.
The whole history takes its roots back to Alan Turing, one of the first scientists working on artificial intelligence. He was one of my heroes during college. From that time on, engineers gradually developed machines that can more “rationally” communicate with the developer’s requests.
“I believe we have not heard about AI tools before because it was of no specific importance for the general public. For instance, we engineers knew about it much earlier. Both AI tools and detectors were just handy for small groups of people.”
Jaime Clark, professional writer at the Computer Science Department
When I studied, AI was mainly on the level of natural language processing, AI frameworks, and autonomous vehicles. I learned about the OpenAI research laboratory in 2016, as they released “OpenAI Gym.” Just a few years later, they had “won a lottery” with ChatGPT. I think it’s a good thing to have a machine to ease your life, write something for you, and then check it for you.
Victoria Davis on the Nature of AI Detection Tools. How Do They Work in the Essay Writing Sphere?
Steven: Victoria, I know that you belong to the other “camp” concerning AI writing detection and anything written by AI. Why so? Can you tell us more about how AI detectors work?
Victoria: I come from the “science settings” at the University of Waterloo, so as a science major, I learned to avoid having anything in common with AI-generated content. In such a background and work as mine, you quickly learn to value academic integrity and well-developed research methods. Also, in preparation for our meeting, I did some research and checked what an AI writing detection tool can and cannot catch when you ask it to check your essay.
“In fact, any AI-detector can’t pass a “reality check.” It is not intelligent enough to grasp all nuances of written text, whether human or AI-written.”
Victoria Davis, TOP-writer at Customwritings.com
Victoria: First, I think something is utterly wrong with the social perception of an AI detector as more intelligent than a human being. For instance, I can tell that an AI writing check works on its built-in assumptions about the average human intelligence. Therefore, if your essay writing style even slightly differs from the norm, AI detection software marks it as machine written.
Here is an example; I bet you will laugh at this one. Just a few months ago, we were talking with Nicolas Evans about what makes a good essay. I took my answer to one of the questions and checked it with Winston AI and Originality.ai. The first AI detection test highlighted my whole speech as “AI detected,” and the second AI wrote that 15% of my speech was not my writing.
Second, the common question of “How accurate are AI detectors” isn’t quite accurate itself, especially for essay writing. We need to specify what we expect from an AI detection tool. For instance, I can list two things that even the best AI detector gets wrong:
- Every AI writing detection tool oddly reacts to complex sentence structure. So, if you think and write in complex sentences, and your ideas sometimes get carried away, as mine do, you are in danger. If teachers and college professors use it to identify AI in students’ writing, they risk falsely accusing a student of using AI for writing.
- You can tell if something is strange about the sentence. AI cannot. AI detection accuracy only covers repetitiveness patterns. However, if AI wrote a text, and you just slightly reduced the repetitiveness, the AI content detector would find nothing. So, can even the best AI content detector be a credible tool to use in education? Most likely, no, it can’t.
How Accurate Are AI Detectors?
Steven: Oh, what a hot confrontation we seem to have here! As far as I understand, AI writing detector tools are a much more recent invention than AI itself. Do you think they can do better with time? To what extent can we trust them now?
Jamie: At this point, mine and Victoria’s positions will most likely align. I totally agree that we can’t actually take AI detector accuracy seriously right now. To detect AI writing precisely, you need to feed a machine way too much information about human speech, stylistics, grammar, and psychology.
“AI detectors are not useless and are not omnipotent, regardless of their accuracy. They are good at checking the general AI-written score, but there is no absolutely accurate AI yet.”
Victoria Davis, TOP-writer at Customwritings.com
Jamie: I will read you two excerpts from essay samples, with a few sentences in each one. I edited one of them and left the other one without any changes. Can you say whether it’s AI-generated or human-edited content? Here they are:
- “Domestic violence often relies on the desire for power and control. Abusers seek to dominate and manipulate their victims, and establish a sense of authority over them. This need for control grows into various forms, such as physical violence, emotional abuse, and financial coercion.”
- “At the heart of domestic violence often lies the desire for power and control. Abusers seek to dominate and manipulate their victims to establish a sense of authority over them. This need for control can manifest in various forms, such as physical violence, emotional abuse, and financial coercion.”
Victoria: I guess the first one is the one you edited a little bit. The second one seems to have more “mechanical” or artificial word structures. Sometimes, people use such phrases as “At the heart of domestic violence often lies…”. Probably, we writers do so for the entire population so that nobody else has to. But if I was to simplify the text, I would reduce such phrases. I guess you did the same, didn’t you?
Steven: I will refrain from answering now; I am just curious about the correct answer.
Jamie: Yes, you are correct. I slightly edited the first one. But I did the same AI detection test as you, Victoria. I used the writer.com ai content detector for that, and here are the results. I will show you screenshots. Steven, will you please add them to the article later?
Steven: Sure, I will.
Jamie: Thanks. So here they are, and here is the fun part. AI detections showed that the edited text has 75% human-generated content, but the initial AI-written text presents as 87% humanly written. Altogether, I will say that an accurate AI detector does not exist yet.
Victoria: But I must also acknowledge that no matter whether an accurate AI detector exists or not; AI-detector existence is definitely a good thing. It’s only a natural step to develop precautions that somewhat limit the existence of how people rely on technologies. I think that while we identify AI as a way of controlling our peers and preventing cheating, we don’t do much for scientific progress. It’s a pity that humanity uses such technologies just to check for AI writing. AI detector accuracy is not its only value.
Can AI Detector Accuracy Progress? Three Prospective Functions of a Good Writing Detector in Essay Writing
Steven: Victoria, I am actually glad that you mentioned precautions and other functions of the AI content detector. Can you think of other possible applications of AI detectors?
Victoria: Well, I can name an important way of using AI detected content to develop the essay writing sphere. Nowadays, we use AI detection accuracy as a way of measuring how likely we are to catch those who violate academic integrity. But an accurate AI detector can also teach us how to write better than a machine does.
For example, we know that if the student tends to use a lot of passive constructions and messy sentence patterns (such as repetitions, overly long sentences, etc.), that student will more likely be accused by an AI detector. But why must it be a punishment? You can turn it into a game to entertain and educate students. You can teach them to learn where the machine can’t.
Jamie: Yes, that’s a good one. I can also add that the question of AI detection accuracy and progress can make a significant difference in the essay writing sphere. For instance, professional writers have a “pool” of writing instruments in their heads all the time. We use them unconsciously and habitually.
We know that citing statistics in the first essay paragraph boosts the text’s credibility, and that narrative essays must influence the reader’s emotions. However, the “mixing capacities” of our brains are limited. But if we teach AI to accurately detect its own footprint in the text, we can teach it many more great things. I see the perspective of using AI detection in creating new genres of writing, new literary devices, and more persuasive writing styles.
Victoria: I also believe that AI detection and the whole AI-related trend can also teach us the value of thinking about people instead of thinking about products and productivity. Academic communities now are highly devoted to checking students’ work with plagiarism and AI-detecting tools. However, I would like to see the trend of the academic community regularly checking on students, their agendas, and their well-being.
AI Writing Detectors Within Education Institutions. How Can Colleges Use Them?
Steven: So, just a few concluding words. What do you think–do we need AI writing detectors in our educational system? Are they actually useful? Can we trust them?
Jamie: We must not trust them in the way we believe in the high almighty. AI detectors are on their first development milestones so far, and they do not show accurate results. However, they have a good potential for boosting both essay writing standards and the educational community.
Victoria: I think we should use all available resources, but use them wisely. Replacing your personality and writing style with AI written content only to be obligated to check and rewrite it will not save you time. Yet, you still can use these same tools to improve your writing and to learn from the machine’s mistakes.
