IN Question
Can Humans Learn To Spot Fake Text?
The most recent generation of chatbots has surfaced longstanding concerns about the growing sophistication and accessibility of artificial intelligence.
Fears about the integrity of the job market — from the creative economy to the managerial class — have spread to the classroom as educators rethink learning in the wake of ChatGPT.
Yet while apprehensions about employment and schools dominate headlines, the truth is that the effects of large-scale language models such as ChatGPT will touch virtually every corner of our lives. These new tools raise society-wide concerns about AI’s role in reinforcing social biases, committing fraud and identity theft, generating fake news, spreading misinformation and more.
A team of researchers, led by Chris Callison-Burch, Associate Professor in Computer and Information Science (CIS), is seeking to empower tech users to mitigate these risks. In the largest-ever human study on AI detection, the group demonstrated that people can learn to spot the difference between machine-generated and human-written text.
“We’ve shown that people can train themselves to recognize machine-generated texts,” says Callison-Burch. “People start with a certain set of assumptions about what sort of errors a machine would make, but these assumptions aren’t necessarily correct. Over time, given enough examples and explicit instruction, we can learn to pick up on the types of errors that machines are currently making.”
“AI today is surprisingly good at producing very fluent, very grammatical text,” adds Liam Dugan, a doctoral student in the Callison-Burch Lab. “But it does make mistakes. We prove that machines make distinctive types of errors — common-sense errors, relevance errors, reasoning errors and logical errors, for example — that we can learn how to spot.”
The study leverages data collected using Real or Fake Text?, an original web-based training game developed by the team.
Daphne Ippolito, the study’s co-leader and a former doctoral student in the Callison-Burch Lab, originated the idea for Real or Fake Text? while co-teaching a special topics course on text generation with Callison-Burch during the spring of 2020. Ippolito suggested to Dugan, then a master’s student in the course, and fellow student Arun Kirubarajan that the duo develop a gamified platform for detecting generated text as their final project. “After Arun and I finished the project and seeing its potential, Daphne worked with us to flesh it out into a bona fide research paper,” says Dugan.
Real or Fake Text? transforms the standard experimental method for detection studies into a more accurate recreation of how people use AI to generate text.
“In standard methods, participants are asked to indicate in a yes-or-no fashion whether a machine has produced a given text,” says Ippolito, now an Assistant Professor at Carnegie Mellon University. “This task involves simply classifying a text as real or fake, and responses are scored as correct or incorrect, which is not ideal for training users.”
In contrast, the Penn team’s model significantly refines the standard detection study into an effective training task by showing examples that all begin as human-written. Each example then transitions into generated text, asking participants to mark where they believe this transition begins. Trainees identify and describe the features of the text that indicate error and receive a score.
The study’s results show that participants scored significantly better than random chance, providing evidence that AI-created text is, to some extent, detectable.
“Our method not only gamifies the task, making it more engaging, it also provides a more realistic context for training,” says Dugan. “Generated texts, like those produced by ChatGPT, begin with human-provided prompts.”
The study speaks not only to artificial intelligence today, but also outlines a reassuring, even exciting, future for our relationship to this technology.
“Five years ago,” says Dugan, “models couldn’t stay on topic or produce a fluent sentence. Now, they rarely make a grammar mistake. Our study identifies the kind of errors that characterize AI chatbots, but it’s important to keep in mind that these errors have evolved and will continue to evolve. The shift to be concerned about is not that AI-written text is undetectable. It’s that people will need to continue training themselves to recognize the difference and work with detection software as a supplement.”
“People are anxious about AI for valid reasons,” says Callison-Burch. “Our study gives points of evidence to allay these anxieties. Once we can harness our optimism about AI text generators, we will be able to devote attention to these tools’ capacity for helping us write more imaginative, more interesting texts.”
“My feeling at the moment is that these technologies are best suited for creative writing,” he continues. “News stories, term papers or legal advice are bad use cases because there’s no guarantee of factuality.”
“There are exciting, positive directions that you can push this technology in,” says Dugan. “People are fixated on the worrisome examples, like plagiarism and fake news, but we know now that we can be training ourselves to be better readers and writers.”
Story by Devorah Fischler, originally published in Penn Engineering Today