The Turing Test

Submitter: Mark Marino and Mary Traester, U of Southern California

——————————————————

The experiment:

Can students tell the difference between AI generated text and human generated text?

In this exercise inspired by the Turing Test, students submit text, either written by an AI or written by themselves for the class to evaluate. The goal of the exercise is to explore students’ skills with prompting while also examining the limits of LLMs. They can work alone or in teams. In the original version of this exercise, Mark used an open-ended question about the impact LLMs will have on the future of writing. Mary extended this exercise by asking students to generate comments on a portion of an article. Students generated one comment on their own and then uploaded the article to an LLM, such as Claude or ChatPDF, to elicit a response tied to a portion. The students posted the comment as an annotation on the article, using the hypothes.is system, which Mary has written about extensively.

Results:

In both versions of the exercise, students soon realized that the best strategy was either to imitate ChatGPT’s generic output or to prompt the LLM to imitate a high school or younger aged student. The game became more a test in imitating style than recognition, perhaps matching the spirit of Turing’s imitation game. Tying the output to a passage of text rather than just soliciting attitudes or opinions helped the students also examine the limits of machine reading through these LLMs. According to Mary, students quickly learned that the LLMs, while good at summary, were bad at engaging the ideas within the piece of writing. If they were going to generate a comment on the idea level, they would have to do that themselves.

Relevant resources: https://read.dukeupress.edu/pedagogy/article-abstract/21/2/329/173415/Pedagogy-to-Disrupt-the-Echo-ChamberDigital?redirectedFrom=fulltext

Contact:

• Email: markcmarino[AT]gmail[DOT]com, traester[AT]usc[DOT]edu
• Twitter: @markcmarino