Term Paper Turing Test

Submitter: Paul Fyfe, North Carolina State U

——————————————————

The experiment:

I assigned students to “cheat” on their final paper by writing with the help of text-generating AI. Their task was to generate and blend any combination of human/machine-written text so that I could not tell which was which. Then students appended reflections on the experience (without AI) in context of several guiding questions: what was their process like? Did they feel they were “cheating”? Why or why not? What ideas about writing, critical thinking, authorship, plagiarism, &c. did this exercise test or challenge for them? Would or should you use these tools again and in what circumstances? Students used a web-based writing platform called Lex, or whatever text-generating AI they were comfortable with. I set no quotas about the amount of generated text required, nor rules about how it should be integrated with their work. Experimentation was strongly encouraged. Students were free to “write” as they pleased, so long as their paper engaged with our course discussions about AI and cited at least a few things from the syllabus. These included introductions to machine learning, articles about large language models and algorithmic bias, and journalistic commentary about the impact of AI on creativity, writing, and education. The goal was to elicit critical reflection through creative experiments.

Results:

Every student initially thinks that this assignment will be easy; they all eventually report the opposite. Students tend to realize that generating text is very different from “writing,” which still requires their involvement across a cognitive spectrum of ideation, composition, selection and arrangement, editing, and revising. The assignment works well to break down myths and presumptions about generative AI, encouraging students to connect to the critiques we’ve read for class. At the same time, the assignment also makes room for students to experiment—importantly without the threat of punishment—in imagining how AI tools might be reasonably integrated in their own work. It invites students to participate in envisioning norms, rules, and guardrails for how AI should be used (or not) in education. Thus, the assignment takes a hands-on approach to developing principles for responsible AI and critical AI literacy.

Ultimately, this assignment moves the discussion beyond cheating to the array of complex issues and questions we have yet to settle about generative AI. For that reason, I have left the assignment open ended, and do not model for them in advance how to use these platforms. While disorientation is part of the point, this presents a challenge for some students unsure how to begin, or how their work will be assessed. In response, I have emphasized the importance of their process reflections and shared a rubric in advance.

Relevant resources:

• https://wac.colostate.edu/repository/collections/textgened/ethical-considerations/the-term-paper-turing-test/
• https://doi.org/10.1007/s00146-022-01397-z

Contact:

• Email: paul[DOT]fyfe[AT]ncsu[DOT]edu
• Faculty page: https://go.ncsu.edu/pfyfe
• Bluesky: https://bsky.app/profile/pfyfe.bsky.social