Asking AI To Write about Its Own Disruption

Submitter: Jon Ippolito with Greg Nelson and Troy Schotter, U of Maine

——————————————————

The experiment:

In fall 2023 I taught a course designed to compare student experiences performing digital tasks with pre- and post-AI approaches in collaboration with Greg Nelson and Troy Schotter, a UMaine Computer Science professor and grad student respectively. In the first half of the semester, 50 students wrote essays, coded software, and created media using familiar digital tools; in the second half, they performed the same tasks using generative AI.

In Week 7, for example, students revisited their first writing assignment from Week 1–identifying winners and losers in a field disrupted by a contemporary technology–but were required to lean on GPT-4 for help brainstorming, drafting, and polishing a similar essay with a different choice of field and technology. Students then wrote a detailed comparison of the two drafts. The class repeated this process for designing logos, coding game avatars, creating illustrations, writing stories, recording soundscapes, and even grading homework.

Results:

My collaborators plan to analyze and share this study with the larger educational community, but we can already see trends in student responses. For the disruptive technology essay, students were impressed with GPT-4’s essay structure, examples, and grammatical prose but felt citations were often unreliable and style sometimes overly elaborate or formal. One described the AI as a student trying really hard to sound smart. Having been coached on incremental prompting to improve results, another student remarked that it took about the same amount of time to coax a good enough essay from the chatbot as to write his original essay.

Even after the lesson, students still felt more confident writing with a traditional approach than with AI. Most felt low-to-moderate confidence about achieving their writing goals with AI, and even less confidence about how to use AI ethically. I hope with future research to figure out whether this insecurity is due to inexperience or endemic to AI tool use.

Relevant resources:

• https://umainenewmedia.net/nmd100
• https://LearnWithAI.org

Contact:

• Email: jippolito[AT]maine[DOT]edu
• Website: https://jonippolito.net
• Twitter: @jonippolito
• Mastodon: @jonippolito@digipres.club
• Bluesky: @jonippolito.bsky.social