Thesis Modeling and Evaluation with ChatGPT

Submitter: Jennifer Duncan, Perimeter C, Georgia State U

——————————————————

The experiment:

As freshman composition students began preparing to write rhetorical analysis essays, this guided activity served as a way to integrate an AI into the planning stage of their writing both to understand the elements of a rhetorical analysis and to evaluate AI generated work.

Students were assigned an essay written within the last year to serve as the basis of their rhetorical analysis. After reading the essay, students asked ChatGPT to write a thesis statement for a rhetorical analysis of that essay. Students then answered the following questions:

• How is the ChatGPT thesis evaluating the essay?
• What rhetorical elements does ChatGPT think are the most important in this essay? Do you agree or disagree? Why? What elements do you think are the most important in this essay?
• Do you agree with what ChatGPT says about the effectiveness of this essay?
• What hallucinations can you find in this thesis statement?

After answering the questions, students developed their own thesis statement for the rhetorical analysis to share with the class.

Results:

Because I used only recent essays from an ebook in our library’s collection, ChatGPT was unable, at the time, to write accurate thesis statements so it developed nice sentences with erroneous content. Several students pointed out that the ChatGPT generated thesis did not seem to be related to the actual essay which gave us the chance to talk about hallucinations and the importance of questioning everything ChatGPT produced.

Because ChatGPT did mimic the correct elements and patterns of a rhetorical analysis essay’s thesis, students were able to see a model of what a thesis requires which helped them understand the elements necessary for their own thesis. Students seemed happy to contradict the AI’s choices which helped them build confidence in their own ideas. ChatGPT generated thesis statements also often took the form of announcements (This essay will…) which students knew were forbidden in my class, so it gave them another opportunity to talk about the limitations of the formulas ChatGPT could produce.

This simple assignment was more effective at helping students craft thesis statements than my previous method of having them write first, then evaluate with their classmates. Because each student had to actively generate their own model, rather than just looking at one I produced, they felt more connected to the assignment, and because they were critiquing “the robot” rather than each other, they were more willing to point out its flaws.

Contact: jduncan[AT]gsu[DOT]edu