Composition Research Literature Review with AI

Submitter: Ehren Helmut Pflugfelder, Oregon State U

——————————————————

The experiment:

In the fall of 2023, I was teaching a course on composition pedagogy. This course includes a larger writing portfolio that focuses on a topic or theme within composition pedagogy that students choose to follow. Topics often range from integrating creative writing in high school English courses, working with students with learning disabilities, understanding 6-12 state-level literacy testing, teaching FYC through genre, and more. The portfolio must include a literature review on the topic, a brief analysis of a recent research article, and a letter to me—other materials are students’ choice. As we develop the literature review, students are often frustrated by the (admittedly challenging) task of researching topics in a field they are new to, so we tend to take things relatively slow, finding materials in existing bibliographies, reference sections, or review essays. As part of this exploratory process, I asked students to (in class) work with a generative AI model and create prompts concerning their topic, asking the AI model to locate and summarize sources for them. We covered how prompts can be made more complex and how they can be “linked” to continue an inquiry within a single AI session.

Results:

I asked students to 1) report their search prompts, 2) turn in any material relevant to their topic, 3) make notes on the materials, sources, etc. that the generative AI found, and 4) explain what they hoped to build upon. We then debriefed about their experience in class.

In general, students found that their AI prompts resulted in sources and summaries that were largely unsurprising. Students looking for materials on critical pedagogy, for example, were shown materials on bell hooks and Paulo Freire—scholars that they had also found with relatively ease working outside of the generative AI. They also found that the AI would locate materials outside of traditional US locations, including open-access journals, blog posts, or white papers. Many of these sources were either helpful or led to additional resources that were helpful. Numerous students found that the generative AI either fabricated (hallucinated) sources, authors, or both, sometimes attributing fabricated articles to real researchers. In general, the generative AI models identified foundational materials for different composition research topics, but either didn’t have access to, or didn’t receive explicit enough prompts for, recent, field-specific materials.

Relevant resources: https://genai.umich.edu/resources/prompt-literacy

Contact: ehren[DOT]pflugfelder[AT]oregonstate[DOT]edu