Testing ChatGPT Response Variety to Introduce Natural Language Processing

Submitter: Elisa Beshero-Bondar, Penn State U, Erie-Behrend

——————————————————

The experiment:

I prepared a series of assignments to introduce natural language processing (NLP) to students that began with repeated prompt experiments with ChatGPT. I was preparing them to work with Python to access libraries of word classifications and embeddings, like the free/open access spaCy library. To help orient us to large language models and the technologies of NLP, I introduced an experiment: What kinds of prompts would generate the most variation in ChatGPT’s responses when repeated? With a little experimentation the students were able to create prompts that generated multiple somewhat different responses. We then applied NLP tools to try to measure their similarity or difference based on the word embeddings data that we could access for each response. The series of assignments also included several readings to orient students to the technologies that drive generative language models to recognize patterns and “find words” that respond to prompts. These assignments succeeded in introducing students to NLP using short generated texts prior to students’ beginning to explore NLP methods with larger collections of texts.

Results:

The experiment was successful in two ways:

1. The experience of experimenting in a sustained way with prompts helped us “wrap our heads” around how ChatGPT works. Students had a goal for interacting with ChatGPT to see if they could make it generate a diverse range of responses with a common thread. What kinds of prompts could best do this? And why might these experiments work? (Students quickly found that asking ChatGPT to tell a story with a few key components would give them responses with some things in common but an interesting diversity. An interesting experiment found that stories about a girl in a white dress would always resolve in having her live in a village by the sea.)

2. It was successful in demystifying the technologies of AI, and to introduce the concepts of word embeddings data. Students also readily understood how the training of a language model would alter the nature of its replies. The idea that one could measure differences in a language model’s responses was a way of seeing how the tools of natural language processing might be used in research.

When I teach this course again in Spring 2023, I plan to expand this assignment to include other LLM chat models like Bing or ClaudeAI. We can try investigating more specific ways to generate response variety beyond exploring their storytelling functions. I’d like to try targeted searches on informational topics, but continue to explore and measure response variety.

Relevant resources: https://wac.colostate.edu/repository/collections/textgened/ai-literacy/testing-chatgpt-response-variety-to-introduce-natural-language-processing/

Contact: ebbondar[AT]gmail[DOT]com, eeb4[AT]psu[DOT]edu