What I learned building the 'Improv with Me' GPT
If you haven't tired the 'Improv with me' GPT, click here.
I created this GPT so that I could test and learn the creative limits of ChatGPT. I thought it would be fun to work with the current AI tech to create an improv scene. Does it know improv rules? Can I prompt it to improve its play and follow all the "guidelines" in improv?
Improv is tricky because there technically is no rules, but there are guidelines, especially for beginners, that typically make it easier to create a scene. Could ChatGPT follow the rules and understand how to play this very complicated game?
Key takeaways
Overall, I was really impressed that after some tweaking, the GPT was able to create fun and lively scenes. Even with general prompting the GPT demonstrated a solid understanding of how to interact, and it enthusiastically tried to create the scene. The experience was similar to playing with a level one improv player. I'll detail the specifics of how it played as a beginner later because I find it fascinating.
After trying a few scenes through text, I had an idea. Let's try it on my phone and use voice-only. So, I pulled up 'Improv with Me' on my phone and started the scene. And WOW. The experience was incredible. Somehow the voice was able to vary how they emphasized certain words. The voice wasn't bland. It felt as if it had a personality. Being able to chat and have a playful and interesting conversation wasn't something I predicted before I started this experiment.
The expressions and tonal voice variations very much felt like someone who had been on stage and acting before. Unlike a new player, the LLM wasn't nervous; you couldn't sense any hesitation in its voice. The confidence it had could take years for a beginner to develop. The confidence aspect may seem trivial, but it's an important quality to have on stage.
How the GPT was similar to a beginner improv player
What most surprised me about playing with the "Improv with me GPT" is how closely it resembled a beginner improv player. It was as if I was in a level one class. Here are a few things it did to resemble a beginner player:
Always was excited, no other emotions
I tried to get around the GPT always being excited by prompting it to think of the "wheel of emotions" and to choose an emotion and stick with it before the scene begins. This direction helped a little a the beginning, but the GPT seemed to slowly more back to excited.
Used pineapple as it's first word suggestion
This feature completely blew me away. It's an ongoing joke that if you ask an audience for a word suggestion, they default to a handful of words, one of which is pineapple. That's why you may hear an improv player ask for a non food word. Well literally the first word the GPT suggested when I prompted it to suggest a random word was pineapple. So weird.
Wanted to use the suggested word
Similar to a beginner player, the GPT made the scene about the suggested word. For example, if the word was umbrella, it would want to use the word in its first reply. Even if I didn't use the word, it felt compelled to incorporate the word. This happens so often in beginner classes that we've often banned using the suggested word in scenes as an exercise to think more broadly.
Wanted to make the scene about the suggested word
Many of the suggestions it made were things. As the scene moved forward, the GPT wanted to make the scene about that thing. In improv, when developing a scene you typically want to focus on relationships, but the thing. Even when I tried to move the scene to relationships, the GPT brought it back to the thing.
Gave way too much information when it talked
I found that the GPT wanted to provide a lot of information when it was its turn to reply. Usually three sentences. It provided a lot of information about the scene instead of letting it happen naturally. I tried to prompt the LLM to vary the amount of sentences it provides. This helped some, but as the convo went on, the information continued to be a lot.
Consistently gave the same about of dialogue
The sentence length for the GPT provided tended to be the same length. The three sentences mentioned about tended to be the same length. I prompted the GPT to vary its sentence length. This helped some, but as the convo went on, it seemed to forget the prompt. I found this "forgetting" to be similar to a beginner player. As if you provide the player feedback on a scene, they begin the scene with your feedback and slowly move back to their bad habits.
Asked a lot of questions
A common habit of new players is to ask a lot of questions. They would start speaking and want to end with a question. Unfortunately, questions tend to lead to more of an interview than building a scene. The GPT would end most of its responses with a question. I prompted the GPT to not ask questions and let the scene build naturally. Again, this helped some but not nearly enough to make a quality scene.
Limited information understanding in the prompt
I think one of the limitations of working with the GPT is its limited ability to take understand all the information in the prompt. I've read that the LLM places greater emphasis on the beginning and end of a more lengthy prompt. The LLM places loses emphasis on the middle of the prompt. That was my experience as well. If I added more improv "guidelines", it typically forgot some other aspect I prompted earlier.
Going forward, I'm curious to know if GPT-5 will be able to take the suggestions and stick with them. Or, if there's a way to train the GPT in a more live environment. For example, telling the GPT, "Instead of saying this, say this instead and here's why." This type of feedback feature would be helpful to train a customer service chatbot as well.
What I learned about Gen AI by creating and updating the GPT prompt
The GPT is infinitely more creative than people
You can ask it questions and it will always come up with answers. Quickly. Typically, you don't want to ask a person a question that puts them on the spot. For example, "Oh yeah, if you like railroad so much, when don't you come up with a rap about it?" Only an experienced player could dive into this with no worries. With the GPT, the creative abilities could come up with a response quickly, without hesitation.
Current gen AI isn't ready for a more complex creative experience When I tried to get the scene to move to a new location, The GPT was reluctant. For example, in one scene were in the library and I was pushing the GPT to come with me to approach two other people in the library. The GPT wanted me to go, but it didn't want to go. It felt like it was stuck in that scene.
Struggled to introduce new characters
Another creative limitation I encountered was when trying to introduce new characters into the scene. Quickly, the scene became confusing with multiple character. So, I prompted the GPT to label general descriptions of the characters and give them a name if their name was introduced. The names would allow us to know what characters were speaking. The character naming sometimes worked well in the beginning, but the naming wasn't consistent. My experience was that the GPT had limited memory capability. Maybe GPT-5 will have a better memory.
The GPT struggled to understand and apply humor
I prompted the GPT to say "Eeennnddd Sceeennnee" on what it believed to be a humorous note. It often tried to end the scene on something weird that happened, but wasn't always humorous. I was impressed by its ability to end a scene, but it still struggles to recognize humor.
The GPT had unlimited confidence
Playing with the GPT brought about a strange feeling. It was making beginner improv mistakes, but there was no nervousness in its voice. Beginners have hesitation in their voice. Subconsciously, they're thinking, "Am I saying the wrong thing?" The GPT has none of self-doubt.
Another important aspect to improv is status. Displaying confidence is a high status trait. Character in a scene are often either higher or lower status. For example, a teacher and their students. Within scenes the GPT was very insistent on its idea. If I introduced a new topic, it would often acknowledge it, but then go back to its original idea. Such as, the weather. Being insistent is a high status, confident move. You typically don't get this type of high status play out of a beginner improv player. It takes someone with irrational confidence to act like this in an improv environment.
I enjoyed verbally countering the GPT's insistent play. However, The interaction made me consider a larger question: In what other areas of our lives will this overly confident technology interact with us? Sales? Therapy? Coaching? Friendship? The AI is relentless in its pursuits. It doesn't get tired. To me, it's a scary thought. This confident, relentless pursuit is something I'll be aware of and monitor as the LLM tech evolves.
Ultimately, after struggling with moving the scene to a new location and introducing new characters, I decided to keep the scene simple. Two people in one location. I prompted the GPT to move the scene if the other player indicated the scene should move. Eg, from inside a garage to inside a car. But still, it struggled to make that move. Similar to naming characters, it will be interesting to see if the GPT will understand more complex improv and be able to move scenes with the player. Or even better (and more fun), make the initiation to move the scene.
Takeaways for online learning
Here are a few ways I think the tech could be applied in a learning environment:
More creative test taking
I envision a future where there is a multiple step test that changes its question depending on how the student answers. For example, maybe the first question is a scenario based question. The question is same for every student. The following questions in the scenario take the previous responses and incorporates the student's response. This will customize the questions and keep the learner engaged by changing the question stems instead of a bland, standard question.
Since the GPT seems to be good with asking questions, I envision a GPT that further investigates the learner's response. For a more challenging quiz or exercise, a Socratic method of questioning could be prompted to challenge the learner's ideas. This type of questioning would be difficult to test, it could certainly be more of a challenging exercise.
Interviews with historical figures
This ongoing conversation format could result in a rich conversation with historical figures. Grading could include creative question asking and getting information from the figure that may not typically be included in textbooks. This is a skills that current student don't develop in skills. It's like the quote, "A wise many doesn't give the right answers, he asks the right questions." Currently, students, particularly before college, aren't trained to ask the right questions.
Speaking exercises instead of writing exercises
Exercises and quizzes, don't need to be only text based. A more conversational format could lend to a different type of learning experience. Similar to how many professions use talking to communicate ideas, there could be learning experiences and discussions through conversations instead of staring at a blank page. The GPT could also assist and provide a prompt to get the learner thinking. Soon, staring at a blank page will be something we no longer have to deal with.
Additionally, if there is a group conversation within a learning environment. The AI could chime in and provide questions to challenge the conversation. From my experience with the improv GPT, I don't think the current AI has this capability to understand at this level. But, the next iteration may.
Wrap-up
Working with the "Improv with me" GPT has me enthusiastic about the future of learning and test taking. Students will be able to learn in so many new engaging formats. Plus, the AI should be able to adapt question difficulty to the learner. For example, the scenario-based questions it proposes could become more abstract and challenge the learner, if the GPT believes the learner is grasping the concept.
I'm also excited by the abilities of the GPT-5 to create an improv scene. With more memory in the prompt and within the chat, I believe an extended improv scene will be much more fun to play.
Regardless of the GPT struggling with the improv details, the voice capabilities, were incredible. The ability of its voice to accentuate tones and place emphasis on certain words felt like I was playing with a trained actor. The voice is much farther ahead than I thought it would be.
Also, incorporating voice and speaking and debating a GPT is a new exercise capability that can also change the way we teach.
I look forward to experimenting with GPTs and building my own no code learning apps that incorporate gen AI. Specifically, I'm curious about how to build interactive open ended quizzes, and scenario based exercises. I'll continue to explore and share.