Goethe’s Puppet Theatre: AI-powered puppet plays inspired by Goethe

Projekte Augmented and Virtual Reality Design

Mixed Reality puppet theater where users create plays with Goethe characters using AI-driven storytelling

A project by Aaron Höhn, Oscar Pohl & Phillip Premper Owor
Supervised by Prof. Dr. Frank Gabler & Prof. Philip Hausmeier
Project done in Study Semester 5 | Winter 2024/2025

About Project

Goethe’s MR Puppet Theater is a Mixed Reality installation that allows users to generate unique puppet plays by selecting historical and fictional characters from Johann Wolfgang von Goethe’s world. Designed for museum visitors and developed for Meta Quest 3, the experience revolves around a virtual puppet theater that users can place physically in their environment using the Quest controller. After placing two puppets into the theater, a unique play is generated by AI, using pre-defined prompts that shape story elements, emotional tone, and dialogue.

The project includes characters such as Goethe, Charlotte, and Mephisto, each with motion-captured animations and character-specific actions. Upon placing puppets, the theater curtains close and a short, approximately 2-minute play begins, featuring interactions entirely generated in real time using ChatGPT.

Technically, the system employs NavMesh for puppet movement and the Meta Voice SDK for real-time text-to-speech synthesis. Characters are treated as unique identities, even if duplicates are selected, which enables AI to deliver emotionally varied interactions and layered storytelling. Over 80 motion-captured animations were integrated to give life to different characters and enhance visual believability.

The experience was developed with the support of Klassik Stiftung Weimar and includes physical measurements and references from the original puppet theater. By combining storytelling, AI, and spatial computing, the project delivers a novel way for audiences to explore Goethe’s legacy through interaction and entertainment.