Evaluating Dynamic Storytelling in Fables
Engineering
University of California Leadership Excellence through Advanced Degrees Program
Narrative structures are a predominant form of communication across many different cultures, and an active area of research. In the Walker Natural Language and Dialogue Systems (NLDS) Lab, we are interested in dialogue and dynamic adaptation of system dialogue behavior, and are currently developing automated storytelling technology that is capable of generating different tellings of the same story and answering questions about the story. Automated generation of storytelling has many broad impacts in question answering, summarization, dynamic retellings of stories from different perspectives, and interactive narrative. We combine technology for natural language generation with a deep story representation called the Story Intention Graph to model classic narratives such as Aesop’s Fables. We collect annotations to train supervised learning algorithms for predicting discourse connectives when generating story retellings, with the goal of improving the coherence of automated storytelling. We evaluate our approach using two web-based annotation experiments. The experiment collects human perceptions of the semantic relations between events in the stories. Participants are required to select the best discourse connective, representing a semantic relation such as cause or contrast, between clauses in twenty Aesop’s Fables. We measured agreement between annotators using Krippendorff’s alpha and are performing ongoing evaluation studies on a clause by clause basis.
