Our work applies the top-down pipeline from procedural content generation using Large Language Models, but also integrates generative models and tools for asset creation.
Digital storytelling, essential in entertainment, education, and marketing, faces challenges in production scalability and flexibility. The StoryAgent framework, introduced in this paper, utilizes Large Language Models and generative tools to automate and refine digital storytelling. Employing a top-down story drafting and bottom-up asset generation approach, StoryAgent tackles key issues such as manual intervention, interactive scene orchestration, and narrative consistency. This framework enables efficient production of interactive and consistent narratives across multiple modalities, democratizing content creation and enhancing engagement. Our results demonstrate the framework's capability to produce coherent digital stories without reference videos, marking a significant advancement in automated digital storytelling.
Our work applies the top-down pipeline from procedural content generation using Large Language Models, but also integrates generative models and tools for asset creation.
Our work offers three benefits:
(1) Fine-grained control over the generated content.
(2) Character interactions with scene objects.
(3) Long-term consistency across scenes and modalities.
(1) An appendix containing all LLM prompts used in this framework;
(2) The videos and storyscripts of the examples in the paper.
(3) a Dev Report that focuses on the technical details of the project.
@inproceedings{words2worldsLi2024,
author = {Li, Danrui and Sohn, Samuel S. and Zhang, Sen and Chang, Che-Jui and Kapadia, Mubbasir},
title = {From Words to Worlds: Transforming One-line Prompts into Multi-modal Digital Stories with LLM Agents},
year = {2024},
isbn = {9798400710902},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3677388.3696321},
doi = {10.1145/3677388.3696321},
booktitle = {Proceedings of the 17th ACM SIGGRAPH Conference on Motion, Interaction, and Games},
articleno = {21},
numpages = {12},
keywords = {Communicative Agents, Digital storytelling, Large Language Models},
location = {Arlington, VA, USA},
series = {MIG '24}
}