Script-to-Storyboard: A new contextual retrieval dataset and benchmark  

作　　者：Xi Tian Yong-Liang Yang Qi Wu 

机构地区：[1]Department of Computer Science,University of Bath,Bath BA27AY,UK [2]Australian Institute for Machine Learning,School of Computer Science,The University of Adelaide,Adelaide,SA 5005,Australia

出　　处：《Computational Visual Media》2025年第1期103-122,共20页计算可视媒体(英文版)

基　　金：supported by RCUK grant CAMERA(EP/M023281/1,EP/T022523/1);the Centre for Augmented Reasoning(CAR)at the Australian Institute for Machine Learning,and a gift from Adobe.

摘　　要：Storyboards comprising key illustrations and images help filmmakers to outline ideas,key moments,and story events when filming movies.Inspired by this,we introduce the first contextual benchmark dataset Script-to-Storyboard(Sc2St)composed of storyboards to explicitly express story structures in the movie domain,and propose the contextual retrieval task to facilitate movie story understanding.The Sc2St dataset contains fine-grained and diverse texts,annotated semantic keyframes,and coherent storylines in storyboards,unlike existing movie datasets.The contextual retrieval task takes as input a multi-sentence movie script summary with keyframe history and aims to retrieve a future keyframe described by a corresponding sentence to form the storyboard.Compared to classic text-based visual retrieval tasks,this requires capturing the context from the description(script)and keyframe history.We benchmark existing text-based visual retrieval methods on the new dataset and propose a recurrent-based framework with three variants for effective context encoding.Comprehensive experiments demonstrate that our methods compare favourably to existing methods;ablation studies validate the effectiveness of the proposed context encoding approaches.

关 键 词：DATASET BENCHMARK text-based image retrieval MOVIE 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]