色婷婷色综合,亚洲天堂2014,亚洲精品2区,亚洲午夜一区二区

<返回

Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment

Hao Fei; Shengqiong Wu; Meishan Zhang; Min Zhang; Tat-Seng Chua; Shuicheng Yan

IEEE Transactions on Pattern Analysis and Machine Intelligence

April 2024

Keywords: Videos, Semantics, Transformers

Abstract:

While pre-training large-scale video-language models (VLMs) has shown remarkable potential for various downstream video-language tasks, existing VLMs can still suffer from certain commonly seen limitations, e.g., coarse-grained cross-modal aligning , under-modeling of temporal dynamics , detached video-language view . In this work, we target enhancing VLMs with a fine-grained structural spatio-temporal alignment learning method (namely Finsta). First of all, we represent the input texts and videos with fine-grained scene graph (SG) structures, both of which are further unified into a holistic SG (HSG) for bridging two modalities. Then, an SG-based framework is built, where the textual SG (TSG) is encoded with a graph Transformer, while the video dynamic SG (DSG) and the HSG are modeled with a novel recurrent graph Transformer for spatial and temporal feature propagation. A spatial-temporal Gaussian differential graph Transformer is further devised to strengthen the sense of the changes in objects across spatial and temporal dimensions. Next, based on the fine-grained structural features of TSG and DSG, we perform object-centered spatial alignment and predicate-centered temporal alignment respectively, enhancing the video-language grounding in both the spatiality and temporality. We design our method as a plug&play system, which can be integrated into existing well-trained VLMs for further representation augmentation, without training from scratch or relying on SG annotations in downstream applications. On 6 representative VL modeling tasks over 12 datasets in both standard and long-form video scenarios, Finsta consistently improves the existing 13 strong-performing VLMs persistently, and refreshes the current state-of-the-art end task performance significantly in both the fine-tuning and zero-shot settings.

View More PDF>>

主站蜘蛛池模板: 大宁县| 上林县| 阿合奇县| 许昌市| 桓仁| 永川市| 新乐市| 军事| 全椒县| 崇义县| 抚州市| 巨野县| 锡林浩特市| 横峰县| 秀山| 景谷| 东阳市| 鄂伦春自治旗| 岢岚县| 克山县| 西畴县| 合山市| 阿城市| 张北县| 二连浩特市| 如东县| 朔州市| 阆中市| 临海市| 新余市| 府谷县| 平谷区| 开平市| 和硕县| 天镇县| 疏勒县| 鄂伦春自治旗| 德阳市| 六盘水市| 新民市| 海原县|