色婷婷色综合,亚洲天堂2014,亚洲精品2区,亚洲午夜一区二区

<返回

Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment

Hao Fei; Shengqiong Wu; Meishan Zhang; Min Zhang; Tat-Seng Chua; Shuicheng Yan

IEEE Transactions on Pattern Analysis and Machine Intelligence

April 2024

Keywords: Videos, Semantics, Transformers

Abstract:

While pre-training large-scale video-language models (VLMs) has shown remarkable potential for various downstream video-language tasks, existing VLMs can still suffer from certain commonly seen limitations, e.g., coarse-grained cross-modal aligning , under-modeling of temporal dynamics , detached video-language view . In this work, we target enhancing VLMs with a fine-grained structural spatio-temporal alignment learning method (namely Finsta). First of all, we represent the input texts and videos with fine-grained scene graph (SG) structures, both of which are further unified into a holistic SG (HSG) for bridging two modalities. Then, an SG-based framework is built, where the textual SG (TSG) is encoded with a graph Transformer, while the video dynamic SG (DSG) and the HSG are modeled with a novel recurrent graph Transformer for spatial and temporal feature propagation. A spatial-temporal Gaussian differential graph Transformer is further devised to strengthen the sense of the changes in objects across spatial and temporal dimensions. Next, based on the fine-grained structural features of TSG and DSG, we perform object-centered spatial alignment and predicate-centered temporal alignment respectively, enhancing the video-language grounding in both the spatiality and temporality. We design our method as a plug&play system, which can be integrated into existing well-trained VLMs for further representation augmentation, without training from scratch or relying on SG annotations in downstream applications. On 6 representative VL modeling tasks over 12 datasets in both standard and long-form video scenarios, Finsta consistently improves the existing 13 strong-performing VLMs persistently, and refreshes the current state-of-the-art end task performance significantly in both the fine-tuning and zero-shot settings.

View More PDF>>

主站蜘蛛池模板: 潜山县| 望城县| 五常市| 正宁县| 铜山县| 昭苏县| 金坛市| 肥西县| 武夷山市| 饶平县| 江川县| 潢川县| 烟台市| 义马市| 慈溪市| 新巴尔虎左旗| 辽阳县| 江北区| 喜德县| 察哈| 桃江县| 陆河县| 廊坊市| 辉县市| 二手房| 望江县| 武夷山市| 屏南县| 恭城| 且末县| 金沙县| 东阳市| 德保县| 历史| 顺昌县| 南郑县| 随州市| 东阿县| 栾城县| 株洲县| 姜堰市|