Mengshi Qi

I am currently a Professor in School of Computer Science at Beijing University of Posts and Telecommunications (BUPT) where I lead the Multimedia Group (BUPT-MM).

Prior to that, I have ever worked as a post-doc research scientist in CVLab at École polytechnique fédérale de Lausanne (EPFL) from 2019 to 2021, working closely with Prof. Pascal Fua and Dr. Mathieu Salzmann. And I have worked in Baidu Research, where I focus on computer vision and deep learning collaborated with Prof. Yi Yang in 2019.

I did my PhD and Master at Beihang University (BUAA) in 2019 and 2014, respectively, where I was advised by Prof. Yunhong Wang. Especially, I had been a visiting PhD at University of Rochester supervised by Prof. Jiebo Luo from 2017 to 2018. I did my bachelors at Beijing University of Posts and Telecommunications (BUPT) in 2012.

Email  /  CV  /  Google Scholar  /  LinkedIn

I'm interested in computer vision and machine learning, especially scene understanding, 3D reconstruction and multimedia analysis. Most of my research are about how to understand the semantic content and infer the physical information from images and videos.

[2024.5.15] Call for papers on IEEE TMM Special Issue on Large Multi-modal Models for Dynamic Visual Scene Understanding, please refer to https://signalprocessingsociety.org/blog/ieee-tmm-special-issue-large-multi-modal-models-dynamic-visual-scene-understanding.

In this paper, we propose a novel semi-supervised method, which can be utilized for better assessment of the AQA task by exploiting a large amount of unlabeled data and a small portion of labeled data.

In this paper, we propose a novel Decomposed Vector-Quantized Variational Autoencoder~(DVQ-VAE) to address this limitation by decomposing hand into several distinct parts and encoding them separately.

In this paper, we propose the semantic graph Transformer (SGT) for 3D scene graph generation. The task aims to parse a cloud point-based scene into a semantic structural graph, with the core challenge of modeling the complex global structure.

In this paper, we propose a novel weakly supervised temporal action localization method by inferring salient snippet-feature.

In this paper, we design a novel two-branch end-to-end fusion network named RDFC-GAN, which takes a pair of RGB and incomplete depth images as input to predict a dense and completed depth map.

In this paper, we propose a novel approach, Mutual Distillation Learning For Person Re-identification (termed as MDPR), which addresses the challenging problem from multiple perspectives within a single unified model, leveraging the power of mutual distillation to enhance the feature representations collectively.

In this paper, we propose a Disentangled Counterfactual Learning~(DCL) approach for physical audiovisual commonsense reasoning. The task aims to infer objects' physics commonsense based on both video and audio input, with the main challenge is how to imitate the reasoning ability of humans.

In this paper, we are the first to introduce an unsupervised way to predict self-driving attention by uncertainty modeling and driving knowledge integration.

In this paper, we propose GaitReload, a post-processing adversarial defense method to defend against AWP for the gait recognition model with sequenced inputs.

In this paper, we design a novel two-branch end-to-end fusion network, which takes a pair of RGB and incomplete depth images as input to predict a dense and completed depth map.

In this paper, we propose a novel binary representation learning framework, named Semantics-aware Spatial-temporal Binaries (S²Bin), which simultaneously considers spatial-temporal context and semantic relationships for cross-modal video retrieval. By exploiting the semantic relationships between two modalities, S²Bin can efficiently and effectively generate binary codes for both videos and texts. In addition, we adopt an iterative optimization scheme to learn deep encoding functions with attribute-guided stochastic training.

In this paper, we present a novel Latent Memory-augmented Graph Transformer (LMGT), a Transformer based framework including a designed graph encoding module and a latent memory unit for visual story generation.

In this paper, we proposed a novel Imitative Non-Autoregressive Modeling method to bridge the performance gap between autoregressive and non-autoregressive models for temporal sequence forecasting and imputation. Our proposed framework leveraged an imitation learning fashion including two parts, i.e., a recurrent conditional variational autoencoder (RC-VAE) demonstrator and a nonautoregressive transformation model (NART) learner.

In this paper, we address few-shot video classification by learning an ensemble of SlowFast networks augmented with memory units. Specifically, we introduce a family of few-shot learners based on SlowFast networks which are used to extract informative features at multiple rates, and we incorporate a memory unit into each network to enable encoding and retrieving crucial information instantly.

In this paper, we present a novel Generative Adversarial Networks-based model (i.e., STC-GAN) for predictive scene parsing. STC-GAN captures both spatial and temporal representations from the observed frames of a video through CNN and convolutional LSTM network. Moreover, a coupled architecture is employed to guide the adversarial training via a weight-sharing mechanism and a feature adaptation transform between the future frame generation model and the predictive scene parsing model.

In this study, we present a novel hierarchical recurrent neural network based framework with an attention mechanism for sports video captioning, in which a motion representation module is proposed to capture individual pose attribute and dynamical trajectory cluster information with extra professional sports knowledge, and a group relationship module is employed to design a scene graph for modeling players’ interaction by a gated graph convolutional network.

In the paper, we present a novel attentive semantic recurrent neural network (RNN), namely stagNet, for understanding group activities and individual actions in videos, by combining the spatio-temporal attention mechanism and semantic graph modeling. Specifically, a structured semantic graph is explicitly modeled to express the spatial contextual content of the whole scene, which is afterward further incorporated with the temporal factor through structural- RNN.

In this study, we propose a novel Attentive Relational Network that consists of two key modules with an object detection backbone to approach this problem. The first module is a semantic transformation module utilized to capture semantic embedded relation features, by translating visual features and linguistic features into a common semantic space. The other module is a graph self-attention module introduced to embed a joint graph representation through assigning various importance weights to neighboring nodes.

In this paper, we propose a novel Knowledge Embedded Generative Adversarial Networks, dubbed as KE-GAN, to tackle the challenging problem in a semi-supervised fashion. KE-GAN captures semantic consistencies of different categories by devising a Knowledge Graph from the large-scale text corpus.

We propose a novel attentive semantic recurrent neural network (RNN), dubbed as stagNet, for understanding group activities in videos, based on the spatio-temporal attention and semantic graph.

In this paper, we present a novel hierarchical recurrent neural network (RNN) based framework with an attention mechanism for sports video captioning. A motion representation module is proposed to extract individual pose attribute and group-level trajectory cluster information.

We propose a new framework for online cross-modal scene retrieval based on binary representations and semantic graph. Specially, we adopt the cross-modal hashing based on the quantization loss of different modalities. By introducing the semantic graph, we are able to extract wealthy semantics and measure their correlation across different modalities.

In this paper, we introduce a novel framework towards scene classification using category-specific salient region(CSSR) with deep CNN features, called Deep-CSSR.

In this paper, we introduce a novel physics-informed temporal network~(PITN) with adversarial contrastive learning to enable precise Blood Pressure estimation with very limited data.

In this paper, we pioneer a cloud-edge collaborative inference framework for ReID systems and particularly propose a distribution-aware correlation modeling network (DaCM) to make the desired image return to the cloud server as soon as possible via learning to model the spatial-temporal correlations among instances.

In this paper, we introduce a novel memory-based method, named Motion Pattern Priors Memory Network. Our method involves constructing a memory bank derived from clustered prior knowledge of motion patterns observed in the training set trajectories.

In this paper, to address the challenges posed by the fine-grained and interrelated vision tool usage of VIoT, we build VIoTGPT, the framework based on LLMs to correctly interact with humans, query knowledge videos, and invoke vision models to accomplish complicated tasks.

In this paper, we propose a Dual-stage Spatial-Channel Transformer for coarse-to-fine video denoising, which inherits the advantages of both Transformer and CNNs.

In this paper, we introduce an effective approach to exploit 3D geometric constraints within a cycle generative adversarial network (CycleGAN) to perform domain adaptation. Furthermore, we propose to enforce short- and long-term temporal consistency to fine-tune the domain-adapted model in a self-supervised fashion.

Thanks to this awesome guy.