Lu Sheng · 盛律

School of Software, Beihang University

prof_pic_v2.png

I am the Associate Professor at the School of Software, Beihang University (BUAA), China. I am also working closely with Shanghai AI Laboratory.

Before I joined BUAA in 2019, I was a postdoctoral researcher (2017-2019) in MMLab@CUHK, working with Prof. Xiaogang Wang. I received my Ph.D. from Department of Electronic Engineering at the Chinese University of Hong Kong (CUHK), advised by Prof. King Ngi Ngan (IEEE Life Fellow).

My research interests include 3D computer vision and embodied AI. My current research lies in developing generalizable models for understanding, interacting with and synthesizing 3D visual world.

To Prospective Students: I am actively looking for highly-motivated students targeted to Master or Ph.D. degree, as well as undergraduate-level research assistants. Please drop me an email with your resume if you are interested in my research.

news

Sep 14, 2024 I will serve as Area Chair of CVPR 2025.
Jun 24, 2024 SketchSampler++ (the extension of the previous ECCV 2022 paper SketchSampler) is accepted at IEEE T-PAMI! Congratulations to all co-authors！
Jun 15, 2024 Fast-BEV is finally accepted at IEEE T-PAMI! Congratulations to all co-authors！
Feb 27, 2024 MP5 and EpiDiff has been accepted at CVPR 2024! Please check out the demos in their webpages!
Jan 17, 2024 I will serve as Area Chair of ECCV 2024.
Jan 16, 2024 Our work Octavius has been accepted at ICLR 2024! It is one of the first mixture-of-experts (MoE) papers about MLLMs. All data and code are now open-sourced, stay tuned for updated!
Dec 19, 2023 Two papers are accepted at AAAI 2024!
Dec 15, 2023 I will serve as Area Chair of CVPR 2024 and ACM Multimedia 2024.
Dec 01, 2023 We have released LAMM, an open-source project for Multi-Modal Large language Models (MLLMs) and Applications as AI agents. Welcome to join us!
Sep 01, 2023 One paper is accepted at NeurIPS 2023, the Dataset and Benchmark Track.

selected publications

  1. Preprint
    neurips24_o3d.png
    Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion
    Hao Wen#, Zehuan Huang# , Yaohui Wang, Xinyuan Chen, Yu Qiao, and Lu Sheng*
    CoRR, 2024
    arXiv Code Website
  2. Preprint
    acmmm_p2w.png
    From Parts to Whole: A Unified Reference Framework for Controllable Human Image Generation
    Zehuan Huang#, Hongxin Fan# , Lipeng Wang#, and Lu Sheng*
    CoRR, 2024
    arXiv Code Website
  3. Preprint
    eccv24_minedreamer.jpg
    MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Control
    Enshen Zhou, Yiran Qin, Zhenfei Yin, Yuzhou Huang , Ruimao Zhang, Lu Sheng, Yu Qiao, and Jing Shao
    CoRR, 2024
    arXiv Code Website
  4. Preprint
    eccv24_ch3ef.png
    Assessment of Multimodal Large Language Models in Alignment with Human Values
    Zhelun Shi , Zhipin Wang, Hongxing Fan , Zaibin Zhang, Lijun Li , Yongting Zhang, Zhenfei Yin, Lu Sheng, Yu Qiao, and Jing Shao
    CoRR, 2024
    arXiv Website
  5. Preprint
    eccv24_rh20tp.png
    RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agents
    Zeren Chen, Zhelun Shi , Xiaoya Lu, Lehan He, Sucheng Qian, Haoshu Fang, Zhenfei Yin, Wanli Ouyang, Jing Shao, Yu Qiao, and 2 more authors
    CoRR, 2024
    arXiv Website
  6. IEEE T-PAMI
    pami24_fast-bev++.png
    Fast-BEV: A Fast and Strong Bird’s-Eye View Perception Baseline
    Yangguang Li, Bin Huang, Zeren Chen, Yufeng Cui, Feng Liang, Mingzhu Shen , Fenggang Liu, Enze Xie, Lu Sheng*, Wanli Ouyang, and 1 more author
    IEEE Trans. Pattern Anal. Mach. Intell., early access , 2024
    arXiv Code
  7. IEEE T-PAMI
    pami24_sketchsampler++.png
    3D Reconstruction from a Single Sketch via View-dependent Depth Sampling
    Chenjian Gao# , Xilin Wang#, Qian Yu*, Lu Sheng, Jing Zhang, Xiaoguang Han, Yi-Zhe Song, and Dong Xu
    IEEE Trans. Pattern Anal. Mach. Intell., early access , 2024
    arXiv Code
  8. CVPR
    cvpr24_epidiff.png
    EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion
    Zehuan Huang#, Hao Wen#, Junting Dong# , Yaohui Wang, Yangguang Li, Xinyuan Chen, Yan-Pei Cao, Ding Liang, Yu Qiao, Bo Dai*, and 1 more author
    In IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024
    arXiv PDF Code Website
  9. CVPR
    cvpr24_mp5.png
    MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception
    Yiran Qin#, Enshen Zhou# , Qichang Liu#, Zhenfei Yin, Lu Sheng* , Ruimao Zhang*, Yu Qiao, and Jing Shao
    In IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024
    arXiv PDF Code Website
  10. ICLR
    iclr24_octavius.png
    Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE
    Zeren Chen# , Ziqin Wang# , Zhen Wang , Huayang Liu, Zhenfei Yin , Si Liu, Lu Sheng*, Wanli Ouyang, Yu Qiao, and Jing Shao*
    In International Conference on Learning Representations , 2024
    arXiv PDF Code Website
  11. NeurIPS
    neurips23_lamm.png
    LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark
    Zhenfei Yin# , Jiong Wang#, Jianjian Cao#, Zhelun Shi# , Dingning Liu, Mukai Li, Xiaoshui Huang , Zhiyong Wang, Lu Sheng, Lei Bai*, and 2 more authors
    In Advances in Neural Information Processing Systems , 2023
    arXiv HTML Video Website
  12. CVPR
    cvpr23_siamese_detr.png
    Siamese DETR
    Zeren Chen#, Gengshi Huang#, Wei Li, Jianing Teng , Kun Wang, Jing Shao, Chen Change Loy, and Lu Sheng*
    In IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023
    PDF Code
  13. CVPR
    cvpr23_vlsat.png
    VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloud
    Ziqin Wang, Bowen Cheng, Lichen Zhao, Dong Xu, Yang Tang*, and Lu Sheng*
    In IEEE/CVF Conference on Computer Vision and Pattern Recognition (Highlight Poster) , 2023
    PDF Code
  14. CVPR
    cvpr22_3djcg.png
    3DJCG: A Unified Framework for Joint Dense Captioning and Visual Grounding on 3D Point Clouds
    Daigang Cai, Lichen Zhao, Jing Zhang*, Lu Sheng, and Dong Xu
    In IEEE/CVF Conference on Computer Vision and Pattern Recognition (Oral Presentation) , 2022
    PDF Code
  15. AAAI
    aaai22_danceformer.png
    DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformer
    Buyu Li, Yongchi Zhao, Zhelun Shi, and Lu Sheng*
    In Thirty-Sixth AAAI Conference on Artificial Intelligence , 2022
    HTML Code
  16. CVPR
    cvpr21_forgerynet.png
    ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis
    Yinan He#, Bei Gan#, Siyu Chen# , Yichun Zhou# , Guojun Yin, Luchuan Song, Lu Sheng, Jing Shao* , and Ziwei Liu
    In IEEE Conference on Computer Vision and Pattern Recognition (Oral Presentation) , 2021
    PDF Code Website
    130 Google Scholar citations
  17. CVPR
    cvpr21_brnet.png
    Back-Tracing Representative Points for Voting-Based 3D Object Detection in Point Clouds
    Bowen Cheng, Lu Sheng*, Shaoshuai Shi, Ming Yang, and Dong Xu
    In IEEE Conference on Computer Vision and Pattern Recognition , 2021
    PDF Code
  18. ICCV
    iccv21_3dvg.png
    3DVG-Transformer: Relation Modeling for Visual Grounding on Point Clouds
    Lichen Zhao, Daigang Cai, Lu Sheng*, and Dong Xu
    In IEEE/CVF International Conference on Computer Vision (1st place at 3D Object Localization Challenge at the CVPR 2021, 1st Workshop on Language for 3D Scenes) , 2021
    PDF Code
    120 Google Scholar citations
  19. ACM MM
    mm21_votehmr.png
    VoteHMR: Occlusion-Aware Voting Network for Robust 3D Human Mesh Recovery from Partial Point Clouds
    Guanze Liu, Yu Rong, and Lu Sheng*
    In Proceedings of the 29th ACM International Conference on Multimedia (Oral Presentation) , 2021
    arXiv HTML Code
  20. IJCV
    ijcv20_highquality.png
    High-Quality Video Generation from Static Structural Annotations
    Lu Sheng#*, Junting Pan#, Jiaming Guo, Jing Shao, and Chen Change Loy
    Int. J. Comput. Vis., 2020
    HTML Code
  21. AAAI
    aaai20_msn.png
    Morphing and Sampling Network for Dense Point Cloud Completion
    Minghua Liu, Lu Sheng, Sheng Yang, Jing Shao, and Shi-Min Hu
    In The Thirty-Fourth AAAI Conference on Artificial Intelligence , 2020
    PDF Code
    330 Google Scholar citations
  22. ECCV
    eccv20_f3net.png
    Thinking in Frequency: Face Forgery Detection by Mining Frequency-Aware Clues
    Yuyang Qian , Guojun Yin, Lu Sheng*, Zixuan Chen, and Jing Shao
    In European Conference on Computer Vision , 2020
    arXiv PDF
    670 Google Scholar citations
  23. IEEE T-PAMI
    tpami19_visibility.png
    Visibility Constrained Generative Model for Depth-Based 3D Facial Pose Tracking
    Lu Sheng, Jianfei Cai, Tat-Jen Cham, Vladimir Pavlovic, and King Ngi Ngan
    IEEE Trans. Pattern Anal. Mach. Intell., 2019
    arXiv HTML
  24. CVPR
    cvpr19_sdgan.png
    Semantics Disentangling for Text-To-Image Generation
    Guojun Yin , Bin Liu, Lu Sheng* , Nenghai Yu, Xiaogang Wang, and Jing Shao
    In IEEE Conference on Computer Vision and Pattern Recognition (Oral Presentation) , 2019
    PDF
    230 Google Scholar citations