COMP6411C: Advanced Topics in Multimodal Machine Learning

Spring 2025, Computer Science and Engineering, Hong Kong University of Science and Technology

Instructor: Long Chen
Class Time: Tuesday 10:30pm - 11:50pm, Thursday 10:30am - 11:50am (Room 4580)
Email: longchen@ust.hk (For course-related queries, please use the subject starting from [COMP6411C])
Teaching Assistant: Mr. Chaolei Tan (ctanak@connect.ust.hk) and Mr. Jiazhen Liu (jliugj@connect.ust.hk)
For those who have enrolled the COMP6411C course, if you want to get the recorded videos for absent classes, you can direct sent emails to the TA.

Course Description: This course provides a comprehensive introduction to recent advances in multimodal machine learning, with a focus on vision-language research. Major topics include multimodal understanding (including translation, multimodal reasoning, multimodal alignment, multimodal information extraction), multimodal generation, multimodal pretraining and adaptation, and recent techniques and trends in multimodal research. The course structure will primarily consist of instructor presentation, student presentation, in-class discussion, and a course final project.

Course Objectives: After completion of this course, students will understand mainstream multimodal topics and tasks, and develop their critical thinking and problem solving, such as identifying and explaining the state-of-the-art approaches for multimodal applications.

Pre-requisite: Basic understanding of probability and linear algebra is required. Familiarity or experience with machine learning (especially deep learning) and computer vision basics are preferred.

Grading scheme:

• Class attendance and in-class discussion: 20%
• Project presentation: 30%
• Final project report: 50%

Reference books/materials:

• Conferences: Proceedings of CVPR/ICCV/ECCV, ICLR/ICML/NeurIPS, ACL/EMNLP/ACM Multimedia
• Book: Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.

Lecture Syllabus / Schedule

Presentation Topics / Schedule

Presentation topcis including: Image-based Multimodal Understanding, Video-based Multimodal Understanding, Image Generation and Editing, Video Generation and Editing, RLHF for Multimodal Generation Model, Multimodal Pretraining, Adapting Pretrained Models, Building Multimodal LLMs (MLLMs), LLM-enhanced Multimodal Understanding, LLM-enhanced Multimodal Generation, Limitations in Today’s MLLM, RLHF for MLLM.

Student Presentation Slides

• 1) Style Aligned Image Generation via Shared Attention. In CVPR, 2024.
  

• 2) Zero-shot Image-to-Image Translation. In Siggraph, 2023.
  

• 3) a) An Image is Worth 32 Tokens for Reconstruction and Generation. In NeurIPS, 2024.
  (b) Language Model Beat Diffusion - Tokenizer is Key to Visual Generation. In ICLR, 2024.
  

• 4) SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformer. In ICLR, 2025.
  

• 5) (a) HunyuanVideo: A Systematic Framework for Large Video Generation Models. In arXiv, 2025.
  (b) SketchVideo: Sketch-based Video Generation and Editing. In CVPR, 2025.
  

• 6) (a) MotionGPT: Human Motion as a Foreign Language. In NeurIPS, 2023.
  (b) The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion. In arXiv, 2024.
  

• 7) (a) One fits all: Power general time series analysis by pretrained lm. In NeurIPS, 2023.
  (b) Time-LLM: Time Series Forecasting by Reprogramming Large Language Models. In ICML, 2024.
  

• 8) Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction. In NeurIPS, 2024.
  

• 9) FairDiffusion: Enhancing Equity in Medical Image Generation via Bayesian Perturbation. Science Advances, 2025.
  

• 10) PuLID: Pure and Lightning ID Customization via Contrastive Alignment. In NeurIPS, 2024.
  

• 11) Muse: Text-To-Image Generation via Masked Generative Trnasformation. In ICML, 2023.
  

• 12) (a) Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation. In arXiv, 2024.
  (b) Sonic: Shifting Focus to Global Audio Perception in Portrait Animation. In arXiv, 2024.
  

• 13) (a) Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold. Siggraph, 2023.
  (b) DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing. In CVPR, 2024.
  (c) MagicQuill: An Intelligent Interactive Image Editing System. In CVPR, 2025.
  

• 14) RF-Diffusion: Radio Signal Generation via Time-Frequency Diffusion. In MobiCom, 2024.
  

• 15) Zero123: Zero-shot One Image to 3D Object. In ICCV, 2023.
  

• 16) HART: Efficient Visual Generation with Hybrid Autoregressive Transformer. In ICLR, 2025.
  

• 17) (a) InstaFlow: One Step is Enough for High-Quality Diffusion-based Text-to-Image Generation. In ICLR, 2024.
  (b) PeRFlow: Piecewise Rectified Flow as Universal Plug-and-Play Accelerator. In NeurIPS, 2024.
  

• 18) VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks. In NeurIPS, 2024.
  

• 19) MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model. In CVPR, 2023.
  

• 20) Accelerating Pre-training of Multimodal LLMs via Chain-of-Sight. In NeurIPS, 2024.
  

• 21) Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training. In CVPR, 2025.
  

• 22)

Acknowledgements

This course was inspired by and/or uses reserouces from the following courses:

Multimodal Foundation Models by Jia-Bin Huang, University of Maryland, Fall 2024.
MultiModal Machine Learning by Louis-Philippe Morency, Carnegie Mellon University, Fall 2023.
Advanced Topics in MultiModal Machine Learning by Louis-Philippe Morency, Carnegie Mellon University, Spring 2023.
Advances in Computer Vision by Bill Freeman, MIT, Spring, 2023.
Deep Learning for Computer Vision by Fei-Fei Li, Stanford University, Spring 2023.
Natural Language Processing with Deep Learning by Christopher Manning, Stanford University, Winter 2023.
Deep Learning for Computer Visionby Justin Johnson, University of Michigan, Winter 2022.