COMP6411C: Advanced Topics in Multimodal Machine Learning

Spring 2025, Computer Science and Engineering, Hong Kong University of Science and Technology

Instructor: Long Chen
Class Time: Tuesday 10:30pm - 11:50pm, Thursday 10:30am - 11:50am (Room 4580)
Email: longchen@ust.hk (For course-related queries, please use the subject starting from [COMP6411C])
Teaching Assistant: Mr. Chaolei Tan (ctanak@connect.ust.hk) and Mr. Jiazhen Liu (jliugj@connect.ust.hk)
For those who have enrolled the COMP6411C course, if you want to get the recorded videos for absent classes, you can direct sent emails to the TA.

Course Description: This course provides a comprehensive introduction to recent advances in multimodal machine learning, with a focus on vision-language research. Major topics include multimodal understanding (including translation, multimodal reasoning, multimodal alignment, multimodal information extraction), multimodal generation, multimodal pretraining and adaptation, and recent techniques and trends in multimodal research. The course structure will primarily consist of instructor presentation, student presentation, in-class discussion, and a course final project.

Course Objectives: After completion of this course, students will understand mainstream multimodal topics and tasks, and develop their critical thinking and problem solving, such as identifying and explaining the state-of-the-art approaches for multimodal applications.

Pre-requisite: Basic understanding of probability and linear algebra is required. Familiarity or experience with machine learning (especially deep learning) and computer vision basics are preferred.

Grading scheme:

  • Class attendance and in-class discussion: 20%
  • Project presentation: 30%
  • Final project report: 50%

Reference books/materials:

  • Conferences: Proceedings of CVPR/ICCV/ECCV, ICLR/ICML/NeurIPS, ACL/EMNLP/ACM Multimedia
  • Book: Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Syllabus / Schedule
Lecture
Data
Reading Materials
Lec: 1.1 Course Introduction and Overview
Course overview
Feb 04
Lec: 2.1 Multimodal Translation
Captioning
Feb 06
Lec: 2.2 Multimodal Reasoning
Visual Question Answering
Feb 11
Lec: 2.3 Multimodal Alignment
Grounding, Matching
Feb 13
Lec: 2.4 Multimodal Information Extraction
Feb 18
Lec: 3.1 Generation Basics
Diffusion Models
Feb 20
Lec: 3.2 Image Generationa and Editing
Feb 25
Lec: 3.3 Video Generation and Editing
Feb 27
Lec: 4.1 RLHF Basics (1)
Mar 04
Lec: 4.2 RLHF Basics (2)
Mar 06
Lec: 5.1 Multimodal Pretraining
Mar 11
Lec: 5.2 Adapting Pretrained Models
Mar 13
Lec: 6.1 In the era of LLMs and MLLMs
Mar 18
Pre: 2.5 Image-based Multimodal Understanding
Mar 20
Pre: 2.6 Video-based Multimodal Understanding
Mar 25
Pre: 3.4 Image Generation and Editing
Mar 27
Pre: 3.5 Video Generation and Editing
Apr 08
Pre: 4.3 RLHF for Multimodal Generation Model
Apr 10
Pre: 5.3 Multimodal Pretraining
Apr 15
Pre: 5.4 Adapting Pretrained Models
Apr 17
Pre: 6.2 Building Multimodal LLMs (MLLMs)
Apr 22
Pre: 6.3 LLM-enhanced Multimodal Understanding
Apr 24
Pre: 6.4 LLM-enhanced Multimodal Generation
Apr 29
Pre: 6.5 Limitations in Today’s MLLM
Hallucination
May 06
Pre: 4.4 RLHF for MLLM
May 08
Acknowledgements

This course was inspired by and/or uses reserouces from the following courses:

MultiModal Machine Learning by Louis-Philippe Morency, Carnegie Mellon University, Fall 2023.
Advanced Topics in MultiModal Machine Learning by Louis-Philippe Morency, Carnegie Mellon University, Spring 2023.
Advances in Computer Vision by Bill Freeman, MIT, Spring, 2023.
Deep Learning for Computer Vision by Fei-Fei Li, Stanford University, Spring 2023.
Natural Language Processing with Deep Learning by Christopher Manning, Stanford University, Winter 2023.
Deep Learning for Computer Visionby Justin Johnson, University of Michigan, Winter 2022.