COMP6411C: Advanced Topics in Multimodal Machine Learning

Spring 2024, Computer Science and Engineering, Hong Kong University of Science and Technology

Instructor: Long Chen
Class Time: Monday 13:30pm - 14:50pm, Friday 9:00am - 10:20am (Room 2504)

Email: longchen@ust.hk (For course-related queries, please use the subject starting from [COMP6411C])
Office Hour: Friday 13:00 pm - 14:00 pm (Room 3508, ZOOM: 986 8247 0232 (Passcode: 6411C))

Teaching Assistant: Mr. Chaoyang Zhu (cy.zhu@connect.ust.hk)
TA Office Hour: Monday 14:00 pm - 15:00 pm (Room 4204)
For those who have enrolled the COMP6411C course, if you want to get the recorded videos for absent classes, you can direct sent emails to the TA.

Course Description: This course provides a comprehensive introduction to recent advances in multimodal machine learning, with a focus on vision-language research. Major topics include multimodal translation, multimodal reasoning, multimodal alignment, multimodal information extraction, and recent deep learning techniques in multimodal research (such as graph convolution network, Transformer architecture, deep reinforcement learning, and causal inference). The course structure will primarily consist of instructor presentation, student presentation, in-class discussion, and a course final project.

Course Objectives: After completion of this course, students will understand mainstream multimodal topics and tasks, and develop their critical thinking and problem solving, such as identifying and explaining the state-of-the-art approaches for multimodal applications.

Pre-requisite: Basic understanding of probability and linear algebra is required. Familiarity or experience with machine learning (especially deep learning) and computer vision basics are preferred.

Grading scheme:

• Class attendance and in-class discussion: 20%
• Project presentation: 30%
• Final project report: 50%

Reference books/materials:

• Conferences: Proceedings of CVPR/ICCV/ECCV, ICLR/ICML/NeurIPS, ACL/EMNLP/ACM Multimedia
• Book: Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.

Student Presentation Slides

• 1) Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data. In ICLR, 2024
  

• 2) Learning Concise and Descriptive Attributes for Visual Recognition. In ICCV, 2023.
  LLMs are Visual Explainers: Advancing Image Classification with Evolving Visual Descriptions. In arXiv, 2024
  

• 3) SDXL: Improving Latent Diffusion Models for High Resolution Image Synthesis. In ICLR, 2024
  Wurstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models. In ICLR, 2024
  

• 4) MultiVI: Deep Generative Model for the Integration of Multimodal Data. Nature Methods, 2023.
  scGPT: Toward Building a Foundation Model for Single-Cell Multi-Omics using Generative AI. Nature Methods, 2023.
  

• 5) 3D-LLM: Injecting the 3D World into Large Language Models. In NeurIPS, 2023.
  

• 6) AudioLDM2: Learning Holistic Audio Generation with Self-supervised Pretraining. In arXiv, 2023.
  AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models. In NeurIPS, 2023.
  

• 7) Link-Context Learning for Multimodal LLMs. In CVPR, 2024.
  

• 8) Towards 3D Molecule-Text Interpretation in Language Models. In ICLR, 2024.
  

• 9) Balanced Multimodal Learning via On-the-fly Gradient Modulation. In CVPR, 2022.
  

• 10) Animatediff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning. In ICLR, 2024.
  Freenoise: Tuning-free Longer Video Diffusion via Noise Rescheduling. In ICLR, 2024.
  

• 11) ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Model. In arXiv, 2023.
  

• 12) A Symbolic Character-Aware Model for Solving Geometry Problems. In ACM MM, 2023.
  

• 13) Grounding Language Models to Images for Multimodal Inputs and Outputs. In ICML, 2023.
  Generating Images with Multimodal Language Models. In NeurIPS, 2023.
  

• 14) PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation. In arXiv, 2023.
  

• 15) Learning Universal Policies via Text-Guided Video Generation, In NeurIPS 2023.
  Large-Scale Actionless Video Pre-Training via Discrete Diffusion for Efficient Policy Learning, In arXiv 2024
  

• 16) MoE-LLaVA: Mixture of Experts for Large Vision-Language Models. In arXiv, 2024.
  

• 17) Hierarchical Adaptive Value Estimation for Multi-modal Visual Reinforcement Learning. In NeurIPS, 2023.
  

• 18) Expert Knowledge-aware Image Difference Graph Representation Learning for Difference-aware Medical Visual Question Answering. In ACM SIGKDD, 2023
  

• 19) MCL-DTI: Using Drug Multimodal Information and Bi-directional Cross-Attention Learning Method for Predicting Drug–Target Interaction. Bioinformatics, 2023.
  

• 20) Few-shot Adaptation of Multi-modal Foundation Models: A Survey. In arXiv, 2024.
  RemoteCLIP: A Vision Language Foundation Model for Remote Sensing. IEEE TGRS, 2024.
  ProtoCLIP: Prototypical Contrastive Language Image Pretraining. IEEE TNNLS, 2023.
  Visual Instruction Tuning with Polite Flamingo. In AAAI, 2024.
  

• 21) Making Large Multimodal Models Understand Arbitrary Visual Prompts. In CVPR, 2024.
  

• 22) SinicVisionLM: Playing Sound with Vision Language Models. In arXiv, 2024.
  Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners. In CVPR, 2024.
  

• 23) Multi-granularity Correspondence Learning from Long-tern Noisy Videos. In ICLR, 2024.
  

• 24) World Model on Million-Length Video and Language with RingAttention. In arXiv, 2024.
  NExT-GPT: Any-to-Any Multimodal LLM. In arXiv, 2023.
  

• 25) Training Language Models to Follow Instructions with Human Feedback. In NeurIPS, 2022.
  

• 26) Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval. In CVPR, 2024.
  Streaming Dense Video Captioning. In CVPR, 2024.

• 27) V-IRL: Grounding Virtual Intelligence in Real Life. In arXiv, 2024.
  

• 28) PMR: Prototypical Modal Rebalance for Multimodal Learning. In CVPR, 2023.
  

• 29) Self-discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generation. In CVPR, 2024.
  

• 30) DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation. In CVPR, 2024.
  

• 31) Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images. In CVPR, 2023.
  

• 32) Beyond Mono to Binaural: Generating Binaural Audio from Mono Audio with Depth and Cross Modal Attention. In WACV, 2022.
  

• 33) InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. In CVPR, 2024.
  

• 34) EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone, In ICCV, 2023.
  

• 35) When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications. In SIGIR, 2024.
  

• 36) Adversarial Diffusion Distillation. In arXiv, 2023.
  

• 37) Multimodal prompting with missing modalities for visual recognition. In CVPR, 2023.
  

• 38) Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities. In CVPR, 2024.
  

• 39) GraphTranslator: Aligning Graph Model to Large Language Model for Open-Ended Tasks. In WWW, 2024.
  

• 40) Improving multimodal datasets with image captioning. In NeurIPS, 2023.
  

• 41) FaceDNeRF: Semantics-Driven Face Reconstruction, Prompt Editing and Relighting with Diffusion Models. In NeurIPS, 2023.
  

• 42) HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion. In arXiv, 2023.
  

Syllabus / Schedule

Acknowledgements

This course was inspired by and/or uses reserouces from the following courses:

MultiModal Machine Learning by Louis-Philippe Morency, Carnegie Mellon University, Fall 2023.
Advanced Topics in MultiModal Machine Learning by Louis-Philippe Morency, Carnegie Mellon University, Spring 2023.
Advances in Computer Vision by Bill Freeman, MIT, Spring, 2023.
Deep Learning for Computer Vision by Fei-Fei Li, Stanford University, Spring 2023.
Natural Language Processing with Deep Learning by Christopher Manning, Stanford University, Winter 2023.
Deep Learning for Computer Visionby Justin Johnson, University of Michigan, Winter 2022.