01Thu Aug 27
Course introduction
Scope of the field, seminar mechanics, how to read and present a technical paper. Presentation sign-ups.
- Affective Computing
Six units26 meetingsAug 27 – Dec 3
Schedule
Classes begin August 26 and end December 4. Fall Break, Election Day, and Thanksgiving remove three of our meetings; November 24 is held online.
The instructor reserves the right to adjust readings. The calendar dates are fixed.
Unit I
Foundations: what is being measured?
Emotion theory and its critics, then the corpora everything downstream is trained on.
02Tue Sep 1
Can emotion be inferred from behavior?
- Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements
03Thu Sep 3
Corpora I: acted and elicited emotion
- IEMOCAP: Interactive Emotional Dyadic Motion Capture Database
- CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset
04Tue Sep 8
Corpora II: naturalistic and in-the-wild data
- Building Naturalistic Emotionally Balanced Speech Corpus by Retrieving Emotional Speech From Existing Podcast Recordings
- AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
Unit II
Learned speech representations
What self-supervised models learn from raw audio, and how much of it is affect.
05Thu Sep 10
Self-supervised speech representation learning
- wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
06Tue Sep 15
Masked prediction and full-stack pretraining
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
07Thu Sep 17
What do these representations actually encode?
- SUPERB: Speech Processing Universal PERformance Benchmark
Fri Sep 18
Project proposal due — 2 pages.
08Tue Sep 22
Prompting and parameter-efficient adaptation
- The Power of Scale for Parameter-Efficient Prompt Tuning
- Contrastive Learning for Speech Emotion Domain Generalization via Soft Prompt Tuning
Unit III
Audio-language models and emotion
General hearing abilities bolted onto language models — and what it takes to make them hear affect.
09Thu Sep 24
Audio-language models I
- Pengi: An Audio Language Model for Audio Tasks
- SALMONN: Towards Generic Hearing Abilities for Large Language Models
10Tue Sep 29
Audio-language models II
- Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
- Listen, Think, and Understand
11Thu Oct 1
Making audio LLMs emotion-aware
- Emotion-Aware Audio Large Language Models with Dual Cross-Attention and Context-Aware Instruction Tuning
12Tue Oct 6
Structured prompting and low-resource affect
- Plug-and-Play Emotion Graphs for Compositional Prompting in Zero-Shot Speech Emotion Recognition
- Role-Guided Annotation and Prototype-Aligned Representation Learning for Historical Literature Sentiment Classification
Thu Oct 8
No class — Fall Break, October 8–11.
13Tue Oct 13
Adapting at test time
- Tent: Fully Test-Time Adaptation by Entropy Minimization
- EMO-TTA: Improving Test-Time Adaptation of Audio-Language Models for Speech Emotion Recognition
Unit IV
Generating emotional speech
Codecs, zero-shot synthesis, style control, and aligning generation with human preference.
14Thu Oct 15
Neural audio codecs
- SoundStream: An End-to-End Neural Audio Codec
- High Fidelity Neural Audio Compression (EnCodec)
15Tue Oct 20
Codecs that preserve affect
- AffectCodec: Emotion-Preserving Neural Speech Codec for Expressive Speech Modeling
- AudioLM: A Language Modeling Approach to Audio Generation
16Thu Oct 22
Zero-shot text-to-speech
- Neural Codec Language Models Are Zero-Shot Text to Speech Synthesizers (VALL-E)
- NaturalSpeech 2: Latent Diffusion Models Are Natural and Zero-Shot Speech and Singing Synthesizers
17Tue Oct 27
Style and expressivity
- StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models
Mon Oct 26
Last day to withdraw.
18Thu Oct 29
Aligning generative models with preferences
- Direct Preference Optimization: Your Language Model Is Secretly a Reward Model
- Diffusion Model Alignment Using Direct Preference Optimization
Fri Oct 30
Midterm progress report due — 4 pages.
Tue Nov 3
No class — Election Day.
19Thu Nov 5
Preference optimization for emotional TTS
- Emo-BPO: Emotion Bidirectional Preference Optimization for Diffusion-based Emotional TTS
- Emotion-Aligned Generation in Diffusion Text-to-Speech Models via Preference-Guided Optimization
Unit V
Interpretability and robustness
What the features mean, and what happens to them outside the training distribution.
20Tue Nov 10
Opening the black box: interpretable emotion control
- Sparse Autoencoders Find Highly Interpretable Features in Language Models
- Sparse Autoencoders for Interpretable Emotion Control in Text-to-Speech
21Thu Nov 12
Domain adaptation
- Domain-Adversarial Training of Neural Networks
- Adversarial Discriminative Domain Adaptation
22Tue Nov 17
Distribution shift in deployed affect systems
- A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
- E-ADDA: Unsupervised Adversarial Domain Adaptation Enhanced by a New Mahalanobis Distance Loss
23Thu Nov 19
Affect sensing in the wild: health deployments
- Emotion Recognition Robust to Indoor Environmental Distortions Using Out-of-Distribution Detection
- Gait-Guard: Turn-aware Freezing of Gait Detection for Non-intrusive Intervention Systems
24Tue Nov 24
Beyond speech: multimodal and physiological affect Online
November 23–24 are university-designated remote instruction days. This meeting is held online.
- Multimodal Transformer for Unaligned Multimodal Language Sequences
- Hierarchical Convolution Multibranch Transformer for EEG Signals
Thu Nov 26
No class — Thanksgiving Break, November 25–29.
Unit VI
Project presentations
No assigned reading. The papers this week are yours.
25Tue Dec 1
Final project presentations, session I
26Thu Dec 3
Final project presentations, session II
Fri Dec 11
Final report due — 8 pages, conference format. No final exam.
December 4 is the last day of classes. The university exam period runs December 7–11 and 14–15.
Supplementary reading
Optional. Useful for project work and for filling gaps in background.
- Tensor Fusion Network for Multimodal Sentiment Analysis
- Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
- Enhancing the Reliability of Out-of-Distribution Image Detection in Neural Networks (ODIN)
- Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities
- MiddleGAN: Generating Inter-domain Images with GANs
- Confidence-Aware Ranker Ensembles for Robust In-Context Knowledge Editing
- ElectroMeter: The Practical Electrolyte Measurement System
- Challenges to Recruiting Dementia Caregiving Dyads in Community-Based Settings