Course Description:
This is a graduate level introduction to information theory, with an emphasis on the ideas and methods that are most useful in language technologies and machine learning. The course will be organized as a seminar, with the participants presenting selected readings and solutions to problems. The seminar will first cover the basic background material, leading to a proof of the channel coding theorem and a discussion of the implications and assumptions behind the joint source-channel coding theorem. Recent developments in codes and graphical models will then be covered, together with connections to supervised and unsupervised learning based on graphical models such as random fields, Bayesian belief networks, and factor graphs. Additional topics will include information geometry, Bregman divergence, and alternating minimization algorithms.

Textbooks:

• Main: Cover and Thomas
• Supplementary: Gallager