Department of Electrical and Computer Engineering
University of Toronto
ECE1502F:
Information Theory
Fall
2014
|
Instructor: |
Lectures: |
|
Professor
Wei Yu |
Tuesday 10-11:30am Bahen
4164 |
|
Email:
weiyu@comm.utoronto.ca |
Thursday
10-11:30am Bahen 4164 |
|
Office:
Bahen 4114 |
First Class: September 4, 2014 |
Textbook:
Elements of
Information Theory
Authors: Thomas M. Cover and Joy A. Thomas,
John Wiley, 2nd Edition, 2006.
Information
theory answers two fundamental questions in communication theory: what is the
ultimate data compression (answer: the entropy H), and what is the ultimate
transmission rate of communication (answer: the channel capacity C) -
Cover & Thomas
This
course is a one-term introduction to information theory at a first-year
graduate level. The aim is to provide a comprehensive coverage of the following
major areas of information theory:
This is
a fundamental course for students interested in digital communications, data
compression and signal processing. It is also of interests to students in
Computer Science, Statistics and related disciplines.
Pre-requisite: An undergraduate course in Probability. A fair level
of mathematical maturity is expected.
A probability refresher is available here. Courtesy of Prof. Frank Kschichang.
Course Outline:
|
Lectures |
Topics |
Text
Reference |
Homework |
|
9/4 |
Introduction.
Entropy. |
Ch.
1.1, 2.1 |
-- |
|
9/9,
9/11 |
Entropy,
Joint Entropy, Relative Entropy, Mutual Information. Jensen's inequality.
Conditional Entropy. |
Ch. 2.2-2.6, 2.8 |
#1
due 9/8 |
|
9/16,
9/18 |
Conditional
Mutual Information, Data Processing Inequality. Entropy Rate of a Stochastic
Process. |
Ch.
4.1-4.2 |
-- |
|
9/23,
9/25 |
Asymptotic
Equipartition Property (AEP) |
Ch. 3 |
#2
due 10/2 |
|
9/30,
10/2 |
Data
Compression, Kraft’s Inequality, Shannon Code, Huffman Code |
Ch.
5.1-5.9 |
-- |
|
10/7,
10/9 |
Arithmetic
Code, Lempel-Ziv Code. Gambling on Horse Races. |
Ch.
6.1-6.3 |
-- |
|
10/16 |
Midterm
on 10/16; (TA office hour during class time on 10/14.) |
|
#3
due 10/14 |
|
10/21,
10/23 |
Channel
Capacity Theorem. |
Ch.
7.1-7.5 |
-- |
|
10/28,
10/30 |
Joint
Typicality. Achievability Proof. |
Ch.
7.6-7.7 |
-- |
|
11/4,
11/6 |
Converse
of the Channel Capacity Theorem. Fano's Inequality.
Channel with Feedback. Source Channel Separation |
Ch.
7.9, 7.13 |
-- |
|
11/11,
11/13 |
Differential
Entropy. AEP. Gaussian Channel |
Ch.
8.1-8.6 |
#4
due 11/11 |
|
11/18,
11/20 |
Maximum
Entropy Distribution. Discrete-Time Gaussian Channel |
Ch.
12.1-12.2 Ch.
9.1-9.2 |
-- |
|
11/25,
11/27 |
Gaussian
Vector Channels. Water-filling. Band-limited Gaussian Channel. Complex
Gaussian Channels. |
Ch.
9.3-9.5 |
-- |
|
12/2 |
Gaussian
Multiple-Access Channel. |
Ch.
15.1 |
#5
due 12/2 |
|
-- |
-- |
-- |
-- |
|
12/16 |
Final
Exam. Dec 16, 2pm-4:30pm. BA3012 |
-- |
-- |
Grades: Midterm (30%), Final (70%). Both exams are open-book open-notes exams.
References:
Last Updated: 2014-12-03