1.2 Timeline of information theory

1.3 Information, Message and Signal

2.1 Self-information &conditional self-information

2.2 Jensen's inequality

2.3 Entropy

2.4 Joint entropy & conditional entropy

2.5 Relative entropy & Mutual information

2.6 Relationship between entropy & Mutual information

2.7 Chain rules for entropy, relative entropy &mutual information

2.8 Data processing inequality

3.1 the Asymptotic Equipartition Property

3.2 Consequences of the AEP: Data compression

3.3 High probability sets and the typical set

4.1 Markov chains

4.2 Entropy Rate

4.3 Example

4.4 Hidden Markov models

5.1 Examples of codes

5.2 Kraft inequality

5.3 Optimal codes

5.4 Bounds on the optimal code length

5.5 Huffman codes

5.6 Optimality of Huffman codes

5.7 Shannon codes

5.8 Shannon-Fano-Elias coding

6.1 History of Communication Systems

6.2 Communication Systems

6.3 Channel Capacity

6.4 Properties of Channel Capacity

6.5 Communication System-Revisited

6.6 Definitions

6.7 Channel Coding Theorem

6.8 Demonstration of Channel Coding

6.9 Examples

7.1 Definitions

7.2 The AEP for continuous random variables

7.3 Relation of differential entropy to discrete entropy

7.4 Joint and conditional differential entropy

7.5 Relative entropy and mutual information

7.6 Properties of differential entropy, relative entropy and mutual information

8.1 The Gaussian Channel: definitions

8.2 Converse to the Coding Theorem for Gaussian Channels

8.3 Band-limited Channels

8.4 Parallel Gaussian Channels

8.5 Channels with Colored Gaussian Noise

8.6 Gaussian Channels with Feedback

9.1 Maximum Entropy Distributions

9.2 Examples

9.3 Anomalous Maximum Entropy

9.4 Spectrum Estimation

9.5 Entropy Rates of A Gaussian Process

9.6 Burg's Maximum Entropy Theorem

10.1 Distortion Definitions

10.2 Rate distortion function

10.3 Rate distortion theorem and the convers

10.4 Calculation of the rate distortion function

11.1 Gaussian Multiple User Channel

11.2 Jointly Typical Sequences

11.3 The Multiple Access Channel

11.4 The Broadcast Channel

11.5 The Relay Channel