# Lecture Notes

This section contains lecture notes and some associated readings.

Complete lecture notes (PDF - 7.7MB)

Lecture notes files.
TOPICS | LECTURE NOTES | READINGS
--- | --- | ---
The role of convexity in optimization Duality theory Algorithms and duality | Lecture 1 (PDF - 1.2MB) |
Convex sets and functions Epigraphs Closed convex functions Recognizing convex functions | Lecture 2 (PDF) | Section 1.1
Differentiable convex functions Convex and affine hulls Caratheodory's theorem | Lecture 3 (PDF) | Sections 1.1, 1.2
Relative interior and closure Algebra of relative interiors and closures Continuity of convex functions Closures of functions | Lecture 4 (PDF) | Section 1.3
Recession cones and lineality space Directions of recession of convex functions Local and global minima Existence of optimal solutions | Lecture 5 (PDF - 1.0MB) | Sections 1.4, 3.1, 3.2
Nonemptiness of closed set intersections Existence of optimal solutions Preservation of closure under linear transformation Hyperplanes | Lecture 6 (PDF - 1.4MB) |
Review of hyperplane separation Nonvertical hyperplanes Convex conjugate functions Conjugacy theorem Examples | Lecture 7 (PDF) | Sections 1.5, 1.6
Review of conjugate convex functions Min common / max crossing duality Weak duality Special cases | Lecture 8 (PDF - 1.2MB) | Sections 1.6, 4.1, 4.2
Minimax problems and zero-sum games Min common / max crossing duality for minimax and zero-sum games Min common / max crossing duality theorems Strong duality conditions Existence of dual optimal solutions | Lecture 9 (PDF) | Sections 3.4, 4.3, 4.4, 5.1
Min common / max crossing Theorem III Nonlinear Farkas' lemma / linear constraints Linear programming duality Convex programming duality Optimality conditions | Lecture 10 (PDF) | Sections 4.5, 5.1, 5.2, 5.3.1–5.3.2
Review of convex programming duality / counterexamples Fenchel duality Conic duality | Lecture 11 (PDF) | Sections 5.3.1–5.3.6
Subgradients Fenchel inequality Sensitivity in constrained optimization Subdifferential calculus Optimality conditions | Lecture 12 (PDF) | Section 5.4
Problem structure Conic programming | Lecture 13 (PDF) |
Conic programming Semidefinite programming Exact penalty functions Descent methods for convex optimization Steepest descent method | Lecture 14 (PDF) | Chapter 6: Convex Optimization Algorithms (PDF)
Subgradient methods Calculation of subgradients Convergence | Lecture 15 (PDF) |
Approximate subgradient methods Approximation methods Cutting plane methods | Lecture 16 (PDF) |
Review of cutting plane method Simplicial decomposition Duality between cutting plane and simplicial decomposition | Lecture 17 (PDF) |
Generalized polyhedral approximation methods Combined cutting plane and simplicial decomposition methods | Lecture 18 (PDF) | Bertsekas, Dimitri, and Huizhen Yu. "A Unifying Polyhedral Approximation Framework for Convex Optimization." SIAM Journal on Optimization 21, no. 1 (2011): 333–60.
Proximal minimization algorithm Extensions | Lecture 19 (PDF) |
Proximal methods Review of proximal minimization Proximal cutting plane algorithm Bundle methods Augmented Lagrangian methods Dual proximal minimization algorithm | Lecture 20 (PDF - 1.1MB) |
Generalized forms of the proximal point algorithm Interior point methods Constrained optimization case: barrier method Conic programming cases | Lecture 21 (PDF) |
Incremental methods Review of large sum problems Review of incremental gradient and subgradient methods Combined incremental subgradient and proximal methods Convergence analysis Cyclic and randomized component selection | Lecture 22 (PDF) | Bertsekas, Dimitri. "Incremental Gradient, Subgradient, and Proximal Methods for Convex Optimization: A Survey." (PDF) Laboratory for Information and Decision Systems Report LIDS-P-2848, MIT, August 2010. Sra, Suvrit, Sebastian Nowozin, and Stephen Wright, eds. Optimization for Machine Learning. MIT Press, 2011. ISBN: 9780262016469.
Review of subgradient methods Application to differentiable problems: gradient projection Iteration complexity issues Complexity of gradient projection Projection method with extrapolation Optimal algorithms | Lecture 23 (PDF) |
Gradient proximal minimization method Nonquadratic proximal algorithms Entropy minimization algorithm Exponential augmented Lagrangian method Entropic descent algorithm | Lecture 24 (PDF) | Beck, Amir, and Marc Teboulle. "Gradient-Based Algorithms with Applications to Signal-Recovery Problems." In Convex Optimization in Signal Processing and Communications. Edited by Daniel Palomar and Yonina Eldar. Cambridge University Press, 2010. ISBN: 9780521762229. Beck, Amir, and Marc Teboulle. "Mirror Descent and Nonlinear Projected Subgradient Methods for Convex Optimization." Operations Research Letters 31, no. 3 (2003): 167–75. Bertsekas, Dimitri. Nonlinear Programming. Athena Scientific, 1999. ISBN: 9781886529007.
Convex analysis and duality Convex optimization algorithms | Lecture 25 (PDF - 2.0MB) |