.. nunggu-web documentation master file, created by
   sphinx-quickstart on Sat Apr  2 21:28:31 2011.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

Optimization in engineering (EE598)
=====================================
	- I created handouts for personal notes and teaching in related courses.
	- The handouts are still under revision.
	- The following contents are part of EE598 (special problem in EE) in 2022.

Lecture notes
---------------
The contents are summarized from the reference textbooks and partly from class notes of Prof. Lieven Vandenberghe. 

Check out the YouTube playlist: `Optimization in engineering and machine learning <https://www.youtube.com/playlist?list=PL5OWEh_D2xktQsIUtV7WpzZULOlen43Uc>`_

0. `General optimization setting <./optim/overview.pdf>`_
	- standard formulation
	- overview of problem types
1. `Unconstrained optimization <./optim/unconstrained.pdf>`_
	- Gradient-descent, Newton, Quasi Newton, Conjugate gradient
	- Accelerated gradient methods for convex problem
	- Momentum-accelerated gradient descent
	- Mini-batch optimization
2. `Constrained optimization <./optim/constraint.pdf>`_
	- Lagrange multiplier theorem
	- contraint elimination
	- convex constraints
3. Linear programming: formulation and algorithms
4. Quadratic programming: formulation and algorithms
5. `Optimization problems in applications <./optim/optim_app.pdf>`_ (more list will be added)
	- portfolio optimization
	- traffic network
	- regression, logistic regression
	- SVM, Neural network
6. `Regularization techniques <./optim/reg.pdf>`_
	- l1 and l2 regularized regression
	- generalized l1 regularization (variants of lasso)
7. Duality theory: dual problem, KKT conditions, examples 
8. Approximation methods
	- proximal algorithms
	- ADMM
	- augmented Lagrangian
	- coordinate descents

References: texbooks, class notes 
-------------------------------------

**Nonlinear Optimization**

	1. D.P. Bertsekas, *Nonlinear Programming*, 2nd edition, Athena Scientific, 1999
	#. J. Nocedal and S.J. Wright, *Numerical Optimization*, 2nd edition, Springer, 2006
	#. D.G. Luenberger and Y. Ye, *Linear and Nonlinear Programming*, 4th edition, Springer, 2008
	#. I. Griver, S.G. Nash, and A. Sofer, *Linear and Nonlinear Optimization*, 2nd edition, SIAM, 2009
 
**Convex Optimization**
	1. S. Boyd and L. Vandenberghe, *Convex Optimization*, Cambridge University Press, 2004
	#. G. Calafiore and L. El Ghaoui, *Optimization Models*, Cambridge University Press, 2014 
	#. D.P. Bertsekas, *Convex Optimization Algorithms*, Athena Scientific, 2015
	#. D.P. Bertsekas, *Convex Optimization Theory*, Athena Scientific, 2009
	#. Y. Nesterov, *Introductory Lectures on Convex Optimization: A Basic Course*, Kluwer Academic Publishers, 2004
	#. D. Bertsimas and J.N. Tsisiklis, *Introduction to Linear Optimization*, Athena Scientific, 1997


**Optimization in Machine Learning**
	1. J. Watt, R. Borhani, and A.K. Katsaggelos, *Machine Learning Refined: Foundations, Algorithms, and Applications*, 2nd edition, Cambridge University Press, 2020
	#. C.C. Aggarwal, *Linear Algebra and Optimization for Machine Learning: A Textbook*, Springer, 2020
	#. D.Bertsimas and J. Dunn, *Machine Learning under a Modern Optimization Lens*, Dynamic Ideas LLC, 2019
	#. S. Boyd, N. Parikh, E. Chu, B. Peleato and J. Eckstein, *Distributed Optimization and Statistical Learnign via the Alternating Direction Method of Multipliers*, Foundations and Trends in Machine Learning, 2011
	#. N. Parikh and S. Boyd, *Proximal Algorithms*, Foundations and Trends in Optimization, 2013

**Linear algebra with applications**
	1. S. Boyd and L. Vandenberghe, *Introduction to Applied Linear Algebra: Vectors, Matrices, and Least squares*, Cambridge, 2018
	#. G. Strang, *Linear Algebra and Learning from Data*, Wellesley-Cambridge Press, 2019

**Statistical learning and ML**
	1. T. Hastie and R. Tibshirani, and J. Friedman, *The Elements of Statistical Learning: Data Mining, Inference, and Prediction*, 2nd edition, Springer, 2009
	#. T. Hastie, R. Tibshirani, and M. Wainwright, *Statistical Learning with Sparsity : The Lasso and Generalizations*, CRC Press, 2015
	#. I. Goodfellow, Y. Bengio, and A. Courville, *Deep Learning*, The MIT Press, 2016

**Class notes**
	1. `Lieven Vandenberghe: <http://www.seas.ucla.edu/~vandenbe/>`_ EE236A, EE236B, EE236C
	2. `Stephen Boyd: <https://web.stanford.edu/~boyd/>`_ EE364a EE364b