Курс будет проходить по вторникам с 19.00 по мск.
Ссылка для подключения
Онлайн-трансляция на ютьюб-канале
(здесь же можно будет посмотреть прошедшие выступления).
This part aims to motivate stochastic optimization problems by statistical learning theory, where the goal is to minimize population risk, possibly based on the minimization of empirical risk.
1.1 Motivation by statistics: stochastic optimization viewpoint, Fisher's theorem, maximum likelihood estimation.
1.2 Motivation by machine learning: stochastic optimization viewpoint, elements of statistical learning theory, empirical risk minimization, regularization.
1.3 Basics of stochastic optimization: online (Stochastic Approximation) and offline (Sample Average Approximation/Monte-Carlo) approaches, setup of parallel and distributed computations.
This part introduces the main algorithmic schemes for stochastic convex optimization with non-smooth objective functions.
2.1 Stochastic SubGradient Descent (SSGD). Convergence rate, lower bounds for convergence rate in non-smooth setting, optimality of SSGD.
2.2 Stochastic Mirror Descent. Convergence rate, flexibility w.r.t. the geometry of the feasible set, lower bounds for convergence rate in non-smooth setting, optimality of MD.
2.3 Stochastic SubGradient Descent and Mirror Descent in Online optimization, connection to stochastic optimization, the flexibility of SSGD and MD.
2.4 Convergence rates with high probability. Stochastic subgradients with heavy-tailed distribution, subgradient clipping technique, convergence rates with high probability.
3.1 Stochastic gradient descent: convergence rate analysis, convergence to the vicinity of the solution with constant step size, convergence rate under interpolation condition, incorporating mini-batch stochastic gradients.
3.2 Catalyst - a universal framework for acceleration of randomized optimization methods. Application to stochastic optimization algorithms.
3.3 Direct acceleration of SGD under weak and strong growth conditions, convergence rate analysis. Over-parameterized learning models and analysis of accelerated methods in this setting.
3.4 Batch-parallel computations and Federated Learning. Lower bounds and optimal algorithms.
3.5 Connection between deterministic methods with inexact oracle for smooth problems and algorithms for stochastic optimization: mini-batch stochastic gradient as an inexact oracle, convergence rate analysis.
3.6 Convergence rates with high probability. Stochastic gradients with heavy-tailed distribution, gradient clipping technique, convergence rates with high probability.
4.1 Motivation for convex-concave stochastic saddle-point problems and monotone stochastic variational inequalities: Equilibrium problems, game theory, robust optimization, robust machine learning problems, e.g. training Generative Adversarial Networks (GAN)
4.2 Variational inequalities and saddle-point problems with non-smooth operators or objectives. Stochastic Mirror Descent and its convergence rate.
4.3 Variational inequalities and saddle-point problems with smooth operators or objectives. Stochastic Mirror Prox and its convergence rate. Stochastic Mirror Prox with batch parallelization, applications to federated learning and distributed computations setting.
5.1 AdaGrad-type algorithms for non-smooth optimization problems. Convergence rate analysis.
5.2 Adam-type methods for smooth problems and non-smooth optimization problems.
5.3 Universal stochastic Mirror Prox for smooth and non-smooth variational inequalities. Convergence rate analysis
6.1 Random gradient-free methods for non-smooth zeroth-order optimization: one-point and two-point feedback, convergence rate analysis.
6.2 Random gradient-free methods for smooth zeroth-order optimization: one-point and two-point feedback. Random (block) coordinate descent methods for smooth optimization problems. A unifying framework for accelerated randomized methods.
6.3 (Accelerated) variance reduced methods for finite-sum convex optimization and saddle-point problems.
6.4 Centralized distributed methods for finite-sum problems that use the statistical similarity of the data.
Pavel Dvurechensky is a research associate at Weierstrass Institute for Applied Analysis and Stochastics in Berlin. He received a Ph.D. in Mathematics in 2014 from the Faculty of Control and Applied Mathematics of Moscow Institute of Physics and Technology and received a B.Sc and M.Sc in Mathematics in 2008 and 2010 from the same university. In 2021 he received (jointly with Alexander Gasnikov) the Award for Young Scientists from the Moscow Government. His main area of research is optimization algorithms. He regularly publishes in top-tier machine learning conferences such as NeurIPS, ICML and has papers in Q1 journals such as EJOR, OMS, and JOTA.
Alexander Gasnikov is a professor at Moscow Institute of Physics and Technology. He received a Doctor degree (Habilitation) in Mathematics in 2016 from the Faculty of Control and Applied Mathematics of Moscow Institute of Physics and Technology and received a B.Sc, M.Sc, Ph.D. in Mathematics in 2004, 2006, and 2007 from the same university. In 2019 he received Yahoo Faculty Research Engagement Program (jointly with Cesar Uribe). In 2020 he received the Yandex Award (Ilya Segalovich Award). In 2021 he received the Award for Young Scientists from the Moscow Government (jointly with Pavel Dvurechensky). His main area of research is optimization algorithms. He regularly publishes in top-tier machine learning conferences such as NeurIPS, ICML and has papers in Q1 journals such as EJOR, OMS, and JOTA.
Eduard Gorbunov is Ph.D. in Computer Science at Moscow Institute of Physics and Technology. He received a B.Sc, a M.Sc and PhD in Applied Mathematics and Physics from Moscow Institute of Physics and Technology. His research interests include stochastic optimization, distributed optimization, federated learning, and derivative-free optimization. Eduard Gorbunov received several awards and scholarships for his research including The Ilya Segalovich Award 2019, Andrei Raigorodskii personal scholarship (main prize), Huawei scholarship for bachelor and master students at MIPT. He regularly publishes in top-tier machine learning conferences such as NeurIPS, ICML, AISTATS, ICLR, receives awards as the best reviewer there, and has papers in Q1 journals such as SIOPT, EJOR.
Dmitry Kovalev is a Ph.D. student at the King Abdullah University of Science and Technology. His research interests include stochastic optimization, distributed optimization, federated learning, and saddle-points problems. Dmitry received several awards and scholarships for his research including The Ilya Segalovich Award 2020, 2021, and best student paper award at ICML 2021 workshop. He regularly publishes in top-tier machine learning conferences such as NeurIPS, ICML, AISTATS, ICLR, and has papers in Q1 journals.