Prof. Sergios Theodoridis
A short introduction to Machine Learning and Deep Learning Networks
(Slides)
Abstract
In this short course, a brief tour to Machine Learning (ML) “land” will be attempted. We will begin from the “beginning”. Two of the major ML pillars will be introduced, namely the regression, in its simplest Least Squares formulation, and the classification tasks. The notion of model overfitting will be outlined, which will bring into the scene Occam’s razor rule and the need for parameter regularization. The Bayes classifier and the logistic regression will offer us their own optimal flavor in grasping the basics. We will move on to present two classical parameter estimation methods, namely the Maximum Likelihood (ML) and Maximum A-Posteriori (MAP) techniques. The basic “secrets” behind the powerful probabilistic view of parameter learning via the Bayesian palette of tools will be briefly established. In the sequel, the tour will follow more recent advances. Neural networks will be “visited”, starting from their late 19th century “spring”, with the discovery of the neuron, and then we will “stop” at the major milestones. The artificial neuron, the perceptron and the multilayer feedforward NN will be the very first to “look” at. Backpropagation and some related optimization algorithms will be discussed. Nonlinear activation functions will be presented, in the context of their effect on the training algorithm convergence. In the sequel, techniques guarding against overfitting will be outlined, such as the dropout approach. The final path will evolve along the most modern advances in the terrain. Convolutional networks (CNN) and recurrent neural networks (RNN) will be “visited” and discussed. If time allows, some “bridges” will be established that bring together deep networks and the Bayesian spirit.
The Classics
- Parametric modelling: Regression and Classification
- Bias Variance Tradeoff and Overfitting
- Ridge Regression
- Bayes Classifiers, Minimum Distance Classifiers and Logistic Regression
- Maximum Likelihood, Maximum a-Posteriori Estimation
- Bayesian Learning: Basic Principles and the EM algorithm
Neural Networks and Deep Learning
- Perceptron and the Perceptron rule
- Feedforward Neural Networks
- The Backpropagation and the Gradient Diminishing/Expanding Effect
- Sigmoid and ReLU Activation Units
- Pruning a network and the Dropout method
- The need for Deep Networks
- Convolutional Networks
- High-way and Residual Networks
- Recurrent Neural Networks and the LSTM
- Building Attention Mechanism in a Network
- Adversarial Examples
- NN with a Bayesian flavor (if time allows)
- Open Issues and Future Challenges.
Biographical notes
Sergios Theodoridis is currently Professor of Signal Processing and Machine Learning in the Department of Informatics and Telecommunications of the National and Kapodistrian University of Athens and he is the holder of a part time Chair at the Chinese University of Hong Kong. His research interests lie in the areas of Adaptive Algorithms, Distributed and Sparsity-Aware Learning, Machine Learning and Pattern Recognition, Signal Processing and Learning for Bio-Medical Applications and Audio Processing and Retrieval.
He is the author of the book “Machine Learning: A Bayesian and Optimization Perspective” Academic Press, 2015, the co-author of the best-selling book “Pattern Recognition”, Academic Press, 4th ed. 2009, the co-author of the book “Introduction to Pattern Recognition: A MATLAB Approach”, Academic Press, 2010, the co-editor of the book “Efficient Algorithms for Signal Processing and System Identification”, Prentice Hall 1993, and the co-author of three books in Greek, two of them for the Greek Open University.
He is the co-author of seven papers that have received Best Paper Awards including the 2014 IEEE Signal Processing Magazine Best Paper award and the 2009 IEEE Computational Intelligence Society Transactions on Neural Networks Outstanding Paper Award.
He is the recipient of the 2017 EURASIP Athanasios Papoulis Award, the 2014 IEEE Signal Processing Society Education Award and the 2014 EURASIP Meritorious Service Award. He has served as a Distinguished Lecturer for the IEEE Signal Processing as well as the Circuits and Systems Societies. He was Otto Monstead Guest Professor, Technical University of Denmark, 2012, and holder of the Excellence Chair, Dept. of Signal Processing and Communications, University Carlos III, Madrid, Spain, 2011.
He has served as Editor-in-Chief for the IEEE Transactions on Signal Processing. He is Editor-in-Chief for the Signal Processing Book Series, Academic Press and co-Editor in Chief for the EReference Signal Processing, Elsevier.
He has served as President of the European Association for Signal Processing (EURASIP), as a member of the Board of Governors for the IEEE Circuits and Systems (CAS) Society, as a member of the Board of Governors (Member-at-Large) of the IEEE SP Society and as a Chair of the Signal Processing Theory and Methods (SPTM) technical committee of IEEE SPS. He currently serves as Vice President IEEE Signal Processing Society.
He is Fellow of IET, a Corresponding Fellow of the Royal Society of Edinburgh (RSE), a Fellow of EURASIP and a Fellow of IEEE.
Prof. Alberto Isidori
Nonlinear Control
(slides)
Abstract
These lectures will review a few basic methods for the design of feedback laws to the purpose of achieving asymptotic stability of an equilibrium in a system modeled by nonlinear differential equations. In particular, the problem of achieving asymptotic with a prescribed region of attraction, via static state-feedback and/or dynamic output feedback, will be discussed.
Syllabus
Some familiarity with a few fundamental concepts in linear system theory (controllability, observability, pole assignment, zeros) is need. Basic concepts equilibrium and of asymptotic stability are assumed to be know. The lectures begin with a short review of the method of Lyapunov for the analysis of stability. The concept of input-to-state stability is also reviewed. Then, it will be shown how the equations describing a nonlinear system, affine in the control, can be reduced to a special form know as “normal form”. Such form is instrumental for the design, under appropriate hypotheses, of static state and/or output feedback law yielding stability with a prescribed region of attraction. Examples and counterexamples will be presented to illustrate the potentials and the limitations of the methods. Then, it will be shown how, if the equations describing a system are reduced to a special form known as “uniform observability for”, a nonlinear observer can be designed. Finally, the problem of stabilization via dynamic output feedback will be discussed and a nonlinear version of the so-called “separation principle” will be presented. In the hours devoted to exercise, the students will be assisted in the solution of some elementary problems.
Suggested reading
Chapters 6 and 7 of the book: A.Isidori, Lectures in Feedbcak Design for Multivariable Systems, Springer Verlag, 2017.
Biographical notes
From 1975 to 2012, Alberto Isidori has been Professor of Automatic Control at this University, where is currently Professor Emeritus. His research interests are primarily in analysis and design of nonlinear control systems. He is author of several books, among which the highly-cited Nonlinear Control Systems (1985, 1989 and 1995) and the more recent Lectures on Feedback Design of Multivariable Systems (2017). He published more than 110 articles on archival journals and more than 200 conference papers. In 1986 he was elected Fellow of IEEE and in 2005 he was elected Fellow of IFAC. He is the recipient of various prestigious awards, which include the Quazza Medal from IFAC (in 1996), the Ktesibios Award (in 2000), the Bode Lecture Award from the Control Systems Society of IEEE (in 2001), the Honorary Doctorate from KTH of Sweden (in 2009), the Galileo Galilei Award from the Rotary Clubs of Italy (in 2009), the Control Systems Award of IEEE in 2012. In 2012 he was also elected corresponding member of the italian Accademia Nazionale dei Lincei. He has been President of the European Community Control Association in 1995-1997 and President of IFAC in the triennium 2008-2011.