Материал из MachineLearning.

Версия 12:37, 6 февраля 2023

Основная статья: Численные методы обучения по прецедентам (практика, В.В. Стрижов)

2021

Bayesian model selection and multimodeling

Course page: https://github.com/Intelligent-Systems-Phystech/BMM-21

The lecture course delivers the main problem of machine learning, the problem of model selection. One can set a heuristic model and optimise its parameters, or select a model from a class, or make a teacher model to transform its knowledge to a student model, or even make an ensemble from a models. Behind all these strategies there is a fundamental technique: the Bayesian inference. It assumes hypotheses about the measured data set, about the model parameters and even about the model structure. And it deduce the error function to optimise. This is called the Minimum Description Length principle. It selects simple, stable and precise models. This course joins the theory and the practical lab works of the model selection and multimodeling.

Grading

• Labs: 6 in total
• Forms: 1 in total
• Reports: 2 in total

The maximum score is 11, so the final score is MIN(10, score)

Syllabus

1. 8.09 Intro
2. 15.09 Distributions, expectation, likelihood
3. 22.09 Bayesian inference
4. 29.09 MDL, Minimum description length principle
5. 6.10 Probabilistic metric spaces
6. 13.10 Generative and discriminative models
7. 20.10 Data generation, VAE, GAN
8. 27.10 Probabilistic graphical models
9. 3.11 Variational inference
10. 10.11 Variational inference 2
11. 17.11 Hyperparameter optimization
12. 24.11 Meta-optimization
13. 1.12 Bayesian PCA, GLM and NN
14. 8.12 Gaussian processes

References

Books

1. Bishop
2. Barber
3. Murphy
4. Rasmussen and Williams, of course!
5. Taboga(to catch up)

Theses

1. Грабововй А.В. Диссертация.
2. Бахтеев О.Ю.. Выбор моделей глубокого обучения субоптимальной сложности git, автореферат, презентация (PDF), видео. 2020. МФТИ.
3. Адуенко А.А. Выбор мультимоделей в задачах классификации, презентация (PDF), видео. 2017. МФТИ.
4. Кузьмин А.А. | Построение иерархических тематических моделей коллекций коротких текстов, | презентация (PDF), видео. 2017. МФТИ.

Papers

1. Kuznetsov M.P., Tokmakova A.A., Strijov V.V. Analytic and stochastic methods of structure parameter estimation // Informatica, 2016, 27(3) : 607-624, PDF.
2. Bakhteev O.Y., Strijov V.V. Deep learning model selection of suboptimal complexity // Automation and Remote Control, 2018, 79(8) : 1474–1488, PDF.
3. Bakhteev O.Y., Strijov V.V. Comprehensive analysis of gradient-based hyperparameter optimization algorithmss // Annals of Operations Research, 2020 : 1-15, PDF.

Mathematical methods of forecasting

The lecture course and seminar introduces and applies methods of modern physics to the problems of machine learning.

Minimum topics to discuss: Geometric deep learning approach.

Optimum topics to discuss are: tensors, differential forms, Riemannian and differential geometry, metrics, differential operators in various spaces, embeddings, manifolds, bundles. We investigate scalar, vector and tensor fields (as well as jets, fibers and shiefs, tensor bundles, sheaf bundles etc.). The fields and spaces are one-dimensional, multidimensional and continuously dimensional.

Grading

• Questionnaires during lectures (4)
• Two application projects (2+2)
• The final exam: problems with discussion (1)
• Bonus for active participation (2)

Deadlines

• October 14th, (October 7th one-week before preliminary show is strongly advised)
• December 2nd, (November 25th one-week before preliminary show is strongly advised)

Lab work report and talk

1. Title and motivated abstract
2. Problem statement
3. Problem solution
4. Link to the code
5. Analysis and illustrative plots
6. References

The report template is here. Please follow the instructions in the template.

Themes

BCI, Matrix and tensor approximation

1. Коренев, Г.В. Тензорное исчисление, 2000, 240 с., lib.mipt.ru.
2.  Roger Penrose, "Applications of negative dimensional tensors," in Combinatorial Mathematics and its Applications, Academic Press (1971). See Vladimir Turaev, Quantum invariants of knots and 3-manifolds (1994), De Gruyter, p. 71 for a brief commentary PDF.
3. Tai-Danae Bradley, At the Interface of Algebra and Statistics, 2020, ArXiv.
4. Oseledets, I.V. Tensor-Train Decomposition //SIAM Journal on Scientific Computing, 2011, 33(5): 2295–2317, DOI, RG, lecture, GitHub, Tutoiral.
5. Wikipedia: SVD, Multilinear subspace learning, HOSVD.

BCI, Feature selection

1. Мотренко А.П. Выбор моделей прогнозирования мультикоррелирующих временных рядов (диссертация), 2019 PDF
2. Исаченко Р.В. Снижение размерности пространства в задачах декодирования сигналов (дисссертация), 2021 PDF

High order partial least squares

1. Qibin Zhao, et al. and A. Cichocki, Higher Order Partial Least Squares (HOPLS): A Generalized Multilinear Regression Method // IEEE Transactions on Pattern Analysis and Machine Intelligence, July 2013, pp. 1660-1673, vol. 35, DOI, ArXiv.

Neural ODEs and Continuous normalizing flows

1. Ricky T. Q. Chen et al., Neural Ordinary Differential Equations // NIPS, 2018, ArXiv, source paper and code
2. Johann Brehmera and Kyle Cranmera, Flows for simultaneous manifold learning and density estimation // NIPS, 2020, ArXiv
3. Flows at deepgenerativemodels.github.io
4. Flow-based deep generative models
5. Variational Inference with Normalizing Flows (source paper, Goes to BME)
6. Знакомство с Neural ODE на хабре, W: Flow-based generative model

Continous time representation

1. Самохина Алина, Непрерывное представление времени в задачах декодирования сигналов (магистерская диссертация): 2021 PDF, GitHub
2. Aaron R Voelker, Ivana Kajić, Chris Eliasmith, Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks // NIPS, 2019, PDF,PDF.
3. Functional data analysis: splines

Вопросы первого семестра

1. Проблемы прогнозирования и постановки задач прогнозирования
2. Авторегрессионные модели
3. Гусеница, выбор компонент
4. Модели прогнозирования временных рядов с высокой дисперсией ошибки
5. Учет ошибки при прогнозировании
6. Снижение размерности, SVD, PCA
7. Метод Сугихары ССМ
8. Метод PLD
9. Тензорные разложения
10. Метод HOPLS
11. LSTM и авторегрессионное модели
12. NeurODE

Navier-Stokes equations and viscous flow

1. Neural PDE
2. Newtonian and Non-Newtonian Fluids in Pipe Flows Using Neural Networks [1], [2]

Metric tensors and kernels

1. Lynn Houthuys and Johan A. K. Suykens, Tensor Learning in Multi-view Kernel PCA // ICANN 2018, pp 205-215, DOI.

fRMI, Riemannian geometry on shapes

1. Xavier Pennec, Stefan Sommer, and Tom Fletcher, Riemannian Geometric Statistics in Medical Image Analysis, 2019 book
2. Surface differential geometry Coursera code video for Image and Video Processing

Spatial time series alignment

1. Titouan Vayer et al., Time Series Alignment with Global Invariances, 2020,ArXiv
2. Marco Cuturi and Mathieu Blondel, Soft-DTW: a Differentiable Loss Function for Time-Series, ArXiv
3. Marcel Campen et al., Scale-Invariant Directional Alignment of Surface Parametrizations // Eurographics Symposium on Geometry Processing, 2016, 35(5), DOI
4. Helmut Pottmann et al. Geodesic Patterns // ACM Transactions on Graphics, 29(4), DOI, PDF

Reproducing kernel Hilbert space

1. Mauricio A. Alvarez, Lorenzo Rosasco, Neil D. Lawrence, Kernels for Vector-Valued Functions: a Review, 2012, ArXiv
2. Pedro Domingos, Every Model Learned by Gradient Descent Is Approximately a Kernel Machine, 2020, ArXiv
3. Wikipedia: RKHS
4. Aronszajn, Nachman (1950). "Theory of Reproducing Kernels". Transactions of the American Mathematical Society. 68 (3): 337–404. DOI.

Convolutions and Graphs

1. Gama, F. et al. Graphs, Convolutions, and Neural Networks: From Graph Filters to Graph Neural Networks // IEEE Signal Processing Magazine, 2020, 37(6), 128-138, DOI.
2. Zhou, J. et al. Graph neural networks: A review of methods and applications // AI Open, 2020, 1: 57-81, DOI, ArXiv.
3. Zonghan, W. et al. A Comprehensive Survey on Graph Neural Networks // IEEE Transactions on Neural Networks and Learning Systems, 2021, 32(1): 4-24, DOI, ArXiv.
4. Zhang, S. et al. Graph convolutional networks: a comprehensive review // Computational Social Networks, 2019, 6(11), DOI.
5. Xie, Y. et al. Self-Supervised Learning of Graph Neural Networks: A Unified Review // ArXiv.
6. Wikipedia: Laplacian matrix, Discrete Poisson's equation, Graph FT
7. GNN papers collection

Higher order Fourier transform

1. Zongyi Li et al., Fourier Neural Operator for Parametric Partial Differential Equations // ICLR, 2021, ArXiv
2. Fourier for fun and practice 1D Fourier Code
3. Fourier for fun and practice nD
   • Fourier analysis on Manifolds 5G page 49
   • Spectral analysis on meshes

Spherical Regression

1. Shuai Liao, Efstratios Gavves, Cees G. M. Snoek, Spherical Regression: Learning Viewpoints, Surface Normals and 3D Rotations on N-Spheres // CVPR, 2019, 9759-9767, ArXiv

Category theory

1. Tai-Danae Bradley, What is Applied Category Theory?, 2018, ArXiv, demo.
2. F. William Lawvere, Conceptual Mathematics: A First Introduction to Categories, 2011, PDF.
3. Картан А. Дифференциальное исчисление. Дифференциальные формы, 1971 lib.mipt.ru
4. Wikipedia: Homology, Topological data analysis

Geometric algebra

1. experior product and quaternions
2. Nick Lucid, Advanced Theoretical Physics, 2019, sample.

Lab works: Serie I

Report consists of

1. Problem motivation in details
2. Formal statement and description
3. Experiment description
4. Plot and its analysis with explanations

The style: use a toolbox to illustrate a practical example with a Figure and its analysis.

One condition is the variety of toolboxes.

Lab work 1

Tensor approximation. Approximate the 2, 3 and 4 index matrices using low-rank decompositions, linear and nonlinear. The data sets are: a picture, a short animation movie (basic variant), a sound spectrogram, an fMRI. Plot the sequence of data approximations with ranks 1,..,n. Plot the error: x-axis is the rank, y-axis is the approximation error. Plot the variance of the error for various samples of data, if possible.

• Code [Tucker, PARAFAC-CANDECOMP, Tensor-train, and variants]
• Data sets [Cartoon, Sound, fMRI, and variants]

Lab work 2

PCA on higher orders. Construct a linear map of pairwise distance matrix to a space of lower dimensionality and plot the data. There are two possibilities: high-rank dimensional reduction and order reduction. If you could demonstrate the order reduction, it would be great. The pictures are appreciated.

• Code [Tucker, Tensor-train, and variants]
• Data sets [Cartoon, Sound, fMRI, and variants]

Lab work 3

Three-dimensional image reconstruction. Use sinograms of the computed tomography scans. Show the sequential approximation like in the lab work 1 with plots.

• Code [linear and NN]
• Data [Sinogram to fMRI]

Lab work 4

Formalize the set of low-rank approximation quality criterions (for multilinear operators with pseudo-inverses): precision, stability, computational complexity. Compare the decomposition methods from hottbox.

• Code [Hottbox]
• Data [Synthetic data with a simple visual/geometric visualization, like TF logo]

Lab work 5

Construct a phase trajectory of a spatial time series (a video, cartoon) and make a forecast with the higher order singular structure analysis.

• Code [SSA, HOSVD]
• Data [Cartoon with a walking hero]

Lab work 6

Construct two phase trajectories of spatial time series and discover a casualty with the higher order convergent cross mapping.

• Code [CCM, HOSVD]
• Data [Cartoon with a walking hero]

Lab work 7

Reduction an index on the tensor representation of the data, along with dimensionality reduction. The data is synthetic video, in more than 4D space, visualised by 3D projections.

• Code [HOSVD, HOPCA]
• Data [Synthetic 4D+ video]

Lab work 8

Higher order spectra analysis versus (higher order) Fourier transform on six-dimensional gyrokinetic data. Reduce number of indexes in tensor data.

• Code [HOSVD]
• Data [Gyrokinetic data 10.1016/j.jcp.2012.02.007]

Lab work 9

Combine higher-order PLS with CCN to predict eye-movement.

• Code [HOPLS]
• Data [Systhesic video and eye-movement]

Lab work 10

Higher order PLS for multiway data

• Code [HOPLS]
• Data [Accelerometer under various conditions to predict gyro data]

Lab works, Serie II

As previously, the report consists of

1. Problem motivation in details
2. Formal statement and description
3. Experiment description
4. Plot and its analysis with explanations

The style: use a toolbox to illustrate a practical example with a Figure and its analysis.

The datasets for all lab works

The datasets for any lab work (if it is not specified) are 1) synthetic, are generated from the explicit solution of some differential equation, 2) from existing datasets, but not too long, 3) the self-made measurements are welcome!!! All of us have accelerometer, microphone, photo-video camera in the mobile phone.

References

• Neural Ordinary Differential Equations by Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, David K. Duvenaud // NeurIPS 2018 [3]
• Learning neural event functions for ordinary differential equations by Ricky T. Q. Chen, Brandon Amos, Maximilian Nickel // ICLR 2021, arXiv:2011.03902v4
• Interpolation Technique to Speed Up Gradients Propagation in Neural ODEs by Talgat Daulbaev, Alexandr Katrutsa, Larisa Markeeva, Julia Gusak, Andrzej Cichocki, Ivan Oseledets // 2020, arXiv:2003.05271v2
• Dynamical Systems with Applications using Python by Stephen Lynch, 2018
• How To Train Your Neural ODE ChrisFinlay et al. [4] or [5]

Code

• PyTorch Implementation of Differentiable ODE Solvers by Ricky Chen [6]
• A PyTorch library dedicated to neural differential equations and implicit models (the simplest implement, 100 lines). Maintained by DiffEqML [7]
• Neural Ordinary Differential Equations by Mikhail Surtsukov (Russian explanation)
• Meta-Solver for Neural Ordinary Differential Equations by Julia Gusak [8], [9]
• Neural ODE Processes code paper
• PyTorch Implementation of Differentiable ODE Solvers [github.com/rtqichen/torchdiffeq]
• Jupyter notebook with Pytorch implementation of Neural Ordinary Differential Equations [github.com/msurtsukov/neural-ode]
• DiffEqFlux.jl – A Julia Library for Neural Differential Equations Julia [10]

Problem 1

Plot the direction field and approximation by ODE-Net near the special points for simple ODEs like y’=y/x (uniform/узел), y’=-y/x saddle, y’=-x/y center, y’=(x+y)/(x-y) spiral.

Problem 2

Plot the stable and unstable phase portraits and approximations for simple ODE.

Problem 3

Find the ODE and plot the solution and ODE-Net approximation to compare and show difference between ODEsolvers: Euler, RK4 and others.

Problem 4

Approximate and plot the electricity consumption time series. Compare the ODE-Net and LSTM or another model you like.

Problem 5

Plot and approximate by ODE-Net the phase portrait of a pendulum with decay.

Problem 5+

Plot and approximate the phase portrait of coupled pendula.

Problem 6

Approximate an accelerometer (walking time series) by a solution of the pendulum ODE.

Problem 6+

Approximate an accelerometer by a solution of the double-pendulum ODE.

Problem 6++

Approximate an accelerometer and a gyroscope (acceleration and velocity) by a solution of the pendulum ODE.

Problem 7

Lorentz attractor, three ODEs. Compare the forecast LSTM (or any model you like) how it differs from the forecast ODE-Net under different initial conditions and under different noise conditions. An option is to immerse the trajectory in a space of a higher dimension by random rotation.

Problem 8

LSTM и ODE-RNN от от Алины Самохиной раздел 2.2.2, только вместо EEG для упрощения временные ряды потребления электроэнергии.

Problem 9

Compare generated and forecaster electricity electricity consumption consumption time series, see section 5 the arXiv:1806.07366v5 and the [ code]

Problem 10

Re-run the normalizing flows from the paper for various examples and analyse properties.

Lab works, Serie III

Various scales, algebraic structures of the design space to construct rating (rank-regression models). The results are published in [Link Colab]

Lab works, Serie IV

The model generation and model forecasting. The goal is to hold an experiment to forecast a structure of an optimal model, given dataset.

List of modules (Problems)

1. Class Model carries trivial methods
2. Select optimal model from the generated set
3. Check the model
4. Propose and compute model complexity and distance between model structures
5. Tune parameters of the model and keep them
6. Get and put subtrees to the structure
7. Generate the data and the model set
8. Recover structure with Prims
9. Recover structure with PCST
10. Plot the trees
11. Plot the distance maps
12. Collect and analyse the errors

Each module is responsible for

1. communication with neighbour models (input and output arguments)
2. acting on the whole system.
3. Each module runs the system!

The report consists of

1. Module description
2. Module test
3. Results of the module (highly depends on the nature of the module )

References

1. Kulunchakov A.S., Strijov V.V. Generation of simple structured Information Retrieval functions by genetic algorithm without stagnation // Expert Systems with Applications, 2017, 85 : 221-230. [11]
2. Бочкарев А.М., Софронов И.Л., Стрижов В.В. Порождение экспертно-интерпретируемых моделей для прогноза проницаемости горной породы // Системы и средства информатики, 2017, 27(3) : 74-87.[12]
3. Сологуб Р.А. Методы трансформации моделей в задачах нелинейной регрессии, презентация PDF, полная версия PDF. 2013. МФТИ.
4. Kuznetsov M., Strijov V. SVM Composer, Matlab code, 2012. [13]
5. Kuznetsov M., Strijov V. Notes on structure learning, 2012.[14]
6. Варфоломеева Анна Андреевна. Методы структурного обучения для построения прогностических моделей, презентация (PDF). 2013.Методы структурного обучения в задаче обобщения структур прогностических моделей, презентация (PDF). 2015.
7. Шибаев Иннокентий Андреевич Бакалаврская диссертация: Прогнозирование оптимальных суперпозиций в задачах регрессии (Презентация). Магистерская диссертация: Алгоритмы индуктивного порождения и упрощения и критерии выбора оптимальной существенно нелинейной регрессионной модели для аппроксимации измеряемых данных, презентация (PDF)
8. Кулунчаков Андрей Сергеевич Бакалаврская диссертация: Порождение структурно простых ранжирующих функций для задач информационного поиска, презентация (PDF)
9. Adam Gaier, David Ha. Weight Agnostic Neural Networks, ArXiv, 2019. [15]
10. Symbolic regression, wikipedia.
11. Wouter Minnebo and Sean Stijven. Empowering Knowledge Computing with Variable Selection not sure
12. Tyler William Hughes. Accelerating Symbolic Regression with Deep Learning, 2017. [16]
13. EJ Vladislavleva, GF Smits, D Den Hertog. Order of nonlinearity as a complexity measure for models generated by symbolic regression via pareto genetic programming // IEEE Transactions on Evolutionary Computation 13 (2), 333-349 [17]
14. Tommi Jaakkola. Scaling structured prediction, 2012. [18]
15. Слайды лекции "Прогнозирование прогностических моделей", часть "Символьная регрессия"

Lab works Serie V

The goal of the lab set is to join several, possible simple modules using joint efforts and information exchange. The following modules make the system work. PLS scheme

Lab 0

• Make the documented demo of the system. List to run all possible models. Make an example of the most interesting models (Linear, Autoencoders, Neur-ODE, Tensor, Graph).
• In: All labs, Lab 4,5
• Out: github.io page with examples

Lab 1

• Load data and put the data to PLS. Two sets of data: synthetic and real (X is time series times sensors, Y is time series times 3D coordinates)
• In: no
• Out: the formats for X, Y, U, V

Lab 2

• Make a set of the error functions. The reconstruction of (X, X), of (Y,Y), and of (Y, X)
• In: Lab 1
• Out: Four functions Qall, QXX, QYY, QXY

Lab 3

• Check the admissibility of any given module in the system. Run the system in all variants
• In: All Labs
• Out: it works in the system, yes/no

Lab 4

• Make the simplest system, linear transformations
• In: Lab 1, Lab 2
• Out: The shell with .predict, .fit stubs, which include interfaces to the models XX, YY, XY

Lab 5

• Select an optimisation algorithm from the set to optimise error functions
• In: Lab 2, Lab 3
• Out: The optimal parameters for the system and predictions

Lab 6

• Plot the predictions and error analysis
• In: Lab 5
• Out: Series of plots

Lab 7 Make YY Neur-ODE model

• In: Lab 5
• Out: Series of plots

Lab 8

• Make XX, YY, stack-auto-encoder model
• In: Lab 5
• Out: The tuned model and forecast

Lab 9

• Make XX, tensor decomposition model
• In: Lab 5
• Out: The tuned model and forecast

Lab 10

• Make XX Graph convolution model
• In: Lab 5
• Out: The tuned model and forecast

Lab 11

• Make XX Graph LSTM model
• In: Lab 5
• Out: The tuned model and forecast

Lab 12

• Make XX Graph GRAND model
• In: Lab 5
• Out: The tuned model and forecast

References

1. neurotycho.org
2. Яушев Ф.Ю., Исаченко Р.В., Стрижов В.В. Модели согласования скрытого пространства в задаче прогнозирования // Системы и средства информатики, 2021, 31(1) : 4-16.
3. Motrenko A.P., Strijov V.V. Multi-way feature selection for ECoG-based brain-computer interface // Expert Systems with Applications, 2018, 114(30) : 402-413.
4. Grabovoy A.V., Strijov V.V. Quasi-periodic time series clustering for human activity recognition // Lobachevskii Journal of Mathematics, 2020, 41 : 333-339.
5. Роман Владимирович Исаченко Снижение размерности пространства в задачах декодирования сигналов git, автореферат, презентация (PDF). 2021. МФТИ
6. Мотренко Анастасия Петровна. Выбор моделей прогнозирования мультикоррелирующих временных рядов, автореферат, презентация (PDF)