Recovery of functional laws from empirical evidence

I. Problem statement

Lecture 1. The problem of dependency reconstruction. Interpretation in terms of a function choice from the given class. Interpretation in terms of a model choice from the given class of models. Interpretation in terms of automata imitation. Criteria of selection.
Lecture 2. Definition of Probability measure (Kolmogorov axioms). Random variables. Distribution functions and their moments. Sums of random values. The Law of large numbers.

II. Empirical risk minimization method

Lecture 3. Least square method for regression estimation (general approach). Maximum likelihood method. (Least square method to find the best linear approximation).
Lecture 4. Search of a decision rule minimizing the number of errors (or, in general case, minimization of average penalty function). Nearest neighbor method. Linear decision rules.
Lecture 5. Perceptron. Potential functions method. Artificial neural nets.
Lecture 6. Support Vector Machine.
Lecture 7. Criticism of empirical risk minimization approach. Examples where it does not work. The problem of the uniform convergence of empirical risk to the true risk (its connection with the uniform convergence of frequencies to probabilities or means to expectations).
Lecture 8. Criteria of the uniform convergence of frequencies to probabilities over a class of events. Growth function. VC-dimension.
Lecture 9. Criteria of the uniform convergence of means to expectations.
Lecture 10. The model choice. Bayesian approach to the problem. General statement.