{"product_id":"learning-theory-from-first-principles-isbn-9780262049443","title":"Learning Theory from First Principles","description":"\u003cb\u003eA comprehensive and cutting-edge introduction to the foundations and modern applications of learning theory.\u003c\/b\u003e\u003cbr\u003e\u003cbr\u003eResearch has exploded in the field of machine learning resulting in complex mathematical arguments that are hard to grasp for new comers. . In this accessible textbook, Francis Bach presents the foundations and latest advances of learning theory for graduate students as well as researchers who want to acquire a basic mathematical understanding of the most widely used machine learning architectures. Taking the position that learning theory does not exist outside of algorithms that can be run in practice, this book focuses on the theoretical analysis of learning algorithms as it relates to their practical performance. Bach provides the simplest formulations that can be derived from first principles, constructing mathematically rigorous results and proofs without overwhelming students. \u003cbr\u003e\u003cbr\u003e\u003cul\u003e\u003cli\u003eProvides a balanced and unified treatment of most prevalent machine learning methods \u003c\/li\u003e\u003c\/ul\u003e\u003cul\u003e\n\u003cli\u003eEmphasizes practical application and features only commonly used algorithmic frameworks \u003c\/li\u003e\n\u003cli\u003eCovers modern topics not found in existing texts, such as overparameterized models and structured prediction \u003c\/li\u003e\n\u003cli\u003eIntegrates coverage of statistical theory, optimization theory, and approximation theory\u003c\/li\u003e\n\u003cli\u003eFocuses on adaptivity, allowing distinctions between various learning techniques\u003c\/li\u003e\n\u003cli\u003eHands-on experiments, illustrative examples, and accompanying code link theoretical guarantees to practical behaviors\u003c\/li\u003e\n\u003c\/ul\u003ePreface ix\u003cbr\u003eI Preliminaries 1\u003cbr\u003e1 Mathematical preliminaries 3\u003cbr\u003e1.1 Linear algebra and differentiable calculus.................. 3\u003cbr\u003e1.1.1 Minimization of quadratic forms................... 3\u003cbr\u003e1.1.2 Inverting a 2 × 2 matrix........................ 4\u003cbr\u003e1.1.3 Inverting matrices defined by blocks, matrix inversion lemma... 4\u003cbr\u003e1.1.4 Eigenvalue and singular value decomposition............ 6\u003cbr\u003e1.1.5 Differential calculus.......................... 7\u003cbr\u003e1.2 Concentration inequalities........................... 7\u003cbr\u003e1.2.1 Hoeffding’s inequality......................... 9\u003cbr\u003e1.2.2 McDiarmid’s inequality........................ 12\u003cbr\u003e1.2.3 Bernstein’s inequality (_x0007_)....................... 14\u003cbr\u003e1.2.4 Expectation of the maximum..................... 16\u003cbr\u003e1.2.5 Estimation of expectations through quadrature (_x0007_)......... 17\u003cbr\u003e1.2.6 Concentration inequalities for matrices (_x0007__x0007_)............. 18\u003cbr\u003e2 Introduction to supervised learning 21\u003cbr\u003e2.1 From training data to predictions....................... 22\u003cbr\u003e2.2 Decision theory................................. 25\u003cbr\u003e2.2.1 Loss functions............................. 25\u003cbr\u003e2.2.2 Risks................................... 26\u003cbr\u003e2.2.3 Bayes risk and Bayes predictor.................... 27\u003cbr\u003e2.3 Learning from data............................... 30\u003cbr\u003e2.3.1 Local averaging............................. 30\u003cbr\u003e2.3.2 Empirical risk minimization...................... 31\u003cbr\u003e2.4 Statistical learning theory........................... 33\u003cbr\u003e2.4.1 Measures of performance....................... 35\u003cbr\u003e2.4.2 Notions of consistency over classes of problems........... 35\u003cbr\u003e2.5 No free lunch theorems (_x0007_).......................... 36\u003cbr\u003e2.6 Quest for adaptivity.............................. 38\u003cbr\u003e2.7 Beyond supervised learning.......................... 39\u003cbr\u003e2.8 Summary - book outline............................ 40\u003cbr\u003e3 Linear least-squares regression 43\u003cbr\u003e3.1 Introduction................................... 43\u003cbr\u003e3.2 Least-squares framework............................ 44\u003cbr\u003e3.3 Ordinary least-squares (OLS) estimator................... 45\u003cbr\u003e3.3.1 Closed-form solution.......................... 45\u003cbr\u003e3.3.2 Geometric interpretation........................ 46\u003cbr\u003e3.3.3 Numerical resolution.......................... 47\u003cbr\u003e3.4 Statistical analysis of OLS........................... 47\u003cbr\u003e3.5 Fixed design setting.............................. 48\u003cbr\u003e3.5.1 Statistical properties of the OLS estimator............. 50\u003cbr\u003e3.5.2 Experiments.............................. 52\u003cbr\u003e3.6 Ridge least-squares regression......................... 53\u003cbr\u003e3.7 Lower-bound (_x0007_)................................ 57\u003cbr\u003e3.8 Random design analysis............................ 59\u003cbr\u003e3.8.1 Gaussian designs............................ 61\u003cbr\u003e3.8.2 General designs (_x0007__x0007_)......................... 61\u003cbr\u003e3.9 Principal component analysis (_x0007_)....................... 63\u003cbr\u003e3.10 Conclusion................................... 65\u003cbr\u003eII Generalization bounds for learning algorithms 67\u003cbr\u003e4 Empirical risk minimization 69\u003cbr\u003e4.1 Convexification of the risk........................... 70\u003cbr\u003e4.1.1 Convex surrogates........................... 71\u003cbr\u003e4.1.2 Geometric interpretation of the support vector machine (_x0007_).... 72\u003cbr\u003e4.1.3 Conditional Φ-risk and classification calibration (_x0007_)........ 74\u003cbr\u003e4.1.4 Relationship between risk and Φ-risk (_x0007__x0007_).............. 76\u003cbr\u003e4.2 Risk minimization decomposition....................... 80\u003cbr\u003e4.3 Approximation error.............................. 80\u003cbr\u003e4.4 Estimation error................................ 81\u003cbr\u003e4.4.1 Application of McDiarmid’s inequality................ 82\u003cbr\u003e4.4.2 Easy case I: quadratic functions.................... 83\u003cbr\u003e4.4.3 Easy case II: Finite number of models................ 84\u003cbr\u003e4.4.4 Beyond finitely many models through covering numbers (_x0007_)... 84\u003cbr\u003e4.5 Rademacher complexity............................ 86\u003cbr\u003e4.5.1 Symmetrization............................. 87\u003cbr\u003e4.5.2 Lipschitz-continuous losses...................... 89\u003cbr\u003e4.5.3 Ball-constrained linear predictions.................. 91\u003cbr\u003e4.5.4 Putting things together (linear predictions)............. 92\u003cbr\u003e4.5.5 From constrained to regularized estimation (_x0007_)........... 93\u003cbr\u003e4.5.6 Extensions and improvements..................... 96\u003cbr\u003e4.6 Model selection (_x0007_)............................... 98\u003cbr\u003e4.6.1 Structural risk minimization...................... 98\u003cbr\u003e4.6.2 Selection based on validation set................... 99\u003cbr\u003e4.7 Relationship with asymptotic statistics (_x0007_)................. 99\u003cbr\u003e4.8 Summary.................................... 101\u003cbr\u003e5 Optimization for machine learning 103\u003cbr\u003e5.1 Optimization in machine learning....................... 103\u003cbr\u003e5.2 Gradient descent................................ 105\u003cbr\u003e5.2.1 Simplest analysis: ordinary least-squares............... 106\u003cbr\u003e5.2.2 Convex functions and their properties................ 110\u003cbr\u003e5.2.3 Analysis of GD for strongly convex and smooth functions..... 112\u003cbr\u003e5.2.4 Analysis of GD for convex and smooth functions (_x0007_)....... 117\u003cbr\u003e5.2.5 Beyond gradient descent (_x0007_)..................... 120\u003cbr\u003e5.2.6 Non-convex objective functions (_x0007_)................. 122\u003cbr\u003e5.3 Gradient methods on non-smooth problems................. 123\u003cbr\u003e5.4 Convergence rate of stochastic gradient descent (SGD)........... 127\u003cbr\u003e5.4.1 Strongly convex problems (_x0007_).................... 132\u003cbr\u003e5.4.2 Adaptive methods (_x0007_)......................... 134\u003cbr\u003e5.4.3 Bias-variance trade-offs for least-squares (_x0007_)............. 135\u003cbr\u003e5.4.4 Variance reduction (_x0007_)........................ 138\u003cbr\u003e5.5 Conclusion................................... 143\u003cbr\u003e6 Local averaging methods 145\u003cbr\u003e6.1 Introduction................................... 145\u003cbr\u003e6.2 Local averaging methods............................ 147\u003cbr\u003e6.2.1 Linear estimators............................ 147\u003cbr\u003e6.2.2 Partition estimators.......................... 148\u003cbr\u003e6.2.3 Nearest-neighbors........................... 150\u003cbr\u003e6.2.4 Nadaraya-Watson estimator a.k.a. kernel regression (_x0007_)...... 151\u003cbr\u003e6.3 Generic “simplest” consistency analysis................... 153\u003cbr\u003e6.3.1 Fixed partition............................. 155\u003cbr\u003e6.3.2 k-nearest neighbor........................... 158\u003cbr\u003e6.3.3 Kernel regression (Nadaraya-Watson) (_x0007_).............. 160\u003cbr\u003e6.4 Universal consistency (_x0007_)........................... 163\u003cbr\u003e6.5 Adaptivity (_x0007__x0007_)................................ 166\u003cbr\u003e6.6 Conclusion................................... 167\u003cbr\u003e7 Kernel methods 169\u003cbr\u003e7.1 Introduction................................... 170\u003cbr\u003e7.2 Representer theorem.............................. 170\u003cbr\u003e7.3 Kernels..................................... 173\u003cbr\u003e7.3.1 Linear and polynomial kernels.................... 175\u003cbr\u003e7.3.2 Translation-invariant kernels on [0, 1]................. 176\u003cbr\u003e7.3.3 Translation-invariant kernels on Rd.................. 179\u003cbr\u003e7.3.4 Beyond vectorial input spaces (_x0007_).................. 182\u003cbr\u003e7.4 Algorithms................................... 184\u003cbr\u003e7.4.1 Representer theorem.......................... 184\u003cbr\u003e7.4.2 Column sampling............................ 185\u003cbr\u003e7.4.3 Random features............................ 185\u003cbr\u003e7.4.4 Dual algorithms (_x0007_).......................... 186\u003cbr\u003e7.4.5 Stochastic gradient descent (_x0007_).................... 187\u003cbr\u003e7.4.6 “Kernelization” of linear algorithms................. 188\u003cbr\u003e7.5 Generalization guarantees - Lipschitz-continuous losses........... 189\u003cbr\u003e7.5.1 Risk decomposition........................... 190\u003cbr\u003e7.5.2 Approximation error for translation-invariant kernels on Rd.... 191\u003cbr\u003e7.6 Theoretical analysis of ridge regression (_x0007_)................. 194\u003cbr\u003e7.6.1 Kernel ridge regression as a “linear” estimator........... 194\u003cbr\u003e7.6.2 Bias and variance decomposition (_x0007_)................ 195\u003cbr\u003e7.6.3 Relating empirical and population covariance operators...... 198\u003cbr\u003e7.6.4 Analysis for well-specified problems (_x0007_)............... 200\u003cbr\u003e7.6.5 Analysis beyond well-specified problems (_x0007_)............. 201\u003cbr\u003e7.6.6 Balancing bias and variance (_x0007_)................... 202\u003cbr\u003e7.7 Experiments................................... 203\u003cbr\u003e7.8 Conclusion................................... 205\u003cbr\u003e8 Sparse methods 207\u003cbr\u003e8.1 Introduction................................... 207\u003cbr\u003e8.1.1 Dedicated proof technique for constrained least-squares...... 209\u003cbr\u003e8.1.2 Probabilistic and combinatorial lemmas............... 210\u003cbr\u003e8.2 Variable selection by the ℓ0-penalty...................... 212\u003cbr\u003e8.2.1 Assuming k is known.......................... 212\u003cbr\u003e8.2.2 Estimating k (_x0007_)............................ 214\u003cbr\u003e8.3 Variable selection by ℓ1-regularization.................... 216\u003cbr\u003e8.3.1 Intuition and algorithms........................ 217\u003cbr\u003e8.3.2 Slow rates - random design...................... 221\u003cbr\u003e8.3.3 Slow rates - fixed design........................ 221\u003cbr\u003e8.3.4 Fast rates (_x0007_).............................. 224\u003cbr\u003e8.3.5 Zoo of conditions (_x0007__x0007_)......................... 225\u003cbr\u003e8.3.6 Random design (_x0007_)........................... 227\u003cbr\u003e8.4 Experiments................................... 228\u003cbr\u003e8.5 Extensions.................................... 229\u003cbr\u003e8.6 Conclusion................................... 230\u003cbr\u003e9 Neural networks 233\u003cbr\u003e9.1 Introduction................................... 233\u003cbr\u003e9.2 Single hidden layer neural network...................... 235\u003cbr\u003e9.2.1 Optimization.............................. 236\u003cbr\u003e9.2.2 Rectified linear units and homogeneity................ 237\u003cbr\u003e9.2.3 Estimation error............................ 239\u003cbr\u003e9.3 Approximation properties........................... 241\u003cbr\u003e9.3.1 Universal approximation property in one dimension........ 242\u003cbr\u003e9.3.2 Infinitely many neurons and variation norm............. 243\u003cbr\u003e9.3.3 Variation norm in one dimension................... 244\u003cbr\u003e9.3.4 Variation norm in arbitrary dimension................ 248\u003cbr\u003e9.3.5 Precise approximation properties................... 249\u003cbr\u003e9.3.6 From the variation norm to a finite number of neurons (_x0007_).... 250\u003cbr\u003e9.4 Generalization performance for neural networks............... 253\u003cbr\u003e9.5 Relationship with kernel methods (_x0007_).................... 255\u003cbr\u003e9.5.1 From a Banach space F1 to a Hilbert space F2 (_x0007_)......... 255\u003cbr\u003e9.5.2 Kernel function (_x0007__x0007_).......................... 257\u003cbr\u003e9.5.3 Upper-bound on RKHS norm (_x0007__x0007_).................. 258\u003cbr\u003e9.6 Experiments................................... 259\u003cbr\u003e9.7 Extensions.................................... 260\u003cbr\u003e9.8 Conclusion................................... 261\u003cbr\u003eIII Special topics 263\u003cbr\u003e10 Ensemble learning 265\u003cbr\u003e10.1 Averaging \/ bagging.............................. 266\u003cbr\u003e10.1.1 Independent datasets.......................... 266\u003cbr\u003e10.1.2 Bagging................................. 268\u003cbr\u003e10.2 Random projections and averaging...................... 269\u003cbr\u003e10.2.1 Gaussian sketching........................... 271\u003cbr\u003e10.2.2 Random projections.......................... 273\u003cbr\u003e10.3 Boosting..................................... 278\u003cbr\u003e10.3.1 Problem set-up............................. 279\u003cbr\u003e10.3.2 Incremental learning.......................... 281\u003cbr\u003e10.3.3 Matching pursuit............................ 282\u003cbr\u003e10.3.4 Adaboost................................ 283\u003cbr\u003e10.3.5 Greedy algorithm based on gradient boosting............ 284\u003cbr\u003e10.3.6 Convergence of expected risk..................... 287\u003cbr\u003e10.3.7 Experiments.............................. 290\u003cbr\u003e10.4 Conclusion................................... 290\u003cbr\u003e11 From online learning to bandits 291\u003cbr\u003e11.1 First-order online convex optimization.................... 292\u003cbr\u003e11.1.1 Convex case............................... 293\u003cbr\u003e11.1.2 Strongly-convex case (_x0007_)....................... 295\u003cbr\u003e11.1.3 Online mirror descent (_x0007_)....................... 295\u003cbr\u003e11.1.4 Lower bounds (_x0007__x0007_)........................... 297\u003cbr\u003e11.2 Zero-th order convex optimization...................... 299\u003cbr\u003e11.2.1 Smooth stochastic gradient descent.................. 301\u003cbr\u003e11.2.2 Stochastic smoothing (_x0007_)....................... 303\u003cbr\u003e11.2.3 Extensions............................... 307\u003cbr\u003e11.3 Multi-armed bandits.............................. 307\u003cbr\u003e11.3.1 Need for an exploration-exploitation trade-off............ 308\u003cbr\u003e11.3.2 “Explore-then-commit”........................ 308\u003cbr\u003e11.3.3 Optimism in the face of uncertainty (_x0007_)............... 310\u003cbr\u003e11.3.4 Adversarial bandits (_x0007_)........................ 312\u003cbr\u003e11.4 Conclusion................................... 314\u003cbr\u003e12 Over-parameterized models 315\u003cbr\u003e12.1 Implicit bias of gradient descent........................ 316\u003cbr\u003e12.1.1 Least-squares.............................. 316\u003cbr\u003e12.1.2 Separable classification......................... 318\u003cbr\u003e12.1.3 Beyond convex problems (_x0007_)..................... 323\u003cbr\u003e12.2 Double descent................................. 325\u003cbr\u003e12.2.1 The double descent phenomenon................... 325\u003cbr\u003e12.2.2 Empirical evidence........................... 326\u003cbr\u003e12.2.3 Linear regression with Gaussian projections (_x0007_)........... 327\u003cbr\u003e12.3 Global convergence of gradient descent.................... 332\u003cbr\u003e12.3.1 Mean field limits............................ 333\u003cbr\u003e12.3.2 From linear networks to positive definite matrices.......... 337\u003cbr\u003e12.3.3 Global convergence for positive definite matrices.......... 338\u003cbr\u003e12.3.4 Special case of Oja flow........................ 340\u003cbr\u003e12.4 Lazy regime and neural tangent kernels (_x0007_)................. 341\u003cbr\u003e12.5 Conclusion................................... 343\u003cbr\u003e13 Structured prediction 345\u003cbr\u003e13.1 Multi-category classification.......................... 346\u003cbr\u003e13.1.1 Extension of classical convex surrogates............... 346\u003cbr\u003e13.1.2 Generalization bound I: stochastic gradient descent......... 348\u003cbr\u003e13.1.3 Generalization bound II: Rademacher complexities (_x0007_)....... 350\u003cbr\u003e13.2 General set-up and examples......................... 352\u003cbr\u003e13.2.1 Examples................................ 352\u003cbr\u003e13.2.2 Structure encoding loss functions................... 354\u003cbr\u003e13.3 Surrogate methods............................... 356\u003cbr\u003e13.3.1 Score functions and decoding step.................. 356\u003cbr\u003e13.3.2 Fisher consistency and calibration functions............. 357\u003cbr\u003e13.3.3 Main surrogate frameworks...................... 357\u003cbr\u003e13.4 Smooth\/quadratic surrogates......................... 358\u003cbr\u003e13.4.1 Quadratic surrogate.......................... 358\u003cbr\u003e13.4.2 Theoretical guarantees......................... 358\u003cbr\u003e13.4.3 Linear estimators and decoding steps................. 359\u003cbr\u003e13.4.4 Smooth surrogates (_x0007_)......................... 360\u003cbr\u003e13.5 Max-margin formulations........................... 362\u003cbr\u003e13.5.1 Structured SVM............................ 362\u003cbr\u003e13.5.2 Max-min formulations (_x0007__x0007_)...................... 363\u003cbr\u003e13.6 Generalization bounds (_x0007_)........................... 365\u003cbr\u003e13.7 Experiments................................... 366\u003cbr\u003e13.7.1 Robust regression............................ 366\u003cbr\u003e13.7.2 Ranking................................. 366\u003cbr\u003e13.8 Conclusion................................... 369\u003cbr\u003e14 Probabilistic methods 371\u003cbr\u003e14.1 From empirical risks to log-likelihoods.................... 371\u003cbr\u003e14.1.1 Conditional likelihoods......................... 373\u003cbr\u003e14.1.2 Classical priors............................. 373\u003cbr\u003e14.1.3 Sparse priors.............................. 374\u003cbr\u003e14.1.4 On the relationship between MAP and MMSE (_x0007_)......... 375\u003cbr\u003e14.2 Discriminative vs. generative models..................... 378\u003cbr\u003e14.2.1 Linear discriminant analysis and softmax regression........ 379\u003cbr\u003e14.2.2 Naive Bayes............................... 379\u003cbr\u003e14.2.3 Maximum likelihood estimations................... 380\u003cbr\u003e14.3 Bayesian inference............................... 381\u003cbr\u003e14.3.1 Computational handling of posterior distributions......... 382\u003cbr\u003e14.3.2 Model selection through marginal likelihood............. 383\u003cbr\u003e14.4 PAC-Bayesian analysis............................. 384\u003cbr\u003e14.4.1 Set-up.................................. 384\u003cbr\u003e14.4.2 Uniformly bounded loss functions................... 385\u003cbr\u003e14.5 Conclusion................................... 387\u003cbr\u003e15 Lower bounds on performance 389\u003cbr\u003e15.1 Statistical lower bounds............................ 390\u003cbr\u003e15.1.1 Minimax lower bounds......................... 390\u003cbr\u003e15.1.2 Reduction to a hypothesis test.................... 391\u003cbr\u003e15.1.3 Review of information theory..................... 393\u003cbr\u003e15.1.4 Lower-bound on hypothesis testing based on information theory. 395\u003cbr\u003e15.1.5 Examples................................ 398\u003cbr\u003e15.1.6 Minimax lower bounds through Bayesian analysis.......... 399\u003cbr\u003e15.2 Optimization lower bounds.......................... 402\u003cbr\u003e15.2.1 Convex optimization.......................... 402\u003cbr\u003e15.2.2 Non-convex optimization (_x0007_)..................... 404\u003cbr\u003e15.3 Lower bounds for stochastic gradient descent (_x0007_).............. 407\u003cbr\u003e15.4 Conclusion................................... 409\u003cb\u003eFrancis Bach \u003c\/b\u003eis a researcher at Inria where he leads the machine learning team which is part of the Computer Science department at Ecole Normale Supérieure. His research focuses on machine learning and optimization.","brand":"The MIT Press","offers":[{"title":"Default Title","offer_id":46302614421733,"sku":"NP9780262049443","price":80.0,"currency_code":"USD","in_stock":false}],"thumbnail_url":"\/\/cdn.shopify.com\/s\/files\/1\/1842\/7735\/files\/9780262049443.jpg?v=1767731241","url":"https:\/\/k12savings.com\/products\/learning-theory-from-first-principles-isbn-9780262049443","provider":"K12savings","version":"1.0","type":"link"}