{"product_id":"perturbations-optimization-and-statistics-isbn-9780262549943","title":"Perturbations, Optimization, and Statistics","description":"\u003cb\u003eA description of perturbation-based methods developed in machine learning to augment novel optimization methods with strong statistical guarantees.\u003c\/b\u003e\u003cbr\u003e\u003cbr\u003eIn nearly all machine learning, decisions must be made given current knowledge. Surprisingly, making what is believed to be the best decision is not always the best strategy, even when learning in a supervised learning setting. An emerging body of work on learning under different rules applies perturbations to decision and learning procedures. These methods provide simple and highly efficient learning rules with improved theoretical guarantees. This book describes perturbation-based methods developed in machine learning to augment novel optimization methods with strong statistical guarantees, offering readers a state-of-the-art overview.\u003cbr\u003e\u003cbr\u003eChapters address recent modeling ideas that have arisen within the perturbations framework, including Perturb \u0026amp; MAP, herding, and the use of neural networks to map generic noise to distribution over highly structured data. They describe new learning procedures for perturbation models, including an improved EM algorithm and a learning algorithm that aims to match moments of model samples to moments of data. They discuss understanding the relation of perturbation models to their traditional counterparts, with one chapter showing that the perturbations viewpoint can lead to new algorithms in the traditional setting. And they consider perturbation-based regularization in neural networks, offering a more complete understanding of dropout and studying perturbations in the context of deep neural networks.Preface ix\u003cbr\u003e 1 Introduction 1\u003cbr\u003e Tamir Hazan, George Papandreou, and Daniel Tarlow\u003cbr\u003e 1.1 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1\u003cbr\u003e 1.2 Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . 4\u003cbr\u003e 1.3 Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9\u003cbr\u003e 1.4 Roadmap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12\u003cbr\u003e 1.5 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14\u003cbr\u003e 2 Perturb-and-MAP Random Fields 17\u003cbr\u003e George Papandreou and Alan L. Yuille\u003cbr\u003e 2.1 Energy-Based Models: Deterministic vs. Probabilistic Approaches . . . . . . . . . . . . . . . .  19\u003cbr\u003e 2.2 Perturb-and-MAP for Gaussian and Sparse Continuous MRFs 23\u003cbr\u003e 2.3 Perturb-and-MAP for MRFs with Discrete Labels . . . . . . . 28\u003cbr\u003e 2.4 On the Representation Power of the Perturb-and-MAP Model 35\u003cbr\u003e 2.5 Related Work and Recent Developments . . . . . . . . . . . . 38\u003cbr\u003e 2.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40\u003cbr\u003e 2.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41\u003cbr\u003e 3 Factorizing Shortest Paths with Randomized Optimum Models 45\u003cbr\u003e Daniel Tarlow, Alexander Gaunt, Ryan Adams, and Richard S. Zemel\u003cbr\u003e 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45\u003cbr\u003e 3.2 Building Structured Models: Design Considerations . . . . . . 47\u003cbr\u003e 3.3 Randomized Optimum Models (RandOMs) . . . . . . . . . . . 48\u003cbr\u003e 3.4 Learning RandOMs . . . . . . . . . . . . . . . . . . . . . . . . 54\u003cbr\u003e 3.5 RandOMs for Image Registration . . . . . . . . . . . . . . . . 56\u003cbr\u003e 3.6 Shortest Path Factorization . . . . . . . . . . . . . . . . . . . 56\u003cbr\u003e 3.7 Shortest Path Factorization with RandOMs . . . . . . . . . . 58\u003cbr\u003e 3.8 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63\u003cbr\u003e 3.9 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 68\u003cbr\u003e 3.10 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70\u003cbr\u003e 3.11 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70\u003cbr\u003e 4 Herding as a Learning System with Edge-of-Chaos Dynamics 73\u003cbr\u003e Yutian Chen and Max Welling\u003cbr\u003e 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74\u003cbr\u003e 4.2 Herding Model Parameters . . . . . . . . . . . . . . . . . . . . 77\u003cbr\u003e 4.3 Generalized Herding . . . . . . . . . . . . . . . . . . . . . . . 99\u003cbr\u003e 4.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109\u003cbr\u003e 4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118\u003cbr\u003e 4.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120\u003cbr\u003e 4.8 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123\u003cbr\u003e 5 Learning Maximum A-Posteriori Perturbation Models 127\u003cbr\u003e Andreea Gane, Tamir Hazan, and Tommi Jaakkola\u003cbr\u003e 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128\u003cbr\u003e 5.2 Background and Notation . . . . . . . . . . . . . . . . . . . . 130\u003cbr\u003e 5.3 Expressive Power of Perturbation Models . . . . . . . . . . . . 131\u003cbr\u003e 5.4 Higher Order Dependencies . . . . . . . . . . . . . . . . . . . 132\u003cbr\u003e 5.5 Markov Properties and Perturbation Models . . . . . . . . . . 134\u003cbr\u003e 5.6 Conditional Distributions . . . . . . . . . . . . . . . . . . . . . 136\u003cbr\u003e 5.7 Learning Perturbation Models . . . . . . . . . . . . . . . . . . 141\u003cbr\u003e 5.8 Empirical Results . . . . . . . . . . . . . . . . . . . . . . . . . 149\u003cbr\u003e 5.9 Perturbation Models and Stability . . . . . . . . . . . . . . . . 152\u003cbr\u003e 5.10 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 155\u003cbr\u003e 5.11 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156\u003cbr\u003e 6 On the Expected Value of Random Maximum A-Posteriori Perturbations 161\u003cbr\u003e Tamir Hazan and Tommi Jaakkola\u003cbr\u003e 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 161\u003cbr\u003e 6.2 Inference and Random Perturbations . . . . . . . . . . . . . . 164\u003cbr\u003e 6.3 Low-Dimensional Perturbations . . . . . . . . . . . . . . . . . 169\u003cbr\u003e 6.4 Empirical Evaluation . . . . . . . . . . . . . . . . . . . . . . . 182\u003cbr\u003e 6.5 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188\u003cbr\u003e 7 A Poisson Process Model for Monte Carlo 193\u003cbr\u003e Chris J. Maddison\u003cbr\u003e 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193\u003cbr\u003e 7.2 Poisson Processes . . . . . . . . . . . . . . . . . . . . . . . . . 196\u003cbr\u003e 7.3 Exponential Races . . . . . . . . . . . . . . . . . . . . . . . . 203\u003cbr\u003e 7.4 Gumbel Processes . . . . . . . . . . . . . . . . . . . . . . . . . 210\u003cbr\u003e 7.5 Monte Carlo Methods That Use Bounds . . . . . . . . . . . . 216\u003cbr\u003e 7.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226\u003cbr\u003e 7.9 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230\u003cbr\u003e 8 Perturbation Techniques in Online Learning and Optimization 233\u003cbr\u003e Jacob Abernethy, Chansoo Lee, and Ambuj Tewari\u003cbr\u003e 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233\u003cbr\u003e 8.2 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . 235\u003cbr\u003e 8.3 Gradient-Based Prediction Algorithm . . . . . . . . . . . . . . 237\u003cbr\u003e 8.4 Generic Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . 245\u003cbr\u003e 8.5 Experts Setting . . . . . . . . . . . . . . . . . . . . . . . . . . 247\u003cbr\u003e 8.6 Euclidean Balls Setting . . . . . . . . . . . . . . . . . . . . . . 252\u003cbr\u003e 8.7 The Multi-Armed Bandit Setting . . . . . . . . . . . . . . . . 254\u003cbr\u003e 8.9 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262\u003cbr\u003e 9 Probabilistic Inference by Hashing and Optimization 265\u003cbr\u003e Stefano Ermon\u003cbr\u003e 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265\u003cbr\u003e 9.2 Problem Statement and Assumptions . . . . . . . . . . . . . . 268\u003cbr\u003e 9.3 Approximate Model Counting via Randomized Hashing . . . . 270\u003cbr\u003e 9.4 Probabilistic Models and Approximate Inference: The WISH Algorithm . . . . . . . . . . .  274\u003cbr\u003e 9.5 Optimization Subject to Parity Constraints . . . . . . . . . . 279\u003cbr\u003e 9.6 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281\u003cbr\u003e 9.7 Open Problems and Research Challenges . . . . . . . . . . . . 282\u003cbr\u003e 9.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284\u003cbr\u003e 9.9 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285\u003cbr\u003e 10 Perturbation Models and PAC-Bayesian Generalization Bounds 289\u003cbr\u003e Joseph Keshet, Subhransu Maji, Tamir Hazan, and Tommi Jaakkola\u003cbr\u003e 10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 290\u003cbr\u003e 10.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . 292\u003cbr\u003e 10.3 PAC-Bayesian Generalization Bounds . . . . . . . . . . . . . 294\u003cbr\u003e 10.4 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296\u003cbr\u003e 10.5 The Bayesian Perspective . . . . . . . . . . . . . . . . . . . . 298\u003cbr\u003e 10.6 Approximate Inference . . . . . . . . . . . . . . . . . . . . . 301\u003cbr\u003e 10.7 Empirical Evaluation . . . . . . . . . . . . . . . . . . . . . . 302\u003cbr\u003e 10.8 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306\u003cbr\u003e 10.9 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307\u003cbr\u003e 11 Adversarial Perturbations of Deep Neural Networks 311\u003cbr\u003e David Warde-Farley and Ian Goodfellow\u003cbr\u003e 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 312\u003cbr\u003e 11.2 Adversarial Examples . . . . . . . . . . . . . . . . . . . . . . 312\u003cbr\u003e 11.3 Adversarial Training . . . . . . . . . . . . . . . . . . . . . . . 329\u003cbr\u003e 11.4 Generative Adversarial Networks . . . . . . . . . . . . . . . . 330\u003cbr\u003e 11.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338\u003cbr\u003e 11.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339\u003cbr\u003e 12 Data Augmentation via L´evy Processes 343\u003cbr\u003e Stefan Wager, William Fithian, and Percy Liang\u003cbr\u003e 12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 343\u003cbr\u003e 12.2 L´evy Thinning . . . . . . . . . . . . . . . . . . . . . . . . . . 349\u003cbr\u003e 12.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361\u003cbr\u003e 12.4 Simulation Experiments . . . . . . . . . . . . . . . . . . . . . 365\u003cbr\u003e 12.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368\u003cbr\u003e 12.6 Appendix: Proof of Theorem 12.4 . . . . . . . . . . . . . . . 369\u003cbr\u003e 12.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371\u003cbr\u003e 13 Bilu-Linial Stability 375\u003cbr\u003e Konstantin Makarychev and Yury Makarychev\u003cbr\u003e 13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 375\u003cbr\u003e 13.2 Stable Instances of Graph Partitioning Problems . . . . . . . 380\u003cbr\u003e 13.3 Stable Instances of Clustering Problems . . . . . . . . . . . . 391\u003cbr\u003e 13.4 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400Tamir Hazan is Assistant Professor at Technion, Israel Institute of Technology.\u003cbr\u003e\u003cbr\u003eGeorge Papandreou is a Research Scientist for Google, Inc.\u003cbr\u003e\u003cbr\u003eDaniel Tarlow is a Researcher at Microsoft Research Cambridge, UK.","brand":"The MIT Press","offers":[{"title":"Default Title","offer_id":46301908173029,"sku":"NP9780262549943","price":70.0,"currency_code":"USD","in_stock":false}],"thumbnail_url":"\/\/cdn.shopify.com\/s\/files\/1\/1842\/7735\/files\/9780262549943.jpg?v=1767734687","url":"https:\/\/k12savings.com\/products\/perturbations-optimization-and-statistics-isbn-9780262549943","provider":"K12savings","version":"1.0","type":"link"}