Machine Learning from Weak Supervision

An Empirical Risk Minimization Approach

by Sugiyama, Bao, Ishida, Lu, Sakai, Niu

ISBN: 9780262370554 | Copyright 2022

Click here to preview

Instructor Requests

Digital Exam/Desk Copy Print Desk Copy Ancillaries

Fundamental theory and practical algorithms of weakly supervised classification, emphasizing an approach based on empirical risk minimization.

Standard machine learning techniques require large amounts of labeled data to work well. When we apply machine learning to problems in the physical world, however, it is extremely difficult to collect such quantities of labeled data. In this book Masashi Sugiyama, Han Bao, Takashi Ishida, Nan Lu, Tomoya Sakai and Gang Niu present theory and algorithms for weakly supervised learning, a paradigm of machine learning from weakly labeled data. Emphasizing an approach based on empirical risk minimization and drawing on state-of-the-art research in weakly supervised learning, the book provides both the fundamentals of the field and the advanced mathematical theories underlying them. It can be used as a reference for practitioners and researchers and in the classroom.

The book first mathematically formulates classification problems, defines common notations, and reviews various algorithms for supervised binary and multiclass classification. It then explores problems of binary weakly supervised classification, including positive-unlabeled (PU) classification, positive-negative-unlabeled (PNU) classification, and unlabeled-unlabeled (UU) classification. It then turns to multiclass classification, discussing complementary-label (CL) classification and partial-label (PL) classification. Finally, the book addresses more advanced issues, including a family of correction methods to improve the generalization performance of weakly supervised learning and the problem of class-prior estimation.

Expand/Collapse All
Contents (pg. v)
I. Machine Learning from Weak Supervision (pg. 1)
1. Introduction (pg. 3)
1.1 Machine Learning (pg. 3)
1.2 Elements of Classification (pg. 7)
1.3 Aspects of Machine Learning (pg. 9)
1.4 Improving Data Collection and Weakly Supervised Learning (pg. 13)
1.5 Organization of This Book (pg. 15)
2. Formulation and Notation (pg. 21)
2.1 Binary Classification (pg. 21)
2.2 Multi-Class Classification (pg. 31)
3. Supervised Classification (pg. 35)
3.1 Positive-Negative (PN) Classification (pg. 35)
3.2 Multi-Class Classification (pg. 56)
II. Weakly Supervised Learning for Binary Classification (pg. 65)
4. Positive-Unlabeled (PU) Classification (pg. 67)
4.1 Introduction (pg. 67)
4.2 Formulation (pg. 68)
4.3 Unbiased Risk Estimation from PU Data (pg. 69)
4.4 Theoretical Analysis (pg. 75)
5. Positive-Negative-Unlabeled (PNU) Classification (pg. 85)
5.1 Introduction (pg. 85)
5.2 Formulation (pg. 86)
5.3 Manifold-Based Semi-Supervised Classification (pg. 87)
5.4 Information-Theoretic Semi-Supervised Classification (pg. 90)
5.5 PU+PN Classification (pg. 94)
5.6 Experiments (pg. 103)
5.7 Extensions (pg. 105)
6. Positive-Confidence (Pconf) Classification (pg. 111)
6.1 Introduction (pg. 111)
6.2 Related Works (pg. 112)
6.3 Problem Formulation (pg. 113)
6.4 Empirical Risk Minimization (ERM) Framework (pg. 114)
6.5 Theoretical Analysis (pg. 116)
6.6 Implementation (pg. 119)
6.7 Experiments (pg. 119)
7. Pairwise-Constraint Classification (pg. 127)
7.1 Introduction (pg. 127)
7.2 Formulation (pg. 128)
7.3 Similar-Unlabeled (SU) Classification (pg. 131)
7.4 Similar-Dissimilar (SD) and Dissimilar-Unlabeled (DU) Classification (pg. 136)
7.5 Similar-Dissimilar-Unlabeled (SDU) Classification (pg. 140)
7.6 Theoretical Analysis (pg. 140)
7.7 Experiments (pg. 143)
7.8 Ongoing Research (pg. 148)
8. Unlabeled-Unlabeled (UU) Classification (pg. 149)
8.1 Introduction (pg. 149)
8.2 Problem Formulation (pg. 150)
8.3 Risk Estimation from UU Data (pg. 152)
8.4 Generative Approach (pg. 164)
III. Weakly Supervised Learning for Multi-Class Classification (pg. 175)
9. Complementary-Label Classification (pg. 177)
9.1 Introduction (pg. 177)
9.2 Risk Estimation from CL Data (pg. 178)
9.3 Theoretical Analysis (pg. 182)
9.4 Incorporation of Ordinary-Labels (pg. 185)
9.5 Experiments (pg. 185)
9.6 Incorporation of Multi-Complementary-Labels (pg. 187)
10. Partial-Label Classification (pg. 193)
10.1 Introduction (pg. 193)
10.2 Formulation and Assumptions (pg. 193)
10.3 Risk Estimation (pg. 195)
10.4 Experiments (pg. 196)
10.5 Proper Partial-Label (PPL) Classification (pg. 198)
IV. Advanced Topics and Perspectives (pg. 205)
11. Non-Negative Correction for Weakly Supervised Classification (pg. 207)
11.1 Introduction (pg. 207)
11.2 Overfitting of Unbiased Learning Objectives (pg. 208)
11.3 Numerical Illustration (pg. 211)
11.4 Non-Negative Correction (pg. 213)
11.5 Theoretical Analyses (pg. 218)
11.6 Experiments (pg. 229)
12. Class-Prior Estimation (pg. 239)
12.1 Introduction (pg. 239)
12.2 Full Distribution Matching (pg. 241)
12.3 Mixture Proportion Estimation (pg. 242)
12.4 Partial Distribution Matching (pg. 248)
12.5 Penalized L1-Distance Minimization (pg. 254)
12.6 Class-Prior Estimation with Regrouping (pg. 262)
12.7 Class-Prior Estimation from Pairwise Data (pg. 273)
13. Conclusions and Prospects (pg. 275)
Notes (pg. 279)
Bibliography (pg. 283)
Index (pg. 293)

Masashi Sugiyama

Masashi Sugiyama is Director of the RIKEN Center for Advanced Intelligence Project and Professor of Computer Science at the University of Tokyo.

Han Bao

Han Bao is a PhD student in the Department of Computer Science at the University of Tokyo and Research Assistant at the RIKEN Center for Advanced Intelligence Project.

Takashi Ishida

Takashi Ishida is a Lecturer at the University of Tokyo and Visiting Scientist at the RIKEN Center for Advanced Intelligence Project.

Nan Lu

Nan Lu is a PhD student in the Department of Complexity Science and Engineering at the University of Tokyo and Research Assistant at the RIKEN Center for Advanced Intelligence Project.

Tomoya Sakai

Tomoya Sakai is Senior Researcher at NEC Corporation and Visiting Scientist at the RIKEN Center for Advanced Intelligence Project.

Gang Niu

Gang Niu is Research Scientist in the Imperfect Information Learning Team at the RIKEN Center for Advanced Intelligence Project.

Go paperless today! Available online anytime, nothing to download or install.


  • Bookmarking
  • Note taking
  • Highlighting