Stochastic chaining and strengthened information-theoretic generalization bounds

被引:0
|
作者
Zhou, Ruida [1 ]
Tian, Chao [1 ]
Liu, Tie [1 ]
机构
[1] Texas A&M Univ, Dept Elect & Comp Engn, College Stn, TX 77845 USA
关键词
SUCCESSIVE REFINEMENT;
D O I
10.1016/j.jfranklin.2023.02.009
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
We propose a new approach to apply the chaining technique in conjunction with information-theoretic measures to bound the generalization error of machine learning algorithms. Different from the deter-ministic chaining approach based on hierarchical partitions of a metric space, previously proposed by Asadi et al., we propose a stochastic chaining approach, which replaces the hierarchical partitions with an abstracted Markovian model borrowed from successive refinement source coding. This approach has three benefits over deterministic chaining: (1) the metric space is not necessarily bounded, (2) facilitation of subsequent analysis to yield more explicit bound, and (3) further opportunity to optimize the bound by removing the geometric rigidity of the partitions. The proposed approach includes the traditional chaining as a special case, and can therefore also utilize any deterministic chaining construction. We illustrate these benefits using the problem of estimating Gaussian mean and that of phase retrieval. For the former, we derive a bound that provides an order-wise improvement over previous results, and for the latter we provide a stochastic chain that allows optimization over the chaining parameter.(c) 2023 The Franklin Institute. Published by Elsevier Inc. All rights reserved.
引用
收藏
页码:4114 / 4134
页数:21
相关论文
共 50 条
  • [1] Strengthened Information-theoretic Bounds on the Generalization Error
    Issa, Ibrahim
    Esposito, Amedeo Roberto
    Gastpar, Michael
    [J]. 2019 IEEE INTERNATIONAL SYMPOSIUM ON INFORMATION THEORY (ISIT), 2019, : 582 - 586
  • [2] Limitations of Information-Theoretic Generalization Bounds for Gradient Descent Methods in Stochastic Convex Optimization
    Haghifam, Mahdi
    Rodriguez-Galvez, Borja
    Thobaben, Ragnar
    Skoglund, Mikael
    Roy, Daniel M.
    Dziugaite, Gintare Karolina
    [J]. INTERNATIONAL CONFERENCE ON ALGORITHMIC LEARNING THEORY, VOL 201, 2023, 201 : 663 - 706
  • [3] Information-theoretic distance measures and a generalization of stochastic resonance
    Robinson, JWC
    Asraf, DE
    Bulsara, AR
    Inchiosa, ME
    [J]. PHYSICAL REVIEW LETTERS, 1998, 81 (14) : 2850 - 2853
  • [4] Information-Theoretic Bounds on the Moments of the Generalization Error of Learning Algorithms
    Aminian, Gholamali
    Toni, Laura
    Rodrigues, Miguel R. D.
    [J]. 2021 IEEE INTERNATIONAL SYMPOSIUM ON INFORMATION THEORY (ISIT), 2021, : 682 - 687
  • [5] Generalization Bounds for Meta-Learning: An Information-Theoretic Analysis
    Chen, Qi
    Shui, Changjian
    Marchand, Mario
    [J]. ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 34 (NEURIPS 2021), 2021, 34
  • [6] Information-Theoretic Generalization Bounds for Meta-Learning and Applications
    Jose, Sharu Theresa
    Simeone, Osvaldo
    [J]. ENTROPY, 2021, 23 (01) : 1 - 28
  • [7] Time-Independent Information-Theoretic Generalization Bounds for SGLD
    Futami, Futoshi
    Fujisawa, Masahiro
    [J]. ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 36 (NEURIPS 2023), 2023,
  • [8] Improved Information-Theoretic Generalization Bounds for Distributed, Federated, and Iterative Learning
    Barnes, Leighton Pate
    Dytso, Alex
    Poor, Harold Vincent
    [J]. ENTROPY, 2022, 24 (09)
  • [9] Formal limitations of sample-wise information-theoretic generalization bounds
    Harutyunyan, Hrayr
    Steeg, Greg Ver
    Galstyan, Aram
    [J]. 2022 IEEE INFORMATION THEORY WORKSHOP (ITW), 2022, : 440 - 445
  • [10] Information-Theoretic Bounds on the Generalization Error and Privacy Leakage in Federated Learning
    Yagli, Semih
    Dytso, Alex
    Poor, H. Vincent
    [J]. PROCEEDINGS OF THE 21ST IEEE INTERNATIONAL WORKSHOP ON SIGNAL PROCESSING ADVANCES IN WIRELESS COMMUNICATIONS (IEEE SPAWC2020), 2020,