Mutual, information, metric entropy and cumulative relative entropy risk

被引:0
|
作者
Haussler, D [1 ]
Opper, M
机构
[1] Univ Calif Santa Cruz, Santa Cruz, CA 95064 USA
[2] Univ Wurzburg, Wurzburg, Germany
来源
ANNALS OF STATISTICS | 1997年 / 25卷 / 06期
关键词
mutual information; Hellinger distance; relative entropy; metric entropy; minimax risk; Bayes risk; density estimation; Kullback-Leibler distance;
D O I
暂无
中图分类号
O21 [概率论与数理统计]; C8 [统计学];
学科分类号
020208 ; 070103 ; 0714 ;
摘要
Assume (P-theta: theta epsilon Theta) is a set of probability distributions with a common dominating measure on a complete separable metric space Y. A state theta* epsilon Theta is chosen by Nature. A statistician obtains n independent observations Y-1,...,Y-n from Y distributed according to P-theta.. For each time t between 1 and n, based on the observations Y-1,...,Yt-1, the statistician produces an estimated distribution (P) over cap(t) for P-theta* and suffers a loss L(P-theta., (P) over cap(t)). The cumulative risk for the statistician is the average total loss up to time n. Of special interest in information theory, data compression, mathematical finance, computational learning theory and statistical mechanics is the special case when the loss L(P-theta*, (P) over cap(t)) is the relative entropy between the true distribution Pg and the estimated distribution IS,. Here the cumulative Bayes risk from time 1 to n is the mutual information between the random parameter Theta* and the observations Y-1,...,Y-n. New bounds on this mutual information are given in terms of the Laplace transform of the Hellinger distance between pairs of distributions indexed by parameters in Theta. From these, bounds on the cumulative minimax risk are given in terms of the metric entropy of Theta with respect to the Hellinger distance. The assumptions required for these bounds are very general and do not depend on the choice of the dominating measure. They apply to both finite- and infinite-dimensional Theta. They apply in some cases where Y is infinite dimensional, in some cases where Y is not compact, in some cases where the distributions are not smooth and in some parametric cases where asymptotic normality of the posterior distribution fails.
引用
收藏
页码:2451 / 2492
页数:42
相关论文
共 50 条
  • [21] Information Distances versus Entropy Metric
    Hu, Bo
    Bi, Lvqing
    Dai, Songsong
    ENTROPY, 2017, 19 (06)
  • [22] Pointwise information entropy for metric spaces
    Galatolo, S
    NONLINEARITY, 1999, 12 (05) : 1289 - 1298
  • [23] A Novel Distance Metric: Generalized Relative Entropy
    Liu, Shuai
    Lu, Mengye
    Liu, Gaocheng
    Pan, Zheng
    ENTROPY, 2017, 19 (06):
  • [24] Relative entropy as a universal metric for multiscale errors
    Chaimovich, Aviel
    Shell, M. Scott
    PHYSICAL REVIEW E, 2010, 81 (06):
  • [25] Entropy and Relative Entropy From Information-Theoretic Principles
    Gour, Gilad
    Tomamichel, Marco
    IEEE TRANSACTIONS ON INFORMATION THEORY, 2021, 67 (10) : 6313 - 6327
  • [26] Cumulative residual entropy: A new measure of information
    Rao, M
    Chen, YM
    Vemuri, BC
    Wang, F
    IEEE TRANSACTIONS ON INFORMATION THEORY, 2004, 50 (06) : 1220 - 1228
  • [27] Metric entropy and minimax risk in classification
    Haussler, D
    Opper, M
    STRUCTURES IN LOGIC AND COMPUTER SCIENCE, 1997, 1261 : 212 - 235
  • [28] Relative information entropy of an inhomogeneous universe
    Morita, Masaaki
    Buchert, Thomas
    Hosoya, Akio
    LiO, Nan
    INVISIBLE UNIVERSE INTERNATIONAL CONFERENCE, 2010, 1241 : 1074 - 1082
  • [29] Information, relative entropy of entanglement, and irreversibility
    Henderson, L
    Vedral, V
    PHYSICAL REVIEW LETTERS, 2000, 84 (10) : 2263 - 2266
  • [30] An information fractal dimensional relative entropy
    Wu, Jingyou
    AIP ADVANCES, 2024, 14 (02)