RANDOM NEURAL NETWORKS IN THE INFINITE WIDTH LIMIT AS GAUSSIAN PROCESSES

被引：7

作者：

Hanin, Boris ^{[1
]}

机构：

[1] Princeton Univ, Dept Operat Res & Financial Engn, Princeton, NJ 08544 USA

来源：

ANNALS OF APPLIED PROBABILITY | 2023年 / 33卷 / 6A期

关键词：

Neural networks; Gaussian processes; limit theorems; PRODUCTS;

D O I：

10.1214/23-AAP1933

中图分类号：

O21 [概率论与数理统计]; C8 [统计学];

学科分类号：

020208 ; 070103 ; 0714 ;

摘要：

This article gives a new proof that fully connected neural networks with random weights and biases converge to Gaussian processes in the regime where the input dimension, output dimension, and depth are kept fixed, while the hidden layer widths tend to infinity. Unlike prior work, convergence is shown assuming only moment conditions for the distribution of weights and for quite general nonlinearities.

引用

页码：4798 / 4819

页数：22

共 50 条

[31] The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization
Li, Mufan
Nica, Mihai
Roy, Daniel M.
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 35, NEURIPS 2022, 2022,
[32] RANDOM BINARY ALLOY .2. GAUSSIAN DISTRIBUTIONS AND THE LIMIT OF INFINITE-RANGE INTERACTIONS
BELL, JM
FRANKEL, NE
INAWASHIRO, S
PHYSICAL REVIEW B, 1986, 33 (01): : 487 - 498
[33] Non-Gaussian processes and neural networks at finite widths
Yaida, Sho
MATHEMATICAL AND SCIENTIFIC MACHINE LEARNING, VOL 107, 2020, 107 : 165 - 192
[34] Deep Neural Networks as Point Estimates for Deep Gaussian Processes
Dutordoir, Vincent
Hensman, James
van der wilk, Mark
Ek, Carl Henrik
Ghahramani, Zoubin
Durrande, Nicolas
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 34 (NEURIPS 2021), 2021, 34
[35] LIMIT-THEOREMS FOR MAXIMA AND CROSSINGS OF A SEQUENCE OF GAUSSIAN-PROCESSES AND APPROXIMATION OF RANDOM-PROCESSES
SELEZNJEV, OV
JOURNAL OF APPLIED PROBABILITY, 1991, 28 (01) : 17 - 32
[36] Infinite-Horizon Gaussian Processes
Solin, Arno
Hensman, James
Turner, Richard E.
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 31 (NIPS 2018), 2018, 31
[37] On the infinite divisibility of squared Gaussian processes
Nathalie Eisenbaum
Probability Theory and Related Fields, 2003, 125 : 381 - 392
[38] INFINITE MIXTURES OF MULTIVARIATE GAUSSIAN PROCESSES
Sun, Shiliang
PROCEEDINGS OF 2013 INTERNATIONAL CONFERENCE ON MACHINE LEARNING AND CYBERNETICS (ICMLC), VOLS 1-4, 2013, : 1011 - 1016
[39] On the infinite divisibility of squared Gaussian processes
Eisenbaum, N
PROBABILITY THEORY AND RELATED FIELDS, 2003, 125 (03) : 381 - 392
[40] Embedded Ensembles: Infinite Width Limit and Operating Regimes
Velikanov, Maksim
Kail, Roman
Anokhin, Ivan
Vashurin, Roman
Panov, Maxim
Zaytsev, Alexey
Yarotsky, Dmitry
INTERNATIONAL CONFERENCE ON ARTIFICIAL INTELLIGENCE AND STATISTICS, VOL 151, 2022, 151

← 1 2 3 4 5 →