Fisher Information and Natural Gradient Learning of Random Deep Networks

A deep neural network is a hierarchical nonlinear model transforming input signals to output signals. Its input-output relation is considered to be stochastic, being described for a given input by a parameterized conditional probability distribution of outputs. The space of parameters consisting of weights and biases is a Riemannian manifold, where the metric is defined by the Fisher information m

[elu_18]

NNのフィッシャー情報行列は同じユニットを終端とするパラメータ間以外はほぼ0であるようなブロック対角行列であることが予想されていたが、パラメータがガウシアンに従うランダム変数でかつ、平均場近似の元でこれ