Bounding the bias of contrastive divergence learning
Research output: Contribution to journal › Journal article › Research › peer-review
Standard
Bounding the bias of contrastive divergence learning. / Fischer, Anja; Igel, Christian.
In: Neural Computation, Vol. 23, No. 3, 2011, p. 664-673.Research output: Contribution to journal › Journal article › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - JOUR
T1 - Bounding the bias of contrastive divergence learning
AU - Fischer, Anja
AU - Igel, Christian
PY - 2011
Y1 - 2011
N2 - Optimization based on k-step contrastive divergence (CD) has become a common way to train restricted Boltzmann machines (RBMs). The k-step CD is a biased estimator of the log-likelihood gradient relying on Gibbs sampling. We derive a new upper bound for this bias. Its magnitude depends on k, the number of variables in the RBM, and the maximum change in energy that can be produced by changing a single variable. The last reflects the dependence on the absolute values of the RBM parameters. The magnitude of the bias is also affected by the distance in variation between the modeled distribution and the starting distribution of the Gibbs chain.
AB - Optimization based on k-step contrastive divergence (CD) has become a common way to train restricted Boltzmann machines (RBMs). The k-step CD is a biased estimator of the log-likelihood gradient relying on Gibbs sampling. We derive a new upper bound for this bias. Its magnitude depends on k, the number of variables in the RBM, and the maximum change in energy that can be produced by changing a single variable. The last reflects the dependence on the absolute values of the RBM parameters. The magnitude of the bias is also affected by the distance in variation between the modeled distribution and the starting distribution of the Gibbs chain.
U2 - 10.1162/NECO_a_00085
DO - 10.1162/NECO_a_00085
M3 - Journal article
C2 - 21162669
VL - 23
SP - 664
EP - 673
JO - Neural Computation
JF - Neural Computation
SN - 0899-7667
IS - 3
ER -
ID: 32089131