NEURAL COMPUTATION 9(1):1-42 (1997)

**Sepp Hochreiter, TUM
Jürgen Schmidhuber, IDSIA
http://www.idsia.ch/~juergen
**

**March 1996**

We present a new algorithm for finding low complexity neural networks with high generalization capability. The algorithm searches for a ``flat'' minimum of the error function. A flat minimum is a large connected region in weight-space where the error remains approximately constant. An MDL-based, Bayesian argument suggests that flat minima correspond to ``simple'' networks and low expected overfitting. The argument is based on a Gibbs algorithm variant and a novel way of splitting generalization error into underfitting and overfitting error. Unlike many previous approaches, ours does not require Gaussian assumptions and does not depend on a ``good'' weight prior - instead we have a prior over input/output functions, thus taking into account net architecture and training set. Although our algorithm requires the computation of second order derivatives, it has backprop's order of complexity. Automatically, it effectively prunes units, weights, and input lines. Various experiments with feedforward and recurrent nets are described. In an application to stock market prediction, flat minimum search outperforms (1) conventional backprop, (2) weight decay, (3) ``optimal brain surgeon'' / ``optimal brain damage''. We also provide pseudo code of the algorithm (omitted from the NC-version).

- BASIC IDEAS / OUTLINE
- TASK / ARCHITECTURE / BOXES
- THE ALGORITHM
- DERIVATION OF THE ALGORITHM

- EXPERIMENTAL RESULTS
- EXPERIMENT 1 - noisy classification.
- EXPERIMENT 2 - recurrent nets.
- EXPERIMENT 3 - stock market prediction (1).
- EXPERIMENT 4 - stock market prediction (2).
- EXPERIMENT 5 - stock market prediction (3).
- DETAILS / PARAMETERS

- RELATION TO PREVIOUS WORK
- LIMITATIONS / FINAL REMARKS / FUTURE RESEARCH
- APPENDIX - THEORETICAL JUSTIFICATION
- A.1. FLAT NETS: THE MOST PROBABLE HYPOTHESES
- A.2. WHY DOES THE HESSIAN DECREASE?
- A.3. EFFICIENT IMPLEMENTATION OF THE ALGORITHM
- A.4. PSEUDO CODE OF THE ALGORITHM

- Bibliography
- About this document ...

Back to Financial Forecasting page