GMDH-type neural networks

1

GMDH• Inductive Learning Algorithms for Complex Systems Modeling • Alexey G. Ivakhnenko (1994)

• Group Method of Data Handling• Inductive learning

• “Survival of the fittest”• Fully automatic structural

• “Self-organization”• “external criterion”

• Parametric optimization

2

Unit neuron• Polynomial neural networks• Solving the nonlinear problem using polynomial instead of linear

combination.• Kolmogorov-Gabor polynomial

3�̂�

𝑥𝑖

𝑥 𝑗

𝑓 (𝑥𝑖 ,𝑥 𝑗 )

• Weight update:

Back propagation:1-hidden-layer:f(XW)2-hidden-layer:f(f(XU)W)

4

U W

Error ε

∆ u=−α 𝜕 ε𝜕u=−𝛼∑ 𝜕 (𝑦− 𝑓 ( 𝑓 ( 𝑋𝑈 )𝑊 ) )𝜕 𝑓 ( 𝑓 ( 𝑋𝑈 )𝑊 )

𝜕 𝑓 ( 𝑓 ( 𝑋𝑈 )𝑊 )𝜕 𝑓 ( 𝑋𝑈 )

𝜕 𝑓 ( 𝑋𝑈 )𝜕𝑢 =𝛼∑ ( 𝑦− 𝑓 ( 𝑓 (𝑋𝑈 )𝑊 )) 𝜕 𝑓 ( 𝑓 (𝑋𝑈 )𝑊 )

𝜕 𝑓 (𝑋𝑈 )𝑥

External Criterion• Principle of self-organization

• Regularity criterion• It consists of a squared error from testing set B

• Symmetric Regularity criterion5

complexity

delta

Δ(B)

ε(A+B)

A: training setB: testing setΔ(): external criterion

Global minimum

Initial model selection

6External criterion

Optimal estimation