ADVANCES IN NEURAL INFORMATION

PROCESSING SYSTEMS 12

Proceedings ofthe 1999 Conference

edited by

Sara A. Solla, Todd ΑΓ. Leen and Klaus-Robert Müller

A Bradford BookThe MIT Press

Cambridge, MassachusettsLondon, England

Contents

Preface xiii

NIPS Committees xv

Reviewers xvii

Part I Cognitive Science

Recognizing Evoked Potentials in a Virtual Environment,

Jessica D. Bayliss and Dana H. Ballard 3

Α Νeurodynamical Approach to Visual Attention, G u s t a v o D e c o a n d J o s e f Z i h l . . . 1 0

Effects of Spatial and Temporal Contiguity on the Acquisition of Spatial

Information, Thea B. Ghiselli-Crippa and Paul W. Munro 17

Acquisition in Autoshaping, Sham Kakade and Peter Dayan 24

Robust Recognition of Noisy and Superimposed Patterns via Selective Attention,

Soo-Young Lee and Michael C. Mozer 31Perceptual Organization Based on Temporal Dynamics,Xiuwen Liu and DeLiang L. Wang 38

Information Factorization in Connectionist Models of Perception,Javier R. Movellan and James L. McClelland 45

Graded Grammaticality in Prediction Fractal Machines,

Shan Parfitt, Peter Tino and Georg Dorffner 52

Rules and Similarity in Concept Learning, Joshua B. Tenenbaum 59

Evolving Learnable Languages, Bradley Tonkes, Alan Blair and Janet Wiles . . . . 66

Learning Statistically Neutral Tasks without Expert Guidance,

Ton Weijters, Antal van den Bosch and Erie Postma 73

A Generative Model for Attractor Dynamics,Richard S. Zemel and Michael C. Mozer . 80

Part II Neuroscience

Recurrent Cortical Competition: Strengthen or Weaken?,P é t e r A d o r j â n , L a r s S c h w a b e , C h r i s t i a n P i e p e n b r o c k a n d K l a u s O b e r m a y e r . . . . 8 9

Effective Learning Requires Neuronal Remodeling of Hebbian Synapses,

Gal Chechik, Isaac Meilijson and Eytan Ruppin 96

Wiring Optimization in the Brain, Dmitri Β. Chklovskii and Charles F. Stevens . . 103

Optimal Sizes of Dendritic and Axonal Arbors, Dmitri Β. Chklovskii 108

ν | · Contents

Neural Representation of Multi-Dimensional Stimuli,

Christian W. Eurich, Stefan D. Wilke and Helmut Schwegler 115

Spiking Boltzmann Machines, Geoffrey E. Hinton and Andrew D. Brown 122

Distributed Synchrony of Spiking Neurons in a Hebbian Cell Assembly,

David Horn, Nir Levy, Isaac Meilijson and Eytan Ruppin 129

Can VI Mechanisms Account for Figure-Ground and Medial Axis Effects?,

ZhaopingLi 136

Channel Noise in Excitable Neural Membranes,Amit Manwani, Peter N. Steinmetz and Christof Koch 143

LTD Facilitates Learning in a Noisy Environment,Paul W. Munro and Gerardina Hernandez 150

Memory Capacity of Linear vs. Nonlinear Models of Dendritic Integration,

Panayiota Poirazi and Bartlett W. Mel 157

Predictive Sequence Learning in Recurrent Neocortical Circuits,Rajesh P. N. Rao and Terrence J. Sejnowski 164

A Recurrent Model of the Interaction Between Prefrontal and InferotemporalC o r t e x i n D e l a y T a s k s , A l f o n s o R e n a r t , N e s t o r P a r g a a n d E d m u n d T . R o l l s . . . . 1 7 1

Information Capacity and Robustness of Stochastic Neuron Models,Elad Schneidman, Idan Segev and Naftali Tishby 178

An MEG Study of Response Latency and Variability in the Human Visual SystemDuring a Visual-Motor Integration Task, Akaysha C. Tang,Barak A. Pearlmutter, Tim A. Hely, Michael Zibulevsky and Michael P. Weisend . 185

Population Decoding Based on an Unfaithful Model,

Si Wu, Hiroyuki Nakahara, Noboru Murata and Shun-ichi Amari 192

Spike-based Learning Rules and Stabilization of Persistent Neural Activity,Xiaohui Xie and H. Sebastian Seung 199

Part III Theory

A Variational Bay stan Framework for Graphical Models, Hagai Attias 209

Model Selection in Clustering by Uniform Convergence Bounds,

Joachim M. Buhmann and Marcus Held 216

Uniqueness ofthe SVM Solution, Christopher J. C. Burges and David J. Crisp . . . 223

Model Selection for Support Vector Machines,

Olivier Chapelle and Vladimir N. Vapnik 230

Dynamics of Supervised Learning with Restricted Training Sets and NoisyTeachers, A. C. C. Coolen and C. W. H. Mace 237

Λ Geometric Interpretation ofv-SVM Classifiers,

David J. Crisp and Christopher J. C. Burges 244

Contents vii

Efficient Approaches to Gaussian Process Classification,

Lehel Csató, Ernest Fokoué, Manfred Opper, Bernhard Schottky and Ole Winther . 251

Potential Boosters?, Nigel Duffy and David Helmbold 258

Bayesian Averag ing is Well-Temperated, Lars Kai Hansen 265

Regular and Irregular Gallager-type Error-Correcting Codes,

Y o s h i y u k i K a b a s h i m a , T a t s u t o M u r a y a m a , D a v i d S a a d a n d R e n a t o V i c e n t e . . . . 2 7 2

Mixture Density Estimation, J o n a t h a n Q . L i a n d A n d r e w R . B a r r o n 2 7 9

Statistical Dynamics of Batch Learning, Song Li and K. Y. Michael Wong 286

Neural Computation with Winner-Take-All as the Only Nonlinear Operation,Wolfgang Maass 293Boosting with Multi- Way Branching in Decision Trees,

Yishay Mansour and David McAHester 300

Inference for the Generalization Error, C l a u d e N a d e a u a n d Y o s h u a B e n g i o . . . . 3 0 7

Resonance in a Stochastic Neuron Model with Delayed Interaction,

Toru Ohira, Yuzuru Sato and Jack D. Cowan 314Understanding Stepwise Generalization of Support Vector Machines: a ToyModel, Sebastian Risau-Gusman and Mirta B. Gordon 321

Lower Bounds on the Complexity of Approximating Continuous Functions bySigmoidal Neural Networks, Michael Schmitt 328

Noisy Neural Networks and Generalizations,Hava T. Siegelmann, Alexander Roitershtein and Asa Ben-Hur 335

The Entropy Regularization Information Criterion, Alexander J. Smola,

John Shawe-Taylor, Bernhard Schölkopf and Robert C. Williamson 342

Probabilistic Methods for Support Vector Machines, Peter Sollich 349

Algebraic Analysis for Non-regular Learning Machines, Sumio Watanabe 356

Semiparametric Approach to Multichannel Blind Deconvolution of NonminimumPhase Systems, L.-Q. Zhang, Shun-ichi Amari and A. Cichocki 363Some Theoretical Results Concerning the Convergence of Compositions ofRegularized Linear Functions, Tong Zhang 370

Part IV Algorithms and Architecture

Robust Full Bayesian Methods for Neural Networks,

Christophe Andrieu, Joäo F. G. de Freitas and Arnaud Doucet 379

Independent Factor Analysis with Temporally Structured Sources, Hagai Attias . . 386

Gaussian Fields for Approximate Inference in Layered Sigmoid Belief Networks,David Barber and Peter Sollich 393Modeling High-Dimensional Discrete Data with Multi-Layer Neural Networks,Yoshua Bengio and Samy Bengio 400

viii Contents

Robust Neural Network Regression for Offline and Online Learning,Thomas Briegel and Volker Tresp 407

Reconstruction of Sequential Data with Probabilistic Models and ContinuityConstraints, Miguel Ä. Carreira-Perpinän 414

Transductive Inference for Estimating Values of Functions,Olivier Chapelle, Vladimir Ν. Vapnik and Jason Weston 421

The Nonnegative Boltzmann Machine,Oliver B. Downs, David J.C. MacKay and Daniel D. Lee 428

Differentiating Functions of the Jacobian with Respect to the Weights,

Gary William Flake and Barak A. Pearlmutter 435

Local Probability Propagation for Factor Analysis, Brendan J. Frey 442

Variational Inference for Bayesian Mixtures of Factor Analysers,

Zoubin Ghahramani and Matthew J. Beal 449

Bayesian Transduction, Thore Graepel, Ralf Herbrich and Klaus Obermayer . . . . 456

Learning to Parse Images,

Geoffrey Ε. Hinton, Zoubin Ghahramani and Yee Whye Teh 463

Maximum Entropy Discrimination, Tommi Jaakkola, Marina Meila and Tony Jebara 470

Topographic Transformation as a Discrete Latent Variable,

Nebojsa Jojic and Brendan J. Frey 477An Improved Decomposition Algorithm for Regression Support Vector Machines,Pavel Laskov 484Algorithms for Independent Components Analysis and Higher Order Statistics,

Daniel D. Lee, Uri Rokni and Haim Sompolinsky 491

The Relaxed Online Maximum Margin Algorithm, Yi L i a n d P h i l i p M . L o n g . . . . 4 9 8

Bayesian Network Induction via Local Neighborhoods,Dimitris Margaritis and Sebastian Thrun 505

Boosting Algorithms as Gradient Descent,

Llew Mason, Jonathan Baxter, Peter Bartlett and Marcus Frean 512

A Multi-class Linear Learning Algorithm Related to Winnow, Chris Mesterharm . 519

Invariant Feature Extraction and Classification in Kernel Spaces,Sebastian Mika, Gunnar Ratsch, Jason Weston, Bernhard Schölkopf,Alexander J. Smola and Klaus-Robert Müller 526

Approximate Inference Algorithms for Two-Layer Bayesian Networks,Andrew Y Ng and Michael I. Jordan 533

Optimal Kernel Shapes for Local Linear Regression,Dirk Ormoneit and Trevor Hastie 54O

Large Margin DAGs for Multiclass Classification,John C. Platt, Nello Cristianini and John Shawe-Taylor 547

The Infinite Gaussian Mixture Model, Carl Edward Rasmussen 554

Contents ix

ν-Arc: Ensemble Learning in the Presence of Outliers, Gunnar Rätsch,Bernhard Schölkopf, Alexander J. Smola, Klaus-Robert Müller, Takashi Onodaand Sebastian Mika 561

Nonlinear Discriminant Analysis Using Kernel Functions,Volker Roth and Volker Steinhage 568

An Analysis of Turbo Decoding with Gaussian Densities,Paat Rusmevichientong and Benjamin Van Roy 575

Support Vector Method for Novelty Detection, Bernhard Schölkopf,Robert C. Williamson, Alexander J. Smola, John Shawe-Taylor and John C Platt . 582

Better Generative Models for Sequential Data Problems: Bidirectional RecurrentMixture Density Networks, Mike Schuster 589

Greedy Importance Sampling, Dale Schuurmans 596

Bayesian Model Selection for Support Vector Machines, Gaussian Processes and

Other Kernel Classifiers, Matthias Seeger 603

Leveraged Vector Machines, Yoram Singer 610

Agglomerative Information Bottleneck, Noam Slonim and Naftali Tishby 617Training Data Selection for Optimal Generalization in Trigonometrie PolynomialNetworks, Masashi Sugiyama and Hidemitsu Ogawa 624

Predictive Approaches for Choosing Hyperparameters in Gaussian Processes,S. Sundararajan and S. Sathiya Keerthi 631

On Input Selection with Reversible Jump Markov Chain Monte Carlo Sampling,Peter Sykacek 638

Building Predictive Models from Fractal Representations of Symbolic Sequences,

Peter Tifio and Georg Dorffner 645

The Relevance Vector Machine, Michael E. Tipping 652

Support Vector Method for Multivariate Density Estimation,Vladimir N. Vapnik and Sayan Mukherjee 659Dual Estimation and the Unscented Transformation,Eric A. Wan, Rudolph van der Merwe and Alex T. Nelson 666

Correctness of Belief Propagation in Gaussian Graphical Models of Arbitrary

Topology, Yair Weiss and William T. Freeman 673

A MCMC Approach to Hierarchical Mixture Modelling, Christopher K. I. Williams 680

Data Visualization and Feature Selection: New Algorithms for NongaussianData, Howard Hua Yang and John Moody 687Manifold Stochastic Dynamics for Bayesian Learning,Mark Zlochin and Yoram Baram 694

χ Contents

PartV Implementation

The Parallel Problems Server: an Interactive Tool for Large Scale MachineLearning, Charles Lee Isbell, Jr. and Parry Husbands 703

An Oculo-Motor System with Multi-Chip Neuromorphic Analog VLSI Control

Oliver Landolt and Steve Gyger 710

A Winner-Take-All Circuit with Controllable Soft Max Property, S h i h - C h i i L i u . . . 7 1 7

A Neuromorphic VLSI System for Modeling the Neural Control of AxialLocomotion, Girish N. Patel, Edgar A. Brown and Stephen P. DeWeerth 724

Bifurcation Analysis of a Silicon Neuron, Girish N. Patel,GennadyS. Cymbalyuk, Ronald L.Calabrese and Stephen P. De Weerth 731

An Analog VLSI Model of Periodicity Extraction, André van Schaik 738

Part VI Speech, Handwriting and Signal Processing

An Oscillatory Correlation Framework for Computational Auditory SceneAnalysis, Guy J. Brown and DeLiang L. Wang 747

Bayesian Modelling of fMRI Ήme Series,

Pedro A. d. F. R. Hojen-Sorensen, Lars Kai Hansen and Carl Edward Rasmussen . 754

Neural System Model of Human Sound Localization, Craig T, Jin and Simon Carlile 761

Spectral Cues in Human Sound Localization,Craig T. Jin, Anna Corderoy, Simon Carlile and André van Schaik 768Broadband Direction-Of Arrival Estimation Based on Second Order Statistics,

J u s t i n i a n R o s c a , J o s e p h Ó R u a n a i d h , A l e x a n d e r J o u r j i n e a n d S c o t t R i c k a r d . . . . 7 7 5

Constrained Hidden Markov Models, Sam Roweis 782

Online Independent Component Analysis with Local Learning Rate Adaptation,Nicol N. Schraudolph and Xavier Giannakopoulos 789

Speech Modelling Using Subspace and EM Techniques,Gavin Smith, Joao F. G. de Freitas, Tony Robinson and Mahesan Niranjan 796

Search for Information Bearing Components in Speech,Howard Hua Yang and Hynek Hermansky 803

Part VII Visual Processing

Audio Vision: Using Audio-Visual Synchrony to Locate Sounds,John Hershey and Javier R. Movellan 813

Bayesian Reconstruction of 3D Human Motion from Single-Camera Video,Nicholas R. Howe, Michael E. Leventon and William T. Freeman 820

Emergence of Topography and Complex Cell Properties from Natural Imagesusing Extensions of ICA, Aapo Hyvärinen and Patrik Hoyer 827

Contents χϊ

An Information-Theoretic Framework for Understanding Saccadic EyeMovements, Tai Sing Lee and Stella X. Yu 834

Learning Sparse Codes with a Mixture-of-Gaussians Prion

Bruno A. Olshausen and Κ. Jarrod Millman 841

Hierarchical Image Probability (HIP) Models, Clay D. Spence and Lucas Parra . . 848

Scale Mixtures of Gaussians and the Statistics of Natural Images,

Martin J. Wainwright and Eero P. Simoncelli 855

A SNoW-Based Face Detector, Ming-Hsuan Yang, Dan Roth and Narendra Ahuja . 862

Managing Uncertainty in Cue Combination, Zhiyong Yang and Richard S. Zemel . 869

Part VIII Applications

Robust Learning of Chaotic Attractors, Rembrandt Bakker, Jaap C. Schouten,Marc-Olivier Coppens, Floris Takens, C. Lee Giles and Cor M. van den Bleek . . . 879

Image Representations for Facial Expression Coding, Marian Stewart Bartlett,Gianluca Donato, Javier R. Movellan, Joseph C. Hager, Paul Ekman andTerrence J. Sejnowski 886

Low Power Wireless Communication via Reinforcement Learning,Timothy X. Brown 893

Learning Informative Statistics: A Nonparametric Approach,John W. Fisher III, Alexander T. Ihier and Paul A. Viola 900

Kirchoff Law Markov Fields for Analog Circuit Design, Richard M. Golden . . . . 907

Learning the Similarity of Documents: An Information-Geometrie Approach toDocument Retrieval and Categorization, Thomas Hofmann 914

Constructing Heterogeneous Committees Using Input Feature Grouping:Application to Economic Forecasting, Yuansong Liao and John Moody 921

From Coexpression to Coregulation: An Approach to Inferring TranscriptionalRegulation among Gene Classes from Large-Scale Expression Data,Eric Mjolsness, Tobias Mann, Rebecca Castano and Barbara Wold 928

Churn Reduction in the Wireless Industry, Michael C. Mozer,Richard Wolniewicz, David B. Grimes, Eric Johnson and Howard Kaushansky . , . 935

Unmixing Hyperspectral Data,Lucas Parra, Clay D. Spence, Paul Sajda, Andreas Ziehe and Klaus-Robert Müller 942

Application of Blind Separation of Sources to Optical Recording of BrainActivity, Holger Schöner, Martin Steuer, Ingo Schießl, John E.W. Mayhew,Jennifer Lund, Niall McLoughlin and Klaus Obermayer 949

Reinforcement Learning for Spoken Dialogue Systems,Satinder Singh, Michael Kearns, Diane Litman and Marilyn Walker 956

Image Recognition in Context: Application to Microscopic Urinalysis,Xubo Β. Song, Joseph Sill, Yaser Abu-Mostafa and Harvey Kasdan 963

xii Contents

Generalized Model Selection for Unsupervised Learning in High Dimensions,Shivakumar Vaithyanathan and Byron Dom 970

Learning front User Feedback in Image Retrieval Systems,Nuno Vasconcelos and Andrew Lippman 977

Part IX Control, Navigation and Planning

An Environment Model for Nonstationary Reinforcement Learning,Samuel P. M. Choi, Dit-Yan Yeung and Nevin L. Zhang 987

State Abstraction in MAXQ Hierarchical Reinforcement Learning,Thomas G. Dietterich 994

Approximate Planning in Large POMDPs via Reusable Trajectories,

Michael Kearns, Yishay Mansour and Andrew Y. Ng 1001

Actor-Critic Algorithms, Vijay R. Konda and John N. Tsitsiklis 1008

Bayesian Map Learning in Dynamic Environments, Kevin P.Murphy 1015

Policy Search via Density Estimation,Andrew Y Ng, Ronald Parr and Daphne Koller 1022

Neural Network Based Model Predictive Control, Stephen Piche, Jim Keeler,Greg Martin, Gene Boe, Doug Johnson and Mark Gerules 1029

Reinforcement Learning Using Approximate Belief States,Andres Rodriguez, Ronald Parr and Daphne Koller 1036

Coastal Navigation with Mobile Robots, Nicholas Roy and Sebastian Thrun . . . . 1043

Learning Factored Representations for Partially Observable Markov DecisionProcesses, Brian Sallans 1050

Policy Gradient Methods for Reinforcement Learning with FunctionApproximation,Richard S. Sutton, David Mc Allester, Satinder Singh and Yishay Mansour 1057

Monte Carlo POMDPs, Sebastian Thrun 1064

Index of Authors 1071

Keyword Index 1075