Neurodynamics - Lecture Slides - Neural Networks | CPSC 636, Study notes of Computer Science

Download Neurodynamics - Lecture Slides - Neural Networks | CPSC 636 and more Study notes Computer Science in PDF only on Docsity! Slide10 Haykin Chapter 14: Neurodynamics CPSC 636-600 Instructor: Yoonsuck Choe Spring 2008 1 Neural Networks with Temporal Behavior • Inclusion of feedback gives temporal characteristics to neural networks: recurrent networks. • Two ways to add feedback: – Local feedback – Global feedback • Recurrent networks can become unstable or stable. • Main interest is in recurrent network’s stability: neurodynamics. • Stability is a property of the whole system: coordination between parts is necessary. 2 Stability in Nonlinear Dynamical System • Lyapunov stablity: more on this later. • Study of neurodynamics: – Deterministic neurodynamics: expressed as nonlinear differential equations. – Stochastic neurodynamics: expressed in terms of stochastic nonlinear differential equations. Recurrent networks perturbed by noise. 3 Preliminaries: Dynamical Systems • A dynamical system is a system whose state varies with time. • State-space model: values of state variables change over time. • Example: x1(t), x2(t), ..., xN (t) are state variables that hold different values under independent variable t. This describes a system of order N , and x(t) is called the state vector. The dynamics of the system is expressed using ordinary differential equations: d dt xj(t) = Fj(xj(t)), j = 1, 2, ..., N. or, more conveniently d dt x(t) = F(x(t)). 4 Autonomous vs. Non-autonomous Dynamical Systems • Autonomous: F(·) does not explicitly depend on time. • Non-autonomous: F(·) explicitly depends on time. F as a Vector Field • Since dx dt can be seen as velocity, F(x) can be seen as a velocity vector field, or a vector field. • In a vector field, each point in space (x) is associated with one unique vector (direction and magnitude). In a scalar field, one point has one scalar value. 5 State Space x1 x2 t=0t=1 t=2 ... dx/dt • It is convenient to view the state-space equation dx dt = F(x) as describing the motion of a point in N-dimensional space (Euclidean or non-Euclidean). Note: t is continuous! • The points traversed over time is called the trajectory or the orbit. • The tangent vector shows the instantaneous velocity at the initial condition. 6 Phase Portrait and Vector Field • Red curves show the state (phase) portrait represented by trajectories from different initial points. • The blue arrows in the background shows the vector field. Source: http://www.math.ku.edu/˜byers/ode/b_cp_lab/pict.html 7 Conditions for the Solution of the State Space Equation • A unique solution to the state space equation exists only under certain conditions, which resticts the form of F(x). • For a solution to exist, it is sufficient for F(x) to be continuous in all of its arguments. • For uniqueness, it must meet the Lipschitz condition. • Lipschitz condition: – Let x and u be a pair of vectors in an open setM in a normal vector space. A vector function F(x) that satisfies: ‖F(x)− F(u)‖ ≤ K‖x− u‖ for some constant K , the above is said to be Lipschitz, and K is called the Lipschitz constant for F(x). – If ∂Fi/∂xj are finite everywhere, F(x) meet the Lipschitz condition. 8 Neurodynamical Models We will focus on state variables are continuous-valued, and those with dynamics expressed in differential equations or difference equations. Properties: • Large number of degree of freedom. • Nonlinearity • Dissipative (as opposed to conservative), i.e., open system. • Noise 17 Manipulation of Attractors as a Recurrent Nnet Paradigm • We can identify attractors with computational objects (associative memories, input-output mappers, etc.). • In order to do so, we must exercise control over the location of the attractors in the state space of the system. • A learning algorithm will manipulate the equations governing the dynamical behavior so that a desired location of attractors are set. • One good way to do this is to use the energy minimization paradigm (e.g., by Hopfield). 18 Hopfield Model • N units with full connection among every node (no self-feedback). • Given M input patterns, each having the same dimensionality as the network, can be memorized in attractors of the network. • Starting with an initial pattern, the dynamic will converge toward the attractor of the basin of attraction where the inital pattern was placed. 19 Discrete Hopfield Model • Based on McCulloch-Pitts model (neurons with +1 or -1 output). • Energy function is defined as E = − 1 2 N∑ i=1 N∑ j=1 wjixixj(i 6= j). • Network dynamics will evolve in the direction that minimizes E. • Implements a content-addressable memory. 20 Content-Addressable Memory • Map a set of patterns to be memorized ξµ onto fixed points xµ in the dynamical system realized by the recurrent network. • Encoding: Mapping from ξµ to xµ • Decoding: Reverse mapping from state space xµ to ξµ. 21 Hopfield Model: Storage • The learning is similar to Hebbian learning: wji = 1 N M∑ µ=1 ξµ,jξµ,i with wji = 0 if i = j. (Learning is one-shot.) • In matrix form the above becomes: W = 1 N M∑ µ=1 ξµξ T µ −MI • The resulting weight matrix W is symmetric: W = WT . 22 Hopfield Model: Activation (Retrieval) • Initialize the network with a probe pattern ξprobe. xj(0) = ξprobe,j . • Update output of each neuron (picking them by random) as xj(n + 1) = sgn ( N∑ i=1 wjixi(n) ) . until x reaches a fixed point. 23 Spurious States • The weight matrix W is symmetric, thus the eigenvalues of W are all real. • For large number of patters M , the matrix is degenerate, i.e., several eigenvectors can have the same eigenvalue. • These eigenvectors form a subspace, and when the associated eigenvalue is 0, it is called a null space. • This is due to M being smaller than the number of neurons N . • Hopfield network as content addressable memory: – Discrete Hopfield network acts as a vector projector (project probe vector onto subspace spanned by training patterns). – Underlying dynamics drive the network to converge to one of the corners of the unit hypercube. • Spurious states are those corners of the hypercube that do not belong to the training pattern set. 24 Storage Capacity of Hopfield Network • Given a probe equal to the stored pattern ξν , the activation of the jth neuron can be decomposed into the signal term and the noise term: vj = ∑N i=1 wjiξv,i = 1N ∑M µ=1 ξµ,j ∑N i=1 ξµ,iξν,i = ξν,j︸︷︷︸ signal + 1 N M∑ µ=1,µ 6=ν ξµ,j N∑ i=1 ξµ,iξν,i︸ ︷︷ ︸ noise • The signal-to-noise ratio is defined as ρ = variance of signal variance of noise = 1 (M − 1)/N ≈ N M • The reciprocal of ρ, called the load parameter is designated as α. According to Amit and others, this value needs to be less than 0.14 (critical value αc). 25 Storage Capacity of Hopfield Network (cont’d) • Given α = 0.14, the storage capacity becomes Mc = αcN = 0.14N when some error is allowed in the final patterns. • For almost error-free performance, the storage capacity become Mc = N 2 loge N • Thus, storage capacity of Hopfield network scales less than linearly with the size N of the network. • This is a major limitation of the Hopfield model. 26 Cohen-Grossberg Theorem • Cohen and Grossberg (1983) showed how to assess the stability of a certain class of neural networks: d dt uj = aj(uj) [ bj(uj)− N∑ i=1 cjiϕi(ui) ] , j = 1, 2, ..., N . • Neural network with the above dynamics admits a Lyapunov function defined as: E = 1 2 N∑ i=1 N∑ j=1 cjiϕi(ui)ϕj(uj)− N∑ j=1 ∫ uj 0 b−j(λ)ϕ′j(λ)dλ, where ϕ′(λ) = d dλ (ϕj(λ)). 27 Cohen-Grossberg Theorem (cont’d) • For the definition in the previous slide to be valid, the following conditions need to be met. – The synaptic weights are symmetric. – The function aj(uj) satisfies the condition for nonnegativity. – The nonlinear activation function ϕj(uj) needs to follow the monotonicity condition: ϕ′j(uj) = d duj ϕj(uj) ≥ 0. • With the above dE dt ≤ 0 ensuring global stability of the system. • Hopfield model can be seen as a special case of the Cohen-Grossberg theorem. 28