In mathematics, and in particular the study of dynamical systems, the idea of *stable and unstable sets* or **stable and unstable manifolds** give a formal mathematical definition to the general notions embodied in the idea of an attractor or repellor. In the case of hyperbolic dynamics, the corresponding notion is that of the hyperbolic set.

The gravitational tidal forces acting on the rings of Saturn provide an easy-to-visualize physical example. The tidal forces flatten the ring into the equatorial plane, even as they stretch it out in the radial direction. Imagining the rings to be sand or gravel particles ("dust") in orbit around Saturn, the tidal forces are such that any perturbations that push particles above or below the equatorial plane results in that particle feeling a restoring force, pushing it back into the plane. Particles effectively oscillate in a harmonic well, damped by collisions. The stable direction is perpendicular to the ring. The unstable direction is along any radius, where forces stretch and pull particles apart. Two particles that start very near each other in phase space will experience radial forces causing them to diverge, radially. These forces have a positive Lyapunov exponent; the trajectories lie on a hyperbolic manifold, and the movement of particles is essentially chaotic, wandering through the rings. The center manifold is tangential to the rings, with particles experiencing neither compression nor stretching. This allows second-order gravitational forces to dominate, and so particles can be entrained by moons or moonlets in the rings, phase locking to them. The gravitational forces of the moons effectively provide a regularly repeating small kick, each time around the orbit, akin to a kicked rotor, such as found in a phase-locked loop.

The discrete-time motion of particles in the ring can be approximated by the Poincaré map. The map effectively provides the transfer matrix of the system. The eigenvector associated with the largest eigenvalue of the matrix is the Frobenius–Perron eigenvector, which is also the invariant measure, *i.e* the actual density of the particles in the ring. All other eigenvectors of the transfer matrix have smaller eigenvalues, and correspond to decaying modes.

The following provides a definition for the case of a system that is either an iterated function or has discrete-time dynamics. Similar notions apply for systems whose time evolution is given by a flow.

Let be a topological space, and a homeomorphism. If is a fixed point for , the **stable set of ** is defined by

and the **unstable set of ** is defined by

Here, denotes the inverse of the function , i.e. , where is the identity map on .

If is a periodic point of least period , then it is a fixed point of , and the stable and unstable sets of are defined by

and

Given a neighborhood of , the **local stable and unstable sets** of are defined by

and

If is metrizable, we can define the stable and unstable sets for any point by

and

where is a metric for . This definition clearly coincides with the previous one when is a periodic point.

Suppose now that is a compact smooth manifold, and is a diffeomorphism, . If is a hyperbolic periodic point, the stable manifold theorem assures that for some neighborhood of , the local stable and unstable sets are embedded disks, whose tangent spaces at are and (the stable and unstable spaces of ), respectively; moreover, they vary continuously (in a certain sense) in a neighborhood of in the topology of (the space of all diffeomorphisms from to itself). Finally, the stable and unstable sets are injectively immersed disks. This is why they are commonly called **stable and unstable manifolds**. This result is also valid for nonperiodic points, as long as they lie in some hyperbolic set (stable manifold theorem for hyperbolic sets).

If is a (finite-dimensional) vector space and an isomorphism, its stable and unstable sets are called stable space and unstable space, respectively.