In the fields of nonlinear optics and fluid dynamics, modulational instability or sideband instability is a phenomenon whereby deviations from a periodic waveform are reinforced by nonlinearity, leading to the generation of spectral-sidebands and the eventual breakup of the waveform into a train of pulses.
It is widely believed that the phenomenon was first discovered − and modeled − for periodic surface gravity waves (Stokes waves) on deep water by T. Brooke Benjamin and Jim E. Feir, in 1967. Therefore, it is also known as the Benjamin−Feir instability. However, spatial modulation instability of high-power lasers in organic solvents was observed by Russian scientists N. F. Piliptetskii and A. R. Rustamov in 1965, and the mathematical derivation of modulation instability was published by V. I. Bespalov and V. I. Talanov in 1966. Modulation instability is a possible mechanism for the generation of rogue waves.
Modulation instability only happens under certain circumstances. The most important condition is anomalous group velocity dispersion, whereby pulses with shorter wavelengths travel with higher group velocity than pulses with longer wavelength. (This condition assumes a focussing Kerr nonlinearity, whereby refractive index increases with optical intensity.)
The instability is strongly dependent on the frequency of the perturbation. At certain frequencies, a perturbation will have little effect, whilst at other frequencies, a perturbation will grow exponentially. The overall gain spectrum can be derived analytically, as is shown below. Random perturbations will generally contain a broad range of frequency components, and so will cause the generation of spectral sidebands which reflect the underlying gain spectrum.
The tendency of a perturbing signal to grow makes modulation instability a form of amplification. By tuning an input signal to a peak of the gain spectrum, it is possible to create an optical amplifier.
The gain spectrum can be derived  by starting with a model of modulation instability based upon the nonlinear Schrödinger equation
which describes the evolution of a complex-valued slowly varying envelope with time and distance of propagation . The imaginary unit satisfies The model includes group velocity dispersion described by the parameter , and Kerr nonlinearity with magnitude A periodic waveform of constant power is assumed. This is given by the solution
where the oscillatory phase factor accounts for the difference between the linear refractive index, and the modified refractive index, as raised by the Kerr effect. The beginning of instability can be investigated by perturbing this solution as
where is the perturbation term (which, for mathematical convenience, has been multiplied by the same phase factor as ). Substituting this back into the nonlinear Schrödinger equation gives a perturbation equation of the form
where the perturbation has been assumed to be small, such that The complex conjugate of is denoted as Instability can now be discovered by searching for solutions of the perturbation equation which grow exponentially. This can be done using a trial function of the general form
where and are the wavenumber and (real-valued) angular frequency of a perturbation, and and are constants. The nonlinear Schrödinger equation is constructed by removing the carrier wave of the light being modelled, and so the frequency of the light being perturbed is formally zero. Therefore, and don't represent absolute frequencies and wavenumbers, but the difference between these and those of the initial beam of light. It can be shown that the trial function is valid, provided and subject to the condition
This dispersion relation is vitally dependent on the sign of the term within the square root, as if positive, the wavenumber will be real, corresponding to mere oscillations around the unperturbed solution, whilst if negative, the wavenumber will become imaginary, corresponding to exponential growth and thus instability. Therefore, instability will occur when
This condition describes the requirement for anomalous dispersion (such that is negative). The gain spectrum can be described by defining a gain parameter as so that the power of a perturbing signal grows with distance as The gain is therefore given by
where as noted above, is the difference between the frequency of the perturbation and the frequency of the initial light. The growth rate is maximum for
Modulation instability of optical fields has been observed in photo-chemical systems, namely, photopolymerizable medium. Modulation instability occurs owing to inherent optical nonlinearity of the systems due to photoreaction-induced changes in the refractive index. Modulation instability of spatially and temporally incoherent light is possible owing to the non-instantaneous response of photoreactive systems, which consequently responds to the time-average intensity of light, in which the femto-second fluctuations cancel out.