The 1995 Shallow Water Acoustics in a Random Medium (SWARM) experiment [Apel et al., IEEE J. Ocean. Eng. 22, 445-464 (1997)] was conducted off the New Jersey coast. The experiment featured two well-populated vertical receiving arrays, which permitted the measured acoustic field to be decomposed into its normal modes. The decomposition was repeated for successive transmissions allowing the amplitude of each mode to be tracked. The modal amplitudes were observed to decorrelate with time scales on the order of 100 s [Headrick et al., J. Acoust. Soc. Am. 107(1), 201-220 (2000)]. In the present work, a theoretical model is proposed to explain the observed decorrelation. Packets of intense internal waves are modeled as coherent structures moving along the acoustic propagation path without changing shape. The packets cause mode coupling and their motion results in a changing acoustic interference pattern. The model is consistent with the rapid decorrelation observed in SWARM. The model also predicts the observed partial recorrelation of the field at longer time scales. The model is first tested in simple continuous-wave simulations using canonical representations for the internal waves. More detailed time-domain simulations are presented mimicking the situation in SWARM. Modeling results are compared to experimental data.