Fluorescent speckle microscopy (FSM) is becoming the technique of choice for analyzing in vivo the dynamics of polymer assemblies, such as the cytoskeleton. The massive amount of data produced by this method calls for computational approaches to recover the quantities of interest; namely, the polymerization and depolymerization activities and the motions undergone by the cytoskeleton over time. Attempts toward this goal have been hampered by the limited signal-to-noise ratio of typical FSM data, by the constant appearance and disappearance of speckles due to polymer turnover, and by the presence of flow singularities characteristic of many cytoskeletal polymer assemblies. To deal with these problems, we present a particle-based method for tracking fluorescent speckles in time-lapse FSM image series, based on ideas from operational research and graph theory. Our software delivers the displacements of thousands of speckles between consecutive frames, taking into account that speckles may appear and disappear. In this article we exploit this information to recover the speckle flow field. First, the software is tested on synthetic data to validate our methods. We then apply it to mapping filamentous actin retrograde flow at the front edge of migrating newt lung epithelial cells. Our results confirm findings from previously published kymograph analyses and manual tracking of such FSM data and illustrate the power of automated tracking for generating complete and quantitative flow measurements. Third, we analyze microtubule poleward flux in mitotic metaphase spindles assembled in Xenopus egg extracts, bringing new insight into the dynamics of microtubule assemblies in this system.