Septins assemble into filaments and higher-order structures that act as scaffolds for diverse cell functions including cytokinesis, cell polarity, and membrane remodeling. Despite their conserved role in cell organization, little is known about how septin filaments elongate and are knitted together into higher-order assemblies. Using fluorescence correlation spectroscopy, we determined that cytosolic septins are in small complexes, suggesting that septin filaments are not formed in the cytosol. When the plasma membrane of live cells is monitored by total internal reflection fluorescence microscopy, we see that septin complexes of variable size diffuse in two dimensions. Diffusing septin complexes collide and make end-on associations to form elongated filaments and higher-order structures, an assembly process we call annealing. Septin assembly by annealing can be reconstituted in vitro on supported lipid bilayers with purified septin complexes. Using the reconstitution assay, we show that septin filaments are highly flexible, grow only from free filament ends, and do not exchange subunits in the middle of filaments. This work shows that annealing is a previously unidentified intrinsic property of septins in the presence of membranes and demonstrates that cells exploit this mechanism to build large septin assemblies.