Large environmental fluctuations often cause mass extinctions, extirpating species and transforming communities [1, 2]. While the effects on community structure are evident in the fossil record, demographic consequences for populations of individual species are harder to evaluate because fossils reveal relative, but not absolute, abundances. However, genomic analyses of living species that have survived a mass extinction event offer the potential for understanding the demographic effects of such environmental fluctuations on extant species. Here, we show how environmental variation since the Pliocene has shaped demographic changes in extant corals of the genus Orbicella, major extant reef builders in the Caribbean that today are endangered. We use genomic approaches to estimate previously unknown current and past population sizes over the last 3 million years. Populations of all three Orbicella declined around 2-1 million years ago, coincident with the extinction of at least 50% of Caribbean coral species. The estimated changes in population size are consistent across the three species despite their ecological differences. Subsequently, two shallow-water specialists expanded their population sizes at least 2-fold, over a time that overlaps with the disappearance of their sister competitor species O. nancyi (the organ-pipe Orbicella). Our study suggests that populations of Orbicella species are capable of rebounding from reductions in population size under suitable conditions and that the effective population size of modern corals provides rich standing genetic variation for corals to adapt to climate change. For conservation genetics, our study suggests the need to evaluate genetic variation under appropriate demographic models.