Animals respond to sensory cues emitted by their conspecifics by initiating a repertoire of sex-specific social behaviors, such as mating, fighting, and parental care. These behavioral responses are thought to be largely driven by genetically pre-programmed circuits. However, they can also vary significantly according to the animal's previous social experience and physiological state, suggesting that the underlying circuits are shaped by adaptive modulatory changes. Modern tools of neuroscience have enabled the direct interrogation of circuits driving sex-specific behaviors and helped unravel key neural mechanisms underlying their function and modulation. In particular, recent work has shown how individual neuronal responses may combine with population activity to encode specific social information in both sexes. In this review, we summarize recent findings on how sex-specific sensory information is processed in social behavioral circuits.