After decades of research, the influence of prenatal testosterone on brain lateralization is still elusive, whereas the influence of pubertal testosterone on functional brain lateralization has not been investigated, although there is increasing evidence that testosterone affects the brain in puberty. We performed a longitudinal study, investigating the relationship between prenatal testosterone concentrations in amniotic fluid, pubertal testosterone concentrations in saliva, and brain lateralization (measured with functional Transcranial Doppler ultrasonography (fTCD)) of the Mental Rotation, Chimeric Faces and Word Generation tasks. Thirty boys and 30 girls participated in this study at the age of 15 years. For boys, we found a significant interaction effect between prenatal and pubertal testosterone on lateralization of Mental Rotation and Chimeric Faces. In the boys with low prenatal testosterone levels, pubertal testosterone was positively related to the strength of lateralization in the right hemisphere, while in the boys with high prenatal testosterone levels, pubertal testosterone was negatively related to the strength of lateralization. For Word Generation, pubertal testosterone was negatively related to the strength of lateralization in the left hemisphere in boys. For girls, we did not find any significant effects, possibly because their pubertal testosterone levels were in many cases below quantification limit. To conclude, prenatal and pubertal testosterone affect lateralization in a task-specific way. Our findings cannot be explained by simple models of prenatal testosterone affecting brain lateralization in a similar way for all tasks. We discuss alternative models involving age dependent effects of testosterone, with a role for androgen receptor distribution and efficiency.