Obesity is a worldwide epidemic, with major health and economic costs. Here we estimate heritability for body mass index (BMI) in 172,000 sibling pairs and 150,832 unrelated individuals and explore the contribution of genotype-covariate interaction effects at common SNP loci. We find evidence for genotype-age interaction (likelihood ratio test (LRT) = 73.58, degrees of freedom (df) = 1, P = 4.83 × 10-18), which contributed 8.1% (1.4% s.e.) to BMI variation. Across eight self-reported lifestyle factors, including diet and exercise, we find genotype-environment interaction only for smoking behavior (LRT = 19.70, P = 5.03 × 10-5 and LRT = 30.80, P = 1.42 × 10-8), which contributed 4.0% (0.8% s.e.) to BMI variation. Bayesian association analysis suggests that BMI is highly polygenic, with 75% of the SNP heritability attributable to loci that each explain <0.01% of the phenotypic variance. Our findings imply that substantially larger sample sizes across ages and lifestyles are required to understand the full genetic architecture of BMI.