Rare, damaging variants in a large number of genes are known to cause monogenic conditions, but have been shown to be incompletely penetrant or cause milder sub-clinical phenotypes in population cohorts. We used exome sequencing and linked health data from >450,000 individuals in UK Biobank to evaluate the penetrance of variants linked with a range of conditions, from severe childhood-onset developmental disorders (DD) through to monogenic diabetes and adult-onset cancer syndromes. We found a markedly lower penetrance in population cohorts compared with clinically-ascertained cohorts across all conditions studied. We further investigated features of the variants and modifiers that could explain this apparent reduction in penetrance, including variant location, rare variant burden, and polygenic contribution. We found that genes that cause DD through haploinsufficiency were more likely to have a non-uniform distribution of predicted loss-of-function variants than other genes (P<2.2x10-6), highlighting various potential molecular rescue mechanisms including translation reinitiation and alternative transcription. We also found that carrying multiple (2-5) rare damaging variants across 599 dominant DD genes had an additive adverse effect on numerous cognitive and socio-economic traits in UK Biobank, which could be partially counterbalanced by a higher educational attainment polygenic score (EA-PGS). Phenotypic deviators from EA-PGS could be partly explained by enrichment or depletion of rare DD variants. Amongst rare DD variant carriers, those with a DD-related clinical diagnosis had a substantially lower EA-PGS and more severe phenotype than those without. Taken together, our results suggest that the location of the variant, as well as the overall burden of rare and common variants, can modify the expressivity of a phenotype and influence whether an individual reaches the threshold for clinical disease.