Mathematical Constraints on F _ST: Biallelic Markers in Arbitrarily Many Populations

Abstract

${\mathit{F}}_{\mathrm{ST}}$ is one of the most widely used statistics in population genetics. Recent mathematical studies have identified constraints that challenge interpretations of $F_{\mathrm{ST}}$ as a measure with potential to range from 0 for genetically similar populations to 1 for divergent populations. We generalize results obtained for population pairs to arbitrarily many populations, characterizing the mathematical relationship between $F_{\mathrm{ST}},$ the frequency M of the more frequent allele at a polymorphic biallelic marker, and the number of subpopulations K. We show that for fixed K, $F_{\mathrm{ST}}$ has a peculiar constraint as a function of M, with a maximum of 1 only if $M=i/K,$ for integers i with $\lceil K/2\rceil \le i\le K-1.$ For fixed M, as K grows large, the range of $F_{\mathrm{ST}}$ becomes the closed or half-open unit interval. For fixed K, however, some $M<\left(K-1\right)/K$ always exists at which the upper bound on $F_{\mathrm{ST}}$ lies below $2\sqrt{2}-2\approx \mathrm{0.8284.}$ We use coalescent simulations to show that under weak migration, $F_{\mathrm{ST}}$ depends strongly on M when K is small, but not when K is large. Finally, examining data on human genetic variation, we use our results to explain the generally smaller $F_{\mathrm{ST}}$ values between pairs of continents relative to global $F_{\mathrm{ST}}$ values. We discuss implications for the interpretation and use of $F_{\mathrm{ST}}.$