Animal production exerts significant demands on land, water and food resources and is one of the most extensive means by which humans modify natural systems. Demand for animal source foods has more than tripled over the past 50years due to population growth and dietary change. As a result, the livestock sector has transitioned towards intensive and concentrated production systems. Typically, studies have divided types of animal production into intensive, mixed and grazing production systems. However, because a large percentage of animal production originates from mixed systems, dividing by such production types can make it difficult to quantify competition for crop production between direct human consumption and use as feed. To this end we employ a calorie-based approach to determine which animal calories were 'free' - in that they did not compete with human consumption for crop use - and consider to what extent alternative scenarios could have reduced this competition between food and feed. We find that growth in non-feed animal systems has only been able to keep pace with population growth and that feed-fed production has necessarily met increases in human dietary demand for animal products. Through solutions such as moderating diets for animal calories, choosing less resource-demanding animal products and maintaining the relative contribution of non-feed systems, between 1.3 and 3.6 billion fewer people would be in competition with feed for crop use. We also estimate that the feed crop calories required to support consumer waste of animal calories could feed an additional 235 million people. With human demand for animal products expected to continue increasing in the coming decades, the findings here provide insights into potential solutions and what the magnitude of their effect may be and suggest that there exist real opportunities for humankind to substantially reduce competition for crop use.