Since the early 1980s, several investigations have focused on developing a vaccine against Mycobacterium avium subspecies paratuberculosis (MAP), the causative agent of Johne's disease in cattle and sheep. These studies used whole-cell inactivated vaccines that have proven useful in limiting disease progression, but have not prevented infection. In contrast, modified live vaccines that invoke a Th1 type immune response, may improve protection against infection. Spurred by recent advances in the ability to create defined knockouts in MAP, several independent laboratories have developed modified live vaccine candidates by transpositional mutation of virulence and metabolic genes in MAP. In order to accelerate the process of identification and comparative evaluation of the most promising modified live MAP vaccine candidates, members of a multi-institutional USDA-funded research consortium, the Johne's disease integrated program (JDIP), met to establish a standardized testing platform using agreed upon protocols. A total of 22 candidates vaccine strains developed in five independent laboratories in the United States and New Zealand voluntarily entered into a double blind stage gated trial pipeline. In Phase I, the survival characteristics of each candidate were determined in bovine macrophages. Attenuated strains moved to Phase II, where tissue colonization of C57/BL6 mice were evaluated in a challenge model. In Phase III, five promising candidates from Phase I and II were evaluated for their ability to reduce fecal shedding, tissue colonization and pathology in a baby goat challenge model. Formation of a multi-institutional consortium for vaccine strain evaluation has revealed insights for the implementation of vaccine trials for Johne's disease and other animal pathogens. We conclude by suggesting the best way forward based on this 3-phase trial experience and challenge the rationale for use of a macrophage-to-mouse-to native host pipeline for MAP vaccine development.