Quantum anomalous Hall (QAH) insulators transport charge without resistance along topologically protected chiral one-dimensional edge states. Yet, in magnetic topological insulators (MTI) to date, topological protection is far from robust, with the zero-magnetic field QAH effect only realised at temperatures an order of magnitude below the N\'eel temperature TN, though small magnetic fields can stabilize QAH effect. Understanding why topological protection breaks down is therefore essential to realising QAH effect at higher temperatures. Here we use a scanning tunnelling microscope to directly map the size of the exchange gap (Eg,ex) and its spatial fluctuation in the QAH insulator 5-layer MnBi\(_2\)Te\(_4\). We observe long-range fluctuations of Eg,ex with values ranging between 0 (gapless) and 70 meV, uncorrelated to individual point defects. We directly image the breakdown of topological protection, showing that the chiral edge state, the hallmark signature of a QAH insulator, hybridizes with extended gapless metallic regions in the bulk. Finally, we unambiguously demonstrate that the gapless regions originate in magnetic disorder, by demonstrating that a small magnetic field restores Eg,ex in these regions, explaining the recovery of topological protection in magnetic fields. Our results indicate that overcoming magnetic disorder is key to exploiting the unique properties of QAH insulators.