All-optical switching of magnetic order presents a promising route toward faster and more energy efficient data storage. However, a realization in future devices is ultimately dependent on the maximum repetition rates of optically induced write/erase cycles. Here, we present two strategies to minimize the temporal separation of two consecutive femtosecond laser pulses to toggle the out-of-plane direction of the magnetization of ferrimagnetic rare-earth transition metal alloys. First, by systematically changing the heat transfer rates using either amorphous glass, crystalline silicon, or polycrystalline diamond substrates, we show that efficient cooling rates of the magnetic system present a prerequisite to accelerate the sequence of double pulse toggle switching. Second, we demonstrate that replacing the transition metal iron by cobalt leads to a significantly faster recovery of the magnetization after optical excitation allowing us to approach terahertz frequency of write/erase cycles with a minimum pulse-to-pulse separation of 7 ps.