The thermal conductivity of the magnetically frustrated, ferroelectric YMnO3 exhibits an isotropic suppression in the cooperative paramagnetic state, followed by a sudden increase upon magnetic ordering. This unprecedented behavior without an associated static structural distortion probably originates from the strong dynamic coupling between acoustic phonons and low-energy spin fluctuations in geometrically frustrated magnets. The replacement of magnetic Ho for Y at the ferroelectrically active site results in an even larger effect, suggestive of the strong influence of multiferroicity.