From an on-going survey of the Galactic bulge, we have discovered a number of compact, steep spectrum radio sources. In this study, we have carried out more detailed observations for two of these sources, located 43 arcmin and 12${_{.}^{\circ}}$7 from the Galactic Centre. Both sources have a very steep spectrum (α ≃ − 3) and are compact, with upper limits on the angular size of 1–2 arcsec. Their flux densities appear to be relatively steady on time-scales of years, months, and hours, with no indications of rapid variability or transient behaviour. We detect significant circularly polarized emission from both sources, but only weak or upper limits on linear polarization. Neither source has a counterpart at other wavelengths and deep, high-frequency searches fail to find pulsations. We compare their source properties with other known compact, non-thermal source populations in the bulge (e.g. X-ray binaries, magnetars, the Burper, and cataclysmic variables). Our existing data support the hypothesis that they are scatter broadened millisecond or recycled pulsars, either at the bulge or along the line of sight. We also consider the possibility that they may be a new population of Galactic radio sources which share similar properties as pulsars but lack pulsations; a hypothesis that can be tested by future large-scale synoptic surveys.