Despite significant efforts in the recent years, the physical processes responsible for the formation of passive galaxies through cosmic time remain unclear. The shape and evolution of the Stellar Mass Function (SMF) give an insight into these mechanisms. Taking advantage from the CANDELS and the deep Hubble Frontier Fields (HFF) programs, we estimated the SMF of total, star-forming and passive galaxies from z=0.25 to z=2.75 to unprecedented depth, and focus on the latter population. The density of passive galaxies underwent a significant evolution over the last 11 Gyr. They account for 60% of the total mass in the nearby Universe against ~20% observed at z~2.5. The inclusion of the HFF program allows us to detect, for the first time at z>1.5, the characteristic upturn in the SMF of passive galaxies at low masses, usually associated with environmental quenching. We observe two separate populations of passive galaxies evolving on different timescales: roughly half of the high mass systems were already quenched at high redshift, while low mass passive galaxies are gradually building-up over the redshift range probed. In the framework of environmental-quenching at low masses, we interpret this finding as evidence of an increasing role of the environment in the build-up of passive galaxies as a function of time. Finally, we compared our findings with a set of theoretical predictions. Despite good agreement in some redshift and mass intervals, none of the models are able to fully reproduce the observations. This calls for further investigation into the involved physical mechanisms, both theoretically and observationally, especially with the brand new JWST data.