With the relaxation of the containment measurements around the globe, monitoring the social distancing in crowded public spaces is of great importance to prevent a new massive wave of COVID-19 infections. Recent works in that matter have limited themselves by assessing social distancing in corridors up to small crowds by detecting each person individually, considering the full body in the image. In this work, we propose a new framework for monitoring the social-distance using end-to-end Deep Learning, to detect crowds violating social-distancing in wide areas, where important occlusions may be present. Our framework consists in the creation of new ground truth social distance labels, based on the ground truth density maps, and the proposal of two different solutions, a density-map-based and a segmentation-based, to detect crowds violating social-distancing constraints. We assess the results of both approaches by using the generated ground truth from the PET2009 and CityStreet datasets. We show that our framework performs well at providing the zones where people are not following the social-distance, even when heavily occluded or far away from the camera, compared to current detection and tracking approaches.