The World Health Organization has declared the ongoing outbreak of COVID-19, which is caused by a novel coronavirus SARS-CoV-2, as pandemic. There is currently a lack of knowledge about the antibody response elicited from SARS-CoV-2 infection. One major immunological question concerns antigenic differences between SARS-CoV-2 and SARS-CoV. We address this question by analyzing plasma from patients infected by SARS-CoV-2 or SARS-CoV, and from infected or immunized mice. Our results show that, while cross-reactivity in antibody binding to the spike protein is common, cross-neutralization of the live viruses may be rare, indicating the presence of non-neutralizing antibody response to conserved epitopes in the spike. Whether such low or non-neutralizing antibody response leads to antibody-dependent disease enhancement needs to be addressed in the future. Overall, this study not only addresses a fundamental question regarding antigenicity differences between SARS-CoV-2 and SARS-CoV, but also has implications for immunogen design and vaccine development.