We develop the quantum Kodaira-Spencer theory on the elliptic curve and establish the corresponding higher genus B-model. We show that the partition functions of the higher genus B-model on the elliptic curve are almost holomorphic modular forms, which can be identified with partition functions of descendant Gromov-Witten invariants on the mirror elliptic curve. This gives the first compact Calabi-Yau example where mirror symmetry is established at all genera.