Carbon ion therapy is a promising evolving modality in radiotherapy to treat tumors that are radioresistant against photon treatments. As carbon ions are more effective in normal and tumor tissue, the relative biological effectiveness (RBE) has to be calculated by bio-mathematical models and has to be considered in the dose prescription. This review (i) introduces the concept of the RBE and its most important determinants, (ii) describes the physical and biological causes of the increased RBE for carbon ions, (iii) summarizes available RBE measurements in vitro and in vivo, and (iv) describes the concepts of the clinically applied RBE models (mixed beam model, local effect model, and microdosimetric-kinetic model), and (v) the way they are introduced into clinical application as well as (vi) their status of experimental and clinical validation, and finally (vii) summarizes the current status of the use of the RBE concept in carbon ion therapy and points out clinically relevant conclusions as well as open questions. The RBE concept has proven to be a valuable concept for dose prescription in carbon ion radiotherapy, however, different centers use different RBE models and therefore care has to be taken when transferring results from one center to another. Experimental studies significantly improve the understanding of the dependencies and limitations of RBE models in clinical application. For the future, further studies investigating quantitatively the differential effects between normal tissues and tumors are needed accompanied by clinical studies on effectiveness and toxicity.