The antiphoto aging property of Zanthoxylum rhetsa obtained from Pangkor Island, Malaysia, was evaluated. Solvent fractions of different polarity obtained from the methanolic extract of the bark material were initially tested for anticollagenase and antielastase activities. The ethyl acetate fraction showed bioactivity against the protease enzymes. Hence, it was subjected to further purification via column chromatography, to yield a major constituent, hesperidin. Subsequently, the ethyl acetate fraction and hesperidin were tested for their effects against UVB-induced cytotoxicity and expressions of inflammatory cytokines (IL-6, IL-1β, and TNF-α), NF-κB, and MMPs (MMP1, 3, and 9) in human dermal fibroblasts (HDF). Both fraction and pure compound prevented UVB-induced cytotoxicity in HDF cells, in a dose dependent manner. Moreover, the ethyl acetate fraction inhibited the increase of pro-inflammatory cytokines induced by UVB to a level similar to the control (without UV treatment). Additionally, the fraction significantly inhibited the expressions of NF-κB, MMP 1, MMP 3, and MMP 9 in HDF cells treated with UVB. Similar effects were observed with hesperidin. The results obtained suggested that the ethyl acetate fraction of Z. rhetsa and its bioactive constituent, hesperidin, have the potential to be used as active ingredients in sunscreen and antiphoto aging formulations.