This paper proposes a fiber Bragg grating sensing-based micro-vibration sensor. The optical fiber has been directly treated as an elastomer to design the micro-vibration sensor, which possesses two FBGs. The mass is fixed on the middle of the fiber, and the vertical vibration of the mass has been converted into the axial tension/compression of the fiber. The principle of the sensor has been introduced, and the experiment conclusions show that the sensor sensitivity is 2362 pm/g within the range of 200–1200 mm/s 2, which is consistent with theoretical analysis sensitivity of 2532.6 pm/g, and it shows an excellent linearity of 1.376%, while the resonant frequency of the sensor is 34 Hz, and the flat frequency range resides in the 0–22 Hz range. When used to measure micro-vibrations, its measured frequency relative error is less than 1.69% compared with the values acquired with a MEMS accelerometer, and the amplitude values of its measured vibration signal are consistent with the MEMS accelerometer under different excitation conditions too, so it can effectively realize the micro-vibration measurements.