A review of entropy measures for uncertainty quantification of stochastic processes

Abstract

Entropy is originally introduced to explain the inclination of intensity of heat, pressure, and density to gradually disappear over time. Based on the concept of entropy, the Second Law of Thermodynamics, which states that the entropy of an isolated system is likely to increase until it attains its equilibrium state, is developed. More recently, the implication of entropy has been extended beyond the field of thermodynamics, and entropy has been applied in many subjects with probabilistic nature. The concept of entropy is applicable and useful in characterizing the behavior of stochastic processes since it represents the uncertainty, ambiguity, and disorder of the processes without being restricted to the forms of the theoretical probability distributions. In order to measure and quantify the entropy, the existing probability of every event in the stochastic process must be determined. Different entropy measures have been studied and presented including Shannon entropy, Renyi entropy, Tsallis entropy, Sample entropy, Permutation entropy, Approximate entropy, and Transfer entropy. This review surveys the general formulations of the uncertainty quantification based on entropy as well as their various applications. The results of the existing studies show that entropy measures are powerful predictors for stochastic processes with uncertainties. In addition, we examine the stochastic process of lithium-ion battery capacity data and attempt to determine the relation between the changes in battery capacity over different cycles and two entropy measures: Sample entropy and Approximate entropy.