Adaptive forecaster for despiking of time-series data. More...

#include <AdaptiveDespiker.h>

Public Member Functions
	AdaptiveDespiker ()
	Constructor.

void	setOutlierProbability (float val)

float	getOutlierProbability () const

void	setDiscLevelMultiplier (float val)

float	getDiscLevelMultiplier () const

float	getDiscLevel () const
	If a value is more than discrLevel * sigma away from the mean, then it is considered a spike, i.e.

float	despike (float u, bool *spike)
	Pass a value u, and return a forecasted value, along with a boolean indicating whether AdaptiveDespiker considers it a spike.

float	forecast () const
	Forecast a value based on the previous point and correlation and mean.

void	reset ()
	Reset the statistics.

Static Public Member Functions
static float	discrLevel (float prob)
	Compute a discrimination level from a Gaussan probability.

static float	adjustLevel (float corr)
	Adjust the discrimination level based on the current correlation.

Private Member Functions
void	initStatistics (float u)

void	incrementStatistics (float u)

void	updateStatistics (float u)

Static Private Member Functions
static bool	staticInit ()

static void	spline (float x, float y, int n, double yp1, double ypn, float *y2)
	Copied from "Numerical Recipies in C", the only changes being arrays are indexed from 0, due to personal taste, and doubles are passed instead of floats.

static double	splint (float xa, float ya, float *y2a, int n, double x)

Private Attributes
float	_prob
	Prob for detection of outlier in good Gaussian data.

float	_levelMultiplier
	Increase detection level with this factor.

float	_maxMissingFreq
	If more than _maxMissingFreq of the recent 10 data points are missing data points, don't substitute forecasted data or update forecast statistics.

float	_u1
	Last point.

double	_mean1
	Previous mean.

double	_mean2
	Current mean.

double	_var1
	Previous variance.

double	_var2
	Current variance.

double	_corr
	Correlation.

double	_initLevel
	Discrimination level.

double	_level

float	_missfreq
	Running ave of freq of missing data.

int	_msize
	Memory size.

size_t	_npts
	Number of points processed.

Static Private Attributes
static const size_t	STATISTICS_SIZE = 100

static const int	ADJUST_TABLE_SIZE = 100

static const int	LEN_ERFC_ARRAY = 140

static float	_adj [ADJUST_TABLE_SIZE][2]

static bool	staticInitDone = AdaptiveDespiker::staticInit()

Detailed Description

Adaptive forecaster for despiking of time-series data.

Algorithim taken from "A Statistical Data Screening Procedure" by Jorgen Hojstrup, Dept of Meteorology and Wind Energy, RISO National Laboratory, Denmark. [Published as: Meas. Sci. Technol (UK). v4 (1993) p. 153-157]

Constructor & Destructor Documentation

◆ AdaptiveDespiker()

AdaptiveDespiker::AdaptiveDespiker ( )

Constructor.

References setDiscLevelMultiplier(), and setOutlierProbability().

Member Function Documentation

◆ adjustLevel()

float AdaptiveDespiker::adjustLevel ( float corr )

static

Adjust the discrimination level based on the current correlation.

References _adj, ADJUST_TABLE_SIZE, and nidas::core::getSample().

Referenced by incrementStatistics(), and updateStatistics().

◆ despike()

float AdaptiveDespiker::despike	(	float	u,
		bool *	spike )

Pass a value u, and return a forecasted value, along with a boolean indicating whether AdaptiveDespiker considers it a spike.

The input value is added to the AdaptiveDespiker statistics for forecasting.

References _level, _maxMissingFreq, _missfreq, _npts, _var2, forecast(), nidas::core::getSample(), incrementStatistics(), initStatistics(), STATISTICS_SIZE, and updateStatistics().

◆ discrLevel()

float AdaptiveDespiker::discrLevel ( float prob )

static

Compute a discrimination level from a Gaussan probability.

If a value is more than discrLevel * sigma away from the mean, then it is considered a spike, i.e. it's probability exceeded the given value.

References nidas::core::getSample(), and LEN_ERFC_ARRAY.

Referenced by setDiscLevelMultiplier(), and setOutlierProbability().

◆ forecast()

float nidas::core::AdaptiveDespiker::forecast ( ) const

inline

Forecast a value based on the previous point and correlation and mean.

References _corr, _mean2, and _u1.

Referenced by despike().

◆ getDiscLevel()

float nidas::core::AdaptiveDespiker::getDiscLevel ( ) const

inline

If a value is more than discrLevel * sigma away from the mean, then it is considered a spike, i.e.

it's probability exceeded getOutlierProbability().

References _level.

Referenced by nidas::dynld::isff::Wind3D::getDiscLevel().

◆ getDiscLevelMultiplier()

float nidas::core::AdaptiveDespiker::getDiscLevelMultiplier ( ) const

inline

References _levelMultiplier.

Referenced by nidas::dynld::isff::Wind3D::getDiscLevelMultiplier().

◆ getOutlierProbability()

float nidas::core::AdaptiveDespiker::getOutlierProbability ( ) const

inline

References _prob.

Referenced by nidas::dynld::isff::Wind3D::getOutlierProbability().

◆ incrementStatistics()

void AdaptiveDespiker::incrementStatistics ( float u )

private

References _corr, _initLevel, _level, _mean1, _mean2, _missfreq, _msize, _npts, _u1, _var1, _var2, adjustLevel(), nidas::core::getSample(), and STATISTICS_SIZE.

Referenced by despike().

◆ initStatistics()

void AdaptiveDespiker::initStatistics ( float u )

private

References _corr, _mean1, _mean2, _missfreq, _npts, _u1, _var1, _var2, and nidas::core::getSample().

Referenced by despike().

◆ reset()

void AdaptiveDespiker::reset ( )

Reset the statistics.

References _initLevel, _level, _missfreq, and _npts.

◆ setDiscLevelMultiplier()

void AdaptiveDespiker::setDiscLevelMultiplier ( float val )

Parameters

val	Discrimination level fudge factor. The computation of discrimination level from the Gaussian probability is typically too small and one can multiply the resultant level by this factor. We've typically used 2.5 in practice.

References _initLevel, _level, _levelMultiplier, _prob, discrLevel(), and nidas::core::getSample().

Referenced by AdaptiveDespiker().

◆ setOutlierProbability()

void AdaptiveDespiker::setOutlierProbability ( float val )

Parameters

val	Probability of an outlier. This is converted to a discrimination level based on a Gaussian distribution.

See also

References _initLevel, _level, _levelMultiplier, _prob, discrLevel(), and nidas::core::getSample().

Referenced by AdaptiveDespiker().

◆ spline()

void AdaptiveDespiker::spline	(	float *	x,
		float *	y,
		int	n,
		double	yp1,
		double	ypn,
		float *	y2 )

staticprivate

Copied from "Numerical Recipies in C", the only changes being arrays are indexed from 0, due to personal taste, and doubles are passed instead of floats.

Given arrays a[0..n-1] and y[0..n-1] containing a tabulated function, i.e. yi = f(xi), with x1 < x2 < ... < xn-1 and given values yp1 and ypn for the first derivative of the interpolating function at points 1 and n-1 respectively, this routine returns an array y2[0..n-1] that contains the second derivatives of the interpolatimng function at the tabulated points xi. If yp1 and/or ypn are equal to 1.e30 or larger, the routine is signalled to set the corresponding boundary condition for a natural spline, with zero second derivative at that boundary.

References nidas::core::getSample().

Referenced by staticInit().

◆ splint()

double AdaptiveDespiker::splint	(	float *	xa,
		float *	ya,
		float *	y2a,
		int	n,
		double	x )

staticprivate

Exceptions

std::range_error

References nidas::core::getSample().

Referenced by staticInit().

◆ staticInit()

bool AdaptiveDespiker::staticInit ( )

staticprivate

References _adj, ADJUST_TABLE_SIZE, nidas::core::getSample(), spline(), and splint().

◆ updateStatistics()

void AdaptiveDespiker::updateStatistics ( float u )

private

References _corr, _initLevel, _level, _mean1, _mean2, _missfreq, _msize, _npts, _u1, _var1, _var2, adjustLevel(), and nidas::core::getSample().

Referenced by despike().

Member Data Documentation

◆ _adj

float AdaptiveDespiker::_adj

staticprivate

Referenced by adjustLevel(), and staticInit().

◆ _corr

double nidas::core::AdaptiveDespiker::_corr

private

Correlation.

Referenced by forecast(), incrementStatistics(), initStatistics(), and updateStatistics().

◆ _initLevel

double nidas::core::AdaptiveDespiker::_initLevel

private

Discrimination level.

Referenced by incrementStatistics(), reset(), setDiscLevelMultiplier(), setOutlierProbability(), and updateStatistics().

◆ _level

double nidas::core::AdaptiveDespiker::_level

private

Referenced by despike(), getDiscLevel(), incrementStatistics(), reset(), setDiscLevelMultiplier(), setOutlierProbability(), and updateStatistics().

◆ _levelMultiplier

float nidas::core::AdaptiveDespiker::_levelMultiplier

private

Increase detection level with this factor.

Referenced by getDiscLevelMultiplier(), setDiscLevelMultiplier(), and setOutlierProbability().

◆ _maxMissingFreq

float nidas::core::AdaptiveDespiker::_maxMissingFreq

private

If more than _maxMissingFreq of the recent 10 data points are missing data points, don't substitute forecasted data or update forecast statistics.

Referenced by despike().

◆ _mean1

double nidas::core::AdaptiveDespiker::_mean1

private

Previous mean.

Referenced by incrementStatistics(), initStatistics(), and updateStatistics().

◆ _mean2

double nidas::core::AdaptiveDespiker::_mean2

private

Current mean.

Referenced by forecast(), incrementStatistics(), initStatistics(), and updateStatistics().

◆ _missfreq

float nidas::core::AdaptiveDespiker::_missfreq

private

Running ave of freq of missing data.

Referenced by despike(), incrementStatistics(), initStatistics(), reset(), and updateStatistics().

◆ _msize

int nidas::core::AdaptiveDespiker::_msize

private

Memory size.

Referenced by incrementStatistics(), and updateStatistics().

◆ _npts

size_t nidas::core::AdaptiveDespiker::_npts

private

Number of points processed.

Referenced by despike(), incrementStatistics(), initStatistics(), reset(), and updateStatistics().

◆ _prob

float nidas::core::AdaptiveDespiker::_prob

private

Prob for detection of outlier in good Gaussian data.

Referenced by getOutlierProbability(), setDiscLevelMultiplier(), and setOutlierProbability().

◆ _u1

float nidas::core::AdaptiveDespiker::_u1

private

Last point.

Referenced by forecast(), incrementStatistics(), initStatistics(), and updateStatistics().

◆ _var1

double nidas::core::AdaptiveDespiker::_var1

private

Previous variance.

Referenced by incrementStatistics(), initStatistics(), and updateStatistics().

◆ _var2

double nidas::core::AdaptiveDespiker::_var2

private

Current variance.

Referenced by despike(), incrementStatistics(), initStatistics(), and updateStatistics().

◆ ADJUST_TABLE_SIZE

const int nidas::core::AdaptiveDespiker::ADJUST_TABLE_SIZE = 100

staticprivate

Referenced by adjustLevel(), and staticInit().

◆ LEN_ERFC_ARRAY

const int nidas::core::AdaptiveDespiker::LEN_ERFC_ARRAY = 140

staticprivate

Referenced by discrLevel().

◆ staticInitDone

bool AdaptiveDespiker::staticInitDone = AdaptiveDespiker::staticInit()

staticprivate

◆ STATISTICS_SIZE

const size_t nidas::core::AdaptiveDespiker::STATISTICS_SIZE = 100

staticprivate

Referenced by despike(), and incrementStatistics().

The documentation for this class was generated from the following files:

nidas/core/AdaptiveDespiker.h
nidas/core/AdaptiveDespiker.cc

Public Member Functions

Static Public Member Functions

Private Member Functions

Static Private Member Functions

Private Attributes

Static Private Attributes

Detailed Description

Constructor & Destructor Documentation

◆ AdaptiveDespiker()

Member Function Documentation

◆ adjustLevel()

◆ despike()

◆ discrLevel()

◆ forecast()

◆ getDiscLevel()

◆ getDiscLevelMultiplier()

◆ getOutlierProbability()

◆ incrementStatistics()

◆ initStatistics()

◆ reset()

◆ setDiscLevelMultiplier()

◆ setOutlierProbability()

◆ spline()

◆ splint()

◆ staticInit()

◆ updateStatistics()

Member Data Documentation

◆ _adj

◆ _corr

◆ _initLevel

◆ _level

◆ _levelMultiplier

◆ _maxMissingFreq

◆ _mean1

◆ _mean2

◆ _missfreq

◆ _msize

◆ _npts

◆ _prob

◆ _u1

◆ _var1

◆ _var2

◆ ADJUST_TABLE_SIZE

◆ LEN_ERFC_ARRAY

◆ staticInitDone

◆ STATISTICS_SIZE