sealed abstract class AnyColumnRegularMetricConfig extends RegularMetricConfig

Base class for regular metrics that works with any number of columns.

final case class ApproxDistinctValuesMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: SingleElemStringSeq, params: ApproxDistinctValuesParams = ApproxDistinctValuesParams(), metadata: Seq[SparkParam] = Seq.empty) extends SingleColumnRegularMetricConfig with Product with Serializable

Approximate distinct values regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric

final case class ApproxSequenceCompletenessMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: SingleElemStringSeq, params: ApproxSequenceCompletenessParams = ApproxSequenceCompletenessParams(), metadata: Seq[SparkParam] = Seq.empty) extends SingleColumnRegularMetricConfig with Product with Serializable

Sequence completeness regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric

final case class ArimaTrendMetricConfig(id: ID, description: Option[NonEmptyString], lookupMetric: ID, rule: TrendCheckRule, order: ThreeElemIntSeq, windowSize: NonEmptyString, windowOffset: Option[NonEmptyString], metadata: Seq[SparkParam] = Seq.empty) extends TrendMetricConfig with Product with Serializable

ARIMA trend metric configuration

id: Metric ID
description: Metric Description
lookupMetric: Metric which historical results to pull for statistic calculation
rule: Window calculation rule: by datetime or by number of records.
order: Sequence of ARIMA order parameters: [p, d, q]
windowSize: Size of the window for average metric value calculation (either a number of records or duration).
windowOffset: Optional window offset (either a number of records or duration)
metadata: List of metadata parameters specific to this metric

final case class AvgNumberMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: SingleElemStringSeq, metadata: Seq[SparkParam] = Seq.empty) extends SingleColumnRegularMetricConfig with Product with Serializable

Average number regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
metadata: List of metadata parameters specific to this metric

final case class AvgStringMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, metadata: Seq[SparkParam] = Seq.empty) extends AnyColumnRegularMetricConfig with Product with Serializable

Average string regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
metadata: List of metadata parameters specific to this metric

final case class AvgTrendMetricConfig(id: ID, description: Option[NonEmptyString], lookupMetric: ID, rule: TrendCheckRule, windowSize: NonEmptyString, windowOffset: Option[NonEmptyString], metadata: Seq[SparkParam] = Seq.empty) extends TrendMetricConfig with Product with Serializable

Average trend metric configuration

id: Metric ID
description: Metric Description
lookupMetric: Metric which historical results to pull for statistic calculation
rule: Window calculation rule: by datetime or by number of records.
windowSize: Size of the window for average metric value calculation (either a number of records or duration).
windowOffset: Optional window offset (either a number of records or duration)
metadata: List of metadata parameters specific to this metric

final case class CastedNumberMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = false) extends AnyColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

Casted number regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class CoMomentMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: DoubleElemStringSeq, metadata: Seq[SparkParam] = Seq.empty) extends DoubleColumnRegularMetricConfig with Product with Serializable

Co-moment regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
metadata: List of metadata parameters specific to this metric

final case class ColumnEqMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: MultiElemStringSeq, metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = false) extends MultiColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

Column equality metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class CompletenessMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, params: CompletenessParams = CompletenessParams(), metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = true) extends AnyColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

Completeness regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class ComposedMetricConfig(id: ID, description: Option[NonEmptyString], formula: NonEmptyString, metadata: Seq[SparkParam] = Seq.empty) extends MetricConfig with ComposedMetric with Product with Serializable

Composed metric configuration

id: Metric ID
description: Metric description
formula: Formula to calculate composed metric
metadata: List of metadata parameters specific to this metric

final case class CovarianceBesselMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: DoubleElemStringSeq, metadata: Seq[SparkParam] = Seq.empty) extends DoubleColumnRegularMetricConfig with Product with Serializable

Covariance Bessel regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
metadata: List of metadata parameters specific to this metric

final case class CovarianceMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: DoubleElemStringSeq, metadata: Seq[SparkParam] = Seq.empty) extends DoubleColumnRegularMetricConfig with Product with Serializable

Covariance regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
metadata: List of metadata parameters specific to this metric

final case class DayDistanceMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: DoubleElemStringSeq, params: DayDistanceParams, metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = false) extends DoubleColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

Day distance regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class DistinctValuesMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, metadata: Seq[SparkParam] = Seq.empty) extends AnyColumnRegularMetricConfig with Product with Serializable

Distinct values regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
metadata: List of metadata parameters specific to this metric

sealed abstract class DoubleColumnRegularMetricConfig extends RegularMetricConfig

Base class for regular metrics that works only with two columns.

final case class DuplicateValuesMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, metadata: Seq[SparkParam] = Seq.empty) extends AnyColumnRegularMetricConfig with Product with Serializable

Duplicate values regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
metadata: List of metadata parameters specific to this metric

final case class EmptinessMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, params: CompletenessParams = CompletenessParams(), metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = true) extends AnyColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

Emptiness regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class EmptyValuesMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = true) extends AnyColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

Empty values regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class FirstQuantileMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: SingleElemStringSeq, params: TDigestParams = TDigestParams(), metadata: Seq[SparkParam] = Seq.empty) extends SingleColumnRegularMetricConfig with Product with Serializable

TDigest First quantile regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric

final case class FirstQuartileTrendMetricConfig(id: ID, description: Option[NonEmptyString], lookupMetric: ID, rule: TrendCheckRule, windowSize: NonEmptyString, windowOffset: Option[NonEmptyString], metadata: Seq[SparkParam] = Seq.empty) extends TrendMetricConfig with Product with Serializable

First Quartile trend metric configuration

id: Metric ID
description: Metric Description
lookupMetric: Metric which historical results to pull for statistic calculation
rule: Window calculation rule: by datetime or by number of records.
windowSize: Size of the window for average metric value calculation (either a number of records or duration).
windowOffset: Optional window offset (either a number of records or duration)
metadata: List of metadata parameters specific to this metric

final case class FormattedDateMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, params: FormattedDateParams = FormattedDateParams(), metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = false) extends AnyColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

Formatted date regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class FormattedNumberMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, params: FormattedNumberParams, metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = false) extends AnyColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

Formatted number regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class GetPercentileMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: SingleElemStringSeq, params: TDigestGeqPercentileParams, metadata: Seq[SparkParam] = Seq.empty) extends SingleColumnRegularMetricConfig with Product with Serializable

TDigest Get percentile regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric

final case class GetQuantileMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: SingleElemStringSeq, params: TDigestGeqQuantileParams, metadata: Seq[SparkParam] = Seq.empty) extends SingleColumnRegularMetricConfig with Product with Serializable

TDigest Get quantile regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric

final case class LevenshteinDistanceMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: DoubleElemStringSeq, params: LevenshteinDistanceParams, metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = false) extends DoubleColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

Levenshtein distance regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class LinregTrendMetricConfig(id: ID, description: Option[NonEmptyString], lookupMetric: ID, rule: TrendCheckRule, windowSize: NonEmptyString, windowOffset: Option[NonEmptyString], metadata: Seq[SparkParam] = Seq.empty) extends TrendMetricConfig with Product with Serializable

Linear regression trend metric configuration

id: Metric ID
description: Metric Description
lookupMetric: Metric which historical results to pull for statistic calculation
rule: Window calculation rule: by datetime or by number of records.
windowSize: Size of the window for average metric value calculation (either a number of records or duration).
windowOffset: Optional window offset (either a number of records or duration)
metadata: List of metadata parameters specific to this metric

final case class MaxNumberMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, metadata: Seq[SparkParam] = Seq.empty) extends AnyColumnRegularMetricConfig with Product with Serializable

Max number regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
metadata: List of metadata parameters specific to this metric

final case class MaxStringMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, metadata: Seq[SparkParam] = Seq.empty) extends AnyColumnRegularMetricConfig with Product with Serializable

Max string regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
metadata: List of metadata parameters specific to this metric

final case class MaxTrendMetricConfig(id: ID, description: Option[NonEmptyString], lookupMetric: ID, rule: TrendCheckRule, windowSize: NonEmptyString, windowOffset: Option[NonEmptyString], metadata: Seq[SparkParam] = Seq.empty) extends TrendMetricConfig with Product with Serializable

Max trend metric configuration

id: Metric ID
description: Metric Description
lookupMetric: Metric which historical results to pull for statistic calculation
rule: Window calculation rule: by datetime or by number of records.
windowSize: Size of the window for average metric value calculation (either a number of records or duration).
windowOffset: Optional window offset (either a number of records or duration)
metadata: List of metadata parameters specific to this metric

final case class MedianTrendMetricConfig(id: ID, description: Option[NonEmptyString], lookupMetric: ID, rule: TrendCheckRule, windowSize: NonEmptyString, windowOffset: Option[NonEmptyString], metadata: Seq[SparkParam] = Seq.empty) extends TrendMetricConfig with Product with Serializable

Median trend metric configuration

id: Metric ID
description: Metric Description
lookupMetric: Metric which historical results to pull for statistic calculation
rule: Window calculation rule: by datetime or by number of records.
windowSize: Size of the window for average metric value calculation (either a number of records or duration).
windowOffset: Optional window offset (either a number of records or duration)
metadata: List of metadata parameters specific to this metric

final case class MedianValueMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: SingleElemStringSeq, params: TDigestParams = TDigestParams(), metadata: Seq[SparkParam] = Seq.empty) extends SingleColumnRegularMetricConfig with Product with Serializable

TDigest Median value regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric

sealed abstract class MetricConfig extends JobConfigEntity

Base class for all metrics configurations.

Base class for all metrics configurations. All metrics are described as DQ entities.

final case class MetricsConfig(regular: Option[RegularMetricsConfig], composed: Seq[ComposedMetricConfig] = Seq.empty, trend: Seq[TrendMetricConfig] = Seq.empty) extends Product with Serializable

Data Quality job configuration section describing all metrics

regular: Regular metrics of all subtypes
composed: Sequence of composed metrics
trend: Sequence of trend metrics

final case class MinNumberMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, metadata: Seq[SparkParam] = Seq.empty) extends AnyColumnRegularMetricConfig with Product with Serializable

Min number regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
metadata: List of metadata parameters specific to this metric

final case class MinStringMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, metadata: Seq[SparkParam] = Seq.empty) extends AnyColumnRegularMetricConfig with Product with Serializable

Min string regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
metadata: List of metadata parameters specific to this metric

final case class MinTrendMetricConfig(id: ID, description: Option[NonEmptyString], lookupMetric: ID, rule: TrendCheckRule, windowSize: NonEmptyString, windowOffset: Option[NonEmptyString], metadata: Seq[SparkParam] = Seq.empty) extends TrendMetricConfig with Product with Serializable

Min trend metric configuration

id: Metric ID
description: Metric Description
lookupMetric: Metric which historical results to pull for statistic calculation
rule: Window calculation rule: by datetime or by number of records.
windowSize: Size of the window for average metric value calculation (either a number of records or duration).
windowOffset: Optional window offset (either a number of records or duration)
metadata: List of metadata parameters specific to this metric

sealed abstract class MultiColumnRegularMetricConfig extends RegularMetricConfig

Base class for regular metrics that works with at least two columns.

final case class NullValuesMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = true) extends AnyColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

Null values regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class NumberBetweenMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, params: NumberIntervalParams, metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = false) extends AnyColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

Number between regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class NumberGreaterThanMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, params: NumberCompareParams, metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = false) extends AnyColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

Number greater than regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class NumberInDomainMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, params: NumberDomainParams, metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = false) extends AnyColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

Number in domain regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class NumberLessThanMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, params: NumberCompareParams, metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = false) extends AnyColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

Number less than regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class NumberNotBetweenMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, params: NumberIntervalParams, metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = false) extends AnyColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

Number not between regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class NumberOutDomainMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, params: NumberDomainParams, metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = false) extends AnyColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

Number out domain regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class NumberValuesMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, params: NumberValuesParams, metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = false) extends AnyColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

Number values regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class QuantileTrendMetricConfig(id: ID, description: Option[NonEmptyString], lookupMetric: ID, quantile: PercentileDouble, rule: TrendCheckRule, windowSize: NonEmptyString, windowOffset: Option[NonEmptyString], metadata: Seq[SparkParam] = Seq.empty) extends TrendMetricConfig with Product with Serializable

Quantile trend metric configuration

id: Metric ID
description: Metric Description
lookupMetric: Metric which historical results to pull for statistic calculation
quantile: Quantile to compute over historical metric results (must be a number in range [0, 1]).
rule: Window calculation rule: by datetime or by number of records.
windowSize: Size of the window for average metric value calculation (either a number of records or duration).
windowOffset: Optional window offset (either a number of records or duration)
metadata: List of metadata parameters specific to this metric

final case class RegexMatchMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, params: RegexParams, metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = false) extends AnyColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

Regex match regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class RegexMismatchMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, params: RegexParams, metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = false) extends AnyColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

Regex mismatch regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

sealed abstract class RegularMetricConfig extends MetricConfig with RegularMetric

Base class for all regular metric configurations (except row count metric).

Base class for all regular metric configurations (except row count metric). All regular metrics must have a reference to source ID over which the metric is being calculated. In addition, regular metrics must have non-empty sequence of columns over which the metric is being calculated. Metric error collection logic might be reversed provided with corresponding boolean flag set to true.

Data Quality job configuration section describing regular metrics

rowCount: Sequence of rowCount metrics
distinctValues: Sequence of distinctValues metrics
approximateDistinctValues: Sequence of approximateDistinctValues metrics
nullValues: Sequence of nullValues metrics
emptyValues: Sequence of emptyValues metrics
duplicateValues: Sequence of duplicateValues metrics
completeness: Sequence of completeness metrics
minString: Sequence of minString metrics
maxString: Sequence of maxString metrics
avgString: Sequence of avgString metrics
stringLength: Sequence of stringLength metrics
stringInDomain: Sequence of stringInDomain metrics
stringOutDomain: Sequence of stringOutDomain metrics
stringValues: Sequence of stringValues metrics
regexMatch: Sequence of regexMatch metrics
regexMismatch: Sequence of regexMismatch metrics
formattedDate: Sequence of formattedDate metrics
formattedNumber: Sequence of formattedNumber metrics
minNumber: Sequence of minNumber metrics
maxNumber: Sequence of maxNumber metrics
sumNumber: Sequence of sumNumber metrics
avgNumber: Sequence of avgNumber metrics
stdNumber: Sequence of stdNumber metrics
castedNumber: Sequence of castedNumber metrics
numberInDomain: Sequence of numberInDomain metrics
numberOutDomain: Sequence of numberOutDomain metrics
numberLessThan: Sequence of numberLessThan metrics
numberGreaterThan: Sequence of numberGreaterThan metrics
numberBetween: Sequence of numberBetween metrics
numberNotBetween: Sequence of numberNotBetween metrics
numberValues: Sequence of numberValues metrics
medianValue: Sequence of medianValue metrics
firstQuantile: Sequence of firstQuantile metrics
thirdQuantile: Sequence of thirdQuantile metrics
getPercentile: Sequence of getPercentile metrics
getQuantile: Sequence of getQuantile metrics
columnEq: Sequence of columnEq metrics
dayDistance: Sequence of dayDistance metrics
levenshteinDistance: Sequence of levenshteinDistance metrics
coMoment: Sequence of coMoment metrics
covariance: Sequence of covariance metrics
covarianceBessel: Sequence of covarianceBessel metrics
topN: Sequence of topN metrics

final case class RowCountMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, metadata: Seq[SparkParam] = Seq.empty) extends RegularMetricConfig with Product with Serializable

Row count metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
metadata: List of metadata parameters specific to this metric

final case class SequenceCompletenessMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: SingleElemStringSeq, params: SequenceCompletenessParams = SequenceCompletenessParams(), metadata: Seq[SparkParam] = Seq.empty) extends SingleColumnRegularMetricConfig with Product with Serializable

Sequence completeness regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric

sealed abstract class SingleColumnRegularMetricConfig extends RegularMetricConfig

Base class for regular metrics that works only with single column.

final case class StdNumberMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: SingleElemStringSeq, metadata: Seq[SparkParam] = Seq.empty) extends SingleColumnRegularMetricConfig with Product with Serializable

Standard deviation number regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
metadata: List of metadata parameters specific to this metric

final case class StdTrendMetricConfig(id: ID, description: Option[NonEmptyString], lookupMetric: ID, rule: TrendCheckRule, windowSize: NonEmptyString, windowOffset: Option[NonEmptyString], metadata: Seq[SparkParam] = Seq.empty) extends TrendMetricConfig with Product with Serializable

Standard deviation trend metric configuration

id: Metric ID
description: Metric Description
lookupMetric: Metric which historical results to pull for statistic calculation
rule: Window calculation rule: by datetime or by number of records.
windowSize: Size of the window for average metric value calculation (either a number of records or duration).
windowOffset: Optional window offset (either a number of records or duration)
metadata: List of metadata parameters specific to this metric

final case class StringInDomainMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, params: StringDomainParams, metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = false) extends AnyColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

String in domain regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class StringLengthMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, params: StringLengthParams, metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = false) extends AnyColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

String length regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class StringOutDomainMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, params: StringDomainParams, metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = false) extends AnyColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

String out domain regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class StringValuesMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, params: StringValuesParams, metadata: Seq[SparkParam] = Seq.empty, reversed: Boolean = false) extends AnyColumnRegularMetricConfig with ReversibleMetric with Product with Serializable

String values regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric
reversed: Boolean flag indicating whether error collection logic should be reversed for this metric

final case class SumNumberMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: NonEmptyStringSeq, metadata: Seq[SparkParam] = Seq.empty) extends AnyColumnRegularMetricConfig with Product with Serializable

Sum number regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
metadata: List of metadata parameters specific to this metric

final case class SumTrendMetricConfig(id: ID, description: Option[NonEmptyString], lookupMetric: ID, rule: TrendCheckRule, windowSize: NonEmptyString, windowOffset: Option[NonEmptyString], metadata: Seq[SparkParam] = Seq.empty) extends TrendMetricConfig with Product with Serializable

Sum trend metric configuration

id: Metric ID
description: Metric Description
lookupMetric: Metric which historical results to pull for statistic calculation
rule: Window calculation rule: by datetime or by number of records.
windowSize: Size of the window for average metric value calculation (either a number of records or duration).
windowOffset: Optional window offset (either a number of records or duration)
metadata: List of metadata parameters specific to this metric

final case class ThirdQuantileMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: SingleElemStringSeq, params: TDigestParams = TDigestParams(), metadata: Seq[SparkParam] = Seq.empty) extends SingleColumnRegularMetricConfig with Product with Serializable

TDigest Third quantile regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric

final case class ThirdQuartileTrendMetricConfig(id: ID, description: Option[NonEmptyString], lookupMetric: ID, rule: TrendCheckRule, windowSize: NonEmptyString, windowOffset: Option[NonEmptyString], metadata: Seq[SparkParam] = Seq.empty) extends TrendMetricConfig with Product with Serializable

Third Quartile trend metric configuration

id: Metric ID
description: Metric Description
lookupMetric: Metric which historical results to pull for statistic calculation
rule: Window calculation rule: by datetime or by number of records.
windowSize: Size of the window for average metric value calculation (either a number of records or duration).
windowOffset: Optional window offset (either a number of records or duration)
metadata: List of metadata parameters specific to this metric

final case class TopNMetricConfig(id: ID, description: Option[NonEmptyString], source: NonEmptyString, columns: SingleElemStringSeq, params: TopNParams = TopNParams(), metadata: Seq[SparkParam] = Seq.empty) extends SingleColumnRegularMetricConfig with Product with Serializable

TopN regular metric configuration

id: Metric ID
description: Metric description
source: Source ID over which metric is being calculated
columns: Sequence of columns which are used for metric calculation
params: Metric parameters
metadata: List of metadata parameters specific to this metric

sealed abstract class TrendMetricConfig extends MetricConfig with TrendMetric

Base class for trend metrics that computes their values based on historical metric results.

Packages

Metrics

object Metrics

Type Members

Value Members

Inherited from AnyRef

Inherited from Any

Ungrouped