Type Members

1.  sealed trait AverageBoundCheckConfig extends AnyRef

   Trait for average bound checks.

   Trait for average bound checks. These type of checks are characterized by presence of following parameters:

   • rule: Window calculation rule: by datetime or by number of records.
   • windowSize: Size of the window for average metric value calculation (either a number of records or duration).
   • windowOffset: Optional window offset (either a number of records or duration)
2.  final case class AverageBoundFullCheckConfig(id: ID, description: Option[NonEmptyString], metric: NonEmptyString, rule: TrendCheckRule, windowSize: NonEmptyString, windowOffset: Option[NonEmptyString], threshold: PercentileDouble, isCritical: Boolean = false, metadata: Seq[SparkParam] = Seq.empty) extends TrendCheckConfig with AverageBoundCheckConfig with Product with Serializable

   'Average bound full' check configuration

   'Average bound full' check configuration

   id

   Check ID

   description

   Check description

   metric

   Reference to a metric ID over which the check is performed

   rule

   Window calculation rule: by datetime or by number of records.

   windowSize

   Size of the window for average metric value calculation (either a number of records or duration).

   windowOffset

   Optional window offset (either a number of records or duration)

   threshold

   Threshold value to calculate upper and lower bounds to compare with.

   isCritical

   Flag if check is critical

   metadata

   List of metadata parameters specific to this check

3.  final case class AverageBoundLowerCheckConfig(id: ID, description: Option[NonEmptyString], metric: NonEmptyString, rule: TrendCheckRule, windowSize: NonEmptyString, windowOffset: Option[NonEmptyString], threshold: PercentileDouble, isCritical: Boolean = false, metadata: Seq[SparkParam] = Seq.empty) extends TrendCheckConfig with AverageBoundCheckConfig with Product with Serializable

   'Average bound lower' check configuration

   'Average bound lower' check configuration

   id

   Check ID

   description

   Check description

   metric

   Reference to a metric ID over which the check is performed

   rule

   Window calculation rule: by datetime or by number of records.

   windowSize

   Size of the window for average metric value calculation (either a number of records or duration).

   windowOffset

   Optional window offset (either a number of records or duration)

   threshold

   Threshold value to calculate lower bound to compare with.

   isCritical

   Flag if check is critical

   metadata

   List of metadata parameters specific to this check

4.  final case class AverageBoundRangeCheckConfig(id: ID, description: Option[NonEmptyString], metric: NonEmptyString, rule: TrendCheckRule, windowSize: NonEmptyString, windowOffset: Option[NonEmptyString], thresholdLower: PercentileDouble, thresholdUpper: PercentileDouble, isCritical: Boolean = false, metadata: Seq[SparkParam] = Seq.empty) extends TrendCheckConfig with AverageBoundCheckConfig with Product with Serializable

   'Average bound range' check configuration

   'Average bound range' check configuration

   id

   Check ID

   description

   Check description

   metric

   Reference to a metric ID over which the check is performed

   rule

   Window calculation rule: by datetime or by number of records.

   windowSize

   Size of the window for average metric value calculation (either a number of records or duration).

   windowOffset

   Optional window offset (either a number of records or duration)

   thresholdLower

   Threshold value to calculate lower bound to compare with.

   thresholdUpper

   Threshold value to calculate upper bound to compare with.

   isCritical

   Flag if check is critical

   metadata

   List of metadata parameters specific to this check

5.  final case class AverageBoundUpperCheckConfig(id: ID, description: Option[NonEmptyString], metric: NonEmptyString, rule: TrendCheckRule, windowSize: NonEmptyString, windowOffset: Option[NonEmptyString], threshold: PercentileDouble, isCritical: Boolean = false, metadata: Seq[SparkParam] = Seq.empty) extends TrendCheckConfig with AverageBoundCheckConfig with Product with Serializable

   'Average bound upper' check configuration

   'Average bound upper' check configuration

   id

   Check ID

   description

   Check description

   metric

   Reference to a metric ID over which the check is performed

   rule

   Window calculation rule: by datetime or by number of records.

   windowSize

   Size of the window for average metric value calculation (either a number of records or duration).

   windowOffset

   Optional window offset (either a number of records or duration)

   threshold

   Threshold value to calculate upper bound to compare with.

   isCritical

   Flag if check is critical

   metadata

   List of metadata parameters specific to this check

6.  sealed abstract class CheckConfig extends JobConfigEntity

   Base class for check configurations.

   Base class for check configurations. All checks are described as DQ entities that must have a reference to metric ID that is being checked. Additionally, it is required to define method for appropriate check calculator initiation.

7.  final case class ChecksConfig(snapshot: Option[SnapshotChecks], trend: Option[TrendChecks], expression: Seq[ExpressionCheck] = Seq.empty) extends Product with Serializable

   Data Quality job configuration section describing all checks.

   Data Quality job configuration section describing all checks.

   snapshot

   Snapshot checks of all subtypes

   trend

   Trend checks of all subtypes

   expression

   List of expression checks

8.  final case class DifferByLtCheckConfig(id: ID, description: Option[NonEmptyString], metric: NonEmptyString, compareMetric: Option[NonEmptyString], threshold: Option[Double], isCritical: Boolean = false, metadata: Seq[SparkParam] = Seq.empty) extends SnapshotCheckConfig with Product with Serializable

   'Differ by less than' check configuration

   'Differ by less than' check configuration

   id

   Check ID

   description

   Check description

   metric

   Reference to a metric ID over which the check is performed

   compareMetric

   Reference to a metric ID to compare with

   threshold

   Threshold value indicating maximum difference between metric and compareMetric

   isCritical

   Flag if check is critical

   metadata

   List of metadata parameters specific to this check

9.  final case class EqualToCheckConfig(id: ID, description: Option[NonEmptyString], metric: NonEmptyString, compareMetric: Option[NonEmptyString], threshold: Option[Double], isCritical: Boolean = false, metadata: Seq[SparkParam] = Seq.empty) extends SnapshotCheckConfig with Product with Serializable

   'Equal to' check configuration

   'Equal to' check configuration

   id

   Check ID

   description

   Check description

   metric

   Reference to a metric ID over which the check is performed

   compareMetric

   Reference to a metric ID to compare with

   threshold

   Explicit threshold value to compare with

   isCritical

   Flag if check is critical

   metadata

   List of metadata parameters specific to this check

10.  final case class ExpressionCheck(id: ID, description: Option[NonEmptyString], formula: NonEmptyString, isCritical: Boolean = false, metadata: Seq[SparkParam] = Seq.empty) extends CheckConfig with Product with Serializable

    Expression check: uses boolean expression to evaluate check condition.

    Expression check: uses boolean expression to evaluate check condition.

    id

    Check ID

    description

    Check description

    formula

    Check formula: boolean expression referring to metric results.

    isCritical

    Flag if check is critical

    metadata

    List of metadata parameters specific to this check.

11.  final case class GreaterThanCheckConfig(id: ID, description: Option[NonEmptyString], metric: NonEmptyString, compareMetric: Option[NonEmptyString], threshold: Option[Double], isCritical: Boolean = false, metadata: Seq[SparkParam] = Seq.empty) extends SnapshotCheckConfig with Product with Serializable

    'Greater than' check configuration

    'Greater than' check configuration

    id

    Check ID

    description

    Check description

    metric

    Reference to a metric ID over which the check is performed

    compareMetric

    Reference to a metric ID to compare with

    threshold

    Explicit threshold value to compare with

    isCritical

    Flag if check is critical

    metadata

    List of metadata parameters specific to this check

12.  final case class LessThanCheckConfig(id: ID, description: Option[NonEmptyString], metric: NonEmptyString, compareMetric: Option[NonEmptyString], threshold: Option[Double], isCritical: Boolean = false, metadata: Seq[SparkParam] = Seq.empty) extends SnapshotCheckConfig with Product with Serializable

    'Less Than' check configuration

    'Less Than' check configuration

    id

    Check ID

    description

    Check description

    metric

    Reference to a metric ID over which the check is performed

    compareMetric

    Reference to a metric ID to compare with

    threshold

    Explicit threshold value to compare with

    isCritical

    Flag if check is critical

    metadata

    List of metadata parameters specific to this check

13.  sealed abstract class SnapshotCheckConfig extends CheckConfig

    Base class for snapshot check configurations.

    Base class for snapshot check configurations. All snapshot checks must have a reference to metric ID over which the check is performed. In addition, snapshot check configuration must have either comparison threshold or a metric ID to compare with. For some checks it is required to set both.

14.  final case class SnapshotChecks(differByLT: Seq[DifferByLtCheckConfig] = Seq.empty, equalTo: Seq[EqualToCheckConfig] = Seq.empty, lessThan: Seq[LessThanCheckConfig] = Seq.empty, greaterThan: Seq[GreaterThanCheckConfig] = Seq.empty) extends Product with Serializable

    Data Quality job configuration section describing snapshot checks.

    Data Quality job configuration section describing snapshot checks.

    differByLT

    Sequence of 'differ by less than' checks

    equalTo

    Sequence of 'equal to' checks

    lessThan

    Sequence of 'less than' checks

    greaterThan

    Sequence of 'greater than' checks

15.  final case class TopNRankCheckConfig(id: ID, description: Option[NonEmptyString], metric: NonEmptyString, targetNumber: PositiveInt, threshold: PercentileDouble, isCritical: Boolean = false, metadata: Seq[SparkParam] = Seq.empty) extends TrendCheckConfig with Product with Serializable

    TopN rank check configuration

    TopN rank check configuration

    id

    Check ID

    description

    Check description

    metric

    Reference to a metric ID over which the check is performed

    targetNumber

    Number of records from TopN metric result (R <= N)

    threshold

    Threshold value representing maximum allowed Jacquard distance between current and previous R-records from TopN metric result.

    isCritical

    Flag if check is critical

    metadata

    List of metadata parameters specific to this check

16.  sealed abstract class TrendCheckConfig extends CheckConfig

    Base class for trend check configurations.

    Base class for trend check configurations. All trend checks must have a reference to metric ID over which the check is performed.

17.  final case class TrendChecks(averageBoundFull: Seq[AverageBoundFullCheckConfig] = Seq.empty, averageBoundUpper: Seq[AverageBoundUpperCheckConfig] = Seq.empty, averageBoundLower: Seq[AverageBoundLowerCheckConfig] = Seq.empty, averageBoundRange: Seq[AverageBoundRangeCheckConfig] = Seq.empty, topNRank: Seq[TopNRankCheckConfig] = Seq.empty) extends Product with Serializable

    Data Quality job configuration section describing trend checks.

    Data Quality job configuration section describing trend checks.

    averageBoundFull

    Sequence of 'average bound full' checks

    averageBoundUpper

    Sequence of 'average bound upper' checks

    averageBoundLower

    Sequence of 'average bound lower' checks

    averageBoundRange

    Sequence of 'average bound range' checks

    topNRank

    Sequence of TopN rank checks