Packages

c

org.checkita.dqf.core.metrics.df

GroupingDFMetricCalculator

abstract class GroupingDFMetricCalculator extends DFMetricCalculator

Base class for metric calculators that require data grouping by metric columns. These metric calculators require data shuffle, and, therefore, are processed as separate Spark Jobs.

Since, we are groupng data by metric columns, then we have to define two types of aggregation functions:

  • aggregation function used to aggregate intermediate metric result per each group.
  • aggregation function used to aggregate final metric result from intermediate per-group results.

Each aggregation function is accompanied with rowData collection for metric increment errors.

Note

There are currently only two metric calculators that inherit from this class: distinctValues and duplicateValues. For both of them final aggregation function is just a summation of per-group results.

Linear Supertypes
DFMetricCalculator, AnyRef, Any
Known Subclasses
DistinctValuesDFMetricCalculator, DuplicateValuesDFMetricCalculator, SequenceCompletenessDFMetricCalculator
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. GroupingDFMetricCalculator
  2. DFMetricCalculator
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new GroupingDFMetricCalculator()

Abstract Value Members

  1. abstract val columns: Seq[String]
    Definition Classes
    DFMetricCalculator
  2. abstract def errorConditionExpr(implicit colTypes: Map[String, DataType]): Column

    Spark expression yielding boolean result for processed row.

    Spark expression yielding boolean result for processed row. Indicates whether metric increment failed or not. Usually checks the outcome of resultExpr.

    colTypes

    Map of column names to their datatype.

    returns

    Spark row-level expression yielding boolean result.

    Attributes
    protected
    Definition Classes
    DFMetricCalculator
  3. abstract def errorMessage: String

    Error message that will be returned when metric increment fails.

    Error message that will be returned when metric increment fails.

    returns

    Metric increment failure message.

    Definition Classes
    DFMetricCalculator
  4. abstract val groupAggregationFunction: (Column) ⇒ Column

    Function that aggregates metric increments into intermediate per-group metric results.

    Function that aggregates metric increments into intermediate per-group metric results. Accepts spark expression groupResultExpr as input and returns another spark expression that will yield aggregated double metric result per each group.

    Attributes
    protected
  5. abstract def groupErrorExpr(rowData: Column, errorDumpSize: Int): Column

    Per-group error collection expression: collects row data for ENTIRE group in case of metric error.

    Per-group error collection expression: collects row data for ENTIRE group in case of metric error. The size of array is limited by maximum allowed error dump size parameter.

    rowData

    Array of row data from columns related to this metric calculator for current group. (source keyFields + metric columns + window start time column for streaming applications)

    errorDumpSize

    Maximum allowed number of errors to be collected per single metric.

    returns

    Spark expression that will yield array of row data per group in case of metric error.

    Attributes
    protected
  6. abstract val metricId: String

    Unlike RDD calculators, DF calculators are not groped by its type.

    Unlike RDD calculators, DF calculators are not groped by its type. For each metric defined in DQ job, there will be created its own instance of DF calculator. Thus, DF metric calculators can be linked to metric definitions by metricId.

    Definition Classes
    DFMetricCalculator
  7. abstract val metricName: MetricName
    Definition Classes
    DFMetricCalculator
  8. abstract def resultExpr(implicit colTypes: Map[String, DataType]): Column

    Spark expression yielding numeric result for processed row.

    Spark expression yielding numeric result for processed row. Metric will be incremented with this result using associated aggregation function.

    colTypes

    Map of column names to their datatype.

    returns

    Spark row-level expression yielding numeric result.

    Attributes
    protected
    Definition Classes
    DFMetricCalculator
    Note

    Spark expression MUST process single row but not aggregate multiple rows.

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. val emptyValue: Column

    Value which is returned when metric result is null.

    Value which is returned when metric result is null. As all grouping metric calculators use summation during aggregation of intermediate per-group results then they should return zero when applied to empty sequence.

    Attributes
    protected
    Definition Classes
    GroupingDFMetricCalculator → DFMetricCalculator
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  9. def errorExpr(rowData: Column)(implicit colTypes: Map[String, DataType]): Column

    Grouping calculators will have different API for collecting per-group arrays with error data.

    Grouping calculators will have different API for collecting per-group arrays with error data. Therefore, an exception will be throw if this method is called.

    rowData

    Array of row data from columns related to this metric calculator (source keyFields + metric columns + window start time column for streaming applications)

    colTypes

    Map of column names to their datatype.

    returns

    Spark expression that will yield row data in case of metric error.

    Attributes
    protected
    Definition Classes
    GroupingDFMetricCalculator → DFMetricCalculator
  10. def errors(implicit errorDumpSize: Int, keyFields: Seq[String], colTypes: Map[String, DataType]): Column

    Final metric errors aggregation expression.

    Final metric errors aggregation expression. Merges all per-group metric errors into final array of error data. The size of array is limited by maximum allowed error dump size parameter.

    errorDumpSize

    Maximum allowed number of errors to be collected per single metric.

    keyFields

    Sequence of source/stream key fields.

    colTypes

    Map of column names to their datatype.

    returns

    Spark expression that will yield array of metric errors.

    Definition Classes
    GroupingDFMetricCalculator → DFMetricCalculator
  11. val errorsCol: String

    Name of the column that will store metric errors

    Name of the column that will store metric errors

    Definition Classes
    DFMetricCalculator
  12. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  13. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  14. def groupErrors(implicit errorDumpSize: Int, keyFields: Seq[String], colTypes: Map[String, DataType]): Column

    Per-group metric errors aggregation expression.

    Per-group metric errors aggregation expression. Collects all metric errors into an array column pear each group. The size of array is limited by maximum allowed error dump size parameter.

    The main difference from one-pass DF-calculators is that error condition depends on the intermediate group aggregation result rather than on individual row result.

    errorDumpSize

    Maximum allowed number of errors to be collected per single metric.

    keyFields

    Sequence of source/stream key fields.

    colTypes

    Map of column names to their datatype.

    returns

    Spark expression that will yield array of metric errors.

    Note

    For streaming applications, we need to collect metric errors in per-window basis. Therefore, error row data has to contain window start time (first element of array).

  15. val groupErrorsCol: String

    Name of the column that will store intermediate metric errors per each group.

  16. def groupResult(implicit colTypes: Map[String, DataType]): Column

    Per-group intermediate metric aggregation expression that MUST yield double value.

    Per-group intermediate metric aggregation expression that MUST yield double value.

    colTypes

    Map of column names to their datatype.

    returns

    Spark expression that will yield double metric calculator result

  17. val groupResultCol: String

    Name of the column that will store intermediate metric results per each group.

  18. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  19. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  20. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  21. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  22. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  23. def result(implicit colTypes: Map[String, DataType]): Column

    Final metric aggregation expression that MUST yield double value.

    Final metric aggregation expression that MUST yield double value.

    colTypes

    Map of column names to their datatype.

    returns

    Spark expression that will yield double metric calculator result

    Definition Classes
    GroupingDFMetricCalculator → DFMetricCalculator
  24. val resultAggregateFunction: (Column) ⇒ Column

    Function that aggregates intermediate metric per-group results into final metric value.

    Function that aggregates intermediate metric per-group results into final metric value. Default aggregation for grouping metrics it is just a summation.

    Attributes
    protected
    Definition Classes
    GroupingDFMetricCalculator → DFMetricCalculator
  25. val resultCol: String

    Name of the column that will store metric result

    Name of the column that will store metric result

    Definition Classes
    DFMetricCalculator
  26. def rowDataExpr(keyFields: Seq[String]): Column

    Row data collection expression: collects values of selected columns to array for row where metric error occurred.

    Row data collection expression: collects values of selected columns to array for row where metric error occurred.

    keyFields

    Sequence of source/stream key fields.

    returns

    Spark expression that will yield array of row data for column related to this metric calculator.

    Attributes
    protected
    Definition Classes
    DFMetricCalculator
  27. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  28. def toString(): String
    Definition Classes
    AnyRef → Any
  29. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  30. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  31. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from DFMetricCalculator

Inherited from AnyRef

Inherited from Any

Ungrouped