Value Members

1.  final def !=(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

2.  final def ##(): Int

   Definition Classes

   AnyRef → Any

3.  final def ==(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

4.  final def asInstanceOf[T0]: T0

   Definition Classes

   Any

5.  def clone(): AnyRef

   Attributes

   protected[lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... ) @native()

6.  val columns: Seq[String]

   Definition Classes

   DuplicateValuesDFMetricCalculator → DFMetricCalculator

7.  val emptyValue: Column

   Value which is returned when metric result is null.

   Value which is returned when metric result is null. As all grouping metric calculators use summation during aggregation of intermediate per-group results then they should return zero when applied to empty sequence.

   Attributes

   protected

   Definition Classes

   GroupingDFMetricCalculator → DFMetricCalculator

8.  final def eq(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

9.  def errorConditionExpr(implicit colTypes: Map[String, DataType]): Column

   Collect error data for groups which contain more than 1 row i.e.

   Collect error data for groups which contain more than 1 row i.e. group has duplicates.

   colTypes

   Map of column names to their datatype.

   returns

   Spark row-level expression yielding boolean result.

   Attributes

   protected

   Definition Classes

   DuplicateValuesDFMetricCalculator → DFMetricCalculator

10.  def errorExpr(rowData: Column)(implicit colTypes: Map[String, DataType]): Column

    Grouping calculators will have different API for collecting per-group arrays with error data.

    Grouping calculators will have different API for collecting per-group arrays with error data. Therefore, an exception will be throw if this method is called.

    rowData

    Array of row data from columns related to this metric calculator (source keyFields + metric columns + window start time column for streaming applications)

    colTypes

    Map of column names to their datatype.

    returns

    Spark expression that will yield row data in case of metric error.

    Attributes

    protected

    Definition Classes

    GroupingDFMetricCalculator → DFMetricCalculator

11.  def errorMessage: String

    Error message that will be returned when duplicate values are found.

    Error message that will be returned when duplicate values are found.

    returns

    Metric increment failure message.

    Definition Classes

    DuplicateValuesDFMetricCalculator → DFMetricCalculator

12.  def errors(implicit errorDumpSize: Int, keyFields: Seq[String], colTypes: Map[String, DataType]): Column

    Final metric errors aggregation expression.

    Final metric errors aggregation expression. Merges all per-group metric errors into final array of error data. The size of array is limited by maximum allowed error dump size parameter.

    errorDumpSize

    Maximum allowed number of errors to be collected per single metric.

    keyFields

    Sequence of source/stream key fields.

    colTypes

    Map of column names to their datatype.

    returns

    Spark expression that will yield array of metric errors.

    Definition Classes

    GroupingDFMetricCalculator → DFMetricCalculator

13.  val errorsCol: String

    Name of the column that will store metric errors

    Name of the column that will store metric errors

    Definition Classes

    DFMetricCalculator

14.  def finalize(): Unit

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

15.  final def getClass(): Class[_]

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

16.  val groupAggregationFunction: (Column) ⇒ Column

    Function that aggregates metric increments into intermediate per-group metric results.

    Function that aggregates metric increments into intermediate per-group metric results. Accepts spark expression groupResultExpr as input and returns another spark expression that will yield aggregated double metric result per each group.

    As each group represents a distinct tuple value of selected columns then, in order to find number of duplicates it is just needed to count number of rows per each group. If group contains only one row, then it represents a unique value in a dataset. In addition, rows where entire tuple of columns is null are filtered out.

    Attributes

    protected

    Definition Classes

    DuplicateValuesDFMetricCalculator → GroupingDFMetricCalculator

17.  def groupErrorExpr(rowData: Column, errorDumpSize: Int): Column

    Per-group error collection expression: collects row data for ENTIRE group in case of metric error.

    Per-group error collection expression: collects row data for ENTIRE group in case of metric error. First element of row data array is removed as it is represents a unique value in the dataset, i.e. if group contains only one row, then error data should be empty.

    The size of array is limited by maximum allowed error dump size parameter.

    rowData

    Array of row data from columns related to this metric calculator for current group. (source keyFields + metric columns + window start time column for streaming applications)

    errorDumpSize

    Maximum allowed number of errors to be collected per single metric.

    returns

    Spark expression that will yield array of row data per group in case of metric error.

    Attributes

    protected

    Definition Classes

    DuplicateValuesDFMetricCalculator → GroupingDFMetricCalculator

18.  def groupErrors(implicit errorDumpSize: Int, keyFields: Seq[String], colTypes: Map[String, DataType]): Column

    Per-group metric errors aggregation expression.

    Per-group metric errors aggregation expression. Collects all metric errors into an array column pear each group. The size of array is limited by maximum allowed error dump size parameter.

    The main difference from one-pass DF-calculators is that error condition depends on the intermediate group aggregation result rather than on individual row result.

    errorDumpSize

    Maximum allowed number of errors to be collected per single metric.

    keyFields

    Sequence of source/stream key fields.

    colTypes

    Map of column names to their datatype.

    returns

    Spark expression that will yield array of metric errors.

    Definition Classes

    GroupingDFMetricCalculator

    Note

    For streaming applications, we need to collect metric errors in per-window basis. Therefore, error row data has to contain window start time (first element of array).

19.  val groupErrorsCol: String

    Name of the column that will store intermediate metric errors per each group.

    Name of the column that will store intermediate metric errors per each group.

    Definition Classes

    GroupingDFMetricCalculator

20.  def groupResult(implicit colTypes: Map[String, DataType]): Column

    Per-group intermediate metric aggregation expression that MUST yield double value.

    Per-group intermediate metric aggregation expression that MUST yield double value.

    colTypes

    Map of column names to their datatype.

    returns

    Spark expression that will yield double metric calculator result

    Definition Classes

    GroupingDFMetricCalculator

21.  val groupResultCol: String

    Name of the column that will store intermediate metric results per each group.

    Name of the column that will store intermediate metric results per each group.

    Definition Classes

    GroupingDFMetricCalculator

22.  final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

23.  val metricId: String

    Unlike RDD calculators, DF calculators are not groped by its type.

    Unlike RDD calculators, DF calculators are not groped by its type. For each metric defined in DQ job, there will be created its own instance of DF calculator. Thus, DF metric calculators can be linked to metric definitions by metricId.

    Definition Classes

    DuplicateValuesDFMetricCalculator → DFMetricCalculator

24.  val metricName: MetricName

    Definition Classes

    DuplicateValuesDFMetricCalculator → DFMetricCalculator

25.  final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

26.  final def notify(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

27.  final def notifyAll(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

28.  def result(implicit colTypes: Map[String, DataType]): Column

    Final metric aggregation expression that MUST yield double value.

    Final metric aggregation expression that MUST yield double value.

    colTypes

    Map of column names to their datatype.

    returns

    Spark expression that will yield double metric calculator result

    Definition Classes

    GroupingDFMetricCalculator → DFMetricCalculator

29.  val resultAggregateFunction: (Column) ⇒ Column

    Function that aggregates intermediate metric per-group results into final metric value.

    Function that aggregates intermediate metric per-group results into final metric value. Default aggregation for grouping metrics it is just a summation.

    Attributes

    protected

    Definition Classes

    GroupingDFMetricCalculator → DFMetricCalculator

30.  val resultCol: String

    Name of the column that will store metric result

    Name of the column that will store metric result

    Definition Classes

    DFMetricCalculator

31.  def resultExpr(implicit colTypes: Map[String, DataType]): Column

    Spark expression yielding numeric result for each row being processed per each group.

    Spark expression yielding numeric result for each row being processed per each group. Metric will be incremented with this result per each group using associated per-group aggregation function.

    Thus, for the purpose of finding number of duplicate values, we just need to assign each row with value 1 in order to count number of rows per-group each group.

    colTypes

    Map of column names to their datatype.

    returns

    Spark row-level expression yielding numeric result.

    Attributes

    protected

    Definition Classes

    DuplicateValuesDFMetricCalculator → DFMetricCalculator

    Note

    Spark expression MUST process single row but not aggregate multiple rows.

32.  def rowDataExpr(keyFields: Seq[String]): Column

    Row data collection expression: collects values of selected columns to array for row where metric error occurred.

    Row data collection expression: collects values of selected columns to array for row where metric error occurred.

    keyFields

    Sequence of source/stream key fields.

    returns

    Spark expression that will yield array of row data for column related to this metric calculator.

    Attributes

    protected

    Definition Classes

    DFMetricCalculator

33.  final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

34.  final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

35.  final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

36.  final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... ) @native()