Type Members

1.  abstract class ConditionalDFCalculator extends DFMetricCalculator with ReversibleDFCalculator

   Abstract class for all conditional DF metric calculators.

   Abstract class for all conditional DF metric calculators. Thus, conditional calculator has a condition defined. When this condition is met for particular column value, then metric value is incremented by one. Otherwise metric value remain unchanged.

   All conditional metrics are reversible: direct error collection logic implies metric increment fails when condition is not met. Correspondingly, for reversed error collection logic, metric increment fails when condition IS met.

2.  abstract class DFMetricCalculator extends AnyRef

   Basic DF metric calculator

   Basic DF metric calculator

   Note

   DF Calculators are intendet to work with Batch applications only. Hence, their functionality may be revised in future to support streaming applications as well.

3.  abstract class GroupingDFMetricCalculator extends DFMetricCalculator

   Base class for metric calculators that require data grouping by metric columns.

   Base class for metric calculators that require data grouping by metric columns. These metric calculators require data shuffle, and, therefore, are processed as separate Spark Jobs.

   Since, we are groupng data by metric columns, then we have to define two types of aggregation functions:

   • aggregation function used to aggregate intermediate metric result per each group.
   • aggregation function used to aggregate final metric result from intermediate per-group results.

   Each aggregation function is accompanied with rowData collection for metric increment errors.

   Note

   There are currently only two metric calculators that inherit from this class: distinctValues and duplicateValues. For both of them final aggregation function is just a summation of per-group results.

4.  trait ReversibleDFCalculator extends AnyRef

   Trait to be mixed in to metric calculator to support reversal of error collection logic.

   Trait to be mixed in to metric calculator to support reversal of error collection logic. Reversible metric calculators can collect metric errors either in direct or in reversed mode depending on provided boolean flag.

   Note

   unlike RDD metric calculators, DF metric calculators essentially require only presence of reversed boolean flag. The rest of reversible logic will be implemented within spark expression.