Type Members

1.  case class CastedNumberRDDMetricCalculator(cnt: Long, reversed: Boolean, failCount: Long = 0, status: CalculatorStatus = CalculatorStatus.Success, failMsg: String = "OK") extends RDDMetricCalculator with ReversibleRDDCalculator with Product with Serializable

   Calculates amount of element that can be cast to numerical (double format)

   Calculates amount of element that can be cast to numerical (double format)

   cnt

   Current count of numeric elements

   returns

   result map with keys: "CASTED_NUMBER"

2.  case class FormattedNumberRDDMetricCalculator(cnt: Long, precision: Int, scale: Int, compareRule: String, reversed: Boolean, failCount: Long = 0, status: CalculatorStatus = CalculatorStatus.Success, failMsg: String = "OK") extends RDDMetricCalculator with ReversibleRDDCalculator with Product with Serializable

   Calculates amount of elements that fit (or do not fit) provided Decimal format: Decimal(precision, scale)

   Calculates amount of elements that fit (or do not fit) provided Decimal format: Decimal(precision, scale)

   cnt

   Current amount of elements which fit (didn't fit) decimal format

   precision

   Precision threshold

   scale

   Required scale

   compareRule

   Either "inbound" or "outbound": defines wither number should fit within provided decimal format or to be outside of the provided precision and scale

   returns

   result map with keys: "FORMATTED_NUMBER"

3.  case class MaxNumberRDDMetricCalculator(max: Double, failCount: Long = 0, status: CalculatorStatus = CalculatorStatus.Success, failMsg: String = "OK") extends RDDMetricCalculator with Product with Serializable

   Calculates maximal value for provided elements

   Calculates maximal value for provided elements

   max

   Current maximal value

   returns

   result map with keys: "MAX_NUMBER"

4.  case class MinNumberRDDMetricCalculator(min: Double, failCount: Long = 0, status: CalculatorStatus = CalculatorStatus.Success, failMsg: String = "OK") extends RDDMetricCalculator with Product with Serializable

   Calculates minimal value for provided elements

   Calculates minimal value for provided elements

   min

   Current minimal value

   returns

   result map with keys: "MIN_NUMBER"

5.  case class NumberBetweenRDDMetricCalculator(cnt: Long, lowerCompareValue: Double, upperCompareValue: Double, includeBound: Boolean, reversed: Boolean, failCount: Long = 0, status: CalculatorStatus = CalculatorStatus.Success, failMsg: String = "OK") extends RDDMetricCalculator with ReversibleRDDCalculator with Product with Serializable

   Calculates count of rows for which column value is within the lowerCompareValue:upperCompareValue interval

   Calculates count of rows for which column value is within the lowerCompareValue:upperCompareValue interval

   cnt

   Current count of appearance

   lowerCompareValue

   Target lower interval bound to compare with

   upperCompareValue

   Target upper interval bound to compare with

   includeBound

   Flag which sets whether interval bounds are included or excluded from the interval.

   returns

   result map with keys: "NUMBER_BETWEEN"

6.  case class NumberGreaterThanRDDMetricCalculator(cnt: Long, compareValue: Double, includeBound: Boolean, reversed: Boolean, failCount: Long = 0, status: CalculatorStatus = CalculatorStatus.Success, failMsg: String = "OK") extends RDDMetricCalculator with ReversibleRDDCalculator with Product with Serializable

   Calculates count of rows for which column value is greater than compareValue

   Calculates count of rows for which column value is greater than compareValue

   cnt

   Current count of appearance

   compareValue

   Target value to compare with

   includeBound

   Flag which sets whether compareValue is included (>=) or excluded (>) from the interval.

   returns

   result map with keys: "NUMBER_GREATER_THAN"

7.  case class NumberInDomainRDDMetricCalculator(cnt: Long, domain: Set[Double], reversed: Boolean, failCount: Long = 0, status: CalculatorStatus = CalculatorStatus.Success, failMsg: String = "OK") extends RDDMetricCalculator with ReversibleRDDCalculator with Product with Serializable

   Calculated amount of elements in provided domain set

   Calculated amount of elements in provided domain set

   cnt

   Current count of elements in domain

   domain

   Set of element that represent requested domain

   returns

   result map with keys: "NUMBER_IN_DOMAIN"

8.  case class NumberLessThanRDDMetricCalculator(cnt: Long, compareValue: Double, includeBound: Boolean, reversed: Boolean, failCount: Long = 0, status: CalculatorStatus = CalculatorStatus.Success, failMsg: String = "OK") extends RDDMetricCalculator with ReversibleRDDCalculator with Product with Serializable

   Calculates count of rows for which column value is less than compareValue

   Calculates count of rows for which column value is less than compareValue

   cnt

   Current count of appearance

   compareValue

   Target value to compare with

   includeBound

   Flag which sets whether compareValue is included or excluded from the interval

   returns

   result map with keys: "NUMBER_LESS_THAN"

9.  case class NumberNotBetweenRDDMetricCalculator(cnt: Long, lowerCompareValue: Double, upperCompareValue: Double, includeBound: Boolean, reversed: Boolean, failCount: Long = 0, status: CalculatorStatus = CalculatorStatus.Success, failMsg: String = "OK") extends RDDMetricCalculator with ReversibleRDDCalculator with Product with Serializable

   Calculates count of rows for which column value is not within the lowerCompareValue:upperCompareValue interval

   Calculates count of rows for which column value is not within the lowerCompareValue:upperCompareValue interval

   cnt

   Current count of appearance

   lowerCompareValue

   Target lower interval bound to compare with

   upperCompareValue

   Target upper interval bound to compare with

   includeBound

   Flag which sets whether interval bounds are included or excluded from the interval.

   returns

   result map with keys: "NUMBER_NOT_BETWEEN"

10.  case class NumberOutDomainRDDMetricCalculator(cnt: Double, domain: Set[Double], reversed: Boolean, failCount: Long = 0, status: CalculatorStatus = CalculatorStatus.Success, failMsg: String = "OK") extends RDDMetricCalculator with ReversibleRDDCalculator with Product with Serializable

    Calculates amount of elements out of provided domain set

    Calculates amount of elements out of provided domain set

    cnt

    Current count of elements out of domain

    domain

    set of element that represent requested domain

    returns

    result map with keys: "NUMBER_OUT_DOMAIN"

11.  case class NumberValuesRDDMetricCalculator(cnt: Long, compareValue: Double, reversed: Boolean, failCount: Long = 0, status: CalculatorStatus = CalculatorStatus.Success, failMsg: String = "OK") extends RDDMetricCalculator with ReversibleRDDCalculator with Product with Serializable

    Calculates count of requested value's appearance in processed elements

    Calculates count of requested value's appearance in processed elements

    cnt

    Current count of appearance

    compareValue

    Target value to track

    returns

    result map with keys: "NUMBER_VALUES"

12.  case class SequenceCompletenessRDDMetricCalculator(uniqueValues: Set[Long], increment: Long, minVal: Long, maxVal: Long, failCount: Long = 0, status: CalculatorStatus = CalculatorStatus.Success, failMsg: String = "OK") extends RDDMetricCalculator with Product with Serializable

    Calculates completeness of incremental integer (long) sequence, i.e.

    Calculates completeness of incremental integer (long) sequence, i.e. checks if sequence does not have missing elements. Works for single column only!

    uniqueValues

    Set of unique values in a sequence

    increment

    Sequence increment

    minVal

    Minimum observed value in a sequence

    maxVal

    Maximum observed value in a sequence

    returns

    result map with keys: "SEQUENCE_COMPLETENESS"

13.  case class StdAvgNumberRDDMetricCalculator(sum: Double, sqSum: Double, cnt: Long, failCount: Long = 0, status: CalculatorStatus = CalculatorStatus.Success, failMsg: String = "OK") extends RDDMetricCalculator with Product with Serializable

    Calculates standard deviation and mean value for provided elements

    Calculates standard deviation and mean value for provided elements

    Works for single column only!

    sum

    Current sum

    sqSum

    Current squared sum

    cnt

    Current element count

    returns

    result map with keys: "STD_NUMBER" "AVG_NUMBER"

14.  case class SumNumberRDDMetricCalculator(sum: Double, failCount: Long = 0, status: CalculatorStatus = CalculatorStatus.Success, failMsg: String = "OK") extends RDDMetricCalculator with Product with Serializable

    Calculates sum of provided elements

    Calculates sum of provided elements

    sum

    Current sum

    returns

    result map with keys: "SUM_NUMBER"

15.  case class TDigestRDDMetricCalculator(tdigest: TDigest, accuracyError: Double, targetSideNumber: Double, failCount: Long = 0, status: CalculatorStatus = CalculatorStatus.Success, failMsg: String = "OK") extends RDDMetricCalculator with Product with Serializable

    Calculates percentiles, quantiles for provided elements with use of TDigest library https://github.com/isarn/isarn-sketches

    Calculates percentiles, quantiles for provided elements with use of TDigest library https://github.com/isarn/isarn-sketches

    Works for single column only!

    tdigest

    Initial TDigest object

    accuracyError

    Required level of calculation accuracy

    targetSideNumber

    Required parameter. For quantiles should be in [0,1]

    returns

    result map with keys:

    • "GET_QUANTILE"
    • "GET_PERCENTILE"
    • "FIRST_QUANTILE"
    • "THIRD_QUANTILE"
    • "MEDIAN_VALUE"