Packages

c

org.checkita.dqf.core.metrics.rdd.regular.AlgebirdRDDMetrics

HLLSequenceCompletenessRDDMetricCalculator

case class HLLSequenceCompletenessRDDMetricCalculator(hLL: HLL, bitsNumber: Int, minVal: Long, maxVal: Long, accuracyError: Double, increment: Long, failCount: Long = 0, status: CalculatorStatus = CalculatorStatus.Success, failMsg: String = "OK") extends RDDMetricCalculator with Product with Serializable

Calculates approximate completeness of incremental integer (long) sequence, i.e. checks if sequence does not have missing elements. Check is performed using variance algorithm HyperLogLog. Works for single column only!

hLL

Initial HyperLogLog monoid

bitsNumber

Size of HLL (calculates automatically for a specific accuracy error)

minVal

Minimum observed value in a sequence

maxVal

Maximum observed value in a sequence

accuracyError

Error of calculation

increment

Sequence increment

returns

result map with keys: "APPROXIMATE_SEQUENCE_COMPLETENESS"

Linear Supertypes
Serializable, Serializable, Product, Equals, RDDMetricCalculator, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. HLLSequenceCompletenessRDDMetricCalculator
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. RDDMetricCalculator
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new HLLSequenceCompletenessRDDMetricCalculator(accuracyError: Double, increment: Long)
  2. new HLLSequenceCompletenessRDDMetricCalculator(hLL: HLL, bitsNumber: Int, minVal: Long, maxVal: Long, accuracyError: Double, increment: Long, failCount: Long = 0, status: CalculatorStatus = CalculatorStatus.Success, failMsg: String = "OK")

    hLL

    Initial HyperLogLog monoid

    bitsNumber

    Size of HLL (calculates automatically for a specific accuracy error)

    minVal

    Minimum observed value in a sequence

    maxVal

    Maximum observed value in a sequence

    accuracyError

    Error of calculation

    increment

    Sequence increment

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. val accuracyError: Double
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. val bitsNumber: Int
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  8. def copyWithError(status: CalculatorStatus, msg: String, failInc: Long = 1): RDDMetricCalculator

    Copy calculator with error status and corresponding message.

    Copy calculator with error status and corresponding message.

    status

    Calculator status to copy with

    msg

    Failure message

    failInc

    Failure increment

    returns

    Copy of this calculator with error status

    Attributes
    protected
    Definition Classes
    HLLSequenceCompletenessRDDMetricCalculator → RDDMetricCalculator
  9. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  10. val failCount: Long
    Attributes
    protected
    Definition Classes
    HLLSequenceCompletenessRDDMetricCalculator → RDDMetricCalculator
  11. val failMsg: String
    Attributes
    protected
    Definition Classes
    HLLSequenceCompletenessRDDMetricCalculator → RDDMetricCalculator
  12. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  13. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  14. def getFailCounter: Long

    Gets current metric failure counts

    Gets current metric failure counts

    returns

    Failure count

    Definition Classes
    RDDMetricCalculator
  15. def getFailMessage: String

    Gets current failure or error message

    Gets current failure or error message

    returns

    Failure message

    Definition Classes
    RDDMetricCalculator
  16. def getStatus: CalculatorStatus

    Gets current metric calculator status

    Gets current metric calculator status

    returns

    Calculator status

    Definition Classes
    RDDMetricCalculator
  17. val hLL: HLL
  18. def increment(values: Seq[Any]): RDDMetricCalculator

    Safely updates metric calculator

    Safely updates metric calculator

    values

    values to process

    returns

    updated calculator

    Definition Classes
    RDDMetricCalculator
  19. val increment: Long
  20. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  21. val maxVal: Long
  22. def merge(m2: RDDMetricCalculator): RDDMetricCalculator

    Merges two metric calculators together

    Merges two metric calculators together

    m2

    second metric calculator

    returns

    merged metric calculator

    Definition Classes
    HLLSequenceCompletenessRDDMetricCalculator → RDDMetricCalculator
  23. val minVal: Long
  24. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  25. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  26. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  27. def result(): Map[String, (Double, Option[String])]

    Gets results of calculator in the current state

    Gets results of calculator in the current state

    returns

    Map of (result_name -> (result, additionalResults))

    Definition Classes
    HLLSequenceCompletenessRDDMetricCalculator → RDDMetricCalculator
  28. val status: CalculatorStatus
    Attributes
    protected
    Definition Classes
    HLLSequenceCompletenessRDDMetricCalculator → RDDMetricCalculator
  29. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  30. def tryToIncrement(values: Seq[Any]): RDDMetricCalculator

    Increment metric calculator.

    Increment metric calculator. May throw an exception.

    values

    values to process

    returns

    updated calculator or throws an exception

    Attributes
    protected
    Definition Classes
    HLLSequenceCompletenessRDDMetricCalculator → RDDMetricCalculator
  31. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  32. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  33. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from RDDMetricCalculator

Inherited from AnyRef

Inherited from Any

Ungrouped