Packages

c

org.checkita.dqf.core.metrics.df.regular.MultiColumnDFMetrics

LevenshteinDistanceDFMetricCalculator

case class LevenshteinDistanceDFMetricCalculator(metricId: String, columns: Seq[String], threshold: Double, normalize: Boolean, reversed: Boolean) extends MultiColumnConditionalDFCalculator with Product with Serializable

Calculates amount of rows where Levenshtein distance between 2 columns is less than threshold.

metricId

Id of the metric.

columns

Sequence of columns which are used for metric calculation

threshold

Threshold (should be within [0, 1] range for normalized results)

normalize

Flag to define whether distance should be normalized over maximum length of two input strings

reversed

Boolean flag indicating whether error collection logic should be reversed for this metric

Linear Supertypes
Serializable, Serializable, Product, Equals, MultiColumnConditionalDFCalculator, ReversibleDFCalculator, DFMetricCalculator, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. LevenshteinDistanceDFMetricCalculator
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. MultiColumnConditionalDFCalculator
  7. ReversibleDFCalculator
  8. DFMetricCalculator
  9. AnyRef
  10. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new LevenshteinDistanceDFMetricCalculator(metricId: String, columns: Seq[String], threshold: Double, normalize: Boolean, reversed: Boolean)

    metricId

    Id of the metric.

    columns

    Sequence of columns which are used for metric calculation

    threshold

    Threshold (should be within [0, 1] range for normalized results)

    normalize

    Flag to define whether distance should be normalized over maximum length of two input strings

    reversed

    Boolean flag indicating whether error collection logic should be reversed for this metric

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. val columns: Seq[String]
    Definition Classes
    LevenshteinDistanceDFMetricCalculator → DFMetricCalculator
  7. val emptyValue: Column

    Value which is returned when metric result is null.

    Value which is returned when metric result is null.

    Attributes
    protected
    Definition Classes
    MultiColumnConditionalDFCalculator → DFMetricCalculator
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def errorConditionExpr(implicit colTypes: Map[String, DataType]): Column

    Spark expression yielding boolean result for processed row.

    Spark expression yielding boolean result for processed row. Indicates whether metric increment failed or not.

    colTypes

    Map of column names to their datatype.

    returns

    Spark row-level expression yielding boolean result.

    Attributes
    protected
    Definition Classes
    MultiColumnConditionalDFCalculator → DFMetricCalculator
  10. def errorExpr(rowData: Column)(implicit colTypes: Map[String, DataType]): Column

    Error collection expression: collects row data in case of metric error.

    Error collection expression: collects row data in case of metric error.

    rowData

    Array of row data from columns related to this metric calculator (source keyFields + metric columns + window start time column for streaming applications)

    colTypes

    Map of column names to their datatype.

    returns

    Spark expression that will yield row data in case of metric error.

    Attributes
    protected
    Definition Classes
    DFMetricCalculator
  11. def errorMessage: String

    For direct error collection logic rows where levenshtein distance between two string values is greater than or equal to the provided threshold are considered as metric failure.

    For direct error collection logic rows where levenshtein distance between two string values is greater than or equal to the provided threshold are considered as metric failure. For reverses error collection logic rows where levenshtein distance between two string values is lower than the provided threshold are considered as metric failure.

    returns

    Metric increment failure message.

    Definition Classes
    LevenshteinDistanceDFMetricCalculator → DFMetricCalculator
  12. def errors(implicit errorDumpSize: Int, keyFields: Seq[String], colTypes: Map[String, DataType]): Column

    Final metric errors aggregation expression.

    Final metric errors aggregation expression. Collects all metric errors into an array column. The size of array is limited by maximum allowed error dump size parameter.

    errorDumpSize

    Maximum allowed number of errors to be collected per single metric.

    keyFields

    Sequence of source/stream key fields.

    colTypes

    Map of column names to their datatype.

    returns

    Spark expression that will yield array of metric errors.

    Definition Classes
    DFMetricCalculator
  13. val errorsCol: String

    Name of the column that will store metric errors

    Name of the column that will store metric errors

    Definition Classes
    DFMetricCalculator
  14. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  15. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  16. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  17. def metricCondExpr: Column

    Row-level spark expression that retrieves day-distance between column values and checks if it is less than provided threshold.

    Row-level spark expression that retrieves day-distance between column values and checks if it is less than provided threshold.

    Attributes
    protected
    Definition Classes
    LevenshteinDistanceDFMetricCalculator → MultiColumnConditionalDFCalculator
  18. val metricId: String

    Unlike RDD calculators, DF calculators are not groped by its type.

    Unlike RDD calculators, DF calculators are not groped by its type. For each metric defined in DQ job, there will be created its own instance of DF calculator. Thus, DF metric calculators can be linked to metric definitions by metricId.

    Definition Classes
    LevenshteinDistanceDFMetricCalculator → DFMetricCalculator
  19. val metricName: MetricName
    Definition Classes
    LevenshteinDistanceDFMetricCalculator → DFMetricCalculator
  20. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  21. val normalize: Boolean
  22. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  23. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  24. def result(implicit colTypes: Map[String, DataType]): Column

    Final metric aggregation expression that MUST yield double value.

    Final metric aggregation expression that MUST yield double value.

    colTypes

    Map of column names to their datatype.

    returns

    Spark expression that will yield double metric calculator result

    Definition Classes
    DFMetricCalculator
  25. val resultAggregateFunction: (Column) ⇒ Column

    Aggregation function for all conditional metrics is just a summation.

    Aggregation function for all conditional metrics is just a summation.

    Attributes
    protected
    Definition Classes
    MultiColumnConditionalDFCalculator → DFMetricCalculator
  26. val resultCol: String

    Name of the column that will store metric result

    Name of the column that will store metric result

    Definition Classes
    DFMetricCalculator
  27. def resultExpr(implicit colTypes: Map[String, DataType]): Column

    Spark expression yielding numeric result for processed row.

    Spark expression yielding numeric result for processed row. Metric will be incremented with this result using associated aggregation function.

    colTypes

    Map of column names to their datatype.

    returns

    Spark row-level expression yielding numeric result.

    Attributes
    protected
    Definition Classes
    MultiColumnConditionalDFCalculator → DFMetricCalculator
    Note

    Spark expression MUST process single row but not aggregate multiple rows.

  28. val reversed: Boolean
    Attributes
    protected
    Definition Classes
    LevenshteinDistanceDFMetricCalculator → ReversibleDFCalculator
  29. def rowDataExpr(keyFields: Seq[String]): Column

    Row data collection expression: collects values of selected columns to array for row where metric error occurred.

    Row data collection expression: collects values of selected columns to array for row where metric error occurred.

    keyFields

    Sequence of source/stream key fields.

    returns

    Spark expression that will yield array of row data for column related to this metric calculator.

    Attributes
    protected
    Definition Classes
    DFMetricCalculator
  30. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  31. val threshold: Double
  32. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  33. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  34. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from MultiColumnConditionalDFCalculator

Inherited from ReversibleDFCalculator

Inherited from DFMetricCalculator

Inherited from AnyRef

Inherited from Any

Ungrouped