Packages

package rdd

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. All

Type Members

  1. abstract class RDDMetricCalculator extends AnyRef

    Basic RDD metric calculator

  2. trait RDDMetricProcessor extends BasicMetricProcessor

    Base functionality for regular metric processor.

    Base functionality for regular metric processor. The concrete implementation of metric processor differs for batch and streaming applications.

  3. trait ReversibleRDDCalculator extends AnyRef

    Trait to be mixed in to metric calculator to support reversal of error collection logic.

    Trait to be mixed in to metric calculator to support reversal of error collection logic.

    • Reversible metric calculators can collect metric errors either in direct or in reversed mode depending on provided boolean flag.

Value Members

  1. object Casting

    Helpers used to convert values of type Any to desirable type.

    Helpers used to convert values of type Any to desirable type.

    The indent of these helpers is to manage values obtained from Spark Row to desired type for use in metric calculators.

    As the Spark Row can stores elements of various type then we need to guess (pattern match) it to provide an appropriate conversion method.

    For that purpose we will follow Spark SQL Type to Java types mapping:

    • BooleanType -> java.lang.Boolean
    • ByteType -> java.lang.Byte
    • ShortType -> java.lang.Short
    • IntegerType -> java.lang.Integer
    • LongType -> java.lang.Long
    • FloatType -> java.lang.Float
    • DoubleType -> java.lang.Double
    • StringType -> String
    • DecimalType -> java.math.BigDecimal
    • DateType -> java.sql.Date if spark.sql.datetime.java8API.enabled is false
    • DateType -> java.time.LocalDate if spark.sql.datetime.java8API.enabled is true
    • TimestampType -> java.sql.Timestamp if spark.sql.datetime.java8API.enabled is false
    • TimestampType -> java.time.Instant if spark.sql.datetime.java8API.enabled is true
    • BinaryType -> byte array
    • ArrayType -> scala.collection.Seq (use getList for java.util.List)
    • MapType -> scala.collection.Map (use getJavaMap for java.util.Map)
    • StructType -> org.apache.spark.sql.Row
  2. object RDDMetricBatchProcessor extends RDDMetricProcessor

    Regular metrics processor for Batch Applications

  3. object RDDMetricProcessor
  4. object RDDMetricStreamProcessor extends RDDMetricProcessor with Logging

Ungrouped