Packages

trait DQJob extends Logging

Base trait defining basic functionality of Data Quality Job

Linear Supertypes
Logging, AnyRef, Any
Known Subclasses
DQBatchJob, DQStreamWindowJob
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DQJob
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. trait RegularMetricsProcessor extends AnyRef

    Metrics processed differently for batch and streaming job.

    Metrics processed differently for batch and streaming job. Therefore, to generalize metric processing API, we introduce this trait that common method to process all regular metrics.

    Attributes
    protected

Abstract Value Members

  1. abstract val checks: Seq[CheckConfig]
  2. abstract val composedMetrics: Seq[ComposedMetricConfig]
  3. abstract val connections: Map[String, DQConnection]
  4. implicit abstract val fs: FileSystem
  5. abstract val jobConfig: JobConfig
  6. implicit abstract val jobId: String
  7. abstract val loadChecks: Seq[LoadCheckConfig]
  8. abstract val metrics: Seq[RegularMetricConfig]
  9. abstract val schemas: Map[String, SourceSchema]
  10. abstract val sources: Seq[Source]
  11. implicit abstract val spark: SparkSession
  12. abstract val storageManager: Option[DqStorageManager]
  13. abstract val targets: Seq[TargetConfig]
  14. abstract val trendMetrics: Seq[TrendMetricConfig]

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def calculateComposedMetrics(stage: String, regularMetricResults: Result[MetricResults]): Result[MetricResults]

    Calculates all composed metrics provided with regular metric results.

    Calculates all composed metrics provided with regular metric results.

    stage

    Stage indication used for logging.

    regularMetricResults

    Map of regular metric results

    returns

    Either a map of composed metric results or a list of calculation errors.

    Attributes
    protected
  6. def calculateTrendMetrics(stage: String)(implicit jobId: String, manager: Option[DqStorageManager], settings: AppSettings): Result[MetricResults]

    Calculates all trend metrics provided with regular and composed metric results.

    Calculates all trend metrics provided with regular and composed metric results.

    stage

    Stage indication used for logging.

    jobId

    Implicit current job ID.

    manager

    Implicit storage manager used to load historical results.

    settings

    Implicit application settings object.

    returns

    Either a map of trend metric results or a list of calculation errors.

    Attributes
    protected
  7. val checksStage: String
    Attributes
    protected
  8. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  9. def combineResults(stage: String, loadCheckResults: Seq[ResultCheckLoad], checkResults: Result[Seq[ResultCheck]], jobState: Result[JobState], regularMetricResults: Result[Seq[ResultMetricRegular]], composedMetricResults: Result[Seq[ResultMetricComposed]], trendMetricResults: Result[Seq[ResultMetricTrend]], metricErrors: Result[Seq[ResultMetricError]])(implicit settings: AppSettings): Result[ResultSet]

    Combines all results into a final result set.

    Combines all results into a final result set.

    stage

    Stage indication used for logging.

    loadCheckResults

    Sequence of load check results

    checkResults

    Sequence of check results (wrapped into Either)

    regularMetricResults

    Sequence of regular metric results (wrapped into Either)

    composedMetricResults

    Sequence of composed metric results (wrapped into Either)

    trendMetricResults

    Sequence of trend metrics results (wrapped into Either)

    metricErrors

    Sequence of metric errors (wrapped into Either)

    settings

    Implicit application settings object

    returns

    Combined results in form of ResultSet

    Attributes
    protected
  10. val composedMetricsMap: Map[String, ComposedMetricConfig]
    Attributes
    protected
  11. def encryptMetricErrors(errors: Seq[ResultMetricError])(implicit settings: AppSettings): Result[Seq[ResultMetricError]]

    Encrypts rowData field in metric errors if requested per application configuration.

    Encrypts rowData field in metric errors if requested per application configuration. Row data field in metric errors contains excerpt from data source and, therefore, these data can contain some sensitive information. In order to protect it, users can configure encryption chapter in application configuration and store encrypted rowData in DQ storage.

    errors

    Sequence of metric errors to encrypt

    settings

    Implicit application settings object

    returns

    Sequence of metric errors with encrypted rowData field (if requested per configuration) or a list of encryption errors.

    Attributes
    protected
    Note

    When metric errors are send via targets rowData field is never encrypted.

  12. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  14. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  15. def finalizeComposedMetrics(stage: String, metricResults: Result[MetricResults])(implicit settings: AppSettings): Result[Seq[ResultMetricComposed]]

    Finalizes composed metric results: selects only composed metrics results and converts them to final composed metric results representation ready for writing into storage DB or sending via targets.

    Finalizes composed metric results: selects only composed metrics results and converts them to final composed metric results representation ready for writing into storage DB or sending via targets.

    stage

    Stage indication used for logging.

    metricResults

    Map with all metric results

    settings

    Implicit application settings object

    returns

    Either a finalized sequence of composed metric results or a list of conversion errors.

    Attributes
    protected
  16. def finalizeJobState(jobConfig: JobConfig)(implicit settings: AppSettings, jobId: String): Result[JobState]

    Finalizes job state: select whether to encrypt or not and converts to the final representation ready for writing into storage DB or sending via targets.

    Finalizes job state: select whether to encrypt or not and converts to the final representation ready for writing into storage DB or sending via targets.

    jobConfig

    Parsed Data Quality job configuration

    settings

    Implicit application settings object

    jobId

    Current Job ID

    returns

    Either a finalized of job state.

    Attributes
    protected
  17. def finalizeMetricErrors(stage: String, metricResults: Result[MetricResults])(implicit settings: AppSettings): Result[Seq[ResultMetricError]]

    Finalizes metric errors: retrieves metrics errors from results and converts them to final metric errors representation ready for writing into storage DB or sending via targets.

    Finalizes metric errors: retrieves metrics errors from results and converts them to final metric errors representation ready for writing into storage DB or sending via targets.

    stage

    Stage indication used for logging.

    metricResults

    Map with all metric results

    settings

    Implicit application settings object

    returns

    Either a finalized sequence of metric errors or a list of conversion errors.

    Attributes
    protected
    Note

    There could be a situations when metric errors hold the same error data. We are not interested in sending repeating data neither to storage database nor to Targets. Therefore, sequence of metric errors is deduplicated by unique constraint.

  18. def finalizeRegularMetrics(stage: String, metricResults: Result[MetricResults])(implicit settings: AppSettings): Result[Seq[ResultMetricRegular]]

    Finalizes regular metric results: selects only regular metrics results and converts them to final regular metric results representation ready for writing into storage DB or sending via targets.

    Finalizes regular metric results: selects only regular metrics results and converts them to final regular metric results representation ready for writing into storage DB or sending via targets.

    stage

    Stage indication used for logging.

    metricResults

    Map with all metric results

    settings

    Implicit application settings object

    returns

    Either a finalized sequence of regular metric results or a list of conversion errors.

    Attributes
    protected
  19. def finalizeTrendMetrics(stage: String, metricResults: Result[MetricResults])(implicit settings: AppSettings): Result[Seq[ResultMetricTrend]]

    Finalizes trend metric results: selects only trend metrics results and converts them to final trend metric results representation ready for writing into storage DB or sending via targets.

    Finalizes trend metric results: selects only trend metrics results and converts them to final trend metric results representation ready for writing into storage DB or sending via targets.

    stage

    Stage indication used for logging.

    metricResults

    Map with all metric results

    settings

    Implicit application settings object

    returns

    Either a finalized sequence of trend metric results or a list of conversion errors.

    Attributes
    protected
  20. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  21. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  22. def initLogger(lvl: Level): Unit

    Initialises logger:

    Initialises logger:

    • gets log4j properties with following priority: resources directory -> working directory -> default settings
    • updates root logger level with verbosity level defined at application start
    • reconfigures Logger
    lvl

    Root logger level defined at application start

    Definition Classes
    Logging
  23. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  24. val loadCheckStage: String
    Attributes
    protected
  25. val loadChecksBySources: Map[String, Seq[LoadCheckConfig]]
    Attributes
    protected
  26. lazy val log: Logger
    Definition Classes
    Logging
    Annotations
    @transient()
  27. def logMetricResults(stage: String, metType: String, mr: MetricResults): Unit

    Logs metric calculation results.

    Logs metric calculation results. Used during metric processing, to immediately log their calculation status.

    stage

    Stage indication used for logging.

    metType

    Type of the metrics being calculated (either regular or composed)

    mr

    Map with metric results

    Attributes
    protected
  28. implicit val manager: Option[DqStorageManager]
  29. val metricStage: String
    Attributes
    protected
  30. val metricsBySources: Map[String, Seq[RegularMetricConfig]]
    Attributes
    protected
  31. val metricsMap: Map[String, RegularMetricConfig]
    Attributes
    protected
  32. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  33. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  34. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  35. def performChecks(stage: String, metricResults: Result[MetricResults])(implicit settings: AppSettings): Result[Seq[ResultCheck]]

    Performs all checks

    Performs all checks

    stage

    Stage indication used for logging.

    metricResults

    Map with metric results (both regular and composed)

    settings

    Implicit application settings object

    returns

    Either sequence of check results or a list of check evaluation errors.

    Attributes
    protected
  36. def performLoadChecks(stage: String)(implicit settings: AppSettings): Seq[ResultCheckLoad]

    Performs all load checks

    Performs all load checks

    stage

    Stage indication used for logging.

    settings

    Implicit application settings object

    returns

    Sequence of load check results.

    Attributes
    protected
  37. def processAll(regularMetricsProcessor: RegularMetricsProcessor, stagePrefix: Option[String] = None)(implicit settings: AppSettings): Result[ResultSet]

    Top-level processing function: aggregates and runs all processing stages in required order.

    Top-level processing function: aggregates and runs all processing stages in required order.

    regularMetricsProcessor

    Regular metric processor used to calculate regular metric results.

    stagePrefix

    Prefix to stage names. Used for logging in streaming applications to indicate window for which results are processed.

    settings

    Implicit application settings object

    returns

    Either final results set or a list of processing errors

    Attributes
    protected
  38. def processTargets(stage: String, resultSet: Result[ResultSet])(implicit settings: AppSettings): Result[Unit]

    Processes all targets

    Processes all targets

    stage

    Stage indication used for logging.

    resultSet

    Final results set

    settings

    Implicit application settings object

    returns

    Either unit or a list of target processing errors.

    Attributes
    protected
  39. def runStorageMigration(stage: String)(implicit settings: AppSettings): Result[String]

    Runs database migration provided with storage manager.

    Runs database migration provided with storage manager.

    stage

    Stage indication used for logging.

    settings

    Implicit application settings object

    returns

    Nothing in case of successful migration or a list of migration errors.

    Attributes
    protected
  40. def saveResults(stage: String, resultSet: Result[ResultSet])(implicit settings: AppSettings): Result[String]

    Saves results into Data Quality storage

    Saves results into Data Quality storage

    stage

    Stage indication used for logging.

    resultSet

    Final results set

    returns

    Either a status string or a list of saving errors.

    Attributes
    protected
  41. val storageStage: String
    Attributes
    protected
  42. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  43. val targetsStage: String
    Attributes
    protected
  44. def toString(): String
    Definition Classes
    AnyRef → Any
  45. val trendMetricsMap: Map[String, TrendMetricConfig]
    Attributes
    protected
  46. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  47. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  48. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped