Value Members

1.  final def !=(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

2.  final def ##(): Int

   Definition Classes

   AnyRef → Any

3.  final def ==(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

4.  final def asInstanceOf[T0]: T0

   Definition Classes

   Any

5.  def checkConnection: Result[Unit]

   Checks connection.

   Checks connection.

   returns

   Nothing or error message in case if connection is not ready.

   Definition Classes

   KafkaConnection → DQConnection

6.  def clone(): AnyRef

   Attributes

   protected[lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... ) @native()

7.  val config: KafkaConnectionConfig
8.  final def eq(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

9.  def finalize(): Unit

   Attributes

   protected[lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( classOf[java.lang.Throwable] )

10.  final def getClass(): Class[_]

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

11.  val id: String

    Definition Classes

    KafkaConnection → DQConnection

12.  def initCheckpoint(sourceConfig: SourceType): CheckpointType

    Creates initial checkpoint for provided Kafka source configuration.

    Creates initial checkpoint for provided Kafka source configuration.

    sourceConfig

    Kafka source configuration

    returns

    Kafka checkpoint

    Definition Classes

    KafkaConnection → DQStreamingConnection

13.  final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

14.  def loadDataFrame(sourceConfig: SourceType)(implicit settings: AppSettings, spark: SparkSession, schemas: Map[String, SourceSchema]): DataFrame

    Loads external data into dataframe given a source configuration

    Loads external data into dataframe given a source configuration

    sourceConfig

    Source configuration

    settings

    Implicit application settings object

    spark

    Implicit spark session object

    schemas

    Implicit Map of all explicitly defined schemas (schemaId -> SourceSchema)

    returns

    Spark DataFrame

    Definition Classes

    KafkaConnection → DQConnection

15.  def loadDataStream(sourceConfig: SourceType, checkpoint: CheckpointType)(implicit settings: AppSettings, spark: SparkSession, schemas: Map[String, SourceSchema]): DataFrame

    Loads stream into a dataframe given the stream configuration

    Loads stream into a dataframe given the stream configuration

    sourceConfig

    Source configuration

    checkpoint

    Checkpoint to start stream from.

    settings

    Implicit application settings object

    spark

    Implicit spark session object

    schemas

    Implicit Map of all explicitly defined schemas (schemaId -> SourceSchema)

    returns

    Spark Streaming DataFrame

    Definition Classes

    KafkaConnection → DQStreamingConnection

16.  final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

17.  final def notify(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

18.  final def notifyAll(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

19.  val sparkParams: Seq[String]

    Attributes

    protected

    Definition Classes

    KafkaConnection → DQConnection

20.  final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

21.  def validateOrFixCheckpoint(checkpoint: CheckpointType, sourceConfig: SourceType): CheckpointType

    Validates checkpoint structure and makes updates in case if checkpoint structure needs to be changed.

    Validates checkpoint structure and makes updates in case if checkpoint structure needs to be changed.

    checkpoint

    Checkpoint to validate and fix (if needed).

    sourceConfig

    Source configuration

    returns

    Either original checkpoint if it is valid or a fixed checkpoint.

    Definition Classes

    KafkaConnection → DQStreamingConnection

22.  final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

23.  final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

24.  final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... ) @native()

25.  def writeData(data: KafkaOutput): Result[String]

    Sends data to Kafka topic.

    Sends data to Kafka topic.

    data

    Data to be send

    returns

    Status of operation: either "Success" string or a list of errors.