Packages

o

org.checkita.dqf.readers

SourceReaders

object SourceReaders

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SourceReaders
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. sealed trait SimpleFileReader extends AnyRef
  2. sealed trait SourceReader[T <: SourceConfig] extends AnyRef

    Base source reader trait

    Base source reader trait

    T

    Type of source configuration

  3. implicit class SourceReaderOps[T <: SourceConfig] extends AnyRef

    Implicit conversion for source configurations to enable read and readStream methods for them.

    Implicit conversion for source configurations to enable read and readStream methods for them.

    T

    Type of source configuration

Value Members

  1. implicit object AnySourceReader extends SourceReader[SourceConfig]

    Generic regular source reader that calls specific reader depending on the source configuration type.

  2. implicit object AvroFileSourceReader extends SourceReader[AvroFileSourceConfig] with SimpleFileReader

    Avro file source reader: reads avro file with optional explicit schema.

    Avro file source reader: reads avro file with optional explicit schema.

    Note

    In order to read avro file source it is required to provide map of source schemas.

  3. implicit object CustomSourceReader extends SourceReader[CustomSource]
  4. implicit object DelimitedFileSourceReader extends SourceReader[DelimitedFileSourceConfig]

    Delimited file source reader: reads delimited (csv, tsv, etc.) file.

    Delimited file source reader: reads delimited (csv, tsv, etc.) file. Schema may be inferred from file header or provided explicitly in job configuration file and referenced in this source by its ID.

    Note

    In order to read delimited file source it is required to provide map of source schemas

  5. implicit object FixedFileSourceReader extends SourceReader[FixedFileSourceConfig]

    Fixed file source reader: reads fixed-width file provided with schema ID corresponding to this file contents.

    Fixed file source reader: reads fixed-width file provided with schema ID corresponding to this file contents.

    Note

    In order to read fixed file source it is required to provide map of source schemas.

  6. implicit object GreenplumSourceReader extends SourceReader[GreenplumSourceConfig]

    Greenplum source reader: reads source from pivotal Connection

    Greenplum source reader: reads source from pivotal Connection

    Note

    In order to read greenplum source it is required to provided map of valid connections

  7. implicit object HiveSourceReader extends SourceReader[HiveSourceConfig]

    Hive source reader: reads table from Hive.

    Hive source reader: reads table from Hive. Can also read only required partitions from table given the list partition columns and their values to read.

  8. implicit object KafkaSourceReader extends SourceReader[KafkaSourceConfig]

    Kafka source reader: reads topic from Kafka broker

    Kafka source reader: reads topic from Kafka broker

    Note

    In order to read kafka source it is required to provided map of valid connections

  9. implicit object OrcFileSourceReader extends SourceReader[OrcFileSourceConfig] with SimpleFileReader

    Orc file source reader: reads orc files.

  10. implicit object ParquetFileSourceReader extends SourceReader[ParquetFileSourceConfig] with SimpleFileReader

    Parquet file source reader: reads parquet files.

  11. implicit object TableSourceReader extends SourceReader[TableSourceConfig]

    Table source reader: reads source from JDBC Connection (Postgres, Oracle, etc)

    Table source reader: reads source from JDBC Connection (Postgres, Oracle, etc)

    Note

    In order to read table source it is required to provided map of valid connections