object SourceReaders
- Alphabetic
- By Inheritance
- SourceReaders
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Type Members
- sealed trait SimpleFileReader extends AnyRef
-
sealed
trait
SourceReader[T <: SourceConfig] extends AnyRef
Base source reader trait
Base source reader trait
- T
Type of source configuration
-
implicit
class
SourceReaderOps[T <: SourceConfig] extends AnyRef
Implicit conversion for source configurations to enable read and readStream methods for them.
Implicit conversion for source configurations to enable read and readStream methods for them.
- T
Type of source configuration
Value Members
-
implicit
object
AnySourceReader extends SourceReader[SourceConfig]
Generic regular source reader that calls specific reader depending on the source configuration type.
-
implicit
object
AvroFileSourceReader extends SourceReader[AvroFileSourceConfig] with SimpleFileReader
Avro file source reader: reads avro file with optional explicit schema.
Avro file source reader: reads avro file with optional explicit schema.
- Note
In order to read avro file source it is required to provide map of source schemas.
- implicit object CustomSourceReader extends SourceReader[CustomSource]
-
implicit
object
DelimitedFileSourceReader extends SourceReader[DelimitedFileSourceConfig]
Delimited file source reader: reads delimited (csv, tsv, etc.) file.
Delimited file source reader: reads delimited (csv, tsv, etc.) file. Schema may be inferred from file header or provided explicitly in job configuration file and referenced in this source by its ID.
- Note
In order to read delimited file source it is required to provide map of source schemas
-
implicit
object
FixedFileSourceReader extends SourceReader[FixedFileSourceConfig]
Fixed file source reader: reads fixed-width file provided with schema ID corresponding to this file contents.
Fixed file source reader: reads fixed-width file provided with schema ID corresponding to this file contents.
- Note
In order to read fixed file source it is required to provide map of source schemas.
-
implicit
object
GreenplumSourceReader extends SourceReader[GreenplumSourceConfig]
Greenplum source reader: reads source from pivotal Connection
Greenplum source reader: reads source from pivotal Connection
- Note
In order to read greenplum source it is required to provided map of valid connections
-
implicit
object
HiveSourceReader extends SourceReader[HiveSourceConfig]
Hive source reader: reads table from Hive.
Hive source reader: reads table from Hive. Can also read only required partitions from table given the list partition columns and their values to read.
-
implicit
object
KafkaSourceReader extends SourceReader[KafkaSourceConfig]
Kafka source reader: reads topic from Kafka broker
Kafka source reader: reads topic from Kafka broker
- Note
In order to read kafka source it is required to provided map of valid connections
-
implicit
object
OrcFileSourceReader extends SourceReader[OrcFileSourceConfig] with SimpleFileReader
Orc file source reader: reads orc files.
-
implicit
object
ParquetFileSourceReader extends SourceReader[ParquetFileSourceConfig] with SimpleFileReader
Parquet file source reader: reads parquet files.
-
implicit
object
TableSourceReader extends SourceReader[TableSourceConfig]
Table source reader: reads source from JDBC Connection (Postgres, Oracle, etc)
Table source reader: reads source from JDBC Connection (Postgres, Oracle, etc)
- Note
In order to read table source it is required to provided map of valid connections