o

org.checkita.dqf.readers.SourceReaders

FixedFileSourceReader

implicit object FixedFileSourceReader extends SourceReader[FixedFileSourceConfig]

Fixed file source reader: reads fixed-width file provided with schema ID corresponding to this file contents.

Note

In order to read fixed file source it is required to provide map of source schemas.

Linear Supertypes
SourceReader[FixedFileSourceConfig], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. FixedFileSourceReader
  2. SourceReader
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  14. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. def read(config: FixedFileSourceConfig)(implicit settings: AppSettings, spark: SparkSession, fs: FileSystem, schemas: Map[String, SourceSchema], connections: Map[String, DQConnection], checkpoints: Map[String, Checkpoint]): Result[Source]

    Safely reads source given source configuration.

    Safely reads source given source configuration.

    config

    Source configuration

    settings

    Implicit application settings object

    spark

    Implicit spark session object

    schemas

    Map of explicitly defined schemas (schemaId -> SourceSchema)

    connections

    Map of existing connection (connectionID -> DQConnection)

    checkpoints

    Map of initial checkpoints read from checkpoint directory

    returns

    Either a valid Source or a list of source reading errors.

    Definition Classes
    SourceReader
  16. def readStream(config: FixedFileSourceConfig)(implicit settings: AppSettings, spark: SparkSession, fs: FileSystem, schemas: Map[String, SourceSchema], connections: Map[String, DQConnection], checkpoints: Map[String, Checkpoint]): Result[Source]

    Safely reads streaming source given source configuration.

    Safely reads streaming source given source configuration.

    config

    Source configuration (source must be streamable)

    settings

    Implicit application settings object

    spark

    Implicit spark session object

    schemas

    Map of explicitly defined schemas (schemaId -> SourceSchema)

    connections

    Map of existing connection (connectionID -> DQConnection)

    checkpoints

    Map of initial checkpoints read from checkpoint directory

    returns

    Either a valid Source or a list of source reading errors.

    Definition Classes
    SourceReader
  17. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  18. def toSource(config: FixedFileSourceConfig, df: DataFrame, readMode: ReadMode, checkpoint: Option[Checkpoint] = None)(implicit settings: AppSettings): Source

    Wraps spark dataframe into Source instance.

    Wraps spark dataframe into Source instance.

    config

    Source configuration

    df

    Spark Dataframe

    readMode

    Mode in which source is read. Either 'batch' or 'stream'

    checkpoint

    Initial source checkpoint (applicable only to streaming sources)

    returns

    Source

    Attributes
    protected
    Definition Classes
    SourceReader
  19. def toString(): String
    Definition Classes
    AnyRef → Any
  20. def tryToRead(config: FixedFileSourceConfig, readMode: ReadMode)(implicit settings: AppSettings, spark: SparkSession, fs: FileSystem, schemas: Map[String, SourceSchema], connections: Map[String, DQConnection], checkpoints: Map[String, Checkpoint]): Source

    Tries to read fixed file source given the source configuration.

    Tries to read fixed file source given the source configuration.

    config

    Fixed file source configuration

    readMode

    Mode in which source is read. Either 'batch' or 'stream'

    settings

    Implicit application settings object

    spark

    Implicit spark session object

    schemas

    Map of explicitly defined schemas (schemaId -> SourceSchema)

    connections

    Map of existing connection (connectionID -> DQConnection)

    checkpoints

    Map of initial checkpoints read from checkpoint directory

    returns

    Source

    Definition Classes
    FixedFileSourceReader → SourceReader
    Note

    When read in stream mode, Spark will stream newly added files only.

  21. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  23. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from SourceReader[FixedFileSourceConfig]

Inherited from AnyRef

Inherited from Any

Ungrouped