final case class CustomSource(id: ID, description: Option[NonEmptyString], format: NonEmptyString, path: Option[URI], schema: Option[ID], persist: Option[StorageLevel], options: Seq[SparkParam] = Seq.empty, keyFields: Seq[NonEmptyString] = Seq.empty, metadata: Seq[SparkParam] = Seq.empty) extends SourceConfig with Product with Serializable

Custom source configuration: used to read from source types that are not supported explicitly.

id

Source ID

description

Source description

format

Source format to set in spark reader.

path

Path to load the source from (if required)

schema

Explicit schema applied to source data (if required)

persist

Spark storage level in order to persist dataframe during job execution.

options

List of additional spark options required to read the source (if any)

keyFields

Sequence of key fields (columns that identify data row)

metadata

List of metadata parameters specific to this source

Linear Supertypes
Product, Equals, SourceConfig, JobConfigEntity, Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. CustomSource
  2. Product
  3. Equals
  4. SourceConfig
  5. JobConfigEntity
  6. Serializable
  7. Serializable
  8. AnyRef
  9. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new CustomSource(id: ID, description: Option[NonEmptyString], format: NonEmptyString, path: Option[URI], schema: Option[ID], persist: Option[StorageLevel], options: Seq[SparkParam] = Seq.empty, keyFields: Seq[NonEmptyString] = Seq.empty, metadata: Seq[SparkParam] = Seq.empty)

    id

    Source ID

    description

    Source description

    format

    Source format to set in spark reader.

    path

    Path to load the source from (if required)

    schema

    Explicit schema applied to source data (if required)

    persist

    Spark storage level in order to persist dataframe during job execution.

    options

    List of additional spark options required to read the source (if any)

    keyFields

    Sequence of key fields (columns that identify data row)

    metadata

    List of metadata parameters specific to this source

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. val description: Option[NonEmptyString]
    Definition Classes
    CustomSource → JobConfigEntity
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. val format: NonEmptyString
  10. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. val id: ID
    Definition Classes
    CustomSource → JobConfigEntity
  12. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  13. val keyFields: Seq[NonEmptyString]
    Definition Classes
    CustomSource → SourceConfig
  14. val metadata: Seq[SparkParam]
    Definition Classes
    CustomSource → JobConfigEntity
  15. val metadataString: Option[String]
    Definition Classes
    JobConfigEntity
  16. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  17. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  18. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  19. val options: Seq[SparkParam]
  20. val path: Option[URI]
  21. val persist: Option[StorageLevel]
    Definition Classes
    CustomSource → SourceConfig
  22. val schema: Option[ID]
  23. val streamable: Boolean
    Definition Classes
    CustomSource → SourceConfig
  24. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  25. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  26. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  27. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Product

Inherited from Equals

Inherited from SourceConfig

Inherited from JobConfigEntity

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped