Packages

case class Source(id: String, df: DataFrame, keyFields: Seq[String] = Seq.empty, parents: Seq[String] = Seq.empty, checkpoint: Option[Checkpoint] = None) extends Product with Serializable

Data Quality Source definition

id

Source ID

df

Spark dataframe with source data

keyFields

Key fields (columns) of this source: uniquely define data row

parents

Sequence of parent sources IDs from which this source is build (applies to virtual sources)

checkpoint

Initial source checkpoint. Applicable only for streaming sources.

Note

Source can hold both static and streaming dataframes and, therefore, both batch source readers and stream source readers return same source definition.

Linear Supertypes
Serializable, Serializable, Product, Equals, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Source
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new Source(id: String, df: DataFrame, keyFields: Seq[String] = Seq.empty, parents: Seq[String] = Seq.empty, checkpoint: Option[Checkpoint] = None)

    id

    Source ID

    df

    Spark dataframe with source data

    keyFields

    Key fields (columns) of this source: uniquely define data row

    parents

    Sequence of parent sources IDs from which this source is build (applies to virtual sources)

    checkpoint

    Initial source checkpoint. Applicable only for streaming sources.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. val checkpoint: Option[Checkpoint]
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  7. val df: DataFrame
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. val id: String
  12. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  13. val isStreaming: Boolean
  14. val keyFields: Seq[String]
  15. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  18. val parents: Seq[String]
  19. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  20. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  21. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped