Type Members

1.  final case class AvroSchemaConfig(id: ID, description: Option[NonEmptyString], schema: URI, validateDefaults: Boolean = false, metadata: Seq[SparkParam] = Seq.empty) extends SchemaConfig with Product with Serializable

   Avro schema configuration

   Avro schema configuration

   id

   Schema ID

   description

   Schema description

   schema

   Path to Avro schema file (.avsc)

   validateDefaults

   Boolean flag enabling or disabling default values validation in Avro schema.

   metadata

   List of metadata parameters specific to this schema

2.  sealed abstract class Column extends AnyRef

   Base class for column retrieved from explicit schemas

3.  final case class DelimitedSchemaConfig(id: ID, description: Option[NonEmptyString], schema: Refined[Seq[GeneralColumn], NonEmpty], metadata: Seq[SparkParam] = Seq.empty) extends SchemaConfig with Product with Serializable

   Delimited schema configuration used for delimited files such as CSV or TSV.

   Delimited schema configuration used for delimited files such as CSV or TSV.

   id

   Schema ID

   description

   Schema description

   schema

   List of columns in order.

   metadata

   List of metadata parameters specific to this schema

4.  final case class FixedFullColumn(name: NonEmptyString, type: DataType, width: PositiveInt) extends Column with Product with Serializable

   Fixed column configuration with full definition (both column type and width are defined).

   Fixed column configuration with full definition (both column type and width are defined). Used to defined columns for fixed-width schema.

   name

   Column name

   width

   Column width

5.  final case class FixedFullSchemaConfig(id: ID, description: Option[NonEmptyString], schema: Refined[Seq[FixedFullColumn], NonEmpty], metadata: Seq[SparkParam] = Seq.empty) extends SchemaConfig with Product with Serializable

   Fixed schema configuration used for files with fixed column width.

   Fixed schema configuration used for files with fixed column width. Columns are fully defined with their name, type and width.

   id

   Schema ID

   description

   Schema description

   schema

   List of columns in order

   metadata

   List of metadata parameters specific to this schema

6.  final case class FixedShortSchemaConfig(id: ID, description: Option[NonEmptyString], schema: Refined[Seq[FixedShortColumn], NonEmpty], metadata: Seq[SparkParam] = Seq.empty) extends SchemaConfig with Product with Serializable

   Fixed schema configuration used for files with fixed column width.

   Fixed schema configuration used for files with fixed column width. Columns are defined in short notation with their name and width only. All columns have StringType.

   id

   Schema ID

   description

   Schema description

   schema

   List of columns in order (format is "column_name:width", e.g. "zip:5")

   metadata

   List of metadata parameters specific to this schema

7.  final case class GeneralColumn(name: NonEmptyString, type: DataType) extends Column with Product with Serializable

   General column configuration used to defined columns for delimited schema

   General column configuration used to defined columns for delimited schema

   name

   Column name

8.  final case class HiveSchemaConfig(id: ID, description: Option[NonEmptyString], schema: NonEmptyString, table: NonEmptyString, excludeColumns: Seq[NonEmptyString] = Seq.empty, metadata: Seq[SparkParam] = Seq.empty) extends SchemaConfig with Product with Serializable

   Schema configuration that is read from hive catalog

   Schema configuration that is read from hive catalog

   id

   Schema ID

   description

   Schema description

   schema

   Hive Schema

   table

   Hive Table

   excludeColumns

   Columns to exclude from schema (e.g. it might be necessary to exclude table partitioning columns)

   metadata

   List of metadata parameters specific to this schema

9.  final case class RegistrySchemaConfig(id: ID, description: Option[NonEmptyString], baseUrls: NonEmptyURISeq, schemaId: Option[Int], schemaSubject: Option[NonEmptyString], version: Option[Int], validateDefaults: Boolean = false, properties: Seq[SparkParam] = Seq.empty, headers: Seq[SparkParam] = Seq.empty, metadata: Seq[SparkParam] = Seq.empty, connectionTimeoutMs: Int = 60000, retryAttempts: Int = 3, retryIntervalMs: Int = 5000) extends SchemaConfig with Product with Serializable

   Schema configuration that is used to read schema from Confluent Schema Registry.

   Schema configuration that is used to read schema from Confluent Schema Registry.

   id

   Schema ID

   description

   Schema description

   baseUrls

   List of urls to connect to Schema Registry

   schemaId

   Schema ID to search in schema registry

   schemaSubject

   Schema subject to search in schema registry

   version

   Schema version (by default latest available version is fetched)

   validateDefaults

   Boolean flag enabling or disabling default values validation in Avro schema.

   properties

   List of additional connection properties: sequence of strings in format key=value.

   headers

   List of additional HTML headers: sequence of strings in format key=value.

   metadata

   List of metadata parameters specific to this schema

   connectionTimeoutMs

   Maximum time in milliseconds to wait for a response from the Schema Registry.

   retryAttempts

   Number of retry attempts in case of a failure.

   retryIntervalMs

   Delay in milliseconds between retry attempts.

10.  sealed abstract class SchemaConfig extends JobConfigEntity

    Base class for all schema configurations.

    Base class for all schema configurations. All schemas are described as DQ entities.