Skip to main content
Schema validation checks incoming JSON or XML files against a formal schema definition before processing. This catches structural issues, missing required fields, and type mismatches early in the pipeline.

​
Requirements

  • The source file must be JSON or XML format.
  • The validation schema file must be stored in MinIO in the {environment}-config bucket under validation-schema/{filename}.

​
Configuration

Set the validationSchema field in the dataQuality block to the filename of the schema. The pipeline resolves the file from the MinIO config bucket at {environment}-config/validation-schema/{filename}. A full s3://bucket/key URL is also accepted and used as-is.

​
Example Schema

Validation uses the Everit org.everit.json.schema library (1.5.1), which implements JSON Schema Draft 4 only. The $schema URI is ignored, and Draft 07-only keywords (const, if/then/else, propertyNames, numeric exclusiveMinimum/exclusiveMaximum as values) are silently ignored rather than enforced. Write schemas to the Draft 4 specification. A JSON Schema file stored in MinIO at {environment}-config/validation-schema/product_catalog_schema.json:

​
Behavior

  1. The pipeline loads the schema file from the configured path in the environment’s config bucket.
  2. The whole file is validated as a single document against the schema. A JSON file containing an array of records therefore needs a schema with "type": "array" and items describing each record; the example above validates a file that holds a single object.
  3. If any record fails validation, the file is rejected and processing stops.
  4. Validation errors are logged with details about which fields failed and why.

​
When to Use

Use schema validation when you need to enforce a strict contract on the structure of incoming JSON or XML data. This is particularly useful when data is provided by external systems where you do not control the format directly.