Skip to main content

​
Prefer protect: {"method": "drop"}

To keep a column out of every later stage and destination, mark it on the source schema with "protect": {"method": "drop"}. It is removed right after the preprocessor, before data quality, transformation, and any AI stage, and it works for every destination, including the object store, MongoDB, and JSON pipelines (top-level keys). See Field Protection for the other methods (hmac, mask, redact).

​
Destination schema projection

You can also exclude columns from the output by defining a destination schema that omits unwanted columns. No special configuration is needed — simply remove the columns from the destination schema that you don’t want in the output.

​
How It Works

For the PostgreSQL, Snowflake, Databricks, and SQL Server destinations, when a destination schema (destination.schemaProperties) is defined, only the columns listed in it are written to the destination. Columns present in the source but absent from the destination schema are dropped. These loaders project each row onto the destination columns by name.

​
Example

Given source data with columns id, name, email, internal_code, created_at:
In this example, internal_code and created_at are present in the source but excluded from the destination schema, so they are dropped from the output.

​
Notes

  • Column dropping applies to delimited/CSV row data loaded into PostgreSQL, Snowflake, Databricks, or SQL Server, where each row is projected to the destination columns by name. The object store (Parquet, ORC, or Iceberg) destination maps values to destination fields by position, so its destination schema must list the same columns in the same order as the source. MongoDB and the vector destinations do not use per-column projection. JSON and XML pipelines use a single raw _json/_xml document that passes through without per-column projection, so column dropping does not apply to them.
  • The destination schema field names must match source schema field names (case-insensitive)
  • Column order in the destination schema determines the output column order