Parquet / Avro Schema Generator

Parquet is a columnar storage format optimized for analytical reads — great for SELECT queries on a subset of columns in data lakes (S3, GCS). Avro is a row-based format optimized for serialization and streaming — ideal for Kafka messages and event schemas. Rule of thumb: Avro for writing/streaming, Parquet for reading/analytics. Many pipelines use Avro in Kafka and convert to Parquet when landing in a data lake.

Schema evolution allows you to change the schema over time while maintaining backward/forward compatibility. Avro supports: adding optional fields (with defaults), removing optional fields, renaming via aliases. Breaking changes: changing field types, removing required fields, reordering fields. Use a Schema Registry (Confluent) to manage schema versions and enforce compatibility rules across producers and consumers.

JSON string → Avro 'string'. JSON number (integer) → Avro 'long' or 'int'. JSON number (float) → Avro 'double' or 'float'. JSON boolean → Avro 'boolean'. JSON null → Avro 'null'. JSON object → Avro 'record'. JSON array → Avro 'array'. Avro also supports 'bytes', 'fixed', 'enum', 'map', and 'union' types not directly representable in JSON.

JSON string → pa.string() or pa.large_string(). JSON integer → pa.int64(). JSON float → pa.float64(). JSON boolean → pa.bool_(). JSON null → optional field (pa.field nullable=True). JSON object → pa.struct(). JSON array → pa.list_(). PyArrow infers the best type from data. Use pa.schema() to enforce strict types and catch schema violations early in your pipeline.

Storing nested JSON as-is in Parquet (using struct/list types) preserves structure but complicates queries in SQL engines. Flattening (expanding nested fields to top-level columns) improves query performance and compatibility but loses the hierarchy. Best practice: flatten one level deep for frequently-queried fields, keep rare nested data as JSON string. BigQuery and Snowflake both support STRUCT natively, making nested Parquet efficient for those platforms.