Parquet Compatibility#

Comet’s Parquet scan offloads decoding to native code and produces Arrow batches for the rest of the plan. Comet falls back to Spark when the scan cannot be converted (for example, due to one of the unsupported features listed below).

Parquet Scan Limitations#

The following features are not supported and cause Comet to fall back to Spark:

  • Decimals encoded in binary format.

  • ShortType columns, by default. When reading Parquet files written by systems other than Spark that contain columns with the logical type UINT_8 (unsigned 8-bit integers), Comet may produce different results than Spark. Spark maps UINT_8 to ShortType, but Comet’s Arrow-based readers respect the unsigned type and read the data as unsigned rather than signed. Since Comet cannot distinguish ShortType columns that came from UINT_8 versus signed INT16, by default Comet falls back to Spark when scanning Parquet files containing ShortType columns. This behavior can be disabled by setting spark.comet.scan.unsignedSmallIntSafetyCheck=false. Note that ByteType columns are always safe because they can only come from signed INT8, where truncation preserves the signed value.

  • Default values that are nested types (e.g., maps, arrays, structs). Literal default values are supported.

  • Spark’s Datasource V2 API. When spark.sql.sources.useV1SourceList does not include parquet, Spark uses the V2 API for Parquet scans. Comet’s Parquet scan only supports the V1 API.

  • _metadata.row_index, _metadata.file_block_start and _metadata.file_block_length. When Spark splits a file, the native scan and Spark can read a row group in different splits, so the block values would differ. Other _metadata columns (file_path, file_name, file_size, file_modification_time) are supported.

  • No support for input_file_name(), input_file_block_start(), or input_file_block_length() SQL functions. Comet’s Parquet scan does not use Spark’s FileScanRDD, so these functions cannot populate their values.

  • No support for ignoreMissingFiles or ignoreCorruptFiles being set to true

  • spark.sql.parquet.enableVectorizedReader=false. Disabling the vectorized reader opts into Spark’s parquet-mr semantics (silent overflow, null-on-narrowing), which Comet’s native reader does not replicate. By default Comet falls back to Spark in this case. Set spark.comet.scan.allowDisabledParquetVectorizedReader=true to opt in to running the Comet Parquet scan regardless.

  • A read schema that repeats a Parquet field id, at the top level or within a struct, when spark.sql.parquet.fieldId.read.enabled=true.

The following limitation may produce incorrect results without falling back to Spark:

  • No support for datetime rebasing. When reading Parquet files containing dates or timestamps written with spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY (which is Spark’s default for data written before Spark 3.0, using the hybrid Julian/Gregorian calendar), Comet reads them as if they were written using the Proleptic Gregorian calendar. This produces silently-wrong values for dates before October 15, 1582 in both projections and predicates. Comet also ignores spark.sql.parquet.datetimeRebaseModeInRead and the file-level org.apache.spark.legacyDateTime metadata that would tell it to rebase. Tracked by #5010.

The following limitations raise an error at scan time rather than falling back to Spark:

  • Selecting a field by name when multiple physical siblings match, including inside structs, arrays, and maps. Comet raises a duplicate-field error instead of resolving the collision. Checks cover referenced columns, including predicates; unselected roots do not prevent reading a unique field by name or field ID. Exact-name projections of unique children in structs and arrays of structs remain supported. Casts that cannot use this pruning reject byte-identical duplicate siblings anywhere in the decoded physical subtree, including maps. Field-ID resolution retains precedence, but selecting a byte-identically duplicated physical root name still raises a duplicate-field error, even when the requested field is renamed. Names in separate groups do not collide. Spark may read a duplicate-bearing file with an explicit schema in case-sensitive mode, but its choice of sibling depends on the field shape and can produce unexpected values. Spark rejects schema inference from a single file with duplicate names; inference across files can depend on merge order. Resolution is tracked in #5884, with mixed-type behavior in #5964.

  • Invalid UTF-8 bytes in STRING columns. Spark permits arbitrary byte sequences in a STRING column (for example from CAST(X'C1' AS STRING)), but Comet’s native execution path is built on Arrow, whose string type is strictly UTF-8. Reading a Parquet file whose STRING column contains non-UTF-8 bytes fails with Parquet error: encountered non UTF-8 data. Disable Comet for the query, or cast the column to BINARY before persisting, if you need to preserve non-UTF-8 bytes. By contrast, Comet decodes non-UTF-8 bytes at the JVM-to-native Arrow FFI boundaries using JVM-compatible replacement semantics. This covers native query input from the JVM-exported Arrow stream (including JVM scans, shuffle reads, and mapInArrow), columnar-to-row conversion, and JVM UDF results. It does not change the native Parquet reader: rejecting invalid UTF-8 there remains Gap A of #4764. See Strings with non-UTF-8 bytes, #4121, and the tracking issue above.

  • Reading TimestampLTZ as TimestampNTZ on Spark 3.x. Spark raises an error per SPARK-36182 because LTZ encodes UTC-adjusted instants that cannot be safely reinterpreted as timezone-free values, and Comet matches this by rejecting the read. This applies to all LTZ physical encodings (INT96, TIMESTAMP_MICROS, TIMESTAMP_MILLIS). On Spark 4.0+, this read is permitted (SPARK-47447) and Comet matches Spark’s behavior.

Schema Mismatch Handling#

The issues in this subsection apply only when the requested read schema differs from the schema written to the Parquet file. They do not affect a plain spark.read.parquet(path) that infers the schema from file metadata, because in that case the requested schema and file schema match by construction. Schema mismatch happens in two real-world scenarios:

  1. The user provides an explicit read schema: spark.read.schema(<schema>).parquet(path) (or the equivalent DataFrame API).

  2. Schema evolution / partitioned reads where files in a single dataset were written at different times with different types, or a table-format catalog (Iceberg, Delta) records a logical schema that has evolved past one or more underlying Parquet files. Spark coerces the file types to the table types at read time.

Spark’s vectorized Parquet reader fully validates these conversions in ParquetVectorUpdaterFactory.getUpdater and throws SchemaColumnConvertNotSupportedException for unsupported pairs. Comet’s Parquet scan mirrors that validation in its schema adapter; the entries below are the remaining gaps.

Note that the exact set of accepted conversions has changed between Spark versions (for example, Spark 3.x’s schemaEvolution.enabled flag gates INT32 → INT64, FLOAT → DOUBLE, and INT32 → DOUBLE widening that Spark 4.0+ accepts unconditionally; TimestampLTZ → TimestampNTZ is rejected by Spark 3.x but accepted by Spark 4.0+). Comet aims to follow the per-version Spark behavior.

  • Conversion errors precede pushed row filters. Empty files and files whose row groups or pages are all pruned read without decode-time conversion errors. When row-filter pushdown is enabled, Comet checks rejected conversions before reading surviving data pages, so a row filter cannot suppress the error by discarding every row. Legacy LIST shape mismatches that Spark cannot clip are rejected when the file is opened, including for empty and fully pruned files.

  • List conversion error paths assume Spark’s standard encoding. Comet inserts list before the element name when reporting a rejected array element conversion. Arrow’s schema omits the repeated group name, so paths for legacy LIST encodings or custom group names may differ from Spark’s Parquet column path.

  • ParquetSchemaConvert errors do not include the file path. The mismatch itself is detected and rejected correctly, but the resulting Spark error message reads Encountered error while reading file . Data type mismatches… (note the empty path). Behavior is consistent across Spark versions. See #4316.

  • Spark 3.x: extra SparkException layer in the cause chain. The native error is translated to a SparkException whose cause is SchemaColumnConvertNotSupportedException (matching what Spark would throw); on Spark 3.x the executor / task error handling re-wraps this once more on the way back to the driver, producing a two-level chain (SparkException → SparkException → SchemaColumnConvertNotSupportedException) instead of the one-level chain Spark’s own vectorized reader produces. Code that catches SparkException and inspects only the immediate cause via e.getCause.isInstanceOf[SchemaColumnConvertNotSupportedException] will see the inner SparkException instead. Walk the cause chain to recover the SchemaColumnConvertNotSupportedException. Spark 4.0+ produces a single-level chain, matching vanilla Spark. See #4354.