Logical read layer

The logical read layer decodes parquet column data into typed C# values. It exposes values a column at a time in temporary decoded buffers. Buffer boundaries are independent of the file's physical page boundaries.

Use it when you want decoded values without constructing rows. If you need raw page bytes and encoding metadata instead, use the physical read layer.

Read with a schema

When you know which columns you need, use a reader generated from a schema. It binds columns by name rather than file order, skips columns outside the schema, and exposes typed column properties on each row group:

using Plank.Reading.Logical;

using var stream = File.OpenRead("events.parquet");
using EventSchema.Reader reader = EventSchema.CreateReader(stream);

foreach (EventSchema.ReadRowGroup rowGroup in reader.RowGroups)
    foreach (ColumnBuffer<int> buffer in rowGroup.IdColumn)
        foreach (int id in buffer.Values)
            Console.WriteLine(id);

If a schema is built at runtime instead, select its leaves through schema.LeafColumns.

Strict validation is enabled by default. A requested column must exist in the file and have a compatible physical type, logical type, repetition, and fixed length. A required file column may be requested as optional, but an optional file column cannot be requested as required.

Set ParquetReaderOptions.Strict to false only when the caller will handle schema mismatches itself.

Read without a schema

When the schema is not known ahead of time, let the reader discover it from the file:

using Plank.Reading.Logical;
using Plank.Schema;

using var stream = File.OpenRead("events.parquet");
using ParquetReader reader = new();

reader.Reset(stream);

foreach (LeafColumn column in reader.Schema.LeafColumns)
    Console.WriteLine($"{column.Path}: {column.PhysicalType}");

Reset(Stream) reads the footer and binds the file schema. Schema is the schema used to read values, while Metadata.Schema always describes the complete schema stored in the file.

Read column values

Enumerate the buffers exposed by the generated column property:

using Plank.Reading.Logical;

foreach (EventSchema.ReadRowGroup rowGroup in reader.RowGroups)
{
    Console.WriteLine($"Rows: {rowGroup.RowCount}");

    foreach (ColumnBuffer<int> buffer in rowGroup.IdColumn)
        foreach (int id in buffer.Values)
            Console.WriteLine(id);
}

RowGroups can be enumerated or indexed. Each generated row group exposes its row count and a strongly typed RowGroupColumn<T> property for every schema property.

Note

Each ColumnBuffer<T> contains one decoded batch. A physical page may produce multiple buffers, so callers must not rely on page-sized or stable batch boundaries. Consume Values before advancing the column enumerator. The span is temporary and may refer to pooled storage that is reused for the next buffer.

Nullable schema properties generate nullable column types such as RowGroupColumn<int?>.

Read binary values

Binary columns use the same generic column API as other unmanaged values. Each generated binary column is a RowGroupColumn<byte>:

foreach (ColumnBuffer<byte> buffer in rowGroup.NameColumn)
{
    for (int i = 0; i < buffer.Count; i++)
    {
        if (buffer.IsNull(i))
            continue;

        Consume(buffer.GetValue(i));
    }
}

Each page stores its internal descriptor table and all referenced bytes in one pooled ParquetBuffer. An empty value has a zero-length span while IsNull(i) returns false; optional nulls return true.

For binary columns, Count is the number of logical byte arrays, GetValue(i) preserves their individual boundaries, and Values is the concatenated non-null byte payload.

Call buffer.GetValue(i).ToArray() only when an owning byte[] is required. That call is the explicit allocation boundary. Runtime schemas can request the same zero-allocation view with rowGroup.Column<byte>(column).

Retain a buffer

Value buffers can be retained beyond the current enumeration step:

using Plank.Reading.Logical;
using Plank.Writing;

foreach (ColumnBuffer<int> buffer in rowGroup.IdColumn)
{
    using ParquetBuffer retained = buffer.Retain();
    foreach (int id in retained.AsSpan<int>())
        Console.WriteLine(id);
}

Retain returns a reference-counted ParquetBuffer; dispose it when it is no longer needed.

Keep a binary ColumnBuffer<byte> value and its retained lease together while accessing values:

ColumnBuffer<byte> buffer = binaryBuffers.Current;
using ParquetBuffer retained = buffer.Retain();
Consume(buffer.GetValue(0));
Edit this page