Class DatasetWriterBase<TRow>

Namespace

Plank.Dataset

Assembly

Plank.dll

Provides the bounded partition state used by generated dataset writers.

public abstract class DatasetWriterBase<TRow>

Type Parameters

TRow

The generated schema row type.

Inheritance

object

DatasetWriterBase<TRow>

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

This unstable API supports Plank-generated code and is not intended for direct use.

Constructors

DatasetWriterBase(ParquetSchema, RowApiColumnDescriptor[], int, IParquetWriteSource[], DatasetFilePath, DatasetWriterOptions)

Initializes the state used by a generated dataset writer that can create file parts.

protected DatasetWriterBase(ParquetSchema schema, RowApiColumnDescriptor[] columns, int rowBufferCapacity, IParquetWriteSource[] files, DatasetFilePath filePath, DatasetWriterOptions options)

Parameters

schema ParquetSchema

The one schema shared by all files in the dataset.

columns RowApiColumnDescriptor[]

The generated column descriptors for the schema.

rowBufferCapacity int

The row capacity of each generated buffer slot.

files IParquetWriteSource[]

The fixed reusable file sources.

filePath DatasetFilePath

Selects the path of each produced file.

options DatasetWriterOptions

The dataset writer options.

DatasetWriterBase(ParquetSchema, RowApiColumnDescriptor[], int, IParquetWriteSource[], DatasetWriterOptions)

Initializes the state used by a generated dataset writer.

protected DatasetWriterBase(ParquetSchema schema, RowApiColumnDescriptor[] columns, int rowBufferCapacity, IParquetWriteSource[] files, DatasetWriterOptions options)

Parameters

schema ParquetSchema

The one schema shared by all files in the dataset.

columns RowApiColumnDescriptor[]

The generated column descriptors for the schema.

rowBufferCapacity int

The row capacity of each generated buffer slot.

files IParquetWriteSource[]

The fixed reusable file sources.

options DatasetWriterOptions

The dataset writer options.

Methods

CopyRow(TRow, int, int)

Copies one schema row into the dataset column buffers.

protected abstract void CopyRow(TRow row, int slotIndex, int rowIndex)

Parameters

row TRow

The source schema row.

slotIndex int

The destination slot identifier.

rowIndex int

The destination row index.

DisposeDataset()

Writes all pending rows and closes all active files.

protected void DisposeDataset()

InitializeSlots()

Creates the fixed set of generated row buffers.

protected void InitializeSlots()

QueueRow(TRow)

Routes and queues one row.

protected void QueueRow(TRow row)

Parameters

row TRow

The row to queue.

SelectPath(TRow, IParquetBufferPool, out ParquetBuffer?)

Gets the UTF-8 path for one schema row.

protected abstract ReadOnlySpan<byte> SelectPath(TRow row, IParquetBufferPool bufferPool, out ParquetBuffer? allocation)

Parameters

row TRow

The schema row.

bufferPool IParquetBufferPool

The writer buffer pool.

allocation ParquetBuffer?

The optional allocation that owns the returned path. Set it to null when the path has another owner.

Returns

ReadOnlySpan<byte>

The full UTF-8 file path.

SetColumnValue<T>(int, int, int, T)

Sets one typed value in a dataset column buffer.

protected void SetColumnValue<T>(int slotIndex, int columnIndex, int rowIndex, T value)

Parameters

slotIndex int

The destination slot identifier.

columnIndex int

The destination column index.

rowIndex int

The destination row index.

value T

The value to set.

Type Parameters

T

The generated column value type.