Package org.apache.sysds.runtime.io

Class DeltaKernelUtils.ScanHandle

java.lang.Object
org.apache.sysds.runtime.io.DeltaKernelUtils.ScanHandle
Enclosing class:
DeltaKernelUtils
public static final class DeltaKernelUtils.ScanHandle extends Object
Opened latest snapshot of a Delta table: the logical schema plus everything needed to (re)read its data files, including the list of per-data-file scan rows. Delta Kernel scan-file rows are self-contained (the kernel's distributed design serializes them to workers), so they can be retained and read independently / in parallel.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    final boolean[]
    hasDeletionVector
    Per-file flag indicating a deletion vector is present (so the live row count differs from DeltaKernelUtils.numRecords(io.delta.kernel.data.Row)), aligned with scanFiles.
    final long[]
    numRecords
    Per-file record counts taken from the Delta numRecords statistic, aligned with scanFiles; -1 where the statistic is absent.
    final io.delta.kernel.types.StructType
    physicalReadSchema
     
    final List<io.delta.kernel.data.Row>
    scanFiles
     
    final io.delta.kernel.data.Row
    scanState
     
    final io.delta.kernel.types.StructType
    schema
     

  • Method Summary

    Modifier and Type
    Method
    Description
    boolean
    hasExactRowCounts()
     

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • schema

      public final io.delta.kernel.types.StructType schema

    • scanState

      public final io.delta.kernel.data.Row scanState

    • physicalReadSchema

      public final io.delta.kernel.types.StructType physicalReadSchema

    • scanFiles

      public final List<io.delta.kernel.data.Row> scanFiles

    • numRecords

      public final long[] numRecords
      Per-file record counts taken from the Delta numRecords statistic, aligned with scanFiles; -1 where the statistic is absent.

    • hasDeletionVector

      public final boolean[] hasDeletionVector
      Per-file flag indicating a deletion vector is present (so the live row count differs from DeltaKernelUtils.numRecords(io.delta.kernel.data.Row)), aligned with scanFiles.

  • Method Details

    • hasExactRowCounts

      public boolean hasExactRowCounts()
      Returns:
      true iff every data file carries a numRecords statistic and none has a deletion vector, i.e. exact per-file row offsets can be derived from metadata without reading the data.