Package org.apache.sysds.runtime.compress.colgroup.dictionary

Class MatrixBlockDictionary

java.lang.Object
org.apache.sysds.runtime.compress.colgroup.dictionary.ADictionary
org.apache.sysds.runtime.compress.colgroup.dictionary.MatrixBlockDictionary
All Implemented Interfaces:
Serializable, IDictionary
public class MatrixBlockDictionary extends ADictionary
See Also:

  • Method Details

    • create

      public static MatrixBlockDictionary create(MatrixBlock mb)

    • create

      public static MatrixBlockDictionary create(MatrixBlock mb, boolean check)

    • createDictionary

      public static MatrixBlockDictionary createDictionary(double[] values, int nCol, boolean check)

    • getMatrixBlock

      public MatrixBlock getMatrixBlock()

    • getValues

      public double[] getValues()
      Description copied from interface: IDictionary
      Get all the values contained in the dictionary as a linearized double array.
      Specified by:
      getValues in interface IDictionary
      Overrides:
      getValues in class ADictionary
      Returns:
      linearized double array

    • getValue

      public double getValue(int i)
      Description copied from interface: IDictionary
      Get Specific value contained in the dictionary at index.
      Specified by:
      getValue in interface IDictionary
      Overrides:
      getValue in class ADictionary
      Parameters:
      i - The index to extract the value from
      Returns:
      The value contained at the index

    • getValue

      public final double getValue(int r, int c, int nCol)
      Description copied from interface: IDictionary
      Get Specific value contain in dictionary at index.
      Specified by:
      getValue in interface IDictionary
      Overrides:
      getValue in class ADictionary
      Parameters:
      r - Row target
      c - Col target
      nCol - nCol in dictionary
      Returns:
      value

    • getInMemorySize

      public long getInMemorySize()
      Description copied from interface: IDictionary
      Returns the memory usage of the dictionary.
      Returns:
      a long value in number of bytes for the dictionary.

    • getInMemorySize

      public static long getInMemorySize(int numberValues, int numberColumns, double sparsity)

    • aggregate

      public double aggregate(double init, Builtin fn)
      Description copied from interface: IDictionary
      Aggregate all the contained values, useful in value only computations where the operation is iterating through all values contained in the dictionary.
      Specified by:
      aggregate in interface IDictionary
      Overrides:
      aggregate in class ADictionary
      Parameters:
      init - The initial Value, in cases such as Max value, this could be -infinity
      fn - The Function to apply to values
      Returns:
      The aggregated value as a double.

    • aggregateWithReference

      public double aggregateWithReference(double init, Builtin fn, double[] reference, boolean def)
      Description copied from interface: IDictionary
      Aggregate all the contained values, with a reference offset.
      Specified by:
      aggregateWithReference in interface IDictionary
      Overrides:
      aggregateWithReference in class ADictionary
      Parameters:
      init - The initial value, in cases such as Max value this could be -infinity.
      fn - The function to apply to the values
      reference - The reference offset to each value in the dictionary
      def - If the reference should be treated as an instance of only as reference
      Returns:
      The aggregated value as a double.

    • aggregateRows

      public double[] aggregateRows(Builtin fn, int nCol)
      Description copied from interface: IDictionary
      Aggregate all entries in the rows.
      Specified by:
      aggregateRows in interface IDictionary
      Overrides:
      aggregateRows in class ADictionary
      Parameters:
      fn - The aggregate function
      nCol - The number of columns contained in the dictionary.
      Returns:
      Aggregates for this dictionary tuples.

    • aggregateRowsWithDefault

      public double[] aggregateRowsWithDefault(Builtin fn, double[] defaultTuple)
      Description copied from interface: IDictionary
      Aggregate all entries in the rows of the dictionary with a extra cell in the end that contains the aggregate of the given defaultTuple.
      Specified by:
      aggregateRowsWithDefault in interface IDictionary
      Overrides:
      aggregateRowsWithDefault in class ADictionary
      Parameters:
      fn - The aggregate function
      defaultTuple - The default tuple to aggregate in last cell
      Returns:
      Aggregates for this dictionary tuples.

    • aggregateRowsWithReference

      public double[] aggregateRowsWithReference(Builtin fn, double[] reference)
      Description copied from interface: IDictionary
      Aggregate all entries in the rows with an offset value reference added.
      Specified by:
      aggregateRowsWithReference in interface IDictionary
      Overrides:
      aggregateRowsWithReference in class ADictionary
      Parameters:
      fn - The aggregate function
      reference - The reference offset to each value in the dictionary
      Returns:
      Aggregates for this dictionary tuples.

    • aggregateCols

      public void aggregateCols(double[] c, Builtin fn, IColIndex colIndexes)
      Description copied from interface: IDictionary
      Aggregates the columns into the target double array provided.
      Specified by:
      aggregateCols in interface IDictionary
      Overrides:
      aggregateCols in class ADictionary
      Parameters:
      c - The target double array, this contains the full number of columns, therefore the colIndexes for this specific dictionary is needed.
      fn - The function to apply to individual columns
      colIndexes - The mapping to the target columns from the individual columns

    • aggregateColsWithReference

      public void aggregateColsWithReference(double[] c, Builtin fn, IColIndex colIndexes, double[] reference, boolean def)
      Description copied from interface: IDictionary
      Aggregates the columns into the target double array provided.
      Specified by:
      aggregateColsWithReference in interface IDictionary
      Overrides:
      aggregateColsWithReference in class ADictionary
      Parameters:
      c - The target double array, this contains the full number of columns, therefore the colIndexes for this specific dictionary is needed.
      fn - The function to apply to individual columns
      colIndexes - The mapping to the target columns from the individual columns
      reference - The reference offset values to add to each cell.
      def - If the reference should be treated as a tuple as well

    • applyScalarOp

      public IDictionary applyScalarOp(ScalarOperator op)
      Description copied from interface: IDictionary
      Allocate a new dictionary and applies the scalar operation on each cell of to then return the new dictionary.
      Specified by:
      applyScalarOp in interface IDictionary
      Overrides:
      applyScalarOp in class ADictionary
      Parameters:
      op - The operator.
      Returns:
      The new dictionary to return.

    • applyScalarOpAndAppend

      public IDictionary applyScalarOpAndAppend(ScalarOperator op, double v0, int nCol)
      Description copied from interface: IDictionary
      Allocate a new dictionary with one extra row and applies the scalar operation on each cell of to then return the new dictionary.
      Specified by:
      applyScalarOpAndAppend in interface IDictionary
      Overrides:
      applyScalarOpAndAppend in class ADictionary
      Parameters:
      op - The operator
      v0 - The new value to put into each cell in the new row
      nCol - The number of columns in the dictionary
      Returns:
      The new dictionary to return.

    • applyUnaryOp

      public IDictionary applyUnaryOp(UnaryOperator op)
      Description copied from interface: IDictionary
      Allocate a new dictionary and apply the unary operator on each cell.
      Specified by:
      applyUnaryOp in interface IDictionary
      Overrides:
      applyUnaryOp in class ADictionary
      Parameters:
      op - The operator.
      Returns:
      The new dictionary to return.

    • applyUnaryOpAndAppend

      public IDictionary applyUnaryOpAndAppend(UnaryOperator op, double v0, int nCol)
      Description copied from interface: IDictionary
      Allocate a new dictionary with one extra row and apply the unary operator on each cell.
      Specified by:
      applyUnaryOpAndAppend in interface IDictionary
      Overrides:
      applyUnaryOpAndAppend in class ADictionary
      Parameters:
      op - The operator.
      v0 - The new value to put into each cell in the new row
      nCol - The number of columns in the dictionary
      Returns:
      The new dictionary to return.

    • applyScalarOpWithReference

      public IDictionary applyScalarOpWithReference(ScalarOperator op, double[] reference, double[] newReference)
      Description copied from interface: IDictionary
      Allocate a new dictionary and apply the scalar operation on each cell to then return a new dictionary. outValues[j] = op(this.values[j] + reference[i]) - newReference[i]
      Specified by:
      applyScalarOpWithReference in interface IDictionary
      Overrides:
      applyScalarOpWithReference in class ADictionary
      Parameters:
      op - The operator to apply to each cell.
      reference - The reference value to add before the operator.
      newReference - The reference value to subtract after the operator.
      Returns:
      A New Dictionary.

    • applyUnaryOpWithReference

      public IDictionary applyUnaryOpWithReference(UnaryOperator op, double[] reference, double[] newReference)
      Description copied from interface: IDictionary
      Allocate a new dictionary and apply the scalar operation on each cell to then return a new dictionary. outValues[j] = op(this.values[j] + reference[i]) - newReference[i]
      Specified by:
      applyUnaryOpWithReference in interface IDictionary
      Overrides:
      applyUnaryOpWithReference in class ADictionary
      Parameters:
      op - The unary operator to apply to each cell.
      reference - The reference value to add before the operator.
      newReference - The reference value to subtract after the operator.
      Returns:
      A New Dictionary.

    • binOpLeft

      public IDictionary binOpLeft(BinaryOperator op, double[] v, IColIndex colIndexes)
      Description copied from interface: IDictionary
      Apply binary row operation on the left side
      Specified by:
      binOpLeft in interface IDictionary
      Overrides:
      binOpLeft in class ADictionary
      Parameters:
      op - The operation to this dictionary
      v - The values to use on the left hand side.
      colIndexes - The column indexes to consider inside v.
      Returns:
      A new dictionary containing the updated values.

    • binOpLeftAndAppend

      public IDictionary binOpLeftAndAppend(BinaryOperator op, double[] v, IColIndex colIndexes)
      Description copied from interface: IDictionary
      Apply binary row operation on the left side with one extra row evaluating with zeros.
      Specified by:
      binOpLeftAndAppend in interface IDictionary
      Overrides:
      binOpLeftAndAppend in class ADictionary
      Parameters:
      op - The operation to this dictionary
      v - The values to use on the left hand side.
      colIndexes - The column indexes to consider inside v.
      Returns:
      A new dictionary containing the updated values.

    • binOpLeftWithReference

      public MatrixBlockDictionary binOpLeftWithReference(BinaryOperator op, double[] v, IColIndex colIndexes, double[] reference, double[] newReference)
      Description copied from interface: IDictionary
      Apply the binary operator such that each value is offset by the reference before application. Then put the result into the new dictionary, but offset it by the new reference. outValues[j] = op(v[colIndexes[i]], this.values[j] + reference[i]) - newReference[i]
      Specified by:
      binOpLeftWithReference in interface IDictionary
      Overrides:
      binOpLeftWithReference in class ADictionary
      Parameters:
      op - The operation to apply on the dictionary values.
      v - The values to use on the left side of the operator.
      colIndexes - The column indexes to use.
      reference - The reference value to add before operator.
      newReference - The reference value to subtract after operator.
      Returns:
      A new dictionary.

    • binOpRight

      public MatrixBlockDictionary binOpRight(BinaryOperator op, double[] v, IColIndex colIndexes)
      Description copied from interface: IDictionary
      Apply binary row operation on the right side.
      Specified by:
      binOpRight in interface IDictionary
      Overrides:
      binOpRight in class ADictionary
      Parameters:
      op - The operation to this dictionary
      v - The values to use on the right hand side.
      colIndexes - The column indexes to consider inside v.
      Returns:
      A new dictionary containing the updated values.

    • binOpRightAndAppend

      public IDictionary binOpRightAndAppend(BinaryOperator op, double[] v, IColIndex colIndexes)
      Description copied from interface: IDictionary
      Apply binary row operation on the right side with one extra row evaluating with zeros.
      Specified by:
      binOpRightAndAppend in interface IDictionary
      Overrides:
      binOpRightAndAppend in class ADictionary
      Parameters:
      op - The operation to this dictionary
      v - The values to use on the right hand side.
      colIndexes - The column indexes to consider inside v.
      Returns:
      A new dictionary containing the updated values.

    • binOpRight

      public MatrixBlockDictionary binOpRight(BinaryOperator op, double[] v)
      Description copied from interface: IDictionary
      Apply binary row operation on the right side as with no columns to extract from v.
      Specified by:
      binOpRight in interface IDictionary
      Overrides:
      binOpRight in class ADictionary
      Parameters:
      op - The operation to this dictionary
      v - The values to apply on the dictionary (same number of cols as the dictionary)
      Returns:
      A new dictionary containing the updated values.

    • binOpRightWithReference

      public MatrixBlockDictionary binOpRightWithReference(BinaryOperator op, double[] v, IColIndex colIndexes, double[] reference, double[] newReference)
      Description copied from interface: IDictionary
      Apply the binary operator such that each value is offset by the reference before application. Then put the result into the new dictionary, but offset it by the new reference. outValues[j] = op(this.values[j] + reference[i], v[colIndexes[i]]) - newReference[i]
      Specified by:
      binOpRightWithReference in interface IDictionary
      Overrides:
      binOpRightWithReference in class ADictionary
      Parameters:
      op - The operation to apply on the dictionary values.
      v - The values to use on the right side of the operator.
      colIndexes - The column indexes to use.
      reference - The reference value to add before operator.
      newReference - The reference value to subtract after operator.
      Returns:
      A new dictionary.

    • clone

      public IDictionary clone()
      Description copied from interface: IDictionary
      Returns a deep clone of the dictionary.
      Specified by:
      clone in interface IDictionary
      Specified by:
      clone in class ADictionary
      Returns:
      A deep clone

    • getDictType

      public IDictionary.DictType getDictType()
      Description copied from interface: IDictionary
      Get the dictionary type this dictionary is.
      Returns:
      The Dictionary type this is.

    • getNumberOfValues

      public int getNumberOfValues(int ncol)
      Description copied from interface: IDictionary
      Get the number of distinct tuples given that the column group has n columns
      Parameters:
      ncol - The number of Columns in the ColumnGroup.
      Returns:
      the number of value tuples contained in the dictionary.

    • getNumberOfColumns

      public int getNumberOfColumns(int nrow)
      Description copied from interface: IDictionary
      Get the number of columns in this dictionary, provided you know the number of values, or rows.
      Parameters:
      nrow - The number of rows/values known inside this dictionary
      Returns:
      The number of columns

    • sumAllRowsToDouble

      public double[] sumAllRowsToDouble(int nrColumns)
      Description copied from interface: IDictionary
      Method used as a pre-aggregate of each tuple in the dictionary, to single double values. Note if the number of columns is one the actual dictionaries values are simply returned.
      Specified by:
      sumAllRowsToDouble in interface IDictionary
      Overrides:
      sumAllRowsToDouble in class ADictionary
      Parameters:
      nrColumns - The number of columns in the ColGroup to know how to get the values from the dictionary.
      Returns:
      a double array containing the row sums from this dictionary.

    • sumAllRowsToDoubleWithDefault

      public double[] sumAllRowsToDoubleWithDefault(double[] defaultTuple)
      Description copied from interface: IDictionary
      Do exactly the same as the sumAllRowsToDouble but also sum the array given to a extra index in the end of the array.
      Specified by:
      sumAllRowsToDoubleWithDefault in interface IDictionary
      Overrides:
      sumAllRowsToDoubleWithDefault in class ADictionary
      Parameters:
      defaultTuple - The default row to sum in the end index returned.
      Returns:
      a double array containing the row sums from this dictionary.

    • sumAllRowsToDoubleWithReference

      public double[] sumAllRowsToDoubleWithReference(double[] reference)
      Description copied from interface: IDictionary
      Method used as a pre-aggregate of each tuple in the dictionary, to single double values with a reference.
      Specified by:
      sumAllRowsToDoubleWithReference in interface IDictionary
      Overrides:
      sumAllRowsToDoubleWithReference in class ADictionary
      Parameters:
      reference - The reference values to add to each cell.
      Returns:
      a double array containing the row sums from this dictionary.

    • sumAllRowsToDoubleSq

      public double[] sumAllRowsToDoubleSq(int nrColumns)
      Description copied from interface: IDictionary
      Method used as a pre-aggregate of each tuple in the dictionary, to single double values. Note if the number of columns is one the actual dictionaries values are simply returned.
      Specified by:
      sumAllRowsToDoubleSq in interface IDictionary
      Overrides:
      sumAllRowsToDoubleSq in class ADictionary
      Parameters:
      nrColumns - The number of columns in the ColGroup to know how to get the values from the dictionary.
      Returns:
      a double array containing the row sums from this dictionary.

    • sumAllRowsToDoubleSqWithDefault

      public double[] sumAllRowsToDoubleSqWithDefault(double[] defaultTuple)
      Description copied from interface: IDictionary
      Method used as a pre-aggregate of each tuple in the dictionary, to single double values. But adds another cell to the return with an extra value that is the sum of the given defaultTuple.
      Specified by:
      sumAllRowsToDoubleSqWithDefault in interface IDictionary
      Overrides:
      sumAllRowsToDoubleSqWithDefault in class ADictionary
      Parameters:
      defaultTuple - The default row to sum in the end index returned.
      Returns:
      a double array containing the row sums from this dictionary.

    • sumAllRowsToDoubleSqWithReference

      public double[] sumAllRowsToDoubleSqWithReference(double[] reference)
      Description copied from interface: IDictionary
      Method used as a pre-aggregate of each tuple in the dictionary, to single double values.
      Specified by:
      sumAllRowsToDoubleSqWithReference in interface IDictionary
      Overrides:
      sumAllRowsToDoubleSqWithReference in class ADictionary
      Parameters:
      reference - The reference values to add to each cell.
      Returns:
      a double array containing the row sums from this dictionary.

    • productAllRowsToDouble

      public double[] productAllRowsToDouble(int nCol)
      Description copied from interface: IDictionary
      Method to product all rows to a column vector.
      Specified by:
      productAllRowsToDouble in interface IDictionary
      Overrides:
      productAllRowsToDouble in class ADictionary
      Parameters:
      nCol - The number of columns in the ColGroup to know how to get the values from the dictionary.
      Returns:
      A row product

    • productAllRowsToDoubleWithDefault

      public double[] productAllRowsToDoubleWithDefault(double[] defaultTuple)
      Description copied from interface: IDictionary
      Method to product all rows to a column vector with a default value added in the end.
      Specified by:
      productAllRowsToDoubleWithDefault in interface IDictionary
      Overrides:
      productAllRowsToDoubleWithDefault in class ADictionary
      Parameters:
      defaultTuple - The default row that aggregate to last cell
      Returns:
      A row product

    • productAllRowsToDoubleWithReference

      public double[] productAllRowsToDoubleWithReference(double[] reference)
      Description copied from interface: IDictionary
      Method to product all rows to a column vector with a reference values added to all cells, and a reference product in the end
      Specified by:
      productAllRowsToDoubleWithReference in interface IDictionary
      Overrides:
      productAllRowsToDoubleWithReference in class ADictionary
      Parameters:
      reference - The reference row
      Returns:
      A row product

    • colSum

      public void colSum(double[] c, int[] counts, IColIndex colIndexes)
      Description copied from interface: IDictionary
      Get the column sum of the values contained in the dictionary
      Specified by:
      colSum in interface IDictionary
      Overrides:
      colSum in class ADictionary
      Parameters:
      c - The output array allocated to contain all column groups output.
      counts - The counts of the individual tuples.
      colIndexes - The columns indexes of the parent column group, this indicate where to put the column sum into the c output.

    • colSumSq

      public void colSumSq(double[] c, int[] counts, IColIndex colIndexes)
      Description copied from interface: IDictionary
      Get the column sum of the values contained in the dictionary
      Specified by:
      colSumSq in interface IDictionary
      Overrides:
      colSumSq in class ADictionary
      Parameters:
      c - The output array allocated to contain all column groups output.
      counts - The counts of the individual tuples.
      colIndexes - The columns indexes of the parent column group, this indicate where to put the column sum into the c output.

    • colProduct

      public void colProduct(double[] res, int[] counts, IColIndex colIndexes)
      Description copied from interface: IDictionary
      Calculate the column product of the dictionary weighted by counts.
      Specified by:
      colProduct in interface IDictionary
      Overrides:
      colProduct in class ADictionary
      Parameters:
      res - The result vector to put the result into
      counts - The weighted count of individual tuples
      colIndexes - The column indexes.

    • colProductWithReference

      public void colProductWithReference(double[] res, int[] counts, IColIndex colIndexes, double[] reference)
      Description copied from interface: IDictionary
      Calculate the column product of the dictionary weighted by counts.
      Specified by:
      colProductWithReference in interface IDictionary
      Overrides:
      colProductWithReference in class ADictionary
      Parameters:
      res - The result vector to put the result into
      counts - The weighted count of individual tuples
      colIndexes - The column indexes.
      reference - The reference value.

    • colSumSqWithReference

      public void colSumSqWithReference(double[] c, int[] counts, IColIndex colIndexes, double[] reference)
      Description copied from interface: IDictionary
      Get the column sum of the values contained in the dictionary with an offset reference value added to each cell.
      Specified by:
      colSumSqWithReference in interface IDictionary
      Overrides:
      colSumSqWithReference in class ADictionary
      Parameters:
      c - The output array allocated to contain all column groups output.
      counts - The counts of the individual tuples.
      colIndexes - The columns indexes of the parent column group, this indicate where to put the column sum into the c output.
      reference - The reference values to add to each cell.

    • sum

      public double sum(int[] counts, int ncol)
      Description copied from interface: IDictionary
      Get the sum of the values contained in the dictionary
      Specified by:
      sum in interface IDictionary
      Overrides:
      sum in class ADictionary
      Parameters:
      counts - The counts of the individual tuples
      ncol - The number of columns contained
      Returns:
      The sum scaled by the counts provided.

    • sumSq

      public double sumSq(int[] counts, int ncol)
      Description copied from interface: IDictionary
      Get the square sum of the values contained in the dictionary
      Specified by:
      sumSq in interface IDictionary
      Overrides:
      sumSq in class ADictionary
      Parameters:
      counts - The counts of the individual tuples
      ncol - The number of columns contained
      Returns:
      The square sum scaled by the counts provided.

    • sumSqWithReference

      public double sumSqWithReference(int[] counts, double[] reference)
      Description copied from interface: IDictionary
      Get the square sum of the values contained in the dictionary with a reference offset on each value.
      Specified by:
      sumSqWithReference in interface IDictionary
      Overrides:
      sumSqWithReference in class ADictionary
      Parameters:
      counts - The counts of the individual tuples
      reference - The reference value
      Returns:
      The square sum scaled by the counts and reference.

    • sliceOutColumnRange

      public IDictionary sliceOutColumnRange(int idxStart, int idxEnd, int previousNumberOfColumns)
      Description copied from interface: IDictionary
      Modify the dictionary by removing columns not within the index range.
      Specified by:
      sliceOutColumnRange in interface IDictionary
      Overrides:
      sliceOutColumnRange in class ADictionary
      Parameters:
      idxStart - The column index to start at.
      idxEnd - The column index to end at (not inclusive)
      previousNumberOfColumns - The number of columns contained in the dictionary.
      Returns:
      A dictionary containing the sliced out columns values only.

    • containsValue

      public boolean containsValue(double pattern)
      Description copied from interface: IDictionary
      Detect if the dictionary contains a specific value.
      Specified by:
      containsValue in interface IDictionary
      Overrides:
      containsValue in class ADictionary
      Parameters:
      pattern - The value to search for
      Returns:
      true if the value is contained else false.

    • containsValueWithReference

      public boolean containsValueWithReference(double pattern, double[] reference)
      Description copied from interface: IDictionary
      Detect if the dictionary contains a specific value with reference offset.
      Specified by:
      containsValueWithReference in interface IDictionary
      Overrides:
      containsValueWithReference in class ADictionary
      Parameters:
      pattern - The pattern/ value to search for
      reference - The reference double array.
      Returns:
      true if the value is contained else false.

    • getNumberNonZeros

      public long getNumberNonZeros(int[] counts, int nCol)
      Description copied from interface: IDictionary
      Calculate the number of non zeros in the dictionary. The number of non zeros should be scaled with the counts given. This gives the exact number of non zero values in the parent column group.
      Parameters:
      counts - The counts of each dictionary entry
      nCol - The number of columns in this dictionary
      Returns:
      The nonZero count

    • countNNZZeroColumns

      public int[] countNNZZeroColumns(int[] counts)
      Description copied from interface: IDictionary
      Count the number of non zero values in each column of the dictionary, multiplied with the counts
      Specified by:
      countNNZZeroColumns in interface IDictionary
      Overrides:
      countNNZZeroColumns in class ADictionary
      Parameters:
      counts - The counts to multiply with.
      Returns:
      The nonzero count of each column in the dictionary.

    • getNumberNonZerosWithReference

      public long getNumberNonZerosWithReference(int[] counts, double[] reference, int nRows)
      Description copied from interface: IDictionary
      Calculate the number of non zeros in the dictionary. Each value in the dictionary should be added to the reference value. The number of non zeros should be scaled with the given counts.
      Specified by:
      getNumberNonZerosWithReference in interface IDictionary
      Overrides:
      getNumberNonZerosWithReference in class ADictionary
      Parameters:
      counts - The Counts of each dict entry.
      reference - The reference vector.
      nRows - The number of rows in the input.
      Returns:
      The NonZero Count.

    • addToEntry

      public void addToEntry(double[] v, int fr, int to, int nCol)
      Description copied from interface: IDictionary
      Adds the dictionary entry from this dictionary to the d dictionary
      Specified by:
      addToEntry in interface IDictionary
      Overrides:
      addToEntry in class ADictionary
      Parameters:
      v - The target dictionary (dense double array)
      fr - The from index is the tuple index to copy from.
      to - The to index is the row index to copy into.
      nCol - The number of columns in both cases

    • addToEntry

      public void addToEntry(double[] v, int fr, int to, int nCol, int rep)
      Description copied from interface: IDictionary
      Adds the dictionary entry from this dictionary to the v dictionary rep times.
      Specified by:
      addToEntry in interface IDictionary
      Overrides:
      addToEntry in class ADictionary
      Parameters:
      v - The target dictionary (dense double array)
      fr - The from index is the tuple index to copy from.
      to - The to index is the row index to copy into.
      nCol - The number of columns in both cases
      rep - The number of repetitions to apply (simply multiply do not loop)

    • addToEntryVectorized

      public void addToEntryVectorized(double[] v, int f1, int f2, int f3, int f4, int f5, int f6, int f7, int f8, int t1, int t2, int t3, int t4, int t5, int t6, int t7, int t8, int nCol)
      Description copied from interface: IDictionary
      Vectorized add to entry, this call helps with a bit of locality for the cache.
      Specified by:
      addToEntryVectorized in interface IDictionary
      Overrides:
      addToEntryVectorized in class ADictionary
      Parameters:
      v - The target dictionary (dense double array)
      f1 - From index 1
      f2 - From index 2
      f3 - From index 3
      f4 - From index 4
      f5 - From index 5
      f6 - From index 6
      f7 - From index 7
      f8 - From index 8
      t1 - To index 1
      t2 - To index 2
      t3 - To index 3
      t4 - To index 4
      t5 - To index 5
      t6 - To index 6
      t7 - To index 7
      t8 - To index 8
      nCol - Number of columns in the dictionary

    • subtractTuple

      public IDictionary subtractTuple(double[] tuple)
      Description copied from interface: IDictionary
      Allocate a new dictionary where the tuple given is subtracted from all tuples in the previous dictionary.
      Specified by:
      subtractTuple in interface IDictionary
      Overrides:
      subtractTuple in class ADictionary
      Parameters:
      tuple - a double list representing a tuple, it is given that the tuple with is the same as this dictionaries.
      Returns:
      a new instance of dictionary with the tuple subtracted.

    • getMBDict

      public MatrixBlockDictionary getMBDict(int nCol)
      Description copied from interface: IDictionary
      Get this dictionary as a MatrixBlock dictionary. This allows us to use optimized kernels coded elsewhere in the system, such as matrix multiplication. Return null if the matrix is empty.
      Specified by:
      getMBDict in interface IDictionary
      Overrides:
      getMBDict in class ADictionary
      Parameters:
      nCol - The number of columns contained in this column group.
      Returns:
      A Dictionary containing a MatrixBlock.

    • getString

      public String getString(int colIndexes)
      Description copied from interface: IDictionary
      Get a string representation of the dictionary, that considers the layout of the data.
      Parameters:
      colIndexes - The number of columns in the dictionary.
      Returns:
      A string that is nicer to print.

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • scaleTuples

      public IDictionary scaleTuples(int[] scaling, int nCol)
      Description copied from interface: IDictionary
      Scale all tuples contained in the dictionary by the scaling factor given in the int list.
      Specified by:
      scaleTuples in interface IDictionary
      Overrides:
      scaleTuples in class ADictionary
      Parameters:
      scaling - The amount to multiply the given tuples with
      nCol - The number of columns contained in this column group.
      Returns:
      A New dictionary (since we don't want to modify the underlying dictionary)

    • write

      public void write(DataOutput out) throws IOException
      Description copied from interface: IDictionary
      Write the dictionary to a DataOutput.
      Parameters:
      out - the output sink to write the dictionary to.
      Throws:
      IOException - if the sink fails.

    • read

      public static MatrixBlockDictionary read(DataInput in) throws IOException
      Throws:
      IOException

    • getExactSizeOnDisk

      public long getExactSizeOnDisk()
      Description copied from interface: IDictionary
      Calculate the space consumption if the dictionary is stored on disk.
      Returns:
      the long count of bytes to store the dictionary.

    • preaggValuesFromDense

      public MatrixBlockDictionary preaggValuesFromDense(int numVals, IColIndex colIndexes, IColIndex aggregateColumns, double[] b, int cut)
      Description copied from interface: IDictionary
      Pre Aggregate values for Right Matrix Multiplication.
      Specified by:
      preaggValuesFromDense in interface IDictionary
      Overrides:
      preaggValuesFromDense in class ADictionary
      Parameters:
      numVals - The number of values contained in this dictionary
      colIndexes - The column indexes that is associated with the parent column group
      aggregateColumns - The column to aggregate, this is preprocessed, to find remove consideration for empty columns
      b - The values in the right hand side matrix
      cut - The number of columns in b.
      Returns:
      A new dictionary with the pre aggregated values.

    • preaggValuesFromDenseDictSparse

      public void preaggValuesFromDenseDictSparse(int numVals, IColIndex colIndexes, IColIndex aggregateColumns, double[] b, int cut, double[] ret)

    • preaggValuesFromDenseDictDense

      public void preaggValuesFromDenseDictDense(int numVals, IColIndex colIndexes, IColIndex aggregateColumns, double[] b, int cut, double[] ret)

    • replace

      public IDictionary replace(double pattern, double replace, int nCol)
      Description copied from interface: IDictionary
      Make a copy of the values, and replace all values that match pattern with replacement value. If needed add a new column index.
      Specified by:
      replace in interface IDictionary
      Overrides:
      replace in class ADictionary
      Parameters:
      pattern - The value to look for
      replace - The value to replace the other value with
      nCol - The number of columns contained in the dictionary.
      Returns:
      A new Column Group, reusing the index structure but with new values.

    • replaceWithReference

      public IDictionary replaceWithReference(double pattern, double replace, double[] reference)
      Description copied from interface: IDictionary
      Make a copy of the values, and replace all values that match pattern with replacement value. If needed add a new column index. With reference such that each value in the dict is considered offset by the values contained in the reference.
      Specified by:
      replaceWithReference in interface IDictionary
      Overrides:
      replaceWithReference in class ADictionary
      Parameters:
      pattern - The value to look for
      replace - The value to replace the other value with
      reference - The reference tuple to add to all entries when replacing
      Returns:
      A new Column Group, reusing the index structure but with new values.

    • product

      public void product(double[] ret, int[] counts, int nCol)
      Description copied from interface: IDictionary
      Calculate the product of the dictionary weighted by counts.
      Specified by:
      product in interface IDictionary
      Overrides:
      product in class ADictionary
      Parameters:
      ret - The result dense double array (containing one value)
      counts - The count of individual tuples
      nCol - Number of columns in the dictionary.

    • productWithDefault

      public void productWithDefault(double[] ret, int[] counts, double[] def, int defCount)
      Description copied from interface: IDictionary
      Calculate the product of the dictionary weighted by counts with a default value added .
      Specified by:
      productWithDefault in interface IDictionary
      Overrides:
      productWithDefault in class ADictionary
      Parameters:
      ret - The result dense double array (containing one value)
      counts - The count of individual tuples
      def - The default tuple
      defCount - The count of the default tuple

    • productWithReference

      public void productWithReference(double[] ret, int[] counts, double[] reference, int refCount)
      Description copied from interface: IDictionary
      Calculate the product of the dictionary weighted by counts and offset by reference
      Specified by:
      productWithReference in interface IDictionary
      Overrides:
      productWithReference in class ADictionary
      Parameters:
      ret - The result dense double array (containing one value)
      counts - The counts of each entry in the dictionary
      reference - The reference value.
      refCount - The number of occurrences of the ref value.

    • centralMoment

      public CmCovObject centralMoment(CmCovObject ret, ValueFunction fn, int[] counts, int nRows)
      Description copied from interface: IDictionary
      Central moment function to calculate the central moment of this column group. MUST be on a single column dictionary.
      Specified by:
      centralMoment in interface IDictionary
      Overrides:
      centralMoment in class ADictionary
      Parameters:
      ret - The Central Moment object to be modified and returned
      fn - The value function to apply
      counts - The weight of individual tuples
      nRows - The number of rows in total of the column group
      Returns:
      The central moment Object

    • centralMomentWithDefault

      public CmCovObject centralMomentWithDefault(CmCovObject ret, ValueFunction fn, int[] counts, double def, int nRows)
      Description copied from interface: IDictionary
      Central moment function to calculate the central moment of this column group with a default offset on all missing tuples. MUST be on a single column dictionary.
      Specified by:
      centralMomentWithDefault in interface IDictionary
      Overrides:
      centralMomentWithDefault in class ADictionary
      Parameters:
      ret - The Central Moment object to be modified and returned
      fn - The value function to apply
      counts - The weight of individual tuples
      def - The default values to offset the tuples with
      nRows - The number of rows in total of the column group
      Returns:
      The central moment Object

    • centralMomentWithReference

      public CmCovObject centralMomentWithReference(CmCovObject ret, ValueFunction fn, int[] counts, double reference, int nRows)
      Description copied from interface: IDictionary
      Central moment function to calculate the central moment of this column group with a reference offset on each tuple. MUST be on a single column dictionary.
      Specified by:
      centralMomentWithReference in interface IDictionary
      Overrides:
      centralMomentWithReference in class ADictionary
      Parameters:
      ret - The Central Moment object to be modified and returned
      fn - The value function to apply
      counts - The weight of individual tuples
      reference - The reference values to offset the tuples with
      nRows - The number of rows in total of the column group
      Returns:
      The central moment Object

    • rexpandCols

      public IDictionary rexpandCols(int max, boolean ignore, boolean cast, int nCol)
      Description copied from interface: IDictionary
      Rexpand the dictionary (one hot encode)
      Specified by:
      rexpandCols in interface IDictionary
      Overrides:
      rexpandCols in class ADictionary
      Parameters:
      max - the tuple width of the output
      ignore - If we should ignore zero and negative values
      cast - If we should cast all double values to whole integer values
      nCol - The number of columns in the dictionary already (should be 1)
      Returns:
      A new dictionary

    • rexpandColsWithReference

      public IDictionary rexpandColsWithReference(int max, boolean ignore, boolean cast, int reference)
      Description copied from interface: IDictionary
      Rexpand the dictionary (one hot encode)
      Specified by:
      rexpandColsWithReference in interface IDictionary
      Overrides:
      rexpandColsWithReference in class ADictionary
      Parameters:
      max - the tuple width of the output
      ignore - If we should ignore zero and negative values
      cast - If we should cast all double values to whole integer values
      reference - A reference value to add to all tuples before expanding
      Returns:
      A new dictionary

    • getSparsity

      public double getSparsity()
      Description copied from interface: IDictionary
      Get the sparsity of the dictionary.
      Specified by:
      getSparsity in interface IDictionary
      Overrides:
      getSparsity in class ADictionary
      Returns:
      a sparsity between 0 and 1

    • multiplyScalar

      public void multiplyScalar(double v, double[] ret, int off, int dictIdx, IColIndex cols)
      Description copied from interface: IDictionary
      Multiply the v value with the dictionary entry at dictIdx and add it to the ret matrix at the columns specified in the int array.
      Specified by:
      multiplyScalar in interface IDictionary
      Overrides:
      multiplyScalar in class ADictionary
      Parameters:
      v - Value to multiply
      ret - Output dense double array location
      off - Offset into the ret array that the "row" output starts at
      dictIdx - The dictionary entry to multiply.
      cols - The columns to multiply into of the output.

    • TSMMWithScaling

      public void TSMMWithScaling(int[] counts, IColIndex rows, IColIndex cols, MatrixBlock ret)
      Description copied from interface: IDictionary
      Transpose self matrix multiplication with a scaling factor on each pair of values.
      Specified by:
      TSMMWithScaling in interface IDictionary
      Overrides:
      TSMMWithScaling in class ADictionary
      Parameters:
      counts - The scaling factor
      rows - The row indexes
      cols - The col indexes
      ret - The output matrix block

    • MMDict

      public void MMDict(IDictionary right, IColIndex rowsLeft, IColIndex colsRight, MatrixBlock result)
      Description copied from interface: IDictionary
      Matrix multiplication of dictionaries note the left is this, and it is supposed to be treated as if it is transposed while it is not physically transposed.
      Specified by:
      MMDict in interface IDictionary
      Overrides:
      MMDict in class ADictionary
      Parameters:
      right - Right hand side of multiplication
      rowsLeft - Offset rows on the left
      colsRight - Offset cols on the right
      result - The output matrix block

    • MMDictScaling

      public void MMDictScaling(IDictionary right, IColIndex rowsLeft, IColIndex colsRight, MatrixBlock result, int[] scaling)
      Description copied from interface: IDictionary
      Matrix multiplication of dictionaries, note the left is this, and it is supposed to be treated as if it is transposed while it is not physically transposed.
      Specified by:
      MMDictScaling in interface IDictionary
      Overrides:
      MMDictScaling in class ADictionary
      Parameters:
      right - Right hand side of multiplication (not transposed)
      rowsLeft - Offset rows on the left
      colsRight - Offset cols on the right
      result - The output matrix block
      scaling - The scaling

    • MMDictDense

      public void MMDictDense(double[] left, IColIndex rowsLeft, IColIndex colsRight, MatrixBlock result)
      Description copied from interface: IDictionary
      Matrix multiplication of dictionaries left side dense and transposed. The right side is this.
      Specified by:
      MMDictDense in interface IDictionary
      Overrides:
      MMDictDense in class ADictionary
      Parameters:
      left - Dense left side (treat as if it is transposed but it is physically not)
      rowsLeft - Offset rows on the left
      colsRight - Offset cols on the right
      result - The output matrix block

    • MMDictScalingDense

      public void MMDictScalingDense(double[] left, IColIndex rowsLeft, IColIndex colsRight, MatrixBlock result, int[] scaling)
      Description copied from interface: IDictionary
      Matrix multiplication of dictionaries left side dense and transposed. The Right side is this, the scaling factor is used to multiply each element with.
      Specified by:
      MMDictScalingDense in interface IDictionary
      Overrides:
      MMDictScalingDense in class ADictionary
      Parameters:
      left - Dense left side (Dense dictionary)
      rowsLeft - Offset rows on the left (That dictionaries column indexes)
      colsRight - Offset cols on the right (This dictionaries column indexes)
      result - The output matrix block, guaranteed to be allocated as dense.
      scaling - The scaling factor to multiply each entry with.

    • MMDictSparse

      public void MMDictSparse(SparseBlock left, IColIndex rowsLeft, IColIndex colsRight, MatrixBlock result)
      Description copied from interface: IDictionary
      Matrix multiplication of dictionaries left side sparse and transposed right side is this.
      Specified by:
      MMDictSparse in interface IDictionary
      Overrides:
      MMDictSparse in class ADictionary
      Parameters:
      left - Sparse left side
      rowsLeft - Offset rows on the left
      colsRight - Offset cols on the right
      result - The output matrix block

    • MMDictScalingSparse

      public void MMDictScalingSparse(SparseBlock left, IColIndex rowsLeft, IColIndex colsRight, MatrixBlock result, int[] scaling)
      Description copied from interface: IDictionary
      Matrix multiplication of dictionaries left side sparse and transposed right side is this.
      Specified by:
      MMDictScalingSparse in interface IDictionary
      Overrides:
      MMDictScalingSparse in class ADictionary
      Parameters:
      left - Sparse left side
      rowsLeft - Offset rows on the left
      colsRight - Offset cols on the right
      result - The output matrix block
      scaling - The scaling

    • TSMMToUpperTriangle

      public void TSMMToUpperTriangle(IDictionary right, IColIndex rowsLeft, IColIndex colsRight, MatrixBlock result)
      Description copied from interface: IDictionary
      Matrix multiplication but allocate output in upper triangle and twice if on diagonal, note this is left
      Specified by:
      TSMMToUpperTriangle in interface IDictionary
      Overrides:
      TSMMToUpperTriangle in class ADictionary
      Parameters:
      right - Right side
      rowsLeft - Offset rows on the left
      colsRight - Offset cols on the right
      result - The output matrix block

    • TSMMToUpperTriangleDense

      public void TSMMToUpperTriangleDense(double[] left, IColIndex rowsLeft, IColIndex colsRight, MatrixBlock result)
      Description copied from interface: IDictionary
      Matrix multiplication but allocate output in upper triangle and twice if on diagonal, note this is right
      Specified by:
      TSMMToUpperTriangleDense in interface IDictionary
      Overrides:
      TSMMToUpperTriangleDense in class ADictionary
      Parameters:
      left - Dense left side
      rowsLeft - Offset rows on the left
      colsRight - Offset cols on the right
      result - The output matrix block

    • TSMMToUpperTriangleSparse

      public void TSMMToUpperTriangleSparse(SparseBlock left, IColIndex rowsLeft, IColIndex colsRight, MatrixBlock result)
      Description copied from interface: IDictionary
      Matrix multiplication but allocate output in upper triangle and twice if on diagonal, note this is right
      Specified by:
      TSMMToUpperTriangleSparse in interface IDictionary
      Overrides:
      TSMMToUpperTriangleSparse in class ADictionary
      Parameters:
      left - Sparse left side
      rowsLeft - Offset rows on the left
      colsRight - Offset cols on the right
      result - The output matrix block

    • TSMMToUpperTriangleScaling

      public void TSMMToUpperTriangleScaling(IDictionary right, IColIndex rowsLeft, IColIndex colsRight, int[] scale, MatrixBlock result)
      Description copied from interface: IDictionary
      Matrix multiplication but allocate output in upper triangle and twice if on diagonal, note this is left
      Specified by:
      TSMMToUpperTriangleScaling in interface IDictionary
      Overrides:
      TSMMToUpperTriangleScaling in class ADictionary
      Parameters:
      right - Right side
      rowsLeft - Offset rows on the left
      colsRight - Offset cols on the right
      scale - Scale factor
      result - The output matrix block

    • TSMMToUpperTriangleDenseScaling

      public void TSMMToUpperTriangleDenseScaling(double[] left, IColIndex rowsLeft, IColIndex colsRight, int[] scale, MatrixBlock result)
      Description copied from interface: IDictionary
      Matrix multiplication but allocate output in upper triangle and twice if on diagonal, note this is right
      Specified by:
      TSMMToUpperTriangleDenseScaling in interface IDictionary
      Overrides:
      TSMMToUpperTriangleDenseScaling in class ADictionary
      Parameters:
      left - Dense left side
      rowsLeft - Offset rows on the left
      colsRight - Offset cols on the right
      scale - Scale factor
      result - The output matrix block

    • TSMMToUpperTriangleSparseScaling

      public void TSMMToUpperTriangleSparseScaling(SparseBlock left, IColIndex rowsLeft, IColIndex colsRight, int[] scale, MatrixBlock result)
      Description copied from interface: IDictionary
      Matrix multiplication but allocate output in upper triangle and twice if on diagonal, note this is right
      Specified by:
      TSMMToUpperTriangleSparseScaling in interface IDictionary
      Overrides:
      TSMMToUpperTriangleSparseScaling in class ADictionary
      Parameters:
      left - Sparse left side
      rowsLeft - Offset rows on the left
      colsRight - Offset cols on the right
      scale - Scale factor
      result - The output matrix block

    • equals

      public boolean equals(IDictionary o)
      Description copied from interface: IDictionary
      Indicate if the other dictionary is equal to this.
      Parameters:
      o - The other object
      Returns:
      If it is equal

    • cbind

      public IDictionary cbind(IDictionary that, int nCol)
      Description copied from interface: IDictionary
      Cbind this dictionary with that dictionary
      Specified by:
      cbind in interface IDictionary
      Overrides:
      cbind in class ADictionary
      Parameters:
      that - the right hand side dictionary to cbind
      nCol - the right hand side number of columns
      Returns:
      The combined dictionary

    • reorder

      public IDictionary reorder(int[] reorder)
      Description copied from interface: IDictionary
      Reorder the elements in the dictionary based on the reorder specification given.
      Specified by:
      reorder in interface IDictionary
      Overrides:
      reorder in class ADictionary
      Parameters:
      reorder - The order to move to.
      Returns:
      A new Dictionary that is reordered.s

    • append

      public IDictionary append(double[] row)
      Description copied from interface: IDictionary
      Return a new dictionary with the given row appended. If possible reuse as much of the old dictionary as possible.
      Specified by:
      append in interface IDictionary
      Overrides:
      append in class ADictionary
      Parameters:
      row - The new row to append.
      Returns:
      A new dictionary with the appended row

    • sliceColumns

      public IDictionary sliceColumns(IntArrayList selectedColumns, int nCol)
      Description copied from interface: IDictionary
      Slice out the selected columns given of this encoded group.
      Parameters:
      selectedColumns - The columns to slice out and return as a new matrix.
      nCol - The number of columns in this dictionary.
      Returns:
      The returned matrix

    • sliceColumns

      public static double[] sliceColumns(MatrixBlock mb, IntArrayList selectedColumns)

    • sort

      public int[] sort()
      Description copied from interface: IDictionary
      Sort the values of this dictionary via an index of how the values mapped previously. In practice this design means we can reuse the previous dictionary for the resulting column group
      Returns:
      The sorted index.