Package org.apache.sysds.runtime.io

Class WriterBinaryBlockParallel

java.lang.Object

org.apache.sysds.runtime.io.MatrixWriter

org.apache.sysds.runtime.io.WriterBinaryBlock

org.apache.sysds.runtime.io.WriterBinaryBlockParallel

public class WriterBinaryBlockParallel
extends WriterBinaryBlock

Constructor Summary

Constructors

Constructor	Description
WriterBinaryBlockParallel(int replication)

Method Summary

Modifier and Type	Method	Description
static int	numPartsFiles(org.apache.hadoop.fs.FileSystem fs, long rlen, long clen, long blen, long nZeros)
long	writeMatrixFromStream(String fname, OOCStream<IndexedMatrixValue> stream, long rlen, long clen, int blen)	Consumes an out-of-core stream of matrix blocks and writes them to the target output path.

Methods inherited from class org.apache.sysds.runtime.io.WriterBinaryBlock

writeEmptyMatrixToHDFS, writeMatrixToHDFS

Methods inherited from class org.apache.sysds.runtime.io.MatrixWriter

createMatrixBlocksForReuse, getMatrixBlockForReuse, setForcedParallel, writeMatrixToHDFS

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

WriterBinaryBlockParallel

public WriterBinaryBlockParallel(int replication)

Method Details

writeMatrixFromStream

public long writeMatrixFromStream(String fname,
 OOCStream<IndexedMatrixValue> stream,
 long rlen,
 long clen,
 int blen)
                           throws IOException

Description copied from class: MatrixWriter

Consumes an out-of-core stream of matrix blocks and writes them to the target output path. Implementations may choose single-file or multipart output depending on format and parallelism.

Overrides:

writeMatrixFromStream in class WriterBinaryBlock

Parameters:

fname - The target output filename

stream - The OOC stream of matrix blocks to consume

rlen - The total number of rows in the matrix

clen - The total number of columns in the matrix

blen - The block size

Throws:

IOException - if an I/O error occurs

numPartsFiles

public static int numPartsFiles(org.apache.hadoop.fs.FileSystem fs,
 long rlen,
 long clen,
 long blen,
 long nZeros)