Package org.apache.sysds.runtime.transform.tokenize.builder

Class TokenizerBuilderWhitespaceSplit

java.lang.Object

org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilder

org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilderWhitespaceSplit

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

TokenizerBuilderNgram

public class TokenizerBuilderWhitespaceSplit
extends TokenizerBuilder

See Also:

• Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
String	regex

Constructor Summary

Constructors

Constructor	Description
TokenizerBuilderWhitespaceSplit(int[] idCols, int tokenizeCol, org.apache.wink.json4j.JSONObject params)

Method Summary

Modifier and Type	Method	Description
void	createInternalRepresentation(FrameBlock in, DocumentRepresentation[] internalRepresentation, int rowStart, int blk)
List<Token>	splitToTokens(String text)

Methods inherited from class org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilder

createInternalRepresentation, getTasks

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

regex

public String regex

Constructor Details

TokenizerBuilderWhitespaceSplit

public TokenizerBuilderWhitespaceSplit(int[] idCols,
 int tokenizeCol,
 org.apache.wink.json4j.JSONObject params)
                                throws org.apache.wink.json4j.JSONException

Throws:

org.apache.wink.json4j.JSONException

Method Details

splitToTokens

public List<Token> splitToTokens(String text)

createInternalRepresentation

public void createInternalRepresentation(FrameBlock in,
 DocumentRepresentation[] internalRepresentation,
 int rowStart,
 int blk)

Specified by:

createInternalRepresentation in class TokenizerBuilder