public class TokenizerFunction extends BaseTokenizerFunction implements org.apache.spark.api.java.function.Function<String,Sequence<VocabWord>>
configurationBroadcast, tokenizerFactory, tokenPreprocessor
Constructor and Description |
---|
TokenizerFunction(@NonNull org.apache.spark.broadcast.Broadcast<VectorsConfiguration> configurationBroadcast) |
Modifier and Type | Method and Description |
---|---|
Sequence<VocabWord> |
call(String s) |
instantiateTokenizerFactory
public TokenizerFunction(@NonNull @NonNull org.apache.spark.broadcast.Broadcast<VectorsConfiguration> configurationBroadcast)
Copyright © 2020. All rights reserved.