org.datavec.nlp.metadata

Interface VocabCache

All Known Implementing Classes:

DefaultVocabCache

public interface VocabCache

Track metadata about vocabs

Author:

Adam Gibson

Method Summary

Modifier and Type	Method and Description
double	idf(String word) Number of documents word has occurred in
void	incrementCount(String word) Increment a word count by 1
void	incrementCount(String word, double by) Increment count for a word
void	incrementDocCount(String word) Increment the doc count for a word by 1
void	incrementDocCount(String word, double by) Increment the document count for a particular word
void	incrementNumDocs(double by) Increment the number of documents
void	initialize(Configuration conf) Configuration for initializing
int	minWordFrequency() The min word frequency needed to be included in the vocab (default 5)
double	numDocs() Number of documents
double	tfidf(String word, double frequency, boolean smoothIdf) Calculate the tfidf of the word given the document frequency
Index	vocabWords() All of the vocab words (ordered) note that these are not all the possible tokens
String	wordAt(int i) Returns a word in the vocab at a particular index
double	wordFrequency(String word) Get the word frequency for a word
int	wordIndex(String word)

Method Detail

incrementNumDocs

void incrementNumDocs(double by)

Increment the number of documents

Parameters:

by -

numDocs

double numDocs()

Number of documents

Returns:

the number of documents

wordAt

String wordAt(int i)

Returns a word in the vocab at a particular index

Parameters:

i - the index to get

Returns:

the word at that index in the vocab

wordIndex

int wordIndex(String word)

initialize

void initialize(Configuration conf)

Configuration for initializing

Parameters:

conf - the configuration to initialize with

wordFrequency

double wordFrequency(String word)

Get the word frequency for a word

Parameters:

word - the word to get frequency for

Returns:

the frequency for a given word

minWordFrequency

int minWordFrequency()

The min word frequency needed to be included in the vocab (default 5)

Returns:

the min word frequency to be included in the vocab

vocabWords

Index vocabWords()

All of the vocab words (ordered) note that these are not all the possible tokens

Returns:

the list of vocab words

incrementDocCount

void incrementDocCount(String word)

Increment the doc count for a word by 1

Parameters:

word - the word to increment the count for

incrementDocCount

void incrementDocCount(String word,
                       double by)

Increment the document count for a particular word

Parameters:

word - the word to increment the count for

by - the amount to increment by

incrementCount

void incrementCount(String word)

Increment a word count by 1

Parameters:

word - the word to increment the count for

incrementCount

void incrementCount(String word,
                    double by)

Increment count for a word

Parameters:

word - the word to increment the count for

by - the amount to increment by

idf

double idf(String word)

Number of documents word has occurred in

Parameters:

word - the word to get the idf for

tfidf

double tfidf(String word,
             double frequency,
             boolean smoothIdf)

Calculate the tfidf of the word given the document frequency

Parameters:

word - the word to get frequency for

frequency - the frequency

Returns:

the tfidf for a word