public class FacetTokenizer
extends TokenStream
Modifier and Type | Field and Description |
---|---|
(package private) Token |
nextToken |
(package private) int |
pos |
(package private) String |
str |
Constructor and Description |
---|
FacetTokenizer(String str)
Construct a token stream to remove accents from the input tokens.
|
Modifier and Type | Method and Description |
---|---|
Token |
next()
Retrieve the next token in the stream.
|