blob: 0cdbbc387a4ab95b3e0484aefa0657f3d641575a (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
[[analysis-lowercase-tokenizer]]
=== Lowercase Tokenizer
A tokenizer of type `lowercase` that performs the function of
<<analysis-letter-tokenizer,Letter
Tokenizer>> and
<<analysis-lowercase-tokenfilter,Lower
Case Token Filter>> together. It divides text at non-letters and converts
them to lower case. While it is functionally equivalent to the
combination of
<<analysis-letter-tokenizer,Letter
Tokenizer>> and
<<analysis-lowercase-tokenfilter,Lower
Case Token Filter>>, there is a performance advantage to doing the two
tasks at once, hence this (redundant) implementation.
|
