Prior to the comparison we use standard stemming and stopword removal on both sentences to increase the morphological uniformity. |
Since the dictionary are aggressively cleaned from stopword, they are less likely stopword. |
There are also problems with extraction and normalization algorithms for search keys with stopword handling and with a host of other messy implementation details. |
However, the use of a single fixed stopword list across different document collections could be detrimental to the retrieval effectiveness. |
To deal with this dimensionality problem, feature selection and dimension reduction methods are applied, such as, stopword removal, stemming and removing less frequent words. |
After perplexed librarians could not retrieve some titles, the initial stopword list was pruned to match the minimal list used for LC's main catalog. |