使用apache lucene取消停用词时的异常

Question

我使用以下代码从输入文本中删除停用词。当tokenStream.incrementToken()运行时，我得到异常。

java.lang.IllegalStateException: TokenStream contract violation: reset()/close() call missing, reset() called multiple times, or subclass does not call super.reset(). Please see Javadocs of TokenStream class for more information about the correct consuming workflow. 

代码：

public static String removeStopWords(String textFile) throws Exception { 
     CharArraySet stopWords = EnglishAnalyzer.getDefaultStopSet(); 
     TokenStream tokenStream = new StandardTokenizer(); 
     tokenStream = new StopFilter(tokenStream, stopWords); 
     StringBuilder sb = new StringBuilder(); 
     CharTermAttribute charTermAttribute = tokenStream.addAttribute(CharTermAttribute.class); 
     tokenStream.reset(); 
     while (tokenStream.incrementToken()) { 
      String term = charTermAttribute.toString(); 
      sb.append(term + " "); 
     } 
     return sb.toString(); 
    } 

Answer 1

实例化的TokenStream如下 -

TokenStream tokenStream = new StandardAnalyzer().tokenStream("field",new StringReader(textFile));