4
我正在尝试使用scikit使用余弦相似性来查找类似的问题。我正在试图在互联网上提供这个示例代码。 Link1和Link2使用scikit-learn时出现属性错误
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from nltk.corpus import stopwords
import numpy as np
import numpy.linalg as LA
train_set = ["The sky is blue.", "The sun is bright."]
test_set = ["The sun in the sky is bright."]
stopWords = stopwords.words('english')
vectorizer = CountVectorizer(stop_words = stopWords)
transformer = TfidfTransformer()
trainVectorizerArray = vectorizer.fit_transform(train_set).toarray()
trainVectorizerArray = vectorizer.
testVectorizerArray = vectorizer.transform(test_set).toarray()
print 'Fit Vectorizer to train set', trainVectorizerArray
print 'Transform Vectorizer to test set', testVectorizerArray
cx = lambda a, b : round(np.inner(a, b)/(LA.norm(a)*LA.norm(b)), 3)
for vector in trainVectorizerArray:
print vector
for testV in testVectorizerArray:
print testV
cosine = cx(vector, testV)
print cosine
transformer.fit(trainVectorizerArray)
print transformer.transform(trainVectorizerArray).toarray()
transformer.fit(testVectorizerArray)
tfidf = transformer.transform(testVectorizerArray)
print tfidf.todense()
我总是得到这个错误
Traceback (most recent call last):
File "C:\Users\Animesh\Desktop\NLP\ngrams2.py", line 14, in <module>
trainVectorizerArray = vectorizer.fit_transform(train_set).toarray()
File "C:\Python27\lib\site-packages\scikit_learn-0.13.1-py2.7-win32.egg\sklearn \feature_extraction\text.py", line 740, in fit_transform
raise ValueError("empty vocabulary; training set may have"
ValueError: empty vocabulary; training set may have contained only stop words or min_df (resp. max_df) may be too high (resp. too low).
我甚至可以检查代码上this link。我有错误AttributeError: 'CountVectorizer' object has no attribute 'vocabulary'
。
如何解决这个问题?
我在Windows 7 32位和scikit_learn 0.13.1上使用Python 2.7.3。
哦!这解决了这个问题..但是,你能告诉我什么是词汇功能...当我尝试使用这个功能时,它为什么会给出属性错误 – 2013-03-05 10:33:08
@AnimeshPandey:错误消息中正确的是:“空的词汇;培训集合可能只包含停用词或min_df(resp.max_df)可能太高(或太低)。“正如我所解释的,默认设置“min_df = 2”太低,因为您只有两个文档。 (请注意,tf-idf在这么少的文档中工作不正常。) – 2013-03-05 10:33:45
在调用fit方法时(除非用户提供了构造函数参数),将提取带有尾部'_'的'vocabulary_'。请参阅[文档](http://scikit-learn.org/stable/modules/feature_extraction.html#text-feature-extraction)。 – ogrisel 2013-03-05 10:34:00