chi squared selectKbest不良输入形状错误

Question

我对scikit和ML有点新。我试图训练一个Adaboost分类器，用于一个vs Rest分类。我使用以下代码

# To Read Training data set 
test = pd.read_csv("train.csv", header=0, delimiter=",", \ 
        quoting=1, error_bad_lines=False) 
num_reviews = len(test["text"]) 
clean_train_reviews = [] 
catlist=[] 
for i in xrange(0,num_reviews): 
    data=processText(test["text"][i]) 
    data1=test["category"][i] 
    clean_train_reviews.append(data) 
    catlist.append(data1.split('.')) 

# To read test dataset 
test = pd.read_csv("test.csv", header=0, delimiter=",", \ 
        quoting=1, error_bad_lines=False) 
num_reviews = len(test["text"]) 
clean_test_reviews = [] 
for i in xrange(0,num_reviews): 
    data=processText(test["text"][i]) 
    clean_test_reviews.append(data) 
X_test=np.array(clean_test_reviews) 


lb = preprocessing.MultiLabelBinarizer() 
Y = lb.fit_transform(catlist) 

classifier = Pipeline([ 
    ('vectorizer', CountVectorizer(ngram_range=(1,2), max_features=1500,min_df=4)), 
    ('tfidf', TfidfTransformer()), 
    ('chi2', SelectKBest(chi2, k=200)), 
    ('clf', OneVsRestClassifier(AdaBoostClassifier()))]) 
classifier.fit(clean_train_reviews, Y) 
predicted = classifier.predict(X_test) 

我使用管道，其中文本插入作为clean_train_reviews和Y是类（多标签，N = 10）。使用TfidfVectorizer()在流水线中提取文本特征，并使用卡方平方特征选择方法进行选择。 Adaboost的分类项：ValueError异常：坏输入形状（1000，10）

File "<ipython-input-10-9dbc8b18e6b8>", line 1, in <module> 
    runfile('C:/Users/Administrator/Desktop/nincymiss/adaboost.py', wdir='C:/Users/Administrator/Desktop/nincymiss') 

    File "C:\Python27\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 601, in runfile 
    execfile(filename, namespace) 

    File "C:\Python27\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 66, in execfile 
    exec(compile(scripttext, filename, 'exec'), glob, loc) 

    File "C:/Users/Administrator/Desktop/nincymiss/adaboost.py", line 179, in <module> 
    classifier.fit(clean_train_reviews, Y) 

    File "C:\Python27\lib\site-packages\sklearn\pipeline.py", line 164, in fit 
    Xt, fit_params = self._pre_transform(X, y, **fit_params) 

    File "C:\Python27\lib\site-packages\sklearn\pipeline.py", line 145, in _pre_transform 
    Xt = transform.fit_transform(Xt, y, **fit_params_steps[name]) 

    File "C:\Python27\lib\site-packages\sklearn\base.py", line 458, in fit_transform 
    return self.fit(X, y, **fit_params).transform(X) 

    File "C:\Python27\lib\site-packages\sklearn\feature_selection\univariate_selection.py", line 322, in fit 
    X, y = check_X_y(X, y, ['csr', 'csc']) 

    File "C:\Python27\lib\site-packages\sklearn\utils\validation.py", line 515, in check_X_y 
    y = column_or_1d(y, warn=True) 

    File "C:\Python27\lib\site-packages\sklearn\utils\validation.py", line 551, in column_or_1d 
    raise ValueError("bad input shape {0}".format(shape)) 

ValueError: bad input shape (1000, 10) 

Answer 1

这是因为，如你所期望的多标签问题的特征选择不起作用。您可以尝试以下选项，分别为每个标签选择“最佳”功能。

classifier = Pipeline([ 
    ('vectorizer', CountVectorizer(ngram_range=(1,2), max_features=1500, min_df=4)), 
    ('tfidf', TfidfTransformer()), 
    ('chi2', SelectKBest(chi2, k=200)), 
    ('clf', AdaBoostClassifier())]) 

clf = OneVsRestClassifier(classifier)