从数据框中删除停用词

Question

dataframe['Text'] = dataframe['Text'].apply(lambda x : ' '.join([item for item in string.split(x.lower()) if item not in stopwords])) 

我正在从数据框中删除停用词。逻辑工作正常，但是当有一些空行时，它会给出错误。

我已经使用了dropna（），但它会删除整行，而不是其他列中有数据。

如何在上述逻辑列的文字要你的逻辑之前不为空

Answer 1

使用，

dataframe.dropna(subset=['Text'], how='all') 

Answer 2

可以更换NaN空list什么是不容易的添加条件 - 通过使用mask或combine_firstSeries由空创建lists：

pos_tweets = [('I love this car', 'positive'), 
('This view is amazing', 'positive'), 
('I feel great this morning', 'positive'), 
('I am so excited about the concert', 'positive'), 
(None, 'positive')] 

df = pd.DataFrame(pos_tweets, columns= ["Text","col2"]) 
print (df) 
           Text  col2 
0     I love this car positive 
1    This view is amazing positive 
2   I feel great this morning positive 
3 I am so excited about the concert positive 
4        None positive 

stopwords = ['love','car','amazing'] 
s = pd.Series([[]], index=df.index) 
df["Text"] = df["Text"].str.lower().str.split().mask(df["Text"].isnull(), s) 
print (df) 
             Text  col2 
0      [i, love, this, car] positive 
1     [this, view, is, amazing] positive 
2   [i, feel, great, this, morning] positive 
3 [i, am, so, excited, about, the, concert] positive 
4           [] positive 

df['Text']=df['Text'].apply(lambda x:' '.join([item for item in x if item not in stopwords])) 
print (df) 
           Text  col2 
0        i this positive 
1      this view is positive 
2   i feel great this morning positive 
3 i am so excited about the concert positive 
4          positive 

另一种解决方案：

stopwords = ['love','car','amazing'] 
df["Text"]=df["Text"].str.lower().str.split().combine_first(pd.Series([[]], index=df.index)) 
print (df) 
             Text  col2 
0      [i, love, this, car] positive 
1     [this, view, is, amazing] positive 
2   [i, feel, great, this, morning] positive 
3 [i, am, so, excited, about, the, concert] positive 
4           [] positive