转换存储为字符串hashIDs的列表，以独特的价值观

Question

我有一个数据帧，其中在一个专栏中，我有存储字符串一样的散列值列表的列：

'[d85235f50b3c019ad7c6291e3ca58093,03e0fb034f2cb3264234b9eae09b4287]' just to be clear. 

数据框看起来像

1 
0 [8a88e629c368001c18619c7cd66d3e96, 4b0709dd990a0904bbe6afec636c4213, c00a98ceb6fc7006d572486787e551cc, 0e72ae6851c40799ec14a41496d64406, 76475992f4207ee2b209a4867b42c372] 
1 [3277ded8d1f105c84ad5e093f6e7795d] 
2 [d85235f50b3c019ad7c6291e3ca58093, 03e0fb034f2cb3264234b9eae09b4287] 

我想创建一个唯一的哈希ID列表存在于此列中。

什么是有效的方法？ 谢谢

Answer 1

选项1
低于最快的选项 见时机

您可以嵌入解析和扁平化的一个理解

[y for x in df['1'].values.tolist() for y in x.strip('[]').split(', ')] 

['8a88e629c368001c18619c7cd66d3e96', 
'4b0709dd990a0904bbe6afec636c4213', 
'c00a98ceb6fc7006d572486787e551cc', 
'0e72ae6851c40799ec14a41496d64406', 
'76475992f4207ee2b209a4867b42c372', 
'3277ded8d1f105c84ad5e093f6e7795d', 
'd85235f50b3c019ad7c6291e3ca58093', 
'03e0fb034f2cb3264234b9eae09b4287'] 

从那里，你可以使用list(set())，pd.unique，或np.unique

pd.unique([y for x in df['1'].values.tolist() for y in x.strip('[]').split(', ')]) 

array(['8a88e629c368001c18619c7cd66d3e96', 
     '4b0709dd990a0904bbe6afec636c4213', 
     'c00a98ceb6fc7006d572486787e551cc', 
     '0e72ae6851c40799ec14a41496d64406', 
     '76475992f4207ee2b209a4867b42c372', 
     '3277ded8d1f105c84ad5e093f6e7795d', 
     'd85235f50b3c019ad7c6291e3ca58093', 
     '03e0fb034f2cb3264234b9eae09b4287'], dtype=object) 

---

选项2
为了简洁，使用pd.Series.extractall

list(set(df['1'].str.extractall('(\w+)')[0])) 

['8a88e629c368001c18619c7cd66d3e96', 
'4b0709dd990a0904bbe6afec636c4213', 
'c00a98ceb6fc7006d572486787e551cc', 
'0e72ae6851c40799ec14a41496d64406', 
'76475992f4207ee2b209a4867b42c372', 
'3277ded8d1f105c84ad5e093f6e7795d', 
'd85235f50b3c019ad7c6291e3ca58093', 
'03e0fb034f2cb3264234b9eae09b4287'] 

---

@ jezrael的list(set())我的理解是最快

解析定时
我保持相同list(set())用于比较分析和扁平化的目的。

%timeit list(set(np.concatenate(df['1'].apply(yaml.load).values).tolist())) 
%timeit list(set([y for x in df['1'].values.tolist() for y in x.strip('[]').split(', ')])) 
%timeit list(set(chain.from_iterable(df['1'].str.strip('[]').str.split(', ')))) 
%timeit list(set(df['1'].str.extractall('(\w+)')[0])) 

1.01 ms ± 45 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) 
6.42 µs ± 219 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) 
279 µs ± 8.87 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) 
941 µs ± 10.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) 

这需要我的理解和使用各种办法，使独特的比较那些速度

%timeit pd.unique([y for x in df['1'].values.tolist() for y in x.strip('[]').split(', ')]) 
%timeit np.unique([y for x in df['1'].values.tolist() for y in x.strip('[]').split(', ')]) 
%timeit list(set([y for x in df['1'].values.tolist() for y in x.strip('[]').split(', ')])) 

57.8 µs ± 3.66 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each) 
17.5 µs ± 552 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) 
6.18 µs ± 184 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) 

Answer 2

IIUC，你想扁平你的数据。使用yaml.load将其转换为一列列表。

import yaml 

df = df.applymap(yaml.load) 
print(df) 
              1 
0 [8a88e629c368001c18619c7cd66d3e96, 4b0709dd990... 
1     [3277ded8d1f105c84ad5e093f6e7795d] 
2 [d85235f50b3c019ad7c6291e3ca58093, 03e0fb034f2... 

最简单的方法是构造一个新的数据帧形式的旧值。

out = pd.DataFrame(np.concatenate(df.iloc[:, 0].values.tolist())) 
print(out) 

            0 
0 8a88e629c368001c18619c7cd66d3e96 
1 4b0709dd990a0904bbe6afec636c4213 
2 c00a98ceb6fc7006d572486787e551cc 
3 0e72ae6851c40799ec14a41496d64406 
4 76475992f4207ee2b209a4867b42c372 
5 3277ded8d1f105c84ad5e093f6e7795d 
6 d85235f50b3c019ad7c6291e3ca58093 
7 03e0fb034f2cb3264234b9eae09b4287 

Answer 3

您需要strip与split第一和flatenning chain：

print (df.columns.tolist()) 
['col'] 

#convert strings to lists per rows 
#change by your column name if necessary  
s = df['col'].str.strip('[]').str.split(', ') 
print (s) 
0 [8a88e629c368001c18619c7cd66d3e96, 4b0709dd990... 
1     [3277ded8d1f105c84ad5e093f6e7795d] 
2 [d85235f50b3c019ad7c6291e3ca58093, 03e0fb034f2... 
Name: col, dtype: object 

#check first value 
print (type(s.iat[0])) 
<class 'list'> 

#get unique values - for unique values use set 
from itertools import chain 
L = list(set(chain.from_iterable(s))) 

['76475992f4207ee2b209a4867b42c372', '3277ded8d1f105c84ad5e093f6e7795d', 
'd85235f50b3c019ad7c6291e3ca58093', '4b0709dd990a0904bbe6afec636c4213', 
'c00a98ceb6fc7006d572486787e551cc', '03e0fb034f2cb3264234b9eae09b4287', 
'8a88e629c368001c18619c7cd66d3e96', '0e72ae6851c40799ec14a41496d64406'] 

---

from itertools import chain 
s = [x.strip('[]').split(', ') for x in df['col'].values.tolist()] 
L = list(set(chain.from_iterable(s))) 
print (L) 

['76475992f4207ee2b209a4867b42c372', '3277ded8d1f105c84ad5e093f6e7795d', 
'd85235f50b3c019ad7c6291e3ca58093', '4b0709dd990a0904bbe6afec636c4213', 
'c00a98ceb6fc7006d572486787e551cc', '03e0fb034f2cb3264234b9eae09b4287', 
'8a88e629c368001c18619c7cd66d3e96', '0e72ae6851c40799ec14a41496d64406']