R - 根据订购因子排序缓慢工作

Question

根据More efficient means of creating a corpus and DTM这个问题，我已经准备好了自己的方法，用于从大型语料库构建术语文档矩阵（我希望）不需要术语x文档内存。

sparseTDM <- function(vc){ 
    id = unlist(lapply(vc, function(x){x$meta$id})) 
    content = unlist(lapply(vc, function(x){x$content})) 
    out = strsplit(content, "\\s", perl = T) 
    names(out) = id 
    lev.terms = sort(unique(unlist(out))) 
    lev.docs = id 

    v1 = lapply(
    out, 
    function(x, lev) { 
     sort(as.integer(factor(x, levels = lev, ordered = TRUE))) 
    }, 
    lev = lev.terms 
) 

    v2 = lapply(
    seq_along(v1), 
    function(i, x, n){ 
     rep(i,length(x[[i]])) 
    }, 
    x = v1, 
    n = names(v1) 
) 

    stm = data.frame(i = unlist(v1), j = unlist(v2)) %>% 
    group_by(i, j) %>% 
    tally() %>% 
    ungroup() 

    tmp = simple_triplet_matrix(
    i = stm$i, 
    j = stm$j, 
    v = stm$n, 
    nrow = length(lev.terms), 
    ncol = length(lev.docs), 
    dimnames = list(Terms = lev.terms, Docs = lev.docs) 
) 

    as.TermDocumentMatrix(tmp, weighting = weightTf) 
} 

它在计算v1时变慢。它运行了30分钟，我停下了。

我已经准备了一个小例子：

b = paste0("string", 1:200000) 
a = sample(b,80) 
microbenchmark(
    lapply(
    list(a=a), 
    function(x, lev) { 
     sort(as.integer(factor(x, levels = lev, ordered = TRUE))) 
    }, 
    lev = b 
) 
) 

结果是：

Unit: milliseconds 
expr  min  lq  mean median  uq  max neval 
... 25.80961 28.79981 31.59974 30.79836 33.02461 98.02512 100 

Id和内容有126522元，Lev.terms有155591元，所以看起来我已经停止处理过早。因为最终我会处理约6M的文件，我需要问...有什么办法可以加速这段代码？

Answer 1

现在我已经加速它与

ind = which(lev %in% x) 
cnt = as.integer(factor(x, levels = lev[ind], ordered = TRUE)) 
sort(ind[cnt]) 

现在更换

sort(as.integer(factor(x, levels = lev, ordered = TRUE))) 

时序为：

expr  min  lq  mean median  uq  max neval 
... 5.248479 6.202161 6.892609 6.501382 7.313061 10.17205 100 

Answer 2

您是否尝试过试用sort method (algorithm)并指定快速排序或shell排序？

类似：

sort(as.integer(factor(x, levels = lev, ordered = TRUE)), method=shell) 

或：

sort(as.integer(factor(x, levels = lev, ordered = TRUE)), method=quick) 

此外，您可以尝试使用一些中间变量的排序算法再重新执行这些步骤的事件来评估嵌套函数并再次：

foo<-factor(x, levels = lev, ordered = TRUE) 
bar<-as.integer(foo) 
sort(bar, method=quick) 

或

sort(bar) 

祝你好运！

Answer 3

我通过创建quanteda::dfm()解决问题的多次迭代去（见GitHub repo here）和最快的解决方案，目前涉及使用data.table个Matrix包索引文件和标记化功能，计数文档中的功能，并直插入结果到稀疏矩阵是这样的：

require(data.table) 
require(Matrix) 

dfm_quanteda <- function(x) { 
    docIndex <- 1:length(x) 
    if (is.null(names(x))) 
     names(docIndex) <- factor(paste("text", 1:length(x), sep="")) else 
      names(docIndex) <- names(x) 

    alltokens <- data.table(docIndex = rep(docIndex, sapply(x, length)), 
          features = unlist(x, use.names = FALSE)) 
    alltokens <- alltokens[features != ""] # if there are any "blank" features 
    alltokens[, "n":=1L] 
    alltokens <- alltokens[, by=list(docIndex,features), sum(n)] 

    uniqueFeatures <- unique(alltokens$features) 
    uniqueFeatures <- sort(uniqueFeatures) 

    featureTable <- data.table(featureIndex = 1:length(uniqueFeatures), 
           features = uniqueFeatures) 
    setkey(alltokens, features) 
    setkey(featureTable, features) 

    alltokens <- alltokens[featureTable, allow.cartesian = TRUE] 
    alltokens[is.na(docIndex), c("docIndex", "V1") := list(1, 0)] 

    sparseMatrix(i = alltokens$docIndex, 
       j = alltokens$featureIndex, 
       x = alltokens$V1, 
       dimnames=list(docs=names(docIndex), features=uniqueFeatures)) 
} 

require(quanteda) 
str(inaugTexts) 
## Named chr [1:57] "Fellow-Citizens of the Senate and of the House of Representatives:\n\nAmong the vicissitudes incident to life no event could ha"| __truncated__ ... 
## - attr(*, "names")= chr [1:57] "1789-Washington" "1793-Washington" "1797-Adams" "1801-Jefferson" ... 
tokenizedTexts <- tokenize(toLower(inaugTexts), removePunct = TRUE, removeNumbers = TRUE) 
system.time(dfm_quanteda(tokenizedTexts)) 
## user system elapsed 
## 0.060 0.005 0.064 

这只是一门课程的片断，但完整的源代码是很容易在GitHub仓库中找到（dfm-main.R）。

我还鼓励您使用软件包中的完整dfm()。在你的文本

devtools::install_github("kbenoit/quanteda") 

怎么看，在性能方面的工作：您可以从CRAN或开发版本使用安装它。