使用R中的Data.Table或Rcpp字符串快速替代NA

Question

我有一个大表：10M行乘33列，其中28列有一些NA值。这些NA值需要使用locf()进行修补。我阅读了有关此主题的一些主题（efficiently locf by groups in a single R data.table和na.locf and inverse.rle in Rcpp）。但是，这些线程正在取代数字向量。我对Rcpp不太熟悉，所以我不知道如何改变他们的代码来迎合字符串---我的数据都是字符串。

这里是我的样本数据：

输入数据

Sample_File = structure(list(SO = c(112, 112, 112, 112, 113, 113, 113, 113), 
    Product.ID = c("AB123", "CD234", "DE345", "EF456", "FG456", 
    "GH567", "HI678", "IJ789"), Name = c(NA, NA, NA, "Human Being", 
    NA, "Lion", NA, "Bird"), Family = c(NA, NA, NA, "Homo Sapiens", 
    NA, NA, NA, "Passeridae"), SL1_Continent = c("Asia", NA, 
    "Asia", "Asia", NA, NA, NA, "Australia"), SL2_Country = c("China", 
    "China", NA, NA, NA, NA, NA, "Australia"), SL3_Direction = c("East", 
    NA, "East", "East", NA, NA, NA, "West"), Expiration_FY = c(2021, 
    NA, 2018, NA, 2012, 2012, NA, 2012), Flag = c("Y", NA, "N", 
    "N", NA, NA, NA, "TBD"), Insured = c("No", NA, NA, NA, NA, 
    NA, NA, "Yes"), Revenue = c(0, 478227.44, 0, 0, 0, 0, 125550.4, 
    44314.51), Quantity = c(1000, 100, 100, 4, 6, 6, 4, 6)), .Names = c("SO", 
"Product.ID", "Name", "Family", "SL1_Continent", "SL2_Country", 
"SL3_Direction", "Expiration_FY", "Flag", "Insured", "Revenue", 
"Quantity"), row.names = c(NA, 8L), class = "data.frame") 

下面是使用data.table我代码：

data.table::setDT(Sample_File) 
cols <- c("Name","Family","SL1_Continent","SL2_Country","SL3_Direction","Expiration_FY","Flag","Insured") 
Sample_File[, (cols):=lapply(.SD, function(x){na.locf(x,fromLast = TRUE,na.rm=TRUE)}), by = SO, .SDcols = cols] 

预期输出：

Output = structure(list(SO = c(112, 112, 112, 112, 113, 113, 113, 113), 
    Product.ID = c("AB123", "CD234", "DE345", "EF456", "FG456", 
    "GH567", "HI678", "IJ789"), Name = c("Human Being", "Human Being", 
    "Human Being", "Human Being", "Lion", "Lion", "Bird", "Bird" 
    ), Family = c("Homo Sapiens", "Homo Sapiens", "Homo Sapiens", 
    "Homo Sapiens", "Passeridae", "Passeridae", "Passeridae", 
    "Passeridae"), SL1_Continent = c("Asia", "Asia", "Asia", 
    "Asia", "Australia", "Australia", "Australia", "Australia" 
    ), SL2_Country = c("China", "China", "China", "China", "Australia", 
    "Australia", "Australia", "Australia"), SL3_Direction = c("East", 
    "East", "East", "East", "West", "West", "West", "West"), 
    Expiration_FY = c(2021, 2018, 2018, 2021, 2012, 2012, 2012, 
    2012), Flag = c("Y", "N", "N", "N", "TBD", "TBD", "TBD", 
    "TBD"), Insured = c("No", "No", "No", "No", "Yes", "Yes", 
    "Yes", "Yes"), Revenue = c(0, 478227.44, 0, 0, 0, 0, 125550.4, 
    44314.51), Quantity = c(1000, 100, 100, 4, 6, 6, 4, 6)), .Names = c("SO", 
"Product.ID", "Name", "Family", "SL1_Continent", "SL2_Country", 
"SL3_Direction", "Expiration_FY", "Flag", "Insured", "Revenue", 
"Quantity"), row.names = c(NA, -8L), class = "data.frame") 

虽然上述代码需要几分之一秒来执行，它需要约10分钟，我的原始数据集，其转换为〜280分钟，处理28分列即使data.table处理一列。

我假设我没有真正利用上面data.table的力量。我不太确定。我衷心感谢任何帮助，以加快na.locf()功能。

有没有更高效的方法取代NA以上？

Answer 1

为了本示例的目的，我简化了这个问题，但我想这很容易推广。下面的代码定义使用C++ 11语法RCPP的locppf功能：

#include <Rcpp.h> 
using namespace Rcpp; 

// [[Rcpp::plugins(cpp11)]] 

using Map = std::unordered_map<double, int> ; 
using Pair = Map::value_type ; 

// [[Rcpp::export]] 
CharacterVector locppf(NumericVector g, CharacterVector s) { 
    auto n = g.size() ; 
    CharacterVector out = clone(s) ; 

    Map map ; 
    for(int i=n-1; i>=0; i--){ 
    double value = g[i] ; 
    auto it = map.find(value) ; 

    if(it == map.end()){ 
     map.insert(Pair(value, i)) ; 
    } else { 
     // if the current value is NA, replace it with the data at correct idx 
     auto current = s[i] ; 
     if(CharacterVector::is_na(current)){ 
     out[i] = s[ it->second ] ; 
     } else { 
     it->second = i ; 
     } 
    } 
    } 
    return out ; 
} 

的想法是定义一个地图来追踪我们已经看到的东西是最后一次，该组中没有NA索引。我使用std::unordered_map<double, int>作为地图，因为您的示例还为该组使用了数字向量。

让我们打破相关掘金：

if(it == map.end()){ 
    map.insert(Pair(value, i)) ; 
} 

在这里，我们检查，如果地图上已经看到的当前值，如果没有，我们保留当前的指数。

 auto current = s[i] ; 
     if(CharacterVector::is_na(current)){ 
     out[i] = s[ it->second ] ; 
     } else { 
     it->second = i ; 
     } 

这里我们检查当前值是否为。

如果是这样，我们用结果向量填充我们之前保留的索引值。

如果不是，我们更改该组地图记住的索引。

现在，让我们给自己一些数据：

library("zoo") 
library("dplyr") 
library("data.table") 

with_holes <- function(x, p = .2){ 
    n <- length(x) 
    x[ sample(n, n*p) ] <- NA 
    x 
} 

n <- 1e6 
x <- sample(as.numeric(1:100), n, replace= TRUE) 
y <- with_holes(sample(letters, n, replace = TRUE)) 
d <- data_frame(x = x, y = y) 

和衡量各种选择时机：

使用dplyr语法与group_by，mutate和na.locf

> system.time(d %>% group_by(x) %>% mutate(y = na.locf(y, fromLast = TRUE, na.rm = FALSE))) 
    user system elapsed 
    0.173 0.023 0.198 

使用data.table语法与na.locf 。我不保证这是最好的data.table方式来做到这一点。

> d2 <- as.data.table(d) 
> system.time(d2[ , y := na.locf(y, fromLast = TRUE, na.rm = FALSE) , x ] ) 
    user system elapsed 
    0.159 0.030 0.188 

随着出定制locppf功能：

> system.time(locppf(d$x, d$y)) 
    user system elapsed 
    0.028 0.001 0.028