如何使用R将大型CSV数据文件分解为单个数据文件？

Question

我有一个CSV文件，其第一行包含变量名称，其余的行包含数据。有什么方法可以把它分解成每个只包含R中一个变量的文件？这个解决方案会变得强大吗？例如。如果输入文件大小为100G会怎样？

输入文件看起来像

var1,var2,var3 
1,2,hello 
2,5,yay 
... 

我要创建3（或包含很多变量）的文件var1.csv，var2.csv，var3.csv ，使文件类似于 文件1

var1 
1 
2 
... 

文件2

var2? 
2 
5 
... 

文件3

var3 
hello 
yay 

我在Python（How to break a large CSV data file into individual data files?）的解决方案，但我不知道是否R可以做同样的事情？ Python代码必须逐行读取csv文件，然后逐行写入一行。 R可以做同样的事吗？命令read.csv一次读取整个文件，这可以减慢整个过程。另外它不能读取100G文件并处理它，因为R试图将整个文件读入内存。我无法在R中找到一个命令，让您逐行读取一个csv文件。请帮忙。谢谢！！

Answer 1

您可以将scan，然后write一次一行地写入一个或多个文件。

i <- 0 
while({x <- scan("file.csv", sep = ",", skip = i, nlines = 1, what = "character"); 
     length(x) > 1}) { 
    write(x[1], "file1.csv", sep = ",", append = T) 
    write(x[2], "file2.csv", sep = ",", append = T) 
    write(x[3], "file3.csv", sep = ",", append = T) 
    i <- i + 1 
} 

编辑!!我正在使用上述数据，复制1000次以上。当我们始终打开文件连接时，我已经对速度进行了比较。

ver1 <- function() { 
    i <- 0 
    while({x <- scan("file.csv", sep = ",", skip = i, nlines = 1, what = "character"); 
     length(x) > 1}) { 
    write(x[1], "file1.csv", sep = ",", append = T) 
    write(x[2], "file2.csv", sep = ",", append = T) 
    write(x[3], "file3.csv", sep = ",", append = T) 
    i <- i + 1 
    } 
} 

system.time(ver1()) # w/ close to 3K lines of data, 3 columns 
## user system elapsed 
## 2.809 0.417 3.629 

ver2 <- function() { 
    f <- file("file.csv", "r") 
    f1 <- file("file1.csv", "w") 
    f2 <- file("file2.csv", "w") 
    f3 <- file("file3.csv", "w") 
    while({x <- scan(f, sep = ",", skip = 0, nlines = 1, what = "character"); 
     length(x) > 1}) { 
    write(x[1], file = f1, sep = ",", append = T, ncol = 1) 
    write(x[2], file = f2, sep = ",", append = T, ncol = 1) 
    write(x[3], file = f3, sep = ",", append = T, ncol = 1) 
    } 
    closeAllConnections() 
} 

system.time(ver2()) 
## user system elapsed 
## 0.257 0.098 0.409