PowerShell的：获取内容从大文件（服务器列表）

Question

我从文本文件（serverlist.txt）服务器100,000名单

当我在一个镜头中运行它会爆我的内存和CPU以及时间需要更长的时间（大约3天）才能完成DNS查找扫描。

我试图拆分下面包含20k个服务器列表的文件，并且可以完成为每个文件扫描最多10分钟。

serverlist1.txt 
serverlist2.txt 
serverlist3.txt 
serverlist4.txt 
serverlist5.txt 

---

$objContainer = @() 
$values = @() 
$domains = Get-Content -path "serverlist1.txt" 
$named = 0 
$timestamp= get-date 

$domains | ForEach-Object { 
    $domain = $_ 
    nslookup $domain 2>&1 | ForEach-Object { 
     if ($_ -match '^Name:\s*(.*)$') { 
      $values += $matches[1] 
      $named = 1; 
     } elseif (($_ -match '^.*?(\d*\.\d*\.\d*\.\d*)$') -and ($named -eq 1)) { 
      $values += $matches[1] 
     } elseif ($_ -match '^Aliases:\s*(.*)$') { 
      $values += $matches[1] 
     } 
    } 

    $obj = New-Object -TypeName PSObject 
    #$obj | Add-Member -MemberType NoteProperty -name 'Domain' -value $domain 
    $obj | Add-Member -MemberType NoteProperty -name 'Name' -value $values[0] 
    $obj | Add-Member -MemberType NoteProperty -name 'IP Address' -value $values[1] 
    $obj | Add-Member -MemberType NoteProperty -name 'Alias' -value $values[2] 
    $obj | Add-Member -MemberType NoteProperty -name 'Timestamp' -value $timestamp 
    $objContainer += $obj 

    $values = @() 
    $named = 0 
} 

Write-Output $objContainer 
$objContainer | Export-csv "dnslog_$((Get-Date).ToString('MM-dd-yyyy_hh-mm-ss')).csv" -NoTypeInformation 

我的问题是，如何执行一次循环，并从文本文件输入生成dnslog（日期时间）后.csv文件

如：

1. 运行powershell脚本。\ filename.ps1
2. 从serverlist1.txt输入
3. 输出dnslog（日期时间）.csv文件
4. 输入从serverlist2.txt
5. 输出dnslog（日期时间）.csv文件从serverlist3.txt
6. 输出dnslog（日期时间）
7. 输入.csv文件
8. 输入从serverlist4.txt
9. 输出dnslog（日期时间）的.csv从serverlist5.txt
10. 输入
11. 输出dnslog（日期时间）的.csv

完成！

如果我有多于5个文本文件列表，它将继续从输入文件循环直到完成。

Answer 1

添加到克里斯的回答我还要补充一个ReadCount标志的获取内容如下所示：

Get-Content -path "serverlist1.txt" -ReadCount 1 | % { 

这将节省不必将整个文件读入内存中。

Answer 2

你应该考虑运行这个并行批处理作业。你已经尝试过吗？

您可以通过删除所有对内存的提交（使用+ =进行变量赋值和数组重写）来处理RAM破坏问题。

$timestamp = get-date 

Get-Content -path "serverlist1.txt" | ForEach-Object { 
    $domain = $_ 

    # You can clear this here. 
    $values = @() 
    $named = 0 

    # There are potentially better options than nslookup. 
    # Needs a bit of care to understand what's an alias here though. 
    # [System.Net.Dns]::GetHostEntry($domain) 
    # And if you don't like that, quite a few of us have written equivalent tools in PowerShell. 
    nslookup $domain 2>&1 | ForEach-Object { 
     if ($_ -match '^Name:\s*(.*)$') { 
      $values += $matches[1] 
      $named = 1; 
     } elseif (($_ -match '^.*?(\d*\.\d*\.\d*\.\d*)$') -and ($named -eq 1)) { 
      $values += $matches[1] 
     } elseif ($_ -match '^Aliases:\s*(.*)$') { 
      $values += $matches[1] 
     } 
    } 

    # Leave the output object in the output pipeline 
    # If you're running PowerShell 3 or better: 
    [PSCustomObject]@{ 
     Domain  = $domain 
     Name   = $values[0] 
     'IP Address' = $values[1] 
     Alias  = $values[2] 
     TimeStamp = $timestamp 
    } 
    # PowerShell 2 is less flexible. This or Select-Object. 
    #$obj = New-Object -TypeName PSObject 
    ##$obj | Add-Member -MemberType NoteProperty -name 'Domain' -value $domain 
    #$obj | Add-Member -MemberType NoteProperty -name 'Name' -value $values[0] 
    #$obj | Add-Member -MemberType NoteProperty -name 'IP Address' -value $values[1] 
    #$obj | Add-Member -MemberType NoteProperty -name 'Alias' -value $values[2] 
    #$obj | Add-Member -MemberType NoteProperty -name 'Timestamp' -value $timestamp 
    # To leave this in the output pipeline, uncomment this 
    # $obj 

    # No version of PowerShell needs you to do this. It's a good way to ramp up memory usage 
    # for large data sets. 
    # $objContainer += $obj 
} | Export-Csv "dnslog_$(Get-Date -Format 'MM-dd-yyyy_hh-mm-ss').csv" -NoTypeInformation