PowerShell管道/ SQL插入查询数据限制（并增加它们？）

Question

对于可以分配给PowerShell中的变量的字符串的大小或对SQL INSERT查询中发送的文本大小的任何限制，是否有任何限制？

我有一个大的CSV文件进入PowerShell，并通过在foreach循环中的字符串构造我为每一行生成SQL INSERT查询。由此产生的INSERT查询; INSERT查询;超过大约4MB。

的SQL服务器有一个完美的模式来接收数据，但是，发送INSERT查询4MB集合（由;每个分隔）我得到我看来像长4MB组插入查询的错误被截断时不知何故。我想我已经遇到了某种限制。

有没有办法解决这个问题（以编程方式在PowerShell中）或增加可接受的SQL INSERT查询集合大小限制的方法？

我的代码是使用System.Data.SqlClient.SqlConnection和System.Data.sqlclient.SqlCommand。

较小的数据集工作正常，但较大的数据集提供了类似以下示例的错误。每个不同的数据集都会给出不同的“不正确语法附近”指示符。

 
Exception calling "ExecuteNonQuery" with "0" argument(s): "Incorrect syntax 
near '('." 
At C:\Users\stuart\Desktop\git\ADStfL\WorkInProgress.ps1:211 char:3 
+   $SQLCommand.executenonquery() 
+   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
    + CategoryInfo   : NotSpecified: (:) [], MethodInvocationException 
    + FullyQualifiedErrorId : SqlException 

Answer 1

根据我的经验，执行此操作的最佳方法是将CSV加载到DataTable中，然后使用SQLBulkCopy。

$ErrorActionPreference = 'Stop'; 

$Csv = Import-Csv -Path $FileName; 

$SqlServer = 'MyServer'; 
$SqlDatabase = 'MyDatabase'; 
$DestinationTableName = 'MyTable'; 

# Create Connection String 
$SqlConnectionString = 'Data Source={0};Initial Catalog={1};Integrated Security=SSPI' -f $SqlServer, $SqlDatabase; 


# Define your DataTable. The column order of the DataTable must either match the table in the database, or 
# you must specify the column mapping in SqlBulkCopy.ColumnMapping. If you have an IDENTITY column, it's a 
# bit more complicated 
$DataTable = New-Object -TypeName System.Data.DataTable -ArgumentList $DestinationTableName; 

$NewColumn = $DataTable.Columns.Add('Id',[System.Int32]); 
$NewColumn.AllowDBNull = $false; 

$NewColumn = $DataTable.Columns.Add('IntegerField',[System.Int32]); 
$NewColumn.AllowDBNull = $false; 

$NewColumn = $DataTable.Columns.Add('DecimalField',[System.Decimal]); 
$NewColumn.AllowDBNull = $false; 

$NewColumn = $DataTable.Columns.Add('VarCharField',[System.String]); 
$NewColumn.MaxLength = 50; 

$NewColumn = $DataTable.Columns.Add('DateTimeField',[System.DateTime]); 
$NewColumn.AllowDBNull = $false; 


# Populate your datatable from the CSV file 
# You may find that you need to type cast some of the fields. 
$Csv | ForEach-Object { 
    $NewRow = $DataTable.NewRow(); 
    $NewRow['Id'] = $_.Id; 
    $NewRow['IntegerField'] = $_.IntegerField; 
    $NewRow['DecimalField'] = $_.DecimalFiled; 
    $NewRow['StringField'] = $_.StringField1; 
    $NewRow['DateTimeField'] = $_.DateTimeField1; 

    $DataTable.Rows.Add($NewRow); 
} 

# Create Connection 
$SqlConnection = New-Object -TypeName System.Data.SqlClient.SqlConnection -ArgumentList $SqlConnectionString; 

# Open Connection 
$SqlConnection.Open(); 

# Start Transaction 
$SqlTransaction = $SqlConnection.BeginTransaction(); 

# Double check the possible options at https://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlbulkcopyoptions(v=vs.110).aspx 
# If you need multiple then -bor them together 
$SqlBulkCopyOptions = [System.Data.SqlClient.SqlBulkCopyOptions]::CheckConstraints; 

# Create SqlBulkCopy class 
$SqlBulkCopy = New-Object -TypeName System.Data.SqlClient.SqlBulkCopy -ArgumentList $SqlConnection, $SqlBulkCopyOptions, $SqlTransaction; 

# Specify destination table 
$SqlBulkCopy.DestinationTableName = $DestinationTableName; 

# Do the insert; rollback on error 
try { 
    $SqlBulkCopy.WriteToServer($DataTable); 
    $SqlTransaction.Commit(); 
} 
catch { 
    # Roll back transaction and rethrow error 
    $SqlTransaction.Rollback(); 
    throw ($_); 
} 
finally { 
    $SqlConnection.Close(); 
    $SqlConnection.Dispose(); 
} 

另一种方法是使用SqlCommand的，做它逐行：

$ErrorActionPreference = 'Stop'; 

$Csv = Import-Csv -Path $FileName; 

$SqlServer = 'MyServer'; 
$SqlDatabase = 'MyDatabase'; 

# Create Connection String 
$SqlConnectionString = 'Data Source={0};Initial Catalog={1};Integrated Security=SSPI' -f $SqlServer, $SqlDatabase; 

# Create Connection 
$SqlConnection = New-Object -TypeName System.Data.SqlClient.SqlConnection -ArgumentList $SqlConnectionString; 

# Create Command 
$InsertCommandText = 'INSERT INTO DestinationTable (Id, IntegerField, DecimalField, StringField, DateTimeField) VALUES (@Id, @IntegerField, @DecimalField, @StringField, @DateTimeField)'; 
$InsertCommand = New-Object -TypeName System.Data.SqlClient.SqlCommand -ArgumentList $SqlConnection; 

[void]$InsertCommand.Parameters.Add('@Id', [System.Data.SqlDbType]::Int); 
[void]$InsertCommand.Parameters.Add('@IntegerField', [System.Data.SqlDbType]::Int); 
[void]$InsertCommand.Parameters.Add('@DecimalField', [System.Data.SqlDbType]::Decimal); 
[void]$InsertCommand.Parameters.Add('@StringField', [System.Data.SqlDbType]::VarChar,50); 
[void]$InsertCommand.Parameters.Add('@DateTimeField', [System.Data.SqlDbType]::DateTime); 

# Open connection and start transaction 
$SqlConnection.Open() 
$SqlTransaction = $SqlConnection.BeginTransaction(); 
$InsertCommand.Transaction = $SqlTransaction; 
$RowsInserted = 0; 

try { 
    $line = 0; 
    $Csv | ForEach-Object { 
     $line++; 

     # Specify parameter values 
     $InsertCommand.Parameters['@Id'].Value = $_.Id; 
     $InsertCommand.Parameters['@IntegerField'].Value = $_.IntegerField; 
     $InsertCommand.Parameters['@DecimalField'].Value = $_.DecimalField; 
     $InsertCommand.Parameters['@StringField'].Value = $_.StringField; 
     $InsertCommand.Parameters['@DateTimeField'].Value = $_.DateTimeField; 

     $RowsInserted += $InsertCommand.ExecuteNonQuery(); 

     # Clear parameter values 
     $InsertCommand.Parameters | ForEach-Object { $_.Value = $null }; 
    } 
    $SqlTransaction.Commit(); 
    Write-Output "Rows affected: $RowsInserted"; 
} 
catch { 
    # Roll back transaction and rethrow error 
    $SqlTransaction.Rollback(); 
    Write-Error "Error on line $line" -ErrorAction Continue; 
    throw ($_); 
} 
finally { 
    $SqlConnection.Close(); 
    $SqlConnection.Dispose(); 
} 

编辑：哦，我忘了一件很重要的一点。如果您需要在数据库中将字段的值设置为空，则需要将其值设置为[System.DBNull]::Value，而不是$null。