VB.Net导入CSV文件到SQL数据库

Question

所以，我需要帮助搞清楚如何使这项工作，并希望开始与其他VB专家的对话。

我有一个CSV文件每小时转储到一个导入文件夹。我编写了一个扫描该文件夹的VB应用程序，获取CSV文件并将其导入到SQL数据库中。那里没有问题。

尝试查询SQL中的数据时出现问题。所有我需要的是一个领域的重要数据，这里的字段内容修改的例子：

Successful Write by DOMAIN\USERLOGIN on /SOMENETWORKSHARE/FOLDER/MyFolder/USERLOGIN/DesktopBackUp/log.txt 

所以基本上我需要一种方法来分割上述行到多个领域，它需要发生在SQL导入的时间。

Successful Write“是事件”通过“不改变以往可作为一个分隔符”

DOMAIN/USERLOGIN字段1

“DOMAIN不改变但用户登陆将”字段2

在“不改变以往任何时候都可以用作分隔符”

/SOMENETWORKSHARE/FOLDER/MyFolder/USERLOGIN/DesktopBackUp/“这是文件路径”栏3

log.txt“已修改的文件名”字段4

有时候没有文件名，只是一个文件路径。

我很乐于提供建议，但是，CSV文件不会更改，我无法更改文件传递给我的方式。

在此先感谢。

这里是数据库结构

SET ANSI_NULLS ON 
GO 
SET QUOTED_IDENTIFIER ON 
GO 
SET ANSI_PADDING ON 
GO 
CREATE TABLE [dbo].[SACL](
    [Unique ID] [varchar](max) NULL, 
    [Event ID] [varchar](max) NULL, 
    [Event Time] [datetime] NULL, 
    [Severity] [varchar](max) NULL, 
    [Workspace] [varchar](max) NULL, 
    [Headline] [varchar](max) NULL, 
    [ID] [int] IDENTITY(1,1) NOT NULL, 
    [Event] [varchar](max) NULL, 
    [Username] [varchar](max) NULL, 
    [Path] [varchar](max) NULL, 
    [Filename] [varchar](max) NULL, 
PRIMARY KEY CLUSTERED 
(
    [ID] ASC 
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] 
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY] 
GO 
SET ANSI_PADDING OFF 
GO 

Answer 1

我希望这可以帮助你。在调用此函数后，可以使用结果将数据库中其他表中的每个字段包含在其中。

Public Function divideThis(CompleteString As String) 

    Dim n As Integer = InStr(CompleteString, " by ") 
    Dim Field1 As String = CompleteString.Substring(0, n - 1) 

    Dim m As Integer = InStr(CompleteString, " on ") 
    Dim Field2 As String = CompleteString.Substring(n + 3, m - n - 4) 

    Dim o As Integer = InStrRev(CompleteString, "/") 
    Dim Field3 As String = CompleteString.Substring(m + 3, o - m - 3) 

    Dim Field4 As String = CompleteString.Substring(o, CompleteString.Length - o) 

    Dim fragmented As String() = {Field1, Field2, Field3, Field4} 

    Return (fragmented) 

End Function 

Answer 2

确定试试这个...它会导入CSV文件，拆分所需的字段并合并它。

我修改了代码以在LINQ语句中包含RegEx来拆分该字段，如果没有文件名，它将工作，但是在那里必须有一个正斜杠并且这必须是唯一字段（在你的CSV文件中），其中包含正斜杠...原因是，如果正斜杠丢失，并且没有文件名，你将丢失一个字段，或者如果csv文件中的其他字段包含斜杠.....

Sub Main() 
    Dim regex As Text.RegularExpressions.Regex = New Text.RegularExpressions.Regex("\d+") 
    Dim readText() As String = File.ReadAllLines("<Path to your CSV file>") 

    Using connection As SqlConnection = New SqlConnection("<DB connection string>") 

     Dim command As SqlCommand = connection.CreateCommand() 
     connection.Open() 

     command.CommandType = System.Data.CommandType.StoredProcedure 
     command.CommandText = "SP_InsertCSVData" 

     For i As Integer = 0 To readText.Count 
      Dim readRecord() As String = readText.ToArray 

      Dim CSVData As XElement = New XElement("Root", 
          From str In readText 
          Let fields = Text.RegularExpressions.Regex.Replace(str, "(by|on|\/(?=[^\/]*$))", ",").Split(",") 
          Select New XElement("CSVDataRecord", 
           New XAttribute("Field_1", fields(0)), 
           New XAttribute("Field_2", fields(1)), 
           New XAttribute("Field_3", fields(2)), 
           New XAttribute("Field_4", fields(3)), 
           New XAttribute("Field_5", fields(4)), 
           New XAttribute("Field_6", fields(5)), 
           New XAttribute("Field_7", fields(6)), 
           New XAttribute("Field_8", fields(7)) 
          ) 
         ) 

      command.Parameters.Clear() 
      command.Parameters.Add(New SqlParameter With 
      { 
       .ParameterName = "@InputXML", 
       .DbType = DbType.Xml, 
       .Value = CSVData.CreateReader 
      }) 

      command.ExecuteNonQuery() 

     Next 
    End Using 

End Sub 

这里是新表...

/****** Object: Table [dbo].[CSV_Import_Table] Script Date: 06/02/2016 01:55:32 ******/ 
SET ANSI_NULLS ON 
GO 

SET QUOTED_IDENTIFIER ON 
GO 

CREATE TABLE [dbo].[CSV_Import_Table]( 
[Field_1] [nvarchar](max) NULL, 
[Field_2] [nvarchar](max) NULL, 
[Field_3] [nvarchar](max) NULL, 
[Field_4] [nvarchar](max) NULL, 
[Field_5] [nvarchar](max) NULL, 
[Field_6] [nvarchar](max) NULL, 
[Field_7] [nvarchar](max) NULL, 
[Field_8] [nvarchar](max) NULL 
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY] 

GO 

这里是SP ...

/****** Object: StoredProcedure [dbo].[SP_InsertCSVData] Script Date: 06/02/2016 01:56:20 ******/ 
SET ANSI_NULLS ON 
GO 
SET QUOTED_IDENTIFIER ON 
GO 
CREATE PROCEDURE [dbo].[SP_InsertCSVData] (@InputXML xml) 
as 
begin 
MERGE CSV_Import_Table AS main 
USING (select Row.id.value('@Field_1','[nvarchar](MAX)') as Field_1, Row.id.value('@Field_2','[nvarchar](MAX)') as Field_2, Row.id.value('@Field_3','[nvarchar](MAX)') as Field_3, Row.id.value('@Field_4','[nvarchar](MAX)') as Field_4, Row.id.value('@Field_5','[nvarchar](MAX)') as Field_5, Row.id.value('@Field_6','[nvarchar](MAX)') as Field_6, Row.id.value('@Field_7','[nvarchar](MAX)') as Field_7, Row.id.value('@Field_8','[nvarchar](MAX)') as Field_8 
      from @InputXML.nodes('/Root/CSVDataRecord') as Row(id)) as stage 
ON main.Field_1=stage.Field_1 
WHEN MATCHED THEN 
    UPDATE SET main.Field_1=stage.Field_1, main.Field_2=stage.Field_2, main.Field_3=stage.Field_3, main.Field_4=stage.Field_4 , main.Field_5=stage.Field_5 , main.Field_6=stage.Field_6, main.Field_7=stage.Field_7, main.Field_8=stage.Field_8  
WHEN NOT MATCHED THEN 
    INSERT (Field_1, Field_2, Field_3, Field_4, Field_5, Field_6, Field_7, Field_8) VALUES (stage.Field_1, stage.Field_2, stage.Field_3, stage.Field_4, stage.Field_5, stage.Field_6, stage.Field_7, stage.Field_8); 
end 

而且我猜你CSV文件看起来是这样的....

记录1场1，通过DOMAIN \ USERLOGIN成功的写上/ SOMENETWORKSHARE /文件夹/ MyFolder中/ USERLOGIN/DesktopBackUp /Log.txt，字段6，字段7，字段8

Answer 3

下面是一个选择，你应该可以在你的表上运行来测试。

更换FROM (....) as tbl位与真表

你可以看到，在选择的表情是相当丑陋，笨拙等都有想想你是否幸福维持这种

有一个WHERE在阻止无效数据进入的最终目标。如果缺少分隔符的数据进入，那么索引将最终为负数，并且会出现各种错误。

有了这些东西，数据质量始终是一个问题。因此，我们不希望通过假设该域永远具有相同的名称，或者假设您只会从一个域导入数据。

所以你需要对你的桌子运行这个，看它是否有效。如果是的话，下一步是：

• 把文件名从路径
• 改变成一个更新

这应该是简单的

SELECT 
SRC, 
LEFT(SRC,CHARINDEX(' by ',SRC)) AS Operation, 
SUBSTRING(
    SRC, 
    CHARINDEX(' by ',SRC)+4, 
    CHARINDEX(' on ',SRC) - CHARINDEX(' by ',SRC)-4 
    ) AS LoginName, 

RIGHT(
    SRC, 
    LEN(SRC) - CHARINDEX(' on ',SRC)-4 
) AS FullPath, 

REVERSE(LEFT(
    REVERSE(SRC), 
    CHARINDEX('/',REVERSE(SRC))-1) 
    ) AS PathAndFileName 
FROM 
(
SELECT 'Successful Write by DOMAIN\USERLOGIN on /SOMENETWORKSHARE/FOLDER/MyFolder/USERLOGIN/DesktopBackUp/log.txt' AS SRC 
UNION ALL 
SELECT 'some invalid stuff' AS SRC 
) as tbl 
WHERE SRC LIKE '% by %\% on %/%' 

的UPDATE语句适用这将是：

UPDATE [dbo].[SACL] 
SET 
[Event]= 
LEFT(Headline,CHARINDEX(' by ',Headline)), 
[Username] = 
SUBSTRING(
    Headline, 
    CHARINDEX(' by ',Headline)+4, 
    CHARINDEX(' on ',Headline) - CHARINDEX(' by ',Headline)-4 
    ), 
[Path] = 
SUBSTRING(
    Headline, 
    CHARINDEX(' by ',Headline)+4, 
    CHARINDEX(' on ',Headline) - CHARINDEX(' by ',Headline)-4 
    ), 

[FileName]= 
REVERSE(LEFT(
    REVERSE(Headline), 
    CHARINDEX('/',REVERSE(Headline))-1) 
    ) 
WHERE 
-- Only update rows that fit the pattern 
Headline LIKE '% by %\% on %/%' 
-- Only update rows that haven't been processed 
AND ([Event] IS NULL OR [Event] = '') 

请注意：

• 我不知道哪个字段包含完整的字符串。我假设Headline
• 我建议你标准化Event字段的内容。 NULL =未处理，空白意味着别的东西（已处理但无法解决）
• 我还建议您添加ImportedDate和ProcessedDate字段，它是datetime类型。 ImportedDate的default为GETDATE()，这意味着这是记录导入的日期时间。 ProcessedDate应该由UPDATE语句更新为GETDATE()，这会告诉您记录何时更新。