Few weeks ago I was working with a small internal project that involves importing CSV file to Sql Server database and thought I'd share
the simple implementation that I did on
the project.
In this post I will demonstrate how to upload and import CSV file to SQL Server database. As some may have already know, importing CSV file to SQL Server is easy and simple but difficulties arise when
the CSV file contains, many columns with different data types. Basically,
the provider cannot differentiate data types between
the columns or
the rows, blindly it will consider them as a data type based on first few rows and leave all
the data which does not match
the data type. To overcome this problem, I used schema.ini file to define
the data type of
the CSV file and allow
the provider to read that and recognize
the exact data types of each column.
Now what is schema.ini?
Taken from
the documentation:
The Schema.ini is a information file, used to define
the data structure and format of each column that contains data in
the CSV file. If schema.ini file exists in
the directory, Microsoft.Jet.OLEDB provider automatically reads it and recognizes
the data type information of each column in
the CSV file. Thus,
the provider intelligently avoids
the misinterpretation of data types before inserting
the data into
the database. For more information see: http://msdn.microsoft.com/en-us/library/ms709353%28VS.85%29.aspx
Points to remember before creating schema.ini:
1.
The schema information file, must always named as 'schema.ini'.
2.
The schema.ini file must be kept in
the same directory where
the CSV file exists.
3.
The schema.ini file must be created before reading
the CSV file.
4.
The first line of
the schema.ini, must
the name of
the CSV file, followed by
the properties of
the CSV file, and then
the properties of
the each column in
the CSV file.
Here's an example of how
the schema looked like:
[Employee.csv]
ColNameHeader=False
Format=CSVDelimited
DateTimeFormat=dd-MMM-yyyy
Col1=EmployeeID Long
Col2=EmployeeFirstName Text Width 100
Col3=EmployeeLastName Text Width 50
Col4=EmployeeEmailAddress Text Width 50
To get started lets's go a head and create a simple blank database. Just for
the purpose of this demo I created a database called TestDB.
After creating
the database then lets go a head and fire up Visual Studio and then create a new WebApplication project.
Under
the root application create a folder called UploadedCSVFiles and then place
the schema.ini on that folder.
The uploaded CSV files will be stored in this folder after
the user imports
the file.
Now add a WebForm in
the project and set up
the HTML mark up and add one (1) FileUpload control one(1)Button and three (3) Label controls.
After that we can now proceed with
the codes for uploading and importing
the CSV file to SQL Server database. Here are
the full code blocks below:
1: using System;
2: using System.Data;
3: using System.Data.SqlClient;
4: using System.Data.OleDb;
5: using System.IO;
6: using System.Text;
7:
8: namespace WebApplication1
9: {
10: public partial class CSVToSQLImporting : System.Web.UI.Page
11: {
12: private string GetConnectionString()
13: {
14: return System.Configuration.ConfigurationManager.ConnectionStrings["DBConnectionString"].ConnectionString;
15: }
16: private void CreateDatabaseTable(DataTable dt, string tableName)
17: {
18:
19: string sqlQuery = string.Empty;
20: string sqlDBType = string.Empty;
21: string dataType = string.Empty;
22: int maxLength = 0;
23: StringBuilder sb = new StringBuilder();
24:
25: sb.AppendFormat(string.Format("CREATE TABLE {0} (", tableName));
26:
27: for (int i = 0; i < dt.Columns.Count; i++)
28: {
29: dataType = dt.Columns[i].DataType.ToString();
30: if (dataType == "System.Int32")
31: {
32: sqlDBType = "INT";
33: }
34: else if (dataType == "System.String")
35: {
36: sqlDBType = "NVARCHAR";
37: maxLength = dt.Columns[i].MaxLength;
38: }
39:
40: if (maxLength > 0)
41: {
42: sb.AppendFormat(string.Format(" {0} {1} ({2}), ", dt.Columns[i].ColumnName, sqlDBType, maxLength));
43: }
44: else
45: {
46: sb.AppendFormat(string.Format(" {0} {1}, ", dt.Columns[i].ColumnName, sqlDBType));
47: }
48: }
49:
50: sqlQuery = sb.ToString();
51: sqlQuery = sqlQuery.Trim().TrimEnd(',');
52: sqlQuery = sqlQuery + " )";
53:
54: using (SqlConnection sqlConn = new SqlConnection(GetConnectionString()))
55: {
56: sqlConn.Open();
57: SqlCommand sqlCmd = new SqlCommand(sqlQuery, sqlConn);
58: sqlCmd.ExecuteNonQuery();
59: sqlConn.Close();
60: }
61:
62: }
63: private void LoadDataToDatabase(string tableName, string fileFullPath, string delimeter)
64: {
65: string sqlQuery = string.Empty;
66: StringBuilder sb = new StringBuilder();
67:
68: sb.AppendFormat(string.Format("BULK INSERT {0} ", tableName));
69: sb.AppendFormat(string.Format(" FROM '{0}'", fileFullPath));
70: sb.AppendFormat(string.Format(" WITH ( FIELDTERMINATOR = '{0}' , ROWTERMINATOR = '\n' )", delimeter));
71:
72: sqlQuery = sb.ToString();
73:
74: using (SqlConnection sqlConn = new SqlConnection(GetConnectionString()))
75: {
76: sqlConn.Open();
77: SqlCommand sqlCmd = new SqlCommand(sqlQuery, sqlConn);
78: sqlCmd.ExecuteNonQuery();
79: sqlConn.Close();
80: }
81: }
82: protected void Page_Load(object sender, EventArgs e)
83: {
84:
85: }
86: protected void BTNImport_Click(object sender, EventArgs e)
87: {
88: if (FileUpload1.HasFile)
89: {
90: FileInfo fileInfo = new FileInfo(FileUpload1.PostedFile.FileName);
91: if (fileInfo.Name.Contains(".csv"))
92: {
93:
94: string fileName = fileInfo.Name.Replace(".csv", "").ToString();
95: string csvFilePath = Server.MapPath("UploadedCSVFiles") + "\\" + fileInfo.Name;
96:
97: //Save
the CSV file in
the Server inside 'MyCSVFolder'
98: FileUpload1.SaveAs(csvFilePath);
99:
100: //Fetch
the location of CSV file
101: string filePath = Server.MapPath("UploadedCSVFiles") + "\\";
102: string strSql = "SELECT * FROM [" + fileInfo.Name + "]";
103: string strCSVConnString = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + filePath + ";" + "Extended Properties='text;HDR=YES;'";
104:
105: // load
the data from CSV to DataTable
106:
107: OleDbDataAdapter adapter = new OleDbDataAdapter(strSql, strCSVConnString);
108: DataTable dtCSV = new DataTable();
109: DataTable dtSchema = new DataTable();
110:
111: adapter.FillSchema(dtCSV, SchemaType.Mapped);
112: adapter.Fill(dtCSV);
113:
114: if (dtCSV.Rows.Count > 0)
115: {
116: CreateDatabaseTable(dtCSV, fileName);
117: Label2.Text = string.Format("
The table ({0}) has been successfully created to
the database.", fileName);
118:
119: string fileFullPath = filePath + fileInfo.Name;
120: LoadDataToDatabase(fileName, fileFullPath, ",");
121:
122: Label1.Text = string.Format("({0}) records has been loaded to
the table {1}.", dtCSV.Rows.Count, fileName);
123: }
124: else
125: {
126: LBLError.Text = "File is empty.";
127: }
128: }
129: else
130: {
131: LBLError.Text = "Unable to recognize file.";
132: }
133:
134: }
135: }
136: }
137: }
The code above consists of three (3) private methods which are
the GetConnectionString(), CreateDatabaseTable() and LoadDataToDatabase().
The GetConnectionString() is a method that returns a string. This method basically gets
the connection string that is configured in
the web.config file.
The CreateDatabaseTable() is method that accepts two (2) parameters which are
the DataTable and
the filename. As
the method name already suggested, this method automatically create a Table to
the database based on
the source DataTable and
the filename of
the CSV file.
The LoadDataToDatabase() is a method that accepts three (3) parameters which are
the tableName, fileFullPath and delimeter value. This method is where
the actual saving or importing of data from CSV to SQL server happend.
The codes at BTNImport_Click event handles
the uploading of CSV file to
the specified location and at
the same time this is where
the CreateDatabaseTable() and LoadDataToDatabase() are being called. If you notice I also added some basic trappings and validations within that event.
Now to test
the importing utility then let's create a simple data in a CSV format. Just for
the simplicity of this demo let's create a CSV file and name it as "Employee" and add some data on it. Here's an example below:
1,VMS,Durano,
[email protected]
2,Jennifer,Cortes,
[email protected]
3,Xhaiden,Durano,
[email protected]
4,Angel,Santos,
[email protected]
5,Kier,Binks,
[email protected]
6,Erika,Bird,
[email protected]
7,Vianne,Durano,
[email protected]
8,Lilibeth,Tree,
[email protected]
9,Bon,Bolger,
[email protected]
10,Brian,Jones,
[email protected]
Now
save the newly created CSV file in some location in your hard drive.
Okay let's run
the application and browse
the CSV file that we have just created. Take a look at
the sample screen shots below:
After browsing
the CSV file.
After clicking
the Import Button
Now if we look at
the database that we have created earlier you'll notice that
the Employee table is created with
the imported data on it. See below screen shot.
That's it! I hope someone find this post useful!
Technorati Tags: ASP.NET,CSV,SQL,C#,ADO.NET
