Import multiple CSV files To MySQL: Complete Guide

It is possible to Import multiple CSV files to MySQL database or table. CSV is referred to as a plain text file. It stores data in a table format. As the name suggests, it is primarily used to transfer data between character-separated values is another name for it. To delimit (separate) data, the comma character is usually used, but other characters, such as semicolons, can be used instead.

CSV (Comma-separated values) is known for its simplicity of the CSV formats which makes it unique. These CSV files are a widely used format for storing and processing data. Also for various scientific and business operations that use this format. Tabular data or data that can be divided into rows and columns can be stored in CSV format. There can be multiple fields in a row, and each row can consist of comma-separated fields in a line. Also, many other symbols, such as spaces, tabs, and hyphens, are used to separate words.

CSV to MySQL software is an easy and fast way to load CSV files into the MySQL database. You can use this program to import the data from CSV files to MySQL server. The program requires only one click to complete the task, and can automatically separate lines of each column. It is such a good helper for you to transfer the old CSV data into the MySQL server.

Whenever we export complex data from one application to a CSV file and then need to import the data from the file into another application, this file comes into play. Using a spreadsheet program such as Microsoft Excel or Google Spreadsheets, we can work with the CSV file. It is not possible to save formulas in a CSV file. The structure of the CSV file is a list of data separated by commas.

Below are some things to check before you Import multiple CSV files to MySQL database –

• Empty table – This is important as the data should be imported into an empty table.
• Create CSV file – Create a CSV file that matches the order, number of columns, and type of table.
• INSERT and FILE privileges are granted to the MySQL user account connected to the database server.

Why Import multiple CSV files to MySQL?

If you have a requirement to Import multiple CSV files to MySQL with table creation, you can utilize this program. Normally the whole process will take less than few minutes when all data is provided in .csv format. The user can add an appropriate amount of records to each file and the program will handle the “Seated Auto Increment” concept with multiple .csv files. 

The truth is that even customer data is not always clean and consistent, and it’s probably more common than not quite If you use an ETL (Extract, Transform, and Load) process, you’ll summary: Inconsistent data needs to be examined first.

However, there are times when it can be difficult to analyze data from hundreds or thousands of CSV files. Our conversion process would require writing a condition to look across all CSV files for parts that do not match a particular pattern to find out which ones do. This requires writing additional analysis and conditional code to look for more conditions. Our data can now be analyzed with the power of SQL since all CSV data has been included in one or more MySQL database tables.

Methods to use CSV files in a database

MySQL supports a special function for importing a single CSV/TSV file into a table. It’s called “LOAD DATA INFILE”. This can be a more efficient way to import large datasets from CSV, TSV, or even your own compiled format using SPLIT() and SUBSTRING(). This article explains how to use the LOAD DATA statement to effectively Import multiple CSV files to MySQL from a CSV file on Linux or Unix systems. 

LOAD DATA INFILE accepts a file from the filesystem or the command line. By default, it expects a CSV file, so if your file is not a CSV file, you can use a different delimiter with a simple option flag. When you do use a delimiter other than a comma, be sure to enclose the entire contents of the file within that alternative character. 

Other command-line options support importing into one table or multiple tables, concurrently. The best thing about this command is its efficiency: MySQL reads data as needed from the input stream, no matter how big the input file is.

It is also possible to read the CSV data into memory and construct INSERT queries for each record. However, this method is usually only useful when only specific records are to be inserted based on specific conditions. To import an entire CSV file into a database table, the LOAD DATA INFILE command is more efficient, easier, and faster.

However, suppose we want to insert multiple CSV files into a database table, or even hundreds of thousands (or more). What can we do to accomplish this? Let us first look at why this is important and why it might make sense to do this.

Import and convert data

It is not uncommon for the data format of the CSV file to be different from that of a database table when it is imported into it. It is possible to convert a column by using the SET clause in conjunction with a LOAD DATA INFILE statement in this scenario. So, let us say you have a CSV file with a date column that is in “dd/mm/yy” format and does not match the corresponding column from STR TO DATE () must be used to convert it to a MySQL date format.

LOAD DATA INFILE ‘C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/address_book.csv’   

INTO TABLE address_book   

FIELDS TERMINATED BY ‘,’  

OPTIONALLY ENCLOSED BY ‘”‘  

LINES TERMINATED BY ‘\r\n’   

IGNORE 1 ROWS  

(Mobile,@Date,Address)  

SET Date = STR_TO_DATE(@Date, ‘%m/%d/%Y’);

Steps to Import multiple CSV files to MySQL –

You can write a tool to help you import your CSV files. And you may need intermediate programming skills to do that. In a single Database Table, it will load all the columns from multiple CSV files, regardless of where the columns are located in the CSV files.

1. Import your CSV file to the MySQL server –

To Import multiple CSV files to MySQL database using the LOAD DATA INFILE statement. It is also possible to read CSV files on the local system by adding the clause LOCAL to a statement such as “LOAD DATA INFILE”. If you are using a Windows computer, this file will be uploaded to C:windowstemp and /tmp on Linux systems. Neither MySQL nor any other database engine will determine or configure this when using MySQL.

When you Import multiple CSV files to MySQL using the Local option, the file may load more slowly. This is because this option takes a long time to transfer the file. It is not required that the MySQL user account has FILE permission when using LOCAL. There are some security issues with the LOAD DATA LOCAL INFILE statement that imports the file from a local server.

LOAD DATA LOCAL INFILE ‘C:/windows/temp/address_book.csv’   

INTO TABLE address_book   

FIELDS TERMINATED BY ‘,’  

OPTIONALLY ENCLOSED BY ‘”‘  

LINES TERMINATED BY ‘\r\n’   

IGNORE 1 ROWS;  

2. Create an empty table –

The first step is to create a new, empty MySQL table. As a result, the table has the same number of columns as our CSV file.

On this page, we have created a table named Users that has six columns: ID, First Name, Last Name, Email Address, and Account Creation. What we have here is a copy of what is in the CSV files.

• When a new record is added to a table, AUTOINCREMENT creates a unique number.
• The data types INT, VARCHAR, and DATE are all data If a column has a specific data type, the column’s data type can be found.
• It is possible to leave a column blank by default (represents a missing value). The NOT NULL statement tells MySQL that NULL values are not allowed in columns. This ensures that columns always contain a value.
• As the name implies, VARCHAR(255) stores a variable The maximum size is specified in parentheses.
• PRIMARY KEY (id) notes that there are no NULL values in a primary key column.

In this case, let us assume that we have a table called “address book” that contains no data, as in:

3. Import File into the table –

It is possible to import a file into a table after it has been opened by clicking on the “Import File” button.

The expected location of a file is determined by the keyword LOCAL. Files are sent from clients to servers, which read and send them from the client’s computer. Otherwise, the file must be on the server’s host and read directly from the server if LOCAL is not specified. This is because the comma is used as a delimiter.

A newline character terminates each line of the CSV file, so LINES TERMINATED BY ‘n’ is used. IGNORE 1 ROWS is used because the header is already contained in the MySQL table.

The default format of the DATE data type is YYYY-MM- DD. There is a discrepancy between the dates in the CSV and Therefore, we need to use the function STR TO DATE () to change the format.

In the case of “address book” –

You can import a CSV file by clicking the Import button. Then click on the Next button.

You can also check whether your data has been imported into your table or not. 

Follow these steps: mysql> SELECT * FROM users;

4. Once we have selected the target table, click on Use existing table or Create table.

5. Import settings must be configured and then Next must be clicked to proceed.

6. When it’s time to finish, we’ll need to keep an eye on the execution process and click on Next->Next>Finish to complete the process.

CONCLUSION

These are the easy steps that can help you Import multiple CSV files to MySQL server. These steps require some basic coding skills, but following the given examples above can make the whole process easier.