CSV files explained: delimiters, quotes, encoding and Excel

Updated:

A CSV file is plain text: one record per line, with the values separated by commas. Almost every database, spreadsheet and online service can export it, which makes it the usual way to move a table from one program to another. It is also easy to get wrong, because CSV was in use long before anyone wrote it down, and programs still disagree on the details.

This part of the series covers the rules CSV has, the separator and quoting problems that split values in the wrong place, character encoding, what spreadsheets change when they open a file, and a risk to keep in mind when you export data that others will open.

The rules CSV actually has

There is no single official standard. The closest is RFC 4180, published in October 2005, which documents the format most programs use and registers the text/csv media type. It is an informational document, not an Internet standard, and it sets out these rules:

Everything else varies. The Library of Congress, which catalogues file formats for long-term preservation, lists common departures: semicolons instead of commas, line breaks with only LF or only CR, backslashes or single quotes used instead of double quotes, and different ways of handling spaces. A file can therefore be a valid CSV for the program that wrote it and still be read wrongly by another.

Commas, semicolons and tabs

Where the comma is the decimal separator, as in much of Europe and South America, a comma between values would clash with numbers such as 1,20. Files made there often use a semicolon instead. Excel follows the computer's regional settings: Microsoft notes that setting the decimal separator to a comma makes Excel use a semicolon as the list separator.

That is the usual reason a file opens with everything in the first column: it was written with one separator and read with another. The same table, with a decimal comma in the price (the slash marks a new line):

Tab-separated files avoid most of this, because tabs rarely appear inside values. They often use the .tsv or .txt extension rather than .csv.

Quotes: the rule that matters most

Many broken CSV files come down to a value that needed quotes and did not get them, or a quote that was never closed. These follow RFC 4180:

If you write CSV yourself, quote any value that contains the separator, a double quote or a line break, and double the quotes inside it. Quoting every value is also allowed and avoids surprises.

Character encoding: why accents turn into ã

A CSV file says nothing about its character encoding. RFC 4180 notes that common usage is US-ASCII, which has no accented letters, and today most files are UTF-8. UTF-8 stores letters such as ã, é or € in two or three bytes. A program that reads those bytes with a different encoding, such as Windows-1252, shows each byte as a separate character:

Microsoft explains that Excel opens a UTF-8 CSV correctly by double-clicking only if the file starts with a byte order mark (BOM), three invisible bytes that announce UTF-8. Without one, Microsoft recommends importing the file instead, with Data > From Text/CSV. The same garbling in a file you created yourself means it was saved in one encoding and opened in another.

What spreadsheets change when you open a CSV

When Excel opens a .csv file directly, it guesses each value's type with its default settings. Microsoft documents what that does to data that only looks like a number or a date:

Value in the CSVAfter opening in ExcelWhy
0077Leading zeros are removed from numbers.
12345678901234561234567890123450, shown in scientific notationExcel keeps 15 significant digits; the rest become zeros.
SEPT22-SepRead as a date.
MARCH11-MarRead as a date.

The changes are kept if the file is saved again. The gene names in the table are real examples: a 2016 study in Genome Biology found such conversions in about one fifth of the papers that published gene lists as Excel files, and in 2019 the HUGO Gene Nomenclature Committee renamed the SEPT and MARCH genes to SEPTIN and MARCHF because of it. Postcodes, phone numbers, product codes and long identifiers are affected the same way.

To keep values as they are, import the file instead of opening it: Data > From Text/CSV, then Edit to open Power Query, select the columns and set their data type to Text. Microsoft also suggests renaming the file from .csv to .txt before opening it, which brings up the Text Import Wizard, where you choose how each column is read. Recent versions of Excel for Microsoft 365 and Excel 2024 also have Automatic Data Conversions settings that can turn some of these changes off.

A risk when exporting: formula injection

A value that starts with =, +, - or @ can be read as a formula by spreadsheet programs. If a website exports text that users typed, such as names or comments, a value written as a formula can be run as one when someone opens the file, instead of being shown as text. MITRE’s catalogue of software weaknesses lists this as CWE-1236, better known as CSV or formula injection.

It concerns software that writes CSV files from untrusted input. The usual defence is to put a single quote (') in front of any value that starts with one of those characters, although MITRE notes that no fix works in every spreadsheet program. If you receive a CSV from an unknown source, look at it in a text editor or import it as text before trusting what a spreadsheet shows.

CSV or JSON?

CSV is a flat table with no types: every value is text until a program decides otherwise. JSON, from the first part of this series, has numbers, true, false, null, lists and nested objects. CSV is smaller and opens in any spreadsheet; JSON is what most web APIs use. Converting between them means deciding what to do with the differences.

The nTools converter keeps every CSV value as text, so 007 and long identifiers survive intact, and turns nested JSON values into JSON text inside one cell. It reads comma, semicolon and tab-separated data, follows the quoting rules above, reports a quote that is never closed, and runs entirely in your browser.

CSV ↔ JSON converter

Frequently asked questions

Why does my CSV open with everything in one column?

The file uses a different separator from the one the program expects, usually a semicolon read as comma-separated or the other way round. Import it and choose the separator instead of double-clicking the file.

Why are the accents wrong after opening a CSV?

The file is UTF-8 and was read with another encoding. In Excel, import it with Data > From Text/CSV instead of double-clicking it, or save the file as UTF-8 with a byte order mark.

How do I keep leading zeros when opening a CSV in Excel?

Import the file instead of double-clicking it, and set the columns that hold codes, postcodes or phone numbers to Text before loading.

Can a CSV file have several sheets or formatting?

No. It holds one table of plain text. When you save a workbook as CSV, Excel saves only the current sheet, and features that plain text cannot hold, such as formatting and charts, are lost.

Is a semicolon-separated file still a CSV?

Strictly, RFC 4180 describes commas only, but semicolon-separated files are common and most programs accept them when you choose the separator.

Does the nTools converter upload my data?

No. The conversion runs in your browser, and the data is not sent anywhere.

Next in the series Text encodings explained: UTF-8, Base64 and URL encoding →

References

  1. RFC 4180 — Common Format and MIME Type for Comma-Separated Values (CSV) Files
  2. Library of Congress — CSV, Comma Separated Values (RFC 4180)
  3. Microsoft Support — Import or export text (.txt or .csv) files
  4. Microsoft Support — Keeping leading zeros and large numbers
  5. Microsoft Support — Opening CSV UTF-8 files correctly in Excel
  6. Ziemann, Eren and El-Osta — Gene name errors are widespread in the scientific literature (Genome Biology, 2016)
  7. HUGO Gene Nomenclature Committee — Summer newsletter 2020
  8. MITRE — CWE-1236: Improper Neutralization of Formula Elements in a CSV File

Related guides