Pandas read_csv: Loading and Parsing CSV Files
The pd.read_csv() function is the workhorse of data loading in pandas. It does far more than just read comma-separated files—it can handle different separators, parse dates, skip rows, select columns, specify data types, and recover from malformed lines. Mastering its parameters keeps your data import clean and predictable. (pandas 2.3.3 used in examples below.)
Basic CSV Reading
Start with a simple example. Here’s a sample CSV file:
Name,Age,Salary,Start_Date,Notes
Alice,28,75000,2020-01-15,Active
Bob,35,85000,2019-03-20,Active
Charlie,42,95000,2018-05-10,On leave
David,29,72000,2021-07-01,Active
Eve,38,88000,2019-11-15,
Frank,31,78000,2020-12-01,New hire
,45,92000,2017-02-28,MalformedRead it with pd.read_csv():
import pandas as pd
df = pd.read_csv("sample.csv")
print(df)Output:
Name Age Salary Start_Date Notes
0 Alice 28 75000 2020-01-15 Active
1 Bob 35 85000 2019-03-20 Active
2 Charlie 42 95000 2018-05-10 On leave
3 David 29 72000 2021-07-01 Active
4 Eve 38 88000 2019-11-15 NaN
5 Frank 31 78000 2020-12-01 New hire
6 NaN 45 92000 2017-02-28 MalformedBy default, pandas treats empty cells as NaN and infers the header from the first row.
Checking What You Loaded
Before doing analysis, verify the shape and data types:
print(df.shape)
print(df.info())Output:
(7, 5)
<class 'pandas.core.frame.DataFrame'>
Data columns (total 5 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Name 6 non-null object
1 Age 7 non-null int64
2 Salary 7 non-null int64
3 Start_Date 7 non-null object
4 Notes 6 non-null objectShape tells you rows and columns. .info() shows data types and missing value counts. This is the first defense against reading the wrong data.
Specifying Data Types
By default, pandas guesses column types. Force specific types with the dtype parameter:
df = pd.read_csv("sample.csv", dtype={'Age': 'int64', 'Salary': 'float64'})
print(df.dtypes)Output:
Name object
Age int64
Salary float64
Start_Date object
Notes object
dtype: objectThis is faster than letting pandas infer and safer than fixing types after the fact.
Parsing Date Columns
Dates often read as strings. Use parse_dates to convert them:
df = pd.read_csv("sample.csv", parse_dates=['Start_Date'])
print(df.dtypes)Output:
Name object
Age int64
Salary int64
Start_Date datetime64[ns]
Notes object
dtype: objectPass a list of column names or indices. The date parser handles common formats automatically.
Selecting Specific Columns
Don’t load columns you won’t use. The usecols parameter loads only what you need:
df = pd.read_csv("sample.csv", usecols=['Name', 'Salary', 'Start_Date'])
print(df)Output:
Name Salary Start_Date
0 Alice 75000 2020-01-15
1 Bob 85000 2019-03-20
2 Charlie 95000 2018-05-10
3 David 72000 2021-07-01
4 Eve 88000 2019-11-15
5 Frank 78000 2020-12-01
6 NaN 92000 2017-02-28This saves memory on large files and speeds up parsing.
Skipping and Limiting Rows
Skip rows at the start or limit the total rows read:
df = pd.read_csv("sample.csv", skiprows=1, nrows=3)
print(df)Output:
Alice 28 75000 2020-01-15 Active
0 Bob 35 85000 2019-03-20 Active
1 Charlie 42 95000 2018-05-10 On leave
2 David 29 72000 2021-07-01 Activeskiprows=1 skips the first data row (not the header). nrows=3 reads only 3 rows. Together they’re useful for testing on a large file before processing all of it.
Handling Different Separators
Not all data is comma-separated. Use the sep parameter for other delimiters:
# Semicolon-separated data
df = pd.read_csv("data.csv", sep=";")
# Tab-separated data
df = pd.read_csv("data.csv", sep="\t")The default is sep=",". Pipe and other delimiters work the same way.
Handling Malformed Lines
By default, read_csv() raises an error on malformed rows. Change behavior with on_bad_lines:
df = pd.read_csv("sample.csv", on_bad_lines='skip')
print(df)Output:
Name Age Salary Start_Date Notes
0 Alice 28 75000 2020-01-15 Active
1 Bob 35 85000 2019-03-20 Active
2 Charlie 42 95000 2018-05-10 On leave
3 David 29 72000 2021-07-01 Active
4 Eve 38 88000 2019-11-15 NaN
5 Frank 31 78000 2020-12-01 New hire
6 NaN 45 92000 2017-02-28 MalformedOptions are 'error' (default), 'warn' (skip and warn), or 'skip' (skip silently).
The index Round-Trip Gotcha
When you write a DataFrame to CSV with .to_csv(), it includes the index by default. Reading it back creates an unwanted Unnamed: 0 column:
# Original data
df = pd.read_csv("sample.csv", nrows=3)
# Write with index (default)
df.to_csv("with_index.csv")
# Read back without index_col
df_back = pd.read_csv("with_index.csv")
print(df_back.columns)
# Index(['Unnamed: 0', 'Name', 'Age', 'Salary', 'Start_Date', 'Notes'], ...)Fix it either way: write with index=False or read with index_col=0:
# Write without index
df.to_csv("without_index.csv", index=False)
# Read with index_col
df_back = pd.read_csv("with_index.csv", index_col=0)Common Parameters at a Glance
sep: Delimiter (default',')dtype: Data types for columnsparse_dates: Columns to parse as datesusecols: Subset of columns to loadskiprows: Rows to skip at the startnrows: Maximum rows to readna_values: Additional strings to treat as NaNon_bad_lines: How to handle malformed rows ('error','warn','skip')encoding: File encoding (default'utf-8')
For the full parameter list and examples, see the pandas read_csv() reference.
Next Steps
Once you’ve loaded and inspected your data with .shape and .info(), use pandas indexing and selection to extract what you need for analysis.
How often does a CSV file surprise you after you load it—wrong columns, wrong types, or extra rows?