🙂

Shane Lynn
Data science, Startups, Analytics, and Data visualisation.

Blog Pandas Tutorials !

! About !
! Contact

Using iloc, loc, & ix to select rows and

columns in Pandas DataFrames Get some data updates!
96 Comments / blog, data science, Pandas, python, Tutorials / By Shane
Enter your email address to subscribe to

this blog and receive notifications of new

posts by email.

Email Address

Subscribe

Pandas Data Selection

There are multiple ways to select and index rows and columns from Pandas DataFrames. I find
Categories
tutorials online focusing on advanced selections of row and column choices a little complex for my
requirements. Select Category

Selection Options
There’s three main options to achieve the selection and indexing activities in Pandas, which can be
confusing. The three selection cases and methods covered in this post are: Pandas Tutorials

1. Selecting data by row numbers (.iloc) Pandas Groupby: Summarising,

2. Selecting data by label or by a conditional statement (.loc) Aggregating, and Grouping data in
3. Selecting in a hybrid approach (.ix) (now Deprecated in Pandas 0.20.1) Python

The Pandas DataFrame – loading,

Data Setup editing, and viewing data in Python

This blog post, inspired by other tutorials, describes selection activities with these operations. The Merge and Join DataFrames with Pandas
tutorial is suited for the general data science situation where, typically I find myself: in Python

Bar Plots in Python using Pandas

1. Each row in your data frame represents a data sample.
DataFrames
2. Each column is a variable, and is usually named. I rarely select columns without their
names. Plotting with Python and Pandas –

3. I need to quickly and often select relevant rows from the data frame for modelling and Libraries for Data Visualisation

visualisation activities. Python Pandas read_csv – Load Data

from CSV Files

For the uninitiated, the Pandas library for Python provides high-performance, easy-to-use data
Using iloc, loc, & ix to select rows and
structures and data analysis tools for handling tabular data in “series” and in “data frames”. It’s
brilliant at making your data processing easier and I’ve written before about grouping and columns in Pandas DataFrames

summarising data with Pandas. Pandas Drop: Delete DataFrame Rows &

Columns

Categories

blog

C++

Cycling

data science

Data Visualisation

Natural Language Processing

Pandas
Summary of iloc and loc methods discussed in this blog post. iloc and loc are operations for python
retrieving data from Pandas dataframes.
R

Selection and Indexing Methods for Pandas ROS

DataFrames Software

Talks
For these explorations we’ll need some sample data – I downloaded the uk-500 sample data set
Tutorials
from www.briandunning.com. This data contains artificial names, addresses, companies and
Uncategorized
phone numbers for fictitious UK characters. To follow along, you can download the .csv
file here. Load the data as follows (the diagrams here come from a Jupyter notebook in the web
Anaconda Python install):

1
2 import pandas as pd

3 import random

4
5 # read the data from the downloaded CSV file.

6 data = pd.read_csv('https://s3-eu-west-1.amazonaws.com/shanebucket/downloads/uk-500.csv')

7 # set a numeric id for use as an index for examples.

8 data['id'] = [random.randint(0,1000) for x in range(data.shape[0])]

9
10 data.head(5)

Pandas Index - Loading Data.py hosted with ❤ by GitHub view raw

Example data loaded from CSV file.

1. Selecting pandas data using “iloc”

The iloc indexer for Pandas Dataframe is used for integer-location based indexing / selection by
position.

The iloc indexer syntax is data.iloc[<row selection>, <column selection>], which is sure to be a
source of confusion for R users. “iloc” in pandas is used to select rows and columns by
number, in the order that they appear in the data frame. You can imagine that each row has a row
number from 0 to the total rows (data.shape[0]) and iloc[] allows selections based on these
numbers. The same applies for columns (ranging from 0 to data.shape[1] )

There are two “arguments” to iloc – a row selector, and a column selector. For example:

1 # Single selections using iloc and DataFrame

2 # Rows:

3 data.iloc[0] # first row of data frame (Aleshia Tomkiewicz) - Note a Series data type output.

4 data.iloc[1] # second row of data frame (Evan Zigomalas)

5 data.iloc[-1] # last row of data frame (Mi Richan)

6 # Columns:

7 data.iloc[:,0] # first column of data frame (first_name)

8 data.iloc[:,1] # second column of data frame (last_name)

9 data.iloc[:,-1] # last column of data frame (id)

Pandas Index - Single iloc selections.py hosted with ❤ by GitHub view raw

Multiple columns and rows can be selected together using the .iloc indexer.

1 # Multiple row and column selections using iloc and DataFrame

2 data.iloc[0:5] # first five rows of dataframe

3 data.iloc[:, 0:2] # first two columns of data frame with all rows

4 data.iloc[[0,3,6,24], [0,5,6]] # 1st, 4th, 7th, 25th row + 1st 6th 7th columns.

5 data.iloc[0:5, 5:8] # first 5 rows and 5th, 6th, 7th columns of data frame (county -> phone1).

Pandas Index - Multi iloc selections.py hosted with ❤ by GitHub view raw

There’s two gotchas to remember when using iloc in this manner:

1. Note that .iloc returns a Pandas Series when one row is selected, and a Pandas DataFrame
when multiple rows are selected, or if any column in full is selected. To counter this, pass a
single-valued list if you require DataFrame output.

When using .loc, or .iloc, you can control the output format by passing lists or single values
to the selectors.

2. When selecting multiple columns or multiple rows in this manner, remember that in your
selection e.g.[1:5], the rows/columns selected will run from the first number to one minus
the second number. e.g. [1:5] will go 1,2,3,4., [x,y] goes from x to y-1.

In practice, I rarely use the iloc indexer, unless I want the first ( .iloc[0] ) or the last ( .iloc[-1] )
row of the data frame.

2. Selecting pandas data using “loc”

The Pandas loc indexer can be used with DataFrames for two different use cases:

a.) Selecting rows by label/index

b.) Selecting rows with a boolean / conditional lookup

The loc indexer is used with the same syntax as iloc: data.loc[<row selection>, <column
selection>] .

2a. Label-based / Index-based indexing using .loc

Selections using the loc method are based on the index of the data frame (if any). Where the index
is set on a DataFrame, using <code>df.set_index()</code>, the .loc method directly selects based
on index values of any rows. For example, setting the index of our test data frame to the persons
“last_name”:

1 data.set_index("last_name", inplace=True)
2 data.head()

Pandas Index - Setting index for iloc.py hosted with ❤ by GitHub view raw

Last Name set as Index set on sample data frame

Now with the index set, we can directly select rows for different “last_name” values using
.loc[<label>] – either singly, or in multiples. For example:

Selecting single or multiple rows using .loc index selections with pandas. Note that the first
example returns a series, and the second returns a DataFrame. You can achieve a single-column
DataFrame by passing a single-element list to the .loc operation.

Select columns with .loc using the names of the columns. In most of my data work, typically I have
named columns, and use these named selections.

When using the .loc indexer, columns are referred to by names using lists of strings, or “:” slices.

You can select ranges of index labels – the selection </code>data.loc[‘Bruch’:’Julio’]</code> will
return all rows in the data frame between the index entries for “Bruch” and “Julio”. The following
examples should now make sense:

1
2 # Select rows with index values 'Andrade' and 'Veness', with all columns between 'city' and 'email'

3 data.loc[['Andrade', 'Veness'], 'city':'email']

4 # Select same rows, with just 'first_name', 'address' and 'city' columns

5 data.loc['Andrade':'Veness', ['first_name', 'address', 'city']]

6
7 # Change the index to be based on the 'id' column

8 data.set_index('id', inplace=True)

9 # select the row with 'id' = 487

10 data.loc[487]

Pandas Index - Select rows with loc.py hosted with ❤ by GitHub view raw

Note that in the last example, data.loc[487] (the row with index value 487) is not equal to
data.iloc[487] (the 487th row in the data). The index of the DataFrame can be out of numeric
order, and/or a string or multi-value.

2b. Boolean / Logical indexing using .loc

Conditional selections with boolean arrays using data.loc[<selection>] is the most common
method that I use with Pandas DataFrames. With boolean indexing or logical selection, you pass
an array or Series of True/False values to the .loc indexer to select the rows where your Series has
True values.

In most use cases, you will make selections based on the values of different columns in your data
set.

For example, the statement data[‘first_name’] == ‘Antonio’] produces a Pandas Series with a
True/False value for every row in the ‘data’ DataFrame, where there are “True” values for the rows
where the first_name is “Antonio”. These type of boolean arrays can be passed directly to the .loc
indexer as so:

Using a boolean True/False series to select rows in a pandas data frame – all rows with first name
of “Antonio” are selected.

As before, a second argument can be passed to .loc to select particular columns out of the data
frame. Again, columns are referred to by name for the loc indexer and can be a single string, a list
of columns, or a slice “:” operation.

Selecting multiple columns with loc can be achieved by passing column names to the second
argument of .loc[]

Note that when selecting columns, if one column only is selected, the .loc operator returns a Series.
For a single column DataFrame, use a one-element list to keep the DataFrame format, for
example:

If selections of a single column are made as a string, a series is returned from .loc. Pass a list to get
a DataFrame back.

Make sure you understand the following additional examples of .loc selections for clarity:

1
2 # Select rows with first name Antonio, # and all columns between 'city' and 'email'

3 data.loc[data['first_name'] == 'Antonio', 'city':'email']

4
5 # Select rows where the email column ends with 'hotmail.com', include all columns

6 data.loc[data['email'].str.endswith("hotmail.com")]

7
8 # Select rows with last_name equal to some values, all columns

9 data.loc[data['first_name'].isin(['France', 'Tyisha', 'Eric'])]

10
11 # Select rows with first name Antonio AND hotmail email addresses

12 data.loc[data['email'].str.endswith("gmail.com") & (data['first_name'] == 'Antonio')]

13
14 # select rows with id column between 100 and 200, and just return 'postal' and 'web' columns

15 data.loc[(data['id'] > 100) & (data['id'] <= 200), ['postal', 'web']]

16
17 # A lambda function that yields True/False values can also be used.

18 # Select rows where the company name has 4 words in it.

19 data.loc[data['company_name'].apply(lambda x: len(x.split(' ')) == 4)]

20
21 # Selections can be achieved outside of the main .loc for clarity:

22 # Form a separate variable with your selections:

23 idx = data['company_name'].apply(lambda x: len(x.split(' ')) == 4)

24 # Select only the True values in 'idx' and only the 3 columns specified:

25 data.loc[idx, ['email', 'first_name', 'company']]

Pandas index - loc selection examples.py hosted with ❤ by GitHub view raw

Logical selections and boolean Series can also be passed to the generic [] indexer of a pandas
DataFrame and will give the same results: data.loc[data[‘id’] == 9] == data[data[‘id’] == 9] .

3. Selecting pandas data using ix

Note: The ix indexer has been deprecated in recent versions of Pandas,

starting with version 0.20.1.

The ix[] indexer is a hybrid of .loc and .iloc. Generally, ix is label based and acts just as the .loc
indexer. However, .ix also supports integer type selections (as in .iloc) where passed an integer.
This only works where the index of the DataFrame is not integer based. ix will accept any of the
inputs of .loc and .iloc.

Slightly more complex, I prefer to explicitly use .iloc and .loc to avoid unexpected results.

As an example:

1
2 # ix indexing works just the same as .loc when passed strings

3 data.ix[['Andrade']] == data.loc[['Andrade']]

4 # ix indexing works the same as .iloc when passed integers.

5 data.ix[[33]] == data.iloc[[33]]

6
7 # ix only works in both modes when the index of the DataFrame is NOT an integer itself.

Pandas index - ix selections.py hosted with ❤ by GitHub view raw

Setting values in DataFrames using .loc

With a slight change of syntax, you can actually update your DataFrame in the same statement as
you select and filter using .loc indexer. This particular pattern allows you to update values in
columns depending on different conditions. The setting operation does not make a copy of the
data frame, but edits the original data.

As an example:

1 # Change the first name of all rows with an ID greater than 2000 to "John"
2 data.loc[data['id'] > 2000, "first_name"] = "John"

3
4 # Change the first name of all rows with an ID greater than 2000 to "John"

5 data.loc[data['id'] > 2000, "first_name"] = "John"

Pandas index - changing data with loc.py hosted with ❤ by GitHub view raw

That’s the basics of indexing and selecting with Pandas. If you’re looking for more, take a look at
the .iat, and .at operations for some more performance-enhanced value accessors in the Pandas
Documentation and take a look at selecting by callable functions for more iloc and loc fun.

← Previous Post Next Post →

! Subscribe !

Join the discussion

{} [+] #

96 COMMENTS " #

Implementare l’algoritmo KNN in Python e Scikit-learn | Lorenzo Govoni

" 1 year ago

[…] maggiori informazioni, si veda il seguente articolo (solo in […]

0 Reply

mariana
" 1 year ago

Really helpful Shane for beginners. Very through and detailed. Looking for more of your blogs on pandas and
python.

8 Reply

Yahor
" 1 year ago

Very helpful content, Shane. Helped me clear my understanding of working with row selections.

2 Reply

Bowen
" 1 year ago

Thank you so much! This is very helpful and illustrative

1 Reply

Maria
" 1 year ago

Very precise and clear. Easy to understand. Thanks for the content

1 Reply

Amol Wadpalle
" 1 year ago

Very detailed explanation! thanks!

when following your examples, i was expecting to get a type = dataframe for the below query: however its
throwing an error
print(df.iloc[[1:4, 2:4]])

3 Reply

Dihao Qi
" 1 year ago

excellent explanations. really helpful

2 Reply

Dung
" 1 year ago

Thank you so much!. Very detailed and helpful

2 Reply

Hari Natarajan
" 1 year ago

Finally, I have a clear picture. Your instructions are precise and self-explanatory. I wish you publish a detailed
book on Python Programming so that it will be of immense help for learners and programmers.

2 Reply

Data Preprocessing with Python | BeingDatum

" 1 year ago

[…] You can read more about the usage of iloc here. […]

1 Reply

srinivas reddy pachika

" 1 year ago

Excellent post. Thank you so much for coming with such awesome content

1 Reply

Marilu
" 11 months ago

Thank you so much, it helped me a lot to understand pandas selection, great article for beginners like me

1 Reply

Elaheh Arjomand
" 7 months ago

Thanks for the content/

0 Reply

Chuck
" 6 months ago

Great job – even greater examples

0 Reply

khoa
" 6 months ago

this is so concise and fully side of selecting element in pandas. Thank you, writer!

0 Reply

Sujay Bhujbal
" 4 months ago

Thank you for the explanation

0 Reply

nick
" 4 months ago

Fantastic explanation. Thanks, Shane!

0 Reply

MARCELLO DISTASIO
" 1 month ago

Exactly what I needed,n this is extremelyhelpful -thank you.

0 Reply

Aurelien
" 26 days ago

Hello!
Thank you very much for this nice article.
I try to use a dataset with scikit-learn M/L algorithm. I have approximatly 4000 samples (Sn), but my dataset is
in this format : (first image, multiple lines for one output); I would like to move it in this format (second image),
to have each sample on 1 raw.

loc and iloc can helps me in moving every 5 raw for column 1 in a single raw please?

Thank you for your help and advises.

0 Reply

Aurelien
" 26 days ago

Hello!
Thank you very much for this nice article.
I try to use a dataset with scikit-learn M/L algorithm. I have approximatly 4000 samples (Sn), but my dataset is
in this format : (multiple lines of input for one output); I would like to move it in this format (second image), to
have each sample on 1 raw.

loc and iloc can helps me in moving every 5 raw for column 1 in a single raw please?

Thank you for your help and advises.

0 Reply

« Previous 1 2 3

Copyright © 2021 Shane Lynn | Powered by Astra WordPress Theme