Remove Unordered Duplicate Elements from a List - Python

Given a list of elements, the task is to remove all duplicate elements from the list while maintaining the original order of the elements.
For example, if the input is [1, 2, 2, 3, 1] the expected output is [1, 2, 3]. Let's explore various methods to achieve this in Python.

Using set

We can initialize an empty set to track the elements we've seen so far and create a new list to store the unique elements in their original order.

# Initializing list of elements
a = [1, 2, 2, 3, 1]

# Initializing an empty set to track seen elements
s = set()

# Initializing an empty list to store unique elements
b = []

# Looping through the list
for x in a:
    # If element is not in the set, add it to unique list and the set
    if x not in s:
        b.append(x)
        s.add(x)

print(b)

[1, 2, 3]

Explanation:

• We initialize an empty set 's' to keep track of elements that have already been encountered.
• We loop through the list and only append elements to the unique list 'b' if they are not in the seen set 's'.

Let's explore some more methods and see how we can remove unordered duplicate elements from a list.

Using list comprehension with set

This method uses list comprehension along with a set for a more compact and precise solution.

# Initializing list of elements
a = [1, 2, 2, 3, 1]

# Initializing an empty set to track seen elements
s = set()

# Using list comprehension to filter out duplicates
b = [x for x in a if not (x in s or s.add(x))]

print(b)

[1, 2, 3]

Explanation:

• We initialize the seen set 's' and use list comprehension to filter out duplicates.
• The expression x in s or s.add(x) ensures that elements are only added to the result if they haven't been seen before.

Using dict.fromkeys()

We can use dict.fromkeys() to remove duplicates while maintaining the order of elements.

# Initializing list of elements
a = [1, 2, 2, 3, 1]

# Using dict.fromkeys() to remove duplicates
b = list(dict.fromkeys(a))

print(b)

[1, 2, 3]

Explanation:

• The dict.fromkeys() method creates a dictionary where the keys are the list elements. Since dictionaries do not allow duplicate keys, it automatically removes duplicates.
• We convert the dictionary back to a list to obtain the result.

Using collections.OrderedDict

We can use OrderedDict from the collections module to remove duplicates and maintain the order of elements.

from collections import OrderedDict

# Initializing list of elements
a = [1, 2, 2, 3, 1]

# Using OrderedDict to remove duplicates while maintaining order
b = list(OrderedDict.fromkeys(a))

print(b)

[1, 2, 3]

Explanation:

• The OrderedDict.fromkeys() method removes duplicates while keeping the order of elements intact.
• It works similarly to dict.fromkeys() but preserves the order of insertion.
• This method is efficient and guarantees that the order is maintained.

Using for loop

This method uses a basic for loop to manually create a new list while checking if an element has already been added.

# Initializing list of elements
a = [1, 2, 2, 3, 1]

# Initializing an empty list to store unique elements
b = []

# Looping through the list
for x in a:
    # Adding element to unique list only if it's not already in it
    if x not in b:
        b.append(x)

print(b)

[1, 2, 3]

Explanation

• We initialize an empty list 'b' and loop through the original list.
• For each element, we check if it is already in the unique list 'b'. If not, we append it.