What is the fastest way to drop consecutive duplicates a List[int] column?

Dropping consecutive duplicates involves removing repeated elements that appear next to each other in a list, keeping only the first occurrence of each sequence.

For example, given the list [1, 2, 2, 3, 3, 3, 4], the result should be [1, 2, 3, 4] because the duplicates immediately following the same value are dropped. Let's discuss various methods to do this in Python.

Using List Comprehension with zip()

This method uses list comprehension along with zip() compares each element with its previous one and includes it in the output only if it differs.

# Input list
a = [1, 2, 2, 3, 3, 3, 4]

# Remove consecutive duplicates
res = [a[0]] + [x for i, x in zip(a, a[1:]) if i != x]
print(res) 

[1, 2, 3, 4]

Explanation:

• We iterate over the list with zip(), pairing each element with the next one.
• If an element differs from the next, it is retained.
• This avoids explicitly iterating twice and is efficient for medium-sized lists.

Let's explore some more methods and see how we can drop consecutive duplicates a List[int] column.

Using itertools groupby

groupby()function groups consecutive identical elements and we take only one element from each group.

from itertools import groupby

# Input list
a = [1, 2, 2, 3, 3, 3, 4]

# Remove consecutive duplicates
res = [key for key, _ in groupby(a)]
print(res)  

[1, 2, 3, 4]

Explanation:

• groupby() function clusters identical elements that are adjacent.
• By taking the key of each group, we eliminate consecutive duplicates.
• This method is efficient and clean, especially for large datasets.

Using reduce from functools

reduce() function allows us to iteratively apply a logic across the elements of a list. In this method, we use reduce to construct a result list by comparing each element with the last one appended.

from functools import reduce

# Input list
a = [1, 2, 2, 3, 3, 3, 4]

# Remove consecutive duplicates
res = reduce(lambda acc, x: acc + [x] if not acc or acc[-1] != x else acc, a, [])
print(res) 

[1, 2, 3, 4]

Explanation:

• reduce() function accumulates elements in a list.
• If the current element is the same as the last one in the accumulator, it is skipped.

Using For Loop

A simple for loop can also achieve this by iterating through the list and appending elements only if they differ from the previous one.

a = [1, 2, 2, 3, 3, 3, 4]

# Initialize result list
res = []
for x in a:
    if not res or res[-1] != x:
        res.append(x)
print(res) 

[1, 2, 3, 4]

Explanation:

• The loop maintains a result list that stores non-duplicate elements.
• If the current element is the same as the last one in the result, it is skipped.
• This method is easy to understand and works well for small to medium-sized lists.