Unique Dictionary Filter in List - Python

We are given a dictionary in list we need to find unique dictionary. For example, a = [ {"a": 1, "b": 2}, {"a": 1, "b": 2}, {"c": 3}, {"a": 1, "b": 3}] so that output should be [{'a': 1, 'b': 2}, {'a': 1, 'b': 3}, {'c': 3}].

Using set with frozenset

Using set with frozenset, we convert dictionary items to immutable sets allowing them to be added to a set for uniqueness filtering. This method ensures only unique dictionaries are retained in list.

a = [
    {"a": 1, "b": 2},
    {"a": 1, "b": 2},
    {"c": 3},
    {"a": 1, "b": 3}
]

# Convert each dictionary to a frozenset of items, ensuring hashable representation
u = [dict(d) for d in {frozenset(d.items()) for d in a}]
print(u)

[{'c': 3}, {'b': 3, 'a': 1}, {'a': 1, 'b': 2}]

Explanation:

• Each dictionary is converted to a frozenset of its key-value pairs, creating a hashable representation that allows duplicates to be removed using a set.
• Unique frozenset items are converted back to dictionaries and result is stored in u ensuring list contains only unique dictionaries.

Using json.dumps

json.dumps converts dictionaries to JSON strings providing a hashable and comparable format. By storing these strings in a set duplicates can be filtered and unique strings are then converted back to dictionaries.

import json

a = [
    {"a": 1, "b": 2},
    {"a": 1, "b": 2},
    {"c": 3},
    {"a": 1, "b": 3}
]
seen = set()
u = []

for d in a:
    
    # Convert dictionary to a JSON string with sorted keys
    di = json.dumps(d, sort_keys=True)
    
    # Add dictionary to result if not seen before
    if di not in seen:
        u.append(d)
        seen.add(di)
print(u)

[{'a': 1, 'b': 2}, {'c': 3}, {'a': 1, 'b': 3}]

Explanation:

• Each dictionary is converted to a JSON string with sorted keys using json.dumps ensuring consistent ordering for comparison.
• A set seen tracks JSON strings to filter out duplicates and unique dictionaries are appended to list u which is then printed.

Using all()

all() checks if all conditions in an iterable are True. It can be used to compare each dictionary with those already added to a result list ensuring no duplicates before appending.

a = [
    {"a": 1, "b": 2},
    {"a": 1, "b": 2},
    {"c": 3},
    {"a": 1, "b": 3}
]
u = []

for d in a:
    
    # Add dictionary if it is not equal to any existing dictionary in the list
    if all(d != existing for existing in u):
        u.append(d)

print(u)

[{'a': 1, 'b': 2}, {'c': 3}, {'a': 1, 'b': 3}]

Explanation:

• For each dictionary d in the list all() ensures it is not equal to any dictionary already in the result list u.
• If d is unique it is appended to u resulting in a list of unique dictionaries which is then printed.