Memory Efficient Python List Creation in Case of Generators

Creating lists in Python can sometimes be memory intensive, especially when dealing with large amounts of data. In such cases, generators offer a solution to reduce memory usage.

In this article, we will look at How we can create a list by using generators.

What are Generators?

Generators in Python are iterable, allowing the creation of iterators in a memory-efficient manner. Unlike lists that store all elements in memory at once, generators produce values on the fly as requested. They are defined using a function with the "the " keyword, which temporarily suspends the function's state, allowing it to resume from where it left off when the next value is requested.

Using List Comprehension

List comprehension is a common and simple method for creating lists in Python. However, it can be memory-heavy when dealing with large data sets because it creates the entire list in memory at once. To optimize memory usage, we will look at using a generator instead.

a = [i * 2 for i in range(10)]  
print(a)

[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

Using Generator Expression

A generator expression is similar to list comprehension but more memory efficient. Instead of generating all the items at once, a generator produces one item at a time. This reduces the memory footprint because it doesn’t store the entire list in memory.

a = (i * 2 for i in range(10))  

print(list(a))

[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

• In this case, a is not a list but a generator. We can use it just like a list but it won’t occupy all the memory. The generator will produce values only when we need them, making it very memory-efficient.

Using map()

map() is another useful tool that works like a generator when combined with functions. It applies a given function to all items in an iterable without storing the entire result at once.

def multiply_by_two(x):
    return x * 2

a = map(multiply_by_two, range(10))  

print(list(a))

[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

Using itertools

For more complex use cases, we can use the itertools module, which provides many tools to handle iterators efficiently. For example, itertools.count() can be used to generate an infinite sequence of numbers, and itertools.islice() allows us to grab specific slices of data from an iterator without loading everything into memory.

import itertools

a = itertools.islice(itertools.count(), 10)  

print(list(a))

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]