Writing Memory-Efficient Python Code Using Generators & Itertools

Have you ever wanted to go through your data without going through the tedious statements?
Then you my friend are in luck with this article, welcome to your one time stop for writing memory efficient code!
In this tutorial we will learn about itertools and generators.

ITERTOOLS

What is Itertool now you may ask? Itertool, just as its name suggests is a module in Python used to maintain complex iterators.For example most of us are familiar with the count() function, if not here is a refresher:

>>> count(8,2) for example starts counting from 8 and up with step of 2 infinitely.

As you can see in this example the function does the job instead of a for loop. In that case you’d write 3 to 4 lines for. Thus providing us with their promised memory efficiency!

There are 3 different types of itertools:

Infinite Itertools

Our favorite count() function appears again as well new functions cycle() and repeat().

>>> cycle(‘xyz’) -> x y z x y z x y z x y z ……

>>> repeat(20) -> 20,20,20,20…..

Terminating

There are various terminating Itertools such as given in the table:

Combinatorics

As implied it consists of combination functions like product(), permutations() and combinations()

For further information on itertools of course make sure to visit Python’s official documentation.

GENERATORS

Generators are how we can create iterators in python. This is really helpful when dealing with large data efficiently.

A generator is a special type of iterator iterates a sequence without storing in memory. It works like normal functions. Instead of returning a single value, generators yield values one at a time and remember their state between calls.

They are also called lazy iterators, meaning they produce items only when requested. Thus making them memory-efficient and useful large datasets or infinite sequences.

1. Creating a Generator using yield
A generator remembers its state and resumes execution when next() is called.

def count_up_to(n):
    count = 1
    while count <= n:
        yield count  # Suspends execution and returns value
        count += 1  # Resumes from here on next call

gen = count_up_to(5)
print(list(gen))  # Output: [1, 2, 3, 4, 5]

2. Generator Expressions
You can create a generator in one line using () instead of [].

gen = (x**2 for x in range(5))  
print(list(gen))  # Output: [0, 1, 4, 9, 16]

I hope this tutorial was helpful! Generators and itertools are like our secret weapons for handling big data . Keep coding, keep optimizing, and let Python do all your heavy lifting!