Iterators & Generators

Previous Next

Every time you write for item in some_list:, Python is quietly running the iterator protocol behind the scenes. Understanding that protocol explains why some objects can be looped over and others can't, and it opens the door to generators — one of the most memory-efficient tools in the language.

Iterables vs. Iterators

An object is iterable if you can loop over it with a for statement — lists, tuples, strings, dictionaries and sets all qualify. Under the hood, an iterable object implements a special method called __iter__() that returns an iterator. The iterator itself implements __next__(), which produces the next value each time it is called, and raises StopIteration once there is nothing left.

You don't normally need to write these methods yourself; built-in types already provide them. But knowing they exist explains exactly what a for loop is doing.

class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        self.current -= 1
        return self.current + 1

for number in Countdown(3):
    print(number)  # 3, 2, 1

Calling iter() and next() Manually

You can call iter() and next() yourself on any iterable to see exactly what a for loop does internally: it first asks for an iterator with iter(), then repeatedly calls next() on it until a StopIteration exception is raised, at which point the loop stops.

numbers = [10, 20, 30]
it = iter(numbers)

print(next(it))  # 10
print(next(it))  # 20
print(next(it))  # 30
print(next(it))  # raises StopIteration

Note: A plain for item in numbers: catches StopIteration for you automatically, so you never see the exception unless you are driving the iterator manually.

Generator Functions with yield

Writing a full class with __iter__ and __next__ just to produce a sequence of values is verbose. A generator function does the same job with far less code: it looks like an ordinary function, except it uses yield instead of return. Every time yield runs, the function's state is paused and the value is handed back to the caller; the next call to next() resumes right where it left off.

def count_up_to(limit):
    count = 1
    while count <= limit:
        yield count
        count += 1

for number in count_up_to(5):
    print(number)  # 1, 2, 3, 4, 5

Calling count_up_to(5) does not run the function body at all — it immediately returns a generator object. The body only executes, one yield at a time, as the generator is iterated.

Why Generators Matter: Memory Efficiency

The biggest advantage of generators is that they produce values one at a time, on demand, instead of building the entire sequence in memory up front. Compare a list comprehension with a generator expression that do the same job:

# builds a list of 1,000,000 items in memory immediately
squares_list = [x * 2 for x in range(1000000)]

# produces one value at a time, using almost no extra memory
squares_gen = (x * 2 for x in range(1000000))

squares_list is a real list holding a million numbers the moment it is created. squares_gen is a lightweight generator object; the multiplications only happen as each value is actually requested, for example inside a for loop. If you only ever need to process the values once, in order, a generator is almost always the better choice for large or unbounded sequences.

Exhausting a Generator

A generator can only be iterated once. After it has produced its last value, calling next() again raises StopIteration, and the generator cannot be "rewound" — you would need to call the generator function again to get a fresh one.

def two_values():
    yield "first"
    yield "second"

gen = two_values()
print(next(gen))  # first
print(next(gen))  # second
print(next(gen))  # raises StopIteration, generator is exhausted

Next steps: Head to Decorators to learn how to wrap functions with extra behavior.

Previous Next