Suppose a pipeline has seven records and groups them in threes. Ordinary batching returns two full batches and a final batch containing one record. That last batch may be acceptable, or it may show that the input is incomplete.
Python's itertools.batched offers a strict=True option for the second case. The name is precise, but easy to overread: it checks the shape of each batch; it does not make the work performed for earlier batches atomic.
I ran a small synthetic experiment on Python 3.14.7. The input was row-1 through row-7, with a batch size of three. Without strict mode, the result was:
['row-1', 'row-2', 'row-3']
['row-4', 'row-5', 'row-6']
['row-7']
With strict=True, the iterator yielded the first two complete batches and then raised ValueError: batched(): incomplete batch when it reached the final record. The six records in those complete batches had already been handed to the loop body.
That distinction matters if the loop does something irreversible after each yield: sends a message, writes a file, or commits a group of rows. The later exception reports a malformed tail, but it does not retract the earlier effects. In the experiment, a synthetic per-batch write list contained six records when the error arrived.
There are three different policies hiding behind the word “batch.” Accept a short final batch and process it. Reject an incomplete input with strict=True, while accepting that earlier work may already have happened. Or validate and stage the complete input before performing external effects, using a transaction or a deliberate compensation strategy where the operation must be all-or-nothing.
The experiment is deliberately small and uses no product, database, or network. It does not establish the atomicity of any particular consumer. It establishes a narrower boundary visible in this interpreter: strict batching detected the incomplete tail, but the detection point came after two complete batches had been yielded.
strict is a useful input check. It is not a rollback mechanism.
No comments:
Post a Comment
Comments are allowed as long as they touch the post in question and they do no contain any spam or crap.