DoFn
Interactive Syntax Reference
DoFnApache Beam Cheatsheet: DoFn
Recommended reading: 4 minsCopy-paste ready Python code snippets
Core Architecture Overview
Define custom element processing logic using the standard DoFn lifecycle.
DoFn Lifecycle Hooks
returns: Generator / YieldPurpose & Description
Manages class lifecycles (setup, start_bundle, process, finish_bundle, teardown) on worker nodes.
Syntax Signature
class MyDoFn(beam.DoFn):Executable Example
import apache_beam as beam
class DBConnectionFn(beam.DoFn):
def setup(self):
# Runs once per class instance initiation (worker start)
self.db_client = "Connected"
def start_bundle(self):
# Runs once per bundle processing window
self.buffer = []
def process(self, element):
# Runs for every element in the collection
yield f"{element}_processed_by_db"
def teardown(self):
# Runs once when worker container shuts down
self.db_client = "Closed"Used In
Optimizing database client pools, metrics tracking, and batch updates.
Common Pitfall
Instantiating heavy API clients or database pools inside process() instead of setup().
Pro Tip
Group elements into batches inside start_bundle / finish_bundle to minimize network request overhead.
More Free Data Engineering Cheatsheets (DataPlayArena Network)Interactive syntax references