1. Introduction
Side Inputs are auxiliary PCollections (e.g., currency exchange rates, country code lookups, or user blacklist maps) passed alongside primary elements into a
ParDo or FlatMap transform for dynamic data enrichment.2. Why This Concept Exists
In distributed data engineering, joining a massive multi-terabyte stream with a small 5 MB lookup table using full shuffle joins (like
CoGroupByKey) is extremely inefficient. Full shuffle joins force the runner to hash, serialize, and network-transfer billions of records across nodes.Side inputs eliminate this network shuffle bottleneck by broadcasting the small lookup dataset to worker nodes, where it is cached in local memory. Each worker processes primary stream elements locally against the in-memory lookup table with zero network hops.
3. Key Terminology
- Primary Input: The main PCollection of elements flowing through the pipeline pipeline stages.
- Auxiliary Side Input: A secondary PCollection converted into a view (
AsSingleton,AsDict,AsList, orAsIter) and passed to worker execution threads. - Broadcast: The mechanism where the runner serializes a small dataset once and distributes a replica to all active worker VMs.
- Slowly Changing Dimension (SCD): Reference data that updates infrequently (e.g., hourly exchange rates) and must be synchronized with streaming windows.
4. Visual Architecture
Comparison between a heavy Shuffle Join and an in-memory Side Input Broadcast:
text┌────────────────────────────────────────────────────────┐ │ Side Input Broadcast Pattern │ │ │ │ [Small Lookup Table: 2 MB] ──(Broadcast Replicas)───┐ │ │ ▼ │ │ [Primary Stream: 10 GB/s] ──► [Worker 1: Local Cache] │ │ ──► [Worker 2: Local Cache] │ │ ──► [Worker 3: Local Cache] │ │ │ │ * Zero shuffle network hops for the 10 GB/s stream │ └────────────────────────────────────────────────────────┘
5. Code Example: Windowed Lookup Table Enrichment
Passing an in-memory lookup table as a dictionary side input to enrich streaming click events:
pythonimport apache_beam as beam from apache_beam.pvalue import AsDict # Primary stream of customer orders: (customer_id, order_value_usd) orders = [ ("cust_101", 150.0), ("cust_102", 45.0), ("cust_103", 890.0), ("cust_999", 12.0) # Unknown customer ] # Lookup reference data: (customer_id, country_code) customer_metadata = [ ("cust_101", "US"), ("cust_102", "DE"), ("cust_103", "UK") ] class EnrichOrderDoFn(beam.DoFn): def process(self, element, country_lookup): cust_id, amount = element # Perform O(1) in-memory lookup against side input dictionary country = country_lookup.get(cust_id, "UNKNOWN") yield { "customer_id": cust_id, "amount_usd": amount, "region": country, "tax_rate": 0.19 if country == "DE" else 0.08 } with beam.Pipeline() as p: # 1. Create the side input view country_view = (p | "CreateMetadata" >> beam.Create(customer_metadata) | "ViewAsDict" >> beam.CombineGlobally(dict)) # 2. Process primary orders using the side input (p | "CreateOrders" >> beam.Create(orders) | "EnrichWithSideInput" >> beam.ParDo(EnrichOrderDoFn(), country_lookup=AsDict(country_view)) | "PrintEnriched" >> beam.Map(print))
6. Production Best Practices & Hazards
- Memory Footprint Limits: Side inputs are materialized into the memory (RAM) of each worker VM. Keep lookup tables under 100 MB. Exceeding worker memory will cause immediate Out-Of-Memory (
OOM / Exit Code 137) worker crashes. - Prefer
AsIterfor Large Sequences: If the side collection is too large to fit in Python dictionary memory at once, usebeam.pvalue.AsIter(pcoll)to iterate through records on-demand without full materialization. - Window Matching in Streaming: In streaming pipelines, the side input window must cover the primary input window. If an incoming element's window has no corresponding side input window, the worker will block waiting for watermark progression.
7. Related Lessons
Related Apache Beam Topics & Lessons
Apache Beam IntroductionLearn the core concepts of unified batch and streaming data processing.
Beam Pipeline BasicsConstruct and execute your first Apache Beam data processing pipeline.
PCollection Data AbstractionMaster distributed data collections in Apache Beam.
ParDo & DoFn TransformationsApply custom element-wise transformations with ParDo and DoFn.
People Also Search For