Skip to main content
intermediate

Side Input Patterns

5 min read

1. Introduction

Side Inputs are auxiliary PCollections (e.g., currency exchange rates, country code lookups, or user blacklist maps) passed alongside primary elements into a ParDo or FlatMap transform for dynamic data enrichment.

2. Why This Concept Exists

In distributed data engineering, joining a massive multi-terabyte stream with a small 5 MB lookup table using full shuffle joins (like CoGroupByKey) is extremely inefficient. Full shuffle joins force the runner to hash, serialize, and network-transfer billions of records across nodes.

Side inputs eliminate this network shuffle bottleneck by broadcasting the small lookup dataset to worker nodes, where it is cached in local memory. Each worker processes primary stream elements locally against the in-memory lookup table with zero network hops.

3. Key Terminology

  • Primary Input: The main PCollection of elements flowing through the pipeline pipeline stages.
  • Auxiliary Side Input: A secondary PCollection converted into a view (AsSingleton, AsDict, AsList, or AsIter) and passed to worker execution threads.
  • Broadcast: The mechanism where the runner serializes a small dataset once and distributes a replica to all active worker VMs.
  • Slowly Changing Dimension (SCD): Reference data that updates infrequently (e.g., hourly exchange rates) and must be synchronized with streaming windows.

4. Visual Architecture

Comparison between a heavy Shuffle Join and an in-memory Side Input Broadcast:

text
┌────────────────────────────────────────────────────────┐
│             Side Input Broadcast Pattern               │
│                                                        │
│  [Small Lookup Table: 2 MB] ──(Broadcast Replicas)───┐ │
│                                                      ▼ │
│  [Primary Stream: 10 GB/s] ──► [Worker 1: Local Cache] │
│                           ──► [Worker 2: Local Cache] │
│                           ──► [Worker 3: Local Cache] │
│                                                        │
│  * Zero shuffle network hops for the 10 GB/s stream    │
└────────────────────────────────────────────────────────┘

5. Code Example: Windowed Lookup Table Enrichment

Passing an in-memory lookup table as a dictionary side input to enrich streaming click events:

python
import apache_beam as beam
from apache_beam.pvalue import AsDict

# Primary stream of customer orders: (customer_id, order_value_usd)
orders = [
    ("cust_101", 150.0),
    ("cust_102", 45.0),
    ("cust_103", 890.0),
    ("cust_999", 12.0)  # Unknown customer
]

# Lookup reference data: (customer_id, country_code)
customer_metadata = [
    ("cust_101", "US"),
    ("cust_102", "DE"),
    ("cust_103", "UK")
]

class EnrichOrderDoFn(beam.DoFn):
    def process(self, element, country_lookup):
        cust_id, amount = element
        # Perform O(1) in-memory lookup against side input dictionary
        country = country_lookup.get(cust_id, "UNKNOWN")
        yield {
            "customer_id": cust_id,
            "amount_usd": amount,
            "region": country,
            "tax_rate": 0.19 if country == "DE" else 0.08
        }

with beam.Pipeline() as p:
    # 1. Create the side input view
    country_view = (p
                    | "CreateMetadata" >> beam.Create(customer_metadata)
                    | "ViewAsDict" >> beam.CombineGlobally(dict))

    # 2. Process primary orders using the side input
    (p
     | "CreateOrders" >> beam.Create(orders)
     | "EnrichWithSideInput" >> beam.ParDo(EnrichOrderDoFn(), country_lookup=AsDict(country_view))
     | "PrintEnriched" >> beam.Map(print))

6. Production Best Practices & Hazards

  • Memory Footprint Limits: Side inputs are materialized into the memory (RAM) of each worker VM. Keep lookup tables under 100 MB. Exceeding worker memory will cause immediate Out-Of-Memory (OOM / Exit Code 137) worker crashes.
  • Prefer AsIter for Large Sequences: If the side collection is too large to fit in Python dictionary memory at once, use beam.pvalue.AsIter(pcoll) to iterate through records on-demand without full materialization.
  • Window Matching in Streaming: In streaming pipelines, the side input window must cover the primary input window. If an incoming element's window has no corresponding side input window, the worker will block waiting for watermark progression.

Related Apache Beam Topics & Lessons