Skip to main content
intermediate

Combine and Aggregation

7 min read

1. Introduction

The Combine transform aggregates elements in a PCollection. You can combine elements globally across the entire collection or group and combine them per key.

2. Why This Concept Exists

While GroupByKey followed by sum gets the job done, it is highly inefficient for large datasets. It requires transferring every single record across the network (shuffling). Combine solves this by performing local pre-aggregation on each worker node before sending the partial results over the network (an optimization known as Combiner Lift).

3. Key Terminology

  • Combiner Lift: The optimization where workers aggregate data locally first, reducing network shuffle volume.
  • Associative & Commutative: A mathematical function where grouping and ordering do not affect the result (e.g. addition: (A + B) + C = A + (B + C)). Combine functions must have these properties.

4. How It Works

  • Local Aggregate: Individual worker machines group their local records and aggregate them.
  • Shuffle: Partial sums are sent to combining nodes.
  • Final Aggregate: The runner merges partial sums to output final results.

5. Visual Diagram

Comparing GroupByKey shuffle versus Combine's local pre-aggregation optimization:

text
GroupByKey:  [1, 2, 3] ──(Shuffle all)──> Network ──(Sum)──> [ 6 ]
Combine:     [1, 2, 3] ──(Local Sum)───> [ 6 ] ──(Shuffle)──> [ 6 ]

6. Code Example

Calculating sum aggregates globally and per key:

python
import apache_beam as beam

with beam.Pipeline() as p:
    scores = p | "CreateScores" >> beam.Create([("A", 10), ("B", 20), ("A", 30)])
    # 1. Combine Per Key
    totals_per_key = scores | "SumPerKey" >> beam.CombinePerKey(sum)

7. Code Explanation

  • scores contains key-value tuples: ("A", 10), ("B", 20), ("A", 30).
  • CombinePerKey(sum) sums the values associated with each key.
  • The worker processes "A" locally (10 + 30 = 40) and yields: [("A", 40), ("B", 20)].

8. Real Production Example: Custom CombineFn

For non-trivial aggregations (like computing averages), you define a custom class subclassing beam.CombineFn:
python
class AverageFn(beam.CombineFn):
    def create_accumulator(self):
        return (0.0, 0) # (sum, count)

    def add_input(self, accumulator, element):
        (sum_val, count) = accumulator
        return sum_val + element, count + 1

    def merge_accumulators(self, accumulators):
        sums, counts = zip(*accumulators)
        return sum(sums), sum(counts)

    def extract_output(self, accumulator):
        (sum_val, count) = accumulator
        return sum_val / count if count > 0 else float("nan")

9. Common Mistakes

  • Applying non-associative operations: Trying to use operations like subtraction or list division inside Combine will yield incorrect values. The operation must be commutative and associative.
  • Assuming ordering guarantees: Because local partial states are merged concurrently, elements are combined in arbitrary order.

10. Best Practices

  • Prefer CombinePerKey over GroupByKey + Map(sum) to leverage Combiner Lift performance.
  • Pin accumulator objects as lightweight tuples or dataclasses to minimize serialization overhead.

11. Summary

  • Combine optimizes aggregations using local pre-combining.
  • Requires associative and commutative functions.
  • Implements create_accumulator, add_input, merge_accumulators, and extract_output.

12. Interactive Challenges

Challenge 1: Global Average Calculator (Beginner)

Write a pipeline segment that takes a PCollection of float integers and uses the built-in Average combiner to compute the mean value globally.

Challenge 2: Max Score Per Student (Intermediate)
Write a pipeline segment that takes a PCollection of student grade tuples (student_name, grade) and uses CombinePerKey to find the maximum grade achieved by each student.
Challenge 3: Custom Ratio Accumulator (Advanced)
Define a custom CombineFn class named ClickThroughRateFn that calculates the ratio of clicks to impressions: clicks / impressions given elements in (clicks, impressions) tuple format (e.g. (1, 10)).

Related Apache Beam Topics & Lessons