Skip to main content
intermediate

Dataflow Runner

6 min read

1. Introduction

The Dataflow Runner is the execution engine that runs Apache Beam pipelines on Google Cloud. It compiles your pipeline code into a managed service of virtual machine workers that execute the processing steps.

2. Why This Concept Exists

While the DirectRunner is excellent for testing pipelines locally, it is limited by the CPU and RAM of your local workstation. To process terabytes of data, you need an engine that can spin up hundreds of servers, distribute data chunks, handle worker crashes, and scale down when finished. The Dataflow Runner does all of this automatically as a fully managed service.

3. Key Terminology

  • Dataflow Service: The serverless Google Cloud service that manages and monitors pipeline execution.
  • Worker VM: The individual Google Compute Engine virtual machines that process records in parallel.
  • Dataflow Shuffle: The high-performance service that handles grouping data across machines without using local worker disks.

4. How It Works

  1. Code: You write your pipeline using the Apache Beam SDK.
  2. Submit: You execute your script passing --runner=DataflowRunner along with GCP project and region details.
  3. Compile: Beam validates the code and uploads the compiled execution graph (DAG) to Google Cloud.
  4. Provision: Dataflow spins up worker VMs and installs your dependencies.
  5. Execute & Scale: Workers pull data, scale up/down depending on load, and write output results.
  6. Teardown: Once finished, VM workers shut down automatically to save cost.

5. Visual Diagram

Submitting local code to Google Cloud for managed execution:

text
[Local Script] ──(Submit CLI)──> [GCP Dataflow API] ──(Autoscales)──> [Worker VMs (1 to N)] ──> [Sinks]

6. Code Example

Configuring a script to run on the Dataflow Runner using command-line arguments:

python
import sys
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions, GoogleCloudOptions

def run():
    # Parse options from command line arguments
    options = PipelineOptions(sys.argv)
    
    # Force GCP configurations
    gcp_options = options.view_as(GoogleCloudOptions)
    gcp_options.project = "my-billing-project"
    gcp_options.region = "us-central1"
    gcp_options.temp_location = "gs://my-bucket/temp"
    
    with beam.Pipeline(options=options) as p:
        (p
         | "Read" >> beam.io.ReadFromText("gs://my-bucket/input.txt")
         | "Uppercase" >> beam.Map(lambda x: x.upper())
         | "Write" >> beam.io.WriteToText("gs://my-bucket/output/clean"))

if __name__ == "__main__":
    run()

7. Code Explanation

  • PipelineOptions(sys.argv) parses CLI flags.
  • GoogleCloudOptions configures required cloud settings: billing project, compute region, and GCS temp_location staging path.
  • When executing the script using python main.py --runner=DataflowRunner, Beam deploys the DAG to GCP.

8. Real Production Example

Deploying pipelines from continuous integration scripts (CI/CD):

bash
python main.py \
  --runner=DataflowRunner \
  --project=prod-data-project \
  --region=us-east4 \
  --temp_location=gs://prod-staging-bucket/temp \
  --num_workers=5 \
  --max_num_workers=50 \
  --machine_type=n2-standard-4
This forces workers to use higher-spec CPU machines (n2-standard-4) and sets the autoscaler limits between 5 and 50 VMs.

9. Common Mistakes

  • Using local file paths: If running on Dataflow, your inputs and outputs must point to cloud paths (like gs://bucket/file.txt), not local folder paths (like /data/input.txt). Local folders do not exist on cloud worker VMs.
  • Omitting temp_location: Dataflow requires a Google Cloud Storage bucket path to stage temporary files. If omitted, the deployment will fail.

10. Best Practices

  • Always define upper autoscaling boundaries using --max_num_workers to prevent runaway billing spikes.
  • Use standard, cost-effective machine types unless your pipeline performs memory-intensive group aggregations.

11. Summary

  • DataflowRunner deploys pipelines to Google Cloud.
  • Spins up, scales, and tears down Compute Engine worker VMs automatically.
  • Requires project, region, and GCS staging folder configurations.

12. Interactive Challenges

Challenge 1: Basic Deployment Configuration (Beginner)
Write a Python code segment that configures PipelineOptions to run on the Dataflow Runner within GCP project "billing-123" and region "us-west1".
Challenge 2: Worker Autoscaling Limits (Intermediate)
Configure pipeline options that set the starting worker count to 3, the maximum worker count limit to 15, and specifies the worker VM size to "e2-standard-4".
Challenge 3: Parse Dataflow Command Line Arguments (Advanced)
Write a Python segment that instantiates a pipeline options object parsing command-line parameters, ensuring that the GoogleCloudOptions view is populated with a GCS temp location if not provided.

Related Apache Beam Topics & Lessons