Skip to main content
intermediate

Logging & Monitoring

5 min read

1. Introduction

Logging & Monitoring inside Apache Beam pipelines involves instrumenting code to capture debug diagnostics, runtime metrics, and health telemetry across distributed worker nodes during batch or streaming execution.

2. Why This Concept Exists

Because Apache Beam pipelines execute across distributed worker clusters (such as Google Cloud Dataflow, Apache Spark, or Apache Flink), traditional interactive debugging via breakpoints or local console output is impossible. When processing millions of events per second, pipeline engineers require structured logging and aggregate metrics to diagnose system lag, identify bottleneck transforms, catch data corruption early, and monitor worker resource utilization.

3. Key Terminology

  • Worker Logging: Standard Python logging output routed automatically to runner diagnostic sinks (such as Google Cloud Logging / Stackdriver or Flink TaskManager logs).
  • Beam Metrics: Distributed aggregators (Metrics.counter, Metrics.distribution, Metrics.gauge) tracked by worker threads and reported back to the runner pipeline dashboard without polluting raw text log streams.
  • Sampling: Selectively logging a controlled fraction of elements (e.g., 1 out of 10,000) to maintain visibility without overwhelming worker I/O or racking up cloud log ingestion costs.
  • System Lag: The delay between event timestamp generation and the current processing watermark, surfaced in runner monitoring graphs.

4. Visual Architecture

How worker log streams and metric counters flow from distributed nodes into centralized monitoring consoles:

text
┌────────────────────────────────────────────────────────┐
│               Distributed Worker Nodes                 │
│  ┌───────────────────────┐   ┌───────────────────────┐ │
│  │   Worker VM 1         │   │   Worker VM 2         │ │
│  │  - logging.warning()  │   │  - logging.error()    │ │
│  │  - Metrics.counter()  │   │  - Metrics.counter()  │ │
│  └───────────┬───────────┘   └───────────┬───────────┘ │
└──────────────┼───────────────────────────┼─────────────┘
               │ Structured Telemetry Push │
               ▼                           ▼
┌────────────────────────────────────────────────────────┐
│     Centralized Cloud Observability (Dataflow / Cloud) │
│  - Cloud Logging (Search by Worker ID, Job ID, Severity)│
│  - Cloud Monitoring (Element throughput, System Lag)   │
│  - Metric Counters (Filtered Elements, DLQ Volume)     │
└────────────────────────────────────────────────────────┘

5. Code Example: Structured Logging & Custom Metrics

Integrating Python's standard logging library alongside Apache Beam distributed metrics inside a custom DoFn:
python
import apache_beam as beam
from apache_beam.metrics import Metrics
import logging

class TelemetryEnrichmentDoFn(beam.DoFn):
    def __init__(self):
        # Declare distributed metrics visible in runner console
        self.processed_counter = Metrics.counter(self.__class__, "elements_processed")
        self.error_counter = Metrics.counter(self.__class__, "parse_errors")
        self.latency_dist = Metrics.distribution(self.__class__, "processing_latency_ms")

    def process(self, element):
        self.processed_counter.inc()
        
        try:
            user_id = element.get("user_id")
            amount = float(element.get("amount", 0))
            
            # Use appropriate log severity levels
            if amount > 50000:
                logging.warning(f"High-value audit threshold reached for user {user_id}: ${amount:,.2f}")
                
            yield {"user_id": user_id, "amount": amount, "status": "APPROVED"}
            
        except Exception as e:
            self.error_counter.inc()
            logging.error(f"Failed to process element {element}: {str(e)}", exc_info=True)

with beam.Pipeline() as p:
    (p
     | "CreateSampleEvents" >> beam.Create([
         {"user_id": "u1", "amount": 120.50},
         {"user_id": "u2", "amount": 75000.00},
         {"user_id": "u3", "amount": "invalid_number"}
     ])
     | "InstrumentedDoFn" >> beam.ParDo(TelemetryEnrichmentDoFn())
     | "LogResults" >> beam.Map(lambda x: logging.info(f"Final output: {x}")))

6. Production Best Practices

  • Never Log Every Record at INFO Level: When running pipelines at 500,000 events/second, logging every item generates gigabytes of text per minute, saturating worker disk I/O, throttling throughput, and ballooning log-storage invoices.
  • Prefer Metrics Over Text Logs for Counting: Use Metrics.counter(...) rather than logging.info("Count: ...") for cumulative stats. Metrics are merged locally via combiner logic and sent to runner dashboards with negligible overhead.
  • Log Contextual Metadata: Always include identifiers like job_id, step_name, and correlation_id in log messages so you can correlate individual worker exceptions with global job runs.
  • Configure Runner Alerting: Set automated alerts on critical metrics like System Lag > 60s or Worker CPU Utilization > 85% to trigger proactive auto-scaling or incident triage.

Related Apache Beam Topics & Lessons