Pub/Sub IO
Interactive Syntax Reference
Pub/Sub IOApache Beam Cheatsheet: Pub/Sub IO
Recommended reading: 3 minsCopy-paste ready Python code snippets
Core Architecture Overview
Integrate with serverless Google Cloud Pub/Sub for messaging.
ReadFromPubSub()
returns: PCollectionPurpose & Description
Reads serverless message publisher queues in real-time.
Syntax Signature
beam.io.ReadFromPubSub(topic=None, subscription=None)Executable Example
import apache_beam as beam
# Read from a Google Cloud Pub/Sub subscription
messages = p | "ReadPubSub" >> beam.io.ReadFromPubSub(
subscription="projects/my-proj/subscriptions/my-sub"
)Used In
Starting point for real-time streaming analytics pipelines.
Common Pitfall
Reading directly from a topic instead of a subscription, which can result in data loss during scaling.
Pro Tip
Pub/Sub automatically assigns element event-time timestamps based on message publication date.
More Free Data Engineering Cheatsheets (DataPlayArena Network)Interactive syntax references