Apache Spark: Distributed Data Processing & PySpark Guide

Apache Spark is the leading unified analytics engine for large-scale data processing, combining batch processing, streaming, SQL, machine learning, and graph computation in a single framework. Process petabytes across thousands of nodes with in-memory speed.

What is Apache Spark?

Apache Spark is an open-source, distributed computing framework originally developed at UC Berkeley's AMPLab in 2009. It provides a unified API for batch processing, streaming, SQL queries, machine learning (MLlib), and graph processing (GraphX). Spark's core innovation is in-memory computation — intermediate results are kept in RAM across transformation steps rather than written to disk like Hadoop MapReduce, making it 10–100x faster for iterative workloads.

Spark can run on a local machine, a standalone cluster, YARN, Mesos, or Kubernetes. Managed services like Databricks, Amazon EMR, Google Dataproc, and Azure HDInsight run Spark at scale without cluster management overhead. PySpark is the Python API and the most widely used interface — it provides the full DataFrame, Spark SQL, and Streaming API with Python syntax.

PySpark DataFrame Quickstart

# pip install pyspark
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, sum as spark_sum, avg, count

# Initialize SparkSession
spark = SparkSession.builder     .appName("Sales Analysis")     .config("spark.sql.shuffle.partitions", "200")     .getOrCreate()

# Read data (Parquet, CSV, JSON, Delta, JDBC...)
df = spark.read.parquet("s3://my-bucket/sales/")

# DataFrame transformations (lazy — nothing executes yet)
result = (
    df
    .filter(col("status") == "completed")
    .filter(col("amount") > 0)
    .withColumn("amount_usd", col("amount") / 100)
    .groupBy("region", "product_category")
    .agg(
        spark_sum("amount_usd").alias("total_revenue"),
        avg("amount_usd").alias("avg_order_value"),
        count("order_id").alias("order_count"),
    )
    .orderBy(col("total_revenue").desc())
)

# Action triggers actual computation
result.show(20)
result.write.mode("overwrite").parquet("s3://my-bucket/output/sales_summary/")

spark.stop()

Spark SQL & Broadcast Joins

# Spark SQL — register temp view and run SQL
from pyspark.sql import SparkSession
from pyspark.sql.functions import broadcast

spark = SparkSession.builder.appName("SparkSQL").getOrCreate()

orders = spark.read.parquet("s3://bucket/orders/")
customers = spark.read.parquet("s3://bucket/customers/")

# Register as temp views
orders.createOrReplaceTempView("orders")
customers.createOrReplaceTempView("customers")

# Run standard SQL
result = spark.sql("""
    SELECT
        c.country,
        DATE_TRUNC('month', o.created_at) AS month,
        COUNT(o.order_id)                 AS orders,
        SUM(o.amount_usd)                 AS revenue
    FROM orders o
    JOIN customers c ON o.user_id = c.user_id
    WHERE o.status = 'completed'
    GROUP BY 1, 2
    ORDER BY 2 DESC, 3 DESC
""")
result.show()

# Broadcast join optimization for small dimension tables
result2 = orders.join(
    broadcast(customers),   # sends small table to all executors
    on="user_id",
    how="left"
)

Performance Optimization

# Spark performance optimization techniques
from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName("Optimization").getOrCreate()

df = spark.read.parquet("s3://bucket/events/")

# 1. Caching — keeps DataFrame in memory across multiple actions
df.cache()                     # MEMORY_AND_DISK by default
df.persist()                   # same as cache()
df.unpersist()                 # free memory when done

# 2. Repartitioning
df_balanced = df.repartition(200)                       # round-robin
df_by_key   = df.repartition(50, col("user_id"))        # hash partition
df_small    = df.coalesce(10)                           # reduce (no shuffle)

# 3. Predicate pushdown — Spark pushes filters to data source
df_filtered = spark.read     .option("pushDownPredicate", "true")     .parquet("s3://bucket/events/")     .filter(col("event_date") == "2024-01-01")

# 4. Explain plan — inspect physical plan
df_filtered.explain(mode="formatted")

# 5. Adaptive Query Execution (AQE) — Spark 3.x auto-tuning
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")

Spark vs Apache Flink vs Dask

Best Practices

• Enable Adaptive Query Execution (spark.sql.adaptive.enabled=true) for automatic shuffle partition optimization
• Broadcast small dimension tables in joins to eliminate expensive sort-merge shuffle
• Use coalesce() to reduce partition count without shuffle; repartition() when you need a balanced distribution
• Always call unpersist() after cached DataFrames are no longer needed to free executor memory
• Store data in columnar Parquet or Delta Lake format to benefit from predicate pushdown and columnar reads
• Profile jobs with Spark UI's SQL tab and Stage details to identify shuffle bottlenecks and spill