What is Apache Spark

Python courses

What is Apache Spark

Apache Spark is an open-source framework for processing large amounts of data quickly across multiple computers.

A simple way to think about it:

Spark is an engine for doing data processing at scale.

A simple example

Imagine you have 5 TB of customer transaction data and want to calculate total sales per customer.

Processing it on one computer could be extremely slow.

Spark can split the data across many machines:

                 5 TB of data
                      │
          ┌───────────┼───────────┐
          ↓           ↓           ↓
       Machine 1   Machine 2   Machine 3
          │           │           │
       Process      Process      Process
       its data     its data     its data
          │           │           │
          └───────────┼───────────┘
                      ↓
                Combine results
                      ↓
                Total per customer

This is called distributed computing.

What can Spark do?

Spark can perform things like:

• ETL — extract, transform, and load data

• Filter and transform huge datasets

• Join large datasets

• Aggregate data

• Run SQL queries

• Machine learning

• Stream processing

• Process files such as CSV, JSON and Parquet

• Read/write from databases and cloud storage

For example, you could write:

df = spark.read.parquet("sales.parquet")

result = (
    df.groupBy("customer_id")
      .sum("amount")
)

result.show()

Spark distributes this work across its cluster.

Spark vs Airflow

This distinction is important if you're learning data engineering:

	Airflow	Spark
Main purpose	Orchestration	Data processing
Question it answers	"When/how should jobs run?"	"How do I process this data?"
Processes huge datasets?	Not its main job	Yes
Schedules jobs?	Yes	Not primarily
Handles dependencies?	Yes	Within processing jobs
Example	Run pipeline every morning	Process 2 TB of data

They are often used together:

                    Airflow
                       │
            "Run the pipeline"
                       ↓
                  Spark job
                       │
              "Process the data"
                       ↓
                 Data Warehouse
                       ↓
                   Dashboard

So, for example:

Airflow:

Run this pipeline at 2 AM.

Spark:

Take these 500 GB of raw data, clean it, join it, aggregate it, and produce the final dataset.

Spark's basic architecture

A simplified Spark cluster looks like this:

                 Driver
                   │
        ┌──────────┼──────────┐
        ↓          ↓          ↓
    Executor    Executor    Executor
        │          │          │
      Data       Data       Data
     partition  partition  partition

The Driver coordinates the work, while Executors actually perform the computations.

One important thing to understand

Python courses

Spark isn't just "a faster Python."

You can use Spark with languages such as Python (PySpark), Scala, Java, and SQL, but the important idea is that Spark can distribute computation across a cluster.

If you're learning Airflow + Spark together, the mental model I'd use is:

Airflow = orchestrator
Spark = processing engine
S3/data lake = storage
Snowflake/BigQuery/Redshift = warehouse
dbt = SQL-based transformation layer

Python courses 

PCWorkshops