What is Apache Airflow

What is Apache Airflow

Learn Python

Apache Airflow is an open-source platform for building, scheduling, and monitoring data workflows.

Think of it as a tool that answers:

“What should run, in what order, when should it run, and what happens if something fails?”

Simple example

Suppose a company has this daily data pipeline:

Get data from API
       ↓
Clean the data
       ↓
Store it in a database
       ↓
Run analytics
       ↓
Send report

Airflow can orchestrate all of those steps:

             ┌── Clean data ──┐
Get API data ┤                ├── Run analytics ── Send report
             └── Validate ────┘

It can run the workflow every day at 6 AM, retry failed tasks, and show you which tasks succeeded or failed.

The key concepts

1. DAG (Directed Acyclic Graph)
A DAG describes your workflow and its dependencies.

For example:

extract >> transform >> load

This means:

extract → transform → load

2. Tasks
Individual pieces of work, such as:

• Run a Python function

• Execute SQL

• Transfer files

• Call an API

• Run a Spark job

• Execute a shell command

3. Scheduler
Airflow's scheduler determines when tasks should run.

For example:

Every day at 02:00
        ↓
    Start DAG
        ↓
  Run task A
        ↓
  Run task B

4. Operators
Operators provide reusable ways to perform tasks, such as Python, SQL, Bash, cloud-service, and other operations.

5. UI / monitoring
Airflow provides a web interface where you can see:

• Workflow status

• Task failures

• Execution history

• Logs

• Task dependencies

• Retry attempts

What Airflow is not

Airflow is primarily an orchestrator, not a database or a data-processing engine.

For example, Airflow might tell Spark:

“Run this Spark job now.”

Spark does the actual large-scale data processing.

Similarly, Airflow might tell a database:

“Execute this SQL query.”

The database does the actual query processing.

Where it's commonly used

Airflow is especially common in data engineering:

                 Airflow
                    │
       ┌────────────┼────────────┐
       ↓            ↓            ↓
     APIs        Databases      Cloud
       │            │            │
       └────────────┼────────────┘
                    ↓
              Data Warehouse
                    ↓
                Analytics

For example, you might use Airflow to orchestrate an ETL/ELT pipeline that pulls data from Salesforce, transforms it, loads it into Snowflake, and refreshes a dashboard.

In one sentence

Learn Python

Apache Airflow is like a control center for automated workflows: it schedules jobs, manages dependencies, handles failures/retries, and lets you monitor everything.

If you're learning data engineering, a good next step is understanding DAGs, tasks, operators, the scheduler, and how Airflow actually executes a Python workflow.

Learn Python

PCWorkshops