Learn Python

What is AWS

 AWS (Amazon Web Services) is a cloud computing platform from Amazon.

The simplest way to think about it is:

AWS lets you rent computing infrastructure over the internet instead of buying and maintaining your own servers.

Without AWS

A company might need to buy:

Physical servers
      ↓
Storage
      ↓
Networking equipment
      ↓
Data center
      ↓
Electricity + cooling
      ↓
IT team to maintain everything

That's expensive and difficult to scale.

With AWS

You can rent these resources from AWS:

                    AWS
                     │
       ┌─────────────┼─────────────┐
       ↓             ↓             ↓
   Computers       Storage      Databases
     EC2             S3          RDS
       │             │             │
       └─────────────┼─────────────┘
                     ↓
               Your application

You pay for the resources you use rather than owning the physical infrastructure.

---

Some important AWS services

There are hundreds of AWS services, but you don't need to learn all of them.

For data engineering, these are especially important:

🖥️ EC2 — computers

EC2 gives you virtual servers.

You can essentially say:

"Give me a Linux machine with 8 CPUs and 32 GB RAM."

AWS provides it.

You could then install Docker, Spark, Python, etc.

---

🗄️ S3 — storage

S3 is AWS's object storage service.

You can store things like:

sales.csv
customers.parquet
logs.json
images
videos
backup files

A data lake might look like:

S3
│
├── raw/
│   ├── sales/
│   └── customers/
│
├── processed/
│   ├── sales/
│   └── customers/
│
└── analytics/

S3 is extremely important in data engineering.

---

🐘 RDS — managed databases

RDS lets you run databases such as PostgreSQL or MySQL without managing all the underlying infrastructure yourself.

Your application
       ↓
      RDS
       ↓
  PostgreSQL

---

⚡ Lambda — run code without managing servers

Lambda lets you run small pieces of code without provisioning a server yourself.

For example:

Someone uploads file to S3
          ↓
       Lambda
          ↓
   Validate the file
          ↓
    Start processing

This is called serverless computing.

---

🔐 IAM — permissions

IAM (Identity and Access Management) controls who can access what.

For example:

Data engineer
      ↓
Can read S3
Can run Spark
Can access database
      ↓
Cannot delete production resources

Understanding IAM is very important when working with AWS.

---

AWS + the technologies you asked about

This is probably the most useful connection to understand:

                    AWS
                     │
       ┌─────────────┼─────────────┐
       ↓             ↓             ↓
      S3            EC2          RDS
   Data storage   Compute      Database
       │
       ↓
     Spark
       │
       ↓
  Process data
       │
       ↓
   Data warehouse

And Airflow can orchestrate the whole thing:

                    Airflow
                       │
             ┌─────────┼─────────┐
             ↓         ↓         ↓
            S3       Spark      RDS
          Read data  Process    Store
             │         │
             └─────────┼─────────┘
                       ↓
                 Final dataset

Docker can package the applications:

AWS
│
├── Docker
│    ├── Airflow
│    └── Application
│
├── S3
│    └── Data
│
└── Compute
     └── Spark

The big picture

Learn Python

If you're learning data engineering, you can think of the technologies you've asked about like this:

Technology	Think of it as
AWS	The cloud/infrastructure platform
S3	Cloud storage
EC2	Cloud computer
Docker	Application containerization
Spark	Big-data processing engine
Airflow	Workflow orchestrator
PostgreSQL	Relational database

A typical real-world pipeline could therefore be:

API → S3 → Spark → PostgreSQL/Data Warehouse, with Airflow coordinating the steps, all running on infrastructure provided by AWS.

Learn Python

PCWorkshops