PCWorkshops

Apache Workflow Terminology by  PCWorkshops Term Explanation DAG Directed Acyclic Graph — the main structure in Airflow. It defines a workflow and the order in which tasks should run. Task A single unit of work in a workflow, such as running a Python function or executing a SQL query.   DAG Run One execution/instance of a DAG. If a DAG runs every day, each day's execution is a separate DAG Run. Task Instance A specific execution of a task within a particular DAG Run. Scheduler Airflow's component that monitors DAGs and determines when tasks are ready to run. Executor Determines how and where Airflow runs tasks, such as locally, in multiple processes, or on Kubernetes. Worker A process or machine that actually executes Airflow tasks. Webserver Airflow's web interface, where you can view DAGs, task statuses, logs, and other information. Trigger Rule A rule that determines when a task is allowed to run based on the status of its upstream tasks. Dependency A rela...

SQL Constraints Primary Key Constraint Step 1:   (first ensure the field is NOT NULL) Syntax to assign to the column the not null constraint: alter table tablename  alter column column int not null   Step 2: Syntax: ALTER TABLE tablename ADD CONSTRAINT constraintname PRIMARY KEY (fieldname ); Example: Alter table Employees   add constraint PK_EmpID  PRIMARY KEY ( EmployeeID ) Foreign key constraint Example: alter table Orders              -- table name add constraint FK_EID      – constraint name foreign key (ShipperID)     -- column that will become a foreign key references Shippers (ShipperID) -– table and column name to which it links   Default constraint Syntax: Alter table tablename add constraint constraintname default value for column Example :  Alter table Orders add constraint df_Constraintname default getdate() for orderdate   Check Con...

What is Streamlit Learn Python   Streamlit is a Python framework that lets you turn Python code into an interactive web application very quickly . The simplest mental model is: Streamlit = an easy way to build a web UI using Python, without needing to learn much HTML, CSS, or JavaScript. Example Learn Python Suppose you have a Python function that predicts house prices: def predict_price(size, bedrooms): return size * 3000 + bedrooms * 10000 With Streamlit, you can turn that into a little web app: import streamlit as st st.title("House Price Predictor") size = st.number_input("House size") bedrooms = st.number_input("Bedrooms") if st.button("Predict"): price = predict_price(size, bedrooms) st.write(f"Predicted price: £{price:,.0f}") The result is a webpage with: ┌─────────────────────────────────┐ │ House Price Predictor │ │ │ │ House size: [ 1200 ] │ │ ...

What is Apache Airflow Learn Python Apache Airflow is an open-source platform for building, scheduling, and monitoring data workflows . Think of it as a tool that answers: “What should run, in what order, when should it run, and what happens if something fails?” Simple example Suppose a company has this daily data pipeline: Get data from API ↓ Clean the data ↓ Store it in a database ↓ Run analytics ↓ Send report Airflow can orchestrate all of those steps: ┌── Clean data ──┐ Get API data ┤ ├── Run analytics ── Send report └── Validate ────┘ It can run the workflow every day at 6 AM, retry failed tasks, and show you which tasks succeeded or failed. The key concepts 1. DAG (Directed Acyclic Graph) A DAG describes your workflow and its dependencies. For example: extract >> transform >> load This means: extract → transform → load 2. Tasks Individual pieces of work, such as: Run a Python function Execute SQ...

Learn Python What is AWS   AWS (Amazon Web Services) is a cloud computing platform from Amazon. The simplest way to think about it is: AWS lets you rent computing infrastructure over the internet instead of buying and maintaining your own servers. Without AWS A company might need to buy: Physical servers ↓ Storage ↓ Networking equipment ↓ Data center ↓ Electricity + cooling ↓ IT team to maintain everything That's expensive and difficult to scale. With AWS You can rent these resources from AWS: AWS │ ┌─────────────┼─────────────┐ ↓ ↓ ↓ Computers Storage Databases EC2 S3 RDS │ │ │ └─────────────┼─────────────┘ ↓ Your application You pay for the resources you use rather than owning the physical infrastructure. Some important AWS services There are hundreds of AWS ...

Python courses   What is Docker Docker is a tool that lets you package an application together with everything it needs to run and then run it consistently anywhere. The simplest mental model is: Docker = a standardized box for your application. The problem Docker solves Imagine you build a Python application on your laptop: Python 3.12 pandas 2.x requests PostgreSQL client some configuration It works perfectly. Then you give it to someone else, and they get: "It doesn't work on my machine." Maybe they have Python 3.10, a different library version, or a missing dependency. Docker helps solve this by packaging the environment: ┌─────────────────────────────┐ │ Docker Container │ │ │ │ Your application │ │ Python │ │ Libraries │ │ Configuration │ │ │ └─────────────────────────────┘ You can then run that container on another machine with much l...