Apache Workflow Terminology

-

Apache Workflow Terminology




TermExplanation
DAG
Directed Acyclic Graph — the main structure in Airflow. It defines a workflow and the order in which tasks should run.

Task
A single unit of work in a workflow, such as running a Python function or executing a SQL query.
 
DAG Run
One execution/instance of a DAG.
If a DAG runs every day, each day's execution is a separate DAG Run.

Task InstanceA specific execution of a task within a particular DAG Run.
Scheduler
Airflow's component that monitors DAGs and determines when tasks are ready to run.
Executor
Determines how and where Airflow runs tasks, such as locally, in multiple processes, or on Kubernetes.

Worker
A process or machine that actually executes Airflow tasks.
Webserver
Airflow's web interface, where you can view DAGs, task statuses, logs, and other information.
Trigger Rule
A rule that determines when a task is allowed to run based on the status of its upstream tasks.
Dependency
A relationship between tasks that defines their execution order.
For example, A >> B means A must run before B.
XCom
Short for cross-communication. A mechanism for tasks to exchange small pieces of data with each other.
Connection
A stored configuration containing information needed to connect to an external system, such as a database or cloud service.
Variable
A key-value setting stored in Airflow that can be accessed by DAGs and tasks. Useful for configuration that may change.
Hook
A Python interface that helps Airflow communicate with external systems such as databases, APIs, or cloud platforms.
Sensor
A special type of task that waits for something to happen, such as a file appearing or a database record becoming available.

Schedule
Defines when a DAG should run, such as every day, every hour, or according to a cron expression.

BackfillRunning a DAG for historical dates that were not previously executed.

Catchup
Airflow's ability to automatically create runs for previous scheduled periods when a DAG is started or its schedule is changed.

TaskFlow APIA modern Airflow approach that lets you define tasks using Python functions with the @task decorator, making dependencies and data passing easier to express.
 




 

By code2j on youtube








Comments

Post a Comment

Popular posts from this blog

Delete vs Truncate in MySQL and MS SQL Server

-
Delete vs Truncate in MySQL and MS SQL Server DELETE DELETE is a DML command. The DELETE statement is executed using a row lock; each row in the table is locked for deletion. We can specify filters in where clause It deletes specified data if a "where" condition exists. Delete activates a trigger because the operations are logged individually. Slower than truncate because it keeps logs. Rollback is possible. TRUNCATE TRUNCATE is a DDL command. TRUNCATE TABLE always locks the table and page but not each row. Cannot use the WHERE condition. It removes all the data. TRUNCATE TABLE cannot activate a trigger because the operation does not log individual row deletions. Faster performance-wise, because it doesn't keep any logs. Rollback is possible. DELETE and TRUNCATE both can be rolled back when used with TRANSACTION (TRUNCATE can be rolled back in SQL Server, but not in MySQL ). if there is a PK with auto increment, truncate will reset the counter Learn SQL with us at PCWorks...
Read more

Building Cross-Platform JavaFX Apps with Gluon

-
Building Cross-Platform JavaFX Apps with Gluon If you love JavaFX for desktop development, you’ll be excited to know that Gluon takes it a step further—bringing your JavaFX applications to mobile and embedded devices with minimal effort. Java Bootcamp   What Is Gluon? Gluon is a powerful suite of tools and libraries that extend JavaFX to run seamlessly across platforms—including iOS , Android , desktop , and even embedded systems . Its mission is simple: to make “write once, run anywhere” truly possible for JavaFX developers. Java Bootcamp Key Highlights of Gluon 1. Cross-Platform Support With Gluon, you can use a single JavaFX codebase to build apps that run natively on mobile, desktop, and embedded devices. This drastically reduces development time and maintenance effort. Java Bootcamp 2. Gluon Mobile Designed for modern devices, Gluon Mobile provides optimized UI controls and themes built for touch interaction—so your app looks and feels natural on smartphones and ta...
Read more

Companies and real-world applications using JavaFX

-
Image
Companies using JavaFX Java Bootcamp Here are some examples of  companies and real-world applications  using JavaFX — this shows it is widely being used in production : Lynden Inc. (Alaska, USA) They used JavaFX for their “Navigator 2.0” desktop application deployed on ~1,000 desktops.   “I rolled out Lynden Navigator 2.0 to 1,000 desktops across the company.”  SkedPal :  “SkedPal … we made the decision to start using JavaFX …”   A commercial time-management / scheduling tool for busy professionals that uses JavaFX for its desktop client.  Carl Zeiss Meditec AG According to one summary of real-world apps, their clinical data management system uses JavaFX (in combination with other tech).   NASA (via one of its applications) An application named “Deep Space Trajectory Explorer” uses JavaFX (or the underlying JavaFX toolkit) for trajectory visualization.  Many more companies: According to a data listing there ar...
Read more