#Airflow streaming how to#
#Airflow streaming code#
#Airflow streaming series#
#Airflow streaming free#

Remote spark-submit to YARN running on EMR - Azhaguselvan walks through submitting Spark jobs to existing EMR clusters with Airflow.

How Apache Airflow Distributes Jobs on Celery workers - A short description of the steps taken by a task instance, from scheduling to success, in a distributed architecture.

#Airflow streaming code#

Just clone and run make start-airflow to get started! Add some CI jobs to deploy your code and you're done.

Airflow Repository Template - A boilerplate repository for developing locally with Airflow, with linting & tests for valid DAGs and plugins.

Start Building Better Data Pipelines With apache Airflow 2020-Oct - Naman Gupta covers the basics of Airflow and its concepts.

Kimaru Thagana covers a practical case of doing an ETL process using Apache Airflow using a dummy ecommerce store's transactional, user and product data.

ETL with Apache Airflow for Data Analysis on Transaction Data.

#Airflow streaming series#

Introduction to Airflow - A web tutorial series by maxcotec for beginners and intermediate users of Apache Airflow.

Apache Airflow Monitoring Metrics - A two-part series by maxcotec on how you can utilize existing Airflow statsd metrics to monitor your airflow deployment on Grafana dashboard via Prometheus.

Airflow-Component - Lightweight installer of federated Airflow-Airflow (RabbitMQ) reference architectrure on Compute node(s).

#Airflow streaming how to#

Introducing KEDA for Airflow - How to use KEDA scaler system to enable autoscaling of celery workers based on data stored in the Airflow metadata database.

Distribute & deploy Apache Airflow via Python PEX files - Example repo with steps to bundle, distribute, & deploy Apache Airflow as PEX files.

A CeleryExecutor docker-compose deployment is available here.

Bitnami Airflow Worker Docker image - A secure and up-to-date docker image for Airflow Worker maintained by Bitnami.

Bitnami Airflow Scheduler Docker image - A secure and up-to-date docker image for Airflow Scheduler maintained by Bitnami.

Bitnami Airflow Docker image - A secure and up-to-date docker image for Airflow maintained by Bitnami.

Astronomer Platform - Apache Airflow as a Service on Kubernetes.

Integrating Apache Airflow with Apache Ambari - Mykola Mykhalov walks through using Apache Ambari to configure and deploy an Airflow instance.

Running Airflow on top of Apache Mesos - Blog describing how to configure Mesos to run all of the Airflow componenents.

airflow-cookbook - Chef cookbook for deploying Airflow.

airflow-k8s-executor-on-GKE - A detailed tutorial to get a scalable, low maintenance airflow kubernetes executor environment deployed on Google Kubernetes Engine with helm.

airflow-on-kubernetes - A guide on all relevant resources, scripts and projects that relate to running Airflow on Kubernetes.

mumoshu's repository has not been recently updated, but there are numerous forks that may be based on more recent releases.

kube-airflow - This repository contains both an Airflow Docker image (that appears to have been based on Puckel's work) and Kubernetes service definition.

Deploys after a few clicks with CloudFormation.

aws-airflow-stack - An AWS based Airflow cluster deployment with CeleryExecutor.

It can be docker pulled at datagovsg/airflow-pipeline.

airflow-pipeline - Airflow Docker container that comes preconfigured for Spark and Hadoop.

Kubernetes Custom Operator for Deploying Airflow - Kubernetes Custom controller (also called operator pattern) for deploying Airflow on Kubernetes.

It is regularly updated and closely tracks the official Apache releases.

Puckel's Docker Image - well-crafted Docker image has become the base for many Airflow installations.

Stable Celery Helm Chart - Curated Helm Chart in the official stable chart repository.

KubernetesExecutor Helm Chart - A lean Helm Chart using the KubernetesExecutor for a more k8s native experience and complementary KubernetesExecutor Docker Image.

#Airflow streaming free#

Apache Airflow Multi-Tier Free Deployment on Azure - A free Azure Resource Manager (ARM) template by Bitnami providing a one-click solution for Airflow deployment on Azure for production use-cases.

Three ways to run Airflow on Kubernetes - Tim van de Keer walks through several methods for deploying Airflow on Kubernetes.

Installing Airflow on IBM Cloud - Quick and easy deployment on IBM Cloud with IBM Bitnami Charts.

Documentation (also the official website).

Source code (latest stable release 1.10.12).

Commercial Airflow-as-a-service providers.

Slide deck presentations and online videos.

Best practices, lessons learned and cool use cases.

Items are generally added at the top of each section so that more fresh items are featured more prominently. Please feel free to contribute any items that should be included. This is a curated list of resources about Apache Airflow.