– easy logs access – catalogue with logs is also mounted as a volume and give us Chance to quickly grep, tai lor simply read log files
AIRFLOW DOCKER COMPOSE UPDATE
– fast DAG update – catalogue with DAGs is mounted as volume, update of DAGs is as fast as values of SCHEDULER_DAG_DIR_LIST_INTERVAL EASY LOGS ACCESS parameter, which can be seconds Main advantages that we discovered using local Airflow environment on Docker-Compose are:
AIRFLOW DOCKER COMPOSE CODE
We have added libraries like black, pylint and pytest in order to check code and scripts helping us with image rebuild and cloud connection settings.Īdvantages of local Airflow environment based on Docker Compose in comparison to remote environment There is official Docker Compose file Airflow Community provided us with, however to better mirror production settings we customize it a little bit. It is great solution for development and testing. With Compose you can set the desired amount of containers counts, their builds, and storage designs, and then with a single set of commands you can build, run and configure all the containers. Docker Compose as a solutionĭocker Compose is a tool for defining and running multi-container Docker applications. We went for a simpler solution and used Docker Compose.
Using Kubernetes locally was for us too much overhead, because we want to focus on DAG development not configuration.
In order to deliver quality pipelines faster we decided to spin up local Airflow environments. Our production and preprod environment works on cloud using Kubernetes Engine, DAGs synchronization took us each time around 2 to 5 minutes, so for efficient development it is far too long. You are not alone, we have the same topic coming up during our retro meetings ? I assume that you are familiar with Airflow, you use it constantly and wondering what is the better way to write and test DAGs. If you are new in this topic I highly recommend to take a look at In the previous article we wrote about use cases and general purpose of the Apache Airflow.
AIRFLOW DOCKER COMPOSE HOW TO
How to make reasonable simplification and make your code testable? Let’s learn on example of Apache Airflow. Sometimes production environments are very complex, for example they use Kubernetes clusters and connections to cloud services, and deployment takes a lot of time. In order to achieve that goal we use local environments to develop, test and optimize our code.
0 kommentar(er)
