Installing Apache Airflow Locally: A Step-by-Step Guide for Developers and Data Engineers

• 3.0.4

Introduction

Apache Airflow is a popular open-source platform designed for authoring, scheduling, and monitoring workflows. Installing Airflow locally offers a flexible environment for development, testing, and debugging workflows without relying on cloud or server dependencies. With local installations, you can experiment with different configurations, troubleshoot issues, and develop workflows offline, creating a controlled environment for hands-on learning and testing.

In this guide, we’ll walk through several methods for installing Apache Airflow locally, covering containerized installation with Docker and language-specific package managers, including pip for Python, npm for Node.js (Airflow plugins and front-end), and Maven or Gradle for Java-related components.

1. Docker Installation

Using Docker for Airflow is ideal because it isolates dependencies and reduces compatibility issues. The official Airflow Docker image simplifies this process by packaging everything needed to run Airflow in one containerized setup.

Steps for Docker Installation

Install Docker: Ensure Docker is installed on your machine. You can download Docker from Docker’s website.

Pull the Airflow Image: Use the following command to pull the official Apache Airflow image:

docker pull apache/airflow:latest

Set Up the Airflow Environment: Create a directory for Airflow’s configuration files and logs:

mkdir -p ~/airflow/{dags,logs,plugins}

Initialize the Database: Airflow needs a metadata database. Initialize it using:

docker run --rm \
    -v ~/airflow:/opt/airflow \
    apache/airflow:latest airflow db init

Start the Airflow Container: Run the container, binding ports and directories:

docker run -d \
    -p 8080:8080 \
    -v ~/airflow:/opt/airflow \
    apache/airflow:latest webserver

Verify the Installation: Open your browser and navigate to http://localhost:8080. You should see the Airflow web interface, confirming that the installation is successful.

2. Python Installation (pip)

Apache Airflow is primarily a Python-based tool, so installing it with pip is a popular method for Python developers.

Steps for Pip Installation

Set Up a Virtual Environment: It’s best to install Airflow in a virtual environment to isolate it from other Python packages.

python3 -m venv airflow_venv
source airflow_venv/bin/activate

Install Apache Airflow: Use pip to install Airflow along with its dependencies.

pip install apache-airflow

Initialize the Database: Similar to the Docker setup, initialize the metadata database:

airflow db init

Start the Web Server: Run the Airflow web server on the default port:

airflow webserver --port 8080

Verify the Installation: Access the Airflow UI at http://localhost:8080 to confirm the setup.

3. Java Installation (Maven/Gradle)

For developers working with Java, Maven or Gradle can manage Airflow’s Java dependencies. While Airflow isn’t written in Java, using Java tools to handle integrations or plugins may be necessary in some environments.

Steps for Maven Installation

Install Maven: Make sure Maven is installed. Check with:

mvn -version

Add Airflow Dependencies: Add the necessary Airflow-related dependencies to your pom.xml file:

<dependencies>
    <dependency>
        <groupId>org.apache.airflow</groupId>
        <artifactId>airflow-api</artifactId>
        <version>latest-version</version>
    </dependency>
</dependencies>

Compile and Run: Run Maven commands to compile and build the necessary files:

mvn clean install

This process ensures that Java applications can communicate with Airflow’s APIs where needed.

4. Node.js Installation (npm)

Airflow’s front-end elements and certain plugins may rely on Node.js components, making npm useful for handling these installations.

Steps for npm Installation

Install Node.js and npm: Download and install Node.js, which includes npm, from Node.js website.

Set Up the Airflow UI Dependencies: Navigate to the airflow/www/ directory and install the required npm packages.

cd airflow/www/
npm install

Run the Development Server: Start the Airflow development server for UI components:

npm run dev

Using npm is optional for most installations, but it can be necessary for those developing or modifying the Airflow front-end.

Managing and Verifying Your Installation

After installation, it’s essential to keep Airflow and its components updated and well-managed. Here are a few tips:

1. Verify Components: After each installation, verify that the components (scheduler, web server, workers) are running smoothly. Use the Airflow UI or airflow info for confirmation.
2. Update Dependencies: Regularly check for updates to Airflow and any related packages, especially security patches.
3. Logging and Monitoring: Airflow’s logs are invaluable for troubleshooting. Ensure logging is configured correctly, particularly for production setups.
4. Use Virtual Environments and Containers: Isolating Airflow installations in virtual environments or containers will reduce conflicts with other software on your machine.

Summary

Installing Apache Airflow locally can seem complex, but Docker, pip, npm, and Maven offer versatile methods for different development needs. Using Docker provides a streamlined containerized installation, while pip, npm, and Maven enable language-specific installs suitable for different types of development. Local installations give you control over your development environment, allowing you to experiment with configurations and workflows effectively.

By following this guide, you should have a working local installation of Apache Airflow that can help streamline your development and testing processes.

Last Releases

• 3.0.4August 8, 2025
  Apache Airflow 3.0.4
• 3.0.3July 14, 2025
  Apache Airflow 3.0.3
• helm-chart/1.18.0July 13, 2025
  Apache Airflow Helm Chart 1.18.0