> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify-poc.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Integrate Apache Airflow with Tiger Cloud

> Author, schedule, and monitor workflows to orchestrate your data pipelines

export const PG = 'Postgres';

export const CONSOLE = 'Tiger Console';

export const CLOUD_LONG = 'Tiger Cloud';

export const SERVICE_LONG = 'Tiger Cloud service';

export const SELF_LONG = 'self-hosted TimescaleDB';

A [DAG (Directed Acyclic Graph)][airflow-dag] is the core concept of Airflow, collecting [Tasks][airflow-task] together,
organized with dependencies and relationships to say how they should run. You declare a DAG in a Python file
in the `$AIRFLOW_HOME/dags` folder of your Airflow instance.

This page shows you how to use a Python connector in a DAG to integrate Apache Airflow with a {SERVICE_LONG}.

## Prerequisites

To follow the steps on this page:

* Create a target [{SERVICE_LONG}][create-service] with Real-time analytics enabled.<p />

  You need [your connection details][connection-info]. This procedure also
  works for [{SELF_LONG}][enable-timescaledb].

[create-service]: /deploy-and-operate/tiger-cloud/get-started/create-services

[enable-timescaledb]: /deploy-and-operate/self-hosted/install-and-update/install-self-hosted

[connection-info]: /integrations/find-connection-details

* Install [Python3 and pip3][install-python-pip]
* Install [Apache Airflow][install-apache-airflow]

  Ensure that your Airflow instance has network access to {CLOUD_LONG}.

This example DAG uses the `company` table you create in [Optimize time-series data in hypertables][ingest-data]

## Install python connectivity libraries

To install the Python libraries required to connect to {CLOUD_LONG}:

1. **Enable {PG} connections between Airflow and {CLOUD_LONG}**

   ```bash theme={"dark"}
   pip install psycopg2-binary
   ```

2. **Enable {PG} connection types in the Airflow UI**

   ```bash theme={"dark"}
   pip install apache-airflow-providers-postgres
   ```

## Create a connection between Airflow and your service

In your Airflow instance, securely connect to your {SERVICE_LONG}:

1. **Run Airflow**

   On your development machine, run the following command:

   ```bash theme={"dark"}
   airflow standalone
   ```

   The username and password for Airflow UI are displayed in the `standalone | Login with username`
   line in the output.

2. **Add a connection from Airflow to your {SERVICE_LONG}**

   1. In your browser, navigate to `localhost:8080`, then select `Admin` > `Connections`.
   2. Click `+` (Add a new record), then use your [connection info][connection-info] to fill in
      the form. The `Connection Type` is `Postgres`.

## Exchange data between Airflow and your service

To exchange data between Airflow and your {SERVICE_LONG}:

1. **Create and execute a DAG**

   To insert data in your {SERVICE_LONG} from Airflow:

   1. In `$AIRFLOW_HOME/dags/timescale_dag.py`, add the following code:

      ```python theme={"dark"}
      from airflow import DAG
      from airflow.operators.python_operator import PythonOperator
      from airflow.hooks.postgres_hook import PostgresHook
      from datetime import datetime

      def insert_data_to_timescale():
          hook = PostgresHook(postgres_conn_id='the ID of the connenction you created')
          conn = hook.get_conn()
          cursor = conn.cursor()
          """
            This could be any query. This example inserts data into the table
            you create in:

            https://www.tigerdata.com/docs/getting-started/latest/try-key-features-timescale-products/#optimize-time-series-data-in-hypertables-with-hypercore
           """
          cursor.execute("INSERT INTO crypto_assets (symbol, name) VALUES (%s, %s)",
           ('NEW/Asset','New Asset Name'))
          conn.commit()
          cursor.close()
          conn.close()

      default_args = {
          'owner': 'airflow',
          'start_date': datetime(2023, 1, 1),
          'retries': 1,
      }

      dag = DAG('timescale_dag', default_args=default_args, schedule_interval='@daily')

      insert_task = PythonOperator(
          task_id='insert_data',
          python_callable=insert_data_to_timescale,
          dag=dag,
      )
      ```

      This DAG uses the `company` table created in [Create regular {PG} tables for relational data][ingest-data].

   2. In your browser, refresh the Airflow UI.

   3. In `Search DAGS`, type `timescale_dag` and press ENTER.

   4. Press the play icon and trigger the DAG:
      ![daily eth volume of assets][daily-eth-volume-of-assets]
2. **Verify that the data appears in {CLOUD_LONG}**

   1. In [{CONSOLE}][cloud-login], navigate to your service and click `SQL editor`.
   2. Run a query to view your data. For example: `SELECT symbol, name FROM company;`.

      You see the new rows inserted in the table.

You have successfully integrated Apache Airflow with {CLOUD_LONG} and created a data pipeline.

[airflow-dag]: https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#dags

[airflow-task]: https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html

[cloud-login]: https://console.cloud.timescale.com/

[connection-info]: /integrations/find-connection-details

[daily-eth-volume-of-assets]: https://assets.timescale.com/docs/images/integrations-apache-airflow.png

[ingest-data]: /getting-started/try-key-features-timescale-products#optimize-time-series-data-in-hypertables-with-hypercore

[install-apache-airflow]: https://airflow.apache.org/docs/apache-airflow/stable/start.html

[install-python-pip]: https://docs.python.org/3/using/index.html