A union of curiosity and data science

Knowledgebase and brain dump of a database engineer


Airflow - access Configuration JSON (Optional) in DAG

Access parameters passed to airflow dag from airflow UI.

 

" To access configuration in your DAG use {{ dag_run.conf }}". This didn't work for me... so, I did the following: 

 

1. To use this data you must setup configs. 

a. add config - airflow.cfg : dag_run_conf_overrides_params=True
b. if Amazon MWAA Configs : core.dag_run_conf_overrides_params=True

 

2. Get the data from kwargs in your function. 

from datetime import datetime
from airflow import DAG
from airflow.operators.python_operator import PythonOperator

def print_configs(ds, **kwargs):
    print("dag_run: ", kwargs['params']) 
    print("ds:", ds)  

dag = DAG('manual_config_run_test', description='manual configs', schedule_interval='0 12 * * *', start_date=datetime(2021, 7, 11), catchup=False)

print_d = PythonOperator(task_id='print_manually_passed_configs', python_callable=print_configs, dag=dag, provide_context=True)

print_d

 

Note the "provide_context=True" in the python operator call. 

 

Result in the log : 

[2021-07-12 18:27:50,651] {{logging_mixin.py:112}} INFO - dag_run: {'end_date': '2021-07-12'}
[2021-07-12 18:27:50,707] {{logging_mixin.py:112}} INFO - ds: 2021-07-12
[2021-07-12 18:27:50,762] {{python_operator.py:114}} INFO - Done. Returned value was: None

 

Comments (1) -

  • rsMsYuFwmxsqtp

    11/22/2021 3:42:30 PM | Reply

    745397 211907A really really intriguing post! I´┐Żll try to track that continues here! Thank you. 738121

Add comment