Skip to content

Airflow development improvements

Fabian Kaelin requested to merge postamble into main

Two changes aimed to improve the development:

  • when running the airflow dev instance as a service user (to be able to use SparkSkeinSubmitHook during development), add an flag to run_dev_instance.sh to install wmf_airflow_common into the conda environment used for the airflow dev instance.
  • add a postamble argument to complement the preamble argument to the SparkSkeinSubmitHook. One use case is to use command_postamble = "|| sleep 600", which keeps the skein application running in case the submit command failed for debugging.

Merge request reports