Skip to content

Simplify the build pipeline and include more components

Btullis requested to merge tweak_airflow_build into main

The previous build did not include some of the necessary airflow providers such as spark, hive, and hdfs. Nor did it include skein.

This commit adds those components and attempts to simplify the build process somewhat, by:

  • Installing all packages from pypi instead of building from source
  • Skipping pipx and hatch by avoiding the use of a venv
  • Using a local constraints file to fix the versions of some packages

Also we use a local requirements file to install additional packages that we will need, such as skein. These are not a standard part of the airflow environment, but we will likely need them.

Bug: T363000

Edited by Btullis

Merge request reports