Skip to content

Prepare the field for using spark3-submit

Brouberol requested to merge T377928 into main

spark3-submit is a custom wrapper we define in conda-analytics to submit jobs via spark-submit. As we use this custom wrapper everywhere in airflow-dags, we make it a symlink to spark-submit (itself provided by pyspark).

We also install openJDK8, to make it possible for spark-submit to run, as it's a bash wrapper around calling the SparkSubmit jar.

We revert to using Bullseye at the moment, as we're struggling to build Bigtop 1.5 for Bookworm.

Signed-off-by: Balthazar Rouberol brouberol@wikimedia.org Bug: T377928

Edited by Brouberol

Merge request reports