Skip to content

Enable customizable logger levels for Spark apps

Aqu requested to merge T381074_quieter_spark_logger into main

Spark apps default to the INFO logging level, which often creates excessive noise. Critical logs can be obscured by Spark's verbose output.

This change allows a custom Log4j properties file to be specified and hosted in the refinery. Users can control logging verbosity for quieter logs.

It only overrides the Spark driver log configuration as Skein is only retrieving those.

https://gerrit.wikimedia.org/r/c/analytics/refinery/+/1100394

Bug: T381074

Merge request reports