Skip to content

Allow spark_kwargs to be passed to all table maintenance tasks.

Xcollazo requested to merge fix-failing-table-maintenance into main

In this MR we:

  • Modify the table maintenance mechanism to allow passing spark_kwargs and spark_conf to all table maintenance tasks.
  • We bump default resources for all maintenance tasks, as they are prone to fail with the cluster's default minimal resources.
  • We override default resources for the case of data deletion for table wmf_dumps.wikitext_inconsistent_rows_rc1.

Bug: T378122

Edited by Xcollazo

Merge request reports