Skip to content

Fix TO_TIMESTAMP() format parameters.

Xcollazo requested to merge fix-emit-dt-issue into main

This MR fixes and issue on the TO_TIMESTAMP() format parameter. We were sending in yyyyMMddkkmmss, while the correct format is yyyyMMddHHmmss, with HH having a range of 00-23 for the hours, while the old kk had a range of 01-24. Whoops!

Fixing this resolves the failure we had seen when we deployed !46 (merged):

spark.sql("""
  SELECT
    revision_id,
    CAST(mw_revision_timestamp AS STRING) as mw_revision_timestamp_as_string,
    TO_TIMESTAMP(mw_revision_timestamp, 'yyyyMMddkkmmss') AS revision_timestamp_kk,
    TO_TIMESTAMP(mw_revision_timestamp, 'yyyyMMddHHmmss') AS revision_timestamp_HH
  FROM  flat_inconsistent_revision_ids_source
  WHERE revision_id = 68598618
""").show(100)

+-----------+-------------------------------+---------------------+---------------------+
|revision_id|mw_revision_timestamp_as_string|revision_timestamp_kk|revision_timestamp_HH|
+-----------+-------------------------------+---------------------+---------------------+
|   68598618|                 20241118003616|                 null|  2024-11-18 00:36:16|
+-----------+-------------------------------+---------------------+---------------------+

Bug: T368755

Edited by Xcollazo

Merge request reports