Intermittent errors on ingestion jobs

[martyngigg]

Occasionally an ingestion job will fail with the error: "pyiceberg.exceptions.CommitFailedException: CatalogCommitConflicts: CatalogCommitConflicts => Requirement failed: Branch or tag main already exists". Rerunning the script fixes it most of the time but the pipelines need to be more robust.

Maybe write_to_table needs wrapping in a retry block?

Details:

File "/home/ubuntu/.cache/uv/environments-v2/accelerator-sharepoint-085545dc75ef208d/lib/python3.13/site-packages/pyiceberg/catalog/rest/response.py", line 111, in _handle_non_200_response
    raise exception(response) from exc
pyiceberg.exceptions.CommitFailedException: CatalogCommitConflicts: CatalogCommitConflicts => Requirement failed: Branch or tag `main` already exists

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/.cache/uv/environments-v2/accelerator-sharepoint-085545dc75ef208d/lib/python3.13/site-packages/dlt/common/destination/client.py", line 424, in run_managed
    self.run()
    ~~~~~~~~^^
  File "/home/ubuntu/.cache/uv/environments-v2/accelerator-sharepoint-085545dc75ef208d/lib/python3.13/site-packages/elt_common/dlt_destinations/pyiceberg/pyiceberg.py", line 64, in run
    self.iceberg_client.write_to_table(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self.load_table_name,
        ^^^^^^^^^^^^^^^^^^^^^
        pq.read_table(self._file_path),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        self._load_table.get("write_disposition"),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/ubuntu/.cache/uv/environments-v2/accelerator-sharepoint-085545dc75ef208d/lib/python3.13/site-packages/elt_common/dlt_destinations/pyiceberg/exceptions.py", line 33, in _wrap
    raise DestinationTerminalException(status_ex) from status_ex