Skip to content

Addition of retry paths that don't have it yet #1197

Description

@vaibhav-datazip

Problem

Retry coverage is uneven. Several network-touching calls are bare return err, so a transient failure surfaces as exit 1 without ever being retried.

Source side

Examples:

  • connector.Setup : protocol/sync.go:95. Opens the driver connection (connect + PingContext, SSH tunnel when configured). No retry wrapper.
  • GetStreamNames : drivers/abstract/abstract.go:81. Live catalog query. No retry wrapper.

Others in the same shape worth checking while in there: PreCDC (drivers/abstract/cdc.go:25) and GetOrSplitChunks (drivers/abstract/backfill.go:22).

Similar retry needs to be checked in destination side for parquet and iceberg destination and handled there as well.

Retry defaults

Minor, but related: postgres, oracle and db2 don't set constants.DefaultRetryCount in their config Validate(), so an unset retry_count gives a single attempt. mysql, mssql, kafka, mongodb and s3 all set it - see drivers/mysql/internal/config.go:122 for the pattern.

Reference

Existing helpers:

  • utils/utils.go:481 : RetryOnBackoff (exponential; short-circuits on constants.ErrNonRetryable)
  • utils/utils.go:452 : RetryWithSkip (linear, caller-supplied predicate)
  • utils/concurrent.go:142 :AddWithRetry

Call sites already doing it right: drivers/abstract/cdc.go:87, drivers/abstract/backfill.go:97, drivers/abstract/incremental.go:63.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    good second issueafter you have solved good first issueintermediateintermediate level issue for contributors with some understanding of the codebase

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions