Description
Is your feature request related to a problem? Please describe.
We are using chains scheduled to run every couple of hours. We are occasionally encountering transient issues while processing a sql task (eg. timeout, deadlock, data yet not available for processing etc). With current implementation we either need to wait until scheduler triggers the chain again in a couple of hours or we need to schedule it more frequently and move some scheduling logic to code.
Describe the solution you'd like
The solution would be to include retry configuration and logic on both task and chain level.
Minimal configuration options would be - maximum_retry_count (smallint) and minimum_retry_timeout (interval).
Ideally, there would be two more options - retry_backoff_enabled(boolean), retry_backoff_steps and maximum_retry_duration (interval). Which would allow for more complex retry configurations with incrementally increasing retry timeouts up to maximum_retry_duration.
Describe alternatives you've considered
Hacking around those tasks and chains by running them on a more frequent schedule (* * * * *) and then resolving retry / scheduling logic based on custom config in db.
Creating a wrapper and invoking those sql functions as a program task.
While both alternatives will work, it would be far better to have this logic built-in in the scheduler itself.
Similar functionality exists in other schedulers we're using for other purposes -> airflow, sql server job agent.
Activity