Distributed Execution

DurableStack is designed for multiple workers competing for work across instances.

It distributes load by combining claim limits, jittered polling, and lease ownership.

Core model

Each worker loop does three things:

• materializes due recurring runs
• claims available due runs up to local capacity
• executes claimed runs with lease heartbeat protection

Local concurrency is bounded by:

• BatchSize / ClaimBatchSize
• MaxConcurrentRuns

Why jitter matters

Without jitter, workers poll at the same interval boundaries and create synchronized load spikes.

With PollJitterEnabled, each worker randomizes poll delay by a bounded ratio:

• delay factor = 1 +/- PollJitterRatio
• default ratio is 0.2 (up to +/-20%)

This naturally spreads claim attempts over time and reduces thundering herd behavior against the store.

Example

If PollIntervalSeconds = 5 and PollJitterRatio = 0.2, each poll is roughly between 4 and 6 seconds.

builder.Services.AddDurableStackPostgres(connectionString, options =>
{
    options.PollIntervalSeconds = 5;
    options.PollJitterEnabled = true;
    options.PollJitterRatio = 0.2;
});

How lease ownership prevents duplicate active execution

When a worker claims a run, the run is leased to that worker for LeaseDuration.

Only the lease owner should be executing that run at that time.

During execution:

• a background heartbeat extends lease ownership periodically
• extension cadence is half lease duration

If the worker crashes or stops heartbeating, lease expires and another worker can reclaim the run.

This model limits execution to one active lease owner per run under normal operation while enabling recovery after failures.

Single execution of each run, practically

The system targets single active execution per run through lease ownership.

As with any distributed lease model, edge cases (crash/network timing) can lead to re-attempts, so handlers must remain idempotent.

Think of the guarantee as:

• single active lease owner in normal operation
• durable recovery and retry when interruptions occur

Worker identity in distributed environments

Set a unique WorkerName per process/container.

DurableStack uses worker identity for lease ownership, diagnostics, and operational traceability.

var workerName = $"{Environment.MachineName}-{Environment.ProcessId}";

builder.Services.AddDurableStackPostgres(connectionString, options =>
{
    options.WorkerName = workerName;
});

Practical tuning guidance

• Start with moderate BatchSize and MaxConcurrentRuns.
• Enable poll jitter in multi-worker deployments.
• Set LeaseDuration above normal execution time for routine jobs.
• Monitor retries and lease extensions before increasing throughput aggressively.