Nonceworks
MenuClose

Relayer: a nonce too high jam that never heals

Written 6 October 2026

Applies to OpenZeppelin Relayer 1.7.0 and 1.8.0 on EVM networks that reject out-of-order nonces at submission (Arbitrum and other strict-nonce rollups). Upstream issue: openzeppelin-relayer #843, Fhenix's follow-up to #818. Reproduction with compose file, proxy, driver and logs from two runs: zachsplat/relayer-843-repro.

Symptom

The relayer's nonce counter runs ahead of the chain and the gap only widens. Records pile up in sent with sent_at: null and status_reason: "Nonce too high (attempt N)". The log repeats:

stuck in Sent, queuing resubmit job with repricing
executing targeted nonce health action
no nonce gaps detected
nonce health action completed ... gaps_filled=0

and the two lines you are waiting for, rewound nonce counter over verified-empty region and nonce gaps confirmed below tx, never come. Throughput falls to whatever the blind resubmits happen to land, regardless of what the chain could take.

Cause

Three things combine, all visible in the 1.8.0 source:

  1. prepare_transaction allocates the nonce, signs, and marks the record Sent before the broadcast, which runs as a separate job. sent_at is only written after a successful eth_sendRawTransaction.
  2. The submit job gets no retries of its own (WORKER_TRANSACTION_SUBMIT_RETRIES is 0 in src/constants/worker.rs). A single transport error, a timeout or a 503 from the provider, ends it: max attempts reached, failing job job_type=Transaction Submission ... max_attempts=0. The record stays Sent, sent_at stays null, and the nonce slot stays occupied by a payload the chain has never seen.
  3. On a strict-nonce chain every later transaction is rejected with nonce too high and deliberately left in Sent (handle_nonce_too_high tracks the attempt and returns). The producer keeps allocating, so the region between the chain nonce and the counter fills with Sent records and stays full.

The 1.7.0 recovery machinery then has nothing to grip. try_rewind_counter (src/domain/relayer/evm/nonce.rs) bounds the rewind by the highest slot holding a record of any status, computes a target equal to the observed counter, and returns without logging. detect_nonce_gaps and detect_nonce_gap_ahead both treat Sent as an active status (is_active_nonce_status in src/domain/transaction/common.rs), so they report no gaps. The only way a slot clears is a blind resubmit of the record that happens to run while the chain sits at exactly that nonce.

Reproduction

On 2026-10-06 we ran the unmodified 1.8.0 image against anvil (a block every 2 s) through a proxy that rejects any raw transaction whose nonce is above the sender's pending nonce with nonce too high, and returns HTTP 503 to eth_sendRawTransaction during a controlled window. One provider, no failover, a producer submitting at a fixed rate.

  • 12 transactions a minute, 90 s window: the 18 transactions prepared during the window were abandoned on their first error and left as sent with null sent_at; everything after them was parked Nonce too high (attempt N); the health job logged no nonce gaps detected on every run; rewound nonce counter never appeared. The drift held at 18 for fifteen minutes, because at that rate the resubmits landed slots as fast as the producer added them, and cleared within two minutes of the producer stopping.
  • 30 transactions a minute, 60 s window: same mechanism with intake above the resubmit-governed drain. Drift 0, 10, 25, 33 during and after the window, flat for three minutes, then 39, 40, 45, 45, 45, 46, 51, 51, 57, 58, 58, 60 over the next six minutes, never decreasing. 69 sent records with null sent_at at the end, 1,807 nonce too high rejections, 1,797 resubmit jobs queued, 215 no nonce gaps detected, zero rewinds.
Nonces the relayer handed out that the chain had not accepted, run 2 of relayer-843-repro t+30 s: 0; t+60 s: 10; t+90 s: 25; t+120 s: 33; t+150 s: 33; t+180 s: 33; t+210 s: 33; t+240 s: 33; t+270 s: 33; t+300 s: 33; t+330 s: 33; t+360 s: 39; t+390 s: 40; t+420 s: 45; t+450 s: 45; t+480 s: 45; t+510 s: 46; t+540 s: 51; t+570 s: 51; t+600 s: 57; t+630 s: 58; t+660 s: 58; t+690 s: 60; t+724 s: 69 RPC errors 0 6 min 12 min 30 60 33 69
Run 2, sampled every 30 seconds. Each column is how many nonces the relayer had handed out that the chain had not yet accepted, one slot for every three. The gap held at 33, then climbed; 69 transactions were stuck as Sent when the run ended.

The second series is the one-way trapdoor the issue describes. The reporter's production numbers were a drift of 263 growing by about nine a minute on Arbitrum Sepolia with a continuously flaky provider.

The relayer log from run 2 at the moment the first submit job was abandoned, and the health job's verdict two minutes later. The lines are shortened: task and request ids are removed, the span names are abbreviated to submit and health, and long lines are cut. The full lines are in logs/relayer-run2-excerpt.log in the repository.

2026-10-06T19:36:28.677090Z  WARN submit: rpc call failed (will retry if retriable) operation_name=send_raw_transaction provider_url=http://proxy:8546 attempt=3 max_re...
2026-10-06T19:36:28.677124Z  WARN submit: all retries exhausted max_retries=3 operation_name=send_raw_transaction provider_url=http://proxy:8546 error=Transport error:...
2026-10-06T19:36:28.677138Z  WARN submit: all retry attempts failed, marking as failed and switching to next provider max_retries=3 provider_url=http://proxy:8546 oper...
2026-10-06T19:36:28.677155Z  INFO submit: Marking current provider as failed attempt=1 job_id=95f7d9ca-a09f-4a98-89f1-eead7bb61805 job_type=TransactionSend attempt=1 t...
2026-10-06T19:36:28.677209Z ERROR submit: rpc call failed after attempts across providers operation_name=send_raw_transaction total_attempts=3 failover_count=1 error=T...
2026-10-06T19:36:28.677290Z  WARN submit: request failed job_type=Transaction Submission error=Underlying provider error: Transport error: HTTP error 503 with body: pr...
2026-10-06T19:36:28.677315Z ERROR submit: max attempts reached, failing job job_type=Transaction Submission max_attempts=0 attempt=1 job_id=95f7d9ca-a09f-4a98-89f1-eea...
2026-10-06T19:36:28.677365Z ERROR task: AbortError: Failed to handle request done_in=671 ms attempt=1
2026-10-06T19:36:28.677406Z ERROR worker encountered an error worker_id=transaction_sender error=AbortError: Failed to handle request
2026-10-06T19:38:19.212056Z DEBUG health: no nonce gaps detected attempt=1
2026-10-06T19:38:19.212089Z  INFO health: nonce health action completed relayer_id=anvil-strict gaps_filled=0 attempt=1

Workarounds on 1.8.0

  • Give the relayer more than one rpc_urls entry with failover. Since submit jobs retry nothing themselves, provider-level failover is the only retry a broadcast gets; with one provider, every transport error is a stuck slot.
  • Alert on max attempts reached, failing job job_type=Transaction Submission, and on sent records whose sent_at is null for more than a minute. Both appear well before the drift becomes visible as throughput.
  • Keep the producer's rate below what the relayer can resubmit through. The jam only grows when intake exceeds the drain, which is governed by the status-check cadence rather than the chain.
  • DELETE /api/v1/relayers/{id}/transactions/pending and RESET_STORAGE_ON_START exist as manual resets; neither was tried in the runs above, so how they behave on a jammed relayer is unknown.

The directions proposed in the issue (treat a Sent record with null sent_at as non-consuming in the rewind, log the silent early return, age out long-rejected Sent records) are consistent with everything above. There is no maintainer reply on the issue as of this writing.