MenuClose
Running a stack
How each stack is checked and upgraded, and the upstream issues tracked for it.
Checks each stack gets
These run from outside the stack. They are built and tested, and will be running before the first stack goes live; no stack is live yet.
Health
The containers, Redis, the relayer's health and readiness endpoints and its
/api/v1/relayersare checked from outside the stack. Monitor 1.6.0 has no health endpoint, so it is judged by the last block it processed: one that stops advancing while the chain moves on raises an alert, and so does a relayer nonce counter that drifts from the chain in either direction.Logs
Three log lines raise an alert:
bumped gas price does not meet minimum requirement(#808),nonce consumed externally, marking as Failed(#817), andmax attempts reached, failing job job_(#843). Each has a note that explains it and what reduces the exposure.type=Transaction Submission Config changes
Before a change to your Monitor files goes live, the control plane runs the Monitor's own
--checkon it and refuses the change if the check fails (judged by its output, since 1.6.0 exits 0 even on failure, monitor #501); then the Monitor alone restarts, since it has no hot reload. A change to the relayer goes through the same validation as the first upload and is applied only when no transaction is in flight: the relayer keeps its state in Redis and loads a changedconfig.jsonwith one reset start, which re-reads the nonce from the chain and replaces policies set through the API with the ones inconfig.json. A pause set through the API is lifted for that start and set again afterwards. The same rule covers a new Google Cloud or Turnkey credential and the webhook signing key. A relayer API key rotation restarts only the relayer.
Releases
A new tag is pulled into a staging copy of your configuration first. That is where stricter secret validation, network field changes, Redis schema moves and rolling-deploy hazards show up. The old replica stops before the new one starts, because overlapping replicas share Redis worker ids until a release carries #892 (merged 2026-09-30, after 1.8.0). Each upgrade gets a written note; the 1.4 to 1.8 notes are an example.
Upstream watch
Upstream issues that matter to anyone running these releases, and what reduces the exposure until a fix ships. Last checked 2026-10-07.
| Item | State | Exposure | Until the fix ships |
|---|---|---|---|
| relayer #843 nonce too high jam | open, reproduced | strict-nonce chains, single RPC | Two rpc_urls per relayer; alert on abandoned submit jobs and on sent records with null sent_at. |
| relayer #817 mined tx marked Failed | open, reproduced; #899 does not cover it | load-balanced RPC pools | One consistent endpoint per relayer; alert on the reconcile line; re-check the hash after a Failed. |
| relayer #808 capped tx never resent | open, reproduced | any relayer with gas_price_cap | Cap with headroom, or no cap; alert on the skip line. |
| relayer #912 sign endpoints outside the policies | opened 2026-10-07 | relayer addresses holding tokens, approvals or signature-based authority | Gas only on relayer addresses; on Turnkey, deny raw-payload signing when the sign endpoints are not used. |
| relayer #913 RPC URLs logged unmasked | opened 2026-10-07 | RPC provider keys inside the URLs end up in the relayer's logs at warn level | Per-team RPC keys with origin or IP limits; logs are not shared outside the stack; rotate RPC keys after an incident. |
| relayer #914 Google Cloud example grants the wrong role | opened 2026-10-07 | a service account set up from the example cannot read the public key | Grant roles/ on the key, as on the how it works page. |
| relayer #892 unique Redis worker ids | merged 2026-09-30, not yet released | rolling deploys with overlapping replicas | Stop the old replica before starting the new one. |
| relayer #757 Google Cloud workload identity | open | Google Cloud KMS signers | A service-account key scoped to one key, rotated on request. |
| relayer PR #899 record hash before broadcast | open | lost resubmission hashes | Staged against your config the day it lands in a release. |
monitor #501 --check exits 0 on failure | opened 2026-10-07 | scripts and CI that trust the exit code | Read the output for Configuration validation completed successfully!, as the control plane does. |
| monitor #498 Failure conditions never match on a busy chain | open; fix PR #499 open | any status condition | A gas_used clause in the expression forces the receipt. |
Leaving
Tell us on Telegram or by email. The how it works page lists what you take with you.
Pinned versions
sha256:9305f80c4e8c735dab52ab4aee5ccf0515adbd6e048ae9c1fc07eb79fc14bcdbRelayer 1.8.0
sha256:a0d8751b065bb269977a91292df8759bc48ffaaecd0a162bda1f96c32c899e77Redis bookworm
sha256:164c759a0c342ee69d08fc99219382b0fd682181465c0df2e0e6911f4c85d73cAlerts
Relayer nonce drift
The three log lines
A signing error in the relayer log (AWS:
AWS KMS sign failed; Google Cloud and Turnkey: Signer error), and transactions pending longer than expected. The relayer keeps accepting requests while KMS refuses, so its API alone does not show it.