---
title: "Retries and idempotency for AI agent tool calls (2026): retry policy, activity idempotency, idempotency keys, and at least once vs exactly once in durable execution — Cadenya"
url: "https://cadenya.com/other-content/retries-and-idempotency-for-agent-tool-calls"
description: "How to avoid running an AI agent's side effect twice in 2026: retry policy and backoff defaults, at least once vs exactly once wording, idempotency keys, and timeouts in Temporal activities, Restate, Inngest steps, DBOS steps, Trigger.dev, BullMQ, Celery, and Sidekiq, and Cadenya, which asks for approval before side effects and keeps a durable record of every tool call."
---

Every retry engine runs a tool call _at least once_, so the only reliable way to avoid a side effect running twice is an idempotency key that the receiving API checks. Temporal, Restate, Inngest, DBOS, and Trigger.dev all retry failed steps. Temporal and Trigger.dev say in their docs that a retried step can run its side effect again, and DBOS calls its steps “at least once.”

So pick Temporal when you want activity retries with no attempt limit by default, Restate or DBOS when you want each finished step stored and skipped on replay, Inngest or Trigger.dev when you want a dedup key on the trigger built in, and a plain queue when you already run one. And if you’d rather run none of them, Cadenya is a hosted agent loop with approvals, duplicate protection, and a durable event log, with nothing to host. [What it offers](#what-does-cadenya-offer-that-durable-execution-engines-and-job-queues-dont) is at the end.

Every price and limit below comes from each vendor’s own pages, checked on October 8, 2026.

## Retries and idempotency side by side

|  | Temporal | Restate | Inngest | DBOS | Trigger.dev | BullMQ, Celery, Sidekiq |
| --- | --- | --- | --- | --- | --- | --- |
| What retries | An Activity | A ctx.run block or the whole invocation | Each step.run(), with its own counter | A step, once you turn retries on | The task’s run(), from the top | The job or task |
| Default attempts | Unlimited | 70, then the invocation pauses | 5 (the first try plus 4 retries) | Retries off. With them on, 3 attempts | You set it in trigger.config or on the task | BullMQ: none until attempts is above 1. Celery: 3 when you opt in. Sidekiq: 25 |
| Default backoff | 1 second, times 2, up to 100 seconds | 50 ms, times 2, up to 60 seconds | Exponential, with jitter | 1 second, times 2 | Exponential, with the factor you set | BullMQ retries at once unless you set a backoff. Sidekiq: about 20 days in total |
| How the docs word the guarantee | ”Observed as completed exactly once”, but “may be executed multiple times” | An invocation “runs exactly once regardless of failures” | A finished step “will not re-run” | Steps “at least once”, transactions “exactly once” | A key stops a task being “triggered twice”, not its code “running twice” | Sidekiq: “at least once, not exactly once” |
| Built-in dedup key | None. Use the Workflow Run ID plus the Activity ID | Idempotency-Key header, 24 hours | Event id or an idempotency expression, 24 hours | The workflow ID | idempotencyKeys, kept 30 days | BullMQ: a custom jobId |

## At least once vs exactly once: what each vendor promises

Read the exact wording. It’s the difference between “the engine records it once” and “your API sees it once.”

Temporal says that for an activity with retries, it “guarantees that the Activity will be observed as completed exactly once.” The next sentence is the one that matters: “the Activity may be executed multiple times and may even partially complete more than once.” The docs name the case. The activity finishes, then “the Worker crashes just before it notifies the Temporal Service,” so the activity runs again.

Restate says each invocation “runs exactly once regardless of failures,” and a crash replays the journal, “skipping completed steps.” But a `ctx.run` block that throws is retried, so a side effect inside it can run again.

DBOS states both halves plainly. “Steps are tried at least once but are never re-executed after they complete.” Database transactions “commit exactly once.”

Inngest saves each step’s result: “When `step.run()` finishes successfully, the response is saved in the function run state and the step will not re-run.” A step that throws halfway through still runs again.

Sidekiq is the bluntest: “Sidekiq will execute your job **at least** once, not **exactly** once.” Celery acknowledges a message before it runs by default, so a started task “is never executed again.” Turn on `acks_late` and the task “may be executed multiple times should the worker crash in the middle of execution.”

**So “exactly once” in a durable engine means the recorded result, not the HTTP call your tool made.**

## Retry policy and backoff: what gets retried, the model call or the tool call

Every option here retries a unit of code, not a model decision. What’s inside the unit decides what repeats.

Temporal tells you to put “failure-prone or non-deterministic operations (API calls, LLM invocations, etc.)” in activities, which retry by default with no attempt limit. Workflows don’t retry by default. Keep each side effect in its own activity: if one activity does three steps and the third fails, “the entire Activity” runs again.

Restate’s AI guide shows two layers stacking. The Vercel AI SDK retries a model call `maxRetries` times (2 by default), then Restate “retries the invocation,” and each Restate retry “goes through `maxRetries` Vercel AI SDK attempts again.” For tool errors, the AI SDK by default turns the error into “a message to the LLM, and the agent decides how to proceed.” A `TerminalError` stops Restate’s retries for good.

Restate also lets you set a retry policy per block:

```
const myRunRetryPolicy = {
  initialRetryInterval: { milliseconds: 500 },
  retryIntervalFactor: 2,
  maxRetryInterval: { seconds: 1 },
  maxRetryAttempts: 5,
  maxRetryDuration: { seconds: 1 },
};
await ctx.run("write", () => writeToOtherSystem(), myRunRetryPolicy);
```

Inngest gives every `step.run()` “its own independent retry counter,” so a model call and a tool call in separate steps retry on their own. `NonRetriableError` skips the rest.

Trigger.dev retries the whole run, and “`run()` executes again from the top.” A model call and a refund in the same task both repeat. `AbortTaskRunError` turns retries off for that attempt.

DBOS steps don’t retry until you set `retries_allowed=True`. Then a step gets 3 attempts, 1 second apart, doubling.

## Idempotency keys and activity idempotency for external side effects

A retry engine can’t stop an API from acting twice. Temporal says it directly: idempotency keys “are enforced by the service you are calling from your Activity, not by the Activity itself.” Trigger.dev agrees: “Only the provider can deduplicate its own API call.”

So the recipe is the same everywhere. Derive a key that stays the same across every attempt, and pass it to the API that does the side effect.

Temporal recommends “a combination of the Workflow Run ID and the Activity ID,” since it’s “consistent across retry attempts but unique among Workflow Executions.” Trigger.dev uses the run ID, because “retries are attempts of the same run, so the ID does not change between them.” Its own example passes it straight to Stripe:

```
import { idempotencyKeys, task } from "@trigger.dev/sdk";
import Stripe from "stripe";
import { sendRefundEmail } from "./send-refund-email";

const stripe = new Stripe(process.env.STRIPE_SECRET_KEY!);

export const refundOrder = task({
  id: "refund-order",
  retry: { maxAttempts: 3 },
  run: async (payload: { orderId: string; paymentIntentId: string; amount: number }, { ctx }) => {
    // The run ID is stable across retries, so Stripe returns the original refund instead of creating a second one
    await stripe.refunds.create(
      { payment_intent: payload.paymentIntentId, amount: payload.amount },
      { idempotencyKey: `refund-${ctx.run.id}` }
    );

    // The Trigger.dev key is run-scoped by default, so a retry does not create a second email run
    const idempotencyKey = await idempotencyKeys.create("refund-email");
    await sendRefundEmail.trigger({ orderId: payload.orderId }, { idempotencyKey });
  },
});
```

Restate gives you `ctx.rand.uuidv4()`, a UUID seeded by the invocation ID that returns “the same result on retries.” Its HTTP ingress also takes an `Idempotency-Key` header, and a repeat “within 24 hours” gets the same response without running again.

The other keys stop a whole run from starting twice, which is a different problem. An Inngest event `id` dedupes “over a 24 hour period.” In DBOS, a workflow ID “acts as an idempotency key,” and the docs name the use case: “making a payment or sending an email.” BullMQ drops a job whose `jobId` already exists: it’s “ignored and not added to the queue at all.”

**The run-level key stops a duplicate trigger. Only the API’s own key stops a duplicate charge.** You need both for a tool that moves money.

## Timeouts: when a stuck tool call counts as failed

A retry only starts once the engine decides the attempt failed. A hung HTTP call needs a timeout, or nothing retries at all.

Temporal says “We strongly recommend setting a Start-To-Close Timeout,” because the server “doesn’t detect failures when a Worker loses communication with the Server or crashes.” It relies on that timeout “to force Activity retries.”

DBOS workflow timeouts are “durable: they are stored in the database and persist across restarts.”

## Plain queues: BullMQ, Celery, and Sidekiq with idempotency keys

A queue works when your tool calls are short and you already run Redis or a broker. You bring the agent loop, the journal, and the keys.

BullMQ doesn’t retry until you set `attempts` above 1, and without a backoff, failed jobs are “retried without delay as soon as they fail.” Celery’s `max_retries` defaults to 3, but it “only applies if the task calls self.retry or if the task is decorated with the autoretry\_for argument.” Sidekiq runs “25 retries over approximately 20 days.”

None of them stores a step result. A job that calls the model, then a tool, then the model again runs all three on retry, so each side effect needs its own key in the API it calls.

## What does Cadenya offer that durable execution engines and job queues don’t?

Temporal, Restate, Inngest, DBOS, Trigger.dev, and the queues retry your code, and your code still has to be the agent and carry every idempotency key. **Cadenya runs the agent loop for you, with approvals before side effects, duplicate protection, and a durable record of every tool call, and nothing to host.** Here’s what that covers:

1.  **Nothing else to run.** No workers, no broker, no Redis, and no journal database. The loop runs on Cadenya, so your deploys and restarts never touch a running objective.
2.  **The agent loop, already written.** Cadenya calls the model, runs the tools, and hands each result back to the model. On the engines above, that’s the code you’d split into activities or steps and key by hand.
3.  **A failed tool call goes back to the agent.** When an HTTP endpoint returns an error, an MCP server rejects the call, or a timeout trips, Cadenya records a `tool_error` event. The objective keeps going: “The agent sees the error text and decides how to proceed.”
4.  **A human says yes before the side effect runs.** Mark a tool as needing approval and Cadenya parks the loop on `toolApprovalRequested`. The `tool_called` event fires only after the approval, so a refund waits for a person, and a denial can carry a memo that steers the agent.
5.  **One objective per record, not two.** Set an External ID when you create an objective. The docs put it plainly: “Use an External ID to prevent duplicate Objectives for a workflow or record in your application.”
6.  **Webhooks with the idempotency key built in.** Every delivery follows Standard Webhooks, signed with an HMAC-SHA256 of the ID, timestamp, and body. The `webhook-id` header is “Unique per delivery. Key your idempotency on it.” Cadenya records each attempt with its status and attempt count.
7.  **A durable event log of every tool call.** “Every Objective carries a durable event log of its agent loop.” Each tool call gets a `toolcall_` ID, and its `toolCalled` event records the tool, configuration, and arguments. Look up a call and its result before you act on it again.
8.  **Timeouts you set on the tool.** A Bare tool set takes a `contentTimeout`: how long to wait for a result “before the tool call errors,” so the agent moves on.
9.  **SDKs for the versions you run today.** TypeScript on Node 18, Python 3.9, Ruby 3.1, and Go 1.22, all at 1.9.0 and Apache-2.0, plus a CLI and a plain HTTP API.
10.  **A price in one line.** Free for 1,000 loops a month, $49 a month for 50,000, then $0.0015 a loop. A loop is one LLM request.

[Webhooks](/docs/guides/the-basics/webhooks) shows the idempotent handler step by step, and the free plan needs no card.

Create an account at [app.cadenya.com/signup](https://app.cadenya.com/signup?utm_source=cadenya.com&utm_medium=article&utm_campaign=retries-and-idempotency-for-agent-tool-calls), then copy your API key from the [API keys page](https://app.cadenya.com/account/apiKeys).