Skip to content
Rapcsány Krisztián Rapcsány Krisztián
hu
← All articles
2 min read

Sending webhooks with retries, without sending twice

Laravel Queues API

At the end of video processing, my media service sends a callback (a webhook) to the operating system. That is harder than it looks for two reasons: the receiving side may be unavailable, and the processing steps finish independently, so the same finished state can occur more than once.

Retries with growing delays

The sending runs on a queue, with at most five attempts, waiting 10 seconds, 30 seconds, 1 minute, then 5 minutes between them. If the response is not successful, I throw an exception, so the queue knows it has to retry.

Do not send it twice

The thumbnail, the stream and the MP4 are built separately, and each of them can trigger the sending. So before sending I read the row under a lock inside a transaction, and record that the sending has happened. If it is already recorded, the second job quietly exits.

SendVideoWebhookJob.php
class SendVideoWebhookJob implements ShouldQueue
{
    use Queueable;

    public $tries = 5;
    public $backoff = [10, 30, 60, 300];   // seconds to wait between attempts
    public $timeout = 30;

    public function handle(): void
    {
        // Several jobs can reach the "finished" state independently:
        // only the first may send, so claim the flag under a row lock.
        $shouldSend = DB::transaction(function () use ($video) {
            $locked = Video::whereKey($video->id)->lockForUpdate()->first();

            if (! $locked || $locked->webhook_sent_at !== null) {
                return false;
            }

            $locked->update(['webhook_sent_at' => now()]);

            return true;
        });

        if (! $shouldSend) {
            return;
        }

        try {
            $response = Http::timeout(10)->post($callbackUrl, $payload);

            if ($response->failed()) {
                throw new \RuntimeException("Webhook endpoint responded with HTTP {$response->status()}");
            }
        } catch (\Throwable $e) {
            // Release the flag, otherwise the retry would think it was already sent.
            $video->update(['webhook_sent_at' => null]);

            throw $e;
        }
    }
}

An excerpt of real code, anonymised.

When sending fails

After a failed send I reset the flag and rethrow the exception. If I leave that out, the retry would think the sending had already happened, and would send nothing. It is easy to forget, because on the happy path everything works.

Working on a similar problem?

If you are stuck, or want a second look at your solution, write to me. I am happy to look.

Write to me →
Get a quote →