Delivery notes sent twice to carriers, and 1,284 messages in the dead-letter queue

Sahara Fret (a fictional freight company operating between Agadir, Laayoune and Dakhla) generates its delivery notes asynchronously: the booking application puts a message in the standard SQS queue bons-livraison; the ECS service worker-bl (4 Python tasks) reads the messages, generates the PDF note, sends it to the carrier, then deletes the message. After 3 failures, a message goes to the dead-letter queue bons-livraison-dlq.

Yesterday at 18:00, version 2.7 of the worker was released: notes are now electronically signed by the customs platform API before being sent.

This morning: some carriers complain about receiving the same note two or three times, others receive none, and the dlq-bons-livraison-non-vide alarm has gone into ALARM. The team suggests “purging the DLQ and doubling the workers”.

You are on your workstation with the AWS CLI v2 (region eu-west-1).