Customers receive the same invoice three or four times, and sending is hours behind

Marsa Colis, an express delivery service based in Tangier, publishes every validated order to the Pub/Sub topic commandes-validees. The worker factures-worker (GKE, 3 pods) consumes the pull subscription sub-factures, generates the invoice, emails it and then acknowledges the message.

Last night, version 2.4.0 of the worker added a payment QR code and a signed timestamp to every invoice. Since then, customer service has received dozens of calls: “I received my invoice four times”. The invoices for this morning's orders, on the other hand, have still not gone out.

The product team suspects the website of publishing orders twice and suggests “running 20 pods”. You are on your workstation with gcloud (project marsa-colis-prod-3307, number 260419738512) and kubectl on the cluster gke-marsa-ew1, namespace facturation.