Deed archiving: document uploads hang for 60 s and then fail, while the storage is “69% full”

Wathiqa Archives (a fictional company, Fez) digitises and stores the deeds of 420 notary offices. The documents are uploaded by the document management application ged-app-01 into S3 object storage served by the RADOS Gateway rgw-wa-01, on top of a Ceph cluster of 12 OSDs on 4 servers (×3 replication).

On Friday, a disk in osd-wa-04 was replaced (osd.11); the operation was declared complete on Monday. Since 08:40 this morning, every upload hangs for 60 seconds and then fails; browsing works. The dashboard shows 69% of space used. The application team blames the gateway, the storage team has seen “slow ops on osd.5” and suggests restarting osd.5.

You are a level-3 storage engineer on bastion-wa (root SSH; the ceph CLI is available on mon-wa-01). Mission: make writes possible again without any risk to the data (no deed may be lost).