Bouregreg Mutuelle: the website, the partner VPNs and the phones have been failing “every other time” since last night

Bouregreg Mutuelle (supplementary health insurance, 140,000 members) hosts everything in the server room of its head office in Rabat. The Internet exit is doubled:

• ISP-A (“FAI-A” in the internal documents, 1 Gb/s fibre, public block 198.51.100.0/29) behind the fw-a firewall: it is through it that the members' portal www.bouregreg-mutuelle.ma (DNAT to the DMZ), the PBX's SIP trunk and the partners' IPsec tunnels (contracted clinics, third-party payment) are published;
• ISP-B (“FAI-B”, 500 Mb/s fibre, block 203.0.113.0/29) behind the fw-b firewall: it is used for web browsing by the 260 workstations (SNAT).

Each firewall (Debian 12, nftables) advertises a default route over eBGP to the core router cr-coeur (FRR), which routes the DMZ 10.20.30.0/24 and the servers 10.20.40.0/24.

Last night at 22:40, change CHG-2291 “use both links in active/active” was applied on the core router. Since this morning: the members' portal “loads every other time”, the partner clinic in Salé can no longer send its coverage requests (“timeout” at random), and the call centre reports calls with no sound in one direction. IT management wants a solution before the branches open at 09:30.

You are on the on-call workstation; you have SSH access to cr-coeur, fw-a, fw-b and srv-web-01. In ~/INC-3516/: the external probe, the network diagram and the change log.