- The first mirror threw a disk;
- For some reason, gmirror became unhappy, rather than running on the second mirror (I'm guessing the controller went unhappy; there wasn't anything logged to indicate the other disk in the mirror set was failing);
- The second mirror started taking load;
- For some weird reason, the second mirror hung hard without any logging to explain why.
The CDN ended up still serving content whilst the masters were down - they just couldn't download uncached content. So it wasn't a -total- loss.
This just highlights that I really do require another mirror master or two located elsewhere. :)