[Themaintainers] why is maintenance important? ask Facebook

[Camille Acey]

"The outage knocked out tools that engineers would normally use to investigate and repair such outages, making the task even more difficult, Facebook said."

This is less about maintenance and more about redundancy as part of an overall disaster recovery plan. At my company we do drills and tabletop activities to ensure we have a plan in case of incidents and outages, and also to identify places where we have holes in the plan. I'm sure FB has the same.

Maintenance doesn't make everything perfect or eliminate human error, but a maintenance culture should be about constant improvement of systems. Tools like trainings, DR playbooks, and post-mortems are part of such a culture.

When it comes to their vast network of distributed systems, FB is far from the "move fast and break things" ethos that drove early success.  For all my criticisms of the company, I tip my Maintainers hat to them when it comes to building and maintaining resiient services across a massive network of data centers.
Camille Emefa Acey https://camilleacey.com

----

/You cannot buy the revolution. You cannot make the revolution. You can only be the revolution. It is in your spirit, or it is nowhere/.
- Ursula K. Le Guin

Oct 9, 2021 08:06:08 Jonathan Coopersmith <j-coopersmith at tamu.edu>:

> https://www.reuters.com/technology/facebook-says-maintenance-error-caused-mondays-6-hour-outage-2021-10-05/
> 
> Stay sane and keep wearing a mask (partially for you; more for others),
> 
> Jonathan
> 
> Jonathan Coopersmith
> Professor
> Department of History
> Texas A&M University
> College Station, TX  77843-4236
> 979.739.4708 (cell)
> 979.862.4314 (fax)
> 
> Voting challenges:  https://theeagle.com/opinion/columnists/texas-should-be-the-model-for-secure-voting/article_4c35a332-e42c-11eb-b0d8-534c59b58ebe.html
> 
> Preserving space archives:  https://www.toboldlypreserve.space/
> 
> International standards battles:  https://spectrum.ieee.org/tech-talk/geek-life/history/lets-thwart-this-terrible-idea-for-standards-setting[https://urldefense.com/v3/__https://spectrum.ieee.org/tech-talk/geek-life/history/lets-thwart-this-terrible-idea-for-standards-setting__;!!KwNVnqRv!R4dbJx9DOnSkoiNAwXjHq9sp5vMvdGEVtOMlWJUc6AH7iQiRA7bw1VEUYeo6i2St9sgf$]
> 
> /FAXED.  The Rise and Fall of the Fax Machine/ (Johns Hopkins University Press) 
> 
> 
> 
> 
> _______________________________________________
> Themaintainers mailing list
> Themaintainers at lists.stevens.edu
> https://lists.stevens.edu/mailman/listinfo/themaintainers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.stevens.edu/pipermail/themaintainers/attachments/20211009/5750105a/attachment.html>