How to roll out updates sensibly

What pitfalls come with updates.

On the one hand, everyone tells us: "upgrade to the latest version," because it fixes bugs and vulnerabilities. But they forget to mention that along with this, new bugs and vulnerabilities may be introduced.

Of course, you've all already heard about the recent backdoor in the xz archiver. As a reminder, the story there was about how one of the developers spent years gaining the trust of the project's maintainer and eventually obtained the privileges needed to add a backdoor to the source code. But the watchful eye of Andres Freund spotted the malicious code while running benchmarks and alerted the community. In the end, versions 5.6.0 and 5.6.1 turned out to be vulnerable.

So updates need to be approached pragmatically; sometimes updates for the sake of updates bring nothing but new problems. It's a different matter if you know for sure that the new version fixes exactly what you need. Then, of course, go ahead and update.

Also keep in mind that, according to folk wisdom, updating on a Friday is a bad idea and bodes nothing good. Although... if you have no plans for the weekend...

Sometimes, by the way, we're simply forced to update, usually by cloud providers. They really don't want to maintain old stuff, so they shift the window of available versions to the right and, when the occasion arises, update the services themselves whether you want it or not.

And that, of course, drags functionality changes along with it, so we devops folks won't be left without work :)