Automating security updates 2724

On Fri, 13 Oct 2006, in the Usenet newsgroup comp.os.linux.misc, in article

That's a good example of a "That depends" type of question. How critical is-are the systems that are being updated. What are they doing? How good are your backups? What is your security model? What is the system usage like?

We're a bit on the paranoid side, and are nearly all rpm based - in this situation, the difference between the Debian and RPM package manager really isn't all that important. For us, an update service looks roughly like this:

1. A cron-job checks several distribution servers, looking in the updates directory for the latest stuff (some FTP servers accept 'ls -lrt' in addition to the 'dir' command). Files that we don't have are downloaded to a quarantine area, and mail is sent to the updates administrator. Note that this download is the package source rather than a pre-compiled binary.

2. The updates admin audits the package, noting what it is, do we use this package, what changed, and so on. This tends to be easier than might appear because typically the update consists of a new patch added to the stuff that makes up the package (source tarball, distribution improvements, and any patches).

3. If things are OK, the updates admin builds the package into a binary, installs that on a test box, and checks one more time - any problems?

4. If no problems are found, the binaries are placed on update servers on the LAN. For "normal" systems, a nightly cron-job looks for anything in those servers, and if there is anything, installs it autonomously, then sends mail to an updates monitor account showing a snapshot of all packages installed on each box. That data gets into a database for the PHB to look at. The packages also go to the 'install' server, replacing the old packages (we install over the wire from a central repository, rather than use CDs or DVDs because very few of our systems have removable media drives - floppies, CDs, what-ever - for security reasons).

5. Some packages require a restart of the application. A few others (kernel, libraries, and the like) require a reboot. We try to schedule these updates for a weekend, when the maintenance crew can handle any disasters.

6. The following day, the updates server transfers the binary packages to an archive directory.

From 'comp.os.linux.announce' a few days ago:

Announcing the version 4.1 release of the "Advanced Bash Scripting Guide." This e-book tutorial and reference is the equivalent of a 700-page print book. With 322 illustrative examples (including such goodies as an anti-spammer script), the book covers virtually every aspect of scripting.

I suppose that wasn't what you were asking, but it is the correct answer. What works for use probably wouldn't be practical for many others. If you don't have an 'updates administrator', I'd probably make up a database of installed packages, and either break that into two (whitelist-blacklist for example), or stick a flag variable in the database. Another database (or database entry) would contain the latest installed version data, that you could compare to the distribution specific list of available updates. Depending on how many systems are involved, you might want to stagger the updates times so as to not overwhelm the network or updates server. Our cron job grabs the local IP address, and has a 'sleep' that is ten times the number of the last octet of the address - simple, effective, and yet allows a common script.

I'd suggest breaking the problem into two as we have - one part being the checking for and grabbing updates from the Debian updates servers, and separating the updates into the 'white-black' categories. This runs on your updates server. The second part runs on your clients, and this would check your 'white' updates directory nightly, installing anything that turns up in that directory. The client should also run a script that grabs a list (from your updates server) and compares what should be installed, verses what actually is. This catches the case where for one reason or another, the nightly cron update on the client either didn't run, or barfed. The script can send mail to someone if a mismatch is detected.

Old guy