OpenSSL, Syslog, and Unexpected Consequences of Usrmerge: Upgrading to bookworm

Rsyslogd, wmnet

Much more harmless than those, but immediately visible after the upgrade, was that my syslog display remained empty. The direct reason was that the rsyslog daemon was not running. The reason for that, in turn, was that there was not even a init script for it in /etc/init.d, let alone rc.d links to it. But the rsyslogd package was installed. What would the purpose be of installing a daemon package without an init script?

The Debian bug tracker had something like an answer: the maintainer took it out, presumably to shed files they considered cruft in the age of systemd. Although I have to concur with Naranyan's remark in the bug report that rsyslog will typically be in place exactly when systemd (with its own log daemon) is not, at least that bug (#1037039) offers the (simple) fix: Install the orphan-sysvinit-scripts package.

Something a bit harder to explain is that the nice wmnet applet for monitoring transfers on network interfaces came up blank after the upgrade. This is fixed by passing a -n option to it, which tells it to draw into a normal window rather than something suitable for the Windowmaker dock. Wmnet (as perhaps other Windowmaker applets, too) tries to guess where to draw based on some divination. Perhaps my window manager sawfish started to pretend it's Windowmaker in bookworm? Or indicated to wmnet in some other way it was living in a Windowmaker dock? Hm: Given that the last changelog entry on sawfish itself is from 2014 (which I consider a sign of quality), that seems unlikely, but then I can't bring myself to investigate more closely.

The usr Merge and Bashism in the Woodwork

Although I had merged the root and usr file systems on that box last time I migrated to a new machine, I had postponed doing the usrmerge thing (i.e., making the content of /bin and /usr/bin identical) on the box until the last possible moment – that is, the bookworm installation – because I had a hunch some hack I may have made 20 years ago would blow up spectacularly.

None did. Except… it turned out I had linked /bin/sh to /bin/bash for some long-forgotten and presumably silly reason; if you had asked me before the upgrade, I'd have confidently claimed that of course all my little glue scripts are executed by Debian's parsimonious dash rather than the relatively lavish bash. Turns out: they weren't.

With the installation of the usrmerge package during the bookworm dist-upgrade that is over. /bin/sh is now dash as I had expected it to be all the time. I have to admit I am a bit disappointed that I do not notice any difference in system snappiness at all.

But I did notice that plenty of my scripts were now failing because they contained a bashism: Comparison for string equality in POSIX-compliant [ ... ] constructs is not the C-like == but the SQL-like = even though bash accepts both. I don't know when I forgot this (or, for that matter, whether I ever knew it), but a dozen or so of my (often rather deeply embedded) shell scripts started to fail with messages like:

script name: 22: [: tonline: unexpected operator

So, repeat after me: In shell scripts, compare strings with = and numbers with -eq. And I have to admit that this experience made me a bit more sympathetic to the zero shell paradigm behind systemd. But for the record: I still think the advantages of having hooks for shell scripts almost everywhere overall outweigh these little annoyances.

The OpenSSL Upgrade

With the bookworm upgrade, a fair number of hashes and ciphers were declared “legacy” in openssl, which means that in the default configuration, it will reject them. That had a few rather disruptive consequences: For one, I needed to update a few snake-oil certificates I had generated for playing with https on my box.

Also, fetchmail failed for a POP server I had configured with a message like:

fetchmail: <hostname> SSL connection failed.
fetchmail: socket error while fetching from <whatever>

I was puzzled for a while until I realised that the recipe said:

with proto TLS1

That was probably valuable in, like, 2004, to suppress ancient (relatively) easily breakable SSL versions, but by now it didn't let fetchmail negotiate crypto that was still allowed by openssl. Removing the proto TLS1 fixed that problem.

The most unnerving breakage, however, was that my preferred disk crypto, encfs (cf. this advocacy in German), broke for some volumes I had created more than a decade ago: they failed to mount because openssl now refuses (I think) the blowfish cipher. I fiddled around a bit with re-enabling legacy algorithms as per Debian bug 1014193 but quickly lost my patience with the slightly flamboyant semantics of openssl.cnf. To my surprise, downgrading to encfs_1.9.5-1+b2_i386.deb from bullseye (by briefly re-adding the sources.list lines) let me mount the old volumes again. I then simply created new encfs volumes and rsync -av-ed from the old decrypted volume into the new decrypted volume. Finally, after unmounting everything encfs, I overwrote the old encrypted volumes with the new encrypted volumes and upgraded back to bookworm encfs.

Since I can't explain why downgrading encfs would have fixed the problem as I've analysed it and hence suspect that a part of my analysis (and fix) is wrong, I'd strongly recommend to run:

encfsctl info <encrypted volume>

on each encfs directory you have before the upgrade. If it says something like:

Filesystem cipher: "ssl/blowfish", version 2:1:1 (using 3:0:2)

or even just:

Version 5 configuration; created by EncFS 1.2.5 (revision 20040813)

(where I have not researched the version where encfs defaults became acceptable for bookworm openssl; 1.9 is ok, at any rate), copy over the decrypted content into a newly created encfs container; it's quick and easy.

Relatedly, bookworm ssh also disallows a few crypto methods now deemed insecure by default, in particular SHA-1 hashes for host keys. Now, I have to connect to a few hosts I cannot upgrade (either because I'm not root or because they are stuck on some ancient kernel because of proprietary kernel components). For these, when trying to connect I now get messages like this:

Unable to negotiate with 192.168.20.21 port 22: no matching host key type found. Their offer: ssh-rsa,ssh-dss

You could reasonably argue I should discard boxes of that type. On the other hand, nobody will spend 50'000 Euro to eavesdrop on my communications with these machines[1] – that's the current estimate for producing a hash collision for an ssh host key, which this is about. Hence, I'm happy to risk man-in-the-middle attacks for these machines.

To deal with such situations, openssh lets you selectively re-allow SHA-1 hashes on RSA host keys. Helpfully, /usr/share/doc/openssh-client/NEWS.Debian.gz gives a recipe to save those hosts; put host stanzas like:

Host ancient-and-unupdatable.some.domain
  HostKeyAlgorithms=+ssh-rsa
  PubkeyAcceptedKeyTypes +ssh-rsa

into ~/.ssh/config (and do read ssh_config (5) if you are not sure what I'm talking about, regardless of whether or not you have this particular problem). Incidentally, just to save that one machine where you forgot to update your ancient DSA public key, you can for a brief moment change the second line to:

PubkeyAcceptedKeyTypes +ssh-rsa,ssh-dsa

If you don't have an RSA key yet, create one (ssh-genkey -t rsa) – RSA keys work even on the most venerable openssh installations that don't yet know about the cool ed25519 keys. Connect to the server, install the RSA public key, and re-remove the ssh-dsa part in the config again.

Kudos to the openssh maintainers for keeping compatibility even in crypto over more than 20 years. And shame on many others – including me – who don't manage to do that even in non-crypto software.

Terrible Font Rendering in Swing

One of the more unexpected breakages after the upgrade was that some Java Swing (a once-popular GUI toolkit) applications suddenly had terribly jagged fonts, such as my beloved TOPCAT:

I cannot exactly say why this looks so terrible[2]. Perhaps in the age of 300 dpi displays font hinting – which is supposed to avoid overly jagged pixelisation when rendering vector fonts at low resolutions – has become out of fashion, perhaps OpenJDK now disables font hinting by default, perhaps it's still something else, but this clearly cannot stand. Surprisingly, the OpenStreetMap editor josm, also built on top of swing for all I can tell, did not suffer from the jagged rendering, but some other Swing clients did, too.

After quickly skimming the changelogs and tentatively trying older JVMs I operated a search engine and found an undated item on some slightly suspicious… um… support forum? They don't explain what broke either, but their suggested fix does indeed bring me back readable fonts. In my words, simply add:

export JDK_JAVA_OPTIONS="-Dawt.useSystemAAFontSettings=lcd"

to your ~/.bashrc or use some other (possibly adapted) means of getting that variable into the Java runtime's environment or the option into the java command line. I expect you could also edit /etc/java-17-openjdk/swing.properties and just add the key/value pair; but I didn't try because the environment variable won't break with the next JDK version (or so I hope).

If you have a plausible hypothesis on the actual genesis of the problem, please write in; I'm mildly curious.

Odds and Ends

In line with the General Encrapification of computing, I am chagrined to have to report that the bookworm Webkit now has a javascript PDF renderer built in. That is highly annoying for any number of reasons, including that PDFs are now broken on sites to which I deny Javascript execution (i.e., almost all of them). I have poked around in Webkit's source code, but it seems this classical crap feature (what business does an HTML renderer have to render paged media like PDF?) can only be disabled at build time at the moment. As of now, I don't have a solution for this problem; but I'll certainly experiment on what's there and otherwise hope for an upstream fix. And no, I won't touch the Webkit source code myself. My inability to meaningfully do that is the most important reason I keep advocating a netsurf-browsable web

With some apprehension I learned that the ISC DHCP client is slated to disappear from bookworm+1. You know, I estimate that that program has fetched some 50'000 IP addresses for my machine (and its predecessors, obviously) in the past 30 years, and so that's no trivial matter. I have tried to immediately replace it with busybox's DHCP client (the udhcpc package in Debian). However, that failed already when the association with an eduroam network took longer than udhcpc's timeout. Hence, I'm saving this migration for another day. Mainly as a note to myself and because the Debian docs in the package don't seem to mention it, let me say that you can pass extra flags from ifupdown to udhcpc via udhcpc_opts lines in interfaces (5) configuration.

What's unpleasant is that mumble now eats 20% CPU or so when I'm speaking and something like 5% even when idle. I suspect there is some weird little setting that would bring that down, but I have not found it yet. I suspect that because in bullseye, mumble caused load like that (even when idle) when you checked RNNoise in the Audio Processing section of the Audio Input settings – and it was checked by default. If you can help me tame mumble's CPU hunger before I dig any deeper: please write in.

Finally, bullseye is the last release in which the the GPE calendar from some long-forgotten Debian release was installable – I think it was a libpng update that finally caused the conflict that has flushed the (“obsolete”, for a long time) GPE packages from my system. This is a bit of a pity (e.g., I could sync this with the equally ancient stuff on my Nokia N900), but then it was inevitable given that what once started as a nice “Palmtop” (does anyone even remember that term?) PIM enviroment has been utterly dead for a decade or so now. Ah well.

As is my standard reaction now when some GUI tool I use(d) disappears, I have migrated to something at least optionally text-based for keeping my appointments: remind. So far, I think I can whole-heartly recommend that, and I'll happily write something up about how I'm using it. In the unlikely case anyone else still needs to migrate away from gpe-calendar, I have published a little script exporting the most common sorts of events from its sqlite-based database to remind-style text files: gpe-calendar-to-remind.

And that's it; I have been on bookworm for about two weeks now, and hence I expect I should have seen all the glaring issues by now.

[1]	I'm undecided whether or not I resent my nagging suspicion that nobody would spend $50'000 on eavesdropping other communication I engage in, but at least there I'm not sure.

[2]	But looking at this gave me some respect for both the designers of bitmap fonts and the designers of font hinting schemes.