Ridiculous unusable download URLs for open source projects

I told myself (and everyone I know) that I wouldn't write another blog post until I moved the blog off Google Blogger, but I can't stay silent on this issue.

UPower, the open source power management software used on Linux (and I believe the *BSD family), has recently changed their download URLs. As the lead of Adélie Linux, I personally maintain a significant chunk of "core" desktop experience packages. We consider UPower to be one of those, because it is important to conserve energy whenever possible.

Today I was notified by Repology that UPower was out of date in Adélie. No big deal, I'll just bump it:

>>> upower: Fetching https://upower.freedesktop.org/releases/upower-0.99.8.tar.xz curl: (22) The requested URL returned error: 404 Not Found

"Hmm", I wondered to myself, "maybe this is a git snapshot package someone uploaded". It turns out it wasn't; Debian, Arch, and Fedora are all shipping 0.99.8 now. What gives?

I looked at Debian's packaging first, since they typically have a good hold on stability. I didn't even understand the change, though, so I looked up Exherbo's packaging and was horrified.

Instead of a simple URL, they are now using a GitLab Upload URL which contains an SHA-1 hash in the URL. That means all of our bump scripts can't work any more. Instead of simply typing a single abump command, for every release of UPower I will now have to:

1. Open their GitLab instance in a Web browser, which isn't even installed on any of the staging computers to minimise security hazards:
2. Wait for all the JavaScript and miscellaneous crap to load;
3. Context-click the link for the UPower tarball;
4. Copy the link;
5. Connect to our staging system remotely from a computer with a Web browser installed;
6. Open vim on the APKBUILD file for UPower;
7. Paste the link into the source= line, replacing what is already there;
8. And then run abuild checksum manually to update the sha512sum in the file.

WHY!? fd.o people, please, out of respect for us packagers that want to give your software to the people who need it, please use your /releases/ directory again!

Fixing the laptop I broke

Sometimes things happen that you don't expect. It can be anything: a power failure during a system upgrade, or maybe a careless chmod 644 /usr/lib/libc.so — in my case, it was the latter (tab completion failure).

Training yourself on the proper way to respond to unexpected failures is the key to recovering them without pain or further data loss. When I had realised my gaffe, the first thing I thought was: How do I chmod it back without the ability to run chmod?

Static-linked rescue binaries are a must-have

The first thing I learned from this experience is that having a set of static-linked rescue binaries somewhere on your system can help in a lot of unexpected situations. We're going to be adding a busybox-static package to Adélie Linux just for such an occasion, and we may put it in the base system depending on community feedback. If I had a static busybox in, say, /var/recovery or such a path, this would have been a ten second fix rather than a few hour fix.

Embrace the system

After a few other attempts, I realised I could drop to assembler. Long ago, I spent my days writing assembler for system-level code. Since assembler is by design writing "below" C, you are not using the C runtime. Theoretically, you should be able to perform the same tasks as any utility on the system as long as there's a matching system call for it. And by luck, there is a single syscall: SYS_chown. Following is x86_64 assembler for Linux to chown /usr/lib/libc.so back to 755 (executable for all users):

_start: mov $90,%rax /* SYS_chown */ mov $str,%rdi /* const char *filename */ mov $493,%rsi /* mode_t mode */ syscall /* do it! */ mov $60,%rax /* SYS_exit */ syscall /* bye */ str: .ascii "/usr/lib/libc.so\0"

Then it was a matter of as -o fixit.o fixit.S; ld -o fixit fixit.o; strip fixit to generate a 440 byte binary file that would solve my issue. The next issue was transferring it to the laptop. I tried to use bash's /dev/tcp; unfortunately however, it does not support binary file transfer without something like `cat` or `dd`. Since I could only use the shell, I did what I had not done in over a decade: echo -n followed by the escape codes. Since a lot of the binary was still padding, I omitted the last 200 or so bytes. The output of the echo command needed to be redirected to a binary that was already executable (otherwise the file created would not have execute permission!), so I chose one I probably wouldn't need urgently: neon-config, a configuration utility for a library I installed for tinkering. The full shell transcript is in my misc Linux directory. This worked! And my laptop ran again...

As I said in the opening of this little musing: I could have made things a lot worse and lost all my open unsaved data by turning off the computer and trying to recover using media. Additionally, that computer is very picky about booting off external media, so that would have wasted even more time. Sometimes all you need is ingenuity and experience, and the only way to acquire either one is by messing about and poking at stuff! Happy hacking.

Configuring a more secure password hash for OpenLDAP

While working on the Galapagos infrastructure, we ran in to an interesting issue: using passwd(1) as an LDAP user would cause it to add another password instead of modifying it. Setting up the slapo-ppolicy(5) overlay then caused passwd(1) to then fail with:

password change failed: Password policy only allows one password value
passwd: Authentication token manipulation error
passwd: password unchanged

After consulting the #openldap channel on Freenode, the problem turned out to be that although OpenLDAP allows you to set olcPasswordHash on the root cn=config node, it does not work correctly when set there; it must be set under olcDatabase={-1}frontend,cn=config. Note, however, that olcPasswordCryptSaltFormat does belong in cn=config directly.

Configuring Apache 2.4 to serve GitLab over TLS / HTTPS

As part of my work assisting in the set up of the infrastructure for Galapagos Linux, I volunteered to install and configure GitLab. My colleagues had attempted to use the Debian Omnibus package, but that failed in spectacular ways, including references to directories in the configuration that did not exist after package installation.

The most important piece of advice I can give is that you absolutely must use Bundler v1.10.6 or older^[1] to ensure that you do not receive Gemfile.lock errors. You will also need to make a small modification to the Gemfile and Gemfile.lock file to ensure that libv8 is present if you wish to precompile the assets.

Now, for the Apache configuration. Note that I assume you have enabled https in GitLab's config/gitlab.yml and set port: 443. You will need to set a forwarding request header^[2] to ensure that GitLab does not throw CSRF authentication errors. Also, if you want to use the recommended Unix sockets of Unicorn, you will need to configure the ProxyPass and ProxyPassReverse to use unix:/path/to/socket|http://HOSTNAME (thanks, Xayto!) - the full VirtualHost for GitLab goes something like this:

<VirtualHost *:443>
        ServerName git.glpgs.io
        ServerAlias code.glpgs.io
        ProxyPass / unix:/home/git/gitlab/tmp/sockets/gitlab.socket|http://git.glpgs.io/
        ProxyPassReverse / unix:/home/git/gitlab/tmp/sockets/gitlab.socket|http://git.glpgs.io/
        SSLEngine on
        SSLCertificateFile /path-to-certificate.crt
        SSLCertificateKeyFile /path-to-key.key
        SSLCertificateChainFile /path-to-ca-chain.crt
        Header always set Strict-Transport-Security "max-age=15768000"
        RequestHeader set X_FORWARDED_PROTO 'https'
</VirtualHost>

<VirtualHost *:80>
        ServerName git.glpgs.io
        Redirect permanent / https://git.glpgs.io/
</VirtualHost>

Additionally, I recommend that you follow Mozilla MozWiki's great TLS advice or use their super handy, easy config generator as a global configuration that applies to all of your VirtualHosts. On Debian, you can pop that in to /etc/apache2/mods-available/ssl.conf, replacing the parameters they already specify.

Happy hacking!

Configuring OpenLDAP to authenticate using X.509 client certificates

This is not meant to be a comprehensive guide by any means, but information on the Web for configuring OpenLDAP to authenticate using X.509 client certificates is lacking. And in some cases, over a decade old! It took me hours to find the documentation I needed, but only minutes to see it working once I had the correct "recipe".

You should probably be running your own Certificate Authority for the purpose of generating client certificates, especially since you need one per user. You can lock it up tightly and only use it for the purposes of LDAP if you like. You can also use a certificate vendor like Thawte or GeoTrust or Comodo. Make sure you pick just one, though, because you will configure OpenLDAP to trust only that single CA to sign all the relevant client certificates. (This ensures that nobody can come in with a forged certificate signed by another vendor, or a self-signed one.)

The Ubuntu guide on making a CA is pretty decent, though unfortunately it uses the inferior GnuTLS package. That's okay, because we are only using it for OpenLDAP. And actually, you can't use OpenSSL generated certificates on Debian's OpenLDAP because they patched it in such a way that the certificates cannot be read. (There are conflicting reports on whether this bug was fixed or not upstream.) Note that you definitely want to set a higher expiration_days than the default 365! 10 or even 15 years isn't unheard of, which is 5475 days if you were wondering.

Once you have either created your CA, or decided on a vendor, you may begin configuring OpenLDAP. Replace authority.pem with the file name for your CA's root certificate, and ldap_cert.pem and ldap_key.pem for the server certificate and its private key. Note that the server certificate must have the FQDN of the LDAP server as its only CN. It may have a wildcard as a subjectAltName (or SAN) but the FQDN (normally something like ldap01.myproject.org) must be the CN.

With slapd.conf

TLSCACertificateFile /etc/ssl/certs/authority.pem
TLSCertificateFile /etc/ssl/certs/ldap_cert.pem
TLSCertificateKeyFile /etc/ssl/private/ldap_key.pem
TLSVerifyCert try

With Dynamic Configuration, aka cn=config, aka "OLC"/on-line configuration, aka ...

dn: cn=config
changetype: modify
add: olcTLSCACertificateFile
olcTLSCACertificateFile: /etc/ssl/certs/authority.pem
-
add: olcTLSCertificateFile
olcTLSCertificateFile: /etc/ssl/certs/ldap_cert.pem
-
add: olcTLSCertificateKeyFile
olcTLSCertificateKeyFile: /etc/ssl/private/ldap_key.pem
-
add: olcTLSVerifyCert
olcTLSVerifyCert: try

Note that if you receive an error such as:

ldap_sasl_interactive_bind_s: Unknown authentication method (-6)
 additional info: SASL(-4): no mechanism available: 

then you most likely forgot the olcTLSVerifyCert like I did the first time :) Note that there is nothing printed after "no mechanism available: ". That was the hardest part to debug! Hopefully this can help a few people out.

Also note that for client certificates to work correctly, the DN of the X.509 certificate must exactly match the DN of the LDAP object. If you cannot meet that requirement, you will need to look at authz-regexp: for cn=config, see this mailing list posting, and for standard configuration see the documentation. Note that I was unsuccessful in making this seemingly-useful feature work, but you may have better luck than I did.

References

• Configuring OpenLDAP for Client Certificate Authentication, The Moose and Squirrel Files
• OpenLDAP using OLC (cn=config), Zytrax LDAP
• Kerberos, GSSAPI, and SASL Authentication using LDAP: useful tips on ACLs for root DSE
• ldif(5) man page: always useful to have for cn=config
• The Source Code: sometimes you just have to see what's going on inside
• GnuTLS certtool man page: for template format to make the CA cert much better than the Ubuntu guide suggested

Going IPv6 native without IPv4

Now that I have finally moved in to my new apartment (which requires a long blog of its own), I have new routing equipment and a new network infrastructure. The native IPv6 on Cox Communications seems to be a bit better than the native IPv6 offered by Comcast Business; namely, Cox seems to be peered more widely and therefore ping times are much lower. Of course, this could be specific to the market I'm in - eastern Oklahoma - so YMMV.

However, because DHCP is a terrible protocol, it is constantly flaking, leaving me with IPv6-only access to the Internet. That is, no access to IPv4 whatsoever. Surprisingly, it's nearly usable. However, I am highly disappointed in a few surprises I've found that do not work over IPv6:

• EVERY SINGLE CODE HOSTING SERVICE ON THE INTERNET. This really, really, really, really upsets me. Luckily, I don't have to care any more, because I run my own now.
• DuckDuckGo. I am incredulous that a modern search engine is not accessible over IPv6.
• eBay and PayPal. This isn't really surprising, I suppose, since eBay were running Windows NT 4 as recently as 2006... they always have been a decade off of the current technologies.
• Any news Web site I tried: Bloomberg, BBC, New York Times, Washington Post.
• The entire StackExchange family of properties, five YEARS after being asked for even a trial of IPv6 access. This is entirely unacceptable. I expect news organisations and e-commerce conglomerates to be woefully behind the times, but a company designed from the ground up for computer scientists by computer scientists? I can't believe this is real.
• Weather.gov. The US government actually has an IPv6 project with real time online completion progress, even available itself via IPv6; however, while NOAA's flashy Web 3.0 marketing pages are available over IPv6, the important research, life-saving data, and forecast information made by the National Weather Service are entirely IPv4-only. I understand that internally, their infrastructure is not entirely ready for IPv6, but they should be able to run the main radar and warning information over IPv6 at least. Americans need not feel singled out, though; the UK's Met Office is also unavailable over IPv6.

At least Wikipedia and the Google properties are usable, so I have music, videos, and a reference library.

Web browsers, music players, workarounds, and PulseAudio

As security researchers have discovered yet another horrible security bug in Chrome, and Google yet again decides to put off fixing it, I decided to finally give up Chrome entirely. I had dwindled down my usage of it from primary browser in 2009; to secondary browser for Flash and videos in 2013; and finally using it solely for streaming Google Play Music and Spotify, along with the occasional site compatibility test for my work, in 2015. Firefox's inspector tools and Firebug are good enough, and I have a Mac running Safari if I need a WebKit test, so I decided it was no longer important to test on Chrome. That left the issue of music streaming.

Can't handle it, can't handle it

Google Play Music, however, has a fatal flaw. It is a mess of terrible "one page" JavaScript. After only a few hours of music streaming, it had already leaked 150 MB(!) worth of orphan DOM nodes, and 282 MB(!!!!) worth of uncollectable JavaScript objects. This basically means it created buttons, links, and so on, and didn't properly remove them when it was done, so that memory is leaked out and I would have to restart Firefox to get that memory back. Restarting Firefox multiple times a day is not an option for me.

What's worse is that one of Firefox's best and most unknown features was also making my life worse. Every 10 seconds, it scans its memory to see if any of it can be reclaimed, to make sure that it does not use too much memory. Since Google Play Music's interface had leaked so much memory, the scan was taking about 2 seconds - during which the browser became completely unresponsive. That means that for about 20% of the entire time it was open, it was unresponsive (frozen, locked, etc), all because Google has no idea how to write JavaScript.

My mother (bless her soul, she's openly embraced Debian) suggested I try Rekonq, but it could not even load Google Play Music's user interface. I also tried Opera Classic (pre-Blink), and it too could not load Google Play Music. At this point I am very upset at Google; why did you write such a cluster#*$@ of terrible code instead of writing a simple multi-page player like YouTube? YouTube does not suffer from any of these issues, and is a Google product!* Anyway, my next goal was to see what I could do for streaming music that did not require a Web browser.

* I am aware that YouTube has a single-page mode, but I found a way to disable it except while using playlists. It works great and does not leak half a gigabyte of memory.

Done, done, and I'm on to the next one

It turns out that Google Play Music has no official API and no non-browser clients. Even Spotify has unofficial ones that are of questionable quality and legality, but Google has done a very good job of making their API so hard to use that nobody bothers to even try with them. (Future project idea.)

Then I realised their Android app is pretty reliable and certainly better than having my browser locked for 20% of the time it's open. However, I still need to be able to hear other things on my computer (if someone links me to a video or presentation, for instance), and I don't want to have to keep flipping back and forth between my phone and desktop.

My work-provided desktop did not come with a sound card (even though we use sound a lot internally...), so I am using a USB Griffin iMic as my sound "card". It works fantastically in Linux/ALSA, but one thing I could not figure out was how to make it play line in as a monitor (i.e. playthrough, listening to line in/mic with headphones/out, whatever you like to call it). Thankfully, I found a very helpful blog post about this very issue, and a solution involving PulseAudio: pactl load-module module-loopback was all it took to listen to crystal-clear, low-latency, glorious Nexus audio on my desktop!

Final thoughts

• While it certainly is great that PulseAudio offers the same great passthrough functionality that OS X had since Jaguar (and lost in Mavericks), they really need to document PulseAudio modules better.
• Google needs to rethink making their music player in one page JavaScript. A native app would be amazing and make me a much happier catfox.
• It just feels like... if Google hadn't royally screwed up Chrome, and they hadn't royally screwed up their music player, then hours of my life would have been saved because then I would not have had to learn how to monitor line in within Linux. It was interesting learning all this, but I still have this feeling that it should be entirely unnecessary, and like this is a very unclean workaround for what amounts to "Google is terrible at writing code".

Oh well, at least Android 6.0 is good. (For now.)

The Joys of Unix Programming: MAP_ANON(YMOUS)

I was trying to do a little late-night hacking last night on SuperGameHerm, the Game Boy emulator my friends and I are writing, and I hit an error in the memory mapper. Specifically, certain OSes that used to be named after cats don't like calling mmap on /dev/zero (neither does Android). I thought it was odd that it was falling back to that code in the first place, though, because Apple's Mac OS X — I mean, certain cat themed OS — has always supported MAP_ANON, and I confirmed that by going to man mmap on a Mac.

What was going on? I dug deeper and saw MAP_ANON was guarded in sys/mman.h, so CMake wasn't finding it and it was instead compiling our fallback code. And so I started digging up other issues related to big endian machines and realised that I had only tested OS X and FreeBSD on big endian, and never tested OS X or FreeBSD on little endian. So was my big mistake.

This is a comprehensive guide to How to Make MAP_ANON(YMOUS) Visible, for every OS I could find information on:

Mac OS X

On 10.3 and below, this is easy; it's always there! It is not guarded by any #ifdef.

On 10.5 and above, it is slighly harder; you must define _DARWIN_C_SOURCE to cause MAP_ANON to be visible in a public scope.

On 10.4 only, it's much harder! It is only protected by #ifndef _POSIX_C_SOURCE, so to use MAP_ANON against the 10.4 SDK, you must completely undefine _POSIX_C_SOURCE. You don't have any other choice.

I suppose that means my overall advice then is to use the 10.5 SDK no matter what, if you have a Leopard computer handy, because it can target as low as 10.0. Otherwise, use the Panther SDK included with Tiger's Xcode Tools. Don't ever use Tiger's SDK if you want MAP_ANON.

FreeBSD

Before 5.0, it's always visible, just as in OS X 10.3. There are no preprocessor options to show or hide MAP_ANON.

On 5.0 or above, the only way to cause MAP_ANON to be visible is to define __BSD_VISIBLE somewhere. Undefining _POSIX_C_SOURCE won't save you here.

Other BSDs (NetBSD, OpenBSD, DragonFly BSD)

It's never guarded. MAP_ANON is always available.

Solaris

I could only get my hands on OpenSolaris, but considering the header having a copyright date of 1989 (by AT&T), I can't imagine it's any different on Real Solaris (or Oracle Solaris). There are no guards here, either; that's to be expected since they invented the damn thing.

Linux

glibc: I don't understand /usr/include/bits in the slightest. It seems to be always available no matter what options I toss to clang, but it is guarded by.. __USE_MISC? I presume this is some sort of feature macro buried deep in glibc that I don't care about or understand.

musl: It's always available, at least on 1.1.11 which is what I have on my test box.

Android: After searching through their spaghetti of includes to get to the actual file that defines constants, it appears they are all completely unguarded, though that isn't surprising since it is Linux and embedded.

In conclusion

Perhaps it's best to avoid anonymous mmap(2) in applications that you want to actually be portable.

Linux 4.1 kernel: Finally, a movement in the upwards direction

No musings yesterday because I was having quite a time getting Linux to play well. I spent yesterday configuring the rest of the kernel until around 21:00, when I started searching for and applying all the patches I wanted. Found a small bug in the CPU optimisation patch that Gentoo uses (via grsecurity); namely, it doesn't enable P6_NOP for anything higher than a Core 2. I manually edited the patch and applied it.

I built the kernel, ran # emerge @module-rebuild, signed the newly built modules, and rebooted in to my new system... Or I would have, if it had been able to read the init RAM disk. Doing the Apple EFI dance to switch between 3.18 to rebuild and 4.1 to test was not my idea of a good time, but after a while, I found the issue. There's something broken somewhere and it would not read XZ-compressed ones. I used my fallback algorithm, LZO (chosen because it's free and very fast), and it booted right up. Only now, I couldn't start KDE.

Typically, I start KDE by using $ X_SESSION=KDE-4 startx; I don't use KDM or XDM because I regularly test breaking changes to i915 that may cause X to hardlock, rendering my laptop useless. This time, however, it complained 'xterm: command not found'. Unsure why it was trying to load xterm, I checked around and I found no answer. I still haven't figured this out. The workaround I'm using is $ XINITRC=/etc/X11/Sessions/KDE-4 xinit which acts the same as the old command, only it's longer and less readable.

Then I had a world of i915 bugs that I will omit from my blog mainly because they were mostly configuration error and the ones that weren't were resolved by re-merging media-libs/mesa.

After all of that was over, I checked on my wireless chip. Lo and behold, for the first time since I've started using draft-n wireless networking on Linux in 2009, it was actually associating successfully to a 5GHz 802.11n AP! The speed is only 150mbit/s; the chip supports 300mbit, so I am not sure why that is, but I am still happy to be finally untethered from my Ethernet cord!

Some benchmarks.

Running emerge -1 openssl

Kernel / Sched	Wall Time	System Time
3.18 / CFQ	5m 3s	59s
4.1 / CFQ	4m 57s	55s
4.1 / BFQ	4m 53s	52s

That's a pretty nice win for BFQ, and a huge improvement over 3.18. Note this is all on the same hardware, on a fresh boot, with nothing in fscache.

Playing 720p MPEG-4 video in VLC

Kernel	On-Die Temperature	CPU % used
3.18	71° C	31.2%
4.1	63° C	46.3%

Each measurement was taken at 2 minutes into playing the video. Not sure why the CPU's a bit more active, but wow, that temperature reduction is serious!

Overall, I am very, very happy with this upgrade. The Linux 4 series seems to be all about making things smoother, faster, and more battery-friendly. I'll update later with a benchmark of battery life.

If you have a MacBook Pro from the 2011 era (or you're just curious), you can view my config file online. Special thanks to Elly and Horst for guidance, patchsets, and keeping me company while I was going insane with menuconfig. :)