Saturday, 7 October 2017

Fixing the laptop I broke

Sometimes things happen that you don't expect. It can be anything: a power failure during a system upgrade, or maybe a careless chmod 644 /usr/lib/ — in my case, it was the latter (tab completion failure).

Training yourself on the proper way to respond to unexpected failures is the key to recovering them without pain or further data loss. When I had realised my gaffe, the first thing I thought was: How do I chmod it back without the ability to run chmod?

Static-linked rescue binaries are a must-have

The first thing I learned from this experience is that having a set of static-linked rescue binaries somewhere on your system can help in a lot of unexpected situations. We're going to be adding a busybox-static package to Adélie Linux just for such an occasion, and we may put it in the base system depending on community feedback. If I had a static busybox in, say, /var/recovery or such a path, this would have been a ten second fix rather than a few hour fix.

Embrace the system

After a few other attempts, I realised I could drop to assembler. Long ago, I spent my days writing assembler for system-level code. Since assembler is by design writing "below" C, you are not using the C runtime. Theoretically, you should be able to perform the same tasks as any utility on the system as long as there's a matching system call for it. And by luck, there is a single syscall: SYS_chown. Following is x86_64 assembler for Linux to chown /usr/lib/ back to 755 (executable for all users):

_start: mov $90,%rax /* SYS_chown */ mov $str,%rdi /* const char *filename */ mov $493,%rsi /* mode_t mode */ syscall /* do it! */ mov $60,%rax /* SYS_exit */ syscall /* bye */ str: .ascii "/usr/lib/\0"

Then it was a matter of as -o fixit.o fixit.S; ld -o fixit fixit.o; strip fixit to generate a 440 byte binary file that would solve my issue. The next issue was transferring it to the laptop. I tried to use bash's /dev/tcp; unfortunately however, it does not support binary file transfer without something like `cat` or `dd`. Since I could only use the shell, I did what I had not done in over a decade: echo -n followed by the escape codes. Since a lot of the binary was still padding, I omitted the last 200 or so bytes. The output of the echo command needed to be redirected to a binary that was already executable (otherwise the file created would not have execute permission!), so I chose one I probably wouldn't need urgently: neon-config, a configuration utility for a library I installed for tinkering. The full shell transcript is in my misc Linux directory. This worked! And my laptop ran again...

As I said in the opening of this little musing: I could have made things a lot worse and lost all my open unsaved data by turning off the computer and trying to recover using media. Additionally, that computer is very picky about booting off external media, so that would have wasted even more time. Sometimes all you need is ingenuity and experience, and the only way to acquire either one is by messing about and poking at stuff! Happy hacking.

Thursday, 12 January 2017

Configuring a more secure password hash for OpenLDAP

While working on the Galapagos infrastructure, we ran in to an interesting issue: using passwd(1) as an LDAP user would cause it to add another password instead of modifying it. Setting up the slapo-ppolicy(5) overlay then caused passwd(1) to then fail with:

password change failed: Password policy only allows one password value
passwd: Authentication token manipulation error
passwd: password unchanged

After consulting the #openldap channel on Freenode, the problem turned out to be that although OpenLDAP allows you to set olcPasswordHash on the root cn=config node, it does not work correctly when set there; it must be set under olcDatabase={-1}frontend,cn=config. Note, however, that olcPasswordCryptSaltFormat does belong in cn=config directly.

Sunday, 8 January 2017

Configuring Apache 2.4 to serve GitLab over TLS / HTTPS

As part of my work assisting in the set up of the infrastructure for Galapagos Linux, I volunteered to install and configure GitLab. My colleagues had attempted to use the Debian Omnibus package, but that failed in spectacular ways, including references to directories in the configuration that did not exist after package installation.

The most important piece of advice I can give is that you absolutely must use Bundler v1.10.6 or older[1] to ensure that you do not receive Gemfile.lock errors. You will also need to make a small modification to the Gemfile and Gemfile.lock file to ensure that libv8 is present if you wish to precompile the assets.

Now, for the Apache configuration. Note that I assume you have enabled https in GitLab's config/gitlab.yml and set port: 443. You will need to set a forwarding request header[2] to ensure that GitLab does not throw CSRF authentication errors. Also, if you want to use the recommended Unix sockets of Unicorn, you will need to configure the ProxyPass and ProxyPassReverse to use unix:/path/to/socket|http://HOSTNAME (thanks, Xayto!) - the full VirtualHost for GitLab goes something like this:

<VirtualHost *:443>
        ProxyPass / unix:/home/git/gitlab/tmp/sockets/gitlab.socket|
        ProxyPassReverse / unix:/home/git/gitlab/tmp/sockets/gitlab.socket|
        SSLEngine on
        SSLCertificateFile /path-to-certificate.crt
        SSLCertificateKeyFile /path-to-key.key
        SSLCertificateChainFile /path-to-ca-chain.crt
        Header always set Strict-Transport-Security "max-age=15768000"
        RequestHeader set X_FORWARDED_PROTO 'https'

<VirtualHost *:80>
        Redirect permanent /

Additionally, I recommend that you follow Mozilla MozWiki's great TLS advice or use their super handy, easy config generator as a global configuration that applies to all of your VirtualHosts. On Debian, you can pop that in to /etc/apache2/mods-available/ssl.conf, replacing the parameters they already specify.

Happy hacking!