Saturday, 7 October 2017

Fixing the laptop I broke

Sometimes things happen that you don't expect. It can be anything: a power failure during a system upgrade, or maybe a careless chmod 644 /usr/lib/ — in my case, it was the latter (tab completion failure).

Training yourself on the proper way to respond to unexpected failures is the key to recovering them without pain or further data loss. When I had realised my gaffe, the first thing I thought was: How do I chmod it back without the ability to run chmod?

Static-linked rescue binaries are a must-have

The first thing I learned from this experience is that having a set of static-linked rescue binaries somewhere on your system can help in a lot of unexpected situations. We're going to be adding a busybox-static package to Adélie Linux just for such an occasion, and we may put it in the base system depending on community feedback. If I had a static busybox in, say, /var/recovery or such a path, this would have been a ten second fix rather than a few hour fix.

Embrace the system

After a few other attempts, I realised I could drop to assembler. Long ago, I spent my days writing assembler for system-level code. Since assembler is by design writing "below" C, you are not using the C runtime. Theoretically, you should be able to perform the same tasks as any utility on the system as long as there's a matching system call for it. And by luck, there is a single syscall: SYS_chown. Following is x86_64 assembler for Linux to chown /usr/lib/ back to 755 (executable for all users):

_start: mov $90,%rax /* SYS_chown */ mov $str,%rdi /* const char *filename */ mov $493,%rsi /* mode_t mode */ syscall /* do it! */ mov $60,%rax /* SYS_exit */ syscall /* bye */ str: .ascii "/usr/lib/\0"

Then it was a matter of as -o fixit.o fixit.S; ld -o fixit fixit.o; strip fixit to generate a 440 byte binary file that would solve my issue. The next issue was transferring it to the laptop. I tried to use bash's /dev/tcp; unfortunately however, it does not support binary file transfer without something like `cat` or `dd`. Since I could only use the shell, I did what I had not done in over a decade: echo -n followed by the escape codes. Since a lot of the binary was still padding, I omitted the last 200 or so bytes. The output of the echo command needed to be redirected to a binary that was already executable (otherwise the file created would not have execute permission!), so I chose one I probably wouldn't need urgently: neon-config, a configuration utility for a library I installed for tinkering. The full shell transcript is in my misc Linux directory. This worked! And my laptop ran again...

As I said in the opening of this little musing: I could have made things a lot worse and lost all my open unsaved data by turning off the computer and trying to recover using media. Additionally, that computer is very picky about booting off external media, so that would have wasted even more time. Sometimes all you need is ingenuity and experience, and the only way to acquire either one is by messing about and poking at stuff! Happy hacking.

Thursday, 12 January 2017

Configuring a more secure password hash for OpenLDAP

While working on the Galapagos infrastructure, we ran in to an interesting issue: using passwd(1) as an LDAP user would cause it to add another password instead of modifying it. Setting up the slapo-ppolicy(5) overlay then caused passwd(1) to then fail with:

password change failed: Password policy only allows one password value
passwd: Authentication token manipulation error
passwd: password unchanged

After consulting the #openldap channel on Freenode, the problem turned out to be that although OpenLDAP allows you to set olcPasswordHash on the root cn=config node, it does not work correctly when set there; it must be set under olcDatabase={-1}frontend,cn=config. Note, however, that olcPasswordCryptSaltFormat does belong in cn=config directly.

Sunday, 8 January 2017

Configuring Apache 2.4 to serve GitLab over TLS / HTTPS

As part of my work assisting in the set up of the infrastructure for Galapagos Linux, I volunteered to install and configure GitLab. My colleagues had attempted to use the Debian Omnibus package, but that failed in spectacular ways, including references to directories in the configuration that did not exist after package installation.

The most important piece of advice I can give is that you absolutely must use Bundler v1.10.6 or older[1] to ensure that you do not receive Gemfile.lock errors. You will also need to make a small modification to the Gemfile and Gemfile.lock file to ensure that libv8 is present if you wish to precompile the assets.

Now, for the Apache configuration. Note that I assume you have enabled https in GitLab's config/gitlab.yml and set port: 443. You will need to set a forwarding request header[2] to ensure that GitLab does not throw CSRF authentication errors. Also, if you want to use the recommended Unix sockets of Unicorn, you will need to configure the ProxyPass and ProxyPassReverse to use unix:/path/to/socket|http://HOSTNAME (thanks, Xayto!) - the full VirtualHost for GitLab goes something like this:

<VirtualHost *:443>
        ProxyPass / unix:/home/git/gitlab/tmp/sockets/gitlab.socket|
        ProxyPassReverse / unix:/home/git/gitlab/tmp/sockets/gitlab.socket|
        SSLEngine on
        SSLCertificateFile /path-to-certificate.crt
        SSLCertificateKeyFile /path-to-key.key
        SSLCertificateChainFile /path-to-ca-chain.crt
        Header always set Strict-Transport-Security "max-age=15768000"
        RequestHeader set X_FORWARDED_PROTO 'https'

<VirtualHost *:80>
        Redirect permanent /

Additionally, I recommend that you follow Mozilla MozWiki's great TLS advice or use their super handy, easy config generator as a global configuration that applies to all of your VirtualHosts. On Debian, you can pop that in to /etc/apache2/mods-available/ssl.conf, replacing the parameters they already specify.

Happy hacking!

Wednesday, 21 December 2016

Ah, wonderful health hazards

I can't tell what has been overall worse for my health in the past few weeks. The bathroom connected to my home office directly sits over the complex's "laundromat station". This did not used to bother me. In fact, I was quite okay with this, because it means I have the closest walking distance of any of my neighbours to it. However, for the past two or three weeks, I can smell — from the office, mind — a very strong odour of laundry detergent every time someone does a load. Turns out a lot of people do loads in the 18:00 to 21:00 time slot on weekdays, which happens to be when I am at my most productive in my office. I cannot imagine this is at all healthy for me.

But then I remember I've spent every day since Saturday spending multiple hours trying to set up OpenLDAP for new project. I've always just used Active Directory on the server-side, so my only experience thus far with OpenLDAP has been client-side. It's a great client library with easy configuration and a great debug mode that will tell you exactly what is happening and what is going wrong. Unfortunately, the server part, at least on Debian, uses "dynamic configuration" which means everything is in LDAP.

Now, look, LDIF and LDAP are fine and great for phone book-style records. It makes perfect sense. That is what it was designed to do. Storing regexp in ASN.1 BER is pushing it. But the way they do HDB/MDB grouping feels to me like trying to fit in with all those cool kids with their NoSQL and their MapReduce and their terrible terribly-great performance by using "shards" everywhere. And our leader wants replication so that it's fault tolerant. Now I get to convert decades-old documentation about an "enterprise" feature to this "dynamic configuration" thing. I cannot imagine this is at all healthy for me.

Configuring OpenLDAP to authenticate using X.509 client certificates

This is not meant to be a comprehensive guide by any means, but information on the Web for configuring OpenLDAP to authenticate using X.509 client certificates is lacking. And in some cases, over a decade old! It took me hours to find the documentation I needed, but only minutes to see it working once I had the correct "recipe".

You should probably be running your own Certificate Authority for the purpose of generating client certificates, especially since you need one per user. You can lock it up tightly and only use it for the purposes of LDAP if you like. You can also use a certificate vendor like Thawte or GeoTrust or Comodo. Make sure you pick just one, though, because you will configure OpenLDAP to trust only that single CA to sign all the relevant client certificates. (This ensures that nobody can come in with a forged certificate signed by another vendor, or a self-signed one.)

The Ubuntu guide on making a CA is pretty decent, though unfortunately it uses the inferior GnuTLS package. That's okay, because we are only using it for OpenLDAP. And actually, you can't use OpenSSL generated certificates on Debian's OpenLDAP because they patched it in such a way that the certificates cannot be read. (There are conflicting reports on whether this bug was fixed or not upstream.) Note that you definitely want to set a higher expiration_days than the default 365! 10 or even 15 years isn't unheard of, which is 5475 days if you were wondering.

Once you have either created your CA, or decided on a vendor, you may begin configuring OpenLDAP. Replace authority.pem with the file name for your CA's root certificate, and ldap_cert.pem and ldap_key.pem for the server certificate and its private key. Note that the server certificate must have the FQDN of the LDAP server as its only CN. It may have a wildcard as a subjectAltName (or SAN) but the FQDN (normally something like must be the CN.

With slapd.conf

TLSCACertificateFile /etc/ssl/certs/authority.pem
TLSCertificateFile /etc/ssl/certs/ldap_cert.pem
TLSCertificateKeyFile /etc/ssl/private/ldap_key.pem
TLSVerifyCert try

With Dynamic Configuration, aka cn=config, aka "OLC"/on-line configuration, aka ...

dn: cn=config
changetype: modify
add: olcTLSCACertificateFile
olcTLSCACertificateFile: /etc/ssl/certs/authority.pem
add: olcTLSCertificateFile
olcTLSCertificateFile: /etc/ssl/certs/ldap_cert.pem
add: olcTLSCertificateKeyFile
olcTLSCertificateKeyFile: /etc/ssl/private/ldap_key.pem
add: olcTLSVerifyCert
olcTLSVerifyCert: try

Note that if you receive an error such as:

ldap_sasl_interactive_bind_s: Unknown authentication method (-6)
 additional info: SASL(-4): no mechanism available: 

then you most likely forgot the olcTLSVerifyCert like I did the first time :) Note that there is nothing printed after "no mechanism available: ". That was the hardest part to debug! Hopefully this can help a few people out.

Also note that for client certificates to work correctly, the DN of the X.509 certificate must exactly match the DN of the LDAP object. If you cannot meet that requirement, you will need to look at authz-regexp: for cn=config, see this mailing list posting, and for standard configuration see the documentation. Note that I was unsuccessful in making this seemingly-useful feature work, but you may have better luck than I did.


Wednesday, 14 December 2016

Let's Encrypt and why I still pay for TLS certificates

I am asked with alarming regularity why I am not using Let's Encrypt for my personal Web sites, and for Adélie's site, and for my mother's art gallery site, and so on. "Why do you pay money for something you could have for free? And then you aren't giving money to those evil CAs!"

TLS certificates are still very much "you get what you pay for". Let's Encrypt is free, and on paper it seems to be a great solution with roots in freedom and socialism. However, it has a number of large issues in practice that prevent me from being able to adopt it.

The first, and most evident, is the failure of the community to provide a single ACME client that is well-supported and provides configuration options. As of this writing, there are 49 different client implementations on the official site. The problems with them are as numerous as the offerings; my main complaint is that most of them require themselves to run as the root user to automatically write to sensitive certificate files that are owned by the Web server user and are chmod 400.

The second large issue I've seen is that most of these 'automatic updates' break. This can be due to administrator error - and since there is not one single option, there cannot be a single repository of knowledge. This can also be due to APIs or endpoints changing. I have seen an official Mozilla blog and Void Linux's repository broken in the last week alone, all by botched ACME cron jobs. This solution is sold as "set and forget", but it requires more effort than simply going to a site every year and inputting a CSR and privkey.

Other issues with Let's Encrypt include: Let's Encrypt lacks a "site seal" which is very important on e-commerce sites to foster user trust. Let's Encrypt does not provide OV (let alone EV), which also compromises trust in people who know what to look for.

All in all, I think going forward Let's Encrypt may be suitable for power users and people who run TLS servers off their home servers. It may even be suitable for some personal sites and blogs. But I don't think it is a long-term solution for person who need trust, or those who have a complicated infrastructure (such as a distro, like Adélie).

Wednesday, 9 November 2016

Trump and change

The ball is in your court now, American Republicans.

I normally avoid politics and other controversial topics on my blog, because I have always felt it is important to keep my audience focused on the technical. Our common ground is unifying and allows us to look past our differences and learn from one another. I feared that if I started talking about politics, people would look at me differently, and I'd lose some of that audience. They wouldn't trust me and I wouldn't be able to enrich their lives.

I feel like that part of life in America is over now. President-Elect Donald Trump talks outlandishly, without filter or censor. People love him, people hate him, people think he's a joke, people think he's the best non-politician the political world has ever seen. As for myself, I'm somewhere between; but if I have learned a single thing from Mr Trump, it is that the world will not end if you speak up and say what is really on your mind. And perhaps this is a good kind of change. Without open discussion, we can't ever heal the divisiveness that permeates the entire country's political landscape, and indeed, the entire world's. There is a not-too-distant past where the words 'conservative' and 'liberal' were words that describe someone's political views, and were not used as slurs or to denigrate someone. Perhaps now that the precedent has been set, we can have open and honest discussions with one another. I'm not sure if that is where we are headed or not. I can only hope that we can learn to be respectful of each other's differences.

Mr Trump has said some things I agree with; per I Side With, I agree with almost 40% of his policies. It's not perfect, but it isn't exactly a disaster either. (For full disclosure, I only had just over 70% of agreement with Clinton.) He has also said a great deal of very offensive things. He has said things that have made some of my friends sick, depressed, and suicidal now that he has become President Elect of the United States. I urge these people especially to remember that first and foremost, Mr Trump is a showman. He knows how to pull in ratings, and was a reality television star. He may think less of Muslims than he should, but I don't think he will actually have every last one deported back to their homelands — especially since some of them were born and raised in the United States. He may think far less of women than he should, but that thinking is common in men from his generation. His objectification of women and misogyny is of course never acceptable, but women have had much worse oppressors than he ever could be.

I have friends of many classes. I have friends who are very well off — the typical Silicon Valley millionaire. I have friends who are destitute and live pay stub to pay stub, and would likely go homeless if they had even a small hiccup in work. I have friends who are in minority classes: African-American people, transgendered people, people with disabilities. We are all Americans. We all deserve a place in general society. Our society is built on the fact, not opinion, that everyone is created equal. There is room in the United States for the rich and poor, and the different races and religions that comprise this great country. No matter who won the United States election this year, our society has been broken, is broken, and will remain broken until it is healed.

Republicans, Democrats, Libertarians, Greens, other party members, independents, and even those disillusioned with the political system as a whole: society will only begin to be fully inclusive when we all learn to love each other. We have to work together. We have to stand up for what we believe in. Conflicting interests only break people into hate when they do not bend to compromise. I plan on writing letters to my state Senator, who is a Republican, and telling him my concerns going forward. I will have my voice heard. My Senator will, of course, have to balance my voice with others in our great state of Oklahoma. But together, I feel that we can find common ground and be able to find peace and happiness no matter what our political views.

Mr Trump. You promised to make America great again. If you can set an example with moderation and fairness, balancing differing viewpoints to create a clear path forward, you just may be able to succeed. I did not vote for you, but I still wish to work with you to create a common good for all of the United States.