Let's Encrypt, broken IPv6

I just had an awful debugging session when let’s encrypt renewals failed on one of my servers, so I will write it up in case someone else comes across the problem.

tldr: Let’s Encrypt failed to fetch tokens from the webserver, but I could fetch them myself. IPv6 was not configured on the server but was in the DNS. My connections worked over IPv4, but they checked both, and did not provide a clear error message.

I was using http-01 / webroot authenticator.

Let’s Encrypt logs / verbose output reported in json: Error getting validation data, "status": 400, "status": "invalid". The 400 apparently was not an http status code, although it looks like one.

It looks like, from this report, that it may only be a problem when there is an http redirect involved too, as there was in my case.

IPv6 configuration failed on the Debian server because of cloud-init interfering with manual IP configuration. /etc/network/interfaces was used manually. cloud-init replaces the main config file. I used a file in the config directory to avoid this, but didn’t know that it also ran some if-pre-up script that tried to run DHCPv6. This caused the manual configuration of the interface to fail.

ifup script causes: RTNETLINK answers: File exists

This seems to happen most commonly when attempting to configure two default gateways, such as ones manually specified on two interfaces, or when one is already auto configured. This could be from SLAAC or DHCPv6.