My Techie-Thoughts

If you run a server of any kind, you know the importance of making sure your clients can securely connect.  Many people rely on Let’s Encrypt since they issue free certificates that make these secure connections possible.  However, those certificates are only good for 90 days and then have to be renewed… that’s a hassle!  Enter Certbot…

Certbot is a super useful program that will run on your server, get your free certificates from Let’s Encrypt and even renew them for you so your connections hum along always encrypted and safe.  Let’s take a look at how we can set this up and give you one less thing to worry about.  In this article, I’m using a Debian 9 (Stretch) server running NGINX. However, the basic methodology should work for most other Linux distros and other web servers too.

How it works…

Let’s Encrypt/ACME (the issuing authority) checks the DNS records of the domains for which you request a certificate.  Those records (almost always an ‘A or AAAA record’) resolve to an IP address that points to your server.  Let’s Encrypt then generates one file for each domain which must be uploaded to your server.  If those files can be accessed via the addresses listed in DNS, then it’s assumed you own/control those servers and a certificate can be issued. Maybe an example would be helpful:

• Your web server answers requests for myserver.net, www.myserver.net & service.myserver.net.
• You would like to secure all 3 hostnames with 1 certificate.
• Let’s Encrypt generates a hash file to be placed in a special directory in the webroot of the server(s) that answer to each of those hostnames.  So, it should map out like this:

• Certbot will create a hidden directory, .well-known/acme-challenge/, in your webroot and place FileA, FileB and FileC in that directory.
• Let’s Encrypt will use DNS and query the server that is supposed to answer each hostname.  As shown in the “LE checks” column of the table above, it will look for a specific file on each server.
• Assuming those files are successfully served, you’ve demonstrated to Let’s Encrypt that DNS points to the correct server and that you are authorized to place files on that server.
• The certificate is issued.
• Certbot cleans up by removing FileA, FileB and FileC.
  
  

Ok, so now let’s make all that happen…

Install Certbot

The default Debian Stretch repo has an older version of Certbot that is no longer recommended. Fortunately, the current version is in the ‘backports’ repo so we can easily access it. Let’s go ahead and add the backports repo to our apt sources lists then let APT do all the heavy-lifting for us.

First, create a new file called ‘stretch-backports.list’ in our /etc/apt/sources.list.d/ directory. You can name the file anything, but it must have the extension .list. You can use a text editor like nano or vi, but I’ll just do it on the command line because it’s easier to show here on the website. You’ll be putting the text “in the quotes” inside that file and saving it. This will add a source to your APT download list that will allow it to find the stretch-backports repo. Please note that if you’re already using Debian 10 (Buster), then just replace ‘stretch’ everywhere you see it with ‘buster’.

# create our sources file pointing to the backports-repo
"deb http://deb.debian.org/debian stretch-backports main" > /etc/apt/sources.list.d/stretch-backports.list
# refresh our APT list
apt update
# install certbot
apt -t stretch-backports install certbot

Now, let’s get the directory structure in your WEBROOT set up and secured for Certbot to work with, replacing [square brackets] with settings appropriate to your setup:

mkdir [webroot]/letsencrypt
chown root:[webuser] [webroot]/letsencrypt
chmod 775 [webroot]/letsencrypt

You can get the correct settings for your webroot and webuser by checking the configuration files for your web server (e.g. /etc/nginx/nginx.conf).  For a default Debian NGINX installation, for example, you’d set up the directory structure like this:

mkdir /usr/share/nginx/html/letsencrypt
chown root:www-data /usr/share/nginx/html/letsencrypt
chmod 775 /usr/share/nginx/html/letsencrypt

this set up is also very popular with various distributions:

mkdir /var/www/letsencrypt
chown root:www-data /var/www/letsencrypt
chmod 775 /var/www/letsencrypt

This gives Certbot somewhere to save the verification files needed by Let’s Encrypt to issue our certificates and then delete those files automatically when the process is completed. We granted permissions to both our root and web server accounts since it depends on your distribution under which account Certbot runs.

Webserver setup

Certbot creates a .well-known/acme-challenge/ directory within your webroot/letsencrypt directory to store generated verification files so they can be confirmed by Let’s Encrypt. A call to myserver.net would result in myserver.net/.well-known/acme-challenge/filename being queried.  Obviously, your web server needs to serve calls to myserver.net/.well-known/acme-challenge/… from [webroot]/letsencrypt/.well-known/acme-challenge/… For NGINX, we’d use a location block something like this in our server block (update webroot on the highlighted line as appropriate for your environment):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# process LetsEncrypt requests
location ^~ /.well-known/acme-challenge {
      # setup logging to review certificate requests
  access_log   /var/log/nginx/LetsEncrypt_access.log main;
  error_log   /var/log/nginx/LetsEncrypt_error.log warn;

  default_type text/plain;
  root /webroot/letsencrypt;
  autoindex on;
}

I don’t use Apache, but I imagine you’d use something quite similar.  If you use Apache, let me know in the comments section and I’ll credit you and add your location block here to help other people. The above section uses a regex match and says that any location URI that starts with ‘/.well-known/acme-challenge’ should be read from our [webroot]/letsencrypt directory with the URI path appended… in other words, [webroot]/letsencrypt/.well-known/acme-challenge/…

Get certificates

While Certbot has an automated mode for Debian/NGINX setups, I find that it requires a generic setup of your NGINX server configuration files and I don’t use generic setups.  So, I’m going to use a more explicit command with Certbot to tell it exactly what to do.  Here’s the command, followed by what it all means (remember to replace stuff in [square brackets]):

certbot certonly ––webroot –w [webroot]/letsencrypt/ --rsa-key-size 4096 –d domain.tld,www.domain.tld,subdomain.domain.tld --deploy-hook "systemctl restart nginx.service" ––agree-tos –m [email protected] --no-eff-email

• certonly: This tells Certbot to obtain/renew the certificate, but don’t install it (i.e. we want to modify our webserver configuration manually)
• –webroot: Treat the path provided by the ‘-w’ parameter as our webroot
• -w: Path to be used as the webroot (required because of ––webroot)
• –rsa-key-size: Number of bits for the RSA key (2048 is the default but modern computer power makes 4096 more reasonable)
• -d: Comma separated list of domains for which to request a certificate (subject-alternate-names)

• –deploy-hook: Command to run after renewing certificates. In this case, we are restarting NGINX so it reads the new certificates. If you’re using Apache, you’d use something like ‘systemctl restart apache2.service’. You can also enter a path to a script file if you need to do something more complex like restart docker containers, etc.
• –agree-tos: Agree to the Let’s Encrypt/ACME terms of service without a prompt
• -m: Email address for email notifications from Let’s Encrypt/ACME
• –no-eff-email: Opt-out of the Electronic Frontier Foundation email mailing list (optional… it’s actually a pretty good newsletter)

So, in plain(ish)-English: Get me a 4096-bit RSA certificate to secure domain.tld, www.domain.tld and subdomain.domain.tld.  Verify that this server is the destination endpoint of the DNS records for those domains by verifying some files in my webroot/letsencrypt folder.  Also, know that I agree to the license terms and you may contact me at the provided email address with any important information but don’t add me to your newsletter mailing list. Once my certs are issued, please restart my webserver for me.

Let’s Encrypt directory structure

Your certificate and key are stored in */etc/letsencrypt/archive/your.domain.tld/* and are symlinked to /etc/letsencrypt/live/your.domain.tld/.  You must ALWAYS reference the symlinked copies since they are the up-to-date renewed certificates.

Confirm your setup

Assuming all went well, you already have certificates for the domains you requested in the step above and renewal has been set up automatically. Let’s just be sure everything is exactly the way we want so we don’t have to worry about it in the future. First, we’ll check our renewal parameters.

cat /etc/letsencrypt/renewal/your.domain.tld.conf

Check the [renewalparms] section, it should look something like this:

[renewalparams]
account = d7j398dlke93kdhf93kd93dl903jf30d
authenticator = webroot
rsa_key_size = 4096
installer = None
renew_hook = systemctl restart nginx.service

Ensure the renew_hook line is correct (change it if needed) and double-check the rsa_key_size value too.  Assuming those are all good, you’re all set.  One more test to ensure everything is ok:

certbot renew ––dry-run

This will generate a renewal request and process domain re-verification, but will not generate new certificates.  It should complete successfully and mention that it is “skipping renewal hook command” which lets you know that it’s aware of the command’s presence in the configuration file.

Renewal timers

Certbot is called twice per day (12 am and 12 pm by default) to check the status of your certificates and if they need to be renewed.  If they are near expiration (within 30 days) then they will be automatically renewed.  This is accomplished via a cronjob or systemd timer.

cronjob

If you check out /etc/cron.d/certbot, you’ll see the following:

0 */12 * * * root test -x /usr/bin/certbot -a \\! -d /run/systemd/system && perl -e 'sleep int(rand(3600))' && certbot -q renew

This means that every 12 hours at midnight and noon, check to see if Certbot is installed and that systemd is NOT installed.  If those conditions are met, run Certbot in renew mode quietly.  If your system is not using systemd, then this is how your renewals will be processed so be aware of this cronjob!

systemd

On most current Debian/Ubuntu systems, you are using systemd and thus, the cronjob above will not run.  Instead, there is a systemd service and timer installed.  Let’s verify all is good with those:

systemctl list-timers

You should see ‘certbot.timer’ listed.  Now, let’s make sure the files that define this timer and it’s accompanying service exist too:

ls -lA /lib/systemd/system/certbot*

You should see both ‘certbot.timer’ and ‘certbot.service’ listed.  Let’s check the status of these components:

systemctl status certbot.service   #should show loaded but dead
systemctl status certbot.timer   #should show loaded and active(waiting)

You should also ensure that when checking the status of the ‘certbot.timer’ the ‘Loaded:’ line lists the timer as ‘enabled’ otherwise it will not be automatically started after a reboot.  If it’s showing ‘disabled’, then run the following:

systemctl enable certbot.timer

Using your certs

Now that everything is set up, it’s time to direct your webserver to use your certificates.  Remember, Certbot stores the active (i.e. current) certificate as a symlink in the live subdirectory.  So you just have to let your web server know that’s where it should be looking.  If you’re using NGINX, you’d add the following to the server configuration (remember to enter your actual domain instead of my placeholders):

ssl_certificate		/etc/letsencrypt/live/domain.tld/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/domain.tld/privkey.pem

By referencing the symlinked ‘live’ files, you do not have to update this configuration as your certificates are renewed, etc.  As a side note, if you need to reference only the certificate chain (such as for OCSP), that file is located at /etc/letsencrypt/live/domain.tld/chain.pem.

Final thoughts…

That’s it!  Pretty simple, just a lot of double-checking since it’s a pretty important service and easy to miss simple configuration steps.  You now have free, trusted SSL certificates for your server on any/all domains and those certificates will renew themselves and work without you having to think about them.  Pretty cool, right?