If you run a server of any kind — mail, web, git, database, whatever — you know the importance of making sure your clients can securely connect.  That all starts with an SSL (or more accurately, TLS) certificate.  For smaller or private installations, many people are relying on Let’s Encrypt since they issue free certificates.  However, those certificates are only good for 90 days and then have to be renewed… that’s a hassle!  Enter Certbot…

Certbot is a super useful program that will run on your server, get your free certificates from Let’s Encrypt and even renew them for you so your connections hum along always encrypted and safe.  Let’s take a look at how we can set this up and give you one less thing to worry about.  This article assumes and gives examples for a Debian/Ubuntu server running NGINX as its webserver.  However, the basic methodology should work for most other Linux distros and other webservers too.

How it works…

Let’s Encrypt/ACME (the issuing authority) checks the DNS records of the domains for which you request a certificate.  Those records (almost always an ‘A record’) resolve to an IP address that points to a particular server — I assume your server in this case.  Let’s Encrypt/ACME then generates one file to be placed on each of those servers.  If those files can be accessed via the addresses listed in DNS, then it’s assumed you own/control those servers and a certificate can be issued to you for use on said server(s).  Let’s run through an example:

    1. Your webserver answers requests for myserver.net, www.myserver.net & service.myserver.net.
    2. You would like to secure all 3 hostnames with 1 certificate.
    3. Let’s Encrypt generates a hash file to be placed in a special directory in the webroot of the server(s) that answer to each of those hostnames.  So, it should map out like this:
      File
      Host/Server
      LE checks:
      FileA
      myserver.net
      http://myserver.net/.well-known/acme-challenge/FileA
      FileB
      www.myserver.net
      http://www.myserver.net/.well-known/acme-challenge/FileB
      FileC
      service.myserver.net
      http://service.myserver.net/.well-known/acme-challenge/FileC
    4. Certbot will create the required special hidden directory .well-known/acme-challenge/ in your webroot and place FileA, FileB and FileC in that directory.
    5. Let’s Encrypt will use DNS and query the server that is supposed to answer each hostname.  As shown in the “LE checks” column in the table in Step 3, it will look for a specific file on each server.
    6. Your webserver will answer for each hostname and steer the query to your webroot and to that special hidden folder Certbot created, find the requested file and thus, confirm to Let’s Encrypt that DNS points to the right server and you have the authority to place files there — so presumably you’re at least an administrator and can be trusted.
    7. The certificate is issued.
    8. Certbot cleans up by removing FileA, FileB and FileC.

Ok, so now let’s make all that happen…

Install Certbot

The default Debian repo has an older but still very functional and safe version of Certbot… so we can go ahead and just use that to make installation super simple:

apt-get install certbot

Now, let’s get the directory structure in your WEBROOT set up and secured for Certbot to work with, replacing stuff in red with settings appropriate to your setup:

mkdir webroot/letsencrypt
chown root:webuser webroot/letsencrypt
chmod 775 webroot/letsencrypt

You can get the correct settings for your webroot and webuser by checking the configuration files for your webserver (e.g. /etc/nginx/nginx.conf).  For a default Debian NGINX installation, for example, you’d set up the directory structure like this:

mkdir /usr/share/nginx/html/letsencrypt
chown root:www-data /usr/share/nginx/html/letsencrypt
chmod 775 /usr/share/nginx/html/letsencrypt

So, we’ve created a directory within our webroot for Certbot to use and granted root and our webserver accounts permission to read/write/execute within that directory.  Now Certbot can place the verification files needed by Let’s Encrypt to issue our certificates and delete them when the process is completed.

Webserver setup

Certbot creates a ‘.well-known/acme-challenge/’ directory within your ‘webroot/letsencrypt directory to store generated verification files.  Calls to all domains to be verified have this path appended to them so the Let’s Encrypt server can see the generated file and verify it, thus confirming that your server answers to that particular DNS call.  So, a call to ‘my.server.com’ would result in ‘my.server.com/.well-known/acme-challenge/filename’ being queried.  Obviously, in this case, your webserver needs to redirect calls to ‘<whatever>/.well-known/acme-challenge/…’ to ‘webroot/letsencrypt/.well-known/acme-challenge/…’.  For NGINX, we’d use a location block something like this in our server block:

# process LetsEncrypt requests
location ^~ /.well-known/acme-challenge {

# setup logging to review certificate requests
access_log   /var/log/nginx/LetsEncrypt_access.log main;
error_log   /var/log/nginx/LetsEncrypt_error.log warn;

default_type   text/plain;
root   /webroot/letsencrypt;
autoindex   on;

}

I don’t use Apache, but I imagine you’d use something quite similar.  If you use Apache, let me know in the comments section and I’ll add your location block here to help other people and give you credit! 🙂  The above section uses a regex match and says that any location URI that starts with ‘/.well-known/acme-challenge’ should be read from our ‘webroot/letsencrypt directory with the URI path appended… in other words, ‘webroot/letsencrypt/.well-known/acme-challenge/…

Get certificates

While Certbot has an automated mode for Debian/NGINX setups, I find that it requires a generic setup of your NGINX vserver configuration files and I don’t use standard setups.  So, I’m going to use a more explicit command with Certbot to tell it exactly what to do.  Here’s the command, followed by what it all means:

certbot certonly ––webroot –w webroot/letsencrypt/ –d domain.tld,www.domain.tld,subdomain.domain.tld ––rsa-key-size 4096 ––agree-tos –m [email protected]
  • certonly: This tells Certbot to obtain/renew the certificate, but don’t install it (i.e. we want to modify our webserver configuration manually)
  • ––webroot: Place the verification files in a webserver’s root directory or path as specified by the –w parameter
  • –w: Path to the webroot directory to be used (required because of ––webroot)
  • –d: Comma separated list of domains for which to request a certificate (subject-alternate-names)
    Warning

    The supplied list is processed in order!  So you should put your root domain first so it is listed as the primary, or ‘common’, name for your certificate.  In other words, you should use ‘example.com,www.example.com,subdomain.example.com’ so that ‘example.com’ is considered the primary identifier for your certificate.  It makes your administrative life easier.

  • ––rsa-key-size: Number of bits for the RSA key (2048 is default, modern computer power makes 4096 more reasonable)
  • ––agree-tos: Agree to the Let’s Encrypt/ACME terms of service without a prompt
  • –m: Email address for email notifications from Let’s Encrypt/ACME

So, in plain(ish)-English: Get me a 4096-bit RSA certificate to secure domain.tld, www.domain.tld and subdomain.domain.tld.  Verify that this server is actually the destination endpoint of the DNS records of those domains by verifying some files in my webroot/letsencrypt folder.  Also, know that I agree to the license terms and you may contact me at the provided email address with any important information.

Let’s Encrypt directory structure

Your certificate and key are stored in /etc/letsencrypt/archive/your.domain.tld/ and are symlinked to /etc/letsencrypt/live/your.domain.tld/.  You must ALWAYS reference the symlinked copies since they are the up-to-date renewed certificates.

Set renewal parameters

Assuming all went well, you already have certificates for the domains you requested in the step above and renewal has been setup automatically (we’ll confirm that soon).  However, there is one big problem… NGINX and other webservers will NOT read a renewed certificate unless it’s reloaded.  Wouldn’t it be great if Certbot could restart our webserver (NGINX in my examples) for us???  It can…

We need to set a post-renewal ‘hook’.  However, we don’t want to wait until our certificates require a renewal in 90 days since, if you’re like me, you’ll very much forget to do it.  So let’s force the issue and take care of it now.

certbot renew ––force-renewal ––renew-hook “systemctl restart nginx.service”

This is pretty self-explanatory and, obviously, if you’re using another webserver you have to alter the hook command as appropriate.  For example, “systemctl restart apache2.service” for Apache, etc.

Warning

If you read the Certbot documentation, you’ll notice that ‘––renew-hook’ has been deprecated in favour of ‘––deploy-hook’.  This is true of all versions 0.21.1+.  The version (as of this post) that is in the Debian Stretch repo is only 0.10.2 and that’s why we are using the old command.

After running the above command, you should see that it was successful.  It will also say that it ran the renew-hook command though it was deployed ‘without reload’, which is confusing since our hook is a service restart/reload… so let’s see what’s really happening:

systemctl status nginx.service

You’ll see that systemd will report that the NGINX service was indeed restarted a few seconds ago, so all is good.  Now we have to check that our renew-hook has been written to the renewal configuration file:

cat /etc/letsencrypt/renewal/your.domain.tld.conf

Check the ‘[renewalparms]’ section, it should look something like this:

account = d7j398dlke93kdhf93kd93dl903jf30d
authenticator = webroot
rsa_key_size = 4096
installer = None
renew_hook = systemctl restart nginx.service

Ensure that ‘renew_hook’ line is correct (change it if needed) and double-check the ‘rsa_key_size’ value too.  Assuming those are all good, you’re all set.  One more test to ensure everything is ok:

certbot renew ––dry-run

This will actually generate a renewal request and domain re-verification, but will not process/create new certificates.  It should complete successfully and mention that it is “skipping renewal hook command” which lets you know that it’s aware of the command’s presence in the configuration file.

Renewal timers

Certbot is called twice per day (12:00am and 12:00pm by default) to check the status of your certificates and if they need to be renewed.  If they are near expiration (within 30 days) then they will be automatically renewed.  This is accomplished via a cronjob or systemd timer.

cronjob

If you check out /etc/cron.d/certbot, you’ll see the following:

0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e ‘sleep int(rand(3600))’ && certbot -q renew

This means that every 12 hours at midnight and noon, check to see if Certbot is installed and that systemd is NOT installed.  If those conditions are met, run Certbot in renew mode quietly.  If your system is not using systemd, then this is how your renewals will be processed so be aware of this cronjob!

systemd

On most current Debian/Ubuntu systems, you are using systemd and thus, the cronjob above will not run.  Instead, there is a systemd service and timer installed.  Let’s just verify all is good with those:

systemctl list-timers

You should see ‘certbot.timer’ listed.  Now, let’s make sure the actual files that define this timer and it’s accompanying service exist too:

ls -lA /lib/systemd/system/certbot*

You should see both certbot.timer and certbot.service listed.  Let’s check the status of these components:

systemctl status certbot.service   #should show loaded but dead
systemctl status certbot.timer   #should show loaded and active(waiting)

You should also ensure that when checking the status of the ‘certbot.timer’, that the ‘Loaded:’ line lists the timer as ‘enabled’ otherwise it will not be automatically started after a reboot.  If it’s showing ‘disabled’, then run

systemctl enable certbot.timer

Using your certs

Now that everything is set up, it’s time to actually direct your webserver to use your certificates.  Remember, Certbot stores the active (i.e. current) certificate as a symlink in the ‘live’ subdirectory.  So you just have to let your webserver know that’s where it should be looking.  If you’re using NGINX, you’d add the following to the server configuration:

ssl_certificate /etc/letsencrypt/live/sub.domain.tld/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/sub.domain.tld/privkey.pem

By referencing the symlinked ‘live’ files, you do not have to update this configuration as your certificates are renewed, etc.  As a side note, if you need to reference only the certificate chain (such as for OSCP), that file is located at ‘/etc/letsencrypt/live/sub.domain.tld/chain.pem’.

Final thoughts…

That’s it!  Pretty simple, just a lot of double-checking since it’s a pretty important service and easy to miss simple configuration steps.  You now have free, trusted SSL certificates for your server on any/all domains and those certificates will renew themselves and work without you having to think about them.  Pretty cool, right? Thanks for reading my techie-thoughts on this issue. Have any comments? Suggestions? Want to add your tips? Things you want me to cover in a future article? Comment below!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Close Menu