Condensing cron emails into a daily digest

(Note: This post is originally from 2017, and is copied to this new site for historical reasons. Opinions, network infrastructure, and other things may have changed since this was first written.)

If you have a non-Windows server on the Internet that does much of anything, you probably have a few cronjobs. And chances are, one or more of those cronjobs put output on standard input – maybe you’re using bash script tracing or something. You care about the output, but you don’t care enough to have a fresh email for every time that command runs (especially if it’s hourly, yeesh). The problem is, cron doesn’t support holding all your command output until a specific time, and then firing a big batch email. If it did that, it would break POSIX compatibility.

In my case, I also wanted to change my sender address. Sure, some cron daemons support MAILFROM, but not all, and apparently the cron daemon that comes with Ubuntu 16.04 is one of those that don’t.

So what you have to do instead is not change your cron daemon’s behavior, but change your crontabs.

Cronic’s influence

Cronic is a program that aims to solve a similar kind of problem: changing cron’s mail behavior. How Cronic achieves its goal is by presenting itself as a sort of “prefix command” – instead of running freshclam, you’d run cronic freshclam. What Cronic does, which is to only send email if something goes wrong (by it’s own definitions, see its site), isn’t particularly helpful for my own goal, so I’m not using it, but how it does it is perfect – instead of just running freshclam, I’d run /usr/local/bin/dbsr.py freshclam instead.

DBSR stands for “Daily Bullshit Report”, the subject line and top header for my daily Cron digest (and whatever else I choose to add to it later).

Outline

Given the following cronjob to focus on:

0 * * * * /usr/local/bin/dbsr.py fallocate -d /sparse/file

Every hour, on the hour, my script runs fallocate -d /sparse/file, to dig holes in a fictional sparse file on the server. If there’s any sort of output, or if the program returns something other than 0, a file is created in /var/cache/dbsr, containing the command name and a POSIX timestamp in the filename. All cronjobs run via dbsr.py have their output dumped in the same place.

Another cronjob exists as the “nobody” user on each server:

0 19 * * * /usr/local/bin/firedbsr.py

Every day, at 19:00 (all my servers are set to America/Los_Angeles, and I will not hear heresy about other time zones), the script runs, reads the contents of /var/cache/dbsr (assuming there are any; the script quits without sending mail if not), wraps all the command output in fancy monospace text boxes (using HTML mail, because I’m using Thunderbird to read my mail anyway, thanks Dovecot!), then sends it in one email.

Implementation details

Step 1: Output collection

dbsr.py is a simple script anyone with half an idea of how their favorite programming language works can implement.

• First, run the command, and collect the output (both stdout and stderr) into a variable.
• Then, dump the contents of that variable

My version, written in Python 3, which also includes the return code and command arguments, is below:

#!/usr/bin/env python3
# Copyright © 2017 Nicole O'Connor. Distributed under the terms of the MIT license.
# https://opensource.org/licenses/MIT

import datetime
import os
import shlex # for shlex.quote
import stat
import subprocess
import sys

# run whatever args are passed into it
# no checks are done, make sure you're not running "rm -fr /" in your crontabs
cmdargs = sys.argv[1:]

# launch it
cmd = subprocess.Popen(cmdargs,
        stdout = subprocess.PIPE,
        stderr = subprocess.STDOUT)

# get output of command
output = cmd.communicate()[0]

if not (output.strip() or cmd.returncode):
    sys.exit(0) # Don't bother appending to the DBSR if there's nothing to append

now = datetime.datetime.now()
timestamp = now.timestamp() # POSIX timestamp, seconds since the epoch

# if directory doesn't already exist, create it and make it writable by everyone
# might want to run "sudo dbsr.py some_command_with_output" on a new install
if not os.path.isdir("/var/cache/dbsr"):
    os.mkdir("/var/cache/dbsr")
    os.chmod("/var/cache/dbsr", # world read/write
        stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO)

# filename: /var/cache/dbsr/echo.123456789.1234
with open("/var/cache/dbsr/{0}.{1}".format(cmdargs[0], timestamp), "w") as outfile:
    # Header - looks like this:
    # Arguments: echo "I am a command line!"
    # Return value: 0
    # ---
    outfile.write("Arguments: " + ' '.join(shlex.quote(arg) for arg in cmdargs) + '\n')
    outfile.write("Return value: " + str(cmd.returncode) + '\n')
    outfile.write("---\n")

    # put command output in file
    outfile.write(output.decode())

Step 2: Sending everything we’ve collected

The script to actually compile the command output into a digest email is a bit more involved, because this is where (in my implementation) I do most of the actual heavy lifting.

• Python has classes in the standard library for building emails. Because I intended to send HTML mail, I needed to use a MIMEText class, and pass an extra argument to its constructor ("html")
• Because this is mail being passed around my internal network, I can skip SMTP (which is good, because my SMTP server is Kerberized, and I didn’t feel like trying to learn SMTP and keytabs in one go). Of the options I had, the one I could make the most sense of was piping it into /usr/sbin/sendmail.
• I need to keep the HTML template as a separate file from my actual script, just so I don’t go insane trying to maintain it. I like Jinja for my templating needs.

This is the template used to generate the email in the screenshot above. I don’t recommend actually using this template for yourself, because it’s not the pinnacle of web design by a long shot. But it’s a reference for what a valid HTML e-mail looks like.

{# vim: ft=jinja :
(This is a Jinja comment block)

Caveats: 
  * The Google results I found said HTML mail cannot use things like linked stylesheets - all styles must be inlined. While I haven't tried
    putting <style> tags in the head, or actually trying linked stylesheets, I'm going to assume you still can't.
  * There doesn't appear to be a solid consensus on whether or not to bother with <html> or <head> tags. One Stack Overflow answer said
    that mail clients will just throw out tags it doesn't want, so I'm including them and letting mail clients deal with them. It
    works in Thunderbird, so ¯\_(ツ)_/¯
#}

<html><head></head>
<body style="font: sans-serif 14pt;">
<h1>The Daily Bullshit Report</h1>
<span style="color: #666666;">for {{ fqdn }}, {{ today }}</span><br />

{% for command in commands %}
<div style="padding-left: 100px; padding-right:100px;">
<tt>{{ command.name }}</tt>, run {{ command.timestamp }}:<br />
<pre style="background-color: #373737; color: #c7c7c7; padding: 20px; border: 2px solid #373737; border-radius: 10px;">{{ command.output }}</pre>
</div>
{% endfor %}
</body>
</html>

And this is the script used to generate the email every night and send it. This one I’m more willing to encourage the use of. At the moment, it doesn’t bother making sure /var/cache/dbsr actually exists, so I don’t know what it would do if it doesn’t. I’m sure I’ll fix that later.

#!/usr/bin/env python3
# Copyright © 2017 Nicholas O'Connor. Distributed under the terms of the MIT license.
# https://opensource.org/licenses/MIT

import datetime
import os
import socket
import subprocess
from email.mime.text import MIMEText

import jinja2

fqdn = socket.getfqdn() # returns "servername.lavacano.net"
shorthost = fqdn.split(".")[0] # returns "servername"

mailfrom = "{0} <{1}@lavacano.net>".format(fqdn, shorthost) # "servername.lavacano.net <servername@lavacano.net>"
mailto = "lavacano@lavacano.net"
subject = "The Daily Bullshit Report"

commands = []

for item in os.listdir("/var/cache/dbsr"):
    # get the command and timestamp from the file
    filesplit = item.split(".")
    cmdname = filesplit[0]
    unix_timestamp = '.'.join(filesplit[1:]) # everything after the first dot (including subsequent dots) is assumed to be a valid float
    dt = datetime.datetime.fromtimestamp(float(unix_timestamp))

    cmd = {}
    cmd["name"] = cmdname
    cmd["timestamp"] = dt.strftime("%Y.%m.%d %H:%M:%S")

    with open("/var/cache/dbsr/" + item) as infile:
        cmd["output"] = infile.read()

    commands.append(cmd)
    os.remove("/var/cache/dbsr/" + item)

if not commands: # list is empty if nothing in directory
    sys.exit(0)  # nothing to do, exit without mail

jinja = jinja2.Environment(
    # tells jinja where our templates live
    loader = jinja2.FileSystemLoader("/usr/local/share/dbsr/templates")
)

mail_template = jinja.get_template("mail_template")

mailargs = ["sendmail", "-i", "-oi", mailto]

mailproc = subprocess.Popen(mailargs, stdin=subprocess.PIPE)
today = datetime.datetime.today()

# render template into MIMEText object
mailmessage = mail.template.render({
    "fqdn": fqdn,
    "today": today.strftime("%Y.%m.%d"),
    "commands": commands
})
mailmime = MIMEText(mailmessage, "html") # changes content type to HTML

# add From and Subject headers
# you can add any other headers the same way
mailmime["From"] = mailfrom
mailmime["Subject"] = subject

# render jinja into stdin of sendmail process
mailproc.communicate(mailmime.as_string().encode())