Check for unread email in all subscribed folders

Table of Contents

Intro
Openssl's s_client program
Create a coprocess
The script
Resources used

Intro

I want to do this on the command line, maybe set up a timer once it works, and display the results via notification and in conky.

It's possible to connect to your mail server with curl but:

there isn't a single command that can achieve that, I need to get a response from the server and send subsequent commands that depend on it
as far as I can see I cannot interact like that through curl
I don't want to send multiple short-lived login commands to my mail server

Openssl's s_client program

Similarly to netcat or telnet, I can log in to the server, then send and receive commands, but securely over SSL/TLS.

For an encrypted connection right from the start:

$> openssl s_client -connect servername:993 -crlf -quiet

For STARTTLS after connecting:

$> openssl s_client -connect servername:143 -starttls imap -crlf -quiet

You will see something like this:

depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = GeoTrust EV RSA CA 2018
verify return:1
depth=0 .....
verify return:1
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN] Dovecot ready.

Now you can login with a login user password, and will receive a reply:

a login user@servername very-secret-password
a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE SEARCH=FUZZY COMPRESS=DEFLATE QUOTA] Logged in

Now there's lots of fun. Links with helpful resources are further down.

Without re-typing the whole communication, if we want to check all subscribed folders, we have to

list all subscriptions with a lsub "" "*"
for each returned folder: a. a select INBOX.somefolder b. a search (unseen) c. count the numerical ids in the SEARCH reply
Save that number, or possibly continue retrieving the subject and other data
and finally, a logout

But we do not have scripting powers within the connection; what we need is to read its output in a script, and be able to send commands in reaction.

Create a coprocess

In Bash, we can use coproc for this(help coproc).

Once you are confident that the manual communication from the previous chapter is working, remember the commands, and try this:

$> coproc SSL { openssl s_client -connect servername:993 -crlf -quiet; }

This gives us an array of which the first two elements are the processes stdout and stdin, so we can send commands to >&${SSL[1]} and listen to <&${SSL[0]}.

Knowing that each response ends with a (that's a tag we chose to start our commands with, we could have chosen X or msg00001, and the server would respond with that), it is easy to establish a communication with scripting powers.

The script

sh
#!/bin/bash

# password previously encrypted:
# openssl enc -aes-256-cbc -iter 99999 -a -pass file:/etc/machine-id -out "$passfile"

sep="################################################"
server="some.server"
user=john
me="${0##*/}"
tag=Zaphod
passfile="$HOME/.local/share/$me.txt"

debug=0
debug() {
    [[ "$debug" != 1 ]] && return
    (($# > 1)) && printf "$@" || echo "$1"
}
remCR() {
    # removes carriage returns from output
    # works directly on $out
    out="${out//$'\r'/}"
}
mimedecode() {
    # only if Perl & its Encode module are available
    # https://superuser.com/a/972248
    # works directly on $out
    x="$(perl -CS -MEncode -e "print decode('MIME-Header', '$out')" 2>/dev/null)"
    [ -n "$x" ] && out="$x"
}
# Start the session
coproc SSL { openssl s_client -connect "$server":993 -crlf -quiet 2>/dev/null; } || exit 1

# Make sure openssl is dead when exiting
trap 'kill $SSL_PID 2>/dev/null' EXIT

# LOGIN
line="$tag login $user@$server"
echo "$line $(openssl enc -aes-256-cbc -iter 99999 -a -pass file:/etc/machine-id -d -in "$passfile")" >&${SSL[1]}
debug "$line super_secret_password"

# Catch output loop. Remove the printf if you want it quiet
while read -r out; do
    debug '%q\n' "$out"
    # This reply tells us the server is done and waiting for more input
    [[ "$out" == "$tag "* ]] && break
done <&${SSL[0]}
debug "$sep"

# Now we're talking to the server
echo "$tag lsub INBOX \"*\"" >&${SSL[1]}
debug "$tag lsub INBOX \"*\""

# Catch output loop. Remove the printf if you want it quiet
subs=(); count=0
while read -r out; do
    debug '%q\n' "$out"
    # This is the first stepof the info we're after
    [[ "$out" == '* LSUB '* ]] && {
        out="${out#*\".\" }"
        remCR # remove carriage return!
        subs[count++]="$out"
        continue
    }
    # Again, this reply tells us the server is done and waiting for more input
    [[ "$out" == "$tag "* ]] && break
done <&${SSL[0]}
debug "$sep"

# Now we're talking to the server
declare -A unseen=()
for i in "${subs[@]}"; do

    # 1. select folder
    echo "$tag select $i" >&${SSL[1]}
    debug "$tag select $i"

    # Catch output loop.
    found=0
    while read -r out; do
        debug '%q\n' "$out"
        # no need to search _each_ folder for unseen messages!
        [[ "$out" == '* OK [UNSEEN '* ]] && found=1
        [[ "$out" == "$tag "* ]] && break
    done <&${SSL[0]}

    # 2. search for unseen mail
    if [[ "$found" == 1 ]]; then
        echo "$tag search (unseen)" >&${SSL[1]}
        debug "$tag search (unseen)"

        # Catch output loop.
        while read -r out; do
            debug '%q\n' "$out"
            # 3. if SEARCH is followed by a space there are unseen messages
            [[ "$out" == '* SEARCH '* ]] && {
                out="${out#*SEARCH }"
                remCR
                # 4. asssign to array
                unseen[$i]="$out"
                continue
            }
            [[ "$out" == "$tag "* ]] && break
        done <&${SSL[0]}
    fi
done
debug "$sep"

final=""
# now also get subject lines
for x in "${!unseen[@]}"; do
    # select folder
    echo "$tag select $x" >&${SSL[1]}
    debug "$tag select $x"
    while read -r out; do
        debug '%q\n' "$out"
        # This reply tells us the server is done and waiting for more input
        [[ "$out" == "$tag "* ]] && break
    done <&${SSL[0]}
    # make array from numerical mail ids
    arr=(${unseen[$x]})
    final="${final}\nFolder: $x"
    for i in "${arr[@]}"; do
        echo "$tag fetch $i body.peek[header.fields (subject)]" >&${SSL[1]}
        debug "$tag fetch $i body.peek[header.fields (subject)]"
        # Catch output loop.
        while read -r out; do
            debug '%q\n' "$out"
            [[ "$out" == "Subject: "* ]] && {
                out="${out#Subject: }"
                remCR
                mimedecode
                final="$final\n - $out"
                continue
            }
            [[ "$out" == "$tag "* ]] && break
        done <&${SSL[0]}
    done
done
final="${final#\\n}"

# That's all we wanted. Logging out
echo "$tag logout" >&${SSL[1]}
debug "$tag logout"
while read -r out; do
    debug '%q\n' "$out"
    [[ "$out" == "$tag "* ]] && break
done <&${SSL[0]}
# at this point the openssl coproc should quit.
debug "$sep"

echo -e "$final"

Note that the server replies with \r carriage returns that need to be removed.

The script is entirely anecdotal: it works with my mail provider's server. There's hardly any quality control here, and I don't currently know enough about the IMAP protocol to provide it.

I suspect making all queries case-insensitive would be a good first step to make this more robust.

If you set debug=1 you will see the whole communication.

Resources used

https://www.atmail.com/blog/imap-101-manual-imap-sessions/
https://www.atmail.com/blog/imap-commands/
https://www.atmail.com/blog/advanced-imap/
https://stackoverflow.com/a/14960067
https://www.openssl.org/docs/man1.0.2/man1/s_client.html
https://www.geeksforgeeks.org/coproc-command-in-linux-with-examples/
https://superuser.com/a/972248