Trouble migrating VCF address books

I have been shuffling my Contacts around devices for many years even before I started serving them via CardDAV (with DAViCal), but so far it has always worked.

For some reason it started amassing enormous amounts of duplicate entries until it grew so large that DAViCal didn't want to serve it anymore. I'm not sure how this happened, but it seems to have gone unnoticed for a while, i.e. my local (mostly phone) contacts were not syncing anymore.

Fixing

First I exported address books as VCF files from all my devices, and from the server, and collected them in one folder.

First I thought I just needed to remove duplicates but the more I tested the more I noticed how crufty these (plain text) VCF files were.

A short list:

• Line endings CR:LF => convert to CR
• Remove unneeded fields and corrupt lines
• Decode RFC 1521 MIME Quoted-Printable strings

Mostly I managed with my code editor's search & replace function but decoding the so-called quoted-printable strings to normal UTF-8 strings (which all my clients are happy to accept) required qprint and a little script:

sh
#!/bin/bash

# massage VCF contacts file with partial email-like QP encoding
# reads stdin, echoes to stdout

exec 2>"$0.log"

# dependency check
type qprint >/dev/null || exit 1

marker=";ENCODING=QUOTED-PRINTABLE"
remove=";CHARSET=UTF-8"
while read line; do
    [[ "$line" != *"$marker"* ]] && echo "${line/"$remove"/}" && continue
    beg="${line%"$marker"*}"
    end="${line#*"$marker"}"
    echo -n "${beg/"$remove"/}"
    echo "$end" | qprint -d | iconv -t "UTF-8//TRANSLIT"
done

Doing this with all my files produces relatively healthy VCF Contact lists (but please check them manually in any case!)

I can now merge them & remove duplicates. I found a java jar that does a good enough job, but it has a little bug, forgetting to put a newline before END:VCARD. In the end I decompiled it with jad, edited and recompiled with javac and jar. The result is here. It also uses the FN: field to check for duplicates - the original uses the N: field.

When importing the final unified and deduplicated addressbook to your CardDAV server make sure to remove the original collection first, then recreate it from scratch. Also make sure other devices are not currently syncing, and delete+recreate the collections there as well.

This proved particularly tricky on my Sailfish OS device; I had to completely remove the CalDAV/Card/DAV account and recreate it, and also make sure there were no local contacts stored.