Nov 9
How-To Guides David
backup, command line, cronjob, e-mail, imap, imapsize, mbox, offlineimap, python, restore, synchronise, thunderbird, ubuntu, windows, wine
OfflineIMAP in action
Some time ago I bemoaned the lack of a way to have my computer run an automated back up of my IMAP email. To summarise, I had a couple of point and click methods to back up my email. Although, these were both manual methods, which means I needed to remember. No matter how good my intentions, sometimes I’ll forget, so I wanted something I could automate. This naturally lead to me wanting something that would run on the command line, which I could hence run from a script.
I’m glad to say I finally discovered a Python program called OfflineIMAP that would do just this. It won’t run on Windows, but it will run on any POSIX type operating system, i.e. Linux, FreeBSD, and Max OS. My only complaint with the thing is that if you run it manually in a terminal, you can no longer type anything into that terminal – you have to shut down and re-open the terminal window. Also, if you want to run this without having to input your log-in details (i.e. unattended executions) then you have to have them stored in a clear-text configuration file.
Other than these niggles, it is perfect for my needs. I can run it from my cron scheduler, and it synchronises with my IMAP server. This is actually better than I wanted, as synchronisation means I can use this to upload any messages I’ve archived away from the server, but want to restore.
If you’re interested in trying this, I’ll just mention that I don’t directly call it from a cronojb, but I call it from a script, because of the above mentioned terminal bug. (And no, running it like this from the cron scheduler didn’t work, it had to be ran from a script).
sleep 2
/usr/local/bin/offlineimap&
echo ""
References:
Jun 29
Personal David
backup, e-mail, imap, imapsize, mbox, restore, thunderbird, ubuntu, windows, wine
Since I began using Ubuntu I have been taking great joy in automating the download of back up copies of the data I have stored on-line, e.g. the database and files for this very website. However, there is one back up job that has refused to be automated – backing up my e-mail. So this post isn’t going to be my usual how to type of post, it’s going to be outlining what I want to do, the problems that have stopped me getting there.
To start with, all of my e-mail is up in the cloud, because I use IMAP. I don’t want to change this, because with POP I would have to have all my e-mail only on my home computer. That’s no good for me trying to find something in my e-mail on my phone or the web when I’m away from home. Although, I trust no server, so I want to know that if my IMAP server goes boom for whatever reason, it’s okay because I have a copy of all my data at home ready to be uploaded to another server.
My favourite IMAP supporting E-Mail client is Thunderbird (because it’s open source and cross platform). Thunderbird has off-line storage for IMAP, which is great. All I have to do to get an up-to-date copy of my e-mail is load up Thunderbird and tell it to go into off-line mode. It will then download any messages it doesn’t have, and also delete any messages I’ve deleted on the server.
This is great, although there’s two problems. Firstly, there’s no way to automate this. Secondly, if for whatever reason you loose all the e-mail on your server, you can’t use your local copies to synchronise back to the server, using Thunderbird. Instead, Thunderbird only synchronises server to client. So if there’s nothing on the server, it’ll wipe your off-line copies.
Thunderbird does support upload of messages to IMAP servers, so I had thought that when needed to do an emergency restore of my e-mail, I could simply use my file manager, navigate to my Thunderbird profile, and copy my IMAP mail files, to Thunderbird’s “local folder” store. Although, for some reason Thunderbird can’t read these properly, and you see that after doing this, the message count is wrong (and some strange malformed messages appear), so this method will cause the loss of some messages.
I had to think of something else.
IMAPSize
If you Google around for “IMAP backup”, you will invariably come across the Windows application “IMAPSize”. This is a great program that I have used before. I even discovered that IMAPSize supports command line options, and if you can run something by command line, you can set up a schedule for automation! I should add here that I’m in Linux so I had to use IMAPSize through WINE, and I found that IMAPSize was very slow to download messages, and this might have been because of using it through WINE, I don’t know.
Performance aside, I hit a short coming of IMAPSize. While it does indeed perform incremental back up of your e-mail, it doesn’t actually remove local messages that have been removed on the server. I verified this by creating test messages, running IMAPSize backup, then going back to the server to delete and expunge them (just to be sure), then re-running the IMAPSize backup. After checking, I could see that the deleted messages were still in my local backup.
So, this is where I am, I am stuck manually backing up my e-mail every few days, by means of having Thunderbird perform the local synchronisation I require (i.e. matching all changes on the server). Then the only progress I made was that I now know that I have IMAPSize to upload/restore the messages that Thunderbird downloads.
Although, this is a two app solution that I have to do through the graphical user interface. If only IMAPSize would remove messages that are not present on server, and if I could get it to play nicely with Linux’s cron scheduler, I would be happy.
It’s amazing to me that there isn’t a command line program that does what I’m looking for. I’ve asked a lot of people, who know far more about such things than I do, but everyone seems to say the same thing to me, “Don’t know of one but it’d sure be useful“.
So, if you know of something that’ll do what I’m looking for, please let me know in the comments.
Twitter
Facebook
LinkedIn
Google
Flickr
YouTube
RSS
