Nov 11
How-To Guides David
alias, backup, backups, cloud, computing, curl, download, e-mail, email, flickr, forwarding, gmail, google, hotmail, scripts, upload, webmail, wget, yahoo
When cloud computing goes bad
Cloud computing is great isn’t it? We can have all our data stored somewhere on the interwebs, accessible to us from any computer or mobile phone. I’m picturing it now, a white fuzzy-bordered soft-focus video with shampoo-advert beautiful people gather around a computer smiling and laughing at work and at home. Yes, I’m being sarcastic, but cloud computing definitely has its advantages, especially given that OSX and Linux are nibbling away at Window’s desktop market share. We’re in more need than ever for cross-platform and platform agnostic software solutions.
So, what’s the catch? Well, if you exercise due diligence on which services to use, then there are not really any significant catches. But there lies the rub, I know most people won’t bother with due diligence, they’ll just sign up to that free email/calendar/documents service and give little thought to what happens when that services breaks down or (wrongly) decides you’re doing naughty things and locks you out, or even worse, disappears from the internet forever. Think about that for a moment, it could be a disaster for you.
The most important thing to ensure with any cloud computing service (e.g. email, calendar, documents, etc.) is to make sure you can get a copy of your own data. If your data only exists in the cloud, then you don’t have your data. If your cloud service ever has a catastrophic failure, goes out of business or locks you out of your account, then you loose access to all of your data. If you have a backup copy, then you’re already in a position to set up an account elsewhere and upload your data. Of course, your cloud service should do their own backups, but downloading your own copy is like having an off-site backup too.
If you use any of the Google services then you should take a look at dataliberation.org. It is a site ran by Google employees, and collects all the information you need on how to extract your data from Google’s cloud services like Gmail, Calendar and Documents, etc. If you use some other cloud service, then it’s up to you to investigate how to get a copy of your data from their servers. If you can’t, then stop using them and find someone else. Being able to manually download backups is okay, but what you really want is to automate backup downloads. This is where command line applications like wget and cURL can come in handy for running scheduled scripts. Email is no different. I recently blogged about how to set up an automated email backup. Although a reliable manual method is to set up Mozilla Thunderbird to keep offline copies of all your IMAP folders. What’s that? You don’t use IMAP? Really, get with the times, please!
Compute in the cloud with peace of mind
Talking of email, there is one extra thing you can do to protect yourself from loosing your email/webmail account. In the even that you loose access to your chosen webmail service, nobody can email you, and if you have to set up a new account you’ll have get in touch with everyone you know to share your new email address. This is an inconvenience for everyone. Therefore, the best way to protect yourself against this is to: first, never let anyone know the email address of your webmail service; and second, get yourself an email alias to redirect to your real email address. The easiest way to get an email alias is to register a domain name an set up email forwarding there. Some organisations offer email aliases as a curtosy to their members (e.g. the Institute of Physics offers user@iop.org aliases). Failing everything else, you could try out myhandle.com. Then, if your regular webmail service ever closes down, you can set up a new account and redirect your public email address to the new webmail account.
Let’s go over this again …
Cloud computing is great, so if you want to enjoy the advantages while avoiding the pitfalls, then just make sure you do the following:
- Make sure you know how to download regular backups of your data.
- Obtain an email alias to use as your public email address.
 | | Share this post |
Nov 9
How-To Guides David
backup, command line, cronjob, e-mail, imap, imapsize, mbox, offlineimap, python, restore, synchronise, thunderbird, ubuntu, windows, wine
OfflineIMAP in action
Some time ago I bemoaned the lack of a way to have my computer run an automated back up of my IMAP email. To summarise, I had a couple of point and click methods to back up my email. Although, these were both manual methods, which means I needed to remember. No matter how good my intentions, sometimes I’ll forget, so I wanted something I could automate. This naturally lead to me wanting something that would run on the command line, which I could hence run from a script.
I’m glad to say I finally discovered a Python program called OfflineIMAP that would do just this. It won’t run on Windows, but it will run on any POSIX type operating system, i.e. Linux, FreeBSD, and Max OS. My only complaint with the thing is that if you run it manually in a terminal, you can no longer type anything into that terminal – you have to shut down and re-open the terminal window. Also, if you want to run this without having to input your log-in details (i.e. unattended executions) then you have to have them stored in a clear-text configuration file.
Other than these niggles, it is perfect for my needs. I can run it from my cron scheduler, and it synchronises with my IMAP server. This is actually better than I wanted, as synchronisation means I can use this to upload any messages I’ve archived away from the server, but want to restore.
If you’re interested in trying this, I’ll just mention that I don’t directly call it from a cronojb, but I call it from a script, because of the above mentioned terminal bug. (And no, running it like this from the cron scheduler didn’t work, it had to be ran from a script).
sleep 2
/usr/local/bin/offlineimap&
echo ""
References:
| | Share this post |
Sep 23
How-To Guides David
backup, browser, export, febe, firefox, passwords, profile, save, speed
My recommended settings for FEBE (click to enlarge)
More people are starting to comment on how Firefox is becoming a slow browser, with all the Add-Ons and the time it take to start up. I have found this too. I think that over time your profile acquires lots of settings and little bits of code here and there, left over from add-ons you may or may not be still using. The best way to deal with this, as with many computer problems is to wipe out and start again. This doesn’t mean I suggest you loose all your saved passwords and bookmarks though.
When I did this with my Firefox profile (which was many years old), I did it the manual way. It didn’t take me long, but you would need to know what you’re doing. I have since found a Firefox Add-On called FEBE (install from here) which will automate the process for you. Although IMHO, you still need to give it some thought when you are changing its settings. You can back up so many things from your profile with this, that you could end up defeating the object of clearing house! Remember, our aim here, is it wipe out as much of your old profile as possible without loosing the things you really need, like your bookmarks, passwords and Add-Ons.
You can set a schedule for FEBE, set whatever suits you. I am trying out having it do a back up everytime I shutdown Firefox, this does slow things down though. You can set daily, weekly or monthly backups. Or even just schedule a reminder to tell you to do it manually.
My recommended settings for saving the most important things with FEBE are in the above screen shot.
Now, in the event that you need to rebuild your Firefox profile due to some sort of fault and malfunction, that is still in your FEBE backup, then you need to manually rebuild your profile. *Deep breath*, this is going to take a little while, but follow my instructions and you’ll get there.
| | Share this post |
Jun 29
Personal David
backup, e-mail, imap, imapsize, mbox, restore, thunderbird, ubuntu, windows, wine
Since I began using Ubuntu I have been taking great joy in automating the download of back up copies of the data I have stored on-line, e.g. the database and files for this very website. However, there is one back up job that has refused to be automated – backing up my e-mail. So this post isn’t going to be my usual how to type of post, it’s going to be outlining what I want to do, the problems that have stopped me getting there.
To start with, all of my e-mail is up in the cloud, because I use IMAP. I don’t want to change this, because with POP I would have to have all my e-mail only on my home computer. That’s no good for me trying to find something in my e-mail on my phone or the web when I’m away from home. Although, I trust no server, so I want to know that if my IMAP server goes boom for whatever reason, it’s okay because I have a copy of all my data at home ready to be uploaded to another server.
My favourite IMAP supporting E-Mail client is Thunderbird (because it’s open source and cross platform). Thunderbird has off-line storage for IMAP, which is great. All I have to do to get an up-to-date copy of my e-mail is load up Thunderbird and tell it to go into off-line mode. It will then download any messages it doesn’t have, and also delete any messages I’ve deleted on the server.
This is great, although there’s two problems. Firstly, there’s no way to automate this. Secondly, if for whatever reason you loose all the e-mail on your server, you can’t use your local copies to synchronise back to the server, using Thunderbird. Instead, Thunderbird only synchronises server to client. So if there’s nothing on the server, it’ll wipe your off-line copies.
Thunderbird does support upload of messages to IMAP servers, so I had thought that when needed to do an emergency restore of my e-mail, I could simply use my file manager, navigate to my Thunderbird profile, and copy my IMAP mail files, to Thunderbird’s “local folder” store. Although, for some reason Thunderbird can’t read these properly, and you see that after doing this, the message count is wrong (and some strange malformed messages appear), so this method will cause the loss of some messages.
I had to think of something else.
IMAPSize
If you Google around for “IMAP backup”, you will invariably come across the Windows application “IMAPSize”. This is a great program that I have used before. I even discovered that IMAPSize supports command line options, and if you can run something by command line, you can set up a schedule for automation! I should add here that I’m in Linux so I had to use IMAPSize through WINE, and I found that IMAPSize was very slow to download messages, and this might have been because of using it through WINE, I don’t know.
Performance aside, I hit a short coming of IMAPSize. While it does indeed perform incremental back up of your e-mail, it doesn’t actually remove local messages that have been removed on the server. I verified this by creating test messages, running IMAPSize backup, then going back to the server to delete and expunge them (just to be sure), then re-running the IMAPSize backup. After checking, I could see that the deleted messages were still in my local backup.
So, this is where I am, I am stuck manually backing up my e-mail every few days, by means of having Thunderbird perform the local synchronisation I require (i.e. matching all changes on the server). Then the only progress I made was that I now know that I have IMAPSize to upload/restore the messages that Thunderbird downloads.
Although, this is a two app solution that I have to do through the graphical user interface. If only IMAPSize would remove messages that are not present on server, and if I could get it to play nicely with Linux’s cron scheduler, I would be happy.
It’s amazing to me that there isn’t a command line program that does what I’m looking for. I’ve asked a lot of people, who know far more about such things than I do, but everyone seems to say the same thing to me, “Don’t know of one but it’d sure be useful“.
So, if you know of something that’ll do what I’m looking for, please let me know in the comments.
| | Share this post |
Jun 13
How-To Guides David
backup, server, upgrade, wordpress
Okay so what the heck happened to the site on Thursday night?
It all boiled down to running out of storage quota on my server. My server account had crept over its quota limit without me knowing. Things were still working like this, but since Wordpress 2.8 (the software that runs this website) came out on Thursday I ran the upgrade, and then we hit problems.
Anyway, lots of technical details later storage quota was extended and the site was put back as it was, and subsequently upgrade to WP 2.8.
Why am I telling you all this? Well the moral of the story is that no matter what you’re doing – keep backups of everything you do! Had I not had backups of this site, it would be trashed now. Moreover, had my backups been better organised, the site would have been up several hours sooner.
I’ve now improved my file backup procedures to be automatic so they’re always up to date. Fortunately, I was already using a Wordpress plugin which e-mailed me a daily backup of my database.
Make sure you keep your stuff backed up too.
| | Share this post |
Twitter
LinkedIn
Facebook
Google
Flickr
YouTube
RSS
