Howto migrate your email to google apps over the weekend
Today I’ve finally moved the emails for my domain ducea.com to google apps for domains. I’m probably one of the few people that still had their own email server these days, and I’m sure anyone would question why would I want to run that on my own server. And the answer to that is that I didn’t, but thought this migration would be more complicated and time consuming so I always put it in the back on my todo list. I wanted to do it for a long time, but never got to it.
Seems like lately I’ve moved everyone I could onto google apps; friends, clients, or even strangers I could easily convince them on how great it is to not worry about your email server and put this into the hands of someone like google; and all this for free. Then why did it take so long for me to move? Well, email is very important to my business and this is why a long time ago (too many years to remember) I’ve made the decision to serve it on my own dedicated server, instead of a cheap vps. This was the main reason I rented the server in a good hosting facility (started with ThePlanet and then moved to SoftLayer about 3 years ago) and was happy to pay for it to know that I have a reliable service and my email will be reliable also, and be sure that if I get an email from a client or some nagios alert that something is not working I will be getting it immediately as expected. I’ve been a big fan of imap and used that all the time so I can check in the emails from different locations and have a central place where the files are and can be easily backed up. As any sysadmin I ended up with a big .procmailrc file with many rules, where some of them are most certainly no longer needed (projects completed, etc.) and with a huge Maildir, as I like to save anything that might be useful in the future. Don’t get me wrong I hit delete probably 80% of the time, but over time this grew to something like 1.2G quite easy. I’m sure many people have much bigger mailboxes than this, but anyway…
With google apps being the best solution for any new startup or basically anyone that is ‘small’ enough to not need anything special (you loose some control of course when you put this on someone else’s infrastructure), I wanted to move my email there for sometime. But it seemed too complicated to move it easily, without downtime and keeping all the existing emails in the same way how I like to see them in my Thunderbird, and my procmail rules still working; I don’t like the labels stuff, and still live in a folder world, and even if occasionally I would go to the web interface, I prefer to use the imap client with my Thunderbird. Naively I tried to move the emails by creating the new account in thunderbird and dragging and dropping the emails to the new location. This works just fine for a small inbox (like my wife’s that had about 500 emails) but for mine it was always timing out and needed to be restarted; this was impossible to be done like that as you always had to see where it stopped and with many emails/folders after 2-3 tries I realized this was not the way to go ;) . Time to get more serious about this…
I sow the light when I found the script called imapsync. This is a script that basically does exactly what I needed and resume if it had any issue and it was obviously more reliable that any thunderbird copy. I installed this locally on the server, and it is basically just a perl script that syncronizes two imap accounts (not google apps specific). Once you download and uncompress it you will probably need some perl modules based on your system setup. In my case this was just Mail::IMAPClient: (you can see this by running perl -c imapsync):
perl -c imapsync
and I installed it from the debian packages with:
aptitude install libmail-imapclient-perl
after this imapsync was happy and I could install it with:
make install
(your setup might be different and you might need other modules like: Digest::MD5, Term::ReadKey, IO:Socket:SSL, Date::Manip, etc. see INSTALL for full details; you can install them using cpan or your system packages whatever you prefer).
Now that I had imapsync installed I was ready to play with this and test it out. I had to tune the command a little but in the end it looked like this:
imapsync --syncinternaldates --host1 <local_server> --port1 993 --ssl1 --user1 <user> --password1 <pass> --host2 imap.gmail.com --port2 993 --ssl2 --user2 <user@domain.com> --password2 <pass> --useheader 'Message-Id' --skipsize --noauthmd5 --reconnectretry1 1 --reconnectretry2 1
and a similar one for the sent folder (that needs a special rewrite rule for the different folder name):
imapsync --syncinternaldates --host1 <local_server> --port1 993 --ssl1 --user1 <user> --password1 <pass> --host2 imap.gmail.com --port2 993 --ssl2 --user2 <user@domain.com> --password2 <pass> --prefix2 '[Gmail]/' --folder 'INBOX.Sent' --regextrans2 's/Sent/Sent Mail/' --skipsize --noauthmd5 --reconnectretry1 1 --reconnectretry2 1
Depending on your mailbox size this will take a while… For me, the initial sync took about 10 hours (about 60,000 emails ~ 1.2GB):
Time : 37329 sec
Messages transferred : 60893
Messages skipped : 142
Messages deleted on host1: 0
Messages deleted on host2: 0
Total bytes transferred : 985592338
Total bytes skipped : 876862
Total bytes error : 0
Average bandwith rate : 25.8 Ko/s
After this I happily changed my MX record from a single mail.ducea.com line to something that looks more redundant like:
@ MX 10 ASPMX.L.GOOGLE.COM.
@ MX 20 ALT1.ASPMX.L.GOOGLE.COM.
@ MX 20 ALT2.ASPMX.L.GOOGLE.COM.
@ MX 30 ASPMX2.GOOGLEMAIL.COM.
@ MX 30 ASPMX3.GOOGLEMAIL.COM.
@ MX 30 ASPMX4.GOOGLEMAIL.COM.
@ MX 30 ASPMX5.GOOGLEMAIL.COM.
and everything is looking good.
Still, I want to be able to use backups to save my emails somewhere else (just in case) as I was doing until now with a simple rsync/tar/gz script. For this, I’ve implemented a solution based also on imapsync that copies back the emails in the reverse direction now from google apps to my own server where I can then backup them up as I want. I hope that this post will be useful to other people that have similar concerns as I had while moving their email to google apps. There is really no reason at this time to run your own email server, deal with spams, etc. when there is a great free solution like google apps.