HowTo Migrate to Chef 11

5-minute read
chef • chef11 • opschef • knife • devops • github

Chef 11 was released earlier in February and it is awesome! Like most people, I love the new features like partial search, chef-apply and knife-essentials inclusions, awesome formatted output, etc. Of course the open source chef 11 server was rewritten completely in erlang with postgresql/mysql support replacing the ruby/couchdb backend stack. solr and rabbitmq are still there ;)… There are many breaking changes meaning you will want to make sure that you fix your cookbooks before upgrading.

When you are ready to upgrade, you will notice that unfortunately there is no official migration path. This howto will document what I’ve used myself for such migrations and hopefully will help you too if you are trying to perform a similar upgrade.

Opscode has done an amazing job with the omnibus installers and starting with Chef 11, the chef server has support for this also. Meaning you can install a new chef server simply by installing the rpm or deb for your platform and everything should be installed for you (ruby/gems, chef, rabbitmq, solr, erlang, postgresql, nginx). Just head over to http://www.opscode.com/chef/install/ and from the chef-server tab download the version for your OS.

In order to migrate to a new chef server we need to migrate from the old server:

  • clients
  • nodes
  • roles
  • environments
  • data bags
  • cookbooks (with all the versions used in each environment)

It is important to have all the clients with their proper public keys because if not we would have to re-register each one of them.

Personally, I’ve migrated using this process several servers from open source chef 0.10.x to chef 11, but theoretically this should work from any chef server implementation (hosted, private, etc.) because we are downloading and uploading the assets using the api calls.

Backup the data from the existing server

You can use my knife-backup plugin for this. Once you install the gem you can just run it and it will backup all the objects from the existing server:

gem install knife-backup
knife backup export

This might take a while depending on your number of nodes/clients, cookbooks, etc. you have. Once completed you will have in .chef/chef_server_backup all the needed files.

Optional: if you have many unused cookbook versions you might want to clean them first before the backup. You can use my knife-cleanup plugin for this:

gem install knife-cleanup
knife cleanup versions -D

Install the new Chef 11 server

I would recommend to setup a new server as this would be the safest approach in case something doesn’t work out well and you don’t have to mess with your current environment. As mentioned earlier you can install the new server very easy with the omnibus installer. For example for Ubuntu 12.04 this would look like:

wget https://opscode-omnitruck-release.s3.amazonaws.com/ubuntu/12.04/x86_64/chef-server_11.0.6-1.ubuntu.12.04_amd64.deb
dpkg -i chef-server*
sudo chef-server-ctl reconfigure

You can also use the chef-server cookbook to install your new server if you prefer that.

Once you have the new chef server up and running, you will need to setup a new admin account and a new knife config. I would recommend to use a special user for this to not interfere with the users that we are trying to import from the old server. I would call it transfer. From the local server this would look like:

mkdir -p ~/.chef
sudo cp /etc/chef-server/chef-webui.pem ~/.chef/
sudo cp /etc/chef-server/chef-validator.pem ~/.chef/

marius@chef:~# knife configure -i
WARNING: No knife configuration file found
Where should I put the config file? [/marius/.chef/knife.rb]
Please enter the chef server URL: [http://localhost:4000] https://localhost
Please enter a clientname for the new client: [transfer]
Please enter the existing admin clientname: [chef-webui]
Please enter the location of the existing admin client's private key: [/etc/chef/webui.pem] ~/.chef/chef-webui.pem
Please enter the validation clientname: [chef-validator]
Please enter the location of the validation key: [/etc/chef/validation.pem] ~/.chef/chef-validator.pem
Please enter the path to a chef repository (or leave blank):
Creating initial API user…
Created client[transfer]
Configuration file written to /marius/.chef/knife.rb

Note: the default server keys will be located in /etc/chef-server/ and not in /etc/chef like they used to be, and this is definitely a welcome change. Also the default server url will still look for http and port 4000, but with chef 11 this works behind a nginx load balancer and listens by default on standard https port.

Restore the data from the old server

Finally, now we can restore all the data from the old server. You can transfer it from the backup and for simplicity drop it in your user .chef folder under .chef/chef_server_backup; be sure to install the knife-backup gem to the server and you should be able to run:

marius@chef:~# knife backup restore
WARNING: This will overwrite existing data!
WARNING: Backup is at least 1 day old
Do you want to restore backup, possibly overwriting exisitng data? (Y/N) y
Restoring clients
...

And this should restore all the data in the new server. Final step would be to regenerate the indexes:

chef-server-ctl reindex

Note: I want to point out that currently knife-backup will skip any clients that already exist on the server as I could not find a way to overwrite them using the api calls. This means that most certainly the validation key will need to be changed as that is a user that for sure will exist on the newly installed server.

Final touches

After the data migration is completed you will probably just have to point your DNS alias to the new server. One issue I’ve noticed is that the chef server when installed will use the local dns record in various places in its config files. When working on a temporary server this has caused problems once changing the dns and activating the server. The chef server will send to the client links from where to download the assets (cookbook parts for ex) and if this was unconfigured at install time then you might have to fix it and correct it to the dns entry the clients can download correctly; check it out:

grep s3_url /var/opt/chef-server/erchef/etc/app.config

and restart the chef server after correcting the s3_url:

chef-server-ctl restart

Hopefully this post will help you migrate to Chef 11. Feel free to let me know in the comments bellow if you had any issues following this process, or if it worked without any problems. Also if you find any problems with the tools used here knife-cleanup or knife-backup please open a ticket on github or submit a patch. Good luck!

comments powered by Disqus