Using Vagrant to create KVM based vms to test Ceph

Ceph, the software defined storage solution has been growing in popularity especially as a cloud storage platform. To learn more about this product, I purchased the book 'Mastering Ceph: Redefine your storage system' by Nick Fisk.

The first hurdle when using the book was that the examples provided in the book rely on using Vagrant with Virtual box to create test machines which themselves were running Ubuntu. I use a Fedora 29 machine and would like to use CentOS on KVM instead for my setup.

To get the test machines setup for my environment, I've had to deviate from the instructions given in the book. This post is based on my notes. These notes should also benefit those who are just looking to use Vagrant with KVM.

First install Vagrant and the libvirt plugin for vagrant.

sudo dnf install -y vagrant vagrant-libvirt

I use the directory ~/vagrant/ceph as the location for the Vagrantfile for my test machines.

My Vagrant file is as follows

storage_pool_name = "vagrant-pool"

nodes = [
  { :hostname => 'ansible', :ip => '192.168.145.40', :box => 'centos/7' },
  { :hostname => 'mon1', :ip => '192.168.145.41', :box => 'centos/7' },
  { :hostname => 'mon2', :ip => '192.168.145.42', :box => 'centos/7' },
  { :hostname => 'mon3', :ip => '192.168.145.43', :box => 'centos/7' },
  { :hostname => 'osd1', :ip => '192.168.145.51', :box => 'centos/7', :ram =>1024, :osd => 'yes' },
  { :hostname => 'osd2', :ip => '192.168.145.52', :box => 'centos/7', :ram =>1024, :osd => 'yes' },
  { :hostname => 'osd3', :ip => '192.168.145.53', :box => 'centos/7', :ram =>1024, :osd => 'yes' },
]

Vagrant.configure("2") do |config|
  config.ssh.forward_agent = true
  config.ssh.insert_key = false
  config.ssh.private_key_path = ["~/.vagrant.d/insecure_private_key","~/.ssh/id_rsa"]
  config.vm.provision :shell, privileged: false do |s|
    ssh_pub_key = File.readlines("#{Dir.home}/.ssh/id_rsa.pub").first.strip
    s.inline = <<-SHELL
      echo #{ssh_pub_key} >> /home/$USER/.ssh/authorized_keys
      sudo mkdir -m 0700 /root/.ssh/
      sudo bash -c "echo #{ssh_pub_key} >> /root/.ssh/authorized_keys"
    SHELL
  end

  nodes.each do |node|
    config.vm.define node[:hostname] do |nodeconfig|
      nodeconfig.vm.box = node[:box]
      nodeconfig.vm.hostname = node[:hostname]
      nodeconfig.vm.network :private_network, ip: node[:ip]

      memory = node[:ram] ? node[:ram] : 512;
      nodeconfig.vm.provider :libvirt do |lv|
        lv.storage_pool_name = storage_pool_name
        lv.driver = "kvm"
        lv.uri = "qemu:///system"
        lv.memory = memory
        lv.graphics_type = "none"
        if node[:osd] == "yes"
          lv.storage :file, :size => '5G'
        end
      end
    end
  end
end

Going through this step-by-step

    storage_pool_name = "vagrant-pool"

I use a variable storage_pool_name to store the name of the storage pool name in libvirt. This pool is created as a 'Filesystem Directory' on my laptop which has been named "vagrant-pool".

nodes = [
  { :hostname => 'ansible', :ip => '192.168.145.40', :box => 'centos/7' },
  { :hostname => 'mon1', :ip => '192.168.145.41', :box => 'centos/7' },
  { :hostname => 'mon2', :ip => '192.168.145.42', :box => 'centos/7' },
  { :hostname => 'mon3', :ip => '192.168.145.43', :box => 'centos/7' },
  { :hostname => 'osd1', :ip => '192.168.145.51', :box => 'centos/7', :ram =>1024, :osd => 'yes' },
  { :hostname => 'osd2', :ip => '192.168.145.52', :box => 'centos/7', :ram =>1024, :osd => 'yes' },
  { :hostname => 'osd3', :ip => '192.168.145.53', :box => 'centos/7', :ram =>1024, :osd => 'yes' },
]

This is an array or dictionaries containing details of the machines to be setup.

    Vagrant.configure("2") do |config|

    ..

    end

The main configuration block used by Vagrant.

I then initialise common settings for all the test machines. I've commented inline.

      #Use my personal ssh key across the test machines without having to copy my private key to the test machines.
      config.ssh.forward_agent = true

      #Do not regenerate a new key for each test machine.
      config.ssh.insert_key = false

      # The private keys I use. This is used for "vagrant ssh".
      config.ssh.private_key_path = ["~/.vagrant.d/insecure_private_key","~/.ssh/id_rsa"]

      #This block reads plublic keys and appends it to .ssh/authorized_keys for the user and root account.
      config.vm.provision :shell, privileged: false do |s|
        ssh_pub_key = File.readlines("#{Dir.home}/.ssh/id_rsa.pub").first.strip
        s.inline = <<-SHELL
          echo #{ssh_pub_key} >> /home/$USER/.ssh/authorized_keys
          sudo mkdir -m 0700 /root/.ssh/
          sudo bash -c "echo #{ssh_pub_key} >> /root/.ssh/authorized_keys"
        SHELL
      end

We then iterate through the list of nodes in the nodes array setting up a test box for each node with the described features in the block.

  #For each node in the nodes array.
  nodes.each do |node|
    #define a new test box with name :hostname.
    config.vm.define node[:hostname] do |nodeconfig|
      #This is set to centos/7 for all the nodes.
      nodeconfig.vm.box = node[:box]
      #This is the name to be set.
      nodeconfig.vm.hostname = node[:hostname]
      #Set an ip address given in the private network.
      nodeconfig.vm.network :private_network, ip: node[:ip]

      #If a value has been provided for ram, we use that or default to 512M
      memory = node[:ram] ? node[:ram] : 512;
      #We configure testmachine in libvirt
      nodeconfig.vm.provider :libvirt do |lv|
        #This is set to "vagrant-pool" we created earlier
        lv.storage_pool_name = storage_pool_name
        #Use KVM.
        lv.driver = "kvm"
        lv.uri = "qemu:///system"
        lv.memory = memory
        lv.graphics_type = "none"

        #Create a new storage of 5G if it is an OSD.
        if node[:osd] == "yes"
          lv.storage :file, :size => '5G'
        end
      end
    end
  end

To create the test machines, we simply call

$ vagrant up

We can also call up individual machines by providing a list of names.

$ vagrant up mon1 osd1

On first run, vagrant will download an image of CentOS 7. Subsequent runs will be faster.

We can suspend and resume using the commands

$ vagrant suspend
$ vagrant resume

This ensures that the virtual machines are available later when you get back to it.

When we are done and no longer need the machines, we can use the following command to stop and delete them.

$ vagrant destroy

As with "vagrant up", we can provide machine names.

To complete the setup, I add the following to my /etc/hosts file.

#Vagrant hosts
192.168.145.40  ansible
192.168.145.41  mon1
192.168.145.42  mon2
192.168.145.43  mon3
192.168.145.51  osd1
192.168.145.52  osd2
192.168.145.53  osd3

Since I will be recreating these test machines several times, I do not want to keep modifying the ~/.ssh/known_hosts because of the default StrictHostKeyChecking I have on my main setup.
I add the following lines to ~/.ssh/config.

#Vagrant hosts
HOST mon?
        StrictHostKeyChecking no
HOST osd?
        StrictHostKeyChecking no

You can now test by sshing into the test machines
$ ssh root@mon1