Stratus3D

Software Engineering, Web Development and 3D Design

Vagrant for Erlang Development

I typically like to do development work on my local machine. Locally I’ve got all my favorite tools, scripts, and aliases along with custom mappings for my editor. Local development is much more pleasant than SSH’ing into a server and running commands. Without all my custom tools and configurations the environment feels foreign to me. Because of this I generally try to avoid solutions to development problems that involve a virtual machine. Even though the VM is running on my laptop it’s really not that much easier to develop on than a regular server.

I’ve known about Vagrant for a long time, but I really wasn’t interested in using it because it was easy to setup development environments on my laptop with asdf. Then I encountered a project at work that I wasn’t able to get working on my laptop. I spent hours trying to figure out what was misconfigured, but to no avail. I reluctantly figured I would give Vagrant a try. It seemed like a better option than using a plain VM. It turned out to be very effective. My development with Vagrant is almost seamless now.

In this blog post I’ll cover a few of the issues I ran into when setting up Vagrant for my Erlang project as well as some things I discovered that improved my workflow with Vagrant.

Installation

First off you’ll need to install Vagrant and a hypervisor for running the actual VM. I like VirtualBox because it is free and open source.

If you are on linux you may be able to use your package manager to install VirtualBox. Version 5.0 is the latest version that Vagrant supports.

1
2
# Install VirtualBox 5, I'm on Debian so I'm using apt-get
$ sudo apt-get install virtualbox-5.0

Then install vagrant. You can download it from the Vagrant website or if you are on linux you can use your package manager to install it.

Installation Hiccup

After installing Vagrant and trying to start it up for one of my projects I realized there was an issue with my VirtualBox installation. It turned out to be due to an option called VT-x being disabled in my BIOS. The error I got when I tried to boot the VM looked like this:

There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["startvm", "dc1a0388-9aab-4ce9-9343-0778af7d1f1d", "--type", "headless"]

Stderr: VBoxManage: error: VT-x is disabled in the BIOS for all CPU modes (VERR_VMX_MSR_ALL_VMX_DISABLED)
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component ConsoleWrap, interface IConsole

I rebooted my machine, went into the BIOS, and enabled that option. When I booted up my VirtualBox installation worked and no errors were printed.

Intel VT-d Feature Enable

Setting Up the Environment on the VM

Once you have Vagrant installed you can begin setting it up for your Erlang project. Navigate to your project on the command line and run vagrant init <box> to generate a Vagrantfile for the project. For this blog post I chose the hashicorp/precise64 box, which is Ubuntu 12.04 and seems to be the default box that is used in the Vagrant documentation. Boxes are the package format for Vagrant environments. Boxes contain the base VM image and other metadata. Available boxes are listed on the Vagrant website. The Vagrantfile in your project root is where you can specify configuration values for your project’s box. Typically there isn’t much that needs to change, but there are plenty of options available. You can set options for network interfaces, synced folders, and the base box image that is used by the VM. I’m not going to cover all that here. The Vagrant documentation and the book Vagrant: Up and Running are great resources.

Once you have a VM up and running you will need to provision the box. For Erlang development you will need Erlang, Rebar/Rebar3, and optionally Elixir. The Vagrantfile allows us to specify a provisioning script that can be run when the VM is created to install all the tools you will need. I use asdf locally, so I figured I would use it on the VM as well.

The provision script I needed would need to install asdf, install all the necessary asdf plugins, and then install the correct versions of Erlang, Rebar, and Elixir for the project. The script I came up with does all of this:

provision.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
#!/usr/bin/env bash

# Unoffical Bash "strict mode"
# http://redsymbol.net/articles/unofficial-bash-strict-mode/
set -euo pipefail
#ORIGINAL_IFS=$IFS
IFS=$'\t\n' # Stricter IFS settings

# Install Git and other asdf dependencies
sudo apt-get install -y git automake autoconf libreadline-dev libncurses-dev \
    libssl-dev libyaml-dev libffi-dev libtool unixodbc-dev \
    build-essential autoconf m4 libncurses5-dev curl

# Install asdf
git clone https://github.com/asdf-vm/asdf.git $HOME/.asdf
(cd $HOME/.asdf; git checkout v0.4.0)
echo -e '\n. $HOME/.asdf/asdf.sh' >> $HOME/.bashrc
echo -e '\n. $HOME/.asdf/completions/asdf.bash' >> $HOME/.bashrc
# Make asdf available in this script
set +u
source "$HOME/.asdf/asdf.sh"
set -u

# Install all the necessary asdf plugins
asdf plugin-add erlang https://github.com/asdf-vm/asdf-erlang.git
asdf plugin-add rebar https://github.com/Stratus3D/asdf-rebar.git
asdf plugin-add elixir https://github.com/asdf-vm/asdf-elixir.git

# Navigate to the directory containing the project (/vagrant is the directory
# that is synced with the project dir on the host)
cd /vagrant
# Make the versions defined .tool-versions file the versions used by the vagrant
# user in any directory
cp .tool-versions $HOME
# Install all correct versions of these packages for the project
asdf install

echo "Completed setup of Erlang environment!"

asdf expects a .tool-versions file in the project root, so before you have Vagrant run the provision script the .tool-versions file must exist in the project. For my project I needed the latest Erlang and Rebar3 versions but not Elixir, so mine looked like:

.tool-versions
1
2
erlang 20.1
rebar 3.4.7

Now you just need to tell Vagrant to use this script to provision your VM. The config.vm.provision parameter allows us to specify the provision method for the Vagrant box. For a shell script like this you need to add a config.vm.provision line like this:

Vagrantfile
1
2
3
4
5
6
Vagrant.configure(2) do |config|

  # ... omitted other options

  config.vm.provision "shell", path: "provision.sh", privileged: false
end

Vagrant will run the provision script after creating the VM, so if you already have a Vagrant box running run vagrant destroy and vagrant up to have Vagrant setup a new VM and then run the provision script. If the provision script finishes without errors you should have a running Vagrant VM configured for Erlang development!

Tighter Integration with My Local Environment

SSH’ing onto the VM to run commands is something I wanted to avoid and it turns out it’s easy to avoid running commands directly on the VM. Vagrant provides the vagrant ssh command which can be used to ssh onto the server, but it can be treated as a regular SSH client, meaning you can use it to run arbitrary commands on the server just like you could with a regular SSH client. To run arbitrary commands use:

$ vagrant ssh -- '<command>'

For example, to see the IP addresses of the VM run:

$ vagrant ssh -- 'ip address'

You can also run scripts on the VM like this:

$ vagrant ssh -- < <script>

This is a lot to type out for simple things so I was eager to find a better way of doing this. It would be nice to not have to type out so much. After asking some questions I found three ways to make running commands on the VM easier.

Shell Alias

The first way to simplify commands is to just create a shell alias for vagrant ssh --. It’s easy to do and makes the commands a lot shorter:

# Add this to your .bashrc
alias vc="vagrant ssh --"

# Then you can use it to run commands on the VM:
$ vc 'ip address'

The downside to this is that you still have to quote the command you want to run.

vagrant-exec

vagrant-exec is a very nice Vagrant plugin that aims to make it easier to run commands on the VM. It offers some very nice features:

  • Uses synced folders to map commands to the right directory on the VM, allowing you navigate around your local environment and run commands in the equivalent on the VM.
  • It has options for generating shims, which you can add to your $PATH and then run commands locally without a prefix.
  • It has options for prepending commands with other commands. For example prepend apt-get with sudo.

vagrant-exec is a much better choice than shell aliases. It offers more features and tighter integration. The downside is it often requires more work to configure.

What vagrant-exec does isn’t that complicated so I wanted to see if I could write a simplified version of it as a shell script.

va script

I was able to write a simple Bash script that works similar to vagrant-exec. It lacks many of the features provided by vagrant-exec, but still makes running commands very easy. I named the script va to make it short enough that no alias would be needed. Using the script is very easy. Going back to the IP address example it would just be:

$ va ip address

Basically all the script does is look at the synced folder mappings configured for the project, and then maps the current directory on the host machine to equivalent directory on the VM. This allows you to easily run directory-specific commands on the host without having to worry about the directory being used on the VM. The output from the command is printed just as if it was run locally.

The source for the script can be found in my dotfile repo on GitHub. All you need to is put it on your $PATH.

Conclusion

Overall Vagrant has been a big help. I was surprised at how much searching I had to do to find a good way of seamlessly running commands on the VM from my local environment. With my va script I’m pretty happy, and I can always use vagrant-exec in the future if I find my va script insufficient.

I still really like developing locally but for times when I can’t run a project locally I’m going to use Vagrant. It’s hard to beat the ease of use and tight integration that Vagrant provides.

References

Octopress Rake Task for Notifying Search Engines

I recently have been trying to automate many of the smaller tasks that I need to perform when publishing a blog post. One those tasks is submitting my new sitemap to the various search engines and blogging services. Right now I either do it manually or wait until the search engines crawl my site again.

I quickly found several rake tasks that can automate submitting my sitemap. While the code for the Rake tasks did work I saw some things that needed to be improved, so I set about refactoring them. I was able to come up with a nice set of well behaved Rake tasks for each of the services I wanted to notify of my new content. I now have Rake tasks for Google, Bing, and Ping-o-Matic.

Each Rake task makes a request to the service’s API then checks the status code in the response. If it’s successful it prints a success message, otherwise it prints an error message along with the response body and fails the Rake task. This ensures the Rake tasks will always print an error and then crash when a submission fails, so there will never be any false positives. The complete set of Rake tasks is shown below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
require 'net/http'
require 'uri'
require 'cgi'
require 'xmlrpc/client'

desc 'Ping pingomatic'
task :pingomatic do
  cfg = YAML.load_file("_config.yml")
  rss_url = "#{cfg['url']}#{cfg['subscribe_rss']}"
  puts '* Pinging ping-o-matic'
  response = XMLRPC::Client.new('rpc.pingomatic.com', '/').call('weblogUpdates.extendedPing', "#{cfg['title']}" , "#{cfg['url']}", rss_url)
  puts response["message"]
  if response["flerror"] == true
    puts "Ping-o-Matic error: #{response}"
    fail
  else
    puts "Successfully RSS feed to Ping-o-Matic"
  end
end

desc 'Notify Google of the new sitemap'
task :sitemapgoogle do
  cfg = YAML.load_file("_config.yml")
  puts '* Pinging Google about our sitemap'
  response = Net::HTTP.get_response('www.google.com', '/webmasters/tools/ping?sitemap=' + URI.escape("#{cfg['url']}/sitemap.xml"))
  if response.code == "200"
    puts "Successfully submitted sitemap to Google"
  else
    puts "Google Sitemap response: #{response.body}"
    fail
  end
end

desc 'Notify Bing of the new sitemap'
task :sitemapbing do
  cfg = YAML.load_file("_config.yml")
  puts '* Pinging Bing about our sitemap'
  response = Net::HTTP.get_response('www.bing.com', '/webmaster/ping.aspx?siteMap=' + URI.escape("#{cfg['url']}/sitemap.xml"))
  if response.code == "200"
    puts "Successfully submitted sitemap to Bing"
  else
    puts "Bing Sitemap response: #{response.body}"
    fail
  end
end

desc 'Notify pubsubhubbub server.'
task :pingpubsubhubbub do
  cfg = YAML.load_file("_config.yml")
  data = 'hub.mode=publish&hub.url=' + CGI::escape("#{cfg['url']}#{cfg['subscribe_rss']}")
  headers = {'Content-Type' => 'application/x-www-form-urlencoded'}
  puts '* Pinging pubsubhubbub server'
  request = Net::HTTP.new('pubsubhubbub.appspot.com', 80)
  response = request.post('http://pubsubhubbub.appspot.com/publish', data, headers)
  if response.code == "200" || response.code == "204"
    puts "Successfully submitted sitemap to pubsubhubbub"
  else
    puts "pubsubhubbub response: #{response.body}"
    fail
  end
end

desc 'Notify all the services about new content'
task :notify => [:pingomatic, :sitemapgoogle, :sitemapbing, :pingpubsubhubbub] do
end

desc "Generate website, deploy, and notify web services of changes"
task :gen_deploy_notify => [:gen_deploy, :notify] do
end

For each of these we check the status code in the HTTP response and then either print the success message, or an error message containing the actual response.

References

Better Vi Mode in Zshell

Many Vim users don’t realize that their shell offers a Vi mode, but both Zshell and Bash do. I’ve been a big fan of Vim for many years now and use it exclusively for all editing work I do. In addition to using Vim for code I also write all my blog post in Vim with vim-pencil and goyo.vim. I only recently decided to switch to Vi mode in my Zsh and Bash shells. While I was happy to move away from Emacs mode, which is the default in Zsh and Bash, I was disappointed with the key mappings present in Zsh’s Vi mode. Many of the features I’d come to love from Zsh’s Emacs mode were gone. After some tinkering and reading several blog posts (links at the bottom of the page) I figured out how to add these features back into Vi mode.

I found that oh-my-zsh offers a Vi mode plugin but I gave up using it when I realized it didn’t offer many of the things I needed. It also made doing some of the customizations I cover in this blog post much harder to do.

Incremental Searching

One of the features I loved most in Zsh’s Emacs mode was the incremental searching. I could type out the start of a command, say tmu, hit the up arrow and scroll through all the tmux commands I had previously run. This feature came in very handy when I needed to frequently rerun a command. All I’d have to do would be to type out the start of the command and then hit the up arrow a couple times. It was much easier than typing out the full command each time or searching back through history with Ctrl-R. Adding this back into Vi mode is easy. Just add something like this to your .zshrc:

# Better searching in command mode
bindkey -M vicmd '?' history-incremental-search-backward
bindkey -M vicmd '/' history-incremental-search-forward

# Beginning search with arrow keys
bindkey "^[OA" up-line-or-beginning-search
bindkey "^[OB" down-line-or-beginning-search
bindkey -M vicmd "k" up-line-or-beginning-search
bindkey -M vicmd "j" down-line-or-beginning-search

First we map the ? and / characters to incremental search when in Vi command mode, so we can use the same key sequences we would use in Vim to find something. Note that these mappings are only in command mode. In insert mode we want to be able to literally type ? or /.

Next we map the up and down arrows (^[OA and ^[OB) to incremental searching when in either insert or command mode. This allows for searching based on a partially typed command like I described above. Since the arrow keys are normally not used in Vi I can safely have these mappings in both insert and command mode.

In addition to mapping the arrow keys to the incremental searching commands I also map k and j in Vi command mode to the same commands, so I can use more Vi like mappings for incremental searching.

Visual Mode Hack

In Vim we have visual mode which makes it easy to select text to yank or manipulate. In Zshell prior to version 5.0.8 there wasn’t a proper visual mode, so if you are using an older version of Zshell you won’t have a way to visualize selections. Instead you can map the v key to open the command in Vim itself for editing. Just add this to your .zshrc:

# Easier, more vim-like editor opening
bindkey -M vicmd v edit-command-line

If you are using Zshell 5.0.8 or newer and have visual mode available you may still want to be able to open Vim to edit a command. I found Ctrl-V to be a mapping that doesn’t interfere with any of the existing mappings.

# `v` is already mapped to visual mode, so we need to use a different key to
# open Vim
bindkey -M vicmd "^V" edit-command-line

Both of these mappings will open the command for editing in the editor you’ve specified in $EDITOR, and from there you can use visual mode to manipulate the command. If you are using Vim and don’t want to load your entire .vimrc just to edit this command you can set $EDITOR to Vim with no config:

export EDITOR='vim -u NONE'

Or you can specify another vim file configuration to use:

export EDITOR='vim -u alternate_profile.vim'

Faster Mode Switching

When you press ESC Zsh normally waits 0.4 seconds before switching to command mode. This is a really long time to wait when typing in a command sequence but we can make it shorter with the KEYTIMEOUT setting. Setting it to 1 makes the wait only 10 milliseconds, which is much more reasonable. Add the following to your .zshrc file:

# Make Vi mode transitions faster (KEYTIMEOUT is in hundredths of a second)
export KEYTIMEOUT=1

Mode Indicator

This isn’t something I use myself, but it is a nice enhancement. With this code the text [NORMAL] or [INSERT] will be shown on the right side of the prompt. The code that adds the mode to the prompt is a bit more involved:

# Updates editor information when the keymap changes.
function zle-keymap-select() {
  zle reset-prompt
  zle -R
}

zle -N zle-keymap-select

function vi_mode_prompt_info() {
  echo "${${KEYMAP/vicmd/[% NORMAL]%}/(main|viins)/[% INSERT]%}"
}

# define right prompt, regardless of whether the theme defined it
RPS1='$(vi_mode_prompt_info)'
RPS2=$RPS1

Basically we call the zle-keymap-select function whenever the input changes, and then vi_mode_prompt_info function determines what mode should be displayed based off of the output from the vicmd, main and viins functions. We assign the resulting string to RPS1 and RPS2 so it’s presented on the right side of the prompt.

Conclusion

With a few tweaks I’ve found Vi mode in Zsh to be very powerful. I’m very happy with my setup and I’m not going back to Emacs mode. I wish I had started using Vi mode earlier as I spent a lot of time learning the Emacs mappings prior to switching. If you’ve got any questions about my configuration you can always find my .zshrc and all my other config files in my dotfiles repository on Github or you can ask me directly.

11/5/2017 Update

Previous versions of this blog post incorrectly stated that the Zshell’s Vi mode lacked a visual mode. That is incorrect. Zshell versions 5.0.8 later have a visual mode that works almost identical to Vi’s actual visual mode. This means the custom v mapping isn’t necessary in versions 5.0.8 and later.

References