Stratus3D

Software Engineering, Web Development and 3D Design

Why Erlang?

Abstract

With web applications today there is an increasing need for systems that can scale up easily and tolerate the network, hardware, and software errors that occur at scale. Building such systems is challenging, if not impossible, to do in many popular programming languages today. Erlang was designed for telephone systems where uptime requirements were law, and as such was designed to solve these challenges. In this talk Trevor will show how Erlang can be used to solve some of the most challenging problems software developers face today. He will introduce concurrency constructs that Erlang provides and show how they can be used to build reliable and scalable applications. This talk will focus mostly on the new concepts introduced by Erlang, but Trevor will also do a few demos and show code samples to illustrate some of the concepts.

Slides

https://speakerdeck.com/stratus3d/why-erlang

Let's Encrypt on WebFaction

I have finally gotten around to installing Let’s Encrypt SSL certificates on all the websites I run. I use WebFaction (affiliate link) for my web hosting and that meant I had use their API for installing certificates. Rather than just placing the certificate files directly on the server I had to use their XML-RPC API to install the certificate. I quickly discovered the letsencrypt-webfaction gem which makes the process of obtaining and installing certificates on WebFaction easy. In this blog post I’ll explain what I did to get everything installed and configured.

Install letsencrypt-webfaction

First install the gem:

$ GEM_HOME=$HOME/.letsencrypt_webfaction/gems RUBYLIB=$GEM_HOME/lib gem2.4 install letsencrypt_webfaction

For ease of use you’ll want to install it in a specific directory under your home directory. On WebFaction you only have permission to write to your home directory, so that rules out most places. I chose to use version 2.4 of the gem executable for installing the letsencrypt-webfaction gem, but you can use older version of Ruby as well. You can check your server for available Ruby versions by typing ruby and then hitting tab in Bash session on your server. Bash should give you autocomplete suggestions for all the executables with that prefix. WebFaction has helpfully suffixed executables with the version numbers.

Remembering the details of your gem installation can be a pain, so add a function to your .bashrc to store all these details and make it easier to use:

function letsencrypt_webfaction {
    PATH=$PATH:$GEM_HOME/bin GEM_HOME=$HOME/.letsencrypt_webfaction/gems RUBYLIB=$GEM_HOME/lib ruby2.4 $HOME/.letsencrypt_webfaction/gems/bin/letsencrypt_webfaction $*
}

After sourcing your .bashrc and you should be able to use the letsencrypt_webfaction function in the shell. Next create config files for each website with the details of the domain and the web server.

Configure letsencrypt-webfaction

There are two ways to run the letsencrypt_webfaction function. You can specify everything the gem needs to know about your website and your WebFaction account with command line arguments, or you can put everything in a config file. I’ve found the config file approach to be much more manageable. It’s easier to make changes to a config file than it is to edit an existing cron job with a long command.

I created a directory called le_webfaction in my home directory to hold my config files. Then I created yaml files in this directory for each website I wanted to create an SSL certificate for. I created files with the domain name as the filename, like this: <domain>.yml. So for stratus3d.com I created a file named stratus3d.com.yml. See the letsencrypt-webfaction README for details on the config file. Mine ended up looking like this:

key_size: 4096
# We need an ACME server to talk to, see github.com/letsencrypt/boulder
endpoint: 'https://acme-v01.api.letsencrypt.org/'
domains: ['stratus3d.com']
public: ['/home/<home dir>/webapps/<app dir>/']
output_dir: '~/certificates/'
letsencrypt_account_email: '<email>'
api_url: 'https://api.webfaction.com/'
username: '<webfaction account username>'
password: '<webfaction account password>'

You’ll need to have all these fields populated, but several can be left with their default values. Now that everything is in a config file, obtaining a certificate is easy. Run the function we defined earlier to issue and install the certificate:

$ letsencrypt_webfaction --config ~/le_webfaction/stratus3d.com.yml

The gem should go through the process of obtaining a certificate and installing it on your WebFaction. I won’t go over how that process works, but you can read more about it here. Once the command has succeeded you may get a message like this:

You will need to change your application to use the stratus3d.com certificate.
Add the `--quiet` parameter in your cron task to remove this message.

This means you haven’t setup a WebFaction “website” that uses the new SSL certificate. You’ll need to do once when creating the certificate initially, but when renewing certificates this will not be necessary. Login to your webfaction account and go to https://my.webfaction.com/websites. Edit an existing website, or create a new one if you need. Click on the “Encrypted website (https)” button, and then choose the certificate letsencrypt-webfaction just obtained.

encrypted website option

After saving the website you should be able to access it over HTTPS! You’ll need to repeat this process with a different config file for every domain that you want to add an SSL certificate to.

Setup Cron Jobs

Running the command again will obtain a new certificate from Let’s Encrypt, but you are limited to 10 per week per domain. Certificates only last 90 days, so it’s best to setup a cron job to automatically renew them every month or two. To edit your cron jobs on WebFaction run crontab -e as you normally would, then add the following:

MAILTO=<your email>
MAILFROM=<your email>

0 7 15 * * PATH=$PATH:$GEM_HOME/bin:/usr/local/bin GEM_HOME=$HOME/.letsencrypt_webfaction/gems RUBYLIB=$GEM_HOME/lib ruby2.4 $HOME/.letsencrypt_webfaction/gems/bin/letsencrypt_webfaction --config $HOME/le_webfaction/stratus3d.com.yml >> $HOME/my_logs/stratus3d_letsencrypt.log

I set mine to renew on the 15th of every month, but I’m not sure what the best renewal strategy is. It’s probably best to refer to the official Let’s Encrypt documentation when deciding when to run your renewal cron jobs. As you can see here the way we invoke the gem in the cron job is different. Since the cron job runs in a different environment the letsencrypt_webfaction function we defined earlier isn’t available, so we have to set some environment variables manually. I also chose to pipe the output of this command to a log file, so I could go back and look at the logs if a certificate failed to renew. And that’s it. You should have a cron job that automatically obtains new certificates and installs them for you automatically.

Conclusion

letsencrypt-webfaction saved me a lot of time on this task. I would have had to written a lot of code to interact with the WebFaction API had it not been for the gem. And as my first experience with Let’s Encrypt I found it easy to understand and work with. The ability to automate renewal like this is also a big time saver. I don’t have to worry about remembering to pay for new certificates every year.

References

Vagrant for Erlang Development

I typically like to do development work on my local machine. Locally I’ve got all my favorite tools, scripts, and aliases along with custom mappings for my editor. Local development is much more pleasant than SSH’ing into a server and running commands. Without all my custom tools and configurations the environment feels foreign to me. Because of this I generally try to avoid solutions to development problems that involve a virtual machine. Even though the VM is running on my laptop it’s really not that much easier to develop on than a regular server.

I’ve known about Vagrant for a long time, but I really wasn’t interested in using it because it was easy to setup development environments on my laptop with asdf. Then I encountered a project at work that I wasn’t able to get working on my laptop. I spent hours trying to figure out what was misconfigured, but to no avail. I reluctantly figured I would give Vagrant a try. It seemed like a better option than using a plain VM. It turned out to be very effective. My development with Vagrant is almost seamless now.

In this blog post I’ll cover a few of the issues I ran into when setting up Vagrant for my Erlang project as well as some things I discovered that improved my workflow with Vagrant.

Installation

First off you’ll need to install Vagrant and a hypervisor for running the actual VM. I like VirtualBox because it is free and open source.

If you are on linux you may be able to use your package manager to install VirtualBox. Version 5.0 is the latest version that Vagrant supports.

1
2
# Install VirtualBox 5, I'm on Debian so I'm using apt-get
$ sudo apt-get install virtualbox-5.0

Then install vagrant. You can download it from the Vagrant website or if you are on linux you can use your package manager to install it.

Installation Hiccup

After installing Vagrant and trying to start it up for one of my projects I realized there was an issue with my VirtualBox installation. It turned out to be due to an option called VT-x being disabled in my BIOS. The error I got when I tried to boot the VM looked like this:

There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["startvm", "dc1a0388-9aab-4ce9-9343-0778af7d1f1d", "--type", "headless"]

Stderr: VBoxManage: error: VT-x is disabled in the BIOS for all CPU modes (VERR_VMX_MSR_ALL_VMX_DISABLED)
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component ConsoleWrap, interface IConsole

I rebooted my machine, went into the BIOS, and enabled that option. When I booted up my VirtualBox installation worked and no errors were printed.

Intel VT-d Feature Enable

Setting Up the Environment on the VM

Once you have Vagrant installed you can begin setting it up for your Erlang project. Navigate to your project on the command line and run vagrant init <box> to generate a Vagrantfile for the project. For this blog post I chose the hashicorp/precise64 box, which is Ubuntu 12.04 and seems to be the default box that is used in the Vagrant documentation. Boxes are the package format for Vagrant environments. Boxes contain the base VM image and other metadata. Available boxes are listed on the Vagrant website. The Vagrantfile in your project root is where you can specify configuration values for your project’s box. Typically there isn’t much that needs to change, but there are plenty of options available. You can set options for network interfaces, synced folders, and the base box image that is used by the VM. I’m not going to cover all that here. The Vagrant documentation and the book Vagrant: Up and Running are great resources.

Once you have a VM up and running you will need to provision the box. For Erlang development you will need Erlang, Rebar/Rebar3, and optionally Elixir. The Vagrantfile allows us to specify a provisioning script that can be run when the VM is created to install all the tools you will need. I use asdf locally, so I figured I would use it on the VM as well.

The provision script I needed would need to install asdf, install all the necessary asdf plugins, and then install the correct versions of Erlang, Rebar, and Elixir for the project. The script I came up with does all of this:

provision.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
#!/usr/bin/env bash

# Unoffical Bash "strict mode"
# http://redsymbol.net/articles/unofficial-bash-strict-mode/
set -euo pipefail
#ORIGINAL_IFS=$IFS
IFS=$'\t\n' # Stricter IFS settings

# Install Git and other asdf dependencies
sudo apt-get install -y git automake autoconf libreadline-dev libncurses-dev \
    libssl-dev libyaml-dev libffi-dev libtool unixodbc-dev \
    build-essential autoconf m4 libncurses5-dev curl

# Install asdf
git clone https://github.com/asdf-vm/asdf.git $HOME/.asdf
(cd $HOME/.asdf; git checkout v0.4.0)
echo -e '\n. $HOME/.asdf/asdf.sh' >> $HOME/.bashrc
echo -e '\n. $HOME/.asdf/completions/asdf.bash' >> $HOME/.bashrc
# Make asdf available in this script
set +u
source "$HOME/.asdf/asdf.sh"
set -u

# Install all the necessary asdf plugins
asdf plugin-add erlang https://github.com/asdf-vm/asdf-erlang.git
asdf plugin-add rebar https://github.com/Stratus3D/asdf-rebar.git
asdf plugin-add elixir https://github.com/asdf-vm/asdf-elixir.git

# Navigate to the directory containing the project (/vagrant is the directory
# that is synced with the project dir on the host)
cd /vagrant
# Make the versions defined .tool-versions file the versions used by the vagrant
# user in any directory
cp .tool-versions $HOME
# Install all correct versions of these packages for the project
asdf install

echo "Completed setup of Erlang environment!"

asdf expects a .tool-versions file in the project root, so before you have Vagrant run the provision script the .tool-versions file must exist in the project. For my project I needed the latest Erlang and Rebar3 versions but not Elixir, so mine looked like:

.tool-versions
1
2
erlang 20.1
rebar 3.4.7

Now you just need to tell Vagrant to use this script to provision your VM. The config.vm.provision parameter allows us to specify the provision method for the Vagrant box. For a shell script like this you need to add a config.vm.provision line like this:

Vagrantfile
1
2
3
4
5
6
Vagrant.configure(2) do |config|

  # ... omitted other options

  config.vm.provision "shell", path: "provision.sh", privileged: false
end

Vagrant will run the provision script after creating the VM, so if you already have a Vagrant box running run vagrant destroy and vagrant up to have Vagrant setup a new VM and then run the provision script. If the provision script finishes without errors you should have a running Vagrant VM configured for Erlang development!

Tighter Integration with My Local Environment

SSH’ing onto the VM to run commands is something I wanted to avoid and it turns out it’s easy to avoid running commands directly on the VM. Vagrant provides the vagrant ssh command which can be used to ssh onto the server, but it can be treated as a regular SSH client, meaning you can use it to run arbitrary commands on the server just like you could with a regular SSH client. To run arbitrary commands use:

$ vagrant ssh -- '<command>'

For example, to see the IP addresses of the VM run:

$ vagrant ssh -- 'ip address'

You can also run scripts on the VM like this:

$ vagrant ssh -- < <script>

This is a lot to type out for simple things so I was eager to find a better way of doing this. It would be nice to not have to type out so much. After asking some questions I found three ways to make running commands on the VM easier.

Shell Alias

The first way to simplify commands is to just create a shell alias for vagrant ssh --. It’s easy to do and makes the commands a lot shorter:

# Add this to your .bashrc
alias vc="vagrant ssh --"

# Then you can use it to run commands on the VM:
$ vc 'ip address'

The downside to this is that you still have to quote the command you want to run.

vagrant-exec

vagrant-exec is a very nice Vagrant plugin that aims to make it easier to run commands on the VM. It offers some very nice features:

  • Uses synced folders to map commands to the right directory on the VM, allowing you navigate around your local environment and run commands in the equivalent on the VM.
  • It has options for generating shims, which you can add to your $PATH and then run commands locally without a prefix.
  • It has options for prepending commands with other commands. For example prepend apt-get with sudo.

vagrant-exec is a much better choice than shell aliases. It offers more features and tighter integration. The downside is it often requires more work to configure.

What vagrant-exec does isn’t that complicated so I wanted to see if I could write a simplified version of it as a shell script.

va script

I was able to write a simple Bash script that works similar to vagrant-exec. It lacks many of the features provided by vagrant-exec, but still makes running commands very easy. I named the script va to make it short enough that no alias would be needed. Using the script is very easy. Going back to the IP address example it would just be:

$ va ip address

Basically all the script does is look at the synced folder mappings configured for the project, and then maps the current directory on the host machine to equivalent directory on the VM. This allows you to easily run directory-specific commands on the host without having to worry about the directory being used on the VM. The output from the command is printed just as if it was run locally.

The source for the script can be found in my dotfile repo on GitHub. All you need to is put it on your $PATH.

Conclusion

Overall Vagrant has been a big help. I was surprised at how much searching I had to do to find a good way of seamlessly running commands on the VM from my local environment. With my va script I’m pretty happy, and I can always use vagrant-exec in the future if I find my va script insufficient.

I still really like developing locally but for times when I can’t run a project locally I’m going to use Vagrant. It’s hard to beat the ease of use and tight integration that Vagrant provides.

References