As developers we strive to maximum our productivity by using tools that streamline workflow and by automating as many things as possible. An Integrated Development Environment (IDE) is software that is intended to streamline a developer’s workflow by handling setup of the development environment and providing an easy to use GUI. While IDEs make many things easy they come at a cost of flexibility, transparency, and even developer productivity. By using a set of individual tools instead of an IDE we can create an environment that is extremely customizable and allows for greater productivity than that of a traditional IDE. In this talk I’ll talk about my decision to not use an IDE and how I came to use the set of tools that I do now. I also demo the setup I have and go over the basics of each tool I use.
You get the idea. There are version managers for almost every language out there. Each of these languages have specific environmental requirements that must be met, each one slightly different from the others. But there are a lot of similarities between these tools. Most of them alter your PATH environment variable based on which version you have selected. They might also set a few other custom environment variables needed for the language to function properly. Some are more invasive and replace shell commands, like cd, with their own versions that handle version switching. But at the core, all of them alter your shell environment so that different executables are used based on the version you select. This is a common characteristic of all these version managers.
Using all these version managers also adds a lot of overhead as each is working to customize your environment. You have to ensure you have the version manager installed and configured correctly. You also have to ensure that the version managers don’t interfere with each other. Since they are all altering the shell environment there is a good chance they will affect each other in some way.
So what’s the solution to all this? Are we stuck juggling different version managers for everything? I did a lot of searching and asked a lot of questions online and found nearly a dozen open source projects out there aimed at solving this problem. But in this post I want to focus on one that I found to be particularly good at solving this issue - asdf.
Note that I was looking for a tool to help manage software versions in my development environment on my laptop. I don’t normally run different versions of the same VM or interpreter on production servers. Only in development do I have a need to for multiple language versions to be installed. As a result, all my thoughts in this post are about asdf in development, not production.
What is asdf?
Extendable version manager with support for Ruby, Node.js, Elixir, Erlang & more
asdf is an extendable version manager with support for Ruby, Node.js, Elixir and Erlang (and now Lua). It was created by Akash Manohar (@HashNuke) and was designed to replace all the language-specific version managers. It’s also 100% shell script, which makes it easy to install and relatively portable. asdf doesn’t include support for any language directly. Each language is supported via a plugin that contains all the language specific version management details. asdf’s plugin system makes it easy to install, upgrade and remove plugins as needed. The plugin system also makes it easy to add support for new languages by third parties. asdf plugins are just plain old git repositories with a few shell scripts in them. Installation of a plugin can be done with the asdf plugin-add command.
Installation of different language versions is very easy as well. Versions can be set globally or on a per-directory basis with a .tool-versions file, which can define the language versions to be used for a directory. If you were working on a Ruby and Elixir project, you would normally set the Ruby version in one place (e.g. the .ruby-version file) and the Elixir version in another (e.g. the .exenv-version file). With asdf you create a .tool-versions file and define all the language version requirements for the project in it.
Generic Version Managers Like This Already Exist, What Makes asdf Different?
asdf has several advantages over the other version managers available:
Simplicity - asdf is by far the most simple and easy to install
Complete set of version management features - asdf has simple commands to install, list, and remove software versions. Commands are also integrated with the .tool-versions file. There are also commands for adding, upgrading and removing plugins.
Simple method of setting defaults - All defaults can be set by creating a .tool-versions file in your home directory. If a subdirectory you are in doesn’t contain a .tool-versions file asdf crawls up the directory tree looking for one. If it gets to your home directory without finding one it uses the defaults you have set there.
Extendable - Plugins are easy to install and write and can be designed to do almost anything you want.
Below I list a few of the more popular projects along with some of the issues I see with them and some of the features asdf has that they lack. These are good projects with a lot of great people behind them. This comparison isn’t to bash the alternatives, but rather to compare their differences. Most of these projects offer more flexibility than asdf in some areas, and as a result often lack some of the features present in asdf.
You have to create Nix profiles for each different project environment you need. Profiles need to be created using nix commands.
While features of many of these tools are similar to asdf, none match the simplicity and easy of use of asdf. asdf is 100% shell script, whereas the alternatives are partially implemented in other languages like Python, Go and C. The addition of another language often makes installation more difficult and usually adds several more dependencies to the tool. Many of these tools also require you to write shell scripts to setup the environment for each project which is something I find very tiresome when setting up small projects.
But asdf Doesn’t Support the Language I Use!
There is a plugin API that is extremely simple and very well documented (I can attest to this since I developed the Lua plugin for asdf). Development of a plugin for a new language or framework is straightforward and can be done in a couple of hours.
How Does It Work?
asdf uses shims to ensure the right executable is run. Shims are stored in a directory that is part of your PATH. When you execute a shim that shim figures out your current directory and looks for a .tool-versions file in it. If one exists it reads the version information and determines which one of the real executables should be executed. For example, suppose you have ruby 2.2.4 and ruby 1.9.3 installed. Each install includes an irb executable but neither are on your PATH. If you are in a project that uses Ruby 2.2.4 (because it’s defined in your .tool-versions file), the irb executable in the 2.2.4 install is the executable that should be run. The shim will see that your in a project that requires Ruby 2.2.4 and look for the irb executable in that Ruby install and execute it. If for some reason you didn’t have Ruby 2.2.4 installed the executable wouldn’t be found (since that Ruby installation doesn’t exist on your machine) and ruby 2.2.4 not installedwill be printed by the shim. Since this only happens when you run an executable the asdf overhead is kept to a minimum. Of course there are many more details in the source code that I won’t bother to mention here.
Next we just cd into our project directory and create a .tool-versions file inside it containing the ruby version:
$ echo'ruby 2.2.4' >> .tool-versions
That’s it! Now we can use all the executables that came with the Ruby 2.2.4 install in our project.
I have been using asdf exclusively at work and at home for almost 4 months now. I have been able to uninstall a lot of software and simplify my dotfiles a lot since switching to asdf. Overall it has provided a big a boost to my productivity. Now I don’t even think about version management most of the time. With the few Erlang version managers available I was always struggling to come up with a good strategy for Erlang version management. With asdf, Erlang version management just works. I hope you will take a look at the project and give it a try. You won’t regret trying it.
I decided to move away from BitBucket Git repository hosting to a self-hosted Git solution. Since I am familiar with Ruby on Rails I decided use GitLab to host my repositories since GitLab version 8 is a Rails 4.1 application. GitLab originally used Gitolite to handle the Git repository hosting but they have moved away from Gitolite to plain old Ruby code. I already had a server running Ubuntu 14.04 and I figured it would be perfect for running GitLab.
It turns out the GitLab installation instructions for Ubuntu are straightforward.
First we need to install OpenSSH server, Postfix, and a few other things:
The curl command configures apt-get with the information it needs to fetch and install the GitLab package. If you want to see the contents of the script before you run it you can pipe it to cat or less instead of sudo bash. apt-get then installs GitLab.
After installation run:
$ sudo gitlab-ctl reconfigure
After installing check the status of GitLab:
$ sudo gitlab-ctl status
Everything should be running. If GitLab is not running try starting it manually by running:
$ sudo gitlab-ctl start
Now that GitLab is running you should be able to navigate to your server’s IP address or hostname in your browser and see the login page.
After getting GitLab installed and the permissions setup the next step is to migrate the BitBucket repositories to the GitLab server. Unfortunately importing BitBucket repositories is a little confusing and it took me quite a while to figure out how to import them. The difficulties are mostly due to the way the BitBucket API works.
Create an OAuth Key
We first need to create an OAuth consumer key so that GitLab can access private repositories. To create an OAuth key sign into your BitBucket account and navigate to settings > Oauth. Then click “Add consumer” and create a consumer for GitLab. The callback URL should be the URL to your GitLab instance. The URL could be anything, but it makes sense to also make it the URL to your GitLab instance. Under the Permissions heading make sure the OAuth consumer has read and admin permissions to repositories. It may seem a little odd to giving GitLab admin permissions to repositories, but this is needed in order to GitLab to be able to clone the repositories. The BitBucket import script will use the BitBucket API to add the public key as a deploy key for each repository your import, then clone down the repository via SSH and remove the deploy key once the import is complete. This is necessary because the BitBucket API doesn’t allow repositories to be cloned by OAuth applications via HTTPS. In order to add a deploy key to a repository GitLab must have admin access to the repositories being imported, since adding keys is an admin-level action.
After creating the OAuth consumer make a note of the key and secret values. You will need them when you configure GitLab to use the OAuth consumer.
Use the key and secret values you got from the BitBucket interface in place of YOUR_KEY and YOUR_APP_SECRET in the BitBucket section of the file.
Create an SSH Public Key for the GitLab Import Script
Next, you need to create a SSH public key on the server you have GitLab installed on. This key must NOT be associated with ANY existing BitBucket accounts. If it is the import will fail with an “Access denied! Please verify you can add deploy keys to this repository.” error. The BitBucket import script will use the BitBucket API to add this public key as a deploy key for each repository you choose to import. If the key is already being used by a BitBucket account, the API call to add the key will fail since the key is already being used. To be safe, generate a brand new key. The key should be generated for the git user:
$ sudo -u git -H ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/var/opt/gitlab/.ssh/id_rsa): bitbucket_rsa
When prompted for the name of the file to store the key in, enter bitbucket_rsa. Mine ended up being stored at /var/opt/gitlab/.ssh/bitbucket_rsa. The documentation says that the location should be /home/git/.ssh/bitbucket_rsa. Apparently my git user’s home directory was /var/opt/gitlab/ instead of /home/git/. It seemed to work just fine in /var/opt/gitlab/. I believe the only thing that is important is that the key is placed in the git user’s home directory. You can verify the home directory of the git user by running echo ~git. This is a known issue with the documentation and apparently /var/opt/gitlab/ is correct.
Your SSH client will also need to be configured to use this key when connecting to BitBucket. You can do that by adding this to the SSH config (mine was located at /var/opt/gitlab/.ssh/config, you may need to create the file):
Next, to verify you have the ssh key setup correctly, run:
$ sudo -u git -H ssh -Tv bitbucket.org
You should see output along the lines of what is shown below and be prompted continue connecting:
OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /var/opt/gitlab/.ssh/config
debug1: Server host key: RSA 97:8c:1b:f2:6f:14:6b:5c:3b:ec:aa:46:46:74:7c:40
The authenticity of host 'bitbucket.org (126.96.36.199)' can't be established.
RSA key fingerprint is 97:8c:1b:f2:6f:14:6b:5c:3b:ec:aa:46:46:74:7c:40.
Are you sure you want to continue connecting (yes/no)?
Type yes to confirm that you want to connect to bitbucket.org. After confirming you should see a permission denied message. If you see a authentication successful message you have done something wrong. The key you are using has already been added to a BitBucket account and will cause the import script to fail. Ensure the key you are using CANNOT authenticate with BitBucket.
Update GitLab with the New Configuration
Next all you need to do is tell GitLab to reconfigure itself with the changes made in gitlab.rb. To reload the configuration run:
$ sudo gitlab-ctl reconfigure
You should now be able to go to the web interface and import a repository. Go to your GitLab server’s homepage and click the “New Project” button. On the page the third row in the form should say “Import project from” and have a list of third party sites. Click on the BitBucket button. You should be taken to the BitBucket OAuth authorization page (You will need to be logged into your BitBucket account for this to work). Allow the GitLab consumer access to your account. After you allow access to your account you should be taken back to your GitLab website. The page should have a list of all your BitBucket repositories with an import button for each one.
Try clicking the import button for an individual repository, or click “Import All Projects”. If everything is working a spinner icon should appear and after a few seconds the import should complete. Once the import is complete the repository will be just like any other repository in GitLab. You can edit the repository and customize it in any way you see fit.
If something goes wrong during import and you get the “Access denied! Please verify you can add deploy keys to this repository.” you can alter the add_deploy_key method in the importer so that more readable error messages are logged to the production log.
$ sudo vi /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/bitbucket_import/client.rb
The additional logging calls will make it easy to diagnose the problem. Restart GitLab and try to import the repositories again. Then check the production log (sudo tail -f /var/log/gitlab/gitlab-rails/production.log).
YOUR_APP_ID and YOUR_APP_SECRET should be set to the Client ID and Client Secret values shown in the GitHub interface. Next reconfigure GitLab:
$ sudo gitlab-ctl reconfigure
Now go to GitLab and click “New Project”. Under the “Import project from” heading click “GitHub”. You will be taken to GitHub and prompted to give the application you created permission to you account. You will need to be logged in to your GitHub account for this to work. After granting permission to the application you will be taken back to GitLab and shown a list of all GitHub repositories. Choose “Import All Projects” or import individual repositories.
Backing Up GitLab Itself
When I was setting up my GitLab server I came across a script that backs up GitLab data to a remote server - Auto GitLab Backup. Since I don’t have another server or service to backup the data on my GitLab server I wasn’t able to use this script. If you want another copy of your Git repositories all you would need to do would be to setup another server (maybe an EC2 instance?) and then add the Auto GitLab Backup script to your crontab on your GitLab server.
GitLab is a good alternative to BitBucket or GitHub. It’s great to have a completely open source piece of software that I can run on my own servers. The BitBucket import tool is definitely be tricky to setup but it’s great that GitLab provides the tools for migrating repositories. Since it does lack some of the features present in BitBucket and GitHub it may be not be suitable for large teams. But it’s perfect for individuals or small teams that want to be independent of BitBucket and GitHub.