Stratus3D

Software Engineering, Web Development and 3D Graphics

Escript Essentials

I’v been working a lot with escripts recently and I’ve discovered quite a few features that I wish I had know about earlier. I’ve written several escripts over the last couple years, but it wasn’t until recently that I read through all the documentation on escript. There are a lot of powerful features tucked away in escript that would have never know about had I not dug deeper into documentation. I also learned a lot from Geoff Cant’s excellent talk on escript at Erlang Factory 2011. In this post I’ll try to cover all the features that I wish I would have known about from day one.

For all the examples in this post I’ll be using Erlang 18.2. Note that the example escripts in this post aren’t meant to be complete escripts but rather simple examples to showcase certain things. While all the scripts should work as is, they don’t handle all the possible return values from some function calls and they don’t handle errors at all. Ideally escripts like these should catch errors and print helpful error messages to the user informing them of their mistake.

Different Execution Modes

Interpreted mode

This is the default mode an escript runs in. In interpreted mode the code is executed very much like in the erl REPL. Unlike the REPL the escript is actually an Erlang module. Even though you don’t define a module in the file the Erlang code in the file is treated as if it’s inside a module. You can include header files, define and use macros, and do anything else you would do in a regular Erlang module. You can even use the ?MODULE in your escripts.

Compiled mode

In compiled mode the Erlang code is compiled down to bytecode before execution to improve performance. To do this set the mode attribute after the shebang and comment lines in the escript:

-mode(compile).

Native mode

Native mode compiles the escript down to native code for even faster execution:

-mode(native).

Of course you will want to make sure this actually improves performance. On a simple escript the overhead of compilation may make the escript slower. If you are concerned about performance you should benchmark the escript in each mode and go with the one that runs the fastest. You can use timer:tc to time the execution of the resource intensive code in your escript. You can also use the unix time utility to record the total run time of the escript.

Enumlator Arguments

Arguments can be passed to escript by having the second line (or third line if the emacs directive line is present) start with %%!. For example, the escript below uses the main/1 function in the my_main_module module (the -escript flag), and runs in interpreted mode (-i).

1
2
3
4
5
#!/usr/bin/env escript
%% -*- coding: utf-8 -*-
%%! -escript main my_main_module -i

...

These are the flags I find most useful:

  • -escript main <module> - use the main function defined in <module> when running the escript
  • -s only do a syntax check of the escript
  • -d Loads the module containing the main/1 function into the debugger, sets a breakpoint in main/1 and invokes main/1.
  • -c run the escript in compiled mode regardless of the mode attribute
  • -i run the escript in interpreted mode regardless of the mode attribute
  • -n run the escript in native mode regardless of the mode attribute

The emulator flags override other settings in the escript. In additional to these flags many of the flags that erl accepts can also be used as well. There are a lot of them. A few that I find useful in escripts are:

  • -setcookie set the Erlang cookie
  • -sname set the short name of the escript node (this also causes net_kernel to start)
  • -noinput ensures that the Erlang runtime system never tries to read any input.
  • -noshell starts the Erlang runtime system with no shell.
  • -path, -pa, -pz tinker with the escript’s Erlang path (this is where Erlang looks to find modules, I don’t recommend relying on this, and I’ll later show a better way of setting your escripts path)

You can also specify the encoding Erlang code in the escript on the second line using a slightly different syntax. This is the line that can contain the emacs directives:

%% -*- coding: utf-8 -*-

Connect to a running Erlang release

Escripts are regular Erlang nodes and they can connect to other nodes if net_kernel is running. There many things you can do once you have net_kernel up and running inside your escript. Below are a few examples.

Ping a Node

You can easily ping a node once net_kernel is running.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#!/usr/bin/env escript

-define(NODE_NAME, pinger).

main([Node, Cookie]) ->
    {ok, _} = net_kernel:start([?NODE_NAME, shortnames]),
    erlang:set_cookie(node(), list_to_atom(Cookie)),
    case net_adm:ping(list_to_atom(Node)) of
        ping ->
            io:format("~s ~p~n", [Node, ping]),
            halt(0);
        Else ->
            io:format("~s ~p~n", [Node, Else]),
            halt(1)
    end.

Note that instead of calling erlang:set_cookie/2 I could have just used the -setcookie flag in the second line of the escript:

%%! -setcookie cookie

To use the escript just run: $ ./pinger <node_name>@<host_name> <cookie>. Note that this script uses short names, so it will only be able to ping other nodes with short names.

Make an RPC Call to a Remote Node

The rpc module makes it easy to call functions on the remote node. This allows you escript to get statistics and performance information from a running node, check on job status, kick off a utility that needs to run, or change the configuration. In the example below I use rpc to get the statistics on the number of reductions that have been performed on the node:

1
2
3
4
5
6
7
8
9
10
11
#!/usr/bin/env escript

-define(NODE_NAME, reduction_stats).

main([Node, Cookie]) ->
    NodeName = list_to_atom(Node),
    {ok, _} = net_kernel:start([?NODE_NAME, shortnames]),
    erlang:set_cookie(node(), list_to_atom(Cookie)),

    {Total, _SinceLastCall} = rpc:call(NodeName, erlang, statistics, [reductions]),
    io:format("Number of reductions on node ~s: ~p~n", [NodeName, Total]).

To use the script just invoke it with the node name you want to check along with the cookie it uses:

$ ./reduction_stats <node>@<host> <cookie>
Number of reductions on node <node>@<host>: 534202

Load Config from a Running Application

Often you will want to verify a release is running with the correct configuration or load the configuration from a release and use it in your escript. Loading the config from a running node is trivial with the rpc module:

1
2
3
4
5
6
7
8
9
10
11
12
#!/usr/bin/env escript

-define(NODE_NAME, config_printer).

main([Target, App, Cookie]) ->
    TargetName = list_to_atom(Target),
    AppName = list_to_atom(App),
    {ok, _} = net_kernel:start([?NODE_NAME, shortnames]),
    erlang:set_cookie(node(), list_to_atom(Cookie)),

    Config = rpc:call(TargetName, application, get_all_env, [AppName]),
    io:format("Config from ~s on node ~s: ~p~n", [AppName, TargetName, Config]).

Same as before, just invoke the escript with the node you want to get config from and the cookie:

$ ./config_printer <node>@<host> <cookie>
Config from your_app on node <node>@<host>: [{param, value}]

Load Modules from a Running Application

You can also pull the Erlang path from a running application and set it as the Erlang path of your escript, making all the Erlang modules in a release available in your escript.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#!/usr/bin/env escript

-define(NODE_NAME, path).

main([Target, Cookie]) ->
    TargetName = list_to_atom(Target),
    {ok, _} = net_kernel:start([?NODE_NAME, shortnames]),
    erlang:set_cookie(node(), list_to_atom(Cookie)),

    Path = rpc:call(TargetName, code, get_path, []),
    % Print it
    io:format("Path on node ~s: ~p~n", [TargetName, Path]),
    % Add it to the path of the escript
    code:add_pathsz(Path).
    % Use the modules in the path directly in your escript...

Again, usage of the escript is the same:

$ ./path <node>@<host> <cookie>
Path from your_app on node <node>@<host>: [".", ...]

Remsh into a Node

Starting a remsh using erl can be a pain sometimes. You can start a remote shell inside an escript with the user_drv:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
#!/usr/bin/env escript
%%! -noshell -noinput

-define(NODE_NAME, remsh).

main([Node, Cookie]) ->
    net_kernel:start([?NODE_NAME, shortnames]),
    erlang:set_cookie(node(), list_to_atom(Cookie)),
    Target = list_to_atom(Node),

    % Traps exits so when the remote shell crashes we can exit gracefully
    process_flag(trap_exit, true),

    % Start the remote shell
    Shell = user_drv:start(['tty_sl -c -e',{Target,shell,start,[]}]),

    % Link to the remote shell so we receive the exit message
    true = erlang:link(Shell),
    io:format("Grabbed a remote shell on ~p~n.", [Target]),

    % Return when we get the exit message
    receive
        {'EXIT', Shell, _} -> ok
    end.

Again, usage of the escript is the same, but this time a remote Erlang shell is started!

$ ./remsh <node>@<host> <cookie>
Erlang/OTP 18 [erts-7.2] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]
Grabbed a remote shell on 'server@trevor-ThinkPad-E550'
                                                    .Eshell V7.2  (abort with ^G)
(<node>@<host>)1>

You can issue commands as you normally would, and just enter <Ctrl>-C to exit.

Locks

Node names can be repurposed as locks. You can ensure only one instance of the escript is running per machine by always starting the Erlang node with the same name:

1
2
3
4
5
6
7
8
9
10
11
12
13
% ...

-define(NODE_NAME, escript_lock).

main(_) ->
    case net_kernel:start([?NODE_NAME, longnames]) of
        {error, _} ->
            error_exit("User state sync already in progress.");
        _ ->
            ok
    end,
    % continue
    % ...

If a second escript with the same name is started when one is already running with the same name the second will fail since the name is already taken. Once the first escript stops the name will become available again, allowing the escript to be invoked a second time.

Package Multiple Files in an Escript

If your writing non-trivial escript it’s usually best to organize the code into multiple modules to keep the project organized. Escripts can contain beam bytecode and a zip archive of other files that are needed in the escript. This makes it possible to package an entire Erlang application in a single, portable, escript.

Building An Escript by Hand

Let’s say we have an Erlang project with a directory structure that looks like this:

src/ % these are the modules we want to include in the escript
    module_1.erl % this module exports a main/1 function
    module_2.erl
priv/ % other resources in here
    index.html

Packaging multiple modules into an escript is done in several steps:

  1. Compile Erlang code. One of the modules must export a main/1 function that the escript can call with the command line arguments. Something along these lines:

     mkdir ebin
     erlc -v -Werror +debug_info +warn_export_vars +warn_shadow_vars +warn_obsolete_guard  -o ebin/ src/module_1.erl src/module_2.erl
    
  2. Build a zip archive of non-beam files:

     zip archive.zip priv/*
    
  3. Create the escript with the beam files and the archive. There are a couple things Erlang’s escript module needs to build the escript, so for convience we can put the code to build the escript in another escript:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
#!/usr/bin/env escript

-define(ESCRIPT, "archive_demo").
-define(SHEBANG, "/usr/bin/env escript").
-define(COMMENT, "").
-define(EMU_ARGS, "-pa . -sasl errlog_type error -escript main module_1").

main(_) ->
    Beams = [{F, read(F)} || F <- filelib:wildcard("ebin/*")],
    Files = [{"archive.zip", read("archive.zip")}|Beams],
    ok = escript:create(?ESCRIPT, [
       {archive, Files, [memory]},
       {shebang, ?SHEBANG},
       {comment, ?COMMENT},
       {emu_args, ?EMU_ARGS}
     ]),
    ok = file:change_mode(?ESCRIPT, 8#755),
    ok.
read(File) ->
    {ok, B} = file:read_file(filename:absname(File)),
    B.

Automation

Rebar, Rebar3, and Erlang.mk all have tools built in that make building multi-module escripts even easier, and are more flexible than a custom build script like the one above.

With Rebar3

With Rebar3 all you need to do is create a rebar.config file specifying the application name, the escript name, the other applications to include in the escript, and finally the emulator arguments:

{escript_main_app, module_1}
{escript_name, escript_demo}.
{escript_incl_apps, []}.
{escript_emu_args, "%%! -escript main module_1"}.

Note that the emulator arguments option must specify the module containing the main/1 function for the escript if the module name containing it differs from the application name. To build the escript just run:

rebar3 escriptize

You should find the completed escript in the _build directory.

With Erlang.mk

With erlang.mk all we need to do is add the escript.mk plugin in the project and include it in the Makefile. We also need to specify a few options for the escript:

PROJECT = escript_example

ESCRIPT_NAME = escript_demo
ESCRIPT_EMU_ARGS = "%%! -escript main module_1"

include erlang.mk
include escript.mk

Then just use the escript target to build the escript:

make escript

Conclusion

Almost anything that can be done in an Erlang release can also be done in an escript with a little work. Since escripts are regular Erlang nodes and can call other modules, there isn’t much you can’t do directly in an escript. In this blog post I’ve really only scratched the surface of what’s possible with escript. While it may be possible to do everything you need inside an escript it’s important to remember escripts aren’t meant to be a replacement for Erlang releases. If you need a long running application you should use an Erlang release. Escript’s real strength it’s ability to provide better interfaces for interacting with Erlang systems. This is especially helpful when you have engineers that aren’t experienced with Erlang maintaining Erlang systems.

Hopefully this blog post has show a few features of escript that you can use to build better escripts.

Oh, and when your writing larger escripts, getopt by Juan Jose Comellas is a essential. I couldn’t live without it.

Resources

Why I Don't Use an IDE

Abstract

As developers we strive to maximum our productivity by using tools that streamline workflow and by automating as many things as possible. An Integrated Development Environment (IDE) is software that is intended to streamline a developer’s workflow by handling setup of the development environment and providing an easy to use GUI. While IDEs make many things easy they come at a cost of flexibility, transparency, and even developer productivity. By using a set of individual tools instead of an IDE we can create an environment that is extremely customizable and allows for greater productivity than that of a traditional IDE. In this talk I’ll talk about my decision to not use an IDE and how I came to use the set of tools that I do now. I also demo the setup I have and go over the basics of each tool I use.

Slides

https://speakerdeck.com/stratus3d/why-i-dont-use-an-ide

Introducing Asdf: The Extendable Version Manager

For Ruby you have rvm, rbenv, ry and a dozen other version managers.
For Elixir you have kiex and exenv.
For Erlang you have evm, erln8 and kerl.
For Node.js you have nvm and nvmw.

You get the idea. There are version managers for almost every language out there. Each of these languages have specific environmental requirements that must be met, each one slightly different from the others. But there are a lot of similarities between these tools. Most of them alter your PATH environment variable based on which version you have selected. They might also set a few other custom environment variables needed for the language to function properly. Some are more invasive and replace shell commands, like cd, with their own versions that handle version switching. But at the core, all of them alter your shell environment so that different executables are used based on the version you select. This is a common characteristic of all these version managers.

Using all these version managers also adds a lot of overhead as each is working to customize your environment. You have to ensure you have the version manager installed and configured correctly. You also have to ensure that the version managers don’t interfere with each other. Since they are all altering the shell environment there is a good chance they will affect each other in some way.

So what’s the solution to all this? Are we stuck juggling different version managers for everything? I did a lot of searching and asked a lot of questions online and found nearly a dozen open source projects out there aimed at solving this problem. But in this post I want to focus on one that I found to be particularly good at solving this issue - asdf.

Note that I was looking for a tool to help manage software versions in my development environment on my laptop. I don’t normally run different versions of the same VM or interpreter on production servers. Only in development do I have a need to for multiple language versions to be installed. As a result, all my thoughts in this post are about asdf in development, not production.

What is asdf?

Extendable version manager with support for Ruby, Node.js, Elixir, Erlang & more

asdf is an extendable version manager with support for Ruby, Node.js, Elixir and Erlang (and now Lua). It was created by Akash Manohar (@HashNuke) and was designed to replace all the language-specific version managers. It’s also 100% shell script, which makes it easy to install and relatively portable. asdf doesn’t include support for any language directly. Each language is supported via a plugin that contains all the language specific version management details. asdf’s plugin system makes it easy to install, upgrade and remove plugins as needed. The plugin system also makes it easy to add support for new languages by third parties. asdf plugins are just plain old git repositories with a few shell scripts in them. Installation of a plugin can be done with the asdf plugin-add command.

Installation of different language versions is very easy as well. Versions can be set globally or on a per-directory basis with a .tool-versions file, which can define the language versions to be used for a directory. If you were working on a Ruby and Elixir project, you would normally set the Ruby version in one place (e.g. the .ruby-version file) and the Elixir version in another (e.g. the .exenv-version file). With asdf you create a .tool-versions file and define all the language version requirements for the project in it.

Generic Version Managers Like This Already Exist, What Makes asdf Different?

asdf has several advantages over the other version managers available:

  • Simplicity - asdf is by far the most simple and easy to install
  • Complete set of version management features - asdf has simple commands to install, list, and remove software versions. Commands are also integrated with the .tool-versions file. There are also commands for adding, upgrading and removing plugins.
  • Simple method of setting defaults - All defaults can be set by creating a .tool-versions file in your home directory. If a subdirectory you are in doesn’t contain a .tool-versions file asdf crawls up the directory tree looking for one. If it gets to your home directory without finding one it uses the defaults you have set there.
  • Extendable - Plugins are easy to install and write and can be designed to do almost anything you want.

Below I list a few of the more popular projects along with some of the issues I see with them and some of the features asdf has that they lack. These are good projects with a lot of great people behind them. This comparison isn’t to bash the alternatives, but rather to compare their differences. Most of these projects offer more flexibility than asdf in some areas, and as a result often lack some of the features present in asdf.

While features of many of these tools are similar to asdf, none match the simplicity and easy of use of asdf. asdf is 100% shell script, whereas the alternatives are partially implemented in other languages like Python, Go and C. The addition of another language often makes installation more difficult and usually adds several more dependencies to the tool. Many of these tools also require you to write shell scripts to setup the environment for each project which is something I find very tiresome when setting up small projects.

But asdf Doesn’t Support the Language I Use!

There is a plugin API that is extremely simple and very well documented (I can attest to this since I developed the Lua plugin for asdf). Development of a plugin for a new language or framework is straightforward and can be done in a couple of hours.

How Does It Work?

asdf uses shims to ensure the right executable is run. Shims are stored in a directory that is part of your PATH. When you execute a shim that shim figures out your current directory and looks for a .tool-versions file in it. If one exists it reads the version information and determines which one of the real executables should be executed. For example, suppose you have ruby 2.2.4 and ruby 1.9.3 installed. Each install includes an irb executable but neither are on your PATH. If you are in a project that uses Ruby 2.2.4 (because it’s defined in your .tool-versions file), the irb executable in the 2.2.4 install is the executable that should be run. The shim will see that your in a project that requires Ruby 2.2.4 and look for the irb executable in that Ruby install and execute it. If for some reason you didn’t have Ruby 2.2.4 installed the executable wouldn’t be found (since that Ruby installation doesn’t exist on your machine) and ruby 2.2.4 not installed will be printed by the shim. Since this only happens when you run an executable the asdf overhead is kept to a minimum. Of course there are many more details in the source code that I won’t bother to mention here.

How Do I Use It?

Installation

Installation is simple:

1
2
3
4
5
6
7
$ git clone https://github.com/HashNuke/asdf.git ~/.asdf

# For Ubuntu or other linux distros
$ echo '. $HOME/.asdf/asdf.sh' >> ~/.bashrc

# OR for Max OSX
$ echo '. $HOME/.asdf/asdf.sh' >> ~/.bash_profile

Then install the dependencies:

  • On OSX install automake autoconf openssl libyaml readline libxslt libtool unixodbc with homebrew.
  • On Ubuntu install automake autoconf libreadline-dev libncurses-dev libssl-dev libyaml-dev libxslt-dev libffi-dev libtool unixodbc-dev with apt-get.

That’s it! You should be able to view asdf help by running asdf in the shell.

Usage

Say we want to install ruby 2.2.4 for a project we are going to build. First we have to install the ruby plugin (you will only need to do this once):

1
$ asdf plugin-add ruby https://github.com/HashNuke/asdf-ruby.git

Now we can install ruby 2.2.4:

1
$ asdf install ruby 2.2.4

Next we just cd into our project directory and create a .tool-versions file inside it containing the ruby version:

1
$ echo 'ruby 2.2.4' >> .tool-versions

That’s it! Now we can use all the executables that came with the Ruby 2.2.4 install in our project.

Summary

I have been using asdf exclusively at work and at home for almost 4 months now. I have been able to uninstall a lot of software and simplify my dotfiles a lot since switching to asdf. Overall it has provided a big a boost to my productivity. Now I don’t even think about version management most of the time. With the few Erlang version managers available I was always struggling to come up with a good strategy for Erlang version management. With asdf, Erlang version management just works. I hope you will take a look at the project and give it a try. You won’t regret trying it.

Resources