I’ve been a maintainer of asdf for about 6 years now. Most of my work these last several years has been fixing bugs, writing automated tests to catch regressions, and ensuring overall correctness across various operating systems. Maintaining a tool written in Bash isn’t easy. For most things there are far more edge cases and compatibility issues than happy paths. Certain commands need to be banned from the codebase to ensure compatibility across operating systems. Bash strict mode also helps but it introduces other issues that have to be worked around.
All this to say I haven’t had a lot of time to step back and assess the other aspects of asdf. Other maintainers have stepped up and helped with documentation and UX, but performance has fallen by the wayside. Over the last couple of months I’ve noticed some asdf commands that I use often seem to have gotten slower. Many asdf commands are fairly slow but with the way asdf was designed it is seldom a problem. asdf current was never fast enough to be used as part of a shell prompt showing the current versions, but that was user experience enhancement that wasn’t critical. But it seems like things are getting worse and I wanted to assess performance. In this post I am going to benchmark several asdf commands and see how their performance has changed over time.
Performance Over Time
I wanted to see how the performance of each command has changed over time since asdf was first released. This can be done by checking out each release tag of asdf with Git, benchmarking each command on that tag and then writing the results to disk, before proceeding to the next tag. Once the benchmarking data is written to disk it can be used to generate graphs of the performance changes over time. There isn’t a tool that does all this so I had to write a Bash script.
For the actual benchmarking I like hyperfine, and it exports results to CSV and JSON so it was perfect for this. All I had to do was write a Bash script that wires up git for-each-ref and git checkout and then runs hyperfine for each command on each version. The resulting shell script is attached. Gathering the data with this benchmarking script is straightforward.
What Sub-Commands to Benchmark
Some commands are important, used often, and expected to be fast. These commands are:
-
asdf current- print the versions specified for the current directory -
asdf which- print the path of the actual executable that a shim resolves to -
asdf where- print the install directory of a tool -
asdf exec- execute a shim (almost the same as executing a shim directly)
I benchmarked all of these commands.
Some commands are expected to be relatively slow and take at least a second or two to finish. These commands are:
-
asdf install <tool> <version>- install a tool version -
asdf plugin add <name> <repo>- install a plugin -
asdf reshim- re-generate all shims
Of these three commands I’ve only received complaints about the performance of asdf reshim. It is by far the slowest of all asdf commands so I benchmarked it. asdf install is often slow as well, but its performance is tied to the plugin’s install script and isn’t controlled by asdf so I skipped it.
Benchmarking
I opted to run my benchmark on my Ubuntu laptop with the following stats.
$ lshw -short
H/W path Device Class Description
======================================================
system 20DF0040US (LENOVO_MT_20DF_BU_Think_FM_ThinkPad E550)
/0 bus 20DF0040US
/0/3 memory 32KiB L1 cache
/0/4 processor Intel(R) Core(TM) i7-5500U CPU @ 2.40GHz
/0/4/5 memory 32KiB L1 cache
/0/4/6 memory 256KiB L2 cache
/0/4/7 memory 4MiB L3 cache
/0/8 memory 8GiB System Memory
/0/8/0 memory 8GiB SODIMM DDR3 Synchronous 1600 MHz (0.6 ns)
/0/8/1 memory DIMM [empty]
/0/30 memory 128KiB BIOSFor asdf commands that need a plugin to operate on, I decided to use my own Lua plugin as Lua is simple and lightweight and should have a very small effect on the resulting numbers.
Results
The benchmarking script produced a directory of CSV files containing the results. Each file represented a single benchmark of a single command at a single version. I wanted to generate graphs of the performance of each command across all versions of asdf and for that I knew I needed to combine the CSV files. I wrote a script that combined all the benchmarking data for each command into a single CSV. This script was a bit sloppy but it worked. Now I have one CSV file for each command, with each row in it containing the benchmark of it at a specific version of asdf.
Graphs
Now that I had the data it was time to generate the graphs. I am somewhat familiar with gnuplot and wrote a little gnuplot code to generate all the graphs.
The asdf current command is very slow and its performance has gotten worse over time. It now takes on average 1 second for the command to finish on my laptop.
asdf exec is probably the most important of the commands I’ve benchmarked as it is invoked when any asdf shim is executed. Whenever a shim is executed it executes asdf exec and asdf exec resolves to the correct version of the command and then executes it with execs. The graph above shows how this resolution process has changed over time. Performance isn’t terrible, but it’s not great either. At least it has been relatively flat over time. Still, 150 milliseconds is a lot of overhead when it affects every shim.
asdf reshim is by far the slowest of all the commands benchmarked. It has also gotten significantly slower in the last couple of releases of asdf. It currently averages 24 seconds on my laptop! Granted, I have a lot of shims on my laptop and the reshim process has a complexity of O(n). Despite this I think there are things we can do to significantly improve these numbers in future releases.
asdf where performs pretty well and has gotten faster! 50 milliseconds isn’t bad for Bash.
asdf which is almost as good as asdf where and has also gotten faster over time.
Conclusion
It’s nice to see that asdf where and asdf which are relatively fast and have gotten a little faster and it will be safe to ignore them in future profiling work. asdf current and asdf reshim are slow and have gotten slower over time. Since these are commands that I frequently run manually it is not surprising to see they have gotten slower over time. Given that asdf reshim takes over 24 seconds to complete it is clear there is a lot of work to be done. I’ve already written a tool for generating flame graphs of Bash scripts so stay tuned.