Stratus3D

A blog on software engineering by Trevor Brown

Asdf Has Been Re-Written in Golang

Over the last year I’ve rewritten asdf in Go. I’m excited to announce that the rewrite was released last Wednesday as asdf version 0.16.0!

asdf 0.16.0 is now a single binary rather than a collection of Bash scripts. All operations are faster than they were in the previous versions. With improvements ranging from 2x-7x faster than asdf version 0.15.0! Numerous long-standing bugs were also fixed during the rewrite. The codebase is also much more approachable for new contributors, and we can now leverage tools in the Go ecosystems to make future releases of asdf even better!

Upgrading

Note

I wrote this blog after asdf 0.16.0 had just been released. A few bugs were found and fixed this last week and released in version 0.16.1. If you upgrade, you should upgrade to 0.16.1, not 0.16.0.

Is it easy to upgrade? The good news is that you can upgrade and not lose your plugins and versions you’ve installed with an older version of asdf. asdf 0.16.0 is backward compatible with plugin and version data from all recent versions of asdf. However, the API has changed some, with some sub-commands being removed entirely, and some replaced. The output of some commands has changed, so if you’ve got any automation or scripts that invoke asdf commands you’ll want to verify that they work with the new version before upgrading. See the full list of breaking changes here.

How do I upgrade? Unfortunately asdf upgrade cannot upgrade to the latest Go implementation. We decided to switch to more mainstream distribution methods: OS package managers, precompiled binaries, and go install. Most of our users will be best served by one of these methods.

Plugin and version data installed by previous versions of asdf will work with the new version, but only if you configure the new version to use the same directory for data. If you choose to use a precompiled binary or go install, you’ll need to configure this manually. If your OS package manager has asdf 0.16.0 available it should take care of this for you. See the migration guide on the website for full instructions for your OS.

If you haven’t used asdf before, now is a great time to give it a try. It is easier to install than it has ever been. It’s now just a single binary installed somewhere on your system, along with a change to your PATH so asdf’s shims can be executed in your shell.

Why a Rewrite?

The rest of this blog post explains why I decided to rewrite asdf in Go. There were two primary reasons why I decided to rewrie asdf, and both of them can attributed to a codebase that was entirely Bash.

Performance

For many years there have been complaints from users on how long certain operations took in asdf. Just running asdf current to see current versions could take 10-20 seconds. Generating shims with asdf reshim typically took several minutes!

Most of the slowness was due to asdf being implemented in Bash. Bash scripts typically invoke other commands to perform much of the work. Invoking a command like grep, sed, or cut means spawning another OS process, which is a relatively slow operation. Commands like asdf reshim loop over every tool and every version, regenerating every shim script. Updating a single shim might require running a dozen commands. Repeating this for dozens or hundreds of shims scripts and it will easily take minutes. All because Bash invokes other programs to do much of the actual work. Unfortunately there wasn’t a lot we could do to speed up the asdf Bash code. The few things we could do to improve performance typically came at the cost of more convoluted code, so we rejected most performance optimizations for the Bash code.

With Go we’re able to invoke functions from Go’s standard library to do most of the work. Rather than using cut to split a string we can use Go’s strings.Cut function. This is much faster than starting an entirely new OS process to split a string.

Maintainability

Early on Bash served us well. Once you learn the esoteric syntax it’s easy to quickly write small scripts that glue a couple of programs together to do something novel on the command line. It is still an easy language to quickly write asdf plugins in, and I doubt we will ever stop using it completely. But for a codebase the size of asdf’s it was limiting to both maintainers and contributors.

The esoteric syntax and quirks of Bash made it hard for users of asdf to contribute bug fixes and features. Many users were not experienced with Bash at all and contributed incorrect bug fixes, or features containing serious bugs. It’s also very hard to debug and test. Maintainers also struggled to review pull requests and bugs often slipped through, even in small bug-fix PRs.

Aside from the performance issues and esoteric syntax, Bash is very limiting when it comes to data structures. By default everything in Bash is just a string. Simple data structures like arrays were challenging to use in Bash. Strings and arrays were the only data structures widely used in asdf Bash code. This often forced us to keep internal and external representations of data the same. The strings we print for the user are the strings used internally. This meant simple feature requests like customizing the output for a sub-command were challenging and resulted in a lot of manual string manipulation. All the essential complexity of asdf got lost in the accidental complexity of the Bash implementation. As time has progressed this has made asdf increasingly unmaintainable.

Conclusion

I’m excited about the future of asdf. The rewrite in Go makes asdf faster, improves the user experience, makes the codebase more maintainable, and makes asdf more approachable to future open-source contributors. Give the new version a try and let me know what you think!

In the next blog post I’ll benchmark the new Go version against the old Bash version and share the results. Subscribe to my mailing list so you don’t miss it!

Discussion on Hackernews: https://news.ycombinator.com/item?id=42961844

References