Stratus3D

A blog on software engineering by Trevor Brown

Elixir 1.9 Configuration Merging

Elixir configuration can be placed in several different files. Config is often stored in config/config.exs and often config.exs is setup to load environment specific configuration files like config/test.exs, config/prod.exs, and so on. And now in Elixir 1.9 runtime configuration can be placed in config/releases.exs.

Configuration Evaluation

These various configuration files are evaluated at different times. config/config.exs is evaluated at compile time. Runtime configuration in config/releases.exs is evaluated inside the release at runtime.

Configuration Merging

If we use multiple configuration files configuration data is merged recursively. We can observe how this merging works by specifying multiple configurations in the same configuration file with the same root key and then loading the configuration values from it.

# Write this to a file named config_test.exs
import Mix.Config

config :config_test,
  param1: :some_value,
  param2: :another_value,
  nested_param: [keyword: :list]

config :config_test, :nested_param, [another: :keyword]

config :config_test,
  param1: :new_value

This file makes three config calls to set the configuration for the :config_test application. These config calls will be evaluated in the order they are defined in the file, but what will the final configuration be?

Loading the configuration using Config.Reader in the iex shell shows us the final values:

iex> Config.Reader.read!("config_test.exs")

[
  config_test: [
    param1: :new_value,
    param2: :another_value,
    nested_param: [keyword: :list, another: :keyword],
  ]
]

Config.Reader recursively merges the configurations in order until they are all combined.

The first config call defines the base configuration. The third changes the value of the :param1 key to :new_value. Even keys inside :nested_param keyword list get merged with the existing keyword list. We can also see that the whole configuration itself is just a set of nested keyword lists.

Conclusion

Elixir merges configuration spread across multiple files in a reasonable way, and makes it easy to combine configuration from config/config.exs, config/releases.exs, and any other file containing configuration data without having to duplicate any of the keys and values.

But remember, configuration files aren’t always best. Application configuration is global and therefore prevents libraries and applications from being used with different configurations in the same release. Avoiding configuration files entirely is usually the best approach for library applications.

Another important change related to configuration is that mix new will no longer generate a config/config.exs file. Relying on configuration is undesired for most libraries and the generated config files pushed library authors in the wrong direction.
— Elixir v1.9 released

References

Bash Errexit Inconsistency

I’m a huge fan of the unofficial Bash strict mode. I use it for all my personal scripts as well as for other projects when I can. One part of Bash strict mode is the errexit option, which is turned on by doing set -e or set -o errexit. With errexit turned on, any expression that exits with a non-zero exit code terminates execution of the script, and the exit code of the expression becomes the exit code of the script. errexit is a big improvement, since any unexpected error ends execution of a script rather than being ignored and allowing execution of the script to continue. If you have a command that you expect to fail, you can do one of two things:

# Use the OR operator
maybe_fails || true

# Use the command as an if condition
if maybe_fails; then
  # success
fi

This ensures every command must succeed and the only commands allowed to fail are that are part of expressions designed to handle non-zero exit codes like if statements. If you have a command that you expect to fail and don’t care about the failure you are best off using the OR operator with true as the second command. If you have a command that may fail, and you want to execute code conditional based on the exit code of the command an if statement is the best option. But as I recently learned, there is one pitfall with the way errexit works with functions invoked from if conditions.

The Problem

errexit isn’t used when executing functions inside an if condition. A function executed normally with errexit would return the exit code of the first expression that returned a non-zero exit code, and execution of the function would stop. A function executed inside an if statement with errexit would ignore non-zero exit codes from commands invoked by the function and would continue to execute until a return or exit command is encountered. Here is an example script:

#!/usr/bin/env bash

set -e

f() {
	false
	echo "after false"
}

if f; then
  echo "f was successful"
fi

f

In this code we invoke the function f twice. Once inside the if statement and once by itself as a single expression. You might think this script will not output anything, since the first expression in the function f is false, which is a command that always returns an exit code of 1. However, the output of this script is actually:

after false
f was success

And the exit code of this script is 1, indicating a failure. When f is executed inside an if condition it is considered successful, and both lines of the function are executed. When it is executed outside of an if statement, as we would expect, only the first line of the function is evaluated and the function returns the exit code of the false command, which is always 1.

To sum up, when using errexit, any expression that returns an non-zero exit code will halt execution of the current function and return the non-zero exit code, except for expressions in the following places:

  • The command list following until or while

  • Part of the test following if or elif

  • Preceding && or ||

  • Any expression in a pipeline except the last, unless you are using set -o pipeline

In these locations non-zero exit codes are ignored.

The Solution

Note

I originally had a solution here that was flat out wrong. I originally stated:

The solution to this is to simply never invoke functions from within if statement conditionals. Instead the function must be executed before the if statement and the return code must be captured in a variable.

Here is the same if statement above modified to use this approach:

# Execute command and capture exit code, regardless of whether the command succeeded or not
f && exit_code=$? || exit_code=$?

# Use the exit code variable as the condition for the if statement.
if [ "$exit_code" = 0 ]; then
  echo "f was successful"
fi

This code does not work because I was using && when executing the function, so it was invoked without errexit set. Thanks to Dr. Björn Kahlert for pointing this and providing me with a complete list of locations where errexit is disabled.

The real solution here is not quite as elegant as I had hoped. The solution is to define a function that disables errexit, runs a subshell, enables errexit inside the subshell, executes provided the function, captures the exit code of the subshell, and then turns errexit back on before returning. The code looks like this:

get_exit_code() {
  # We first disable errexit in the current shell
  set +e
  (
    # Then we set it again inside a subshell
    set -e;
    # ...and run the function
    "$@"
  )
  exit_code=$?
  # And finally turn errexit back on in the current shell
  set -e
}

exit_code=0
get_exit_code f

if [ "$exit_code" = 0 ]; then
  echo "f was successful"
else
  echo "f failed"
fi

While this is not very elegant it is easy to use in practice. Since the function is executed outside of the if statement it will always be executed with errexit set as expected. Output from the original function can still be captured and used if desired.

Update 8/6/2022

Olivier wrote in and shared his own solution to this problem:

#!/usr/bin/env bash

set -e

g(){
  return 42
}

f() {
  g || return
  echo "after false"
}

if f; then
  echo "f was successful"
fi

This works about the same as my solution above with two significant differences:

  • It doesn’t run code in a subshell so a separate process is not spawned. It may therefore be slightly faster.

  • It doesn’t capture the exit code to a variable. The if statement will only be able to indicate success or failure unless you capture and compare the exit code yourself inside the if condition with something like if (f; [ $? -eq 42 ]); then.

References

My Favorite Talks From Strange Loop 2019

I was able to go Strange Loop for the first time this year. I’ve watched a lot of videos of talks from Strange Loop over the last couple years, and was excited to be able to attend this year. It was a great conference with excellent talks on a good variety of topics. I learned a lot and I am really glad I went. The conference was multi-track with up to 8 talks scheduled during each time slot, so there were many talks I missed. Below are my 6 favorite talks from those I attended at the conference. I am sure there are many excellent talks I missed out on, but these were my favorites among the ones I attended.

Probabilistic Scripts for Automating Common-Sense Tasks

https://www.youtube.com/watch?v=MiiWzJE0fEA

https://thestrangeloop.com/2019/probabilistic-scripts-for-automating-common-sense-tasks.html

My favorite talk from the conference by far. Coming into the talk I didn’t have an experience with probabilistic programming so this talk introduced me to the concepts and as well as Gen, a probabilistic programming system. I was able to learn a lot from this talk and it got me excited about probabilistic programming. I have decided to dedicate some time to learning probabilistic programming.

Beyond Alt-Text: Trends in Online Accessibility

https://www.youtube.com/watch?v=pNcB7ChyO1E

https://thestrangeloop.com/2019/beyond-alt-text-trends-in-online-accessibility.html

This talk was a good reminder of the importance of web accessibility. I really enjoyed the topic as it forced me to look at the web differently. Realizing that not everyone can have the same experience on the web reminds me to be more careful about how I design things for the web. This talk introduced my to Pa11y and I’ve started running it on this website.

How to Teach Programming (and Other Things)?

https://www.youtube.com/watch?v=g1ib43q3uXQ

https://thestrangeloop.com/2019/how-to-teach-programming-and-other-things.html

This was an excellent and fun talk. This keynote was probably one of the most thought provoking talks I listened to at the conference. I think it was also one of the most controversial talks that was given at the conference. I had numerous discussions with people at the conference with people about the things I agreed with and the things I didn’t agree with in this talk. Overall it was very thought provoking and very well presented.

New programming constructs for probabilistic AI

https://www.youtube.com/watch?v=xNutxms6SH4

https://thestrangeloop.com/2019/new-programming-constructs-for-probabilistic-ai.html

Another interesting talk about probabilistic computing. This talk also featured example code written with Gen just like the other talk on probabilistic scripting. This talk presented some very interesting problems and showed how probabilistic programming constructs could be used to solve them without a strong understanding of the underlying mathematics.

Uptime 15,364 days - The Computers of Voyager

https://www.youtube.com/watch?v=H62hZJVqs2o

https://thestrangeloop.com/2019/uptime-15-364-days---the-computers-of-voyager.html

An all around fun and informative talk about the design and development of the Voyager space probes. Aaron covers the requirements the project had as well challenges that were encountered in the design and development of the hardware for the probes. I found the details of the redundant computer hardware and the probes' power system to be particularly interesting.

Parser Parser Combinators for Program Transformation

https://www.youtube.com/watch?v=JMZLBB_BFNg

https://thestrangeloop.com/2019/parser-parser-combinators-for-program-transformation.html

A very interesting talk on tool called comby, which allows for programmatic refactoring of code in any language. comby has been run on code in the top 100 Github projects and produced refactorings that resulted in around 40 merged PRs all without any manual manipulation of the code. comby seems very powerful and I hope ideas like this will become more popular.

References