Bulletproof App Settings with Malli & Aero

21st June 2024

Aero is a great library for pulling external application settings into your application, but even if you don’t go overboard with what you put in your config files, it has weaknesses. But pairing it together with Malli we have a recipe for bulletproof application settings; let’s take a look at how they can work together…

Let’s see some code

First let’s see how we can read app settings using the two libraries, then we’ll dig into the detail of what the benefits are.

Given an Aero config.edn file that looks like

{:http-port
 #env PORT

 :db-url
 #profile
 {:default [#env DATABASE_URL]
  :dev "some-db-url"}

 :db-password
 #profile
 {:default [#env DATABASE_URL]
  :dev "some-db-url"}

 :auth-client
 #profile
 {:default #or [#env AUTH_CLIENT :okta]
  :dev #or [#env AUTH_CLIENT :dummy]}

 :okta-client-id
 #profile
 {:default [#env OKTA_CLIENT_ID]
  :dev  "dev-client-id"}

 :metrics-enabled
 #profile
 {:default [#env METRICS_ENABLED]
  :dev #or [#env METRICS_ENABLED false]}}

We can combine it with the following Clojure code - making use of Malli - like so:

(require '[aero.core :as aero]
         '[clojure.string :as str]
         '[clojure.java.io :as io]
         '[malli.core :as m]
         '[malli.error :as me]
         '[malli.transform :as mt]
         '[malli.util :as mu])

(def non-blank-string
  [:and
   [:string {:min 1}]
   [:fn {:error/fn
         (constantly "must not be blank")}
    (fn not-blank? [s]
      (not (str/blank? s)))]])

(def Settings
  "A Malli schema representing all
  the application settings;
  i.e. values that can be set externally
  at runtime via environment variables."
  (let [schema-common
        [:map
         [:http-port
          {:default 8080}
          :int]
         [:db-url non-blank-string]
         [:db-password non-blank-string]
         [:auth-client
          {:default :okta}
          [:enum :okta :dummy]]
         [:metrics-enabled
          {:default false}
          [:boolean]]]]
    ;; Multi-schema; add in different
    ;; extra fields depending on
    ;; auth client type
    [:multi
     {:dispatch :auth-client
      :decode/string
      (fn [string-settings]
        (update
         string-settings
         :auth-client
         #(or (some-> % name
                      str/lower-case
                      keyword)
              :okta)))}
     [:okta
      (mu/merge
       schema-common
       [:map
        [:okta-client-id non-blank-string]])]
     [:dummy schema-common]]))

(defn decode-string-settings
  [string-settings]
  (m/decode Settings
            string-settings
            (mt/transformer
             mt/string-transformer
             (mt/default-value-transformer))))

(defn coerce-and-validate-string-settings
  "Parses a raw map of string-based
  settings, coercing the values into
  their respective types and throwing
  an exception on any validation errors."
  [string-settings]
  (let [settings
        (decode-string-settings
         string-settings)

        settings-errors
        (me/humanize (m/explain Settings
                                settings))]
    (if settings-errors
      (throw
       (Exception.
        (format "Invalid settings: %s"
                settings-errors)))
      settings)))

(defn read-config
  ([]
   (read-config :default))
  ([profile]
   (let [string-settings
         (-> (io/resource "config.edn")
             (aero/read-config
              {:profile profile}))]
     (coerce-and-validate-string-settings
      string-settings))))

With this in place, we can easily ready our config and get the coerced values we want:

(read-config :dev)
;; =>
{:http-port 8080,
 :db-url "some-db-url",
 :db-password "password123"
 :auth-client :dummy,
 :okta-client-id "dev-client-id",
 :metrics-enabled false}

While automatically validating incorrect inputs:

(coerce-and-validate-string-settings
 {:http-port "foo"
  :db-url "some-db-url"
  :db-password "password123"
  :okta-client-id "okta-client-123"
  :metrics-enabled "bar"})
 ;; =>
;; Invalid settings: {:http-port ["should be an integer"],
;; :metrics-enabled ["should be a boolean"]}

Why bother?

Aero already offers to do some coercion for us, so what do we gain here over using just using Aero, like so?

{:http-port
 #long
 #profile
 {:default [#env PORT]
  :dev #or [#env PORT 8080]}

 :db-url
 #profile
 {:default #env DATABASE_URL
  :dev #or [#env DATABASE_URL
            "some-db-url"]}

 :auth-client
 #profile
 {:default #or [#env AUTH_CLIENT :okta]
  :dev #or [#env AUTH_CLIENT :dummy]}

 :okta-client-id
 #profile
 {:default [#env OKTA_CLIENT_ID]
  :dev  "dev-client-id"}

 :metrics-enabled
 #boolean
 #profile
 {:default [#env METRICS_ENABLED]
  :dev #or [#env METRICS_ENABLED true]}}

This is probably fine to get started - after all, when creating a new project you probably want to focus on more interesting things than configuration.

But in a day and age where most of us are working on large fleets of microservices, with configuration passed down through multiple layers of abstraction, it can be worth taking some time making your application settings bulletproof.

Let’s take a look at what we can gain here.

Fail-fast on simple mistakes

These days it’s not so likely that we’ll be deploying jars straight onto servers; more likely, we have to contend with various layers such as Docker, Kubernetes and AWS Secrets Manager. This means that just one typo somewhere can potentially leave you with an application with a blank connection string, flailing away as it attempts to connect to a database or message broker. And if your logging and/or app healthchecks aren’t quite up to scratch, that broken service may not be so easy to find.

Using plain Aero, we don’t get any such validation for free. We could write such code ourselves, but why do that when Malli lets us do that far more succinctly?

For example, in our example code above we make sure that if the db-url and/or okta-client-id are missing, we’ll know about it straight away as an exception on app startup:

(coerce-and-validate-string-settings
 {:http-port "8080"
  :db-url ""
  :db-password "password123"
  :okta-client-id "okta-client-123"
  :metrics-enabled "false"})
;; =>
;; Invalid settings:
;; {:db-url ["should be at least 1 character"
;; "must not be blank"]}

We could guard against nil or empty strings just by declaring db-url in our Malli schema as

[:db-url [:string {:min 1}]]

But by creating our own non-blank-string schema via

(def non-blank-string
  [:and
   [:string {:min 1}]
   [:fn {:error/fn
         (constantly "must not be blank")}
    (fn not-blank? [s]
      (not (str/blank? s)))]])

we guard against strings containing only whitespace, too.

Friendly error messages

Using Aero on its own would still catch certain errors such as type coercion issues - but the resulting error messages wouldn’t be nearly as helpful.

For example, compare a bad port number in plain Aero…

;; config.edn
{;; Not a valid long!
 :http-port #long "foo"
 :db-url "some-db-url"
 :db-password "password123"
 :auth-client #keyword "okta"
 :okta-client-id "dev-client-id"
 :metrics-enabled #boolean "false"}

(aero/read-config (io/resource "config.edn") {:profile :default})
;; =>
;; Unhandled java.lang.NumberFormatException
;; For input string: "foo"

…with a bad port number when using Malli…

(coerce-and-validate-string-settings
 {:http-port "foo"
  :db-url "some-db-url"
  :db-password "password123"
  :okta-client-id "dev-client-id"
  :metrics-enabled "false"})
;; =>
;; Unhandled java.lang.Exception
;; Invalid settings:
;; {:http-port ["should be an integer"]}

We can now see which config value has the problem - much better!

Secure error messages

Another thing to note with Malli’s human-friendly error messages is that by default they don’t report the values in question. While you lose some convenience with this, they key thing is that it makes it safe to use across your entire config, which almost certainly will contain sensitive values such as passwords.

If you were to use Spec for example, most of it’s informative validation output also includes the complete value received. This is desirable in a development context, but not so much when your data potentially contains passwords. With Malli’s approach you can apply schemas to all your application settings without fear of leaking sensitive data.

Don’t fall at the first hurdle

Another issue with depending too much on Aero for picking up config issues is that it will fall over at the first issue it comes across:

;; config.edn
{;; Not a valid long!
 :http-port #long "foo"
 :db-url "some-db-url"
 :db-password "password123"
 :auth-client #keyword "okta"
 :okta-client-id "dev-client-id"
 ;; Not a valid boolean!
 :metrics-enabled #boolean "bar"}

(aero/read-config
 (io/resource "config.edn")
 {:profile :default})
;; =>
;; Unhandled java.lang.NumberFormatException
;; For input string: "foo"

Aero identified the bad HTTP port, but is still none the wiser about the bad boolean.

Making use of our Malli schema, however, tells us everything that’s gone wrong in one go:

(coerce-and-validate-string-settings
 {:http-port "foo"
  :db-url "some-db-url"
  :db-password "password123"
  :okta-client-id "dev-client-id"
  :metrics-enabled "bar"})
;; =>
;; Invalid settings:
;; {:http-port ["should be an integer"],
;; :metrics-enabled ["should be a boolean"]}

This helps avoid having to go through multiple fix-build-deploy-test cycles to sort out teething issues with config.

Easier to test

Because Aero introduces its own reader tags it understandably makes you isolate your Aero config within separate EDN files. But this is awkward to test because you can’t as easily create test cases inline within a test file.

Worse, if you bundle your system component map into your config.edn, you effectively have logic which you can’t test without either making duplicate copies of your config.edn data, or else setting environment variables on the fly.

However if you lean on Aero only for obtaining your settings as strings, then that leaves the parts left to Malli as being much easier to test:

(deftest db-url-cannot-be-blank
  (is (thrown-with-msg?
       Exception
       #".*must not be blank.*"
       (sut/coerce-and-validate-string-settings
        {:http-port "8080"
         :db-url "   "
         :db-password "password123"
         :auth-client "okta"
         :okta-client-id "okta-client-123"
         :metrics-enabled "true"}))))

Conditional validation

By making use of Malli’s multi-schemas functionality, we can add polymorphic validation in a similar way to Clojure’s multimethods. This makes it easy to adapt the validation for multiple modes or subcomponent types.

We saw this in action in the below part of the example code:

;; Multi-schema; add in different
;; extra fields depending on
;; auth client type
[:multi
 {:dispatch :auth-client
  :decode/string
  (fn [string-settings]
    (update
     string-settings
     :auth-client
     #(or (some-> % name
                  str/lower-case
                  keyword)
          :okta)))}
 [:okta
  (mu/merge
   schema-common
   [:map
    [:okta-client-id non-blank-string]])]
 [:dummy schema-common]]

Which in our example, means that an okta-client-id is required when the auth client is set to okta…

(coerce-and-validate-string-settings
 {:http-port "8080"
  :db-url "some-db-url"
  :db-password "password123"
  :auth-client "okta"
  :metrics-enabled "false"})
;; =>
;; Unhandled java.lang.Exception
;; Invalid settings:
;; {:okta-client-id ["missing required key"]}

…but isn’t required when the auth client is set to dummy:

(coerce-and-validate-string-settings
 {:http-port "8080"
  :db-url "some-db-url"
  :db-password "password123"
  :auth-client "dummy"
  :metrics-enabled "false"})
;; =>
;; All OK :)
{:http-port 8080,
 :db-url "some-db-url",
 :db-password "password123",
 :auth-client :dummy,
 :metrics-enabled false}

Simple global defaults

Malli lets us simply and more clearly state our intended defaults. In our example our Malli schema contains [:http-port {:default 8080} :int], meaning that the HTTP port always has a default of 8080.

Aero’s #or functionality does much the same, though it does have an odd quirk; for some reason Aero converts false to nil when #or is used.

For example, an Aero config.edn file of

{:some-boolean #or [#env SOME_FLAG false]}

would result in

{:some-boolean nil}

This is OK most of the time, so long as the receiving code in question nil-puns it. However, especially when it comes to application configuration, we’re often passing these values down to pure-Java libraries which fall over when passed a nil rather than false.

Why use Aero?

At this point you may be wondering why we’re still using Aero if all we’re basically using it for is grabbing values? The key reason for me is that Aero’s profile functionality lets you have your cake and eat it when it comes to prod VS development defaults.

There’s often a tension between making an application convenient to start up on a dev machine VS making it reliable in production. A common approach is to set global defaults; e.g. the database is assumed to be on localhost if no configuration is specified.

Where the ‘global default’ approach falls down though is in reliability, if for example a typo is accidentally put into a production env variable name. The application falls back to the local default, and so if its container healthchecks are absent or broken it will just uselessly sit there until someone notices the problem in some other way. (Sadly, I’ve seen this happen!)

However, with Aero profiles you get given a simple approach to having it both ways; you can set up convenient defaults that only apply to dev and/or test, whereas any properties that don’t have a sensible default in a production context will be caught at app start-up if they are misconfigured.

Lastly, Aero provides the benefit of a commonly-used convention for getting external values into your application. Especially when pared down to the limited subset of its functionality we’re using here, the things Aero does wouldn’t be very hard to write yourself. But since Aero is so ubiquitous, it provides a lot of value simply by the virtue of putting all the external inputs into a single, commonly-used location.

Summary

Validation of application configuration is a sadly neglected area of development. This is a shame when it’s such a low-cost way of preventing faulty application deployments, as well as making it easier for a dev new to a project to get an app running locally on their machine for the first time.

Thankfully, the combination of Malli & Aero gives you a simple & easy way of making your application config rock-solid.