tech

Templating with Bash

2018-08-14 6 min read

While scripting in Bash, pretty often there is a need to generate a file or few. Some of the configuration options might not be known until the just-before application start or might need to be derived from the environment or context.

Environment Variable Substitution

One off

One of the most common ways to generate a configuration file is by using heredoc literal:

$ SERVER_IP=0.0.0.0 SERVER_PORT=12345 cat > config.json << EOF
> {
>   "ip": "${SERVER_IP}",
>   "port": ${SERVER_PORT}
> }
> EOF

as simple as that, it’s hard to beat it in terms of convenience as you don’t need to worry about many things such as escaping quotes, worry about newlines and so on. We should remember that EOF we see on the first line is an arbitrary string literal and whatever it is, it should be used to terminate the heredoc.

We can verify the statement above very easily:

$ ls -1
config.json

$ cat config.json
{
  "ip": "0.0.0.0",
  "port": 12345
}

Multiple files

The heredoc literal works great until there is a need to deal with an arbitrary number of config files that have to be processed. When that happens, the one needs some script to render a bunch of files by substituting environment variables in them. Luckily, envsubst from gettext package is to the rescue.

Let’s say we want to create a script called render.sh that would take a path to a directory as an argument, find all files like foo.in inside and render them by substituting environment variables into foo.

$ cat > render.sh << 'EOF'
> #!/usr/bin/env bash
> target="${1}"
> templates=$(find ${target} -type f -name "*.in" | xargs echo)
> for sourcePath in ${templates}; do
>   targetPath=$(echo ${sourcePath} | rev | cut -c4- | rev)
>   rm -f ${targetPath}
>   envsubst < ${sourcePath} > ${targetPath}
>   chmod --reference=${sourcePath} ${targetPath}
> done
> EOF

Here we use the same heredoc literal but wrap the EOF on the 1st line into quotes (both single and double quotes can be used). It would prevent environment variables from substitution - exactly what we want when use heredoc to write a script to a file.

Now let’s make it executable:

$ chmod +x render.sh

And create a sample template:

$ cat > config.json.in << 'EOF'
> {
>   "ip": "${SERVER_IP}",
>   "port": ${SERVER_PORT}
> }
> EOF

At this point we’re ready to give it a try:

$ SERVER_IP=0.0.0.0 SERVER_PORT=12345 ./render.sh .

Let’s make sure that config.json.in was rendered into config.json:

$ ls -1
config.json
config.json.in
render.sh

And check its content:

$ cat config.json
{
  "ip": "0.0.0.0",
  "port": 12345
}

So far so good!

Command Evaluation

Unfortunately, sometimes that’s not enough.

Occasionally the one needs to execute some commands to derive values from the runtime environment or to make a more sophisticated decision about the value that should be used.

Let’s say there is an environment variable ${MNEMONIC} that we want to look at and depending on its value we want to adjust our port number to be 1000 in dev, 2000 in prod or equal to our user id otherwise. On top of that, we want ip to be the address assigned to the interface that is used to access the default gateway. Crazy, right? Let’s add one more field and call it proto - it should default to http unless another value is provided via environment variable called PROTOCOL.

Each of the above can be done with Bash quite easily.

We would use case statement for the 1st problem:

$ MNEMONIC=dev bash -c 'case ${MNEMONIC} in dev) echo 1000;; prod) echo 2000;; *) echo $(id -u);; esac'
1000

$ MNEMONIC=prod bash -c 'case ${MNEMONIC} in dev) echo 1000;; prod) echo 2000;; *) echo $(id -u);; esac'
2000

$ bash -c 'case ${MNEMONIC} in dev) echo 1000;; prod) echo 2000;; *) echo $(id -u);; esac'
3501

The last output would be different for everybody.

And the 2nd one is a cascade of ip and grep invocations:

$ ip a show $(ip r | grep -oP 'default .* \K.+') | grep -oP 'inet \K[\d\.]+'
10.75.53.12

The last one is the easiest one - we just going to use “default value” construct available in bash:

$ bash -c 'echo ${PROTOCOL:-http}'
http

$ PROTOCOL=https bash -c 'echo ${PROTOCOL:-http}'
https

One-off

How would we fit the above commands into our template when dealing with a single file? We would use the same heredoc literal as before!

$ cat > config.json << EOF
> {
>   "ip": "$(ip a show $(ip r | grep -oP 'default .* \K.+') | grep -oP 'inet \K[\d\.]+')",
>   "port": $(case ${MNEMONIC} in dev) echo 1000;; prod) echo 2000;; *) echo $(id -u);; esac),
>   "proto": "${PROTOCOL:-http}"
> }
> EOF

And let’s just verify that everything is as expected:

$ ls -1
config.json

$ cat config.json
{
  "ip": "10.75.53.12",
  "port": 3501,
  "proto": "http"
}

Multiple files

As one would expect from its name, command evaluation is beyond envsubst capabilities. Not only that, even “default value” construct as in bash is not supported.

Luckily, there is still a way around. We would use eval builtin function from bash and will feed it with cat and another heredoc literal. This is what it takes to process an arbitrary input while making sure that we will preserve all the quotes, whitespaces etc:

$ cat > render.sh << 'EOF'
> #!/usr/bin/env bash
> target="${1}"
> templates=$(find ${target} -type f -name "*.in" | xargs echo)
> for sourcePath in ${templates}; do
>   targetPath=$(echo ${sourcePath} | rev | cut -c4- | rev)
>   rm -f ${targetPath}
>     eval "cat <<END
> $(<${sourcePath})
> END
> " > ${targetPath}
>   chmod --reference=${sourcePath} ${targetPath}
> done
> EOF

While it should be obvious, we still must stress that “templates” processed with such a rendered are as good as an arbitrary script and therefore one should not accept them from untrusted sources.

Now let’s write an updated template:

$ cat > config.json.in << 'EOF'
> {
>   "ip": "$(ip a show $(ip r | grep -oP 'default .* \K.+') | grep -oP 'inet \K[\d\.]+')",
>   "port": $(case ${MNEMONIC} in dev) echo 1000;; prod) echo 2000;; *) echo $(id -u);; esac),
>   "proto": "${PROTOCOL:-http}"
> }
> EOF

And then render it:

$ MNEMONIC=dev PROTOCOL=https ./render.sh .

The only thing that is left is to verify the generated file:

$ ls -1
config.json
config.json.in
render.sh

And its content:

$ cat config.json
{
  "ip": "10.75.53.12",
  "port": 1000,
  "proto": "https"
}

Done like a terribly cooked steak!