Adventures in the Terraform DSL, Part I: Structured data
by Alex Harvey
This is a blog series aimed at experienced developers who want to learn the Terraform DSL quickly. I assume that the reader has finished the official Getting Started tutorial, has created and destroyed some cloud resources using Terraform, and probably also knows other high-level programming languages like Python or Ruby.
In this first part, I look at Terraform’s data types, the lookup()
and element()
functions, how to address the elements of lists inside maps of lists, and how to address the keys of maps inside lists of maps. Along the way I introduce Terraform’s three types of variables and its data types.
- What is the problem
- Input variables, output variables and local values
- Terraform data types
- Addressing structured data
- Good news - Terraform 0.12-beta2
- The lookup and element functions
- Summary
What is the problem
Addressing structured data in languages like Python, Ruby or Perl is trivial. Given a list mylist
, its n
th element can be addressed using the familiar notation mylist[n]
. And given a map mymap
, the value associated with a key key
can be addressed using the notation mymap[key]
. Then, nested data can be addressed by combining these notations. Thus, the first element of a list associated with the key key
in a map mymap
can be addressed as mymap[key][0]
; and the value associated with key key
in the map in the third element of a list mylist
can be addressed as mylist[2][key]
. And so on.
In Terraform 0.11, the notations mylist[n]
and mymap[key]
are supported, but, when combined, are not. For example, you might expect this to work:
locals {
foo = [{bar = "baz"}]
}
output "qux" {
value = "${local.foo[0]["bar"]}"
}
Testing:
▶ terraform apply
Error: Error loading test.tf: Error reading config for output foo:
parse error at 1:15: expected "}" but found "["
In Terraform 0.12-beta2 however it works fine:
▶ terraform012 apply
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
bar = baz
This post, therefore, is about the workaround. How to use the lookup()
and element()
functions to workaround these problems, and how in general to access nested data in Terraform.
Input variables, output variables and local values
Definitions
Terraform has three different kinds of variables: input variables which serve as parameters in Terraform modules; output values that are returned to the user when Terraform applies and can be queried using the terraform output command; and local values which, strictly-speaking, assign a name to an expression. Or, if an analogy to a function is preferred, then, as the docs note, input variables are analogous to function arguments, output values are analogous to function return values, and local values are analogous to a function’s local variables.
Declaring input variables
Here is an example of declaring an input variable in a module:
variable "zones" {
type = "list"
default = ["us-east-1a", "us-east-1b"]
}
If that variable is an input to a module, mymodule
, then the module can be called and a value passed to that variable like this:
module "mymodule" {
source "./mymodule"
zones = ["ap-southeast-2"]
}
And if nothing was specified for zones, then the default would be used. Specification of the variable type is optional but recommended. It will cause Terraform to fail with a more helpful message if data of an unexpected type is passed in.
Addressing an input variable
To address an input variable, use the notation var.<NAME>
. For example:
variable "key" {
type = "string"
}
output "key" {
value = "${var.key}"
}
Declaring a local value
Now, here is an example of declaring a local value instead:
locals {
common_tags = {
PowerMgt = "business_hours"
Environment = "production"
}
}
There is no way to declare the type of a local value, and Terraform will infer its type.
Note that local values are also referred to in the docs as local named variables, or as variables, or as temporary variables.
Addressing a local value
To address a local value, use the notation local.<NAME>
. For example:
locals {
foo = "bar"
}
output "baz" {
value = "${local.foo}"
}
Declaring an output value
The examples above have already introduced the output value. Once again:
output "addresses" {
value = ["${aws_instance.web.*.public_dns}"]
}
(Note that the splat notation there will be covered in part II on iteration.)
Terraform data types
As we have seen by now, Terraform has a number of data types. These are:
string
: a sequence of Unicode characters representing some text, like “hello”.number
: which, perhaps surprisingly, can represent both integers like 42, and floats like 3.14.bool
: a boolean, i.e. true or false.list
: a sequence of ordered values, like["us-west-1a", "us-west-1c"]
.set
: similar to a list, but an unordered collection of unique values.map
: a collection of values identified by keys, like a dict in Python or a Hash in Ruby, Perl etc, e.g.{name = "Mabel", age = 52}
.
Terraform’s documentation refers to strings, numbers and bools as primitive types and to lists, maps and sets as collection types. In addition to these, there are also structural types:
- object: a map with a schema, for example
object({ name=string, age=number })
. - tuple: a list with a schema, for example
tuple([string, number, bool])
.
The purpose of these types is, much as in Puppet for those familiar with Puppet, validating input data.
In this post, however, I am only interested in strings, lists and maps.
Addressing structured data
For the remainder of this article, I explore the various permutations of the problem of addressing structured data in Terraform 0.11 and 0.12-beta2. As mentioned already, I do this because I take it to be the problem that an experienced developer will need to get their head around when picking up the Terraform DSL.
Addressing a list
In this example I declare a local list and then address the whole list in an output value. (I use output values throughout this post, because Terraform prints those during a terraform apply. It is the closest there is to a “print” statement in Terraform.)
locals {
foo = ["bar", "baz", "qux"]
}
output "quux" {
value = "${local.foo}"
}
Note also the requirement in Terraform 0.11 and earlier to interpolate local.foo
inside a string. As I’ll explain later, that is no longer required in Terraform 0.12.
Now, to test:
▶ terraform apply
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
quux = [
bar,
baz,
qux
]
Addressing an element in a list
Addressing an element in a list is also no big deal. Here’s how it’s done:
locals {
foo = ["bar", "baz", "qux"]
}
output "quux" {
value = "${local.foo[1]}"
}
Testing:
▶ terraform apply
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
quux = baz
Addressing a map
Next, I am going to create a local map variable and reference it to address a whole map:
locals {
foo = {
bar = "baz"
qux = "quux"
}
}
output "quuz" {
value = "${local.foo}"
}
Testing:
▶ terraform apply
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
quuz = {
bar = baz
qux = quux
}
Addressing a key in a map
To address a key in a map, we can again use notation familiar to us from other languages like Python and Ruby. Assuming the same local foo
, I can address the bar
key as:
output "quuz" {
value = "${local.foo["bar"]}"
}
Note also that, like Bash, Terraform allows interpolation of double quotes inside double quotes.
Addressing a list of maps
The fun starts when there is nested data, that is, lists of maps, maps of lists and so on. In this example, I create an output value that outputs a list of maps:
locals {
foo = [
{
bar = "baz"
qux = "quux"
},
{
quuz = "corge"
grault = "garply"
},
]
}
Now, addressing the whole data structure is, as usual, fine:
output "waldo" {
value = "${local.foo}"
}
Addressing an element of a list of maps
Addressing one element of a list of maps is also fine:
output "waldo" {
value = "${local.foo[0]}"
}
Testing:
▶ terraform apply
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
waldo = {
bar = baz
qux = quux
}
Addressing a key in an element of a list of maps
But, if I try to address the key bar
, I get a parse error:
output "waldo" {
value = "${local.foo[0]["bar"]}"
}
Applying:
▶ terraform apply
Error: Error loading terraform-test/test.tf: Error reading config for output waldo:
parse error at 1:15: expected "}" but found "["
Addressing an element in a key in a map of lists
Before I continue, I also wish to introduce the parallel problem of the element within a key of a map of lists. Consider:
locals {
foo = {
bar = ["baz", "qux", "quux"]
quuz = ["corge", "grault", "garply"]
}
}
Once again, addressing the whole structure is fine; addressing one key of that structure is fine:
output "waldo" {
value = "${local.foo["bar"]}"
}
Which leads to:
▶ terraform apply
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
waldo = [
baz,
qux,
quux
]
But addressing an element of that key within the structure also leads to a parse error:
output "waldo" {
value = "${local.foo["bar"][1]}"
}
And:
▶ terraform apply
Error: Error loading terraform-test/test.tf: Error reading config for output waldo:
parse error at 2:19: expected "}" but found "["
Good news - Terraform 0.12-beta2
In the forthcoming Terraform 0.12 this problem appears to be resolved. If I switch to my Terraform 0.12-beta2 binary:
locals {
foo = [
{
bar = "baz"
qux = "quux"
},
{
quuz = "corge"
grault = "garply"
},
]
}
output "waldo" {
value = "${local.foo[0]["bar"]}"
}
And I get:
▶ terraform012 apply
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
waldo = baz
In fact, in Terraform 0.12 there are first-class expressions, meaning it is no longer necessary to wrap the expression in double quotes. Thus, this also works:
output "waldo" {
value = local.foo[0]["bar"]
}
And if I try the other case:
locals {
foo = {
bar = ["baz", "qux", "quux"]
quuz = ["corge", "grault", "garply"]
}
}
output "waldo" {
value = local.foo["bar"][1]
}
I get:
▶ terraform012 apply
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
waldo = qux
Even something arbitrarily complex like this works in 0.12:
locals {
foo = {
bar = [
{
baz = "qux"
quux = ["quuz", "corge"]
}
]
}
}
output "waldo" {
value = local.foo["bar"][0]["quux"][1]
}
The lookup and element functions
Using lookup
In the mean time, the normal way of working around this problem in Terraform is via the lookup()
and element()
functions. Switching back to Terraform 0.11 and this example:
locals {
foo = [
{
bar = "baz"
qux = "quux"
},
{
quuz = "corge"
grault = "garply"
},
]
}
I can access the key “bar” using the lookup function as follows:
output "waldo" {
value = "${lookup(local.foo[0], "bar")}"
}
That’s quite inconvenient and ugly of course and I expect that the release of Terraform 0.12 will lead to code like this being gradually refactored. For the moment, that’s how it’s done.
Using element
Switching to the other example:
locals {
foo = {
bar = ["baz", "qux", "quux"]
quuz = ["corge", "grault", "garply"]
}
}
And I can use element as:
output "waldo" {
value = "${element(local.foo["bar"], 1)}"
}
More complex
Returning to the arbitrarily complex example from before:
locals {
foo = {
bar = [
{
baz = "qux"
quux = ["quuz", "corge"]
}
]
}
}
Unfortunately, it appears impossible to deal with that even using lookup/element. Because:
output "waldo" {
value = "${element(local.foo["bar"], 0)}"
}
Leads to:
▶ terraform apply
Error: output.waldo: element: element() may only be used with flat lists, this list contains elements
of type map in:
${element(local.foo["bar"], 0)}
Similar restrictions apply on the lookup function too:
variable "foo" {
type = "list"
default = [
{
foo = ["bar", "baz", "qux"]
}
]
}
output "foo" {
value = "${lookup(var.foo[0], "foo")}"
}
And this leads to:
▶ terraform apply
Error: output.foo: lookup: lookup() may only be used with flat maps, this map contains elements of
type list in:
${lookup(var.foo[0], "foo")}
As mentioned in the docs here:
…This function only works on flat maps and will return an error for maps that include nested lists or maps.
See also this comment in GitHub here:
…Terraform’s type system doesn’t currently allow a function to return a different type depending on its input arguments and thus, as you’ve seen, the functions are all defined to return strings.
Summary
That is the end of part 1. If you are new to Terraform, you may like to just wait for Terraform 0.12! If on the other hand, you need to, or want to, learn Terraform 0.11 and earlier, I have shown in this post how to address complex, nested data in the Terraform DSL, and some of the limitations. Along the way, I covered the Terraform variable types and its data types.
Stay tuned for Part II, where I will look at iteration in the Terraform DSL.
tags: terraform