Add templating to porter config file (#1879)

* Add templating to porter config file

Support templating in the Porter config file, e.g.
~/.porter/config.json|toml|yaml.

The template syntax uses a yaml-friendly delimiiter, ${}. For example,
${env.NAME} or ${secret.NAME}. If you are using toml or json for your
config file format, then you may need to use quotes around template
values to ensure that the config file has valid syntax when it is
loaded.

We load the config file first with only environment variables
substituted, then we initialize the secrets plugin, and finally do a
second pass of the config file to replace any secrets. So that first
pass needs to be a valid file based on the current format.

For example, in a porter.toml file, secrets should be quoted:

```toml
[[storage]]
  name = "dev"
  plugin = "mongodb"

  [storage.config]
    url = "${secret.connection-string}"
```

Only environment variables and secrets can be substituted. The secrets
are resolved from the default secret storage.

I am implementing this with the liquid template engine (from shopify
that was ported to Go). I don't want us to commit to supporting liquid
templates at this time, it's just an implementaiton detail that liquid
is used at the moment and it could change later.

Signed-off-by: Carolyn Van Slyck <me@carolynvanslyck.com>

* Review feedback

Signed-off-by: Carolyn Van Slyck <me@carolynvanslyck.com>

Author: carolynvs

.gitignore

@@ -1,4 +1,5 @@
 /docs/public/
+/docs/content/operator
 /bin
 /porter
 .DS_Store

cmd/porter/main.go

@@ -23,7 +23,12 @@ var usageText string
 
 func main() {
 	run := func() int {
+		ctx := context.Background()
 		p := porter.New()
+		if err := p.Connect(ctx); err != nil {
+			fmt.Fprintln(os.Stderr, err.Error())
+			os.Exit(1)
+		}
 
 		rootCmd := buildRootCommandFrom(p)
 
@@ -116,8 +121,9 @@ Try our QuickStart https://porter.sh/quickstart to learn how to use Porter.
 			case "porter", "help", "version", "docs":
 				return nil
 			default:
+				// Reload configuration with the now parsed cli flags
 				p.DataLoader = cli.LoadHierarchicalConfig(cmd)
-				err := p.LoadData()
+				err := p.Connect(cmd.Context())
 				if err != nil {
 					return err
 				}

docs/content/configuration.md

@@ -3,99 +3,171 @@ title: Configuration
 description: Controlling Porter with its config file, environment variables and flags
 ---
 
+Porter has a hierarchical configuration system that loads configuration values in the following precedence order:
+
+* Flags (highest)
+* Environment Variables
+* Config File (lowest)
+
+You may set a default value for a configuration value in the config file, override it with an environment variable, and then override both for a particular command with a flag.
+
 * [Flags](#flags)
 * [Environment Variables](#environment-variables)
 * [Config File](#config-file)
 * [Experimental Feature Flags](#experimental-feature-flags)
   * [Build Drivers](#build-drivers)
   * [Structured Logs](#structured-logs)
+* [Common Configuration Settings](#common-configuration-settings)
+  * [Set Current Namespace](#namespace)
+  * [Enable Debug Output](#debug)
+  * [Debug Plugins](#debug-plugins)
+  * [Output Formatting](#output)
+* [Allow Docker Host Access](#allow-docker-host-access)
 
-Porter's configuration system has a precedence order:
+## Flags
 
-* Flags (highest)
-* Environment Variables
-* Config File (lowest)
+Nearly all of Porter's configuration, except global configuration such as secret accounts, storage accounts, or telemetry, are configurable by flags.
+Use the `porter help` command to view available flags.
 
-You may set a default value for a configuration value in the config file,
-override it in a shell session with an environment variable and then override
-both in a particular command with a flag.
+## Environment Variables
 
-* [Set Current Namespace](#namespace)
-* [Enable Debug Output](#debug)
-* [Debug Plugins](#debug-plugins)
-* [Output Formatting](#output)
-* [Allow Docker Host Access](#allow-docker-host-access)
+Flags have corresponding environment variables that you can use so that you don't need to manually set the flag every time.
+The flag will default to the value of the environment variable, when defined.
+Global configuration settings can also be specified with an environment variable.
+For example, the experimental config file setting maps to PORTER_EXPERIMENTAL, and accepts a comma-separated list of values.
 
-## Flags
+\--flag maps to the environment variable of PORTER_FLAG.
+Dashes in the flag name are represented as underscores in the environment variable name.
+So \--another-flag maps to the environment variable PORTER_ANOTHER_FLAG
 
-### Namespace
-`--namespace` specifies the current namespace.
+For example, you can set PORTER_OUTPUT=json and then all subsequent porter commands will act as though the \--output=json flag was passed.
 
-### Debug
+## Config File
 
-`--debug` is a flag that is understood not only by the porter client but also the
-runtime and most mixins. They may use it to print additional information that
-may be useful when you think you may have found a bug, when you want to know
-what commands they are executing, or when you need really verbose output to send
-to the developers.
+Porter's configuration file is located in the PORTER_HOME directory, by default ~/.porter/.
+The file name should be config.FILE_EXTENSION, where the file extension can be json, toml, yaml, or hcl.
+For example, If you defined the configuration in YAML, the file is named config.yaml.
 
-### Debug Plugins
+Do not embed sensitive data in the configuration file.
+Instead, use templates to inject environment variables or secrets in the configuration file.
+Environment variables are specified with ${env.NAME}, where name is case-sensitive.
+Secrets are specified with ${secret.KEY} and case sensitivity depends upon the secrets plugin used.
 
-`--debug-plugins` controls if logs related to communication
-between porter and its plugins should be printed when debugging. This can be _very_
-verbose, so it is not turned on by default when debug is true.
+Below is an example configuration file in TOML:
 
-### Output
+```toml
+# ~/.porter/config.toml
 
-`--output` controls the format of the output printed by porter. Each command
-supports a different set of allowed outputs though usually there is some
-combination of: `table`, `json` and `yaml`.
+# Set the default namespace
+namespace = "dev"
 
-### Allow Docker Host Access
+# Include debug logs
+debug = true
 
-`--allow-docker-host-access` controls whether the local Docker daemon
-should be made available to executing bundles. This flag is available for the
-following commands: [install], [upgrade], [invoke] and [uninstall]. When this
-value is set to true, bundles are executed in a privileged container with the
-docker socket mounted. This allows you to use Docker from within your bundle,
-such as `docker push`, `docker-compose`, or docker-in-docker.
+# Include debug logs from the plugins
+debug-plugins = true
 
-🚨 **There are security implications to enabling access! You should trust any
-bundles that you execute with this setting enabled as it gives them elevated 
-access to the host machine.**
+# Default command output to JSON
+output = "json"
 
-⚠️️ This configuration setting is only available when you are in an environment 
-that provides access to the local docker daemon. Therefore it does not work with
-the Azure Cloud Shell driver.
+# Allow all bundles access to the Docker Host
+allow-docker-host-access = true
 
-## Environment Variables
+# Enable experimental v1 features
+experimental = ["build-drivers", "structured-logs"]
 
-Flags have corresponding environment variables that you can use so that you
-don't need to manually set the flag every time. The flag will default to the
-value of the environment variable, when defined.
+# Use Docker buildkit to build the bundle
+build-driver = "buildkit"
 
-`--flag` has a corresponding environment variable of `PORTER_FLAG` and `--another-flag`
-corresponds to the environment variable `PORTER_ANOTHER_FLAG`.
+# Use the storage configuration named devdb
+default-storage = "devdb"
+
+# When default-storage is not set, use the mongodb-docker plugin.
+# This mode does not support additional configuration for the plugin.
+# If the plugin requires configuration, use default-storage and define
+# the configuration in the storage section.
+default-storage-plugin = "mongodb-docker"
+
+# Use the secrets configuration named mysecrets
+default-secrets = "mysecrets"
+
+# When default-secrets is not set, use the kubernetes.secret plugin.
+# This mode does not support additional configuration for the plugin.
+# If the plugin requires configuration, use default-secrets and define
+# the configuration in the secrets section.
+default-secrets-plugin = "kubernetes.secret"
+
+# Defines storage accounts
+[[storage]]
+  # The storage account name
+  name = "devdb"
+
+  # The plugin used to access the storage account
+  plugin = "mongodb"
+
+  # Additional configuration for storage account
+  # These values vary depending on the plugin used
+  [storage.config]
+    # The mongodb connection string
+    url = "${secret.porter-db-connection-string}"
+    
+    # Timeout for database queries
+    timeout = 300
+
+# Define secret store accounts
+[[secrets]]
+  # The secret store name
+  name = "mysecrets"
+  
+  # The plugin used to access the secret store account
+  plugin = "azure.keyvault"
+
+  # Additional configuration for secret store account
+  # These values vary depending on the plugin used
+  [secrets.config]
+    # The name of the secret vault
+    vault = "topsecret"
+    
+    # The subscription where the vault is defined
+    subscription-id = "${env.AZURE_SUBSCRIPTION_ID}"
+
+# Log command output to a file in PORTER_HOME/logs/
+[logs]
+  # Log command output to a file
+  enabled = true
 
-For example, you can set `PORTER_DEBUG=true` and then all subsequent porter
-commands will act as though the `--debug` flag was passed.
+  # Sets the log level for what is written to the file
+  # Allowed values: debug, info, warn, error
+  level = "info"
 
-## Config File
+# Send trace and log data to an Open Telemetry collector
+[telemetry]
+  # Enable trace collection
+  enabled = true
 
-Common settings can be defaulted in the config file. The config file is located in
-the PORTER_HOME directory (**~/.porter**), is named **config** and can be in any
-of the following file types: JSON, TOML, YAML, HCL, envfile and Java Properties
-files.
+  # Send telemetry via the grpc protocol
+  # Allowed values: http/protobuf, grpc
+  protocol = "grpc"
 
-Below is an example configuration file in TOML
+  # The Open Telemetry collector endpoint
+  endpoint = "127.0.0.1:4318"
 
-**~/.porter/config.toml**
-```toml
-namespace = "dev"
-debug = true
-debug-plugins = true
-output = "json"
-allow-docker-host-access = true
+  # Specify if the collector endpoint is secured with TLS
+  insecure = true
+
+  # Specify a certificate to connect to the collector endpoint
+  certificate = "/home/me/some-cert.pem"
+
+  # The compression type used when communicating with the collector endpoint
+  compression = "gzip"
+ 
+  # The timeout enforced when communicating with the collector endpoint
+  timeout = "3s"
+
+  # Additional headers to send to the open telemetry collector
+  [telemetry.headers]
+    environment = "dev"
+    owner = "myusername"
 ```
 
 ## Experimental Feature Flags
@@ -104,11 +176,11 @@ Porter sometimes uses feature flags to release new functionality for users to
 evaluate, without affecting the stability of Porter. You can enable an experimental
 feature by:
 
-* Using the experimental global flag `--experimental flagA,flagB`.
+* Using the experimental global flag \--experimental flagA,flagB.
   The value is a comma-separated list of strings.
-* Setting the PORTER_EXPERIMENTAL environment variable like so `PORTER_EXPERIMENTAL=flagA,flagB`.
+* Setting the PORTER_EXPERIMENTAL environment variable like so PORTER_EXPERIMENTAL=flagA,flagB.
   The value is a comma-separated list of strings.
-* Setting the experimental field in the configuration file like so `experimental = ["flagA","flagB"]`.
+* Setting the experimental field in the configuration file like so experimental = ["flagA","flagB"].
   The value is an array of strings.
 
 ### Build Drivers
@@ -117,11 +189,11 @@ The **build-drivers** experimental feature flag enables using a different
 driver to build OCI images used by the bundle, such as the installer.
 
 You can set your desired driver with either using `porter build --driver`,
-`PORTER_BUILD_DRIVER` environment variable, or in the configuration file with
-`build-driver = "DRIVER"`
+PORTER_BUILD_DRIVER environment variable, or in the configuration file with
+build-driver = "DRIVER".
 
-The default driver is [Docker], and the full list of available drivers
-is below.
+The default driver is docker, and the full list of available drivers
+is below:
 
 * **Docker**: Build an OCI image using the [Docker library], without buildkit support.
   This requires access to a Docker daemon, either locally or remote.
@@ -212,4 +284,62 @@ experimental = ["structured-logs"]
     owner = "me"
 ```
 
-[otel]: https://github.com/open-telemetry/opentelemetry-specification/blob/v1.8.0/specification/protocol/exporter.md
+[otel]: https://github.com/open-telemetry/opentelemetry-specification/blob/v1.8.0/specification/protocol/exporter.md
+
+## Common Configuration Settings
+
+Some configuration settings are applicable to many of Porter's commands and to save time you may want to set these values in the configuration file or with environment variables.
+
+### Namespace
+\--namespace specifies the current namespace.
+It is set with the PORTER_NAMESPACE environment variable.
+
+```toml
+namespace = "dev"
+```
+
+### Debug
+
+\--debug is a flag that is understood not only by the porter client but also the runtime and most mixins.
+They may use it to print additional information that may be useful when you think you may have found a bug, when you want to know what commands they are executing, or when you need really verbose output to send
+to the developers.
+It is set with the PORTER_DEBUG environment variable.
+
+```toml
+debug = true
+```
+
+### Debug Plugins
+
+\--debug-plugins controls if logs related to communication between porter and its plugins should be printed when debugging.
+This can be _very_ verbose, so it is not turned on by default when debug is true.
+It is set with the PORTER_DEBUG_PLUGINS environment variable.
+
+```toml
+debug-plugins = true
+```
+
+### Output
+
+\--output controls the format of the command output printed by porter.
+It is set with the PORTER_OUTPUT environment variable.
+Each command supports a different set of allowed outputs though usually there is some combination of: plaintext, json, and yaml.
+
+```toml
+output = "json"
+```
+
+### Allow Docker Host Access
+
+\--allow-docker-host-access controls whether the local Docker daemon or host should be made available to executing bundles.
+It is set with the PORTER_ALLOW_DOCKER_HOST_ACCESS environment variable.
+
+This flag is available for the following commands: install, upgrade, invoke, and uninstall.
+When this value is set to true, bundles are executed in a privileged container with the docker socket mounted.
+This allows you to use Docker from within your bundle, such as `docker push`, `docker-compose`, or docker-in-docker.
+
+🚨 **There are security implications to enabling access!
+You should trust any bundles that you execute with this setting enabled as it gives them elevated access to the host machine.**
+
+⚠️️ This configuration setting is only available when you are in an environment that provides access to the local docker daemon.
+Therefore, it does not work with the Azure Cloud Shell driver.

go.mod

@@ -19,6 +19,10 @@ replace (
 	// go.mod doesn't propogate replacements in the dependency graph so I'm copying this from github.com/moby/buildkit
 	github.com/jaguilar/vt100 => github.com/tonistiigi/vt100 v0.0.0-20190402012908-ad4c4a574305
 
+	// expose-ast
+	// https://github.com/osteele/liquid/pull/59
+	github.com/osteele/liquid => github.com/carolynvs/liquid v1.2.5-0.20220131221838-2e107bef298f
+
 	// Fixes https://github.com/spf13/viper/issues/761
 	github.com/spf13/viper => github.com/getporter/viper v1.7.1-porter.2.0.20210514172839-3ea827168363
 )
@@ -61,6 +65,7 @@ require (
 	github.com/moby/term v0.0.0-20201216013528-df9cb8a40635
 	github.com/olekukonko/tablewriter v0.0.4
 	github.com/opencontainers/go-digest v1.0.0
+	github.com/osteele/liquid v1.2.4
 	github.com/pelletier/go-toml v1.9.1
 	github.com/pivotal/image-relocation v0.0.0-20191111101224-e94aff6df06c
 	github.com/pkg/errors v0.9.1

go.sum

@@ -251,6 +251,8 @@ github.com/carolynvs/cnab-go v0.20.2-0.20210805155536-9a543e0636f4 h1:w6gndqIhqX
 github.com/carolynvs/cnab-go v0.20.2-0.20210805155536-9a543e0636f4/go.mod h1:u/Y7piTNJuFs2KfQqmda6uqIx4rtqQ73H6IW7gudz7E=
 github.com/carolynvs/datetime-printer v0.2.0 h1:Td3FU4YGzx0OogCMhCmLBTUTDPQcq0xlgCeMhAKZmMc=
 github.com/carolynvs/datetime-printer v0.2.0/go.mod h1:p9W8ZUhmQUOVD5kiDuGXwRG65/nTkZWlLylY7s+Qw2k=
+github.com/carolynvs/liquid v1.2.5-0.20220131221838-2e107bef298f h1:VQBTZqr7lKJ7I5aibSzwlDd9QiPIJ2vZCYlHB4tUfuQ=
+github.com/carolynvs/liquid v1.2.5-0.20220131221838-2e107bef298f/go.mod h1:w8U5mURyI2WkBkOqadQ8C2W+oK+8TDGwo8V612sRSAI=
 github.com/carolynvs/magex v0.6.0 h1:rzz4RnBiR8hr2WYEsmq+mqkRLEstPnEK8ZP9MgxNY9Y=
 github.com/carolynvs/magex v0.6.0/go.mod h1:hqaEkr9TAv+kFb/5wgDiTdszF13rpe0Q+bWHmTe6N74=
 github.com/casbin/casbin/v2 v2.1.2/go.mod h1:YcPU1XXisHhLzuxH9coDNf2FbKpjGlbCg3n9yuLkIJQ=
@@ -1184,6 +1186,8 @@ github.com/openzipkin/zipkin-go v0.1.3/go.mod h1:NtoC/o8u3JlF1lSlyPNswIbeQH9bJTm
 github.com/openzipkin/zipkin-go v0.1.6/go.mod h1:QgAqvLzwWbR/WpD4A3cGpPtJrZXNIiJc5AZX7/PBEpw=
 github.com/openzipkin/zipkin-go v0.2.1/go.mod h1:NaW6tEwdmWMaCDZzg8sh+IBNOxHMPnhQw8ySjnjRyN4=
 github.com/openzipkin/zipkin-go v0.2.2/go.mod h1:NaW6tEwdmWMaCDZzg8sh+IBNOxHMPnhQw8ySjnjRyN4=
+github.com/osteele/tuesday v1.0.3 h1:SrCmo6sWwSgnvs1bivmXLvD7Ko9+aJvvkmDjB5G4FTU=
+github.com/osteele/tuesday v1.0.3/go.mod h1:pREKpE+L03UFuR+hiznj3q7j3qB1rUZ4XfKejwWFF2M=
 github.com/pact-foundation/pact-go v1.0.4/go.mod h1:uExwJY4kCzNPcHRj+hCR/HBbOOIwwtUjcrb0b5/5kLM=
 github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
 github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k=