Skip to content

e2e-test-guide.md

Latest commit

 

History

History
178 lines (128 loc) · 5.64 KB

e2e-test-guide.md

File metadata and controls

178 lines (128 loc) · 5.64 KB

Argo CD Operator E2E Test Guide

E2E tests are written using KUTTL.

Requirements

This test suite assumes that an Argo CD Operator is installed on the cluster or running locally using ARGOCD_CLUSTER_CONFIG_NAMESPACES=argocd-e2e-cluster-config make install run.

The system executing the tests must have following tools installed:

  • kuttl kubectl plugin (>= v0.11.1)
  • oc and kubectl client
  • jq for parsing JSON data
  • curl

There should be a kubeconfig pointing to your cluster, user should have full admin privileges (i.e. kubeadm).

!!! note E2E tests utilize GNU Grep under the hood. Please make sure that you have the GNU compatible grep installed.

If you are on OSX you can install GNU compatible grep using the below command. The package is installed as ggrep by default. Please set this(ggrep) as an alias to grep.

brew install grep

Use the below commands to install GNU compatible grep on OSX.

Also, note that the e2e tests for Redis HA mode require a cluster with at least three worker nodes. A local three-worker node cluster can be created using k3d

Running the tests

In any case, you should have set up your kubeconfig in such a way that your default context points to the cluster you want to test. You can use the kubectl login ... command to set this up for you.

Run e2e tests

make e2e

Run Operator locally and execute e2e tests

make all

Running manual with kuttl

kubectl kuttl test ./tests/k8s --config ./tests/kuttl-tests.yaml

Running single tests

Sometimes (e.g. when initially writing a test or troubleshooting an existing one), you may want to run single test cases isolated. To do so, you can pass the name of the test using --test to kuttl, i.e.

kubectl kuttl test ./tests/k8s --config ./tests/kuttl-tests.yaml --test 1-004_validate_namespace_scoped_install

The name of the test is the name of the directory containing its steps and assertions.

If you are troubleshooting, you may want to prevent kuttl from deleting the test's namespace afterwards. In order to do so, just pass the additional flag --skip-delete to above command.

Writing new tests

Name of the test

Each test comes in its own directory, containing all its test steps. The name of the test is defined by the name of this directory.

The name of the test should be short, but expressive. The format for naming a test is currently <test ID>_<short description>.

The <test ID> is the serial number of the test as defined in the Test Plan document. The <short description> is exactly that, a short description of what happens in the test.

Name of the test steps

Each test step is a unique YAML file within the test's directory. The name of the step is defined by its file name.

The test steps must be named XX-<name>.yaml. This is a kuttl convention and cannot be overriden. XX is a number (prefixed with 0, so step 1 must be 01), and <name> is a free form value for the test step.

There are two reserved words you cannot use for <name>:

  • assert contains positive assertions (i.e. resources that must exist) and
  • errors contains negative assertions (i.e. resources that must not exist)

Refer to the kuttl documentation for more information.

Documentation

Documentation is important, even for tests. You can should provide inline documentation in your YAML files (using comments) and a README.md in your test case's directory. The README.md should provide some context for the test case, e.g. what it tries to assert for under which circumstances. This will help others in troubleshooting failing tests.

Recipes

kuttl unfortunately neither encourages or supports re-use of your test steps and assertions yet.

Generally, you should try to use assert and errors declaration whenever possible and viable. For some cases, you may need to use custom scripts to get the results you are looking for.

Scripts general

Scripts can be executed in a kuttl.dev/TestStep resources from a usual test step declaration.

Your script probably will retrieve some information, and asserts it state. If the assertion fails, the script should exit with a code > 0, and also print some information why it failed, e.g.

apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- script: |
    # Get some piece of information...
    if test "$result" != "expected"; then
      echo "Expectation failed, should 'expected', is '$result'"
      exit 1
    fi

Also, you may want to use set -e and set -o pipefail at the top of your script to catch unexpected errors as test case failures, e.g.

apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- script: |
    set -e
    set -o pipefail
    # rest of your script

Getting values of a resource's environment variables

YAML declarations used in assert or errors files unfortunately don't handle arrays very well yet. You will always have to specify the complete expectation, i.e. the complete array.

If you are just interested in a certain variable, and don't care about the rest, you can use a script similar to the following using jq. E.g. to get the value of a variable named FOO for the argocd-server deployment in the test's namespace:

apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- script: |
    val=$(kubectl get -n $NAMESPACE deployments argocd-server -o json \
      | jq -r '.spec.templates.spec.containers[0].env[]|select(.name=="FOO").value')
    if test "$val" != "bar"; then
      echo "Expectation failed for for env FOO in argocd-server: should 'bar', is '$val'"
      exit 1
    fi