Design prototype: Replacement for shell tests

[ffrank]

We need a robust and powerful mechanism to implement end-to-end acceptance tests, i.e. make mgmt run actual code, and examine the state of the system afterwards (possibly even DURING the run).

This prototype was written quick-and-dirty. It was specifically not engineered and I am not looking for feedback on the code.

Instead, I'd like thoughts on the approach and design, specifically:

• is a YAML specification the right approach?
• obvious flaws in the way tests are specified?

But any discussion based on this starting point are welcome.

As said, quick and dirty, if this results in a suggestion for a fundamentally different approach, that is fine also. Happy to scrap this prototype in favor of a whole new one.

[ffrank]

Can be tested using

go run test/mgmtest/mgmtest.go test/mgmtest/file-mode.yml

The mount.yml does not work with current master because MountRes does not support tmpfs. (It does work when rebased to the remount branch as seen in #744 ). It does show how this approach improves upon shell scripting (I hope).

I went for YAML because why not. We could also write something that is more native to golang, but frankly, the information density in YAML is pretty nice, and I expect that we can do anything we need here.

What I still have not built is an example of running mgmt, converging, then messing up the state from shell commands or whatever, and converging again. But we can do lots of funky things.

Currently the prototype relies on mgmt writing nothing to stderr while converged, but this can be expanded to a filter for specific log lines e.g. CheckApply.

[purpleidea]

There is already an integration package which could be greatly expanded if that's the right approach.

There is already a step test system for resources ( https://github.com/purpleidea/mgmt/blob/master/engine/resources/resources_test.go#L66 ) which could be greatly expanded if that's the right approach.

Lastly there is TestAstFunc3 which could be greatly expanded if that's the right approach.

I think the next_big_step in testing real-time systems is actually using mgmt itself to do so, and to do that we need more pieces first.

In the meantime we probably need a better mechanism for testing resources in pure golang.

I'm closing this PR for now, please keep design discussions in matrix until our mailing list is back up.

Thanks!

[ffrank]

There is already an integration package which could be greatly expanded if that's the right approach.

Integration testing is not acceptance testing; both are useful.

There is already a step test system for resources ( https://github.com/purpleidea/mgmt/blob/master/engine/resources/resources_test.go#L66 ) which could be greatly expanded if that's the right approach.

Yes, that one is also a really good integration/unit test approach that we should keep.

Lastly there is TestAstFunc3 which could be greatly expanded if that's the right approach.

Ah cheers, TIL. This is also a different level of granularity though.

I think the next_big_step in testing real-time systems is actually using mgmt itself to do so, and to do that we need more pieces first.

Interesting idea, fair enough. If that's on the horizon, doesn't really make sense to invest in the approach sketched in this PR.

In the meantime we probably need a better mechanism for testing resources in pure golang.

Yes, definitely agree.

I'm closing this PR for now, please keep design discussions in matrix until our mailing list is back up.

Chat is a terrible place for that, or is there some kind of indexed archive where we could later look up what was said wrt. a certain topic?

[purpleidea]

Integration testing is not acceptance testing; both are useful.

👍

Yes, that one is also a really good integration/unit test approach that we should keep.

There's obviously some value in the ideas you PR'ed here, but please read through some of the existing code first, and then we can circle back here if needed.

Interesting idea, fair enough. If that's on the horizon, doesn't really make sense to invest in the approach sketched in this PR.

Not sure how far away it is. Not short though.

Chat is a terrible place for that, or is there some kind of indexed archive where we could later look up what was said wrt. a certain topic?

I agree. Matrix is the best we have for now. I'm waiting on a mailing list host.