Skip to content

Create simple system test framework in Python

Currently, all of our system tests are fully in bash. They have a few problems:

  • They produce a lot of logs, often without the distinction of where a specific test case started/ended
  • A lot of bash commands are not checked properly for failures, making tests sometimes fail silently
  • Most importantly, the scripts are an absolute nightmare to maintain and extent

Moving to Python should solve these issues. I don't want to suggest moving fully to Python, but at least rewriting the top-level scripts (e.g. test_client.sh) in Python should provide a lot of benefits.

Two important things should be kept in mind:

  • The testing setup should remain somewhat portable. E.g. executing this on a (pre)production instance should be possible. On the machines/pods themselves, we should not need to install any additional packages
  • The performance should not be impacted negatively. This is especially important for the stress test. This should be easy provided we keep the performance critical commands in bash

I'll look into creating a simple framework with some convenience functions and make a proof-of-concept with a test-client.sh replacement