Timeline support for runltp

[priyama2]

Since runltp is getting deprecated and we need to move with https://github.com/linux-test-project/kirk/
It would be helpful if it is known till how long the runltp is supported

[chetjain]

@pevik @metan-ucw We have some wrapper tool, written based on runltp and in case, if this moves out of support, our tool breaks. Please help us in knowing the support and timeline

[metan-ucw]

@chetjain we are not in a hurry for the replacement, there will be at least two LTP releases before runltp is dropped.

I had a quick look a the tooling you have and it does not seem that replacing runltp with kirk would be that difficult, you just need to build a different command to run and then instead of parsing the LTP output with grep you will have a json file generated by kirk that is actually easier to use.

[chetjain]

@metan-ucw @pevik

The wrapper tool what we have been using to test the Linux kernel has an excellent way of dispatching the individual testcases independently. It also has an intelligence to know what can be run together and what cannot.
Now, limiting the tool just for the suite level execution breaks the integration and stress part during system test.

We are concerned how much of this could remain intact with kirk knowing all these limitations. I understand, kirk is done for a different purpose, but we are losing the legacy method of execution, which might start inducing the stress & integration test gaps going forward. We need to be cautious before retiring runltp and extend the support for few more years.

kirk_runltp_limitations (2).xlsx

[pevik]

It also has an intelligence to know what can be run together and what cannot.

I wonder what exactly you mean. Also, I did not get why you don't run the entire runtest file as this is expected.

FYI We plan with kirk and metadata from struct tst_test to run in the future tests in paralell, see: https://people.kernel.org/metan/towards-parallel-kernel-test-runs
We store that data in metadata/ltp.json (built runtime). Maybe you're trying to solve something we are on track to solve.

IMHO it'd be better to adapt your way of working to what LTP expects as you'll loose much of improvements in the future. But you really depend on some runltp behavior you may just endup just vendoring runltp. But FYI Red Hat and Linaro already moved to kirk, we (SUSE) too.

[metan-ucw]

Rather than discussing how long is runltp going to be maintained we should focus on getting kirk better so that runltp does not have to be maintained anymore.

[chetjain]

It also has an intelligence to know what can be run together and what cannot.

I wonder what exactly you mean. Also, I did not get why you don't run the entire runtest file as this is expected.

FYI We plan with kirk and metadata from struct tst_test to run in the future tests in paralell, see: https://people.kernel.org/metan/towards-parallel-kernel-test-runs We store that data in metadata/ltp.json (built runtime). Maybe you're trying to solve something we are on track to solve.

IMHO it'd be better to adapt your way of working to what LTP expects as you'll loose much of improvements in the future. But you really depend on some runltp behavior you may just endup just vendoring runltp. But FYI Red Hat and Linaro already moved to kirk, we (SUSE) too.

During integration stress test, we plan to run the valid combinations of individual testcase from multiple testsuite in parallel (guarded with the real-time resource check). This way, we see the behavior of the system from all components perspective. But, I agree running the whole suite makes sense during functional/component testing.

Anyway, we can try to align and use kirk.

Thank you for agreeing to give support on the below issues. Looking forward to see the code changes.
linux-test-project/kirk#29
linux-test-project/kirk#31

BTW, I am interested to know how you are developing "-parallel-kernel-test-runs", but I see runltp-ng is archived/closed. Is the work still in plan or thrashed?

[acerv]

kirk already supports parallel execution, but we can't execute every test in parallel due to their nature. Please take a look at the algorithm used to select what tests can run in parallel: https://github.com/linux-test-project/kirk/blob/e0d6a2dd39f93b1bd694571b05b4f251709d61bf/libkirk/ltp.py#L151

Executing every test in parallel is a mistake due to how tests might interact to each other, that's why there's not a blindly parallelization as it was in runltp. In any case, kirk also supports --force-parallel if someone wants to get hurt :-)