Meet OpenQA

OpenQA is different.

It is also a demostration of the power emerges from the open source world and the beast that it can create. A bit ambiquous and chaotic if you ask me. It is also slow. And if you are looking for a 21st century UI, i doubt it would impress you.

The first time i heard about it, I was like i should certainly try it. I gave up. Everything i knew at the time, i was not able to apply any of them. I used to have an easy approach to start, right? Run some script, setup dev environment, run the software, check out the code and stuff. And some fancy documentation some times.

Instead… Not sure where the documentation is. Too many links, too long pages. Shit do i have to go through all of them just to see my dev env working? The code is in perl. What? really? Any way i gave up quite fast. But i was always curious.

So to get started with the framework is quit disturbing, especially for new contributors. And the testing as a process can also be slow. But it can pay back and testing the installation of a an OS, you do not focus on time matters. nevertheless there are ways to speed up the jobs. Where its power lays is on the things that you can do and is unlimited. Despite it is a distribution installation tool, in fact you can setup and test many other things. For instance, you can test UI as well as backend, you can run test with other tools like selenium, pytest or LTP.

Grab a coffee and Get to know your API

So how is it to work with OpenQA. Every day i can go to the OpenQA instance and check the summary of my test. Ideally i can see them all green which mean they run successfully. This is part of my review, like everyones. Going to each test suite individual you can see each module that the test jobs have scheduled. And each of them has captured a screenshot of the execution that runs. This is depended from your setup. For instance there are tests that use serial console and less screens. In this case you can review what the test did, going to the “Logs & Assets”>serial_terminal.txt. The test suite can be configured to run hidding the terminal output from the “Details” tab, if you want. Check if _QUIET_SCRIPT_CALLS is enabled in the “Settings” tab, where we can find most of the settings of the test suites(but not all*).

In case of the screen matching, you can click and check the screenshot directly. quite intuitive here. Each screen(called a needle) has some tags that they used in the code to be looked up. More on this in another post maybe.

When things go wrong we have a few helping tools to investigate the defect. First you can find the “Investigation” tab which contains some info regarding the changes from the last successful run. Then i will usually check the autoinst-log.txt from “Logs & Assets”. That is the logs from the test runner itself. In case it is not clear or i want to debug it on my local instance, i edit the module in the code, adding some prints or a sleep to make it stop in that particular point before run anything else. Then i can connect into it, usually via vnc and execute commands or check the system. Of course, you are able to do quite the same in the OpenQA instance but it is easier to do changes in your local repo.

And so many things can go wrong. Which makes the debugging hard.

Things can work just fine. Until one small line is changed. And boom. Here is where the fun begins. Everywhere else this works.

Once you have an OpenQA instance up and running on your computer the beginning can be quite easy. Easy as writing linux commands for many cases. Clone a job and start adding linux code through the API. The API gives you perl wrappers like script_run, select_console, mouse_click, save_screenshot or check_screen. This is small amount of the available subroutine. Other times you will create those needles and you will check if this is displayed and act accordandly.

Make more coffee and make things done

I said that OpenQA is slow. And to make this more clear the reasoning is that the when you do a change or add a new module, then you have to shit back and watch the test running. Some modules depends on others so you can not skip them. If your module or change is very late in the flow, good luck. There is not easy way to overcome this. Now think what happens when you work in two or three tasks at the moment. or you want to check something very fast. or you have some requests for changes on your PR!! Better make sure you wont. :) And this is where maybe you spend most of your time.

Anyways… Once it works, you are ready to create a pull_request and ask for reviews.

In this point, you may be sure that the script works for your test case, right? One of the problem with OpenQA is that there is one master branch for any crazy test idea. So that simply means that

• When you change a module, this would probably be used by other test scenarios and it is difficult to know in advance. This would be worse if the module make use of test variables for instance which will change the behaviour. And put it plainly, there are too many of them.
• Every team uses different logic to implement their test. From how the test is scheduled or is triggered to what is run and how the results would produce. Thus, you can see different output of a module, based on whether they want to use a serial terminal or not. Other teams based their tests on needling and other not. Others use yaml scheduling and other lib/main_common.pm. Some creates modules of one-specific purpose and others have long monolithic modules having it do many test in once. Or run some other test framework. I dont even know if that was all but you get the idea.

I guess the question is how can you collaborate efficient between all those teams and projects. Hard to tell in the OpenQA world. There is a lot of legacy code and propably you will not notice until you hit it. But What principle can and usually helps

• find cross references of your methods, libs and modules which you are changing.
• Run as many VRs you can against those tests that you find above.
• Follow the Factory-first principle. Which means write and test first against OpenSUSE versions. Another good practice is to add the test first to the development job group until you make sure that works for some time and then move it to the official job group.

As i said the documentation is not the strongest part of OpenQA. It is not that it doesnt exist but it is scattered and i would say that it is very likely to find out of date context or missing new features. So it is important to be online present and active in the various chat rooms. This is where the most of the communication is happening.

One things that troubles me is the review process. on the OpenQA side there is a very nice coverage with tests and everything. In the other side of the os-autoinst-distri-opensuse things are a bit different. Is the part where the engineer has more responsibility as it is easier to break things. The CI exists but the focus is for things like the static analysis of the code and the scheduling. In additional, as a big legacy code, the review relays to how experience the reviewer is to catch conflicted snippets or enhancements against the big picture.

Having said that, the review is depended from who will give you feedbacks and in general you do not have control over it. It is possible to have one review and merge or go through as many reviewer as they are happened to take a look. From my point of view, a proper review process and quality assurance approach has not been established, resulting with the code to be diversed and is not uniformed. You can get feedback or requests or even the smallest things. The main focus for the current process is the flexibility allegedly but there is space for improvement for sure, without loosing the freedom and the speed to do things here.

Knowledge base for beginners

There are still things i am learning every day. And every day can be different. I think that it is safe to say that you never know what you will have to deal with in the next problem.

But i believe there are things i wish i knew earlier on. And advice anyone who is willing to do something more than just write a sequence of commands to verify an output, to learn more about the serial terminals, vnc over ssh, the linuxrc and the boot process. Obvious perl is a thing but not so worry if you spend some time. What i found also useful to learn and do some homework is with qemu. Like launching an image manually. Something that the openQA actually does underneath to run a test on an ISO. Finally i had to read a few books about virtualisation. Not yet expert but it helps in my opinion. Then the rest is depended from the area of focus. Containers, PublicCloud, security etc.

I usually turn to https://github.com/os-autoinst/os-autoinst/tree/master/doc and i will suggest to take a look at docs (i told you is long) and especially at troubleshooting and test_development_tricks.

TL;DR

Having said a lot and nothing in the same time, i really believe that OpenQA is a unique tool and needs some love. It is powerful and i wish many other distros based their tests on it. There are already a few well-known distros though which i do not know a lot about the work flow and how do they use it. But surelly they do. So every piece of code or distribution on it makes me really proud.