Automate Qt apps using Spix and PyAutoGUI
In this post, I show how to use Spix together with PyAutoGUI to control a QtQuick/QML based app.
To make this possible, I recently update Spix, my open-source test automation library for Qt, so that it can work together with 3rd party tools. This allows you to use Spix to find UI elements in the QtQuick application and then use a different tool to post events on the system level, rather than the Qt-internal level. You could even use this to simulate touch events on the hardware layer of an embedded device.
Why not just use Spix alone?
Spix can be used standalone and doesn’t require an external tool to generate events, so why would you use one? The important difference is external vs. internal events, where Spix on its own can only generate internal ones:
Internal Events
Spix simulates mouse clicks or other user interaction by creating fake QEvents and sending them directly to a QtQuick item on the screen. The advantage of this is that it works the same on every system your app runs on, as it doesn’t rely on any system calls. You also don’t need to have an additional application running with special privileges to post events on the system level.
The downside is that not all user interactions can be simulated reliably this way.
External Events
The alternative approach is to not post events inside of Qt, but rather post events on the system level. This means that the mouse will actually move on the screen and Qt will receive the events from the system. The advantage of this approach is that all kinds of interaction can be simulated as Qt will receive the events the same way as if they were the result of real user interaction.
The downside of this approach is that you need another application running in the background that has the right privileges to post events on the system level. This is what we use PyAutoGUI for in this example.
How to use Spix in your project
If you have never used Spix before, you can have a look at the ui testing examples in the Spix repo to get started.
Usually, you only need a few lines in your main() to have Spix running. Use the following code to control your app via the network or from another tool (like PyAutoGUI):
spix::AnyRpcServer server;
auto bot = new spix::QtQmlBot();
bot->runTestServer(server);
This is also done in the RemoteCtrl example.
Now, when you run your application, it opens a port through which it can be controlled via xml rpc.
Connecting to the xml rpc interface with python
Connecting to your Spix enabled app from python is straight forward using the xmlrpc package:
import xmlrpc.client
s = xmlrpc.client.ServerProxy('http://localhost:9000')
print("Available Methods:")
print(s.system.listMethods())
You could use this interface to issue ui commands directly to Spix, but here we want to use PyAutoGUI instead.
Clicking buttons with PyAutoGUI
Before we can tell PyAutoGUI to click a button, we first need to know where on the screen it is located. For this, we can ask Spix using
bounds = s.getBoundingBox("mainWindow/Button_1")
bounds is a list with the screen coordinates [top left x, top left y, width, height] of the item Button_1. Items are identified through their path of objectNames, so in this case, it is Button_1 in window mainWindow. You will find those objectName properties in the qml file.
From this, you can calculate the center of the button as
center_x = bounds[0] + bounds[2] / 2
center_y = bounds[1] + bounds[3] / 2
and use PyAutoGUI to click it:
pyautogui.moveTo(center_x, center_y, duration=1)
pyautogui.click()
You can find a fully working example of this script here