How to get a job at Kolibri Games – QA

Games as a Service (GaaS) are great for players. We’re constantly providing updates, expanding game features and delivering new content. Looking at Idle Miner Tycoon alone, we’ve shipped nearly 300 game updates and we’ve yet to run out of ideas! It’s exciting, having a game that’s constantly growing and improving. It’s also nerve-wracking! With nearly one million active players on any given day, each of whom logs into the game three or four times per day, if we ship an update with an exciting new feature, and it doesn’t work as expected, a lot of players are going to know! 

It’s crucial for a GaaS company to make sure that every update is stable and makes the game better for our players – and that’s where QA comes in. 

Quality Assurance, Game Testing or Bug Detecting?

All three. It’s a trope that game QA Testers play games all day long, and it’s not entirely inaccurate. QA Testers at Kolibri Games spend a lot of time within the game. By some measures, our Senior QA Testers know our games better than anyone else, which is what makes them an invaluable part of our product teams. They know our games so well, not just from logging hours in them, but by playing them in very specific and controlled ways. Our testers are our last line of defense before we send our game out to millions of players, confident that they will love all the new content we’ve added.

Methodology

Simulation games, which all of our titles are, are inherently complex, with features interacting in dozens of ways behind the scenes. Using IMT as an example, we have permanent boosts to mineshafts, elevators, warehouses, and even full continents. We have time-limited boosts. We have boosts that stack on other boosts and one-off rewards that expire immediately. Virtually every variable in our game can be modified by player actions. This makes our games challenging and rewarding, and it opens up a lot of opportunities for a player to do something unexpected – leading to unpredictable results. 

“There is an infinite amount of testing to do, and only a week in which to do it.” 

Because we can’t always test every variable in how our games can be played, we have to be smart and prioritize our testing time. Enter: the test map. Test maps, also called “test plans”, are prioritized lists of each feature, interaction, dependency, or known (previously reported) bug that’s to be tested in a certain sprint. These “testable items” can be incredibly granular, like the color of a UI element, or more macro, such as does a continent unlock when the intended conditions are met. Our test maps are extremely detailed and shared with the entire delivery team so that at any point, everyone can see what stage of testing we’re at, what’s passed testing and where there are roadblocks. 

Our QA testers are tightly integrated into our development teams. They’re part of our sprint kick-offs, along with PMs, game designers, devs and artists, so they know right from the start what new features or changes are coming and can start to anticipate and share possible breakpoints. Because our sprints are generally only a week long, our testers need to be able to prioritize their, theoretically infinite, to-do lists. A brand-new feature may need more extensive testing than the return of a previously-used Supercash offer – unless something has changed since the last time that Supercash offer was used. It’s a bit like Jenga, if your Jenga tower were constantly rearranging itself as you played.

What is a bug?

When a player’s game crashes, that’s obviously the result of a bug (or several). Not all bugs are so obvious. Game-breaking bugs are rare and in order to fully QA a game it’s valuable to think about bugs not as things that are broken, but as conditions in the game that don’t meet some stakeholders’ requirements or expectations. 

Game stakeholders include, but aren’t limited to, players, game designers, UX artists, monetisation managers, and project managers. Players expect a game to be fun, easy to understand, and rewarding. Something in the game that’s confusing, or decreases player enjoyment could be a bug. Game designers might expect, for example, a 2x boost to boost income 2x across every mine, restaurant or fire station. If it affects only one mine, that’s probably a bug. UX artists and monetisation managers may expect a change in the store layout to increase sales of a particular offer. If sales for that offer decrease, that may be a bug. 

The difference between looking for things that are “broken” versus looking for things that don’t work as expected can be subtle, but it’s a crucial difference for performing proper software testing.

During sprints, our QA testers both create bug tickets, which are sent on to the relevant members of their game team, and respond to player-generated bug tickets, trying to replicate the bugs and identify their sources. With all of these bug tickets, prioritization is key. Is a bug game-breaking, or just annoying? Does a certain bug affect one UI panel only, or does it ripple through several systems and have widespread effects?

As our testers work so closely with the other members of their delivery team, they need to have an overview and basic understanding of nearly every other role contributing to shipping a live game weekly. The more a tester learns about these other roles, their responsibilities, constraints, and ways their domains overlap, the better they are at finding the causes of bugs and even proactively predicting where a bug might pop up. The best testers start looking for problems well before the problems actually occur. 

As our sprints wind down and we’re nearing release, it’s time for a smoke test. Here we run through all the basic functionality of the game, as if we were a new player opening it up for the very first time. This allows us to see if there were any unexpected changes to our core loop or if we somehow made the Supermanager panel inaccessible. 

Our sprint weeks end with retros, which are meetings with our delivery teams and smaller QA-only meetings where we discuss how the sprint went, recognize where we did good work  and look for ways to improve the next sprint. As you can imagine, QA testers are always looking for bugs or chokepoints, at work or in the game, and motivated to eliminate or smooth them out going forward. A good QA department QAs itself, as well as the game!

Joining the team

We hire both junior and more seasoned QA testers. QA is often seen as a stepping stone into the gaming industry and that can be the case – but it’s not an immediate springboard into game design, programming, or art. We take QA seriously within Kolibri Games and offer a QA career path with increasing responsibility and opportunities to impact both live game and personal development. 

Junior testers can have nearly any educational or professional experience, though we get lots of applicants from game schools and university game study programs. More experienced testers frequently come from other game companies or have done UI/UX testing for business-to-consumer apps. What’s important to note here is that we primarily do manual testing. Some of our testing is automated, but candidates who have done a lot of automated testing may find themselves in less familiar waters. 

We’re also incredibly motivated to hire testers from diverse backgrounds. Our games are played by millions of players around the world, in 29 different languages. Our player-base is diverse in every way possible and for that reason we want our QA team to be as well. Experience has shown us that different testers have different approaches and strengths, and for that reason find different bugs. It’s an adage that you want your testers to look like your players and we give that an enthusiastic +1. 

So, you’ve applied. What’s next? You’ll start off with a recruiter screen, to determine where you are in your career, whether there appears to be a fit with the role, and to let you ask any questions you have about the role or our company. From there it’s a QA screening test. In this test, you’ll be asked to QA an aspect of Idle Miner Tycoon. This test is a great opportunity for you to show us how you prioritize your time, what testing approaches you choose, and how well you document your steps. We’ll be looking to see if you’ve covered the basics and then maybe gone a bit beyond and included some approaches we’ve not thought of. Don’t be afraid to flex!

After a successful test, we’ll hold two more Zoom calls, one with our QA Lead and one with a Senior QA Tester. Here we’re interested in getting to know you better and answering all your questions about us. As our QA Testers work closely with their delivery teams and interact with devs, designers, community managers, marketing managers, and artists daily, it’s important to find a good fit for everyone. This is also a great opportunity to learn more about the team you’d be joining, about their motivations and D&D class preferences.

Next up are two trial days. The trial task is similar to the first tech test, but will also more closely replicate the feeling of working in one of our delivery teams. During the trial we’ll schedule a series of Zoom calls with different team members, and we’ll expect you to proactively reach out to the team with questions and concerns. We don’t imagine that you know our games as well as we do, but you should be able to put yourself in the shoes of our average players! If you’re wondering whether the availability of a specific Supermanager is linked to progression somewhere else, ask us. We won’t make you guess. 

Following a successful trial, we’d love to welcome you to the team. Be forewarned, we get a lot of submissions for our testing roles. It’s an important role, one that’s helped us build games played by millions, and games we’ll keep updating for years to come. 

Summing it all up

If this article’s only just whetted your appetite for QA, have a listen to this episode of our podcast where we talk to Evgeniia, a working student QA tester here at Kolibri Games and this day-in-the-life blog post with Jonas, QA lead. 

If you want to be part of a player-centric, motivated and fast-moving mobile games company, we want to hear from you. Have a look at our featured open positions below and head over to our careers page to find out more.