Choosing a Test Management System in 2019

Despite the enormous capabilities of modern spreadsheet editors, many development teams sooner or later face the question of implementing specialized tools (systems) to manage the testing process on their projects. Which one should you choose?

This question has been raised more than once in my practice. it's encouraged me to write this overview article, also in order to save some time for myself and all those who face this issue in the future. And even if there is no concrete answer in this article (sorry for the spoiler), your subconscious mind will tell you what to pick for your team or project at the right moment.

While searching the Web, I came across many similar articles (in one of which I borrowed the structure of the comparison table), but they all listed briefly main features from the websites of these systems and 1-2 screenshots of the user interface and often contain subtle or direct advertisement of one of these tools. Therefore I wanted to test them to get a firsthand knowledge of each system and share my impartial experience.

Disclaimer:
The article contains only my subjective opinion, some things might have changed (considering the time that passed from idea to publication - 9 months). I have no relation to the listed companies and products, all screenshots are presented for informational purposes. The position in the table or text is not the position in the ranking; you determine “best” and “worst” tool by yourself. For "test case", I will omit "test" and write only "case", also I prefer to write "testrun" as one word.

Below is the final comparison table, and I if you are interested in and have a bit time you can read more details about each tool after the table.

Paid Tools:
TestRail
QA Touch
Test Collab
qTest Manager
PractiTest
Zephyr Standalone
TestLodge
EasyQA

Open Source Tools:
TestLink
Squash TM
Kiwi TCMS

---

TestRail

Website: www.gurock.com/testrail
Developer: Gurock Software GmbH
Price: TestRail Cloud $30 for user per month (1-20 users)

UI screenshots:

Like: convenient and understandable UI; quick pasting of images; Markdown support; adding custom fields; setting the display order of case fields; the ability to assign each case (or testrun) to an individual user; quick copying and arrangement of cases; grouping several testruns into a test plan; quick integration with bug trackers (via tracker URL or using plugins); export of both cases and reports to XML and CSV formats; extensive API capabilities for integration with your internal systems (synchronizing information in cases, saving autotest results, etc.); visual reports; neat print forms; when creating a project, you can choose three types of structure for storing test cases (one of which allows you to divide cases into branches corresponding to development).

Dislike: there are no built-in requirements; there is no bug tracker; the price is quite high (especially for small companies).

QA Touch

Website: www.qatouch.com
Developer: DCKAP
Price: Professional $69 for 50 users per month

UI screenshots:

Like: extensive features both available and planned at an adequate price; nice UI; built-in bug tracker and requirements; integration with external bug trackers JIRA, GitHub, GitLab; testrun planning by release version; it is possible to create a re-run with cases of a certain status; voice typing and auto-completion of case and bug title; reusable case steps.
Unfortunately, I was not able to try and take a screenshot of some components of the product, because they are only available with subscription plan.

Dislike: the parts of a case (main fields, steps, attachments) are divided into tabs when editing; no text formatting; although a picture can be added to a separate step, but only to one and as a downloadable file; no case versioning; text areas of steps and expected results become shifted during editing (not on the same line), when resizing them manually; the form for selecting cases for testrun is slightly different when creating and editing it; the list of cases and bugs includes only their name and ID (additional custom fields would be useful for a quick overview of the list); the status mark during a run is also placed on a separate tab from the general information of the case; there are small mistakes in the displayed data (the total number of cases on the dashboard and the strange “Assigned to” field - I attribute this to the fact that I registered back in past year (2018), since then the vendor has significantly improved the product).

Test Collab

Website: testcollab.com
Developer: Test Collab
Price: Startups от $25 for user per month (over 50 users - $10)

UI screenshots:

Like: extensive customization of email notifications (with the ability to edit email templates); combining testruns into test plans; nice user interface with hints (which can be disabled); reusable steps; the ability to comment on a case while editing; tags; selecting the type of text formatting (Markdown, HTML/WYSIWYG, Plain Text); adding custom fields; customizable templates for selecting cases in testrun; built-in requirements; price (for small projects there is a free plan with 200 cases and 400 executed cases).

Dislike: you cannot edit system fields (in particular Priority); there is no way to save displayed fields in filtered lists (pages are reset to default when refreshed); there are minor flaws in the display of UI elements; there are no printed forms for either cases or reports (probably, they are rarely used now, but other tools offer this function); data on the general dashboard is updated with a little delay; export of cases only for one test suite (at the same time, nested suites are processed only by export to XML); inconvenient pasting of images into case steps (via a link); there is no bug tracker.

qTest Manager

Website: www.tricentis.com
Developer: Tricentis (acquired QASymphony in 2018)
Price: от $99 for user per month (price is approximate and depends on the number of users)

UI screenshots:

Like: customization of system fields and addition of custom fields to various objects (with the ability to preview each object and even change the color for the run status); informative history of case changes; built-in HTML editor in the text fields of a case; commenting on a specific case; you can subscribe to a separate case and receive email notifications about changes; interesting implementation of case auto-versioning (with major and minor components); case quick run function (without setting the status for each step); built-in requirements with case binding and target build tag, there is also built-in bug tracking; reusable cases; custom testrun configurations; rich integration with bug trackers and CI/CD services; API for even more flexible interaction and integration; you can save search queries for cases, requirements, etc.; various reports; flexible configuration on the administrator side (users, notifications, rights, groups, etc.); built-in demo project with all the data to familiarize yourself with the capabilities of the product; test plans.

Dislike: there are no printed forms for cases; export of cases only in Excel format (but it's readable for humans); slightly inconvenient viewing of pictures in the steps of a case during testrun; there is no information on the website about the price of the product (you need to make a separate request) and it is high relative to other tools in this overview.

PractiTest

Website: www.practitest.com
Developer: PractiTest
Price: ENTERPRISE $49 for user per month (minimum 3 users)

UI screenshots:

Like: the ability to comment on a test case; tags (keywords) for a test; quick jump to a test by its number; generation of testrun permutations of different testers, versions, etc.; using parameters in case steps (with the ability to change the value on each run); inclusion of steps (call to test) of one case into another (sometimes this is really necessary and there is no need to jump on links to other cases); two-way integration with Jira and YouTrack (by creating an API token); addition of custom fields; Kanban board for cases by their statuses; own bug tracker and requirements (but I didn’t take screenshots of them).

Dislike: complex UI/UX in general (in particular, the division of the case structure into tabs: description and steps on different tabs); scant, unpresentable printed version of both the cases themselves and the reports; although there is Markdown support, pictures are added to the steps only as attachments; you cannot change the status of the entire case if some step has failed (this is correct, but sometimes you need to "green" the entire test); you cannot save the sorting order in the cases list (sorting is alphabetical); price (overpriced, even considering the unique features); support chat widget on every page.
At the time of publication, the company’s website was not accessible from a Russian IP, but the web platform was accessible.

Zephyr Standalone

Website: www.getzephyr.com
Developer: SmartBear Software
Price: от $50 for user per month (price taken from the past, not currently listed on the website)

UI screenshots:

Like: attractive planning of testing stages (similar to a Gantt chart) with the addition and configuration of various testing activities; built-in requirements and bug tracker, which seamlessly integrates with JIRA (with the ability to search and view tickets (issues) directly from Zephyr); it is possible to add custom fields to objects (project, cases, requirements), change the name and color of execution statuses (the entire case and an individual step); you can add widgets for the project with various characteristics (diagrams), which will be updated after a specified time; export of cases to HTML and PDF format with selecting the required fields.

Dislike: unusual user interface, it takes some time to get used to the UX/UI; there is no direct link to a separate case (you can only generate a URL for it through a search query); the block with the help widget at the bottom cannot be removed; you cannot change system fields; pictures to the steps of a case can only be added as attachments to the entire case; focused only on JIRA; the price must be found out upon request (and only by phone; this information is not provided by email).

TestLodge

Website: www.testlodge.com
Developer: TestLodge Limited
Price: Basic $49 for 600 cases per month (data subscription)

UI screenshots:

Like: uncluttered user interface; creation and selection a configuration when adding a testrun; quick integration with popular bug trackers; creating a test plan (with a default template); coverage matrix (if requirements are written using built-in feature); subscription plan based on the amount of data (for some projects this is a pros); export of test suite to HTML (aka print form).

Dislike: there is no division of the case into separate steps; user fields cannot be moved above system fields; a picture can only be inserted as an attachment or a link to it; cases export only each set separately; a test suite can be specified for only one test plan; there is no bug tracker.

EasyQA

Website: geteasyqa.com
Developer: ThinkMobiles
Price: $10 for user per month

UI screenshots:

Like: a feature for monitoring and testing mobile builds (APK and IPA files), integration with GitHub or GitLab repositories; own bug tracker with the ability to add a plugin for integration with other systems (GitHub, GitLab, YouTrack, JIRA, RedMine, Pivotal Tracker, Trello); ease of use (sometimes this is a pros) of test objects (plans, testruns, test cases, reports); visual comparison of fields when exporting cases from a CSV file (but it requires attention, you can end up in an endless export loop).

Dislike: there is no way to customize objects in any way; there is no expected result for each step of a case; you cannot format text or attach pictures in cases; there are questions about the UI in some places (especially when displaying tables); I did not receive an invitation link for the second participant via email with personal domain; you cannot change rights for roles (+ I could not change the project role for an unverified user).
Judging by the activity on social media and their blog on the website, they have suspended development.

---

Free (open source) systems attract many people because there are no monthly fees. But they will certainly require certain costs. First, hosting rental (computing resources). Secondly, the labor costs of the system administrator to install, configure and maintain the system. But you may already have a server (or a small virtual machine), furthermore, the availability of production-ready Docker images simplifies deployment, so launching a free system from scratch may be easier than a registration in a paid system (of course I'm exaggerating, but there's a grain of joke in every joke). And on top of that, a trend of “total control” over corporate data. So, open source tools become increasingly attractive for adoption. I took three such systems for review: TestLink, Squash TM and Kiwi TCMS.

TestLink

Website: testlink.org
Development language: PHP

UI screenshots:

Like: built-in requirements and tracking them through cases; various text reports; ability to add custom fields; flexible configuration of user roles; integration with bug trackers (JIRA, YouTrack, GitLab, Bugzilla, etc.); a test lead can additionally set urgency for each case in a testrun, that will affect the order of cases; managing the list of platforms being tested; inventory list of hosts (a small, but nice, thing); HTML editor with the ability to paste pictures, links, tables, lists; in my opinion, it's the most popular open-source tool with many tutorials and articles about setup and use; having a PHP-developer in a team allows you to change the product to suit your needs (if the standard functions seem insufficient or they are not so convenient).

Dislike: the user interface requires a little getting used to; you will have to tinker with setting up some components for full operation (fine-tuning of configuration files, emails setup, integration with a bug tracker); browser spell checking function does not work in text windows of the HTML editor; you cannot edit system fields and the order in which they are displayed on the case form; manually changing the order of steps (you have to enter numbers instead of drag'n'dropping); after paid tools, assigning cases for certain build will seem inconvenient.

Squash TM

Website: www.squashtest.org
Development language: Java

UI screenshots:

Like: requirements management with linking to test cases; HTML editor; using of parameters in case steps and preconditions; the ability to create a test case with a Gherkin script (BDD); UI function for collapsing the contents of case steps; there is the mode for running cases with displaying the website being tested in a frame (in most cases this is no longer relevant, but here it is); re-using an existing case ("Call a test case"); adding custom fields to objects; color differentiation of product components; the color status indicator in the case library also seemed convenient; plugins for integration with JIRA and Mantis bug trackers; there is detailed documentation (though some sections are not in the right order) .

Dislike: in order to paste a picture as an attachment, you must first add extensions for such formats to the white list of system settings (by default, PNG and JPG are absent); you cannot edit system fields (category and priority); you cannot edit the existing seven roles with access permissions (however, 7 levels are often enough); I could not generate some reports (I found out from logs that there was a problem with class initialization); unusual terminology for test plans and testruns (here, they are called Campaign and Iteration); splitting the UI into tabs.

Kiwi TCMS

Website: kiwitcms.org
Development language: Python

UI screenshots:

Like: flexible configuration of user groups (permissions); integration with bug trackers like JIRA, Bugzilla, GitHub and GitLab; website performance; Markdown support with quick preview option; the ability to connect plugins (some, however, only in development plans) to collect autotest results; API for external tools; detailed product documentation; active development of new features and plans for the future.

Dislike: there is no expected result for each step of test case (scenario); some counters in the UI are not updated after an action is completed (a page needs to be refreshed); own spelling highlighting in the text editor; only one report - by created cases (but there are plans to add several more); there is no division into projects within the system.

---

Conclusion

This is only a small part of the systems available on the market. However, the review is already getting quite long. It does not include plugins for JIRA ( TM4J , Zephyr , Xray , TestFLO , etc.), because they are in a different league. Although JIRA is now used on many projects, it is still not a panacea. Therefore, this is a topic for a new longread. Test IT and Qase were also not mentioned, but they came out quite recently, when the main part of this article had been written already. Perhaps more systems were “born” during this time or there are existing ones which are worthy to review - write in the comments. People seem to love the series - let's write a sequel!

In the meantime, I encourage you to take part in the discussion “What test management system do you use on your project?” and state your choice in the poll on Habr.com page (in Russian).
(I understand that this is not always the “desired” system, then share your pain: which one you work with, and which one you would like to work with.)

Click on picture to open the article (in Russian) on Habr.com (with a poll)