Should you keep your UI Regression suite in a separate repository?

Well, for those who do not know what a UI Regression suite is,

UI Regression testing compares a screenshot of a given page to a baseline of what it should be, and outputs a diff image when applicable. When a difference is found, you have many options on how to proceed. You can set up your CI/CD pipeline so that the build fails, trigger a pager duty alert, or send a slack message to the team responsible. Ref: https://medium.com/disney-streaming/ui-regression-testing-71b2ef1bd9b4

This article is for tech teams, developers, SDETs, or anyone who fall in the software development lifecycle who are starting to create a UI regression test suite for your application.

So, the question is, should I create a subfolder in my existing repository or should I create a new one for this? There is not a single good answer. It depends on multiple factors. Here, I have tried to lay down some of my findings with my experience.

My vote is for separate repository and you would find it resonating in my detailing. 😁

The framework the regression suite runs on is generally an entirely different one than the application source code. It also uses separate tools for building and executing. Some of the popular frameworks are Cypress, Selenium, Playwright, etc.

It gets executed from a UI perspective as a user and automates user action workflows. And the team that works on these activities are generally different. It could be the same team, but they have a very little that can be shared. Also, the pipeline scripts for running the regression suite and sending out reports will be different from the application pipeline and it’s more time consuming which would run overnight.

However, this choice is always debatable and could change opinions as the application grows. For some applications, it might look good having all in one place and for some it might be maintainable to have them separately. The general trend nowadays is to have it separated. All of this depends on how the team works, number of team members, lifecycle followed, etc.

From my experience, following are the pros and cons of keeping regression test suites in a separate repository.

Pros:

• Reduces clutter when application and team grows / scales
• It brings in “separation of concern” principle allowing the team to scale without dependency
• The two repositories can use their own framework, dev lifecycle and pipelines. Clubbing two different frameworks and their pipelines together would create a large monolith pipeline architecture. Also, maintaining stories of each project and branch naming might create confusions.
• DevOPS can execute regression suites in parallel without any dependencies.
• There wouldn’t be changes in the regression suite for every change we make on the application. The suite just needs to be executed on the pipeline after a new deployment to the environment. The regression suites generally will take more time to execute and are triggered as nightly activities.
• The reviewers, maintainers and approvers of the two repositories will be generally different and this makes it easier.
• The commits, pull requests of developers and testers as the team grows gets mixed up and complicated.
• Smaller repositories make the work faster in most of the IDEs. Generally all IDEs keep scanning all files for changes saved.
• Allows to use a separate branching scheme, versioning, tagging, separate setup instructions, etc. for both code bases.
• UI Regression suites generally are triggered to run as a by-product of a deployment pipeline of the actual application into a test or critical environment. They run independently based on a given config with application url under test, service url to connect to, etc. which makes it detached from actual application source code.

Cons:

• Harder to set up CI/CD pipelines to trigger regression suites after a deployment as it needs cross calls against repository pipelines.
• If the same team develops the application as well as the regression suite, the team would need to switch between repositories
• Single repo gives the feel of working with one folder and multiple sub-folders than complicating to two repositories
• You can always split your code to two repositories later if you don’t see a reason to do it now. We could keep operations and processes simple for now. You could have some code that could be shared. (DRY, YAGNI and KISS principle)
• You can’t reuse any code. (Generally there is no code that can be reused between application source code and regression suites other than some test data)

Summary

Personally I would favour a repo for each application, as I like to version my tests separately like I do for my application. Keeping separate code base makes if cleaner and maintainable in my perspective. There are also less chances of people creating dependency between two unwanted pieces of code.

But I always advocate the YAGNI approach, which suggest to start with solving the problem you have now and refractor later when you encounter the situation. Only add complexity if you really need it. Still I prefer separate repositories 😁.