Preparing for a UAT Period - The Importance of Prioritising UAT for a Replatforming Project?

Key considerations for UAT

It is really important to discuss the UAT phase with your chosen agency partner as part of the initial commercial and timeline discussions - it’s vital to ensure there is adequate time allocated within the timeline for a full UAT phase. It’s worth noting that I’m focusing primarily on the front-end side of things, as back-end UAT would generally happen throughout the project.

For a standard Shopify Plus build we would typically expect ~2 weeks allocation, depending on the complexities of the project, and with Magento we would generally expect a minimum of 2-3 weeks to allow for end to end testing, due to the nature of the platform. This period should typically include allocation for bug fixes.

The QA phase should generally be discussed with your development agency too - we would expect an agency to do a fairly comprehensive first pass test to validate that the requirement is as the scope determined (be this user acceptance criteria, user stories or any other documentation). For builds that are more complex, I would typically expect every section and feature of a build to be validated against ‘test cases’ by the agency, as a first round pass.

A development agency should always test based on your business data; using GA to extract the top devices, browsers and screen resolutions that your customers use and, where possible, using physical devices for testing. Where physical devices cannot be accessed, you can use simulators such as BrowserStack or Sizzy which tend to give you near-realistic results every time. Ideally you’d agree the devices and browsers with the agency pre signing the project off, but this is also a good topic to cover when vetting agencies.

We would also expect a code freeze before you enter a UAT phase - this will allow you to test a fully ready site, without moving parts.

Approach to UAT - what you should expect

What I will demonstrate here is a very simplified version of UAT handling. Some of you may be used to a more complex process with documentation from the enterprise world - however given our client and agency base and seeing what systems and resources there is access to, it makes sense to simplify where possible.

There is a difference between testing and full UAT. During the course of a project, I would encourage testing. However, during the build process there will (or there definitely should be) a point where there is a code freeze, to allow proper UAT to start. The reason behind this is that you need to be able to test as if it were a live site - with all aspects of the site fully configured, so you can simulate as close to a live user experience as possible. This is something that would likely need to be allowed for if you’re working with an agency that strictly follows agile principles, however we always try to insist on it.

Regarding timelines, as demonstrated above, the UAT phase is expected towards the end of the project - generally just a couple of weeks before a full site launch. There should be enough time to capture and log all bugs, followed by a period to be able to work through fixes, then review these before go-live.

How do I prepare for the UAT phase internally?

This may feel daunting but as soon as the design and functional aspects of a build have been pinned down, you will be able to produce a skeleton master testing document. You can manage this with many solutions but our clients have generally found working with Google Docs is extensive enough. Involve all stakeholders from internal to external teams, as this will impact almost everyone. It’s key to get the people with the most context involved - ring-fencing their time early on is very important.

Steps to take:

1. Build out test scripts for every area of the site, confirming key devices.

2. Organise the team responsible for supporting validation of each area and line item.

3. Firm up your approach to document findings from UAT, alongside your agency. When, where and how should bugs be raised?

4. Firm up priority status between the testing team so the team can allocate the severity of the ticket into an appropriate bucket.

5. Have your team prepared to be on-hand for validating fixes to bugs too.

6. Arrange to have physical devices where possible.

What do I need to validate and where do I start?

In short, everything that touches the platform (anything feeding in or out) needs to be tested in a replatforming project, therefore test scripts should cover all aspects of a website; remember to include all apps, integrations (ERP, PIM etc) plus any customisation work which needs to be signed off.

Please remember to tackle this from a front end and back end (admin set up) perspective. I would see this as a working document - extract areas that need special attention due to complexities into separate tabs - for example, SEO, Google Analytics, orders and payments etc. Assign owners to areas where possible, so that relevant responsibility is shared.

Just to reiterate, it is extremely important that you test against your key devices as per your unique site data - you may choose to do this based on traffic or revenue but we would suggest to combine these two metrics. You could wish to drill down as far as screen resolution, but we would suggest device x browser as a minimum.

Every test script will be unique to a client. The expected outcome needs to reflect what has been discussed as a part of the build, so it’s important that the detail is captured here to allow any external testers to know what to expect.

I would suggest creating a skeleton document during discovery, documenting all pages, features, apps etc., and actively updating this as functionality is confirmed throughout the process.

Examples of test scripts we use at Vervaunt can be sent directly to you upon request - please reach out to us.

Ensure that everyone is set up with access ahead of time, with the correct permissions to test their area. Log something like the below and ensure your agency has visibility - to ensure they are aware of which individual needs access to specific areas.

UAT across multiple stores

Please remember if you are a multi-store site, you will need to capture store nuances as a part of your test scripts and ensure these areas are reviewed on the store, once these are available. This generally happens as a separate workflow, after finalising your ‘main’ production store, so please take time out for this in your timeline to allow for this additional testing and review, per store.

Languages

Considerations need to be made for language sites too - this could impact timeline further depending on the team available to run the set up and UAT on this store. The main point to usually consider on a language site is text length - which can affect design and page templates. In addition, a localised language site may house additional apps, specific for the store which again, would need to have test scripts built out.

How do I document bugs?

When raising bugs, you would typically be doing this within a project board on a ‘bug form’ to document the issue - this is usually decided by the agency. A bug form would have fields to populate - an example below.

Please ensure there is one person owning the bug tracking system - to review and remove duplicate issues found or issues which conflict (and therefore need further discussion). This is an extremely important step, to avoid slowing down the process of back and forth discussion between bug fixes.

Bugs with a detailed explanation are great but you should incorporate a screenshot or video recording where possible to document the issue - this will give additional clarity to help pinpoint the issue in the first place. You will find free Google Chrome extensions to allow you to manage this or you can pay for services - internally, we use Droplr which we would highly recommend.

There are too many bugs logged and fixing these will impact my timeline - what options do I have?

Always raise all issues, no matter how big or how small you think they are; don’t be worried to document everything as ‘minor’ as a design tweak. Personally I think the best way to manage these is via labels and tags, but again, please speak with your agency to ensure they incorporate anything you may want into their existing process.

Outsourcing UAT- is this a meaningful approach?

Not all in-house teams are going to be able to resource a full replatforming project alone and we have seen some clients in the past using external agencies to support additional usability testing for example. This is great if you don’t have the means to do this internally as it will add a layer of testing beyond an agency doing some QA.

The only negative to this approach is that the teams will not be as close to the project and therefore can lack context and general build information; however it is still important to get a layer of general best practice nuances picked up, that someone close to the project could miss.

What does a successful UAT phase look like?

A successful UAT phase can be seen when all tickets have been cleared or at least organised into buckets for delivery, before launch. The key thing here is organisation - to allow visibility of the delivery plan for each ticket raised. Any bugs remaining pre launch should have an agreed delivery date so they do not remain as ongoing concerns.

Conclusion

Your UAT phase will be unique to your project and your company, but the fundamentals will always apply. If you have any questions about the details in this guide, you can connect with me on LinkedIn or get in touch at [email protected]