Earlier this year, those using Shopify may have been affected by a technical issue when evaluating new vs returning customers, where an additional (and confusing) ‘N/A’ category was included. Our team investigated the matter and recounted how this was examined, highlighting the relevance of the issue.

Classifying customers as ‘New’ or ‘Returning’ in Shopify is integral to merchants and advertisers to allow for accurate reporting on new customer acquisition, and therefore key performance indicators (KPIs) such as New Customer Acquisition Cost (CaC), ‘New’ vs ‘Returning’ customer split, and Returning Customer Rate (RCR).

Given the importance of these metrics, accuracy of segmentation is key. When inconsistencies arise, this can create an environment of uncertainty and lack of trust in data being used to inform topline business decisions and investment into paid media.

What is the Shopify N/A issue?

When segmenting data in Shopify’s Analytics reports, it is expected to see two discrete categories: ‘First-Time’ and ‘Returning’, as seen below.

However, it was discovered that in certain stores, a third category appeared: ‘N/A’. Although this represented a very small percentage of total orders and customers, some showed greater proportions appearing under this category. This was a concern, as Vervaunt use Supermetrics, a technology that collates data for automated reporting. When this was checked, ‘N/A’ values were incorrectly included as ‘First-Time’ customers in the export.

As a result, Vervaunt needed to investigate whether customers pulling in as ‘N/A’ were truly ‘First-Time’ customers.

How was this investigated?

In the same reporting section, we filtered orders by ‘Customer Type’, including orders only classified as ‘N/A’. ‘Order ID’ and ‘Customer ID’ were pulled in as columns, allowing purchase history of specific orders to be reviewed, to ensure this was the first purchase of a given customer. After spot checking many order IDs from ‘N/A’ customers for a particular client, this proved that all were indeed ‘First-Time’ customers.

However, this N/A category was noticed on another client, which could not be resolved in the same way. When pulling in Order ID and Customer ID as columns, the Customer ID was blank. Further investigation showed all the purchases being attributed as ‘N/A’ were from in-store purchases captured through the PoS. Therefore, the only way to resolve this is by acquiring email addresses or phone numbers of customers purchasing in store, which is not always possible. Incentives to improve this data quality would be offering digital receipts, discounts on second purchases and more.

This therefore provided reassurance to merchants who saw this specific issue, that new customer volume, CaC, and RCR were being calculated accurately. However, it is advised all clients should conduct similar checks to ensure this is also true for them - it may not be.

Why is this relevant?

Accuracy of data directly impacts how precisely KPIs can be monitored and whether the correct decisions are made off the back of it. Therefore, the Shopify N/A issue represented a threat to the accuracy of data being used in business decisions. By investigating and addressing, our agency ensured that reporting remained reliable, allowing the team to continue to use this data as a source of truth.

Final thoughts

While the N/A issue was a niche aspect of analytics in Shopify, it highlighted the importance of regular evaluations and data health checks to avoid making decisions based on inaccurate information. Additionally, it illustrated no system is immune to errors, and by prioritising data accuracy, brands can make informed decisions that drive growth and success. We found that conducting the following health checks has helped to identify and resolve these inconsistencies:

1. Check data in Shopify against your API reporting tool’s export (such as Supermetrics) regularly, as to not assume it is always correct.
2. Data deduping as sometimes customers may purchase for the first time on a UK store or domain, then return a second time in the US store. This would count a customer as ‘First-Time’ twice, and therefore deduping is required to ensure accuracy here. We then would utilise deduped data in our reporting rather than Shopify’s classification (which has proven problematic), to further improve the accuracy of new customer acquisition data.
3. In-store purchases: This is difficult to resolve without acquiring first party data of people purchasing in store, via digital receipts or other means (10% off next purchase). This will help improve data gaps, but will be difficult to resolve fully.

This proactive approach ensures that business strategies are guided by reliable data, ultimately supporting better outcomes for merchants and advertisers.

If you would like support in building a paid media strategy, please feel free to get in touch.