How to Generate Ideas for CRO Tests

Brands tend to think of Conversion Rate Optimisation (CRO) as being about data, quantitative metrics and statistics. While all of those things are important to CRO, it doesn’t tell the whole story.

CRO, at its core, focuses on changing user behaviour to act in a more desirable way. We measure this behaviour with the above, but we use human psychology and creativity to create that change.

Experimentation (or A/B testing) allows us to measure the impact of the changes we make, but before you can run a test, you need an idea.

The below will explore how to come up with impactful ideas and how to give your experiments a higher chance of winning and delivering value to your brand and customers.

Start With Insights

At Experimentation Elite in December, Steven Pavlovich from Conversion.com shared an interesting insight from an internal study:

• 35% win rate for experiments based on gut feel

• 51% win rate for experiments backed by qualitative research

This is a clear indication that behavioural user research plays a key role when it comes to the creation of winning tests.

There are plenty of tools and methods you can use to conduct user research. From heatmaps and session recordings to user interviews and surveys, these methods give us glimpses into user behaviours and challenges. We partner with Mouseflow to access these insights for our clients.

We often try to corroborate any insight found by one method with another. See the example below.

Insight: From GA4, we noticed a high frequency of sessions who add an item to the cart, but do not complete the checkout. When we surveyed users about why they didn’t complete their purchase, we discovered that a lack of clarity about returns was a common problem.

The first step in creating impactful experiments is distilling insights like the one above into clear, actionable problem statements.

Define your Problem Statements

Using the insight example above, you could probably think of a few ideas that could potentially solve the problem. But if those ideas don’t work when you run your A/B test, then you’ll need to start iterating or generating more ideas to test.

To help you source more ideas, we recommend defining a clear problem statement. This problem statement can be shared with other stakeholders so they can start thinking of ideas. There are many different ways that you can define your problem statement. This guide from the Interaction Design Foundation goes into detail about the ‘Define’ step of the Design Thinking process.

An easy definition to start with is the ‘How might we? (HMW)’ method.

The “How Might We” (HMW) method is a framework that encourages teams to think collaboratively and creatively while maintaining a clear focus on the user.

HMW Method Example:

Let’s use the example insight above:

• Insight: Users abandon carts due to confusion about returns.

• HMW Question: How might we make the returns process more transparent prior to purchase?

You can use the HMW method in a cross-functional workshop setting, send as a quick survey on Slack, or even as part of a remote user survey. The problem statement should be universally understood so anyone can answer.

Turn Problem Statements Into Experiment Ideas

Once your problem statement is defined, you can begin to ideate. The key here is to maintain a clear connection between the insight, the problem, and the idea. It’s easy to get excited about a test idea and iterate any relevance out of it before you even run the test.

If you don’t have the resources to pull a cross-functional team together for a workshop or time to wait for survey responses, there are plenty of resources available that you can use as a source of inspiration.

• “Competitor” Analysis: You’re never just competing with your direct competitors, you’re competing with all the major platforms that users interact with every day. You can take inspiration from Instagram, Google, Booking, AirBnB, Amazon, Apple etc. These companies set the benchmark for what good UX and interaction design looks like.

• CRO / UX libraries: There are several directories you can access, some for free, that contain well documented ideas that may solve the problems you’ve identified. Baymard Institute, Guess the Test, GoodUI and Evidoo. Filter by the category / page you want to design for and use these as a source of inspiration.

• UX Design Galleries: Explore platforms like Dribbble, Behance, or Awwwards for more creative ideas.

Leveraging UX Principles

The UX Principles or Laws of UX are a collection of well researched ‘best practices’ that designers can consider when building user interfaces. Although ‘best practices’ are somewhat at odds with the CRO methodology, they can be useful as a source of inspiration.

We use principles from the following table to articulate what we’re trying to achieve with a design change or solution. For example, we may want to reduce the complexity of choice by framing available options in a different way.

If you’re going to use these resources make sure that you don’t just lift ideas from elsewhere and apply them. Use them to inspire your own ideas and check in with your problem statement to ensure any idea you plan on testing remains relevant to your users’ problem.

Using AI to generate ideas

With the right prompts, ChatGPT can also help you to generate ideas. The clear problem statement acts as a guard rail against AI hallucination. Iqbal Ali shared his AI playbook at Experimentation Elite. He shared an example where he used AI to generate ideas in response to a problem statement. He even uses it to write the Problem Statements themselves. Check out the playbook here.

Designing your experiment

Once you have generated and prioritised your ideas, you’ll need to design your experiment. You may need to collaborate with designers to visualise your idea, but in this case we mean design in terms of setting your experiment to measure impact.

Write a hypothesis, usually in an ‘If, Then, Because’ format and include the metrics you’ll measure to determine the impact of your idea. This is your final opportunity to check that the idea is clearly linked to your problem statement and insight.

You should have the following information to support each of your test ideas:

• Insight: Users abandon carts due to confusion about returns.

• HMW Question: How might we make the returns process more transparent prior to purchase?

• Experiment Idea: Place a short, reassuring statement about the returns policy directly under or next to the "Add to Cart" button.

• Hypothesis: If we place a short, reassuring statement about the returns policy directly under or next to the "Add to Cart" button, then we expect to reduce the cart abandonment rate by reducing hesitation caused by uncertainty about returns

You can now run your experiment with confidence that you’ve increased the chances of making an impact.

Key takeaways

User research is the key to impactful experimentation. However, transforming research insights into actionable experiments requires a structured approach.

1. Distill Clear Problem Statements
   Use multiple user research methods to uncover pain points and opportunities. Ensure you couple quantitative and qualitative research methods and corroborate any interesting findings from one method with another. Once you have a solid insight, turn it into a clear problem statement that can be shared with anyone and understood.
   
   

2. Ideate Relevant Solutions
   Leverage cross-functional workshops, competitor UX analysis, libraries like Baymard Institute, or tools like ChatGPT to brainstorm experiment ideas aligned with user needs. It is important to maintain a clear connection between the insight and the idea to give your experiment the best chance of creating the change in behaviour you’re looking for.
   
   

3. Create a Data-Driven Hypothesis
   Craft hypotheses in an “If, Then, Because” format, linking insights to expected outcomes. This step will help you to connect your insight with your experiment and ensure that you can measure the impact.

By aligning experiments closely with user insights, you can drive meaningful change and ensure your ideas solve real customer problems.