When data breaks your conviction
How to challenge your strongest beliefs (and why)?
Have you ever been so sure about something only to have data prove you wrong?
Pictarine's Product team proposed adding an extra step to our funnel: one that I was 100% convinced would be unnecessary and even harmful to our funnel by slowing down the experience and deceiving users who expect an easy, fast and convenient experience.
Well… I was wrong! The lifetime value (LTV) actually doubled!
Listen to the podcast version generated by NotebookLM:
The context
Our customers use our experience to print their photos for same-day pickup in the US.
We decided to answer the question: does adding an extra step to our funnel will enhance the customer experience?
- Legacy funnel (version A): users would come, select a print size by clicking on the product image directly on the homepage (let's say 5x7), and a photo picker appears for them to select photos. And then voilà, the order is ready for pickup.
- Alternative funnel (version B): users clicks on a product image (let's say a 24x30 poster), land on the dedicated product page and then select their photos.
Left: extra step (product page). Right: photo picker triggered from the HP.
The results
We broke down the results by comparing desktop vs. mobile users and created two distinct funnels depending on the type of products:
- Prints.
- Creatives (such as canvas).
At each step, we measured user engagement and used confidence intervals to ensure statistical accuracy:
- Prints: the results were the most surprising here; the LTV remained the same, whatever the version used.
- Creatives: less surprising as we expected users to want to see the product details; the LTV doubled.
Challenging yourself on your own ideas
With experience, we develop strong convictions as well as a gut feeling. The mistake would be to hold onto them indefinitely as if they’re eternal truths.
By revisiting these beliefs and allowing them to be challenged, even those that have been proven right in the past, we create room for growth.
Don’t fall in love with your convictions, celebrate results instead
One key lesson I’ve learned is that holding onto these beliefs without challenging them is risky. Sometimes what produced top results in the past may not produce the same uplift and can be less effective today.
Ask yourself:
- When was the last time you tested one of your core assumptions?
- Has your market changed?
- Have user behaviors evolved?
If you’re relying on gut feelings without validating them through fresh data, you could be missing out on growth opportunities.
Encourage disagreement
One of the most valuable parts of this process wasn’t just the test itself; it was the conversations that came out of it. Initially, I insisted on running an A/B test to avoid damaging our funnel, but it sparked something more: open dialogue and debate on our own convictions and beliefs. We shared our own vision and understanding on the vertical of photo printing in the US: how we lecture the market, the audience and our current customers.
By encouraging disagreement, we were able to elevate the discussion and challenge each other’s ideas. In the end, the test was more than just a safeguard: it was an opportunity to challenge the entire flow and learn something new.
- Regularly test your strongest convictions: just because something worked in the past doesn’t mean it’s still the best approach. Choose one assumption and test it against fresh data.
- Segment your tests: break down your A/B test results by key segments such as desktop vs. mobile, product types, or high-margin vs. best-selling. This helps you uncover hidden patterns in user behavior.
- Foster a culture that encourages constructive disagreement: don’t be afraid to raise questions or encourage debates within your team. You make an impact when you do this. Healthy disagreement can lead to better, more informed decisions.
Final thoughts
Beyond A/B testing, one of the best ways to challenge your assumptions is to engage with peers who fundamentally disagree with you (it works well with peers outside your company or even outside your industry). These conversations can offer fresh perspectives and help you step back from your daily routine, making it easier to challenge long-held beliefs and develop more innovative solutions.