Bad news: Website analytics are never fully accurate

Website analytics tools like Google Analytics, Microsoft Clarity, and Shopify data are incredibly useful for gaining insights into user behavior. They can show us traffic patterns, user demographics, and how visitors interact with our sites.

Unfortunately, these tools are far from perfect. They often miss data, can be affected by cookies and ad blockers, and sometimes misinterpret user actions. While analytics are beneficial, they’re not always fully accurate, and treating them like a crystal ball can give you an inaccurate idea about the state of your website.

Comparing this year’s conversion stats to last year is sketchy

Usually, when we peek at our analytics, we want to know if we’re doing better than we did last year, or last quarter, or last month.

Can it give us a helpful snapshot into how we’re doing? Sure. But there are so many things that change in 365 days: consumer behavior, your offer, your competitor’s offer, and more.

So if you’re trying to track and measure — if you want to know “Will this tweak on my website yield an uplift in conversions?” — comparing one range of time to a previous range of time will never be fully accurate.

To get a much more accurate answer to your question: “Will this change on my website lead to more conversions?” — you want to compare now data to now data.

What about qualitative research?

Talking directly to customers and clients provides invaluable insights that numbers alone can’t offer. These conversations can reveal the why behind user behavior and highlight pain points that data might miss.

But qualitative research isn’t foolproof either. It can be biased based on who you talk to and how you interpret their feedback. Just like with analytics, it’s a piece of the puzzle, but not the whole picture.

Don’t assume: Experimentation allows you to actually test out your ideas

Website experimentation — also called conversion rate optimization, or CRO — lets you test your ideas in real-world scenarios rather than making assumptions.

Instead of comparing now to last month or last year, with the classic A/B test experiment, you can compare now to now.

Here’s the basic A/B testing process:

• Look at your data: Identify areas of your website that could be improved.
• Make a hypothesis: Based on your data, come up with a theory for what might improve user experience or conversions.
• Design a test: Create an A/B test to compare your current website (control) with the new version (variant).
• Run the test: See how the variant performs against the control.
• Analyze results: Determine if the changes had a positive impact.

Experimentation provides concrete evidence of what works and what doesn’t. Instead of assuming your data is correct, you’re testing it out — and getting far more clarity on what actually converts.

Do you have enough traffic to A/B test?

To get statistically significant results from A/B testing, you need a decent amount of traffic and conversions. Generally, you should have at least 1,000 conversions a month to try A/B testing and get statistically significant/accurate results. Without this volume, your results might not be reliable.

How to experiment on a smaller website

If your website doesn’t have a lot of traffic, you can still experiment effectively.

A few ideas:

• Focus on high-impact areas: Test changes on key pages like your homepage or primary landing pages where traffic is highest.
• Sequential testing: Instead of A/B testing, try changes one at a time and measure their impact over a set period.
• User feedback: Gather insights through surveys or feedback forms to identify potential improvements.
• Usability testing: Watch how a small group of users interact with your site to uncover issues and test changes. (You can use tools like usertesting.com)

3 practical takeaways for your website

1. Don’t rely solely on analytics: Use data as a guide, but remember it can be flawed.
2. Combine qualitative research with testing: Definitely talk to users, but validate their insights with real-world tests.
3. Experiment wisely: Whether you have a large or small website, don’t waste your time hyper-focusing on small changes like button colors. Instead, try to gather good qualitative feedback and make real improvements to your copy and structure.

‍

‍