A/B testing a landing page: how not to fool yourself

An A/B test of a landing page makes sense only once you have accumulated enough data to tell a real effect from coincidence. The test itself is a comparison of two variants of the same page with split traffic, and the yardstick is the order. On most Bulgarian websites the problem is exactly the volume: the test is stopped after three days, the variant with eight conversions against five wins, and this decision is carried over to all the traffic. But eight against five is a difference that can also occur between two completely identical variants. Three rules prevent self-deception: you change one variable at a time, you determine the required volume in advance and you let the test run for whole weeks, not individual days.

My name is Hristiyan Panov — a performance marketing consultant with over 5 years’ experience and over €2 000 000 advertising budget managed for over 90 companies in Google and Meta. Below is what to test first and which mistakes make the result useless.

How much traffic is needed?

It depends on your current conversion rate and on how big a difference you want to detect. The practical rule: the smaller the expected difference, the more visitors are needed — and small differences require volumes that most websites do not have.

That is why with less traffic you should test big changes. A different offer, a different structure, a different price. Changing the colour of a button requires a volume you will not collect.

Why whole weeks, not days?

Because behaviour differs by day of the week. A test that runs from Tuesday to Friday measures the working week, not your audience. Every test must cover whole weekly cycles — a minimum of one, preferably two.

The practical rule is for the test to start and end on the same day of the week and at the same hour. That way Saturday and Sunday each enter once in every variant, rather than once in only one of them.

How to tell that the cycle was broken:

• The difference between the variants flips when you add or remove the last day from the report.
• The daily conversions of the same variant differ several times over between a weekday and a day off.
• Part of the period coincides with payday, a holiday or a vacation.

The two variants run at the same time, not one after the other. Comparing this week with last week is not an A/B test — there you change both the time and the traffic, and you cannot separate the two.

What should you test first?

Ordered by the usual size of the effect:

• The offer. What exactly the person gets and in exchange for what. The biggest change, the least often tested.
• The headline and the first screen. Do they answer the question the person came with.
• The proof. Results, reviews, concrete numbers — and where they sit on the page.
• The form. The number of fields and what follows after submission. See what breaks the form.
• Layout and colours. Last, and only with enough traffic.

Which mistakes make the test meaningless?

• Several variables at once. If you change the headline, the image and the button, all you know is that something worked.
• Peeking and stopping early. Every result at the start looks convincing. Determine the volume in advance and do not stop before it.
• Different traffic sources in the two variants. Then you are measuring the source, not the page.
• Changes to the advertising during the test. New creatives mean a new audience, and the test becomes meaningless.
• A test during a promotion or a season. The result does not carry over to normal conditions.

What should you measure — clicks or orders?

Orders. A variant that raises clicks to the cart and lowers completed orders is a loss presented as a gain.

So look at the whole chain in one report: visitors, add to cart, checkout start, completed order. That way you see at which step the variant wins and at which it loses. Add revenue per visitor to the number of orders — a change that raises orders but lowers the average value often comes out at zero or at a minus.

If the volume is not enough to judge by orders, measure the closest event to them — checkout start, a form submitted — and accept that the conclusion is weaker.

Before the start, check three things: the two variants send the same event with identical parameters; the definition of a conversion does not change during the test; cancelled and returned orders are deducted in the same way in both variants.

When does an A/B test make no sense?

When the traffic is small, when the page has an obvious defect, and when the tracking has not been reconciled.

An obvious defect is fixed, not tested. This category includes a form that does not submit from a phone, a price or delivery terms that cannot be seen without searching, a button three screens down, a page that loads slowly on mobile data. There is nothing to compare here — the second variant is simply the working page.

Unreconciled tracking shows in a mismatch between the conversions in the advertising platform and the real orders in the store’s admin, in duplicated events, and in a conversion that is recorded on page load instead of on an order. A test on inaccurate data gives a confident wrong answer — and it looks exactly like the right one.

A test whose result will change nothing is also ruled out, because the decision has already been made — then the test serves as a justification, not as an answer.

And what if I do not have enough traffic?

Then work with observation instead of a test: session recordings, scroll maps, the questions you get on the phone. They do not prove, but they point the way — and at low volume this is the more honest tool.

What exactly is looked at:

• How far most visitors scroll and what remains below that line.
• At which form field people drop off and how many times a person comes back to correct it.
• Which questions are repeated on the phone and in messages — every repeated question is information missing from the page.
• What people type in the website’s internal search.
• Which objections are heard by the person who takes the orders.

If you do change the page anyway, change all of it and compare equally long periods before and after — with the caveat that the season is included there. This is not a test and is not presented as a test, but it is enough to remove the most obvious brakes.

FAQ — Frequently asked questions about A/B tests

How long does a typical test take?

Usually between two and four weeks. Shorter almost always means insufficient volume.

Does it harm SEO?

No, when it is done with the generally accepted mechanisms and the variants are not disguised, different content for search engines.

Can I test in the advertising platform instead of on the website?

Yes, for creatives and messages. For the page itself you need a tool that splits the website’s traffic.

How many variants at once?

Two. Every additional variant splits the traffic and lengthens the test.

What do I do with the losing variant?

Record what you have learned, not just who won. Lost tests build the understanding of the customer faster than winning ones.

Conclusion

An A/B test is not a way to prove to yourself that you are right. It is a way to find out that you were wrong — and that works only when the rules are set before you see the numbers.

If you are not sure whether your data even allows you to test, start with an independent review.

Request a quote for a digital audit →

Related articles: product page · leakage at checkout · enquiry form