Guide
The 3 Kinds of A/B Tests for Marketing Sites—and Which is Best For You
Modern tools are expanding the kinds of A/B tests that marketers and designers can deploy self-sufficiently, allowing more rapid experimentation, and ultimately higher conversion.
Mar 27, 2024
Creating high-conversion web sites is no trivial task. With each visitor, you get a precious few seconds to:
✔️ Communicate a clear value proposition
✔️ Build trust with your brand
✔️ Differentiate your offering
✔️ Motivate the visitor to take action now
Most of us are not mind readers. To discover what resonates with the market, we need to test messaging on real leads and gather data on how it drives them to either bounce or engage and convert.
A/B testing is a proven tool for conversion optimization. Modern tools for building web sites are expanding the kinds of A/B tests that marketers and designers can deploy self-sufficiently, allowing more rapid experimentation, and ultimately higher conversion.
Let's look at a couple of established approaches to A/B testing before introducing the newest approach, Native Component Testing, which is gaining steam.
Split URL Testing
In a split URL test, a fraction of visitors that land on a particular path are redirected to a different page.
Example: When a visitor navigates to /pricing
50% stay on
/pricing50% are redirected to
/pricing-tiers
The alternate page may contain a few subtle messaging tweaks, or be a wildly different experience with different offers, value props, and design.
By keeping track of which version of the page each user is shown, along with whether or not that user converts, we can determine if one of the two alternatives performs significantly better.
How to Implement a Split URL Test
To implement a split URL test, you start by publishing the alternate page to your site. Then in your A/B testing tool, you configure the traffic split and redirect.
Here how you do it in Sumatra:
For the detailed steps, see Sumatra Redirect Docs.
Pros and Cons of Split URL Tests
Pros
✅ Familiar to implement. Publish the alternate page like any other.
✅ Allows big changes. No restriction on how different the page can be
✅ Universal. Works regardless of your web site tech stack.
Cons
❌ Slower page load. An anti-flicker snippet is typically needed, slowing page load.
❌ Obvious experiment. Visitor is aware from the URL that they are seeing an alternate page.
❌ Hard to test page sections. If the page has 3 sections with 2 variants each, you need to test 2 x 2 x 2 = 8 pages.
❌ Harder to maintain. To update an element common to all variants, each page must be updated separately.
Visual Override Testing
In a visual override test, a fraction of visitors see a modified version of the site, where certain visual and messaging elements have been changed from the original.
Example: For the home page headline
50% see "The internet is your canvas"
50% see "Build beautiful websites"
Possible overrides include swapping of text, images, and style as well as showing or hiding elements. A variant comprises possibly many overrides grouped together to create the desired experience.
Again, by keeping track of which variant each user is shown, along with whether or not that user converts, we can determine which variant is best.
How to Implement a Visual Override Test
To implement a visual override test, your A/B testing tool typically provides a visual (WYSIWYG) editor, which allows you to select and modify page elements in an embedded browser.
Here is how you do it in Sumatra:
For the detailed steps, see Sumatra Visual Editor Docs.
Pros and Cons of Visual Override Tests
Pros
✅ Fast to implement. Visual editor allows variants to be created quickly.
✅ Self-service. Marketer can ship end-to-end experiment without redeploying site.
✅ Universal. Works regardless of your web site tech stack.
Cons
❌ Hard to make big changes. Restricted to a few kinds of changes: text, images, style, etc.
❌ Slower page load. An anti-flicker snippet is needed for above-the-fold changes, slowing page load.\
❌ May not support breakpoints. Some changes may not work across screen sizes.
Native Component Testing
In a native component test, a fraction of visitors are presented with alternate versions of various sections and elements of the page, created natively in the tool used to build the web site.
Example: For the home page hero
50% see this:
And 50% see this:
The designer or marketer is free to change as much or as little as they'd like regarding the component's content, assets, styling, and layout. It is also possible to show or hide a component as part of the test.
A single component may be evaluated for its impact on conversion or a collection of components may be combined into a single test.
How to Implement a Native Component Test
To implement a native component test, you create variants of components in your web site builder or CMS. Then you link those variants to feature flags or experimental variations in your A/B testing tool.
Here is how you do it in Sumatra:
For the detailed steps, see Sumatra Framer Docs.
Pros and Cons of Component Tests
Pros
✅ Fast and familiar to implement. Build variants in the same tool used to build your site.
✅ Allows big changes. Make big layout and design changes or minor tweaks to messaging.
✅ Fast site performance. No anti-flicker snippet required.
✅ Preserves breakpoints and effects. Test variants without sacrificing responsiveness or aesthetics.
Cons
❌ Tech stack dependent Only works on sites built with particular tools, e.g. Framer.
Component testing is gaining traction as tools like headless CMS's allow non-engineers to make substantial site changes without visual overrides. For an example, see How to setup A/B tests in Storyblok, with Google Optimize.
Today, Sumatra integrates exclusively with Framer for component testing because of its native support for component variants and its mature design tools.
Together, Sumatra and Framer provide a uniquely powerful and intuitive solution for designers and marketers to drive conversion with A/B testing.
To try it yourself, check out https://sumatra.ai/framer.
Creating high-conversion web sites is no trivial task. With each visitor, you get a precious few seconds to:
✔️ Communicate a clear value proposition
✔️ Build trust with your brand
✔️ Differentiate your offering
✔️ Motivate the visitor to take action now
Most of us are not mind readers. To discover what resonates with the market, we need to test messaging on real leads and gather data on how it drives them to either bounce or engage and convert.
A/B testing is a proven tool for conversion optimization. Modern tools for building web sites are expanding the kinds of A/B tests that marketers and designers can deploy self-sufficiently, allowing more rapid experimentation, and ultimately higher conversion.
Let's look at a couple of established approaches to A/B testing before introducing the newest approach, Native Component Testing, which is gaining steam.
Split URL Testing
In a split URL test, a fraction of visitors that land on a particular path are redirected to a different page.
Example: When a visitor navigates to /pricing
50% stay on
/pricing50% are redirected to
/pricing-tiers
The alternate page may contain a few subtle messaging tweaks, or be a wildly different experience with different offers, value props, and design.
By keeping track of which version of the page each user is shown, along with whether or not that user converts, we can determine if one of the two alternatives performs significantly better.
How to Implement a Split URL Test
To implement a split URL test, you start by publishing the alternate page to your site. Then in your A/B testing tool, you configure the traffic split and redirect.
Here how you do it in Sumatra:
For the detailed steps, see Sumatra Redirect Docs.
Pros and Cons of Split URL Tests
Pros
✅ Familiar to implement. Publish the alternate page like any other.
✅ Allows big changes. No restriction on how different the page can be
✅ Universal. Works regardless of your web site tech stack.
Cons
❌ Slower page load. An anti-flicker snippet is typically needed, slowing page load.
❌ Obvious experiment. Visitor is aware from the URL that they are seeing an alternate page.
❌ Hard to test page sections. If the page has 3 sections with 2 variants each, you need to test 2 x 2 x 2 = 8 pages.
❌ Harder to maintain. To update an element common to all variants, each page must be updated separately.
Visual Override Testing
In a visual override test, a fraction of visitors see a modified version of the site, where certain visual and messaging elements have been changed from the original.
Example: For the home page headline
50% see "The internet is your canvas"
50% see "Build beautiful websites"
Possible overrides include swapping of text, images, and style as well as showing or hiding elements. A variant comprises possibly many overrides grouped together to create the desired experience.
Again, by keeping track of which variant each user is shown, along with whether or not that user converts, we can determine which variant is best.
How to Implement a Visual Override Test
To implement a visual override test, your A/B testing tool typically provides a visual (WYSIWYG) editor, which allows you to select and modify page elements in an embedded browser.
Here is how you do it in Sumatra:
For the detailed steps, see Sumatra Visual Editor Docs.
Pros and Cons of Visual Override Tests
Pros
✅ Fast to implement. Visual editor allows variants to be created quickly.
✅ Self-service. Marketer can ship end-to-end experiment without redeploying site.
✅ Universal. Works regardless of your web site tech stack.
Cons
❌ Hard to make big changes. Restricted to a few kinds of changes: text, images, style, etc.
❌ Slower page load. An anti-flicker snippet is needed for above-the-fold changes, slowing page load.\
❌ May not support breakpoints. Some changes may not work across screen sizes.
Native Component Testing
In a native component test, a fraction of visitors are presented with alternate versions of various sections and elements of the page, created natively in the tool used to build the web site.
Example: For the home page hero
50% see this:
And 50% see this:
The designer or marketer is free to change as much or as little as they'd like regarding the component's content, assets, styling, and layout. It is also possible to show or hide a component as part of the test.
A single component may be evaluated for its impact on conversion or a collection of components may be combined into a single test.
How to Implement a Native Component Test
To implement a native component test, you create variants of components in your web site builder or CMS. Then you link those variants to feature flags or experimental variations in your A/B testing tool.
Here is how you do it in Sumatra:
For the detailed steps, see Sumatra Framer Docs.
Pros and Cons of Component Tests
Pros
✅ Fast and familiar to implement. Build variants in the same tool used to build your site.
✅ Allows big changes. Make big layout and design changes or minor tweaks to messaging.
✅ Fast site performance. No anti-flicker snippet required.
✅ Preserves breakpoints and effects. Test variants without sacrificing responsiveness or aesthetics.
Cons
❌ Tech stack dependent Only works on sites built with particular tools, e.g. Framer.
Component testing is gaining traction as tools like headless CMS's allow non-engineers to make substantial site changes without visual overrides. For an example, see How to setup A/B tests in Storyblok, with Google Optimize.
Today, Sumatra integrates exclusively with Framer for component testing because of its native support for component variants and its mature design tools.
Together, Sumatra and Framer provide a uniquely powerful and intuitive solution for designers and marketers to drive conversion with A/B testing.
To try it yourself, check out https://sumatra.ai/framer.
Creating high-conversion web sites is no trivial task. With each visitor, you get a precious few seconds to:
✔️ Communicate a clear value proposition
✔️ Build trust with your brand
✔️ Differentiate your offering
✔️ Motivate the visitor to take action now
Most of us are not mind readers. To discover what resonates with the market, we need to test messaging on real leads and gather data on how it drives them to either bounce or engage and convert.
A/B testing is a proven tool for conversion optimization. Modern tools for building web sites are expanding the kinds of A/B tests that marketers and designers can deploy self-sufficiently, allowing more rapid experimentation, and ultimately higher conversion.
Let's look at a couple of established approaches to A/B testing before introducing the newest approach, Native Component Testing, which is gaining steam.
Split URL Testing
In a split URL test, a fraction of visitors that land on a particular path are redirected to a different page.
Example: When a visitor navigates to /pricing
50% stay on
/pricing50% are redirected to
/pricing-tiers
The alternate page may contain a few subtle messaging tweaks, or be a wildly different experience with different offers, value props, and design.
By keeping track of which version of the page each user is shown, along with whether or not that user converts, we can determine if one of the two alternatives performs significantly better.
How to Implement a Split URL Test
To implement a split URL test, you start by publishing the alternate page to your site. Then in your A/B testing tool, you configure the traffic split and redirect.
Here how you do it in Sumatra:
For the detailed steps, see Sumatra Redirect Docs.
Pros and Cons of Split URL Tests
Pros
✅ Familiar to implement. Publish the alternate page like any other.
✅ Allows big changes. No restriction on how different the page can be
✅ Universal. Works regardless of your web site tech stack.
Cons
❌ Slower page load. An anti-flicker snippet is typically needed, slowing page load.
❌ Obvious experiment. Visitor is aware from the URL that they are seeing an alternate page.
❌ Hard to test page sections. If the page has 3 sections with 2 variants each, you need to test 2 x 2 x 2 = 8 pages.
❌ Harder to maintain. To update an element common to all variants, each page must be updated separately.
Visual Override Testing
In a visual override test, a fraction of visitors see a modified version of the site, where certain visual and messaging elements have been changed from the original.
Example: For the home page headline
50% see "The internet is your canvas"
50% see "Build beautiful websites"
Possible overrides include swapping of text, images, and style as well as showing or hiding elements. A variant comprises possibly many overrides grouped together to create the desired experience.
Again, by keeping track of which variant each user is shown, along with whether or not that user converts, we can determine which variant is best.
How to Implement a Visual Override Test
To implement a visual override test, your A/B testing tool typically provides a visual (WYSIWYG) editor, which allows you to select and modify page elements in an embedded browser.
Here is how you do it in Sumatra:
For the detailed steps, see Sumatra Visual Editor Docs.
Pros and Cons of Visual Override Tests
Pros
✅ Fast to implement. Visual editor allows variants to be created quickly.
✅ Self-service. Marketer can ship end-to-end experiment without redeploying site.
✅ Universal. Works regardless of your web site tech stack.
Cons
❌ Hard to make big changes. Restricted to a few kinds of changes: text, images, style, etc.
❌ Slower page load. An anti-flicker snippet is needed for above-the-fold changes, slowing page load.\
❌ May not support breakpoints. Some changes may not work across screen sizes.
Native Component Testing
In a native component test, a fraction of visitors are presented with alternate versions of various sections and elements of the page, created natively in the tool used to build the web site.
Example: For the home page hero
50% see this:
And 50% see this:
The designer or marketer is free to change as much or as little as they'd like regarding the component's content, assets, styling, and layout. It is also possible to show or hide a component as part of the test.
A single component may be evaluated for its impact on conversion or a collection of components may be combined into a single test.
How to Implement a Native Component Test
To implement a native component test, you create variants of components in your web site builder or CMS. Then you link those variants to feature flags or experimental variations in your A/B testing tool.
Here is how you do it in Sumatra:
For the detailed steps, see Sumatra Framer Docs.
Pros and Cons of Component Tests
Pros
✅ Fast and familiar to implement. Build variants in the same tool used to build your site.
✅ Allows big changes. Make big layout and design changes or minor tweaks to messaging.
✅ Fast site performance. No anti-flicker snippet required.
✅ Preserves breakpoints and effects. Test variants without sacrificing responsiveness or aesthetics.
Cons
❌ Tech stack dependent Only works on sites built with particular tools, e.g. Framer.
Component testing is gaining traction as tools like headless CMS's allow non-engineers to make substantial site changes without visual overrides. For an example, see How to setup A/B tests in Storyblok, with Google Optimize.
Today, Sumatra integrates exclusively with Framer for component testing because of its native support for component variants and its mature design tools.
Together, Sumatra and Framer provide a uniquely powerful and intuitive solution for designers and marketers to drive conversion with A/B testing.
To try it yourself, check out https://sumatra.ai/framer.