AI-powered, Human-reviewed.
The 12-question scorecard for choosing an ecommerce CRO agency
Use these questions to compare agencies on fit, not reputation, and find the one built for your store.
| # | Ask this | A strong answer sounds like | Walk away if |
| 1 | What will you look at before proposing a single test? | Analytics, session recordings, post-purchase survey data, support tickets, and a technical audit | They already have test ideas for you |
| 2 | What is our baseline, and what lift is realistic at our traffic? | A range, with the caveat that low traffic means fewer, bigger tests | A percentage guarantee |
| 3 | Who writes the code that ships the test? | We do, in-house, in a preview theme, with QA before it goes live | Your dev team implements the winners |
| 4 | What happens to a losing test? | It is documented, and the learning feeds the next hypothesis | Vague answers, or no losing tests in their case studies |
| 5 | How do you decide when a test is finished? | A pre-set sample size and a significance threshold agreed before launch | We stop it when the winner is clear |
| 6 | Which metric do you optimize for? | Revenue per visitor, with conversion rate and AOV read together | Conversion rate, full stop |
| 7 | Can we see a redacted roadmap from another client? | Yes, here is one, with the reasoning behind the priority order | We cannot share client work |
| 8 | How many tests will actually launch per month? | A number tied to your traffic, not their capacity | As many as you need |
| 9 | Who owns the testing tool and the data? | You do, on your account, and it stays yours after we part ways | The agency’s account, agency’s license |
| 10 | What do you need from our team, and how often? | Named owner, a recurring call, access to analytics and staging | Nothing, we handle everything |
| 11 | What is explicitly not included? | A clear list: creative production, dev outside tests, tracking rebuilds | Everything is included |
| 12 | What does month 13 look like? | Fewer, larger structural tests, because the obvious wins are gone | The same plan as month one |
The four questions that actually break deals
Lead with question 3: who writes the code that ships the test?
Agencies that hand off winning variants to your developers are selling advice, not outcomes. The winner sits in a backlog for six weeks, ships slightly wrong, and the measured lift evaporates.
Agencies with design and development in the same room ship in days. This is why teams increasingly shortlist partners like craftberry that keep research, design, build, QA, and measurement under one roof rather than splitting them across vendors.
Separate statistics from vibes with question 5: how do you decide when a test is finished?
If an agency cannot tell you the sample size and significance threshold before a test launches, it will stop tests when the numbers look good.
That is how a store accumulates a folder of wins and a flat revenue line. A 90-95% significance band, agreed in advance, is the standard worth holding people to.
Most programs quietly go wrong at question 6: which metric do you optimize for?
A discount popup will lift conversion rate on day one and cost margin every day after. Most published agency evaluation frameworks still lead with conversion rate, which is exactly why it is worth asking about explicitly.
Ask which metric goes on the monthly report first. If conversion rate is the only one there, the incentive is misaligned from the start.
Question 12 tells you whether they have run a program for more than a year: what does month 13 look like?
The first six months of CRO are the easy part. Anyone can find broken mobile navigation and a missing shipping threshold.
What happens after that requires a partner who understands merchandising, offer structure, and the store’s technical architecture.
How five CRO agencies answer these questions
These five agencies were picked because they answer the 12 questions differently, not because they compete for the same job: an in-house Shopify Plus team, a boutique specialist, a documented process-led shop, a media-and-conversion hybrid, and a post-launch execution specialist.
Every one has a verifiable public track record, a case study, a stated methodology, or published numbers.
1. craftberry
craftberry is a certified Shopify Plus agency with more than 11 years in the Shopify ecosystem and an in-house team of 30-plus designers, developers, and CRO strategists.
craftberry’s clearest edge on this list, as an ecommerce conversion rate optimization agency, is who ships the code: the same team that writes the hypothesis writes the Liquid, so there is no handoff and no queue.
Its Shopify CRO practice runs the full loop, from research through theme implementation to post-test measurement, which is why brands stay past the low-hanging-fruit phase: craftberry restructures a PDP template or checkout flow, not just recolours a button.
Published work backs the process rather than the promise, including a 73% conversion rate increase and 20% sales lift for Westbourne, and a 21% conversion rate increase for Enhance from a single landing page test.
With 800-plus launched projects and internal benchmarks drawn from 100-plus Shopify audits, craftberry suits established Shopify and Shopify Plus brands that want one accountable partner for build and optimization, particularly where migrations, localization, B2B logic, or ERP integrations sit in the same roadmap.
2. Easycomm Innovations
EasyComm Innovations is a Shopify-focused digital solutions company helping ecommerce brands build, optimize, and scale high-performing online stores. The company provides Shopify development, Shopify Plus solutions, ecommerce development, UX optimization, and conversion rate optimization (CRO) services designed to improve customer experiences and turn more store visitors into customers. By combining design, development, analytics, and strategic optimization, EasyComm helps merchants address technical and user-experience challenges that can impact ecommerce performance.
With expertise across Shopify CRO, ecommerce CRO, Shopify store optimization, custom integrations, and digital marketing, EasyComm works with businesses looking to improve their online presence and achieve sustainable growth. The company provides solutions to ecommerce businesses across the USA, UK, UAE, Australia, India, and other global markets, helping brands identify opportunities, implement improvements, measure performance, and continuously optimize their stores for better conversions and revenue.
3. CRO Lab
CRO Lab is a specialist Australian agency that focuses narrowly on CRO: it publishes its pricing tiers and states plainly that it does not do PPC or social.
The engagement starts with onboarding, tool installation, and a behaviour-led audit, then moves to a prioritized roadmap where the merchant approves every test before it ships.
Suits smaller Australian merchants in the AUD 500k to 3M range who want pure CRO and a veto on the roadmap.
4. DVISIONMEDIA
DVISIONMEDIA is a Denmark-based growth agency that ties conversion work to paid media and retention rather than treating it as an isolated discipline.
It has been a certified Shopify Partner since 2021 and publishes operational counts including 937-plus split tests run. The company focuses on optimizing revenue per visitor, the metric that matters most once the same team controls both the ad spend and the storefront.
Suits brands already spending heavily on acquisition who want onsite efficiency before they raise budgets.
5. Beyond Borders
Beyond Borders is a Korean agency focused on what happens after launch: campaign planning, bundle and gift-with-purchase mechanics, Klaviyo automation, and always-on performance marketing.
Its published work with Beauty of Joseon describes CRM driving over 25% of DTC revenue.
It gives the clearest answer on this list to how much ongoing involvement it needs from your team, because the model is built around continuous execution rather than periodic recommendations.
Suits K-beauty and cross-border brands scaling into North America.
Key takeaway
You are not choosing a vendor. You are choosing who gets to decide what your store looks like for the next two years. The 12 questions above are designed to surface the difference between an agency that produces recommendations and one that produces revenue.
Score your shortlist honestly, weight execution, stopping discipline, the right metric, and the long game double, and pick the partner whose answers were specific enough to be checked later.
For most established Shopify and Shopify Plus brands, that comes down to whether one team can research, build, ship, and measure the same test, which is the bar craftberry was built around and the one worth holding every candidate to.
