CTR is usually the first personalization metric ecommerce teams check. It sits on the dashboard, changes after an update, and resembles engagement. Engagement resembles relevance, and relevance seems to be personalization's goal.
CTR can rise without helping the business. Strong CTR on products with little purchase intent shows clicks, not purchases. A team can improve CTR for months while revenue per session remains flat or declines. CTR still has value, but it measures something different from personalization's intended business outcome.
What Click-Through Rate Actually Measures
CTR shows whether a shopper found a product impression interesting enough to click. That is a real signal. Products receiving no clicks, however long they hold a ranking position, are plainly failing to resonate.
Clicks are not purchases. One shopper may click fifteen products and buy none, while another clicks two and buys one. Surfacing products that attract clicks without converting measures attention, not purchase-intent matching. The signals overlap, but they are not equivalent.
CTR also reflects presentation, not just recommendation quality. Images, price display, badges, and page position influence clicks. The same product will usually receive more clicks in position one than position five, regardless of relevance. CTR optimization can therefore reward position bias instead of match quality.
Why Revenue Per Session Is the Right Primary Metric
Revenue per session measures the outcome that matters: revenue generated by a shopper in a session. Averaged across sessions in a measurement window, it shows whether catalog ranking helps shoppers find and purchase what they came for.
It covers the funnel in one figure. A clicked but unpurchased product adds nothing to revenue per session. An engine can produce high CTR while revenue per session stays flat or falls. When the metrics separate, revenue per session decides whether the system is working.
Revenue per session exposes noise that CTR can miss. More viewed but unpurchased products may lengthen sessions and preserve CTR, yet revenue per session records the missing purchase lift. That makes weak business results harder to conceal behind engagement.
The Problem with 30-Day Conversion Windows
Some teams instead track conversion over a longer period, since a shopper clicking today may purchase three days later after comparing options. That makes sense in some high-consideration categories, but extended windows introduce other complications.
Attribution gets unclear. If a shopper clicks a personalized recommendation, sees a retargeting ad for that product two days later, and buys on a third visit, how much credit belongs to the recommendation? Long windows also hide real-time ranking response. Accurate current-session signals should produce revenue effects within the first few visits, not after thirty days.
In our early-access program, we use a 14-day revenue per session window beginning when personalization activates. A holdout group keeps the previous static category ranking. Fourteen days captures shoppers needing a couple of visits, while remaining short enough for reasonably clean attribution.
How to Set Up a Clean Measurement Baseline
Lift requires a baseline, though teams often skip this step. Comparing post-activation revenue per session with a previous period mixes personalization with seasonality, promotions, launches, and every other change in that period.
Use a concurrent holdout instead. At activation, randomly assign 20-25% of sessions, our usual early-validation recommendation, to the previous ranking logic. Compare revenue per session between personalized and holdout groups over the same period. Without a lab, this is about as clean a lift measurement as possible.
One detail matters: randomize at session level, not shopper level. If shoppers are assigned to groups and return several times, treatment must remain consistent across their sessions during the test. Otherwise behavior under one condition can affect the other, creating cross-contamination.
The Secondary Metrics Worth Watching
Revenue per session is primary. Secondary metrics explain the ranking engine's behavior and help diagnose slower-than-expected movement.
Catalog surface area, the number of unique products shown in at least one ranked result per day, indicates whether the engine explores the catalog or relies on a small group of high-confidence products. Healthy per-shopper ranking should broaden coverage beyond static category pages. If it does not, historical bestsellers may be driving the model too strongly.
Zero-result search rate is the share of queries returning no products. Per-shopper ranking does not directly change search, but an integrated discovery layer that presents relevant products in browse should gradually reduce searches for catalog products that category pages make difficult to find.
Split revenue per session between new and returning visitors. New visitors lack purchase history, creating a harder cold-start ranking problem, so their early-weeks lift will usually be smaller. Strong returning-visitor lift with modest new-visitor lift is healthy initially. Improvement in both groups over time indicates genuine learning.
A Note on Statistical Significance
Revenue per session varies more than CTR, since one large purchase can raise a small sample's result. Consequently, its tests need more sessions for statistical significance. As a rough guide, use at least 5,000 sessions per arm, ideally 10,000 when average order values vary widely.
That variance helps explain CTR's appeal: results arrive sooner and look cleaner. Still, run revenue per session long enough to produce a meaningful result, despite the pull of early CTR wins. We have seen teams call success after two weeks of CTR data, then discover flat or negative revenue per session over six weeks. The CTR result was not false, it answered another question.