Integrating AI as a path to product optimization, not the product itself.
Three stages of AI, one validated, one partially invalidated, one pivot.
Role
Sole product designer; partnered with two PMs (prompts, templates), marketing and engineering. I owned the design surface end-to-end and drove the Stage 3 pivot from research through ship.
Problem
Discovery for standalone AI buttons came in at 10–13%, less than half what we'd expect for very prominent UI at those entry points.
The deeper structural problem: native Vidalytics features get discovered at half the rate of industry-standard ones (12.34% vs 24.81%) while adoption-of-discovered is statistically identical. Discovery is the gate, not value. AI was tried as one lever against that.
Solution
Three stages, each scoped from the previous stage's data:
• Stage 1 - AI inside an existing feature. One-click AI captions embedded in the captions flow.
• Stage 2 - AI as standalone surfaces. Dedicated AI Script Analyser + Vid Stats AI chat.
• Stage 3 - AI integrated into dashboards users already visit. A pivot I drove from cross-feature funnel research between Stages 2 and 3.
Outcome
Stage 1 validated the friction hypothesis (+48% relative lift). Stage 2 partially invalidated the standalone-differentiation hypothesis - the AI worked for users who reached it, but discovery hit a 10–13% ceiling. Self-initiated research between stages found that analytics engagement didn't drive change-making, which reframed Stage 3 around the data-to-action gap.
The structural problem AI was running against: discovery is the bottleneck.
Native features are unique to Vidalytics; non-native are industry-standard tools available in competing platforms. Adoption = share of users who used a feature consistently (the amount of uses per time period is defined per each feature individually) among those who discovered it. Native features (Smart Vids, Experiments, Conversions, Training Centre, Segmentations): features that don't exist in other tools average 12.34% get discovered and 27.26% relative adoption. Industry-standard features in the same product average 24.81% discovery and 28.72% adoption. Adoption is statistically equivalent (the ~1.5% is small enough that we're treating adoption as equivalent). Discovery is roughly half.
The strategy and the testing-ground approach
Leadership picked AI for two reasons: competitive parity (direct competitors were rolling out AI features) and business impact (AI was a plausible lever on feature engagement, value-per-action, and retention). The target landed as: move feature engagement and value-per-action, with AI as one of several levers.
Some features like AI captions are no longer differentiation in our segment. By the time we shipped, multiple competitors had rolled out AI captions.
We had hopes for each stage - these were real attempts at value, not hypothesis tests dressed up as features. But AI is still relatively unexplored for product integration in our kind of work, so we prioritized shipping cheap and learning between stages over heavy investment in any one approach. None of the stages were planned upfront, each was scoped from the prior stage’s data.
Vidalytics is a profitable B2B platform with around 50 employees, roughly 2200 monthly paying customers. Core audience: VSL marketers - businesses of different scale, ecom brands, and agencies whose livelihoods depend on video performance.
Stage 1: AI as automation inside existing features
Hypothesis at stage 1 sounded like this: pre-AI captions had low adoption because the setup effort was high. Remove the friction with one-click AI, and more users will use them.
What shipped. AI generation embedded directly inside the existing captions as a one-click solution. Open captions, get the AI option. AI was the way to use the feature, not a feature in itself.
AI captions generation button added to captions feature.
AI captions reached 6.77% adoption vs 4.56% pre-AI which is a 48% relative lift, measured at 2 months separated by two weeks after launch compared to 2 months separated by 2 weeks before launch (users who enabled the feature on at least one video).
What this validated and what it didn't:
Frictionless AI is a real lever. Less setup effort drives more usage when the underlying feature is already wanted.
Frictionless AI works on features users already want. It doesn't work on features users don't see themselves in. 4.56% to 6.77% is a real lift, but it's below our internal benchmark for embedded automation (roughly 15%) - which we read as a ceiling on what "easier" alone can do.
Stage 2: AI as standalone differentiation
Our direct VSL/video marketing competitors have mostly invested in generative video AI and AI captions/transcription as standalone selling points. Stage 3’s integrated-insights direction isn’t where they’re pushing - Vidalytics is a performance/analytics product, not a creation product, and the AI work follows that positioning.
Hypothesis sounded like this: standalone AI features could differentiate Vidalytics by delivering value users couldn’t get elsewhere. Stage 1 was one-click automation inside existing flows, Stage 2 was a capability of its own - dedicated entry points, its own surface. Stage 2 run after Stage 1 but not informed by it, both parts of the same OKR.
AI Script Analyser flow.
Vid stats AI assistant. Floating button on vid stats. Chat with pre-made quick actions where users could ask questions about their video performance.
Vid Stats AI feature flow.
Both placed contextually. Buttons were big, bright, prominent - neither was hidden. That last part matters; it’s what made the data later surprising.
Numbers (in-environment funnels, two months in). Both measured against 100% of users who reached the environment the button lives in:
What funnel analysis uncovered for two standalone AI surfaces with same model under the hood: discovery within ~2.5% of each other (10.56% script analyser vs 13.01% Stats AI Analyser). But once users clicked: 21.05% adoption for the analyser, 53.13% for the chat - a 2.5x gap. Users found both and adopted one way more than another. However, according to our internal benchmark 21.05% adoption is ok, 53.13% is amazing.
Our original hypothesis on placement was that vid settings was the right environment for the analyzer because settings is where everything related to the video itself lives, and script falls under that. Stats was kept for stats. But users seem to expect settings to hold settings, and analysis to live where stats live (even analysis of the video itself).
Three plausible reads on the analyser gap:
Settings is the wrong room for analysis (most testable, gets priority next iteration)
The feature was too Vidalytics-native - users opened it without grasping what it offered.
"Script Analyser" didn't communicate the value clearly enough
All three likely contributed. Placement is the cheapest hypothesis to test, so that's where we'd start.The chat’s 53.13% adoption confirms the AI itself was doing real work - same model across both surfaces. The gap is contextual fit.
The standalone-differentiation hypothesis was substantially invalidated, but for a non-obvious reason. The features were doing real work for the people who reached them but it failed because standalone surfaces with prominent dedicated entry points couldn’t reach enough users to make the differentiation real.
Between stages: the research that opened Stage 3
Coming out of Stage 2, the team agreed the standalone direction had ceiling problems. Where opinions split was on what to do next: more entry points to the existing surfaces, or a more drastic re-think.
Two pieces of research between Stage 2 and Stage 3 reframed the next move. I ran both on my own initiative, and they came together in the Stage 3 pitch.
Finding 1: analytics tools inform, but don't drive action
Out of standard performance analysis on launched analytics features (assigned work, but I run it looking for cross-feature patterns), one stat kept surfacing: in sessions where users engaged with any analytics feature, 28.56% included a video change. In sessions without analytics engagement, 35.96% did.
The cleanest read: checking stats and making changes are different use cases, not one connected flow. Users open analytics, look at numbers, and leave. Then they make changes in a different session driven by something else. Stats are purely informative rather than action-informing. The actionable framing for Stage 3: stats should inform changes, not exist separately from them. Closing the gap means making analytics tell users what to do to increase their videos performance, not just show them what happened.
Alternative reads I considered:
Selection effect (analytics-checkers and fixers may be different cohorts)
Measurement window (action may take longer than the session captured)
I don’t think these explain the gap fully because the magnitude is consistent across cohorts but I still went to Mixpanel to check them out. I used filtering by profiling answer: who user is to check out the first alternative read and it checked out with no sufficient difference being noticed between cohorts. To check out second alternative read i just used bigger conversion window in funnel analysis and the difference wasn't sufficient either.
I raised the finding in a weekly performance review. It sat in the team's open-problem list for a while before Stage 3 picked it up. The question wasn't whether the gap was real, it was what to do about it.
Finding 2: AI discoverability research
Some time into Stage 2, I ran Mixpanel research on AI feature performance. I built funnels comparing AI features to non-AI features in the same environments and similar complexity. The pattern showed what most non-AI features looked like. Non-In Vid Stats, AI had lower discovery but good adoption numbers. In Vid Settings, AI had mid-range discovery and the lowest adoption. The Vid Stats pattern is the one Stage 3 was built around; the Vid Settings result reinforces that the analyser had a deeper problem than discovery alone.
Cohort analysis on the group that did adopt AI features showed a recurring pattern: led in by curiosity, stayed because they saw value.
If discoverability was the dominant blocker for AI specifically and not value or quality, then more entry points to the same standalone pattern not presenting the value clearly enough probably wouldn’t change the situation as much as we would want to.
The banner blindness surprise (and the alternative reads I worked through)
I worked through three alternative reads:
Placement (not isolable without A/B test we couldn't run).
Function clarity: "Script Analyser" not communicating what it did (label test deferred to post-launch).
AI novelty effect (cohort comparison planned at 30/60/90 days).
None isolated cleanly without label A/B tests we couldn't run at production scope. Stage 3's design hedges against all three at once, so clean attribution wasn't needed to move forward.
My read on the dominant cause: the entry points named the wrong thing. They named a category (“AI”) at the gate to value. The most plausible mechanism is that users had absorbed a pattern across other products - AI buttons as vague-capability markers and our buttons inherited the association even though our AI did real work. The button promised a category many users had learned to skip; the value behind it stayed invisible until they clicked. Stage 3’s design hedges against all three reads at once, so a clean attribution wasn’t needed to move forward.
Stage 3: AI as integrated enhancement
I brought both findings to the weekly product department meeting. Together they pointed at a different framing: instead of standalone capability, AI should fill the data-to-action gap directly, inside the interfaces where users already look at their data. The diagnosis landed. Disagreement showed up in execution.
Stage 3: pivot story and the pushback
My first attempt wasn’t a dedicated section. I wanted to embed AI directly inside the existing metric cards in vid stats - on hover, each card would surface an “improve this metric” CTA. Maximum integration: AI lives inside the unit users were already active.
First proposed solution
The senior PM pushed back, and the argument was substantive. Variance in our data: content, traffic, audience profile, marketing channel is so big that we can’t tie a specific setting change to a specific metric outcome with confidence high enough. A hover CTA promising “improve this metric” would set up users to expect a deterministic outcome the system couldn’t reliably deliver. If the metric didn’t move after the user accepted the suggestion, we’d have damaged trust at the most engaged surface in the product. The variance argument was right, integrity of the user relationship matters more than tighter integration.
I redesigned around it. The shipped version is a dedicated AI insights section between metric modules in the existing interface but separated from individual metric cards. Cards surface an observation and a suggested action the user reviews and decides on. The AI doesn’t claim more than it can deliver.
Vid stats before and after implementing proposed AI insights solution.
Other alternatives we ruled out: a more prominent chat than Stage 2’s (same blank-page problem); more entry points to the existing standalone surfaces (the discoverability research said the ceiling was about category-named gates, not entry-point count); and contextual notifications (off-brand for an analytics product where users open the dashboard to look at their data - cards meet users in the rhythm they’re already in, notifications interrupt it).
Cards don't ask users to commit before showing value
Stage 2's chat depended on the user knowing what to ask. Cards don't. The user doesn't have to know what to ask, click anything, or commit to "AI" before reading what's there. The value is on the surface.
This was structurally important given Finding 1: when users don't know what to ask in the first place, a chat surface is the wrong primitive. Cards meet users in the rhythm they already have with the dashboard. Notifications would have interrupted that rhythm which is off-brand for an analytics product where users come to look at their data.
We label cards as "AI Insights" (observations) and "AI Experiment Suggestion" (proposed tests), with a gradient that signals AI in our visual language. We don't hide what's algorithmic, users should know what's AI so they can calibrate trust. The difference from Stage 2 is where the labeling sits relative to the value: in Stage 2, users had to click an AI-labeled button before seeing what was inside. In Stage 3, each card already shows its specific outcome. The AI label is descriptive context on a card whose value is already visible, not an entry point.
Placement and hierarchy
Within vid stats, the AI insights section lives between metric modules. It had to hold two content types: insights (observations) and experiment suggestions (proposed actions) and I kept them in the same section because splitting them would have created two AI-flavored zones, undermining the “one integrated layer” intent. On top of that, our approach is to create a section connecting data to action as a one-stop solution.
The A/B test handoff
When a user accepts an AI experiment suggestion, they don’t get a one-click “test launched” outcome. They go through the regular experiment setup flow with every field pre-filled: variant, hypothesis, success metric, traffic split, target page. The user reviews, edits if needed, launches it themselves.
This is the user’s money. Vidalytics users run high-traffic VSLs where a wrong A/B test can cost real revenue. A one-click handoff would have been faster but would have stripped the user’s ability to confirm what they’re testing and why. That’s not a UX simplification, it’s a transfer of accountability that doesn’t belong with the product. If the AI’s suggestion is wrong and we let the user one-click it, we’ve cost them money via a flow we designed.
The review step builds confidence. Going through the pre-filled flow makes the user understand what’s being proposed and why. By the time they hit launch, they own the decision.
I argued for the review step on accountability grounds and the team agreed. I also proposed marking each pre-filled field with a small AI indicator. The PM declined for v1: the indicator interaction model would have required logic-heavy handling for each field type, and development cost outweighed the benefit on the launch-fast-and-cheap principle. If users get confused about field provenance, the indicator is the first thing I’d add.
How AI decides what to show
Two generation approaches under the hood, scoped to match how reliable each content type needs to be. The PMs owned prompt and template design in collaboration with engineering. I’ll cover the structure since it shapes the surface:
Experiment suggestions come from a predefined library of test types agreed with engineering — CTA changes, thumbnail tests, intro length variants, pricing display tests. The model picks from a known set rather than generating novel ideas. The library will broaden based on data over time.
Insights use templates with AI-filled variables. The structure is fixed (observation → metric → recommendation); the LLM fills the specifics for each user’s data. Specificity per user without unbounded variance.
The split is deliberate. Experiment suggestions need to be safe — a bad one can cost real money if it slips past review, so a bounded library is the safer pattern. Insights are lower-stakes and benefit from the LLM’s ability to reflect user-specific data.
The split was invisible to users but shaped card structure: experiment cards have a fixed schema (variant, hypothesis, success metric, traffic split, target page) so review is fast and trustable; insight cards have variable layouts so specificity reads as specificity rather than noise.
What we cut to ship cheap and fast
Stage 3 shipped on the OKR’s minimum-effort principle. The biggest cut — the hover-CTA-in-metric-cards approach — is covered above. The others:
Visual richness of cards. Original mocks had embedded micro-charts (sparklines, tiny conversion graphs). Engineering flagged rendering cost as too high for v1. Dropped in favor of cleaner text-and-number cards.
AI onboarding. I proposed contextual onboarding for the section. PM declined for v1 on cost grounds. Plan was to ship without it, watch the data, revisit.
AI-pre-filled field indicators. Covered above — proposed, declined for development complexity.
Coverage. Vid stats only, not every dashboard. Vid stats had the clearest AI use case and the highest engagement.
Insight count and types. Capped at one insight per video and a smaller library than the roadmap envisioned. Ship a working set, expand based on data rather than speculation.
Building a measurement framework for AI
Standard adoption metrics tell you whether AI is being noticed, not whether it's delivering value. I built a four-layer framework so the team could read AI's contribution to product outcomes, not AI's contribution to AI engagement.
Adoption: clicks on AI surfaces. The floor: necessary but not sufficient.
Engagement depth. Suggestions accepted, insights full view opened, follow-up actions taken. Tells you whether the cohort that’s paying attention is finding value. Adoption without engagement depth means users are finding the feature but not getting value from it — a quality problem at the surface, or the wrong value to surface. Adoption with engagement depth is the early product-market-fit signal.
Downstream behavior. Did an AI suggestion lead to an A/B test being created? The metric that closes the loop. It ties AI to a downstream product action rather than measuring AI in isolation. AI features that don’t change behavior outside their own surface aren’t doing the job, regardless of how well they engage on-surface. The layer most teams skip because it’s hardest to measure.
Layer 4 is the hardest and least clean. Current plan: matched-cohort comparison on AI-insights adopters vs non-adopters, with the explicit caveat that other concurrent launches will confound attribution.
The framework’s most useful property was layer three. It forced the team to think about AI as a means to a product outcome, not as a product outcome in itself.
What's next
Stage 3 went live on May 25th, 2026. Measurement is in flight via the four-layer framework above. Coming updates to this case:
Analyse outcomes for stage 3: Layer 1–4 reads at 30, 60, and 90 days.
What I'd do differently, once the cohort-level data is in.
What this initiative taught me about designing under AI uncertainty
If you want the in-progress numbers before they're published here, ask me via email.