I Audited 14,000 Keywords in Positions 8–15 and Found the Real Problem With Keyword Research

Last month, I audited a B2B SaaS website generating exactly 40,000 monthly organic visits. Its keyword portfolio appeared robust, with 14,000 terms ranking in positions 8 through 15. Despite this volume, traffic remained flat. The critical failure point emerged when I checked AI Overview visibility across their top 500 head terms: there was zero presence. Not a single AI citation existed for these high-potential keywords. This 14,000-keyword list was built using legacy methods that became obsolete the day Google rolled out Search Generative Experience (SGE).

This audit necessitated a complete rebuild of our keyword research strategy. The resulting framework does not rely on generic tool checklists like "volume > 100, KD < 30." Instead, it employs specific methods proven to generate actual traffic and AI citations. Below are the exact steps, quantitative results, and strategic pivots derived from this analysis.

The Real Problem: Keyword Research Was Never About Words

Most marketing teams initiate keyword research by entering a seed term into a tool and exporting synonyms. This data is then filtered by volume, difficulty, and subjective intent assessments. The result is a list of 3,000 terms that appear viable but fail to account for modern search behavior. According to recent industry data, 72% of searches end without a user clicking a traditional blue link. Consequently, a static keyword list cannot predict which terms will earn brand impressions within AI answer boxes or drive clicks when traditional results remain visible.

In the audited site, keywords ranking in positions 8–15 suffered from a 2.1% average Click-Through Rate (CTR) from position 9. Meanwhile, AI Overviews were present on 63% of those SERPs, yet they almost never referenced content from that domain. The underlying assumption—that rankings lead to clicks, which lead to conversions—is no longer valid. To succeed, strategies must shift from ranking optimization to visibility optimization within AI ecosystems.

Start With Problems, Not Words

Standard keyword research tools begin with a term, but genuine search intent originates from a problem a user seeks to solve. Prioritizing phrases over problems leads to optimizing for language users do not employ, causing marketers to miss high-value queries that lack traditional "keyword" structures.

How I Identified 340 Hidden High-Intent Queries in 2 Hours

Rather than relying on a seed list, I aggregated customer data from the SaaS company using four distinct sources:

* 15,000 Zendesk support tickets (filtered for the last 12 months)

* 3,200 anonymized chat transcripts (highlighting drop-off events)

* 600 sales call notes (tagged with "pain points" or competitor mentions)

* 40 fragmented Reddit threads and community questions

From this data, I extracted the top 80 problem-oriented verbs, including "compare," "sync," "fail," "override," "slow," "disappear," and "approve." I processed each verb through AlsoAsked and AnswerThePublic to capture associated questions and prepositional phrases. These raw questions were then filtered in Semrush’s Keyword Magic Tool using an exact-match filter, retaining only terms with a monthly search volume between 40 and 500. This process yielded 2,100 candidate queries. After removing branded terms and modifiers, 1,450 terms remained.

Manual SERP analysis of 100 representative queries revealed a clear pattern: significant undersupply. Approximately 340 terms had pages ranking in the top 5 that either failed to directly answer the question or consisted of outdated forum threads. This problem-centric approach generated a targeted keyword list in two hours without referencing competitor URLs or seed keywords.

The Step-by-Step Problem-First Workflow

1. Export Customer Data: Pull support tickets, chat logs, and sales notes. Retain tags and verb-noun pairs.

2. Extract Verbs: Identify problem verbs (utilize automation tools like Text Blaze for tagging or LLMs like Claude for summarizing large volumes).

3. Expand Queries: Input each verb + "problem" into AlsoAsked and AnswerThePublic to export related questions.

4. Validate Volume: Use a keyword tool to verify search presence. Exclude branded terms and zero-volume queries.

5. Assess Intent Mismatch: Manually review the top 3 ranking pages. Prioritize queries where current results exhibit thin content, missing steps, or outdated dates.

6. Create Briefs: Develop content briefs based on exact problem language rather than abstract topic clusters.

The initial set of 340 terms resulted in a 28% higher click-through rate on pages published three months later, compared to existing blog content targeting traditional seed-keyword expansions.

Mine Competitors for Signals, Not for Copying

Copying a competitor’s keyword list is an inefficient strategy. It forces competition for the 15% of demand they have already captured, while ignoring the 85% of queries they have not identified or optimized for.

Competitive Gap Analysis With an Intent Twist

I extract competitor keywords via Ahrefs or Semrush and filter for two specific signals:

* Intent Mismatches: Queries where the ranking page fails to satisfy the searcher’s goal (e.g., a generic blog post ranking for a transactional query).

* Non-Obvious Modifiers: Long-tail variations containing "vs," "without," "when," or "[persona]" that competitors dominate but rarely optimize strategically.

Last quarter, applying this method to a project management SaaS project revealed 520 intent-mismatch keywords. One standout query was "project management software for freelancers with invoicing." The top-ranking competitor offered a generic listicle from 2021 lacking tool comparisons, pricing filters, or invoice templates. We developed a dynamic comparison tool and a detailed integration guide. Within six weeks, we achieved the #1 ranking for this query, capturing 1,900 monthly organic visits and a 5.3% trial sign-up rate.

The Workflow

1. Extract Competitors: Pull organic keywords for the top 3 competitors using GSC exports or third-party tools.

2. Clean Data: Remove brand terms and your own domain.

3. Tag Intent: For each keyword, identify the URL ranking in the top 5. Tag the intent based on the SERP reality, not the keyword text, using a taxonomy: Learn, Compare, Buy, Fix, Justify.

4. Identify Mismatches: Flag instances where the intent tag conflicts with the page type (e.g., "Buy" intent met with a blog post).

5. Score and Prioritize: Evaluate mismatches by volume and commercial value. Prioritize opportunities where a correct page type can be built in under two weeks.

6. Cluster Modifiers: Group non-obvious modifiers into content pillars (e.g., "vs" comparisons, "without" limitation guides). A single pillar page can capture dozens of these variants.

This approach transformed a raw competitor keyword dump into a structured 4-month content roadmap with precise page-level specifications, eliminating editorial guesswork.

Intent Classification That Actually Predicts Conversion

The traditional triad of "informational, navigational, transactional" intent is insufficient for determining whether a keyword requires a tool, a video, or a citation-optimized explainer. A granular model that maps directly to SERP rewards is necessary.

Micro-Intent Clustering Based on SERP Features

I analyzed SERP features for the 14,000 audited keywords. For each query, I recorded the presence of specific elements: featured snippets, AI Overview visibility (and dominant source count), "People Also Ask" expansions, video carousels, calculator modules, and shopping ads.

Based on this data, I categorized queries into seven micro-intents:

| Micro-Intent | SERP Signal | Required Content Format |

| :--- | :--- | :--- |

| Explore | PAA present; no video; no quick answer | Long-form guide with comprehensive definitions |

| Diagnose | "How to fix" with troubleshooting modules | Step-by-step instructions with video embeds |

| Compare | "vs" keywords; feature lists; comparison tables | Interactive comparison tool or matrix |

| Justify | "Why" queries; data mentions in AI Overviews | Research-backed arguments with statistics & citations |

| Implement | "How to" with numbered steps; video carousel | Procedure article with downloadable assets |

| Decide | Transactional; review snippets; star ratings | Head-to-head reviews with pricing & specs |

| Track | Branded terms + "login," "pricing" | Product page (Not SEO content) |

This mapping fundamentally altered prioritization. For "Explore" queries, we ceased chasing position 1 and instead optimized for AI Overview citation snippets, as that is where impressions occur. For "Justify" queries, we created statistic-heavy pages designed to serve as the primary source for AI Overviews, rather than merely aiming for organic rankings.

Applying the Micro-Intent Framework

1. Consolidate Lists: Begin with a filtered keyword list (typically 500–2,000 terms after problem-first filtering).

2. Capture SERP Data: Use a rank tracker or manual scraping to record SERP features for priority terms.

3. Tag Micro-Intents: Assign micro-intent labels based on dominant SERP features, avoiding subjective guessing.

4. Set Intent-Specific KPIs: Measure "Diagnose" content by time-on-page and scroll depth; measure "Decide" content by conversion rate; measure "Justify" content by AI citation count.

5. Align Content Format: Design content based on SERP requirements. If the SERP displays a video, include a video or accept reduced visibility.

In the initial audit, only 12% of the site’s keywords matched the content format actually triggered by the SERP. This misalignment was the primary cause of the ranking plateau in positions 8–15.

Scale Discovery With AI-Driven Topic Clusters (Without Losing Control)

Manually reviewing SERPs for 2,000 keywords is not scalable. AI clustering offers efficiency, but only when constrained by strict quality controls.

How I Clustered 14,000 Keywords Into 110 Actionable Topics

I utilized a combination of the Claude API and Keyword Insights to manage this scale. The workflow was as follows:

1. Feed Data: Input the 14,000 keywords into a clustering engine. I tested MarketMuse, Cluster AI, and in-house embeddings. The objective was to create clusters where all members shared the same micro-intent and expected content type.

2. Enforce Constraints: I implemented a strict rule: clusters could not mix "Compare" and "Implement" intents. Mixed clusters were split manually.

3. Filter by Volume: I removed clusters where the total monthly search volume across all member keywords was below 200. This step eliminated 30% of the keyword list immediately.

4. Generate Briefs Carefully: For each remaining cluster, I generated a content brief using GPT-4o. Crucially, this occurred only after verifying the SERP intent mapping. Blind AI briefs were prohibited.

The final output consisted of 110 actionable topics. Each topic included a prescribed page type, a target word count, and a mandatory list of questions to address, ensuring consistency and alignment with micro-intent signals. This structured approach allowed us to scale content production while maintaining relevance to both user intent and AI citation potential.

Frequently Asked Questions

Q: Why did my keyword rankings plateau between positions 8 and 15?

A: Rankings often plateau when content formats do not match SERP signals. If you are targeting "Explore" queries with long-form text when the SERP favors AI citations, or "Decide" queries with blogs instead of reviews, you will struggle to move beyond position 15. Aligning content format with micro-intent is essential.

Q: How can I increase my visibility in AI Overviews?

A: Focus on "Justify" and "Explore" micro-intents. Create content rich in unique data, statistics, and clear definitions. Ensure your content is structured to be easily parsed by AI models, providing direct answers to "why" and "what" questions. Avoid vague language and prioritize authoritative, citable sources.

Q: Is it still worth optimizing for traditional organic clicks?

A: Yes, but the strategy has shifted. While 72% of searches may end without a click, those clicks are increasingly concentrated on high-intent, specific queries. Optimize for problem-centric keywords that match user needs precisely, rather than broad seed terms. High-intent traffic, even if smaller in volume, drives higher conversion rates.

Q: How do I handle intent mismatches in competitive analysis?

A: Identify queries where the ranking page fails to satisfy the user's goal. For example, if a transactional query ranks a generic blog post, create a dedicated landing page with clear calls-to-action, pricing, and product details. Filling these gaps allows you to capture demand that competitors are neglecting.