{
"title": "We Ran LLMs on 10k Pages. Here’s What Broke.",
"content": "# We Ran LLMs on 10k Pages. Here’s What Broke.\n\nLast Tuesday, I pushed a script to ingest our entire historical archive—about 10,000 support articles—into a local vector database. The goal was simple: build a retrieval-augmented generation (RAG) layer for our internal help desk. We wanted faster answers for support agents who were drowning in ticket volume.\n\nIt failed. spectacularly.\n\nThe LLM hallucinated product features that didn’t exist. It cited obsolete API endpoints from 2021. Worse, it merged two different troubleshooting guides into a coherent but factually wrong narrative. The \"large-scale model\" hype cycle had us believing that if we just threw enough tokens at the problem, accuracy would follow. It didn’t.\n\nWe spent three weeks fixing the pipeline. Not because the model was dumb, but because our data structure was naive. We learned that scaling up doesn’t mean scaling out. It means getting precise.\n\nIf you are planning to integrate large language models (LLMs) into your content or support workflows, stop treating them like magic boxes. Treat them like junior engineers who read too fast and forget everything by lunch. Here is how we fixed our implementation, what tools actually work, and where the bleeding stops.\n\n## The Chunking Trap\n\n### Problem: Context Window Noise\n\nOur initial approach was to chunk text by paragraph. Standard procedure, right? Wrong. Large models have massive context windows, but they still suffer from the \"lost in the middle\" phenomenon. When we fed a 5,000-word document split into arbitrary paragraphs, the model ignored the first third and the last third. It only paid attention to the middle.\n\nThis caused massive inconsistencies. A customer asking about \"account recovery\" at the top of the chunk got different advice than one asking at the bottom, depending on which sub-section the model focused on.\n\n### Solution: Semantic Chunking with Overlap\n\nWe switched to semantic chunking. Instead of splitting by whitespace, we used embeddings to identify natural breaks in topic flow. We kept chunks between 300-500 tokens. Crucially, we added a 10% overlap between chunks. This ensures that context carrying over from one section to the next isn’t severed abruptly.\n\nThe result? Accuracy jumped from 68% to 94% in our internal testing. The model stopped guessing and started retrieving.\n\nFor those building similar ingestion pipelines, understanding the nuance of citation sources is critical. The Citation Gap: Why Your Google Rankings Won’t Get You Into AI Search And 7 Steps To Fix It details the exact metadata structures needed to prevent these hallucinations.\n\n## Prompt Engineering Is Dead. Structure Is Alive.\n\n### Problem: The \"Helpful Assistant\" Fallacy\n\nMost tutorials tell you to prompt the model: \"Be helpful, concise, and accurate.\" This is useless noise. Large models are overfit to being polite. They don’t care about your business logic unless you enforce constraints.\n\nIn our early tests, the model would generate beautiful, well-formatted responses that were technically incorrect. It prioritized fluency over facts. It sounded smart while lying.\n\n### Solution: Few-Shot Prompting with Strict JSON Schemas\n\nWe stopped using natural language instructions for the final output. Instead, we defined strict JSON schemas for the response. We provided five \"few-shot\" examples where the input was complex and the output was a rigid structure: `{ \"issue\": string, \"steps\": string], \"citations\": [int] }`.\n\nBy forcing the model to fill a schema, we eliminated the rambling. More importantly, we forced it to cite specific document IDs for every claim. If it couldn’t find a citation, it returned a null value instead of inventing one.\n\nThis shift reduced hallucination rates significantly. It also made debugging easier. We could trace exactly which document ID the model retrieved and why it failed to extract the right step.\n\n## RAG Isn’t Just Retrieval. It’s Ranking.\n\n### Problem: The First-Chunk Bias\n\nRetrieval-Augmented Generation assumes the most relevant document is always the first one returned. Our vector search was returning topically similar documents, but not necessarily the *correct* ones. \n\nFor example, a query about \"iPhone 15 battery drain\" would return a generic \"General Battery Tips\" article because the semantic similarity score was high. The specific troubleshooting guide for the iPhone 15 was buried on page 3 of results. The LLM read page 1, got the wrong info, and moved on.\n\n### Solution: Hybrid Search with Recency Weighting\n\nWe implemented hybrid search. This combines vector similarity (semantic meaning) with keyword matching (exact terms). We weighted exact matches higher for product-specific queries. Additionally, we introduced a recency decay factor. Documents updated in the last 30 days received a boost. If a document hadn’t been updated in two years, its score dropped by 40%, assuming the tech landscape had likely changed.\n\nThis simple adjustment aligned our search results with user intent much better. It’s not about finding words; it’s about finding *current* truth.\n\nFor a deeper dive into how search algorithms are shifting under the weight of AI, check out [Zero-Click Survival Guide: How GEO Reclaims Your Brand Visibility When 72% Of Searches End Without A Click. It explains why traditional relevance metrics are failing in the age of generative answers.\n\n## Evaluating What \"Good\" Looks Like\n\n### Problem: Subjective Quality Scores\n\nHow do you know if your LLM is doing a good job? We initially relied on human review. This is slow, expensive, and inconsistent. One reviewer might think a vague answer is \"acceptable,\" while another marks it as a failure. We needed objectivity.\n\n### Solution: LLM-as-a-Judge with Guardrails\n\nWe built an evaluation harness. For every 100 queries, we used a separate, smaller LLM to grade the output against the ground truth. We defined clear rubrics: *Did the response contain a direct answer? Were all cited documents relevant? Was there any contradictory information?* \n\nThis automated grading allowed us to run thousands of variations of chunk sizes, prompt templates, and temperature settings overnight. We found that lowering the temperature from 0.7 to 0.2 drastically reduced creativity but increased factual consistency. For support scenarios, consistency is king. Creativity is a liability.\n\n## Scaling Costs vs. Performance\n\n### Problem: The Token Bill Spiral\n\nRunning large models is expensive. Our initial setup used a 70B parameter model hosted on a cloud GPU. We were burning through $2,000 a day for moderate traffic. It wasn’t sustainable for long-term retention or personalized recommendations.\n\nWe analyzed the query distribution. 80% of queries were simple FAQs. Only 20% required complex reasoning across multiple documents. Using the giant model for the FAQs was financial suicide.\n\n### Solution: Model Routing and Distillation\n\nWe implemented a router. Simple queries were sent to a distilled 7B parameter model. Complex, multi-step queries went to the 70B model. This cut our average inference cost by 60%. \n\nFurthermore, we fine-tuned a small open-source model on our specific domain data. While the large general-purpose models are impressive, they lack deep domain expertise. A fine-tuned smaller model often outperforms a generic large model on niche tasks because it has seen more of *our* specific jargon and edge cases.\n\nIf you are looking to optimize your existing content strategy to support these lower-cost models, reviewing SEO Content Optimization Tools 2026: Surfer SEO, Clearscope, MarketMuse, Frase, and SilkGeo Compared can help you align your editorial workflows with AI-readiness.\n\n## The Hidden Infrastructure Debt\n\n### Problem: Latency and Time-to-First-Token\n\nAccuracy means nothing if the user waits 15 seconds for an answer. Our RAG pipeline involved: 1) Embedding the query, 2) Searching the vector DB, 3) Retrieving top 5 docs, 4) Formatting the prompt, 5) Generating the response.\n\nEach step added latency. The total time-to-first-token (TTFT) was unacceptable for real-time chat interfaces. Users abandoned the chat after 4 seconds.\n\n### Solution: Asynchronous Pre-fetching and Caching\n\nWe stopped doing everything in sequence. We implemented aggressive caching for common queries. But more importantly, we pre-computed embeddings for our most popular 1,000 articles. When a user typed a query, we didn’t wait for the full retrieval process to finish before showing *something*. We streamed the response token by token while simultaneously refining the context window in the background.\n\nWe also moved the vector database to a local SSD-backed instance rather than a networked storage solution. This reduced retrieval time from 400ms to 20ms. Small infrastructure tweaks yielded massive UX improvements.\n\nFor teams struggling with site performance alongside these new AI integrations, remember that speed matters. Core Web Vitals Are Not Dead: How I Saved A 30% Traffic Drop By Fixing The Invisible Metrics covers the non-negotiable technical baselines you need before layering on heavy JS or API calls.\n\n## From Pipelines to Autonomous Agents\n\n### Problem: Static Answers for Dynamic Problems\n\nEven with perfect RAG, the model could only answer what was in the database. If a user asked, \"Can you check my account status?\", the model couldn’t do anything. It was a librarian, not an assistant.\n\nWe needed to bridge the gap between information retrieval and action execution. This is where \"Agents\" come in. But most definitions of agents are vaporware.\n\n### Solution: Tool-Use Functions with Human-in-the-Loop\n\nWe exposed internal APIs as \"tools\" to the LLM. The model didn’t execute the code; it generated a function call. For example, it would output `{"function": "get_account_status", "args": {"user_id": "123"}}`. \n\nOur backend validated this call. If it required sensitive data access, we routed it to a human agent via Slack. If it was low-risk, like checking order status, the backend executed it and passed the result back to the LLM to formulate a natural language response.\n\nThis hybrid approach gave us the power of automation without the risk of autonomous errors. It turned our static FAQ bot into a functional support layer.\n\nBuilding the right architecture for these interactions requires a mindset shift. You aren’t just building a content pipeline; you’re building a workflow engine. Read Build Agents Not Pipelines: My 6-Month Experiment With Autonomous Workflow Automation to see how we moved from linear scripts to decision trees.\n\n## The New SERP Reality\n\n### Problem: Content Being Erased from Index\n\nAs we integrated LLMs, we noticed a correlation with organic traffic drops. Our branded queries were still strong. But informational queries were disappearing. Why? Because users were getting their answers directly from the AI overview in search results, not by clicking through to our site.\n\nLarge scale models are changing the discoverability landscape. If your content isn’t structured to be cited, it becomes invisible.\n\n### Solution: Structured Data as the Source of Truth\n\nWe audited our top-performing content. We added explicit `QAPage` and `HowTo` schema markup. We broke down long paragraphs into bullet points that could be easily parsed by scrapers and LLMs alike. We also created dedicated \"source\" pages for every major claim we made, linking to primary research or official documentation.\n\nThis didn’t just help AI citations. It improved our visibility in traditional SERPs as well. Google’s systems prioritize structured, authoritative data. By feeding the machines what they want, we kept our brand visible even as the click-through rate dropped.\n\nUnderstanding this shift is vital for survival. The New SERP Reality: How AI Overviews Are Reshaping Search Industry Trends In 2024 breaks down the exact mechanism behind these traffic shifts.\n\n## Final Thoughts on Scale\n\nScaling large language models isn’t about buying bigger GPUs. It’s about cleaning your data, constraining your outputs, and routing your queries intelligently.\n\nWe started this project thinking we needed more intelligence. We ended up realizing we needed more discipline. The models are ready. Your data probably isn’t. Fix the foundation, then build the house.\n\nIf you want to explore how AI is reshaping the broader ecosystem beyond just internal tools, look into [AI Agent Reality Check: Why Google's New RAG Era Demands A Fresh SEO Strategy](https://silkgeo.com/blog/ai-agent-reality-check-why-googles-new-rag-era
> 说实话写这篇的时候我反复确认了三遍数据，因为搞错了会被同行笑话。