As you read this, an LLM is crawling your website. Perplexity processes over 400 million searches per month, ChatGPT is estimated to process 1.1 billion, and Google AI overview accounts for a growing proportion of the 255 billion searches per month. But are they getting the right information?
Why you should think of your website as a database
The development of Large Language Models–LLMs–reflects an increasing shift in the perception and use of websites: both for humans and robots, including LLMs. This places demands on how we work with websites, both technically and tactically, if we want to keep up with developments and ride the wave.
For a long time, we’ve designed and developed websites as a kind of digital brochure. These were more or less static “pages” optimized for humans. But AI systems do not read websites visually, as humans do. They search for data in order to structure and make sense of the content. A structure we normally build APIs for.
With LLMs, we need to rethink this. Data and representation must be separated. Headless CMS has been an important step in this direction, which has also found its way into larger organizations and companies. But not just because it is a technical trend. No, it is a strategic opportunity for companies.
LLMs create new business opportunities and change the playing field. This applies both in terms of visibility, when potential customers “prompt” topics where you’re an expert, think of Google AI Overview, Perplexity, and ChatGPT, but also in terms of usability.
Your website will become much more than a showcase. It’s marketing, user research, and business development all rolled into one. Based on user behavior, your data and content can be optimized for relevance using, for example, vector embedding, so that it is presented in a way that best matches the presumed user profile. Your website will be able to exchange actions with other AI agents. Agents that will act on your behalf, for example, comparing products and potentially completing the purchase process, just as you’ll be able to orchestrate enormous amounts of knowledge and media internally and externally in different models.
Need for a new standard
How do we handle the way LLMs request data from your website?
Model Context Protocol (MCP) seems to be the way forward. American company Anthropic has developed this open standard, which after just eight months has already been adopted by Google and OpenAI . My experience so far indicates great potential, and I don't recall ever seeing a standard adopted so quickly and so widely.
It’s intended as a standardized way for AI systems to communicate with your data. So, just as robots.txt guides Google's crawlers, MCP.txt, if it becomes a standard, will be able to guide future AI services to access and use website data and functionality optimally.
There’s already another proposal for a standard: llms.txt. In my opinion, however, this solution is short-sighted, as it targets a specific model type. With the developments we have seen in AI, we should be open to the idea of development in model types as well. Here, MCP has a clear advantage, as it makes data and functionalities available to all models that have an MCP client.
The future belongs to those who are prepared
Just as responsive design became mandatory after the mobile revolution, AI-ready architecture is now becoming essential for survival. A path that ensures optimized, secure, and efficient communication between your server, your data, and LLMs.
I hope and assume that MCP will become the industry standard. So it’ll require you to define “context” with MCP. You need to control what and how much data is exchanged with LLMs and other models.
If you want to ride the LLM wave, you need to be in control of your data, your architecture, your security, and free yourself from decades of classic website understanding. The question is not whether AI will become your biggest user—it will—so it's more about whether you can manage and take advantage of your most frequent user.
