The emergence of the web data infrastructure layer for AI

A new web data infrastructure layer is emerging to enable AI models to discover and map the ever-expanding digital realm, addressing a fundamental bottleneck in keeping pace with dynamic, unstructured web data [1]. While early AI breakthroughs were driven by scaling training data and model size, organizations now encounter a bottleneck: grounding outputs in current, verifiable information [1]. AI performance increasingly depends on a system’s compute, networking, retrieval, and data engineering capabilities [1]. Traditional model training relies on snapshots of information collected at a particular point in time, which is no longer sufficient for tracking fluctuations such as competitor pricing, consumer sentiment, and market trends [1]. Large language models (LLMs) are neural networks trained on vast amounts of text, but biased or inaccurate training data can make an LLM's output less reliable [2]. Foundation models, which are trained on massive datasets for wide-ranging use cases, are often highly resource-intensive to build, costing hundreds of millions of dollars for data acquisition, curation, and compute power [3]. Delayed data retrieval can reduce the usefulness of an otherwise sophisticated model [1]. Using live, high-quality web data can reduce AI hallucinations and build user trust [1]. One survey found that 56% of AI practitioners said businesses need access to real-time web data to improve trust in AI outputs [1]. According to Gartner, 60% of AI projects that are not supported by AI-ready data will be abandoned by the end of the year [1]. Some research has found that 97% of AI organizations depend on real-time web data infrastructure, but 90% feel boxed in by various restrictions [1]. “If it can't retrieve real-time information, it lacks context. In a business setting, that's not acceptable anymore. Stale answers lead to bad decisions and disappointed consumers,” said Or Lenchner, CEO of Bright Data [1]. Lenchner described the required infrastructure as one that can mimic a web user with identifying information—IP address, location, and 1,000 more parameters—at scale, doing that 80 billion times a day for millions of websites [1]. Continuous retrieval introduces data governance challenges [1]. Platforms can enforce compliance protocols aligned with global privacy frameworks, such as the EU’s General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA) [1]. The field of artificial intelligence, which encompasses these systems, has seen increased interest and funding since 2012 with advances in deep learning and the transformer architecture, leading to an AI boom in the 2020s [4]. Lenchner noted that when this becomes critical infrastructure, doing it in-house becomes a full-time engineering problem that competes with the actual AI work [1]. “Think of the trained model as intelligence and relevant data as knowledge. A powerful intelligence layer sitting on top of a hollow knowledge layer is like a genius who knows nothing—useless in practice. Intelligence and knowledge have to come together,” Lenchner said [1].