The Data Horizon

I was looking at an access log.

There was no particular reason to do it. The server was working, the application was responding, and nothing appeared to be on fire — which, in server administration, is usually a good sign.

But the log was moving quickly.

Google was there.

So was Amazon.

Then Meta.

Anthropic.

OpenAI.

Different IP addresses. Different user agents. Different crawling patterns. All arriving at the same small corner of the public Web.

And they all seemed to be in a hurry.

A busy evening on the Web

The website had recently gained a large collection of structured public data.

Nothing secret. Nothing obtained from behind a login screen. Public procurement notices, organizations, contractors, CPV classifications, locations and relationships between them.

The interesting part was not any individual document.

It was the structure.

A procurement notice led to a contracting authority. The authority led to a municipality. The notice contained CPV codes. Those codes led to categories, regions and other notices. Results led to contractors. Contractors led to their profiles.

One page became ten paths.

Ten paths became thousands.

And the crawlers noticed.

Watching the access log for a few minutes produced a peculiar picture. One crawler was walking through old procurement notices. Another was exploring CPV categories. Another jumped between regions. Yet another discovered company profiles.

Some requested pages one after another.

Others arrived from many IP addresses at once.

It looked less like someone reading a website and more like several machines discovering a new deposit of something valuable.

Data.

That was when an entirely different image came to mind.

A black hole.

The event horizon

A black hole does not need to chase matter.

Matter simply gets close enough.

Once it crosses the event horizon, the rest follows from physics.

The comparison with Web crawlers is obviously imperfect. A server does not lose a document when somebody downloads it. Information can be copied without disappearing from its original location.

And yet the image was difficult to ignore.

Imagine several enormous data centres as dark objects somewhere beyond the visible surface of the Internet.

Around them is the public Web: documents, databases, discussions, photographs, product descriptions, scientific papers, government records, source code and billions of ordinary pages.

And between them lies something resembling a new boundary.

A data horizon.

Cross it by publishing something openly on the Web and, sooner or later, a machine may arrive.

Then another.

Then several at once.

Identifying the visitors

For decades, recognizing Web crawlers was mostly a familiar exercise.

Googlebot was Google.

Bingbot was Microsoft.

Search engines crawled pages so that people could later find them.

The relationship was reasonably easy to understand.

Today an access log can look slightly different.

BOT_SIGNATURES = {
    "Googlebot": "Google",
    "Claude-SearchBot": "Anthropic",
    "Amzn-SearchBot": "Amazon",
    "meta-externalagent": "Meta",
    "ChatGPT-User": "OpenAI",
}

def identify(user_agent):
    for signature, operator in BOT_SIGNATURES.items():
        if signature in user_agent:
            return operator

    return "Unknown civilization"

The last case has not caused any problems yet.

But it seems prudent to keep it.

Why the hurry?

This is where the access log stops providing answers.

It tells us who requested a URL.

It tells us when.

It tells us how often.

Sometimes it gives us a reasonably descriptive user agent.

It does not tell us what ultimately happens to every piece of information after it has been retrieved.

Search indexing is one possibility.

AI-powered search and retrieval is another.

Building representations of the Web, refreshing existing indexes, discovering relationships between entities or supporting products that do not yet exist are others.

Different crawlers may have completely different purposes.

So it would be tempting — and wrong - to look at a few thousand HTTP requests and announce that we know exactly why every company is collecting every page.

We do not.

But there is another observation we can make without knowing their internal architecture.

They want the data.

Or, more precisely, their systems consider discovering and retrieving it useful enough to spend bandwidth, computing resources and infrastructure doing so.

That alone is interesting.

Data as a raw material

We have spent years talking about data as the new oil.

The metaphor has become so overused that it barely means anything anymore.

Perhaps the better analogy is not oil but matter.

Matter has no single purpose.

It can become a bridge, a processor, a house, a battery or a spacecraft.

Its value depends on what can be built from it.

The same increasingly seems true of public information.

A procurement notice by itself is a document.

Ten thousand procurement notices form a dataset.

Connect them to organizations, locations, classifications, contractors and results, and something else appears.

A network.

And networks are particularly interesting to machines.

They provide context.

A company is no longer merely a name.

It becomes a company that won a particular contract from a particular institution in a particular place for a particular type of work at a particular price.

The individual facts may have been public all along.

The connections between them are where much of the value begins to emerge.

The old bargain

The public Web developed around a remarkably simple bargain.

You published something.

A search engine copied enough information to understand it.

In return, the search engine helped people discover your page.

The crawler consumed resources, but it potentially returned visitors.

For millions of websites, that exchange became so normal that considerable effort was spent encouraging robots to crawl more.

Sitemaps were created.

Internal links were improved.

Structured data was added.

Entire industries emerged around making machines understand websites better.

Website owners spent decades asking:

How do we get the robots in?

That question may not disappear.

But another one is beginning to sit beside it.

How many robots do we want inside at the same time?

Because the economics of the relationship may be changing.

A machine can now retrieve information, interpret it, combine it with other information and present an answer somewhere else.

The human reader may never need to visit the original page.

That does not automatically make crawling harmful.

It does make the old bargain less obvious.

Several black holes

There is another problem with the astronomical metaphor.

There is not one black hole.

There are several.

They compete.

Google wants fresh information.

Microsoft wants fresh information.

Meta wants information.

Amazon wants information.

Anthropic wants information.

OpenAI wants information.

Other systems will want it too.

And if useful public data appears somewhere on the Web, waiting politely for a competitor to discover it first is not necessarily an attractive strategy.

Perhaps that explains some of the apparent urgency.

Perhaps not.

But competition changes the geometry.

Instead of imagining the Web slowly falling toward one enormous centre of gravity, imagine several centres moving through the same space, each building its own representation of the world.

The public Web sits between them.

Every new document slightly changes the field.

Every new connection makes the structure more interesting.

And every crawler wants to know what changed.

The Web is acquiring new readers

For most of its history, the Web was written by humans, for humans, with machines acting largely as intermediaries.

That description is becoming incomplete.

Machines are becoming readers in their own right.

Not readers in the human sense. They do not sit down with coffee and become fascinated by municipal procurement.

At least there is no evidence of that in the access logs.

But they navigate.

They discover.

They classify.

They extract relationships.

They return.

And increasingly, one machine may publish information primarily because another machine will consume it.

APIs already work this way.

Structured data works this way.

Parts of the ordinary Web may gradually begin to work this way too.

That creates an unusual possibility.

The future Web may contain enormous amounts of information that humans can read but that is economically valuable primarily because machines can read it at scale.

Beyond the horizon

There is something slightly absurd about watching all this from a terminal.

The grand discussions about artificial intelligence usually happen at the scale of billions of parameters, enormous GPU clusters, trillion-dollar companies and predictions about the future of civilization.

At the other end of the system there is an ordinary Web server.

It receives:

GET /something HTTP/2

It looks for the requested information.

It returns:

200 OK

Then another request arrives.

And another.

Google.

Meta.

Amazon.

Anthropic.

OpenAI.

Perhaps someday:

Unknown civilization

The server does not know whether the request is historically significant.

It just serves the page.

And perhaps that is what makes watching the logs so interesting.

Large technological changes do not always announce themselves with a new device appearing on a desk.

Sometimes they appear as a small change in who - or what - is reading.

For years, we built the Web and waited for people to arrive.

Then we invited search engines to help them find us.

Now a different population is appearing at the door.

It is automated.

It is persistent.

It can read very quickly.

And judging by the access logs, several of its members have already found the entrance.

Nothing disappeared.

The server was still there.

The data was still there.

But something had crossed the horizon.

READY.