Some Of Its Parts

A Company That Could Benefit From A Breakup

When AI Threatens Internet Search

This is Porter & Co.’s The Big Secret On Wall Street, our flagship publication that we publish every Thursday at 4 pm ET. Once a month, we provide to our paid-up subscribers a full report on a stock recommendation, and also a monthly extensive review of the current portfolio… At the end of this week’s issue, paid-up subscribers can find our Top 3 Best Buys, three current portfolio picks that are at an attractive buy price. You can go here to see the full portfolio of The Big Secret. Every week in The Big Secret, we provide analysis for non-paid subscribers. If you’re not yet a paid subscriber, to access the full paid issue, the portfolio, and all of our Big Secret insights and recommendations, please click here.

Six-year-old Michael Calce didn’t know it… yet… but his parents’ run-of-the-mill divorce would ultimately cost Amazon, eBay, and Yahoo about $1.7 billion.

In the meantime, the custody battle took a human toll. In classic late-1980s fashion, Michael’s workaholic dad got the kid every other weekend, along with a sparsely-furnished bachelor condo in downtown Montreal. 

It wasn’t much fun for either of them.

Unsure how to entertain a mopey first-grader who missed his mom and friends, Michael’s father unplugged a spare computer, carted it home from his office, and said, “Here you go.”

For Michael, it was love at first sight. “I remember how the screen lit up in front of my face,” he wrote years later. “There was something intoxicating about the idea of dictating everything the computer did, down to the smallest of functions. The computer gave me, a six-year-old, a sense of control and command. Nothing else in my world operated that way.”

By the time he was nine, Michael was hacking, in a modest way. He figured out how to extend his dad’s 30-day AOL free trial indefinitely, so he could stay on the internet forever. And Michael (screen name “Mafiaboy”) plus the internet was a very, very bad combination.

Back then, the web was small. And the robots that crept through it – examining it, cataloguing it, and retrieving information – were small, too. They were called crawlers, spiders, ants, or worms. For the most part, they were benign little programs, written by geeks who wanted to index the internet and itemize the fascinating sites that lived there. 

A page must be indexed to appear in search results. If it isn’t indexed, it’s effectively invisible to search engine users. (For a while, in the early ’90s, it was possible to list every single site on the web – in 1993, there were about 100.)

Too many spiders (or, spiders in the wrong hands) could be deadly, though. A flood of incoming bots could paralyze a site with too much information, slow it down drastically, or crash it altogether – something called a “DOS attack,” for denial of service. (Folks like Mafiaboy do this for fun – and sites like Amazon and eBay lose billions of dollars.)

The first-ever DOS attack, though, happened purely by accident – in 1994, when Charlie Stross, an English computer geek, made a little spider-bot and set it free in the wilds of the internet. The spider crawled onto a small company server, indexed all the pages at once, slowed down the system, and royally pissed off the webmaster, a Dutchman named Martijn Koster.

Martijn emailed Charlie and reamed him out. Then the Dutchman wrote a little program called “robots.txt” – a few lines of code that’s known today as the unofficial Constitution Of The Internet.

Robots.txt’s directive (like Isaac Asimov’s Prime Directive, the ethics code that all science-fiction robots must follow) is very simple. It tells robots to get the hell off the page. If the webmaster includes this request in the page code, a mannerly crawler will steer clear of that page – and will only index the pages the site chooses to allow. 

Of course, it’s usually in websites’ best interests to allow indexing. In a nice tit-for-tat arrangement, search crawlers – like Googlebot, Bingbot, and so on – make indexed sites visible on the web and drive moneymaking traffic to indexed pages. But this only works if everyone is polite and goes where they’re invited… hence, robots.txt.

Bots Gone Wild

By mid-1994, robots.txt was adopted across most of the web – and, in 2019, became the official standard under the Internet Engineering Task Force. But, functionally, it’s only a handshake agreement. It’s not binding – and it assumes that everyone on the internet wants the best for this big web we crawl around in.

Mafiaboy just wanted chaos. In the mid to late ’90s, as he entered his teen years, he figured out how to seize control of bigger networks of computers – and in 2000, just for a lark, he decided to crash the servers of five or six of the world’s biggest websites. He’d ignore robots.txt and pummel every page on every site with an overload of crawlers. 

He set up his program, let his spiders loose, and headed to school. By the time class was out, he was wanted by the FBI.

Mafiaboy’s crawlers took down Amazon, eBay, Yahoo, and several other major websites, costing almost $2 billion in damages and attracting the joint attention of U.S. and Canadian law enforcement. When 16-year-old Michael headed home from school and saw the white van idling at the end of his street, he knew the worm had turned. 

Michael Calce’s trial was nationally publicized and ended with 50 criminal charges (mostly related to unauthorized access of a computer) and an eight-month stint in a youth correctional center. (By then, his father surely wished he’d brought home a puppy instead of a computer after the divorce.)

Mafiaboy’s was not the only big DOS attack around the turn of the millennium: in 1996, New York internet provider Panix had been famously shoved offline for 36 hours by malicious crawlers. Moving forward into the 2000s, bad actors found ever more sophisticated ways to overload sites and wreak havoc, such as the “DDOS,” (Distributed Denial of Service), a mega-DOS attack with bots coming from multiple networks. Dubbed the “Year of the DDOS,” 2016 weathered five separate high-profile attacks on Reddit, CNN, Twitter, and other sites. 

For the most part, though, web-dwellers have obeyed the Constitution Of The Internet and respected the rule of robots.txt. If a site says “Keep Out,” the crawlers keep out.

Until, well, right about now.

Twilight Of The Worms

Mafiaboy hails from Canada, and few real supervillains come out of the sweet land of maple syrup. Today he’s a cybersecurity consultant and has learned valuable life lessons (and gotten a book deal) from his eight months in juvie.

But the internet, it seemed, learned nothing about what happens when web crawlers ignore the “Keep Out” sign.

Martijn Koster – the Dutch father of robots.txt – was robot-agnostic at best. “Robots are one of the few aspects of the web that cause operational problems and cause people grief,” he acknowledged in 1994. “At the same time, they do provide useful services.” Three decades later, with artificial intelligence (“AI”) exploding in popularity, robots are now offering all the grief, without any of the useful services.

AI crawlers work just like the web’s early worms, but with exponentially greater speed and force. Where the polite index crawlers of the early internet obeyed the “Keep Out” sign of robots.txt, the giant AI scrapers of ChatGPT, Perplexity, Anthropic, and others steamroll across every site, every page, and every paywall, harvesting data without permission.

They grab search results without offering the quid-pro-quo of profitable site traffic, the way Google and other search engines have always done. At scale, these industrial-grade scrapers sometimes overwhelm sites by their sheer presence, creating a de facto DOS attack that can sweep smaller websites offline altogether.

The New York Times, the BBC, Amazon, Facebook, and countless other big sites have explicitly updated their robots.txt to bar ChatGPT’s voracious crawlers from entry. A fat lot of good it’s done.

AI respects neither God, man, nor robots.txt. And it’s killing the internet – specifically, search engines. 

In April 2025, Google searches on Apple’s browser, Safari, fell for the first time in history… along with Alphabet’s market cap, which immediately lost $150 billion, or 5% of its value. A recent Pew Research study finds that after seeing an AI answer pinned at the top of the page, the likelihood that people will click on a Google link drops by half. Traffic to major news and finance sites has plummeted – for instance, Business Insider has laid off a fifth of its staff after a 55% drop in traffic over the past three years – and that’s likely only the beginning…

Mafiaboy’s mischievous crawlers shut down a part of the web, for a part of one day, and destroyed $2 billion. AI’s huge, malevolent spiders are fixing to shut down the whole internet, for all time – although these non-human bots make up 35% of pageviews, they account for about 65% of the most resource-intensive traffic, costing hundreds of billions of dollars of wasted bandwidth and ad revenue. “The nature of the internet has completely changed,” the chief executive of coding site Stack Overflow tells The Economist. “AI is basically choking off traffic to most content sites.”

We actually don’t have any assurance that search, as a function, will survive the AI apocalypse.

But we do have our finger on the pulse of the one search engine that we believe will survive the twilight of the worms… not by fighting them, but by joining them.