Lens 2: Fake threats and Real threats to Google's supremacy over the years.
In the first part of this series, I discussed the various categories of search intents and how the search UI (read: Google UI) has evolved in the past twenty years. I concluded by saying:
Until late 2022, interactive browsing of the Knowledge Graph; and “Featured Snippets” and “People also ask”, both powered by deep natural-language understanding; were the pinnacle of Search user experiences.
Until, that is, ChatGPT arrived.
That is, until ChatGPT brought interactivity and natural-language understanding together so beautifully — ChatGPT put a stake in the ground, saying, ‘Here’s what a compelling new search experience could look like.’ Does this constitute a moment of significant vulnerability to Google?
I thought I’d start by outlining what, over the years, were possible moments of real and imagined vulnerability for Google’s supremacy in Search. This is mostly an after-the-fact perspective, and a personal one at that (classic Monday-morning quarterback stuff). As a long-time (now ex-) Googler, I have strong technical knowledge of how core systems like Search and Ads work, but this essay is not about any technical details or secret sauces, but my opinions on the Search ecosystem and its competitive landscape.
Let’s start all the way back — when Google was in its earliest years.
(Image credit: 1000 Logos and, of course, Google)
When Google came on the scene, there were two prevailing paradigms for discovering useful content on the Web, respectively from Yahoo! and AltaVista. The former was a hand-curated directory of websites built by Jerry Yang and David Filo in 1994, soon augmented with search functionality to search the directory itself. AltaVista, conceived of by Digital’s Paul Flaherty, who is credited with the idea of indexing webpages, and who built it together with Louis Monier and Mike Burrows. Soon after, Yahoo! went public in 1996 and added proper Web search functionality by adding a search box powered by AltaVista.
Surprisingly, neither Yahoo! nor AltaVista was the first search engine on the Web — that honor (likely) goes to the “WWW Worm” built by Oliver McBryan, which had some clever ideas (like attaching the anchor text of a hyperlink to the page/object it points to — this was a key idea in Google as well, and appears in the Brin-Page paper from 1998). The idea of using hyperlinks to rank webpages had also been foreseen (or independently discovered) in a 1997-submitted patent by Robin Li, who went on to found Baidu.
When Google was launched in 1998, it brought two stunning improvements over the AltaVista generation of search engines (of which quite a few were brewing).
The first stunning improvement was search quality — that is, providing highly relevant answers to the search queries typed into it. PageRank, of course, was a breakthrough — and not just for the elegant mathematical underpinnings. One of the key benefits of incorporating PageRank into search was to be able to take advantage of a global notion of a high-quality page, thus largely weeding out what Brin and Page called “junk pages” in their paper. The introduction of PageRank as a search signal rephrased the promise from the search engine to the user as “here are the best pages on the Web, sorted by how relevant they are to your query” (as opposed to “here are some pages from the Web, sorted by how many of your query terms they contain”). Besides these two benefits (authoritative pages, less junk), Google benefited from the fact that indexing the anchor text for webpages has a pleasing positive interaction with PageRank. Since search was still a new endeavor and a significant fraction of queries were likely navigational (looking up a company or a person or a broad topic), these features worked remarkably well, and Google’s search quality seemed almost magical (compare, for example, to this video of AltaVista in 1996).
The second stunning improvement that Google delivered over AltaVista was a lovably clean user interface. From 1995 to 1998, AltaVista had already evolved into a mess of too many irrelevant and distracting links, commercial placements, etc. that aren’t unlike what this parody video (of Microsoft packaging the iPod) shows. The “plenty of white space” design that Google delivered was a breath of fresh air, and enough has been said and written about it.
Of course, great quality even in a messy interface might’ve worked well, but poor search quality would not have survived even with the cleanest user interface. As this 2003 NYT article notes: “[as] Google rises, some of its new challenges are in the form of imitation. Yahoo and Ask Jeeves, a search engine, have tried to match the spartan simplicity of Google's home page.”
The first real threat: Google underestimates itself
The story used to be apocryphal and not well-known or well-believed, but Vinod Kholsa confirmed that in 1999, Brin and Page wanted to sell Google to Excite for $1M, were willing to be talked down to $750K, but the Excite CEO refused to spend that much. If that had happened, who knows if Google would’ve survived as we know it today.
By 2000, though, Google had signed a deal to provide Yahoo! with search as a service (for a few million dollars a year), and a just a year later, would turn down a $1B offer from Yahoo! to be bought out!
What changed Brin and Page’s minds about the commercial value of Google?
Between 1999 and 2001, Google had launched a cost-per-impression model of advertising where advertisers could bid for specific keywords, and the highest-paying advertiser won the best spots. Google supposedly made $70M in the first year of this business. No wonder Brin and Page turned down a billion dollars!
The first fake threats — in retrospect — Yahoo! and Bing
What is surprising though, is that even with $70M circa 2000, Google was not the company that made the most money off search advertising. It was Overture (later acquired by Yahoo!), which was generating the most search advertising revenue, less than a dollar at a time, through its pay-per-click model.
Not unlike how all competitors incorporated PageRank into their algorithms and started mimicking Google’s clean UI, Google imitated Overture’s pay-per-click model and launched (in 2002) the form of search advertising that lives to this day. It came with a brilliant twist: advertisers can bid on keywords, and the best slot wouldn’t be given to the highest bidder, but to the bidder whose ad is likely to bring the most revenue to Google. This design, attributed by Google’s Chief Economist Hal Varian to Google engineer Eric Veach, fixes a core problem with Overture’s pay-per-click model — how do we let market forces figure out what a click is really worth and, at the same time, ensure that the most relevant ads are ranked highly?
The reason I spent a fair bit of ink talking about advertising is this: the period 2001-2003 should’ve been a time of real vulnerability for Google. A whole bunch of search engines, search technology providers, and search advertising companies — through a number of acquisitions — were landing at Yahoo!. This included AltaVista, Alltheweb, Fast, Goto, Inktomi, Overture, etc. At this point, Yahoo! had nearly all the ingredients to outdo Google, which it was using to power its Search. The ingredients for good search quality were also becoming clearer, with a flurry of (good) academic research starting to appear. If anything, it was Google that was copying a Yahoo! trick (pay-per-click mentioned above).
How, then, did Google survive — scratch that, thrive, given this existential threat?
I wasn’t at Google during this time, so I can’t speak authoritatively. But I’ll offer my honest opinion:
Google thrived because Google put the user experience first, while its most credible rival had too many things going on to care much about the user.
To be fair, Yahoo! did a fantastic job of absorbing a plethora of technical stacks, people, and processes to produce a good search engine whose quality — when you strip away the logos, the distractions, the ads, etc. — was not perceptibly worse than Google, say around late 2004. How is it that the user experience was compromised? I’ll offer two answers to this.
First, I believe that poor-quality ads on a search result page lead to an extremely bad user experience. When this happens often enough, and you heard (or tried) this other search engine whose overall user experience is much better, you defect. Yes, competition is always a click away. I believe that the little twist that Google made with their ads ranking (while Yahoo! continued to show search ads sorted by bid, as noted in this NBER article by Edelman, Ostrovsky, and Schwarz) made a fairly significant difference to user experience — especially when you consider that search is a repeated activity, and even if a user defects only with 0.1% probability, they’re likely to have defected after doing one search a day for a year.
Secondly, and equally importantly — Yahoo! could never make up its mind about whether they are a “technology company” or a “media/content company”. The news and blog media of the 2000s attributed this split personality to the hiring of Terry Semel from Hollywood as Yahoo!’s CEO in 2001. I believe that the trouble goes deeper: Yahoo! evolved from being a “portal” on the Web into a strong technology company with strong search and advertising chops, excellent “big data” technology, and beloved products like Yahoo! Mail. However, they could never shed the portal mentality entirely: it was too risky a bet to have yahoo.com show a white page with a search box, since the bulk of its users came to yahoo.com to read their email or catch up on the day’s Kim Kardashian stories. The same Edelman et al. article notes that while Google made 98% of its revenue (then) from search advertising, Yahoo! made only about 50% of its revenue from Search; the rest came from display advertising on its highly popular properties. In fact, it was Yahoo!’s strong “content” brand that drew people there; this had the unfortunate side effect that the search engine didn’t need to stand on its own merits; there was always traffic coming — until it wasn’t. (Microsoft’s Bing search engine, rebranded from MSN/Live Search suffered from this malaise as well.) Could Yahoo! have launched a search engine under a different brand and grown it organically using the technical assets it had? We’ll never know. (Interestingly, Apple seems to be struggling with similar questions now — except Apple also has to contend with losing the huge sum of money it gets from Google for being the search provider on iOS devices.). A key lesson from Yahoo!’s demise that is relevant for Google is the classic innovator’s dilemma: should you cannibalize yourself by out-innovating your successful products? (There’s a second one — nothing good comes in the long run from money-making products that, on average, hurt user experience. In Yahoo!’s specific case, it was the dollars from display advertising.)
Two very different reasons were also very much at play — the shenanigans of Steve Ballmer and “activist investor” Carl Icahn, the unsolicited Microsoft bids to buy Yahoo!, etc., and various distribution deals (e.g., with Netscape) — but I’ll restrict my discussion to aspects related to search and ads technology and experience.
Enough about Yahoo! What did Google do to deliver an exceptional user experience, besides not repelling them with irrelevant or annoying ads?
A whole bunch of things, it turns out. Let’s look at some of the key ones during the 2003-2010 period: the start of this window coincides approximately with the time Yahoo! switched out of Google to its own search engine and Microsoft went through the painful branding/rebranding exercise, going from MSN Search to Windows Live Search to Live Search to Bing; the end of this period coincides roughly with when Google’s search share reached its “steady state” in the US of nearly 65-70%.
First, Google invested heavily in Infrastructure: the MapReduce framework for scaling the indexing system to eye-popping levels, serving infrastructure that was unimaginably better than the competition (and what could be bought on the market) and that led to lightning-fast search responses; the evaluation infrastructure for search quality — two variants of the search engine presented to raters side by side to collect high-quality feedback based on which to tune the search engine; live experimentation framework for search and ads quality; the list goes on. The investment in core infrastructure like this meant very fast updates, more experiments and more analysis that could power more features like spell correction and dealing with synonyms gracefully.
Second, the emphasis of Google’s search quality team on the relevance of the results. Among other things, this meant two very concrete things: (a) ranking algorithms that were crafted with a huge emphasis on debuggability — if a search quality engineer spots a bad result for a search query, within minutes, they could get to the bottom of what caused it; by cataloguing these systematically, they could see patterns that they would then fix with one or two clever ideas; (b) the search launch process was obsessively focused not just on automatically trackable metrics (click rates, bounce rates or short clicks, etc.), but also analysis by expert analysts and engineers of tens or hundreds of actual search queries on which the ranking produced by the “new algorithm” differed from the incumbent.
Third, Google led the pack in inventing new search features. My favorite example is a feature that Google introduced experimentally in 2004 that we’ve now come to take for granted in every search box, anywhere on the Web: autocompletions. Later in 2010, Google would take this to another level: not just show you search completions, but actually show you results for the top search completion (this feature was discontinued in 2017). Also in the period between 2003 and 2010, Google introduced a variety of additional one boxes (weather, stocks, dictionary, etc.) and started to incorporate news and videos much more prominently in the Universal Search I mentioned in the previous article.
I’ll conclude discussing the dominance of Google over Yahoo! and Bing in the 2005-2010 period with another anecdote about search advertising. In a January 2011 article talking about the biggest features in 2010 in search advertising, the #1 story is the “market pooling” of advertisers as well as search traffic that was engineered in the Bing—Yahoo! “search alliance”. This turned out to be a disaster for a number of reasons that we won’t go into, but more germane to our discussion is the contrast between the business-first thinking (prevalent at Bing and Yahoo!) and the user-first thinking prevalent at Google as illustrated by the #2 story in the same article. This one — the broad match modifier feature of Google Search ads — is a personal favorite of mine, since the sole engineer on the project described there is my co-founder at Tonita. It was a very simple advertiser-facing feature with an elegant idea. Google ads had two types of keyword matching — exact match and broad match (technically, there was a third one called phrase match but let’s leave that aside). The two types offered advertisers a lever between having their ads shown on the most relevant keywords (exact) and having their ads reach as large a pool of keywords as possible (broad) as long as they are somewhat related to their ad. The trouble was, this lever wasn’t sharp enough: if you’re selling wall-mount kits for LCD TVs and bid on the exact keyword “lcd tv wall-mount kit”, your ad will not be shown if I search for “lcd tv wall mounting brackets”; if you use broad matching for the same keyword, you had the danger that your ad will be shown to people searching for “lcd tv”, and your ad will lose out to the high-bid ads from TV makers, and your ad’s “quality score” will go down, your costs will go high, and so on. The insight in broad match modifier is to allow the advertiser to add a ‘+’ sign to some of the terms in the keyword they bid on, that will cause the matching to happen only on keywords that contain those words (or close variants). Thus, for example, you could bid on “lcd +tv +wall +mounting kits” and still match “lcd tv wall mount brackets” but avoid getting shown for “lcd tv”. This feature is very simple, but advertisers flocked to it, and over the next decade, would go on to drive a large fraction of Google’s search ads revenue!
Fake threat: Facebook
When Facebook shot to prominence in the late 2000s, fueled by media articles and Google’s own ill-fated attempt at building a social network, there was a belief that Facebook was a serious threat to Google Search. There was one very valid point in that thinking — that an alternative to the “open Web” was evolving within the walled garden of Facebook. Also, in many ways, FaceBook, Twitter, and later, Instagram, took attention away from some Google products (such as YouTube). Facebook even rolled out “Graph Search’ in 2013, touting the value of searching for information/opinion from one’s own social network of (Facebook) connections. Unsurprisingly, it turned out not to make any dent on Google’s position in Search. The reason is simple: there was only so much content your social network could generate or endorse (e.g., via Likes or Shares) that was worth searching for, relative to the ocean out there on the Web. This is despite the fact that Facebook incorporated Bing search into a Web search box inside Facebook, one that it quickly dropped since it wasn’t core to the Facebook user experience; users still went to Google when they were looking for information from the Web. In fact, to the extent that social media aroused people's curiosity about the world — news, sports, politics, products, everything — it often reflected in more Google searches, not fewer!
Facebook did (and does) compete with Google majorly on a different business — display advertising, where Facebook’s strength as a “publisher” (of content alongside which graphical ads can be shown) soon evolved into an “ad network” through which advertisers could reach a broader set of websites and apps (including Instagram). Display advertising and search advertising serve two very different purposes — the former is a demand creation mechanism and the latter is a demand satisfaction mechanism (or “top of the marketing funnel” vs. “bottom of the funnel”). It is my personal opinion (probably a very naive one that is not shared widely by anyone knowledgeable about the search and advertising business) that despite generating 12+% of Google’s revenue (2021 numbers) — a whopping $30+B — involvement in display advertising (the whole works: AdSense, Ad Exchange, Ad Network, mobile display advertising) has been a net negative for Google. But that’s a topic for another day — this post is long enough without it.
Mobiles happen — a real threat
When the iPhone was announced in 2007 and Android followed shortly after, it was a watershed moment for supremacy in Search. While the media was obsessed with Facebook as a “Google competitor”, there was a fundamental shift going on in the world: web traffic overall, and certainly Search query traffic, from mobile devices was growing rapidly and often surpassing desktop traffic.
The shift of Search traffic to mobile devices had an enormous implication: it was a great opportunity for a competitor (if not dinosaurs like Yahoo! or Microsoft, some small scrappy start-up) to create a magical new Search experience for Mobile devices. Stunningly, this never happened! It’s possible that nobody felt they could take on Google, even for a sufficiently focused and useful slice of searches. More probable, in my opinion, is nobody had a crystal ball with a very specific form of insight: what are the classes of search intents that were growing rapidly on mobile devices, and which ones are worth crafting new experiences for? If some company had created a slick, and blazingly fast, app for “quick information searches” — news, weather, live sports results, recipe searches, easy access to Wikipedia information about prominent entities — I believe it would have gained a fair bit of mindshare and given the company a foothold to start climbing the Search mountain. Alternatively, a slick shopping search app could’ve given that foothold (recall the excerpt from Meeker’s report, linked above, that nearly a quarter of online shopping, at least, on shopping-heavy days like the day after Thanksgiving, was happening on mobile devices).
The lack of a fearless competitor who was willing to rethink Search from scratch definitely helped Google. A slightly different flavor of threat to Google Search came from the idea known as “disintermediation”: that users will directly go to the Fox News or New York Times or CNN or WSJ or ESPN app and bypass Google; that users will directly go to Amazon to shop for something; that they'll go to Yelp to find a plumber, and so on. — essentially bypassing Google. This is certainly happening, but a number of factors have helped Google unwittingly: poor-quality apps and mobile websites (incidentally, a lot of it because of annoying display ads, but also lack of UI innovation even by market leaders like Amazon), the unspoken fear in users’ minds that any one app for a category of use cases might not be comprehensive enough to bypass Google.
Google, of course, wasn’t going to give up its supremacy easily. Over the five years or so during the rapid mobile growth (2011-2016, say), Google Search launched a huge number of features that made the Google app on the iPhone and Android a pleasure to use. Advanced spell correction that was tuned to dealing with the numerous typos that happened on mobile keyboards, launching the microphone button with increasingly sophisticated and accurate voice recognition, anticipating a number of needs and proactively serving them on the Search results page, paying attention to the mobile-friendliness of the landing pages in ranking them, etc., all ensured that Google dealt with this vulnerability before it could mushroom and unseat Google from the pinnacle of that Search mountain. The acquisition of Metaweb in 2010 brought Freebase to become the foundation of Google's Knowledge graph, a significant source of information to serve factoid searches. What is surprising is that Metaweb itself didn't really have any competitor, one that could've created a compelling app for the same slice of search intents, and bootstrapped a serious search engine from there.
Shopping, or, more broadly, choice-driven searches, are a hugely important segment (in the “transactional search” bucket in Broder's taxonomy mentioned in Part 1 of this series) that, to this day, is a poor experience (even on Google — don't even get me started on the crummy Amazon app). I’ll say more about this in Part 6; for now, I’ll merely note that despite the fact that there were numerous greenfield opportunities, nobody stepped up and challenged Google’s supremacy in Shopping Search when the mobile revolution unfolded. Ironically, Google Search didn’t do anything particularly innovative or impressive for shopping searches during this period; it was an innovation in search advertising that may have helped save Google here: the introduction of product listing ads in 2010 started to streamline the search experience for product search queries — not by some massive rethinking of the search page, but by a well-crafted carousel of actual products that addressed user needs in useful ways and, despite being an ads feature, improved the overall user experience for this class of searches!
Competing with Alexa and Siri: Fake or Real Threat?
It pains me to say this, but the Google Assistant has been an unmitigated disaster. Nearly seven years since its launch, at least personally for me, it's a glorified voice-controlled kitchen timer, and on rare occasions, a quick way to get basic factoids from Google through a voice interface when I'm eating dosas with my hands. There are just so many things wrong with it that I don't even know where to begin.
The biggest problem with the Assistant, in my opinion, is that it jams so many features and use cases into one “product” — it can be activated by speaking into a device or typing into an app, the responses can be delivered through a speaking device or visually on a screen; it can be used to close your garage door or increase the volume on your car speakers while you're driving, and it can serve as a voice interface to Google Search; it can be somewhat interactive (stateful) or it can be a bunch of one-off activations. Reflecting all this confusion, it also has one of the most bizarre mix of user bases: some of the use cases are not even a first-world thing but a first-world one-percenter thing (“turn off the Sonos speaker in the kids' bedroom”); and other dramatically different use cases come from the “Google button” on Reliance Jio phones (a very basic phone model in India that comes with a 4G or 5G plan at extraordinarily low prices) — the Assistant fields searches looking for music in languages like Haryanvi or Bhojpuri that are severely under-represented on the Web relative to their number of speakers, often from very underdeveloped rural parts of the world. (And, of course, YouTube has the right results!)
One possible reason for this multiple-personality disorder in the Google Assistant is that it wasn’t conceived as a voice interface to Google — rather, it was built (consciously or sub-consciously, I don’t know) as a competitor to inferior products like Alexa or Siri. I say “inferior” — even though Alexa and Siri are fairly good products for what they are designed for — for a simple reason: neither one has Google Search as an asset to turn to in their tech stack. Google’s inability in taking advantage of this asset boggles my mind.
In particular, lost in all the drama around what the Google Assistant is, and what it can be, is the golden opportunity for Google to showcase the primacy of three enormously valuable elements of user experience in Search:
communicating through natural language;
being stateful; and
allowing access to personal “data structures” (my watchlist, my calendar, my contacts…).
All of these will become fundamental to the Search experience, given the possibilities that ChatGPT has hinted at — we’ll continue to explore those possibilities and the challenges underlying them in the next articles in this series. Stay tuned!
More to come:
Lens 3: Trust.
Lens 4: Antitrust.
Lens 5: The challenge ahead: Strings and Things, Take 2.
Lens 6: Commercial searches and monetization.