How Google Makes Improvements to Its Search Algorithm

Rajan Patel: The Google search algorithm is made up of several hundred signals that we try to put together to serve the best results for each user. Amit Singhal: Just last year, we launched over 500 changes to our algorithm. So, by some count, we change our algorithm almost every day or almost twice over. Scott Huffman: We really analyze each potential change very deeply to try to make sure that it's the right thing for users. Mark Paskin: The first step in improving Google Search is coming up with an idea. Scott Huffman: There are almost always a set of motivating searches and these searches are not performing as well as we would like. Ranking engineers then come up with a hypothesis about what signal, what data, could we integrate into our algorithm. Amit Singhal: We test all these reasonable ideas through rigorous scientific testing. Mark Paskin: The first is with raters. These are external people that have been trained to judge whether one ranking is more relevant and higher quality than another. Rajan Patel: We showed these raters side-by-side for queries that the engineer's experiment might be affecting. We also confirm these changes with live experiments on real users. Mark Paskin: And we do this in something that's called a Sandbox. We send a very small fraction of actual Google traffic to the Sandbox. We compute lots of different metrics. Scott Huffman: In 2010, we ran over 20,000 different experiments. All the data from the human evaluation and the live experiment are then rolled out by a search analyst. Sangeeta Das: For each project, it's usually one analyst assigned from the moment that we're talking to the engineers trying to learn about their change. And the impact is quite small, as you see. Mark Paskin: We then have a launch decision meeting where the leadership of the search team then looks at that data and makes a decision. Amit Singhal: That surely we should fix. Sangeeta Das: Ultimately, the goal of the search eval analyst team is just to provide an informed, data-driven decision and present an unbiased view. Amit Singhal: OK? So, not approved. The team will understand what's happening. If our scientific testing says this is a good idea for Google users, we will launch it on Google. Mark Paskin: For many years now, Google has been offering spelling suggestions for queries that contained typos or misspellings. So, sometimes you'll type a query and you might see, "Did you mean," and then an alternate query. If you type a misspelling of your medicine and you don't click on the "Did you mean" you might be getting results that contain that misspelling. And they tend not to be high-quality results. So, we thought about a different kind of interface that we call "full page replacement." And instead of "Did you mean," you'll see at the top of your page, "Showing results for." And in the case that we made a mistake, there's another link, "Search instead for" and it has the query that you typed. We call that link the "escape hatch." For every time a user had to click that escape hatch because the spelling algorithm made a mistake, we wanted to make sure that there were 50 other times that they got the right spelling suggestion and they didn't have to click the "Did you mean." And they were also looking to see in the live traffic data how often were users clicking on that escape hatch to make sure that the user signal that we get from live experiments was lining up with the signal that we get from our regular evaluations. We brought it to Launch Committee and based on the rater evaluations and the live experiments, it was pretty clear that engineers had done what they were supposed to do. And so we launched it. Amit Singhal: When you align Google's interest with user's interest as we have aligned, good things happen. Mark Paskin: We've put a huge investment into understanding what works for users. Rajan Patel: Is this change gonna help users not only in the United States or in English, but all over the world? Scott Huffman: I think we get excited when we feel like we've hit on an idea that really helps a lot of users. Amit Singhal: Users keep coming back to Google even though they have a choice of a search engine every time they open a browser.