Amazon’s Secret Ranking Algorithm Exposed: How I Tricked an A.I. into Revealing the Hidden Truth!

Amazon’s A9 algorithm is easily one of the most fascinating and inquired about topics in Amazon seller circles. People who seem to understand its inner workings are almost regarded as wizards or secret geniuses in the space.

Much work has been done to disambiguate this topic, particularly the dissection of Daria Sorokina’s The Joy of Ranking Products paper and presentation, which the likes of Danny McMillan and Anthony Lee provided many insights on.

And every year, with the publishing of a new paper on Amazon.science or elsewhere, whether it be about Amazon’s inventory algorithm, or the Bestseller rank, interest in A9 and keyword relevance spikes again. 

Experts from all corners of Amazon typically weigh in on the topic, however this year things are a bit…different.

(Get the Ultimate Guide to ChatGPT for Amazon Sellers Here ==> signalytics.ai/chatgpt)

ChatGPT: The “Clippy” of the 21st Century

Enter Artificial Intelligence. Nowadays A.I. is accessible to everyone, and as such, it allows us all to consult with the smartest person in the room.

A.I. may not be able to do everything, but one of its strengths is analyzing data. So we decided to feed GPT (through both Chat GPT-3.5, Bing Chat, and GPT-4) the science and data available and see if it could glean any insights our limited human minds may have missed.

Our first attempt was with GPT-3.5 through the ChatGPT interface, but unfortunately that didn’t work as there was too much text. Between The Joy of Ranking Products paper and the transcript from Daria’s presentation, we only got an error that we were exceeding token limits.

Thankfully we had access to Bing Chat, so that was our next attempt. Since Bing has access to the internet, we only had to reference the paper as its text is indexed. However, through a painstaking process we managed to feed the transcript from the video presentation to Bing.

After Bing had consumed the information, the responses from the A.I. were very interesting.

Artificial Intelligence Claims We Are Trying Too Hard

We asked for summaries (which we will share later), as well as insights. Interestingly, when we asked for the hidden insights, things started turning in an unexpected direction.

We asked Bing to reveal the complex hidden secrets of the information that may be beyond most people reading. Then we asked it to take that information and give us actionable steps that a seller could take to rank above competitors.

Its response was…to put it plainly…vanilla.

Bing told us that to gain a competitive advantage on our competitors we should:

• Optimize our listing.
• Choose relevant keywords.
• Choose the proper category.
• Obtain positive reviews.

Essentially…the fundamentals

Confused, we then asked it if having this knowledge could give sellers an edge. Bing candidly responded that knowing this information about hunger scores, in-category ranking, cold starts, etc. would likely not be of any benefit at all.

The reason is because knowing those things wouldn’t change the actions necessary to gain relevance and rank.

🤯🤯🤯

That wasn’t the answer we’d expected. However, we can understand that point of view, and honestly it makes sense. We sellers can often suffer “shiny object syndrome” and the moment we hear highly technical information about Amazon’s secret algorithms we voraciously consume it. But ultimately, this knowledge is about internal metrics that we likely have no control over.

HOWEVER, we would not be deterred.

And then, our prayers were answered…

(Get the Ultimate Guide to ChatGPT for Amazon Sellers Here ==> signalytics.ai/chatgpt)

Then GPT-4 Was Released

The moment we received the email about GPT-4 we went to a computer and requested API access. While GPT-4 was available through ChatGPT to Plus subscribers, we knew that character limitations would likely still be an issue.

Through the Playground, however, we quickly got access to GPT-4 and over 8k token limits. Now we would be able to feed ALL of the science to GPT and hopefully get some good responses. Below are the prompts and chat completions provided by GPT-4:

System Prompt – You are an algorithm expert. You help deconstruct and explain the inner workings of complex algorithms.

Prompt – Below is the text from the paper The Joy of Ranking Products by Daria Sorokina. This paper describes the research that led to the development of Amazon’s keyword ranking algorithm and the ML algorithms that determine keyword relevance. Summarize the paper and outline the most technical and pertinent aspects of it:

*copy/pasted text from the paper*

Response – In the paper “The Joy of Ranking Products” by Daria Sorokina, the authors discuss the Amazon search algorithm and describe the various machine learning techniques applied to improve product ranking, matching queries to products, and identifying user intent.

1. Ranking Models: Gradient boosted trees are used as the main method for ranking because they can discover complex feature interactions and work well without significant tuning. Models are trained with pairwise objectives, and nDCG serves as the default objective function. They also perform feature selection in two stages and then evaluate the chosen model through A/B tests on several metrics.
2. Behavioral Features: Amazon relies heavily on behavioral features since product descriptions can be very similar. To correct for position bias (where users tend to click more often on top results), they developed a scheme that adapts the bias correction from day to day.
3. Matching Products and Queries: The authors use a probabilistic context-free grammar (PCFG) to determine product types in queries and assign them to appropriate products in the catalog. They then train a series of logistic regression models to predict multilabel classification for proper product description. These detected product types are used to create powerful features in the ranking models.
4. Finding Preferred Edition: Gradient boosted trees are also used to predict the preferred edition for each book based on query-independent features. They pre-calculate most of the score offline.
5. Ranking in Fashion: Due to the unique challenges presented by Amazon Fashion, they optimize for a fused target that combines purchases with clicks. They also identify strategic categories of fashionable customers and amplify their influence during training set design.
6. Blending Across Categories: To handle the “Search All” option, the paper discusses interleaving results from different categories based on an estimate of the probability a given query is associated with a particular category. For common queries, this probability is based on observed user actions, and for tail queries, a trigram language model is used.

Overall, the paper presents a comprehensive understanding of the machine learning techniques and algorithms that serve as the backbone for Amazon’s product search engine, with specific focus on ranking, behavioral features, matching products with queries, and dealing with specific transactional contexts such as books or fashion products.

Prompt – Below is a transcript from a presentation given by Daria Sorokina on the topic of the paper. Please summarize and highlight any new information from the transcript that adds to the details of the Joy of Ranking Products summary:

*copy/pasted transcript*

Response – In the presentation, Daria Sorokina provides an in-depth overview of the algorithms used for ranking products on Amazon, their models, feature selection, handling cold start problems, and non-relevant sort challenges. Key additional points and insights from the presentation include:

1. Amazon has over a hundred machine-learned models in their system, with each model using about 20 of the 150 available features.
2. The positive examples for training data include products that are clicked, added to cart, or purchased. For negative data points, they use negative seen (products implicitly rejected) and negative unseen (a random sample).
3. They use a mix of clicks and purchases as training targets, including in cases where customers’ intentions vary significantly.
4. Feature selection is done in two phases, with the second one involving rounds of backward elimination and forward selection.
5. They use a combination of query-category scores, hunger scores, and original relevance scores to blend results from different categories for the “Search All” option.
6. Cold start problems are of three types: day-zero phase (relying on other features), day-one phase (dealing with infrastructural delays by propagating fast changes), and fixing features for new products.
7. For non-relevant sorts, they calculate scores between the query and product type, filtering the match set by leaving only up to three of the most important product types.

After elaborating on how Amazon handles ranking products and addressing various challenges, the speaker goes on to mention the importance of continuous improvement and optimization of their models and algorithms to ensure customer satisfaction.

1. A/B Testing: Regularly conducting A/B tests helps measure the performance of different models or algorithms against each other, ensuring that the best-performing model gets deployed. This involves comparing various metrics such as click-through rate (CTR), revenue, and customer conversion rates.
2. Monitoring and Addressing Customer Feedback: The relevance team at Amazon continuously monitors customer feedback and complaints to identify potential issues with their ranking models. Adjusting the models based on this feedback helps in enhancing the user experience and minimizing customer dissatisfaction.
3. Staying Up-to-date with Market Trends: Ensuring that the ranking algorithms accurately represent customer interest in new products, trends, and seasonal items is crucial. Regularly updating models to account for any changes in consumer behavior and preferences helps maintain relevancy.
4. Collaboration with Other Teams: Working closely with teams such as Query Understanding, where they perform advanced text processing to improve query matching, is essential for improving ranking models. By sharing knowledge and best practices, different teams can deliver a more comprehensive and accurate search experience for customers.
5. Handling Diverse User Preferences: Providing personalized user experiences plays an essential role in maintaining customer satisfaction. Developing models and algorithms catered to individual customer’s preferences creates a more streamlined and customized search experience, further improving user engagement.

In conclusion, Amazon’s search team continuously optimizes and refines their ranking models and algorithms to ensure that customers receive relevant search results and an improved shopping experience. By addressing various challenges such as cold start issues, non-relevant sorts, and blending results, the team can better cater to the diverse preferences and intentions of Amazon’s customers.

Prompt – Using the summary from the paper The Joy of Ranking Products, as well as the transcript from the presentation on the same topic, please provide a list of action items a seller can take to leverage this information in order to increase rank for their products. Also point out what specific details from this science can help give a seller a competitive advantage.

Response – Based on the information provided in the paper and the presentation, sellers can take the following action items to increase the rank of their products on Amazon:

1. Optimize product listings: Ensure product titles, descriptions, and keywords are relevant and accurately describe the product. This will help the Amazon ranking algorithm match the product with appropriate user queries.
2. Focus on behavioral features: Since Amazon relies heavily on behavioral features such as clicks, purchases, and add-to-carts, encourage customers to engage with your product listing by offering incentives, discounts, or special promotions.
3. Improve product quality and selection: High customer satisfaction will lead to better reviews and ultimately a higher relevance score in Amazon’s ranking algorithm. Offer top-quality products that meet customers’ needs and expectations.
4. Monitor product type and category: Make sure your products are correctly listed in the appropriate product types and categories to increase the chances of being included in relevant search results.
5. Leverage customer reviews: Encourage satisfied customers to leave positive reviews and ratings, as these can impact the ranking of your product. For instance, sorting by “customer reviews” can bring your product to the top if it has received consistently high ratings.
6. Stay up-to-date with market trends: Regularly update your product offerings according to current market trends and demand, as Amazon’s ranking algorithms account for the popularity and demand of products.
7. Use Amazon’s A+ Content (enhanced brand content) feature: Provide high-quality images, informative videos, and engaging descriptions that accurately represent your products. This can inspire more customer interaction with your listings, improving visibility and click-through rates.

Specific details from the research that can give sellers a competitive advantage include:

1. Understanding the importance of behavioral features in Amazon’s ranking algorithm and focusing on optimizing these aspects of your product listing.
2. Recognizing the significance of matching products and queries, ensuring your products are accurately described and categorized so they appear in relevant search results.
3. Leveraging Amazon’s blending process, which combines rankings from different categories, by ensuring your products have strong query-category scores and original relevance scores.
4. Keeping track of cold start problem solutions implemented by Amazon to ensure newly launched products gain traction quickly, and making necessary adjustments to your product listings accordingly.

What Did the A.I. Teach Us?

After consulting the strongest, smartest A.I. available to the public, with the most robust input capabilities, we’ve learned some valuable insights.

First, GPT-4 reiterated what Bing Chat attempted to explain; that knowing the science behind the internal metrics of Amazon’s ranking algorithm doesn’t change the actions we all must take to gain a leg up in ranking our listings.

We learned that it all comes down to fundamentals. Listing optimization, keyword relevance, good product, good support, customer reviews. Things we’ve been told time and time again from the very first day we started selling.

And while understanding what a trigram even is is cool as heck, it won’t give us a competitive advantage without taking those fundamental steps.

A Word On the Evolution of Amazon’s A9 (….A10?)

Every year or so the seller community sets abuzz on the topic of A10 again. What is A10? Apparently the next iteration of A9.

Now, we’ve been in the space for quite some time and are unaware of any such iteration. Furthermore, there have been no significant updates to the ranking algorithm or relevance engine that fuels it since Daria’s work in 2016 (not that the algorithm hasn’t changed, but no new science or techniques has been added to the way it works).

The bulk of the development focus has been on voice search, inventory management, and other elements. Don’t believe us? Just look at Amazon.science yourself. They update all their projects there.

However, in the spirit of this article’s original aim, we decided to consult A.I, on this topic as well. We asked the big three A.I. Chat platforms “What is Amazon A10?” Here’s what they had to say:

GPT-4…

GPT trained up to 2021 had never heard of such a thing….not too shocking.

Then we have Bing Chat…

At first glance it definitely looks like Bing Chat is confirming the existence of this shift in the algorithm. Thankfully, Bing provides references. Each of the cited references are from the same three articles, none from Amazon, and none that cite sources from Amazon.

Essentially, zero credible sources. Even moreso, one of the articles outright says A10 isn’t a thing, but rather just a name sellers made up:

We also asked Google’s Bard, who gave an almost identical response to Bing, only without the references.

So, if you believe that a real shift in the algorithm warranting the change in name to A10 exists, it’s likely you’ve fallen victim to clickbait and marketing-speak. The term was made up to gain attention (and it clearly worked).

One thing we DO know; Amazon’s algorithm does change regularly, but mostly in subtle ways. Most of the changes are also for internal efficiency and may not impact sellers much. The things that do impact sellers only close loopholes that made manipulation easier (mostly).

The bulk of changes that Amazon makes is to their layout and how they gather behavioral intelligence. This helps to increase shoppers on their site making purchases. All of those actions tend to lead to MORE sales overall for sellers (without taking into account competition from new sellers joining the platform).

We think the moral of this story is…

• Stick to the fundamentals.
• Beware shiny objects.
• Don’t believe everything that you read.
• And as always, put customers first.

And you’ll increase the odds of finding success on the Amazon.com platform.

(Get the Ultimate Guide to ChatGPT for Amazon Sellers Here ==> signalytics.ai/chatgpt)