AI-Powered SEO with Botify: Automation in Prevention, Execution, and Implementation at Scale – Webinar Recap

by | May 3, 2022

AI-Powered SEO with Botify Automation in Prevention, Execution, and Implementation at Scale - Webinar Recap
15 min read

For our last Data-Driven Digital community webinar, we spoke about AI-powered SEO with Botify and automation in prevention, execution and implementation at scale with Agata Adamczak, SEO Specialist and client partner director at Botify.

In this joint webinar with Botify and In Marketing We Trust learn how to automate common and complex SEO tasks to free up your time and headspace.

Historically, SEOs have had to operate extensive spreadsheets to build reports and analyse data and conduct manual checks to stay on top of their website’s technical health. With most time spent on reporting and audits that left little time for action and strategy.

In this webinar, you’ll learn:

  • How to crawl (including JavaScript) at scale
  • How to use Botify to determine what tasks will drive the biggest impact
  • How to automate technical recommendations
  • How to protect your website and resolve issues before you’re impacted in rankings, useability or performance
  • How to automate keyword performance reporting at scale
  • And much more

Plus learn how In Marketing We Trust puts it all into practice on enterprise websites with millions of pages.

AI-Powered SEO with Botify: Automation in Prevention, Execution, and Implementation at Scale

What is Botify?

Botify is the enterprise SEO platform that unlocks revenue potential for your website by serving as the interface that automatically connects search engines and websites.

The Power of SEO

“Every single day, everyone touching the website is doing SEO, whether they know it or not. They are either making multi-million dollar decisions that help SEO or hurt it.”

– Jessica Bowman, SEO Guru


14% of enterprises are categorised as “Optimisers,” investing in SEO across people, process, and technology… and driving revenue!

The Power of SEO

  • ⅔ work closely with their dev teams
  • ½ + have a company-wide SEO strategy
  • 75% believe SEO is the most effective revenue generation strategy 
  • 63% have 10 or more people on the SEO team

Quick Facts: 

  1. According to Conversion Guru, 90% of searchers do not go past the first page of search results. (If you’re not there, your competitors are).
  2. Higher Visibility found out that 80% of users ignore paid ads, focusing only on organic results (paid media has diminishing returns).
  3. Lastly, Google says there are more than 3,200 search engine algorithm updates each year. This means that It’s impossible to predict and uncover the impact on your site without the proper data infrastructure.

SEO Challenges

1st SEO sin: 

Your organic search performance depends on Google’s algorithm and search result updates.

When my organic traffic increases but I did absolutely nothing. A Blessing from the Lord. SEO meme

Make sure to be a proactive SEO. Monitor your top technical KPIs across the site and always know what is going on.

2nd SEO sin: 

You practice SEO without any visibility into Google’s activities on your website.

Keep in mind to monitor how the changes you’re making to the site impact your performance and KPIs. Understand how Google is reacting to your updates and new content.

3rd SEO sin: 

You use free or cheap tools and mix data sets from sources of different methodologies.

Waiting for the rankings from my $100 SEO package - SEO Meme

Use first-party data that talks to each other in one environment to provide the most accurate and relevant insights & recommendations.

SEO Automation with Botify

Large sites struggle to scale

On an average website, 77% of pages are not getting any traffic from search engines. In other words, only 23% of your website is working towards driving conversions from search traffic.

Large Sites Struggle to Scale - AI-Powered SEO with Botify

Based on Botify’s analysis of 6.2B Googlebot requests across 413M web pages in June 2018 “How Does Google Crawl the Web?

How can this be true?

It’s impossible to know this about your site, and why, if you don’t have all your data (crawls, log files, SERP performance, website analytics) in a single location, over time. 

Do you have the data you need to make important decisions?

What is causing all this?

How do you web crawler - seo meme

Simple, the lack of time and manpower.

  • Millions of URLs 
  • Multiple site versions
  • Thousands of metrics
  • Lots of data. Limited resources.
  • Limited budget

Here’s where Botify comes in handy

SEO automation tool Botify can:

What does Botify do? SEO Automation

  • Crawl – enable search engine bots to find all webpages, not just some of them
  • Render – optimise page rendering from JavaScript to HTML, so search engine crawls everything on pages
  • Index – get your pages indexed on the search engines. They will officially show up now in the search result
  • Rank – URLs need to rank against strategic keywords and topics
  • Convert – once the URL ranks in SERP, it drives traffic to the site and converts to a sale

Botify’s solution is built the way search works

Botify’s solution is built the way search works

With the way Botify’s solution is built and by knowing where the opportunities lie, you can maximise your site potential, and the revenue it delivers. 

Why do SEOs value Botify?

Why do SEOs value Botify?

  1. Comprehensive technical analysis – crawl data enhanced by log files and crawl budget analysis
  2. Availability of data – unlimited crawls at scale
  3. Keyword performance and analysis – cross-analysis of keywords and crawl data
  4. Segmentation – ability to analyse data by segment
  5. Automation – AI-powered recommendations and always-on alert system

Botify Customer Lifecycle

Botify Customer Lifecycle

Site Migration

  • Enterprise-level crawl capability
  • Daily logfiles analysis
  • Always on SEO issues alert system

Crawl Budget Optimisation

  • Visibility into how Google crawls the website 
  • A list of orphan pages
  • HTML errors report

Content Quality Boost

  • Visibility into a sitewide content structure
  • Content v template ratio
  • Sitewide content quality and optimisation analysis

Internal Linking Optimisation

  • Inlinks status codes report
  • Canonical links not pointing to self-report
  • Internal Pagerank by Segment report 

Generic Rankings Growth

  • Access to all the ranking keywords
  • Keyword groups analysis
  • Daily keyword ranking data

What’s next for Botify in 2022

What's coming to Botify in 2022

  1. Core web vitals dashboard – not available anywhere else, trended Core Web Vitals metrics by segment
  2. Crawl on crawl trended reports – major revamp of the core Botify data experience. Monitor your technical KPIs over time
  3. Third-party data integration – integrate your competitive and backlinks data
  4. ROI calculations for the SEO recommendations – let Botify do the math and provide scenarios for each of the recommendations
  5. Custom reports storytelling – enhanced reporting experience allowing for further customisation of the reports

Case Studies

Global sportswear brand

The brand partnered with Botify in 2018 to gain visibility into their technical health and understand how Google was crawling the website. At the time, the site was comprised of 265M URLs and the SEO team knew reducing site bloat would be key to success. In the first crawls, Botify provided the team with important new learnings about organic search performance such as:

  1. Google was not crawling ~90% of the global sportswear website,
  2. 75% of URLs were non-indexable and
  3. Only 1% of the global sportswear’s URLs were driving search traffic

Maximising crawl budget

Reduced site bloat has led to Google crawling more (80%) of the global sportswear website.

Botify Case Studies - Global Sportwear Brand - Maximising Crawl Budget

Crawl Ratio: The percentage of URLs in the website structure that have been crawled by Google.

Increased indexable URLs

A smaller site and higher crawl rates mean more of the website is able to appear in search results.

Increase in indexable URLs - Botify Case Study - Global Sportswear Brand

Indexable URLs: URLs that have an opportunity to be indexed by search engines, and drive traffic from organic traffic from search results.

Increased URLs driving traffic

A smaller site, higher crawl rates, and more indexable URLs have led to a higher number of pages receiving organic traffic.

Increased URLs driving traffic - Botify Case Study - Global Sportswear Brand

Active pages: pages that have received at least one organic visit over a specific period of time. 

Global beauty brand

The brand is a global leader in the beauty industry with 14 e-commerce sites and 56 brands. The brand offers prolific product offerings with new launches each year and invests a lot in content creation across all media. Main concerns include:

  1. an impossible to manage the sheer scale of domains and their technical performance,
  2. reliant on paid channels as a quick fix for visibility and
  3. most of the content is not driving any new visitors

Beauty Brand #1

Botify Case Study - Beauty Brand 1Result: 45% of the known pages for Beauty Brand #1 crawled by Google are indexable (+ 22 points)

Beauty Brand #2

Botify Case Study - Beauty Brand 2

Result: 50% of the known pages for Beauty Brand #2 crawled by Google are indexable (+ 12 points)

Artificial Intelligence in action

Recommendations for improvements generated using Machine Learning:

Artificial Intelligence in Action - Botify Intelligence ActionBoard

  • 25 recommendations generated through machine learning
  • 16 actions taken by Botify SpeedWorkers
  • 3 actions taken by the brand

140 possible optimisation recommendations analysed each time 

Global technical SEO dashboard

Consolidated view with a single destination for all organic traffic optimisation for all the brands

Botify Global Technical SEO Dashboard

2022 SEO resolution: AUTOMATE

2022 SEO Resolution AUTOMATE


  1. Stand out from the crowd
  2. Gain visibility
  3. Automate data analysis
  4. Automate reporting
  5. Make your life easier

How In Marketing We Trust use Botify

Identify quick wins and tidy up work

Having well targeted internal links is pointless if the site is broken.

The Botify Action Board tells us where internal linking needs to be tightened up before we begin to work on targeting links better.

Botify ActionBoard

Example: Cutting Down Huge Sites


  • 280M pages known by Google (that’s 11 pages per every human in Australia).
  • With only 100M crawled, and less than 1% of the pages crawled producing traffic in a given period.


  • In Marketing We Trust dropped 60% of the ‘crap’ pages for Google.
  • Increased ‘the good URLs’ by 5M by redirecting crawl resources.


Identifying and fixing these issues gives us a solid foundation to build fun stuff on.

Post hoc selection of test and baseline groups after launching experiments

You should always design your experiments well before you launch them.

Design your experiments first


But sometimes things go awry…

Implementation Plan and Measurement Plan

Or sometimes you find yourself in the midst of an accidental test.

One sometimes finds, what one is not looking for. When I woke up just after dawn on September 28, 1928, I certainly didn’t plan to revolutionize all medicine by discovering the world’s first antibiotic, or bacteria killer. But I suppose that was exactly what I did.

— Alexander Fleming


In Marketing We Trust have experienced both of these scenarios. Botify to the rescue!

Botify to the rescue - AI-Powered SEO with Botify

Splitting test/baseline groups for internal linking

Splitting test/baseline groups for internal linking

We can then run causal impact analyses on these results.

Causal impact analysis

Data supply for machine learning models

Problem: Can we identify the pages which will respond best to receiving more internal links?

In Marketing We Trust Data and Engineering Team - AI-Powered SEO with Botify

In Marketing We Trust Approach: Train a machine learning model to predict if a page will rank in the top 10 search results for a given keyword.

A bunny vs machine learning algorithm

Then: Test if the prediction changes if we test the page with different page ranks and internal link counts.

Internal Linking Simulator

With the help of Botify, In Marketing We Trust is able to provide huge dumps of data – both for on-page and internal links.

  • Full URL
  • No. of Internal Inlinks (From Distinct URLs)
  • Inlinks Percentile
  • No. of Incoming Redirects
  • Internal Pagerank
  • Internal Pagerank Position
  • LCP: Agg
  • FID: Agg
  • CLS: Agg
  • No. of Similar Indexable Pages (Score ≥ 75%)
  • No. of Words (Total)
  • …And 240 more!

In Marketing We Trust then combined this with data from other sources.

Combined data

The model was then trained with this data and was quite successful in predicting page performance.  And we used that to plan link distribution.

Internal Linking Optimisation Simulator

Kirsten Tanner

Recommended for you

Get Our Newsletter

Sign up for our newsletter and receive monthly updates on what we’ve been up to, digital marketing news and more.

Your personal information will not be shared, and we don’t like mail spam or pushy salesmen either!