For our last Data-Driven Digital community webinar, we spoke about AI-powered SEO with Botify and automation in prevention, execution and implementation at scale with Agata Adamczak, SEO Specialist and client partner director at Botify.
In this joint webinar with Botify and In Marketing We Trust learn how to automate common and complex SEO tasks to free up your time and headspace.
Historically, SEOs have had to operate extensive spreadsheets to build reports and analyse data and conduct manual checks to stay on top of their website’s technical health. With most time spent on reporting and audits that left little time for action and strategy.
In this webinar, you’ll learn:
- How to crawl (including JavaScript) at scale
- How to use Botify to determine what tasks will drive the biggest impact
- How to automate technical recommendations
- How to protect your website and resolve issues before you’re impacted in rankings, useability or performance
- How to automate keyword performance reporting at scale
- And much more
Plus learn how In Marketing We Trust puts it all into practice on enterprise websites with millions of pages.
AI-Powered SEO with Botify: Automation in Prevention, Execution, and Implementation at Scale
What is Botify?
Botify is the enterprise SEO platform that unlocks revenue potential for your website by serving as the interface that automatically connects search engines and websites.
The Power of SEO
“Every single day, everyone touching the website is doing SEO, whether they know it or not. They are either making multi-million dollar decisions that help SEO or hurt it.”
– Jessica Bowman, SEO Guru
14% of enterprises are categorised as “Optimisers,” investing in SEO across people, process, and technology… and driving revenue!
- ⅔ work closely with their dev teams
- ½ + have a company-wide SEO strategy
- 75% believe SEO is the most effective revenue generation strategy
- 63% have 10 or more people on the SEO team
Quick Facts:
- According to Conversion Guru, 90% of searchers do not go past the first page of search results. (If you’re not there, your competitors are).
- Higher Visibility found out that 80% of users ignore paid ads, focusing only on organic results (paid media has diminishing returns).
- Lastly, Google says there are more than 3,200 search engine algorithm updates each year. This means that It’s impossible to predict and uncover the impact on your site without the proper data infrastructure.
SEO Challenges
1st SEO sin:
Your organic search performance depends on Google’s algorithm and search result updates.
Make sure to be a proactive SEO. Monitor your top technical KPIs across the site and always know what is going on.
2nd SEO sin:
You practice SEO without any visibility into Google’s activities on your website.
Keep in mind to monitor how the changes you’re making to the site impact your performance and KPIs. Understand how Google is reacting to your updates and new content.
3rd SEO sin:
You use free or cheap tools and mix data sets from sources of different methodologies.
Use first-party data that talks to each other in one environment to provide the most accurate and relevant insights & recommendations.
SEO Automation with Botify
Large sites struggle to scale
On an average website, 77% of pages are not getting any traffic from search engines. In other words, only 23% of your website is working towards driving conversions from search traffic.
Based on Botify’s analysis of 6.2B Googlebot requests across 413M web pages in June 2018 “How Does Google Crawl the Web?”
How can this be true?
It’s impossible to know this about your site, and why, if you don’t have all your data (crawls, log files, SERP performance, website analytics) in a single location, over time.
Do you have the data you need to make important decisions?
What is causing all this?
Simple, the lack of time and manpower.
- Millions of URLs
- Multiple site versions
- Thousands of metrics
- Lots of data. Limited resources.
- Limited budget
Here’s where Botify comes in handy
SEO automation tool Botify can:
- Crawl – enable search engine bots to find all webpages, not just some of them
- Render – optimise page rendering from JavaScript to HTML, so search engine crawls everything on pages
- Index – get your pages indexed on the search engines. They will officially show up now in the search result
- Rank – URLs need to rank against strategic keywords and topics
- Convert – once the URL ranks in SERP, it drives traffic to the site and converts to a sale
Botify’s solution is built the way search works
With the way Botify’s solution is built and by knowing where the opportunities lie, you can maximise your site potential, and the revenue it delivers.
Why do SEOs value Botify?
- Comprehensive technical analysis – crawl data enhanced by log files and crawl budget analysis
- Availability of data – unlimited crawls at scale
- Keyword performance and analysis – cross-analysis of keywords and crawl data
- Segmentation – ability to analyse data by segment
- Automation – AI-powered recommendations and always-on alert system
Botify Customer Lifecycle
Site Migration
- Enterprise-level crawl capability
- Daily logfiles analysis
- Always on SEO issues alert system
Crawl Budget Optimisation
- Visibility into how Google crawls the website
- A list of orphan pages
- HTML errors report
Content Quality Boost
- Visibility into a sitewide content structure
- Content v template ratio
- Sitewide content quality and optimisation analysis
Internal Linking Optimisation
- Inlinks status codes report
- Canonical links not pointing to self-report
- Internal Pagerank by Segment report
Generic Rankings Growth
- Access to all the ranking keywords
- Keyword groups analysis
- Daily keyword ranking data
What’s next for Botify in 2022
- Core web vitals dashboard – not available anywhere else, trended Core Web Vitals metrics by segment
- Crawl on crawl trended reports – major revamp of the core Botify data experience. Monitor your technical KPIs over time
- Third-party data integration – integrate your competitive and backlinks data
- ROI calculations for the SEO recommendations – let Botify do the math and provide scenarios for each of the recommendations
- Custom reports storytelling – enhanced reporting experience allowing for further customisation of the reports
Case Studies
Global sportswear brand
The brand partnered with Botify in 2018 to gain visibility into their technical health and understand how Google was crawling the website. At the time, the site was comprised of 265M URLs and the SEO team knew reducing site bloat would be key to success. In the first crawls, Botify provided the team with important new learnings about organic search performance such as:
- Google was not crawling ~90% of the global sportswear website,
- 75% of URLs were non-indexable and
- Only 1% of the global sportswear’s URLs were driving search traffic
Maximising crawl budget
Reduced site bloat has led to Google crawling more (80%) of the global sportswear website.
Crawl Ratio: The percentage of URLs in the website structure that have been crawled by Google.
Increased indexable URLs
A smaller site and higher crawl rates mean more of the website is able to appear in search results.
Indexable URLs: URLs that have an opportunity to be indexed by search engines, and drive traffic from organic traffic from search results.
Increased URLs driving traffic
A smaller site, higher crawl rates, and more indexable URLs have led to a higher number of pages receiving organic traffic.
Active pages: pages that have received at least one organic visit over a specific period of time.
Global beauty brand
The brand is a global leader in the beauty industry with 14 e-commerce sites and 56 brands. The brand offers prolific product offerings with new launches each year and invests a lot in content creation across all media. Main concerns include:
- an impossible to manage the sheer scale of domains and their technical performance,
- reliant on paid channels as a quick fix for visibility and
- most of the content is not driving any new visitors
Beauty Brand #1
Result: 45% of the known pages for Beauty Brand #1 crawled by Google are indexable (+ 22 points)
Beauty Brand #2
Result: 50% of the known pages for Beauty Brand #2 crawled by Google are indexable (+ 12 points)
Artificial Intelligence in action
Recommendations for improvements generated using Machine Learning:
- 25 recommendations generated through machine learning
- 16 actions taken by Botify SpeedWorkers
- 3 actions taken by the brand
140 possible optimisation recommendations analysed each time
Global technical SEO dashboard
Consolidated view with a single destination for all organic traffic optimisation for all the brands
2022 SEO resolution: AUTOMATE
- Stand out from the crowd
- Gain visibility
- Automate data analysis
- Automate reporting
- Make your life easier
How In Marketing We Trust use Botify
Identify quick wins and tidy up work
Having well targeted internal links is pointless if the site is broken.
The Botify Action Board tells us where internal linking needs to be tightened up before we begin to work on targeting links better.
Example: Cutting Down Huge Sites
Discoveries:
- 280M pages known by Google (that’s 11 pages per every human in Australia).
- With only 100M crawled, and less than 1% of the pages crawled producing traffic in a given period.
Actions:
- In Marketing We Trust dropped 60% of the ‘crap’ pages for Google.
- Increased ‘the good URLs’ by 5M by redirecting crawl resources.
Results:
- 47% increase in traffic.
- Added 1.6M non-branded visits per month (view the case study).
Identifying and fixing these issues gives us a solid foundation to build fun stuff on.
Post hoc selection of test and baseline groups after launching experiments
You should always design your experiments well before you launch them.
But sometimes things go awry…
Or sometimes you find yourself in the midst of an accidental test.
“One sometimes finds, what one is not looking for. When I woke up just after dawn on September 28, 1928, I certainly didn’t plan to revolutionize all medicine by discovering the world’s first antibiotic, or bacteria killer. But I suppose that was exactly what I did.”
— Alexander Fleming
In Marketing We Trust have experienced both of these scenarios. Botify to the rescue!
Splitting test/baseline groups for internal linking
We can then run causal impact analyses on these results.
Data supply for machine learning models
Problem: Can we identify the pages which will respond best to receiving more internal links?
In Marketing We Trust Approach: Train a machine learning model to predict if a page will rank in the top 10 search results for a given keyword.
Then: Test if the prediction changes if we test the page with different page ranks and internal link counts.
With the help of Botify, In Marketing We Trust is able to provide huge dumps of data – both for on-page and internal links.
- Full URL
- No. of Internal Inlinks (From Distinct URLs)
- Inlinks Percentile
- No. of Incoming Redirects
- Internal Pagerank
- Internal Pagerank Position
- LCP: Agg
- FID: Agg
- CLS: Agg
- No. of Similar Indexable Pages (Score ≥ 75%)
- No. of Words (Total)
- …And 240 more!
In Marketing We Trust then combined this with data from other sources.
The model was then trained with this data and was quite successful in predicting page performance. And we used that to plan link distribution.
