Get more actionable data from your website.
What is IMWT-Bot?
IMWT-Bot is In Marketing We Trust’s web crawler. It allows us to gather vast amounts of information about your website so we can analyse it and suggest improvements. IMWT-Bot allows us to find, analyse and resolve technical issues you may or may not be aware of.
Examples of the kind of analysis we might use IMWT-Bot for include:
- Understanding the layout of your site
- Auditing the internal links on your website to ensure the most important pages are prioritised
- Content quality and similarity checks
And much more!
How does IMWT-Bot work?
IMWT-Bot works by scraping pages in the order they’re encountered. After scraping your home page it will identify links to other pages of your site and crawl the first page it sees, then the second page from the home page and so on. This is known as a “breadth first” traversal.
Unlike Google Bot which will crawl the most important pages of your site more than other pages, IMWT-Bot will only crawl each page once – no matter how many links that page receives.
Unless explicitly told otherwise IMWT-Bot will also assume that URLs with query strings (e.g. ?p1=imwt&p2=bot) canonicalise to the query-stringless version of the page. The intent is to cut down on unnecessary crawling and save us time and your website resources.
Identifying IMWT-Bot
You can identify crawls from IMWT-Bot by checking the user agent of requests. IMWT-Bot crawls with the following user-agent:
Mozilla/5.0 (compatible; IMWT-Bot/1.0; +https://inmarketingwetrust.com.au/imwt-bot/)
Although we don’t crawl from a pre-defined set of IP addresses it’s likely we will have been in contact with your traffic control team in order to have one or more IPs whitelisted. If you’re seeing unexpected traffic coming from something identifying itself as IMWT-Bot please get in touch.
IMWT-Bot plays nicely with your site
IMWT-Bot was built to crawl huge sites. That means it can make thousands of requests per second. We realise, however, that it may be detrimental to your site’s performance for it to be crawled this heavily.
To account for this we have a number of configuration options which we use to ensure we don’t disrupt your website’s capacity to serve human traffic.
The simplest of these is crawl rate – before we begin a crawl we will discuss with your team to understand how many pages per second your website can support us crawling.
If you are using auto-scaling we also support setting a ramp up time on our crawl rates. This means that if we’re going to crawl at 200 pages per second we can configure the crawler to ramp up to this speed over a given time period, meaning we’re not unexpectedly and suddenly hammering your servers.
How many web pages can IMWT-Bot crawl?
IMWT-Bot has the capacity to crawl thousands of pages per second and store detailed information about the content and links found on those pages. Our toolset allows us to work efficiently on the enormous datasets that are produced by crawling at this scale.
See the Results: Expedia
Machine learning solution for internal link structure on 18m pages site.
Trusted by Leading Brands
50% of the most valuable 12 companies in the world trust us to get faster results.
Get A Free Consultation
Speak to us today to find out how IMWT-Bot can improve your marketing performance and receive a response from our Commercial Director within 24 hours.
