Get more actionable data from your website.
How does IMWT-Bot work?
IMWT-Bot works by scraping pages in the order they’re encountered. After scraping your home page it will identify links to other pages of your site and crawl the first page it sees, then the second page from the home page and so on. This is known as a “breadth first” traversal.
Unlike Google Bot which will crawl the most important pages of your site more than other pages, IMWT-Bot will only crawl each page once – no matter how many links that page receives.
Unless explicitly told otherwise IMWT-Bot will also assume that URLs with query strings (e.g. ?p1=imwt&p2=bot) canonicalise to the query-stringless version of the page. The intent is to cut down on unnecessary crawling and save us time and your website resources.
You can identify crawls from IMWT-Bot by checking the user agent of requests. IMWT-Bot crawls with the following user-agent:
Mozilla/5.0 (compatible; IMWT-Bot/1.0; +https://inmarketingwetrust.com.au/imwt-bot/)
Although we don’t crawl from a pre-defined set of IP addresses it’s likely we will have been in contact with your traffic control team in order to have one or more IPs whitelisted. If you’re seeing unexpected traffic coming from something identifying itself as IMWT-Bot please get in touch.
IMWT-Bot plays nicely with your site
IMWT-Bot was built to crawl huge sites. That means it can make thousands of requests per second. We realise, however, that it may be detrimental to your site’s performance for it to be crawled this heavily.
To account for this we have a number of configuration options which we use to ensure we don’t disrupt your website’s capacity to serve human traffic.
The simplest of these is crawl rate – before we begin a crawl we will discuss with your team to understand how many pages per second your website can support us crawling.
If you are using auto-scaling we also support setting a ramp up time on our crawl rates. This means that if we’re going to crawl at 200 pages per second we can configure the crawler to ramp up to this speed over a given time period, meaning we’re not unexpectedly and suddenly hammering your servers.
How many web pages can IMWT-Bot crawl?
IMWT-Bot has the capacity to crawl thousands of pages per second and store detailed information about the content and links found on those pages. Our toolset allows us to work efficiently on the enormous datasets that are produced by crawling at this scale.
See the Results: Expedia
Machine learning solution for internal link structure on 18m pages site.
The team at In Marketing We Trust, led by the very knowledgeable Frederic, has been working with Expedia since 2014 and has helped us uncover and fixed a number of on site issues across our 18 million page website. Their deep understanding of complex sites’ issues and pragmatic approach to enterprise SEO in an autonomous fashion helped me deliver results while allowing me to focus on some other matters.
I would highly recommend his skills & the team at In Marketing We Trust to anyone looking to succeed in Digital, and Search in particular.Julien PSEO Manager, Expedia
Trusted by Leading Brands
50% of the most valuable 12 companies in the world trust us to get faster results.
Get A Free Consultation
Speak to us today to find out how IMWT-Bot can improve your marketing performance and receive a response from our Commercial Director within 24 hours.
Always willing to help and offer advice and has even offered fantastic training. We don’t feel like a client, we feel that we work together as they are passionate about what they do in order for us to achieve our business goals. IMWT staff is highly skilled and are experts in their field.
I especially appreciate that IMWT are so proactive in ensuring that we are kept up to date with reporting and have been a major help with migration. One of the best agencies we’ve worked with. Cecilia is wonderful! Very easy to bounce ideas off, will offer advice and is not afraid to give constructive criticism which we need at times. We’ve been low in staff numbers and when we express this to Cecilia, she has always been quick to offer help from their end – pulls in resources from everywhere to ensure we meet our deadlines.Rachelle DDigital Marketing Coordinator, Globus Family of Brands
We engaged IMWT a few months ago after considering a few different digital agencies. They stood out from the crowd as they didn’t just tell us how we should spend our marketing budget but more importantly what the return would be. IMWT provided us with an impressive proposal that included a detailed forecast of the results we could expect and so far they’ve exceeded this!
We’ve really enjoyed working with the team of digital experts at IMWT who’ve gone above and beyond to help us achieve our objectives.Lucy BHead of Marketing, Geo Workforce Solutions
I worked with IMWT on a number of very diverse projects and was impressed by the variety of skills and expertise IMWT brought to the table. IMWT really took the time to understand our business and our needs, and delivered some highly effective solutions as a result.Antje LMarketing Manager International, Columbus Direct