Crawling

OpenAI is reportedly scaling up its crawling infrastructure for the holiday shopping season. The folks at Vercel noticed OpenAI adding a lot of new IP ranges for its bots and…

Recently, Google said that no AI system is currently using the LLMS.txt file. But maybe some are starting to? OpenAI may be starting to discover and crawl LLMS.txt files on…

Gary Illyes from Google described how search engine crawlers have changed over the years. This came up in the latest Search Off the Record podcast with Martin Splitt and Gary…

Last April 2024, Gary Illyes from Google said he was on a mission to make web crawling more efficient, he wanted to “figure out how to crawl even less, and…

Some sites, hosted on some CDNs (content delivery networks), are experiencing a big spike in server response times for crawling, while seeing a drop in total crawl requests. So technically,…

Google’s John Mueller said that there is no persistent shortcut to faster crawling. Yes, you can expedite crawling for specific situations and times, but there is no way to just…

Google has slightly updated its Google crawlers and fetchers documentation to say that it will pick the protocol, HTTP/1.1 and HTTP/2, that “provides the best crawling performance” for Googlebot. In…

Google’s Gary Illyes and John Mueller, along with Bing’s Fabrice Canel and probably other representatives from these search engines spent time in Dublin this week to attend the IETF 121…

Google posted a new “Search Off The Record” podcast yesterday on the topic of crawling where John Mueller, Lizzi Sassman, and Gary Illyes spoke about how Google crawls, some ideas…

Google’s Gary Illyes posted on LinkedIn with two common examples of when a spike in Googlebot activity, crawling, is a bad thing. The short answer is when Googlebot gets to…

Google posted a public service announcement saying you should disallow Googlebot from crawling your action URLs. Gary Illyes from Google posted on LinkedIn, “You should really disallow crawling of your…

When it comes to crawl budget and Google not crawling your site too much or too little, Google takes into account all Googlebot activity across all verticals. So that includes…

Google’s John Mueller said on Reddit that disallowing URLs with UTM parameters in them won’t help you to improve crawling or rating with Google Search. He added that a site…

Google has updated its favicon search developer documentation to remove the section for the Google Favicon user agent and to clarify that if you want Google to show your favicon…

Google’s John Mueller said again that its quality updates, like core updates and the others in the family, can not just impact the ranking of a page in Google Search…

In 2018, John Mueller of Google said Google does not use cache-control headers when crawling. He said then that the has no impact on GoogleBot and how it crawls your…

I always wondered why Google had status dashboards for hundreds of other products across Gmail, Drive, Google Ads, etc., but not for Google Search – its most important product/service. But…

John Mueller of Google said in a post on Reddit that disallowing Googlebot to crawl your site would not immediately lead to the site being deindexed. He added that “it…

Google’s John Mueller confirmed that GoogleBot will crawl and pick up on URL patterns that simply do not work on your site. I mean, we have all seen this happen…

At the Google NYC SEO Meetup Lily Ray mentioned that when it comes to interviewing new employees for the Amsive Digital SEO team, she asks a question you must get…

The Google algorithm ranking tracking tools seems to be having fun, or maybe they are off, I am seeing mixed signals of Google ranking updates. Google is testing a way…

Last week, we saw reports that some Shopify sites were showing massive declines in crawling activity from Google’s Googlebot. John Mueller of Google responded this morning that this was “temporary…

In this show we got everything for you from creepy crawling to blood sucking vampires and and walking zombies. First, we had a few unconfirmed Google search algorithm updates this…

Google’s Gary Illyes said on the last Search Off The Record Podcast that Google in 2022 is looking to make crawling more efficient and environmentally friendly. And while Google is…

Google’s John Mueller said again that a spike in crawling activity on your site is unrelated to an upcoming search ranking algorithm update. John Mueller said on Twitter “it’s unrelated”…

It seems like Google is having a major indexing or crawling issue this morning – again. Google seems to have not picked up on new content in the past hour…

Google’s John Mueller confirmed that Googlebot is not yet crawling over HTTP/3 yet. He said if you do implement HTTP/3 for your site, it doesn’t mean it won’t benefit your…

Happy Thanksgiving – we had a surprise from Google again, where the second wave of the Google November 2021 core update happened on Wednesday and Thursday – yes, the busiest…

Google’s John Mueller confirmed Google had a crawling issue – specifically saying “crawling for the caches had slowed down for some sites.” Google said it was fixed “a while” ago…

Olivier Papon from Seolyzer, a log analysis toolset, reported the other week that Google seemed to slow or stop crawling most of the web between around November 11 and November…

Sites hosted on SiteGround last week found themselves not being crawled by Googlebot, Google’s crawler. The issue seemed to have been an DNS issue between the provider’s partners (AWS) and…

Google has a legacy Search Console feature to let you limit, or slow down, how fast Google can crawl your site under the crawl rate setting. This feature, as we…