{"id":1095,"date":"2020-03-26T16:08:28","date_gmt":"2020-03-26T16:08:28","guid":{"rendered":"https:\/\/mb30.tetsua.xyz\/2020\/03\/26\/how-do-crawlers-work-what-crawlability-stands-for\/"},"modified":"2026-04-01T17:48:29","modified_gmt":"2026-04-01T17:48:29","slug":"how-do-crawlers-work-what-crawlability-stands-for","status":"publish","type":"post","link":"https:\/\/mb30.tetsua.xyz\/en\/how-do-crawlers-work-what-crawlability-stands-for\/","title":{"rendered":"How Do Crawlers Work &#038; What Crawlability Stands For"},"content":{"rendered":"<p><span style=\"font-weight: 400;\">Ranking in the search engines requires a website with flawless technical SEO. If you really want to get the most out of your website and keep on outranking the competition, some basic knowledge of technical SEO is a must. In this post, we\u2019ll talk about one of the most important concepts of technical SEO: crawlability.<\/span><\/p>\n<h2><b>#1 What is a Crawler<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">A search engine like Google consists of a crawler, an index, and an algorithm. The crawler follows the links. When Google\u2019s crawler finds your website, it\u2019ll read it and its content is saved in the index. A crawler follows the links on the web. A crawler is also called a robot, a bot, or a spider. It goes around the internet 24\/7. Once it comes to a website, it saves the HTML version of a page in a gigantic database, called the index. This index is updated every time the crawler comes around your website and finds a new or revised version of it. Depending on how important Google deems your site and the number of changes you make on your website, the crawler comes around more or less often.<\/span><\/p>\n<h2><b>#2 What Crawlability Means?<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">Crawlability has to do with the possibilities Google has to crawl your website. Crawlers can be blocked from your site. There are a few ways to block a crawler from your website. If your website or a page on your website is blocked, you\u2019re saying to Google\u2019s crawler: \u201cdo not come here\u201d. Your site or the respective page won\u2019t turn up in the search results in most of these cases.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">There are a few things that could prevent Google from crawling (or indexing) your website:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">If your robots.txt file blocks the crawler, Google will not come to your website or specific web page.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Before crawling your website, the crawler will take a look at the HTTP header of your page. This HTTP header contains a status code. If this status code says that a page doesn\u2019t exist, Google won\u2019t crawl your website. In the module about HTTP headers of our Technical SEO training, we\u2019ll tell you all about that.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">If the robots meta tag on a specific page blocks the search engine from indexing that page, Google will crawl that page, but won\u2019t add it to its index.<\/span><\/li>\n<\/ul>\n<h2><b>#3 Conclusion<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">Although crawlability is just the very basics of technical SEO (it has to do with all the things that enable Google to index your site), for most people it\u2019s already pretty advanced stuff. Nevertheless, if you\u2019re blocking \u2013 perhaps even without knowing! \u2013 crawlers from your site, you\u2019ll never rank high in Google. So, if you\u2019re serious about SEO, this should matter to you.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Also, if you need technical support with SEO for your brand new-or not so new website\u00a0 &#8211; there is always Mobiteam: The Top Web Agency from Berlin. We are happy to help anyone with SEO issues, Web Design, Web Development, and Consulting.\u00a0<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Ranking in the search engines requires a website with flawless technical SEO. If you really want to get the most out of your website and keep on outranking the competition, some basic knowledge of technical SEO is a must. In this post, we\u2019ll talk about one of the most important concepts of technical SEO: crawlability. <a href=\"https:\/\/mb30.tetsua.xyz\/en\/how-do-crawlers-work-what-crawlability-stands-for\/\" class=\"more-link\">&#8230;<span class=\"screen-reader-text\">  How Do Crawlers Work &#038; What Crawlability Stands For<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":615,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1095","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.2 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>How Do Crawlers Work &amp; What Crawlability Stands For | Mobiteam<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/mb30.tetsua.xyz\/en\/how-do-crawlers-work-what-crawlability-stands-for\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How Do Crawlers Work &amp; What Crawlability Stands For | Mobiteam\" \/>\n<meta property=\"og:description\" content=\"Ranking in the search engines requires a website with flawless technical SEO. If you really want to get the most out of your website and keep on outranking the competition, some basic knowledge of technical SEO is a must. In this post, we\u2019ll talk about one of the most important concepts of technical SEO: crawlability. ... How Do Crawlers Work &#038; What Crawlability Stands For\" \/>\n<meta property=\"og:url\" content=\"https:\/\/mb30.tetsua.xyz\/en\/how-do-crawlers-work-what-crawlability-stands-for\/\" \/>\n<meta property=\"og:site_name\" content=\"Mobiteam\" \/>\n<meta property=\"article:published_time\" content=\"2020-03-26T16:08:28+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-04-01T17:48:29+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/mb30.tetsua.xyz\/wp-content\/uploads\/2026\/04\/Mobiteam-Crawl-1020x480-1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1020\" \/>\n\t<meta property=\"og:image:height\" content=\"480\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Serhii\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Serhii\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/how-do-crawlers-work-what-crawlability-stands-for\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/how-do-crawlers-work-what-crawlability-stands-for\\\/\"},\"author\":{\"name\":\"Serhii\",\"@id\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/#\\\/schema\\\/person\\\/868c9641ec8412e6767d2477a6611cbe\"},\"headline\":\"How Do Crawlers Work &#038; What Crawlability Stands For\",\"datePublished\":\"2020-03-26T16:08:28+00:00\",\"dateModified\":\"2026-04-01T17:48:29+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/how-do-crawlers-work-what-crawlability-stands-for\\\/\"},\"wordCount\":507,\"commentCount\":0,\"image\":{\"@id\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/how-do-crawlers-work-what-crawlability-stands-for\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/wp-content\\\/uploads\\\/2026\\\/04\\\/Mobiteam-Crawl-1020x480-1.jpg\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/how-do-crawlers-work-what-crawlability-stands-for\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/how-do-crawlers-work-what-crawlability-stands-for\\\/\",\"url\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/how-do-crawlers-work-what-crawlability-stands-for\\\/\",\"name\":\"How Do Crawlers Work & What Crawlability Stands For | Mobiteam\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/how-do-crawlers-work-what-crawlability-stands-for\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/how-do-crawlers-work-what-crawlability-stands-for\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/wp-content\\\/uploads\\\/2026\\\/04\\\/Mobiteam-Crawl-1020x480-1.jpg\",\"datePublished\":\"2020-03-26T16:08:28+00:00\",\"dateModified\":\"2026-04-01T17:48:29+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/#\\\/schema\\\/person\\\/868c9641ec8412e6767d2477a6611cbe\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/how-do-crawlers-work-what-crawlability-stands-for\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/how-do-crawlers-work-what-crawlability-stands-for\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/how-do-crawlers-work-what-crawlability-stands-for\\\/#primaryimage\",\"url\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/wp-content\\\/uploads\\\/2026\\\/04\\\/Mobiteam-Crawl-1020x480-1.jpg\",\"contentUrl\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/wp-content\\\/uploads\\\/2026\\\/04\\\/Mobiteam-Crawl-1020x480-1.jpg\",\"width\":1020,\"height\":480},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/how-do-crawlers-work-what-crawlability-stands-for\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How Do Crawlers Work &#038; What Crawlability Stands For\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/#website\",\"url\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/\",\"name\":\"Mobiteam\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/#\\\/schema\\\/person\\\/868c9641ec8412e6767d2477a6611cbe\",\"name\":\"Serhii\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/22018943a6d60a783f53a4165a70fe390e6c8438bb252508e11efde3f1379cfa?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/22018943a6d60a783f53a4165a70fe390e6c8438bb252508e11efde3f1379cfa?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/22018943a6d60a783f53a4165a70fe390e6c8438bb252508e11efde3f1379cfa?s=96&d=mm&r=g\",\"caption\":\"Serhii\"},\"description\":\"I elaborate and manage IT products, startups, and groups. Drafting the product and business development, planning and executing. I\u2019m able to simplify complex issues to benefit from everyone\u2019s contribution.\",\"sameAs\":[\"https:\\\/\\\/mb30.tetsua.xyz\"],\"url\":\"https:\\\/\\\/mb30.tetsua.xyz\\\/en\\\/author\\\/serhii\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How Do Crawlers Work & What Crawlability Stands For | Mobiteam","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/mb30.tetsua.xyz\/en\/how-do-crawlers-work-what-crawlability-stands-for\/","og_locale":"en_US","og_type":"article","og_title":"How Do Crawlers Work & What Crawlability Stands For | Mobiteam","og_description":"Ranking in the search engines requires a website with flawless technical SEO. If you really want to get the most out of your website and keep on outranking the competition, some basic knowledge of technical SEO is a must. In this post, we\u2019ll talk about one of the most important concepts of technical SEO: crawlability. ... How Do Crawlers Work &#038; What Crawlability Stands For","og_url":"https:\/\/mb30.tetsua.xyz\/en\/how-do-crawlers-work-what-crawlability-stands-for\/","og_site_name":"Mobiteam","article_published_time":"2020-03-26T16:08:28+00:00","article_modified_time":"2026-04-01T17:48:29+00:00","og_image":[{"width":1020,"height":480,"url":"https:\/\/mb30.tetsua.xyz\/wp-content\/uploads\/2026\/04\/Mobiteam-Crawl-1020x480-1.jpg","type":"image\/jpeg"}],"author":"Serhii","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Serhii","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/mb30.tetsua.xyz\/en\/how-do-crawlers-work-what-crawlability-stands-for\/#article","isPartOf":{"@id":"https:\/\/mb30.tetsua.xyz\/en\/how-do-crawlers-work-what-crawlability-stands-for\/"},"author":{"name":"Serhii","@id":"https:\/\/mb30.tetsua.xyz\/en\/#\/schema\/person\/868c9641ec8412e6767d2477a6611cbe"},"headline":"How Do Crawlers Work &#038; What Crawlability Stands For","datePublished":"2020-03-26T16:08:28+00:00","dateModified":"2026-04-01T17:48:29+00:00","mainEntityOfPage":{"@id":"https:\/\/mb30.tetsua.xyz\/en\/how-do-crawlers-work-what-crawlability-stands-for\/"},"wordCount":507,"commentCount":0,"image":{"@id":"https:\/\/mb30.tetsua.xyz\/en\/how-do-crawlers-work-what-crawlability-stands-for\/#primaryimage"},"thumbnailUrl":"https:\/\/mb30.tetsua.xyz\/wp-content\/uploads\/2026\/04\/Mobiteam-Crawl-1020x480-1.jpg","inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/mb30.tetsua.xyz\/en\/how-do-crawlers-work-what-crawlability-stands-for\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/mb30.tetsua.xyz\/en\/how-do-crawlers-work-what-crawlability-stands-for\/","url":"https:\/\/mb30.tetsua.xyz\/en\/how-do-crawlers-work-what-crawlability-stands-for\/","name":"How Do Crawlers Work & What Crawlability Stands For | Mobiteam","isPartOf":{"@id":"https:\/\/mb30.tetsua.xyz\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/mb30.tetsua.xyz\/en\/how-do-crawlers-work-what-crawlability-stands-for\/#primaryimage"},"image":{"@id":"https:\/\/mb30.tetsua.xyz\/en\/how-do-crawlers-work-what-crawlability-stands-for\/#primaryimage"},"thumbnailUrl":"https:\/\/mb30.tetsua.xyz\/wp-content\/uploads\/2026\/04\/Mobiteam-Crawl-1020x480-1.jpg","datePublished":"2020-03-26T16:08:28+00:00","dateModified":"2026-04-01T17:48:29+00:00","author":{"@id":"https:\/\/mb30.tetsua.xyz\/en\/#\/schema\/person\/868c9641ec8412e6767d2477a6611cbe"},"breadcrumb":{"@id":"https:\/\/mb30.tetsua.xyz\/en\/how-do-crawlers-work-what-crawlability-stands-for\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/mb30.tetsua.xyz\/en\/how-do-crawlers-work-what-crawlability-stands-for\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/mb30.tetsua.xyz\/en\/how-do-crawlers-work-what-crawlability-stands-for\/#primaryimage","url":"https:\/\/mb30.tetsua.xyz\/wp-content\/uploads\/2026\/04\/Mobiteam-Crawl-1020x480-1.jpg","contentUrl":"https:\/\/mb30.tetsua.xyz\/wp-content\/uploads\/2026\/04\/Mobiteam-Crawl-1020x480-1.jpg","width":1020,"height":480},{"@type":"BreadcrumbList","@id":"https:\/\/mb30.tetsua.xyz\/en\/how-do-crawlers-work-what-crawlability-stands-for\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/mb30.tetsua.xyz\/en\/"},{"@type":"ListItem","position":2,"name":"How Do Crawlers Work &#038; What Crawlability Stands For"}]},{"@type":"WebSite","@id":"https:\/\/mb30.tetsua.xyz\/en\/#website","url":"https:\/\/mb30.tetsua.xyz\/en\/","name":"Mobiteam","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/mb30.tetsua.xyz\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/mb30.tetsua.xyz\/en\/#\/schema\/person\/868c9641ec8412e6767d2477a6611cbe","name":"Serhii","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/22018943a6d60a783f53a4165a70fe390e6c8438bb252508e11efde3f1379cfa?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/22018943a6d60a783f53a4165a70fe390e6c8438bb252508e11efde3f1379cfa?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/22018943a6d60a783f53a4165a70fe390e6c8438bb252508e11efde3f1379cfa?s=96&d=mm&r=g","caption":"Serhii"},"description":"I elaborate and manage IT products, startups, and groups. Drafting the product and business development, planning and executing. I\u2019m able to simplify complex issues to benefit from everyone\u2019s contribution.","sameAs":["https:\/\/mb30.tetsua.xyz"],"url":"https:\/\/mb30.tetsua.xyz\/en\/author\/serhii\/"}]}},"_links":{"self":[{"href":"https:\/\/mb30.tetsua.xyz\/en\/wp-json\/wp\/v2\/posts\/1095","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mb30.tetsua.xyz\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mb30.tetsua.xyz\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mb30.tetsua.xyz\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mb30.tetsua.xyz\/en\/wp-json\/wp\/v2\/comments?post=1095"}],"version-history":[{"count":1,"href":"https:\/\/mb30.tetsua.xyz\/en\/wp-json\/wp\/v2\/posts\/1095\/revisions"}],"predecessor-version":[{"id":1096,"href":"https:\/\/mb30.tetsua.xyz\/en\/wp-json\/wp\/v2\/posts\/1095\/revisions\/1096"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mb30.tetsua.xyz\/en\/wp-json\/wp\/v2\/media\/615"}],"wp:attachment":[{"href":"https:\/\/mb30.tetsua.xyz\/en\/wp-json\/wp\/v2\/media?parent=1095"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mb30.tetsua.xyz\/en\/wp-json\/wp\/v2\/categories?post=1095"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mb30.tetsua.xyz\/en\/wp-json\/wp\/v2\/tags?post=1095"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}