05-26-2023
4 min read
Auto Check Websites for Noindex Tags
As marketing and SEO professionals and business owners, we understand the importance of ensuring our clients' websites are crawled and indexed by search engines. An accidental no-index tag can severely impact their online visibility and overall success. However, monitoring and managing indexability across multiple sites can be time-consuming and prone to human error. That's why we built the _No-Index Checker _- an automated tool designed to avoid accidental noindexes and streamline the process for marketing and SEO teams.
Why Did We Build the No-Index Checker?
We developed the No-Index Checker to solve a pressing issue faced by many marketing and SEO professionals: ensuring clients' websites are indexable by search engines. The importance of indexability cannot be overstated, as it directly affects a site's search engine rankings and performance.
We have experienced firsthand the negative consequences of accidentally deploying a no-index tag on a live site. In one case, our website suffered from low search engine rankings for months due to an unchecked no-index tag. This situation inspired us to create a tool that could not only help us, but also our fellow professionals in the industry.
By automating the indexability monitoring process, the No-Index Checker empowers marketing and SEO teams to focus on other critical aspects of their clients' online success, all while having the peace of mind that their sites are being effectively crawled and indexed.
Manually checking for noindex tags
If you just want to check manually for noindex tags here's how you do it.
-
Manual Inspection of the Source Code:
You can do this by:- Right-click on the webpage you want to inspect and select "View Page Source" or "Inspect Element".
- Use the search function (CTRL + F on Windows or Command + F on Mac) and look for the following lines:
<meta name="robots" content="noindex"> <meta name="googlebot" content="noindex"> -
Use of Browser Extensions:
Before we built this tool we built a Chrome Browser extension to check for the same noindex problems. You can install the browser extension via the Chrome Store: [https://chrome.google.com/webstore/detail/noindex-alerter/ehbhjonpgdlegenahbadgbpafopcncii](Noindex Checker Chrome Extension) -
Checking the HTTP Headers:
Use tools like HTTP Header Check, REDbot, or a command-line tool like CURL to see the headers. Look for the following line in the HTTP response headers: -
Checking the robots.txt file:
This file, usually located at the root of a website (e.g., www.fruition.net/robots.txt), can also contain noindex directives.
How Does the No-Index Checker Work?
Our No-Index Checker evaluates a site's indexability by examining three key components:
- Robots Meta Value: This tag, found in the <head> of some websites, either encourages or discourages search engine crawling.
- Robots Headers Value : Similar to the meta robots value, this tag is located in the response headers of some websites and can affect crawling behavior.
- Robots.txt Value : A file located at the root of some websites, providing general rules for site accessibility and more detailed rules for blocking specific URLs from being crawled.
Our tech stack for the checker
The No-Index Checker is built on a robust tech stack, including:
- Airtable: Used for storing website entries and handling Slack messaging for team alerts.
- Datafetcher Pro: Works in conjunction with Airtable to parse records, make API requests, and update records accordingly.
- AWS Lambda/Gateway API: Responsible for running the function and managing the API endpoint.
- Browserless.io /fetch API: Handles the site crawling process and bypasses bot detection measures, such as Cloudflare.
Stay Informed with Automated Alerts
Our No-Index Checker runs daily, seven days a week, and sends alerts to an internal Slack channel. If a site fails the no-index check, an alert is generated to keep your team informed and ready to address any issues. This proactive approach ensures that potential problems are quickly identified and resolved. Not only will you safeguard your clients' SEO success, but you’ll also free up your team to focus on higher-level strategies to better support your clients’ digital presence.
The noindex checker works!
The No-Index Checker is a powerful tool designed to alleviate the stress and uncertainty associated with indexability monitoring. By automating this process, marketing and SEO professionals can now focus on optimizing their clients' online performance and success. With the No-Index Checker, you can have confidence in your clients' site indexability and take a significant step toward safeguarding their SEO future.
To get started just complete a form and we'll reach out. We can add one to 10,000 sites to the noindex checker.