# What does the ideal website look like to Google?

At Fruition we have a greater focus on statistics and the use of probability than any other SEO services firm. So what is statistics and how does it apply to SEO? Statistics is the study and presentation of data. Probability is the mathematical foundation of all statistical properties. In most cases we use statistics by making assumptions about our data. Common assumptions range from underlying distributions to equality of variance to the nature of our data being random. All assumptions are necessary for the underlying probability laws to apply to our data. Unfortunately in practice, our assumptions are often false. The reason our assumptions are so often false is the nature of the data we collected in the first place. Assumptions are like stereotypes, though they always include some amount of truth, you can never really know how true they are. Though we say we test our assumptions we are really just making sure that our assumptions are not patently false, we are not verifying that they are true. Translated into what this means for your website - at Fruition we knock out assumptions that are time wasters in SEO projects. From a statistical perspective, Search Engine Optimization provides wonderful data. We are not measuring human variables. Though Google will tell you that their ranking is about "good" sites, the truth is no computer can make a judgment call like determining if a site is "good" or what the "perfect" site is. A computer can only measure the presence and extent of different, predefined variables. An algorithm is simply a complex set of equations that incorporate many variables. Google has developed an algorithm that they constantly update to approximate their definition of a "good" site based on many, undisclosed, variables. To this end they often employ human reviewers of sites and incorporate their findings into their ranking results. In order to work with numbers, statistics requires data sets. One number is never enough. Rather, we look at groups of related numbers. These groups are often a single measurement made multiple times, like visits on a website. We could measure visits daily, and then compare those daily visits to the daily visits from another site. Then we can compare the two websites. Similarly, we can compare a single websites performance before and after a specified event. This event could be a known Google update (like Penguin, Panda, or Hummingbird), or simply to check to see if modification of the website have had a measureable impact. In searching for updates we can look at groups of sites and see how they perform across a certain date. When many sites have a similar reaction to a single day, something outside of the site has likely occurred. The Monday after Thanksgiving will show increased site traffic to many retail sites, the cause is the shopping season. When such a change is unexplainable, we can conclude that an update has occurred. So we look for dates where substantial decreases in site traffic have occurred across many websites to find unannounced Google updates. There are many variables that are likely a part of Google's algorithm. We can build statistical models to determine the relationship between these variables and site performance. The result is simply showing that these variables are related to, though not necessarily causing changes in site rank. Establishing which variables cause a site’s rank to improve requires studies. Studies can easily be done within single websites across multiple pages. By making the same change to a subset of the pages available and tracking the impact those changes have we can begin to build a profile as to the impact each variable has. As a result we can begin to determine not only which variables matter, but how they matter. **This is the strength in Fruition’s SEO methodology!** When we look at search results we are not seeing which sites are the best. Rather we are seeing the result of Google's algorithm being applied to the sites and search terms in question. These variables lack any human element; they are purely the result of numerical and probabilistic properties. Unlike other applications of statistical research at Fruition we find ourselves collecting data that many of our assumptions may truly apply to. If you’re interested in learning more about Fruition’s SEO methodology and how you can make your website look good to Google just click the contact us button. If you haven't tried the Google penalty checker you can try it to get a complimentary look into our in-house SEO tools.

Unlock the secret of Google's ranking system. Empower your business to create an optimized website for better search engine visibility. Dive in now.

Generic-Blog-Hero-1.jpg

Boost Your Google Rankings: The Ideal Website According to Google

# Google Holiday Update Information

Google rolled out an update on December 19th, 2013 that impacted many websites across nearly all verticals. Looking at websites that were most heavily impacted Fruition believes the update was related more to Panda vs Penguin (onsite content vs offsite back links). Below you'll see how this website (fruition.net) was impacted by the December 19th Google update.

Let us know how your website was impacted by this Google update.

Discover Fruition's case study on the impact of Google's December 2013 Panda update. Explore how this affected traffic across various verticals.

Google Panda Update Effects on Websites - Fruition Case Study

## Is Your City Website Penalized by Google? Case Study from California
Google penalties aren't confined to SEO-driven businesses. Even city websites might fall prey to this trouble, as a major Californian city experienced when their site vanished from Google's organic search results. The website facilitated multiple significant functions like ticket payment for parking violations. Investigations into the situation revealed a serious issue pointing to a server outage, a website configuration problem, or a site hack, resulting in alarming organic traffic drops. 

## Playbook: Steps to Diagnose Google Penalties
The potential for a hack was precarious, so we focused on ruling this out by analyzing the referring organic traffic in Google Analytics, checking the anchor text for websites linking back, and running security scans of the site. Thankfully, no overt hacking signs were revealed in the city website's case. 

## Going Beyond Obvious Hacks: The Limited Access Penalty
We named this the "GoogleBot Limited Access Penalty," as it entails accessibility issues and site configuration problems. We ensured that no faulty robots.txt file was preventing Google from indexing the website. Checks on 'Googlebot crawl errors,' 'server connectivity issues,' 'access denied errors,' and 'general server' issues did not reveal any anomalies. 

## Is Googlebot Still Locked Out? 
We dove deeper into the situation as Googlebot's inability to access the website usually triggers warnings. With no such message, we began reviewing possible causes like DNS routing issues, hardware firewall problems, or software firewall issues. The client's IT manager identified a change in firewall policies corresponding to the penalty date, which was rectified leading to solution of the issue.

## Firewalls and Googlebot: How to Prevent Inadvertent Blocking
When setting up a firewall, consider that blocking all bots, including search engines, has consequences. In this particular case, the absence of search engines in the exceptions led to falling off Google's index. This was rectified by ensuring Googlebot could access the robots.txt file. 

## Diagnosis Yields Results
Following the detailed guide above, this particular Google penalty was identified as being caused by a simple firewall configuration issue. Once sorted, the website reappeared in Google's index within about 12 hours. By promptly identifying and addressing the issue, we managed to restore the city website's Google ranking and re-enabled user's access to important city services.

Effect of Google penalties on city websites and how a California city managed the crisis. Explore Fruition's steps to resolve the issue.

2023-12-16T00:12:03.282Z_913.png

Resolve Google Penalties: A Case Study of Major California City

# How to quickly remove 10,000 PDF, XLS, and HTML files from Google & Bing

What a difference a week makes. Last week we got a call from a large city that had accidentally dropped their main city domain name from Google. This week we tackled the exact opposite. A C-level IT director at a middle-market company called when they realized the content of their intranet was accidentally made public. The company needed assistance in getting the confidential information out of Google and Bing search results as fast as possible. This article details the steps needed to remove 10,000 PDF, Microsoft Word, Excel, and other documents from Google and Bing indexes fast while preserving evidence to meet disclosure requirements. It showcases that SEO experts are actually good at something aside from getting websites ranked; they know how to take down content from Google and Bing quickly as well. The lessons contained are also valuable to understand how not to accidentally block Google bot. Sometimes it helps to reverse engineer what you are actually trying to do in order to understand the right processes. Here, we learn about how Google Bot accesses a site which can help with your traditional SEO efforts.

### How can so many documents get on Google and Bing?
A little forward to the story. Disclosure of corporate documents can happen by accident or it can happen maliciously. This article focuses on fixing the accidental disclosures but is important to note the two main types of malicious disclosures. The first kind of malicious disclosure is from hackers stealing the entire contents of intranets and posting them online. Unfortunately, network security is very difficult and becoming even more difficult. There are custom malware shops in Russia, Romania, China, and elsewhere that specialize in creating software programs that are very difficult to detect using traditional anti-virus and anti-malware technologies. The second kind of malicious disclosure is by insiders. Insider disclosure of malicious documents is also unfortunate in that it happens quite frequently. Both of these types of malicious disclosures are subjects of many other great articles. Here we are focusing solely on using our SEO skills to remove the accidental disclosures on a domain that is owned and controlled by a company.

### Checklist for Google and Bing Content Removal
- Step 1 - Lock Down the Opening
- Step 2 - Spider the Publicly Available Documents
- Step 3 - Setup Webmaster Tools in Google and Bing
- Step 4 - Submit URL Removals
- Step 5 - Do manual searches to verify site:domain.com
- Step 6 - Save Server Logs
- Step 7 - Run network scans of the IPs
- Step 8 - Setup Google Alerts
- Step 9 - Analyze. Asses. Repeat.

### Step 1 - Locking Down the Opening(s) and Damage Assessment
This step has two components that must happen in parallel.
**Step 1.a - Damage Assessment**
To do the damage assessment correctly you need experts in IT, legal, marketing, PR, and SEO. As an attorney, IT directory, and SEO with 15 years experience I cringe at the number of issues that this type of issue raises. If you are a publicly traded company you have serious disclosure issues to worry about, if you are a health care company you have HIPPA, if you are a financial services company you have many regulatory issues, and these are just the public disclosure requirements. Contractually you likely have Non Disclosure Agreements ("NDA") and confidentiality agreements, employee privacy issues, and just straight forward trade secrets that maybe at risk. To understand the scope of the damage Fruition has developed a checklist for health care, defense, financial services, and retail stores to work off of. The checklist in this article provides a jump start to documenting what items were made public and the reporting requirements. This can help if the accidental disclosure draws the attention of states attorneys. If you are in-house counsel you probably have a good handle on the legal, if you are IT you probably know how to quickly shut down the opening, and if you are SEO you are relishing the opportunity to get something done quickly that has a definitive end point and is clearly successful. Working together the damage to the company is minimized and customer trust can be saved.

**Step 1.b - Lock Down the Opening**
Corporate intranets can be built using many different platforms. Fruition builds intranets for Fortune 1000 enterprises using Drupal and dedicated servers with Watchguard firewalls. Fruition also builds corporate intranets using Microsoft IIS. This particular case involved a Microsoft IIS server. Assuming that the actual access has been blocked to the concerned files we move on to the configuration changes to speed removal of the documents. This is typically done by changing permissions from any or all to only the corporate IP addresses. The next step is an important one that is often overlooked. It involves properly configuring the robots meta tag and X-Robots-Tag HTTP header specifications. Without this step Google and Bing may think that your website is not crawlable because of an unknown problem. Thus, Google and Bing may delay removing the content. You also have to remember that with IIS and most Apache flavors it is easiest to "deny all" which means the robots.txt file will not be accessible. The robots.txt is another trick in the tool bag that generally will not work in this specific use case. Unlike the robots.txt file, x-robots shows in the header so even pages that are 403d (permission denied) will get reached by Google and Bing. Google's developers have an excellent document explaining how the x-robots works: https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag. In this case we set the x-robots to the following: X-Robots-Tag: noarchive, noindex, unavailable_after: 23 Jul 2007 15:00:00 PST, nofollow, noydir, nopreview, noimageindex, nosnippet, noodp Once you do this you can check your work by using any of the header check tools such as this one http://web-sniffer.net/.

### Step 2: Spider All URLs
Spider all urls that are in the Google and Bing index. Note that this goes against Google's webmaster guidelines (no automated bots used on search results). However, it is important to get a list of the urls that are included in Google and Bing's indexes. Both Google Webmaster Tools and Bing Webmaster Tools do a good job of giving you many of the urls but it has been Fruition's experience that both engines often leave out some urls. Furthermore, when you are dealing with tens of thousands or hundreds of thousands of Word, Excel, PDF, and other documents it is not feasible to do this documentation process by hand. There are several third party tools that do this. At Fruition we have an in-house tool that tracks which urls are driving traffic to a site. By tweaking it to just list the urls we generate a list of all documents that are indexed in Google and Bing. Again, this is not recommended to do frequently but in this case it is the only reliable way to catalog the potential exposure.

### Step 3 - Setup Webmaster Tools in Google and Bing
Since this is likely an intranet or similar service there probably was not a reason to include the site in webmaster tools. Now there is! That is to get the content removed from Google and Bing, FAST! Here are the links to both engines webmaster tools:
- https://www.bing.com/toolbox/webmaster
- https://www.google.com/webmasters/tools/

### Step 4 - Submit URL Removals
Now that the root domain that you want to remove content from is authorized you can use Google and Bing's content removal service. Hopefully, your content is in directories so that you can use wild cards. You can also do domain.com/*/* which seems effective in getting most of the content. If not you have to exclude each url by hand. Google's content removal tool is not linked from within webmaster tools. This is probably a good move to prevent novice webmasters from submitting urls by accident. Here are direct links to both Google and Bing's cached content removal tools.

- https://www.google.com/webmasters/tools/removals
- https://ssl.bing.com/webmaster/home/dashboard/ Note: With Bing it is linked directly in the Bing Webmaster Tools. The image on the left shows you where it is.

It is interesting that Bing's block expires. I really cannot think of a case where you would want to block a url and then have it reappear. Nevertheless, since we have already blocked bots at the server level when Bing bot goes back to check it will not get a 403 error and the page will not be added back into Bing's index.
This feature allows you to block a URL from appearing in the Bing search results. This block will remain in place for 90 days. If the URL still returns a 200 OK code when visited by our crawler after 90 days, it will reappear in our search results. There are no limits on the number of consecutive blocks you can apply to a URL. We will alert you 8 days in advance of the block being removed so you can decide if you wish to renew the block.
Removing cached content is important because Google caches a lot of these documents so even if you have stopped direct access the document lives on Google's servers. There is also the description of the site (search snippet) that often reveals confidential information.

### Step 5 - Do Manual Searches to Verify site:domain.com
Searching for site:domain.com (note there is no space between : and domain.com) provides you with a list of a urls (including Word, Excel, and PDF) documents that are included in the Google index. By adding a space to the search site: domain.com you see all sites linking to your exposed domain. Often these are just informational sites such as domaintools.com and other services that list domain registrant information.

### Step 6 - Save Server Logs
It is important to save as big of a log set as possible to try and understand who accessed the data and how bad the damage is. The server logs can give you an ideas of how long the files were up and who accessed the data.

### Step 7 - Run Network Scans of the IPs
Using Nexus and meta-sploit you can scan your block IP addresses to see what other data is leaking.

### Step 8 - Setup Google Alerts
Setup Google Alerts to catch when your private stuff leaks. To do this first create a text file with a random phrase that does not appear in the Google index. Pick your phrase, go to Google, search for it and the results should be blank. Stick your phrase in a text file anywhere on your intranet that you are concerned about private documents leaking. Next setup a Google alert to tell you when that file is in the wild. Alert: "Random phrase" Set what you want to search for (hint leave at default) The default alert type is Everything. Everything alerts include results from Google Web Search, Google Blog Search and Google News. If you are only interested in one type of results, you can select a single alert type. You can also do one for site:concerned-domain.com to alert you if a configuration issue occurs again and the content gets picked back up by Google bot.

### Step 9 - Analyze, Assess, and Repeat
Even with setting up the alerts and doing everything needed in webmaster tools you are still going to have to go to Google and Bing and do manual searches to find documents that got picked up else where or have not been removed from the indexes. This is tough process that you can write books about.

### Summary of How to Quickly Remove 10,000 Excel, Word, and PDF documents from Google & Bing
There is nothing simple about getting this much data removed from Google and Bing quickly. Each step takes attention to detail and an understanding of what signals that Google and Bing bots use to find content and then what the search engine algorithms use to list the data. The total time it took to remove 98% of the Excel, PDF, and Word documents was approximately 48 hours for Google and 96 for Bing. A few files straggled out in the indexes for several days. Nevertheless, in this case, the damage was contained quickly and the harm caused to the company was minimal. I hope that this article helps you avoid having to use these issues or if you found this article because you are facing the same issues feel free to contact Fruition. We are available 24/7 for issues such as this.

### Why This Was a Great Project
This project was interesting because it brought together my skills as attorney, IT director, and owner of SEO company allowing me to apply my 20 years of knowledge with a group of talented individuals to minimize damage to the client. It is the exact opposite of the previous weeks project that required Fruition to get a site re-index quickly.

Find out how our SEO experts help businesses swiftly remove large amounts of accidentally exposed private files from Google and Bing. Reach out to our team for immediate assistance.

hoe-to-quickly-remove.jpg

Swift Removal of Bulk Confidential Files from Google and Bing Search

# March 7th Google Web Spam Notifications

On March 7th, 2014 many websites received a "manual web spam" notice for unnatural inbound links from Google's web spam team. We found that several websites in our Google Penalty Checker tool that didn't do any link building for several years and they still received the notice. After a little excel magic we found just a handful of backlinks that all of the sites shared. We filtered out some common domains e.g. domaintools.com, alexa.com, and other generic whois sites.

- Doc-txt.com
- foccorepresentacoes.net.br
- mrwhatis.net
- prlog.ru
- similarpages.com
- topalternate.com
- Articlesnatch.com
- Askives.com
- Blogspot.com

**Do you have links from these sites?** The message on the 7th came in at least two varieties. The first said that Google applied an action to just the inbound links i.e. no penalty. The second stated that some pages of the website were impacted. Let us know if you received the notice(s) on March 7th from Google and if you had links from the same sites. You can also send us your domain name and we'll add it to our filters. It's possible that you needed to have links from some of these sites, all of them, or maybe some of them and then other domains could knock out the penalty. We'll post an update after we get some responses.

Dive deep into Google's March 7 manual web spam notices, understand the penalties and find the best methods to tackle and even prevent them.

Navigating Google's Manual Web Spam Notices from March 7: Essential Steps

# Panda 4.0

Google announced Panda 4.0. This impacts a significant number of searches. Data for your site will not be available on Fruition's Google Penalty Checker dashboard until we have a strong statistical basis for the impact. If you're a paid subscriber to GPC Panda 4.0 will be available within two weeks. If you're a free subscriber it'll be several weeks: You can login to GPC again here: https://fruition.net/sem/



Grasp the implications of Google's latest algorithm update, Panda 4.0, and learn to navigate its potential effects on your website effectively.

Strategy & Research

Digital Marketing

Web Development

Creative

AI Data & Analytics

Brad Anderson

CTO

More articles by Brad Anderson

Boost Your Google Rankings: The Ideal Website According to Google

Google Panda Update Effects on Websites - Fruition Case Study

Resolve Google Penalties: A Case Study of Major California City

Swift Removal of Bulk Confidential Files from Google and Bing Search

Navigating Google's Manual Web Spam Notices from March 7: Essential Steps

Unravel Panda 4.0 - Understand Its Impact On Your Website