not able to open website
1 - 10 of 10 Posts
Joined
·
1,058 Posts
Same thing for me since pas wednesday. I can't open it at work from IE or Chrome, but it works fine at home with any browser.
Joined
·
963 Posts
A few days ago we installed a trap for rouge web crawlers/spammers. They are links which humans can't see, but crawlers can. Search engines are automatically excluded from these traps. The robots.txt file is a file telling crawlers, accelerators, content filters, etc. what pages they are allowed and/or disallowed to fetch.
Our robots.txt file is dynamic - search engines get one version anything else fetching that file is going to be a proxy server, caching server, accelerator, content filter, etc. These are given a version of the file telling them not to fetch any pages on the server in general, and specifically not to fetch the trap page. The reason is these types of things tend to hit a server very hard with page requests and slow the site down for everyone else or they are more often "rouge" crawlers such as content thieve crawlers, spammers, etc.
What's happened here is there is a web accelerator (likely at AOL), caching server (also likely at AOL), content filter (often found at work places), etc. on your end which is following links without respecting the robots.txt file on this server. They eventually follow the invisible trap link on the pages and BAM the IP gets instantly blocked. In other words, they are poorly written - good tools of this type fetch and respect the contents of robots.txt Not your fault, and I apologize for the inconvenience.
Now, I can fix this, our trap code can white list browser user agents, IP addresses, etc.
What I'll need you to do from the computer and browser(s) you're having trouble with is visit this page:
ADMIN NOTE: test url removed for now
After you do, post here that you have and I'll look at the server logs. This will give me your IP and browser user agent so I add them to the white list.
As to anyone here who's now curious and decides to look at our robots.txt file or look for the hidden link on the pages - fetch the hidden link page or the page disallowed in robots.txt at your own peril - there is almost zero reason for human to be looking at these and the trap will instantly block your IP address.
Our robots.txt file is dynamic - search engines get one version anything else fetching that file is going to be a proxy server, caching server, accelerator, content filter, etc. These are given a version of the file telling them not to fetch any pages on the server in general, and specifically not to fetch the trap page. The reason is these types of things tend to hit a server very hard with page requests and slow the site down for everyone else or they are more often "rouge" crawlers such as content thieve crawlers, spammers, etc.
What's happened here is there is a web accelerator (likely at AOL), caching server (also likely at AOL), content filter (often found at work places), etc. on your end which is following links without respecting the robots.txt file on this server. They eventually follow the invisible trap link on the pages and BAM the IP gets instantly blocked. In other words, they are poorly written - good tools of this type fetch and respect the contents of robots.txt Not your fault, and I apologize for the inconvenience.
Now, I can fix this, our trap code can white list browser user agents, IP addresses, etc.
What I'll need you to do from the computer and browser(s) you're having trouble with is visit this page:
ADMIN NOTE: test url removed for now
After you do, post here that you have and I'll look at the server logs. This will give me your IP and browser user agent so I add them to the white list.
As to anyone here who's now curious and decides to look at our robots.txt file or look for the hidden link on the pages - fetch the hidden link page or the page disallowed in robots.txt at your own peril - there is almost zero reason for human to be looking at these and the trap will instantly block your IP address.
Joined
·
1,058 Posts
Just opened the test page.
Windows 7 SP1 with Chrome 15.0.874.120, if that helps
Windows 7 SP1 with Chrome 15.0.874.120, if that helps
Joined
·
963 Posts
Thank you! I'll check into it now
1 - 10 of 10 Posts
