Mustang and Ford Performance Forums banner
1 - 10 of 10 Posts

· Registered
Joined
·
1,521 Posts
Discussion Starter · #1 ·
like the title says i usally use aol browser but if i try to open modular fords it says forbidden that the website wont let me access it but does fine on exporler is it on my end or did some thing change on the site?

thanks
Josh
 

· Registered
Joined
·
1,521 Posts
Discussion Starter · #2 ·
well im on a diff computer and same problem anybody got any suggestions?
 

· Registered
Joined
·
1,058 Posts
Same thing for me since pas wednesday. I can't open it at work from IE or Chrome, but it works fine at home with any browser.
 

· Registered
Joined
·
1,521 Posts
Discussion Starter · #4 ·
hopefully someone has some info i hate exporler for some reason
 

· Premium Member
Joined
·
963 Posts
A few days ago we installed a trap for rouge web crawlers/spammers. They are links which humans can't see, but crawlers can. Search engines are automatically excluded from these traps. The robots.txt file is a file telling crawlers, accelerators, content filters, etc. what pages they are allowed and/or disallowed to fetch.

Our robots.txt file is dynamic - search engines get one version anything else fetching that file is going to be a proxy server, caching server, accelerator, content filter, etc. These are given a version of the file telling them not to fetch any pages on the server in general, and specifically not to fetch the trap page. The reason is these types of things tend to hit a server very hard with page requests and slow the site down for everyone else or they are more often "rouge" crawlers such as content thieve crawlers, spammers, etc.

What's happened here is there is a web accelerator (likely at AOL), caching server (also likely at AOL), content filter (often found at work places), etc. on your end which is following links without respecting the robots.txt file on this server. They eventually follow the invisible trap link on the pages and BAM the IP gets instantly blocked. In other words, they are poorly written - good tools of this type fetch and respect the contents of robots.txt Not your fault, and I apologize for the inconvenience.

Now, I can fix this, our trap code can white list browser user agents, IP addresses, etc.

What I'll need you to do from the computer and browser(s) you're having trouble with is visit this page:

ADMIN NOTE: test url removed for now

After you do, post here that you have and I'll look at the server logs. This will give me your IP and browser user agent so I add them to the white list.

As to anyone here who's now curious and decides to look at our robots.txt file or look for the hidden link on the pages - fetch the hidden link page or the page disallowed in robots.txt at your own peril - there is almost zero reason for human to be looking at these and the trap will instantly block your IP address. :(
 

· Registered
Joined
·
1,521 Posts
Discussion Starter · #6 ·
whatever u did mine works now and didnt need to go to ur page thanks
 

· Premium Member
Joined
·
963 Posts
It won't stay fixed. I removed this particular check until we determine what your browser which is getting blocked is sending us. So when you get a moment please check that page, so I can resolve the issue and re-enable our rouge crawler trap. Thanks.
 

· Registered
Joined
·
1,521 Posts
Discussion Starter · #8 ·
went to it just says test page right ?
 
1 - 10 of 10 Posts
This is an older thread, you may not receive a response, and could be reviving an old thread. Please consider creating a new thread.
Top