Python Networking Problems

[FuegoFro]

Hi, I've been looking on a website that uses Ajax requests to update multiple timers a couple of times a second, resulting in probably 10 to 15 ajax requests per second. I was trying to write a program in python that would access the same url that the Ajax is requesting and save the data coming in. I am using httplib for the actual requests, maintaining persistent connections, getting data about twice a second. I am also using multithreading to run multiple requests at once. Each one looks like this:

import httplib,time

web_access = httplib.HTTPConnection("www.website.com")
while True:
    web_access.request("/my_url")
    response = web_access.getresponse().read()
    time.sleep(.5)
     ## do stuff with response

The program works fine for about 20-30 min, then I get a timeout error and the sever I am connecting to blocks me out (aka, even firefox can't get to it, but I can browse all other websites). I used to use urllib to access the data, but it would time out way faster (I thought it was because it created too many tcp connections, so I switched to persistent ones with httplib). However, if I open the website in firefox, it will run for hours, continually updating timers without blocking me. I was wondering what I am doing wrong and how Ajax.request is different from httplib.HTTPConnection.request()

 

[Lyerae]

The site may think your DoSing it.

 

[FuegoFro]

Yeah, I thought about that, which is why I was wondering what the differences (and similarities) were between Ajax and httplib. I mean, why would ajax not set off that response, but httplib does? Either way, the client computer is requesting data via http a couple of times per second.

 

[codemonkey]

Yeah, I thought about that, which is why I was wondering what the differences (and similarities) were between Ajax and httplib. I mean, why would ajax not set off that response, but httplib does? Either way, the client computer is requesting data via http a couple of times per second.

They probably check the referrer.

 

[Xorifelse]

The site may think your DoSing it.

It's more likely that the ISP is temporarily blocking your connecting to the website, many websites aren't protected against DOS attacks while ISP suffer from it's effects as well.

 

[FuegoFro]

I figured it out. I was just being dumb

I don't know if it was the ISP or the website itself, but the reason that the website I am accessing doesn't set off the response is because rather than sending separate requests for each timer on-screen, it has a different interface for getting the data for multiple timers in one request. I was interpreting the Javascript file from the website wrong. :banghead: