I spent a few minutes today debugging the reason why one of my “rake” tasks was getting 503 connections (Service Unavailable) from Yahoo’s API.
I had mistakenly thought this was due to the fact that Yahoo limits the number of requests to their API in a given day. I decided to put a “sleep” call in my code to be sure it wasn’t being denied access due to server-side throttling. Sure enough — all requests succeeded after adding a brief call to the “sleep” function.
One of my rails applications relies on daily updated statistics from a variety of APIs for a large set of domains (all within a loop). After adding a 2 second pause just before making the relevant API calls was enough to allow all my requests to succeed.
for dir in @directories
sleep 2
# do the api_call()
end
The caveat here is that the number of requests you are making will depend on how long it takes to complete…adding a 2 second pause will substantially increase the completion time. In my case it takes about 33 minutes (2 seconds x 991 domains)….obviously with twice as many domains, it would take over an hour to complete.
I have not tried, but using a fractional sleep (ie: .5 seconds) would probably be enough to succeed, depending on whatever Yahoo’s servers are willing to allow (ie: sleep .3). Also, I have considered extending the amount of time in between batch runs. I currently am collecting stats every 3 hours. I could decrease this crontab job to run once a day, or only run the report on domains that haven’t had a new stat created in the last 7 days (for example).