Chovy's Blog

Web Development with Node.js, PHP, Javascript, and HTML5


Sleeping Between API Requests

I spent a few minutes today debugging the reason why one of my “rake” tasks was getting 503 connections (Service Unavailable) from Yahoo’s API.

I had mistakenly thought this was due to the fact that Yahoo limits the number of requests to their API in a given day. I decided to put a “sleep” call in my code to be sure it wasn’t being denied access due to server-side throttling. Sure enough — all requests succeeded after adding a brief call to the “sleep” function.

One of my rails applications relies on daily updated statistics from a variety of APIs for a large set of domains (all within a loop). After adding a 2 second pause just before making the relevant API calls was enough to allow all my requests to succeed.

The caveat here is that the number of requests you are making will depend on how long it takes to complete…adding a 2 second pause will substantially increase the completion time. In my case it takes about 33 minutes (2 seconds x 991 domains)….obviously with twice as many domains, it would take over an hour to complete.

I have not tried, but using a fractional sleep (ie: .5 seconds) would probably be enough to succeed, depending on whatever Yahoo’s servers are willing to allow (ie: sleep .3). Also, I have considered extending the amount of time in between batch runs. I currently am collecting stats every 3 hours. I could decrease this crontab job to run once a day, or only run the report on domains that haven’t had a new stat created in the last 7 days (for example).