Archive for the ‘Development’ Category

Sleeping Between API Requests

Tuesday, December 22nd, 2009

I spent a few minutes today debugging the reason why one of my “rake” tasks was getting 503 connections (Service Unavailable) from Yahoo’s API.

I had mistakenly thought this was due to the fact that Yahoo limits the number of requests to their API in a given day. I decided to put a “sleep” call in my code to be sure it wasn’t being denied access due to server-side throttling. Sure enough — all requests succeeded after adding a brief call to the “sleep” function.

One of my rails applications relies on daily updated statistics from a variety of APIs for a large set of domains (all within a loop). After adding a 2 second pause just before making the relevant API calls was enough to allow all my requests to succeed.


for dir in @directories
  sleep 2
  # do the api_call()
end

The caveat here is that the number of requests you are making will depend on how long it takes to complete…adding a 2 second pause will substantially increase the completion time. In my case it takes about 33 minutes (2 seconds x 991 domains)….obviously with twice as many domains, it would take over an hour to complete.

I have not tried, but using a fractional sleep (ie: .5 seconds) would probably be enough to succeed, depending on whatever Yahoo’s servers are willing to allow (ie: sleep .3). Also, I have considered extending the amount of time in between batch runs. I currently am collecting stats every 3 hours. I could decrease this crontab job to run once a day, or only run the report on domains that haven’t had a new stat created in the last 7 days (for example).

VN:F [1.8.4_1055]
Rating: 9.7/10 (3 votes cast)
VN:F [1.8.4_1055]
Rating: +1 (from 1 vote)

Docbook Customization Layer

Monday, July 13th, 2009

I decided to use Docbook as the place to store user interface documentation for my employer’s web applications.

The UI documentation has the “customization layer” implemented as described in the Docbook XSL book available online.

The customization layer, generally speaking, is a way to override the default docbook xsl stylesheet rules and templates in favor of our own.

For example a docbook paragraph may be defined as:

sample.xml fragment:

<book>
	<section>
		<title>User Interface Help</title>
		<para>Content goes here...</para>
	</section>
</book>

The default docbook xsl template for “section” might produce something unwieldy like this:

<div class="section">
	<div class="title">
    		<font size="4">User Interface Help</font>
	</div>
	<div class="para">
		<p>Content goes here...</p>
	</div>
</div>

A customization layer for a docbook xsl template enables us to write our own templates producing cleaner markup after the xml -> html transformation:

<div class="help">
	<h2>User Interface Help</h2>
	<p>Content goes here...</p>
</div>

The UI docbook customization layer can be found here in trunk:

	/myproject/ui/source/xslt/docbook

Only customized docbook xslt files belong in the above directory.

The “default handling” for docbook XML tags are pulled from the file system’s location of docbook as installed on the operating system (and should *never* be modified because changes will not be distributed):

	/usr/share/xml/docbook/stylesheet/nwalsh

This location may be different based on your local environment, and is defined in a configuration file somewhere under /etc and picked up by the xml processor from the DOCUMENT type declaration in a given docbook file (ie: the user interface’s glossary.xml file).

VN:F [1.8.4_1055]
Rating: 0.0/10 (0 votes cast)
VN:F [1.8.4_1055]
Rating: 0 (from 0 votes)

Error Checking Pattern for Form Fields

Monday, February 2nd, 2009

The below is written in Perl, but is simple enough to adapt to any web programming language.

Our simple form has two fields. Lets say a phone number as an optional field, and another required field for first name.

For simplicity, I’ll assume a first name can contain spaces and uppercase or lowercase letters, and a phone number can contain parenthesis, hyphens, spaces, and digits.

We will assume your application framework already has a mechanism in place to create or update an application user (Application::User), and a means of notifying the user interface (UI::Message) with the appropriate methods and constructors.

my $user = new Application::User;
my $msg = new UI::Message;
my $first_name = $ARGS{'first_name'};
my $phone_number = $ARGS{'phone_number'};

#required field first name must always be present.
unless ( $first_name && $first_name =~ m/^[a-zA-Z ]+$/ ) {
   $msg->err('first_name', "First name is a required field and can only contain upper and lowercase letters with spaces.");
}

#validate phone number if we have data for this field (optional)
if ( $phone_number && $phone_number !~ m/^[0-9\-\(\) ]+$/ ) {
   $msg->err( 'phone_number, "Phone number should be in the form: (555) 555-5555." ).
}

#stop if we encountered errors
return if $msg->has_errors;

#continue saving the data if we get this far because we have a valid First Name and valid Phone Number

$user->set( 'phone_number', $phone_number ) if $phone_number;
$user->set( 'first_name', $first_name );

if ( $user->save() ) {
    $msg->ok("You have successfully updated your contact details.");
}

Its an important security practice to “white list” all acceptable values prior to saving data or handling it within a web application.

Perl regular expression are a good way to validate strings of values. Simple regexes should always be tied to the beginning and end of the variable in question, otherwise you’re only matching against a substring.

For example:

$phone_number =~ m/\d+/; #matches at least one digit (but could contain any number of nefarious characters)

Better example:

$phone_number =~ m/^\d+$/; #tied to beginning "^" and end "$" of the string

…ONLY digits are allowed with this rule.

VN:F [1.8.4_1055]
Rating: 0.0/10 (0 votes cast)
VN:F [1.8.4_1055]
Rating: 0 (from 0 votes)