Windows Azure Tables: Expect Continuation Tokens, Seriously!

A few weeks ago, my blog appeared suddenly empty.  More recently, a customer reported that all of his data had seemingly disappeared.  In his case and in mine, we were forgetting an important part of using Windows Azure Tables: continuation tokens.  Read on to understand why continuation tokens are so important to handle properly in your application.

What are continuation tokens?

I’ve talked before about continuation tokens in Windows Azure Tables, in the context of paging over data.  Basically, continuation tokens are the way you can pick up a query where it left off.  In the paging example, you’re explicitly querying for a subset of the results (using the $top parameter in the REST API, or .Take(n) syntax in LINQ).  The storage service returns those results, along with a continuation token.  When you’re ready for the next page, you query again, passing in the continuation token you received from the first query.  This gives you the next set of results and the next continuation token.

Another common place you’ll see continuation tokens is when the results of a query exceed 1000 entities.  Windows Azure Tables returns up to a maximum of 1000 entities in a single request and returns a continuation token when more results are available.

The final place most people expect to see a continuation token is when the request to the server timed out.  In practice, I have yet to see a query exceed the 30-second timeout, but it could, particularly in the case of a scan (non-indexed query) over a large set of entities.

What if you’re not doing those things?

Well, you still need to be ready for continuation tokens.  When my blog appeared empty, I was querying for the top five posts (or in some queries, one post), and I was receiving no results at all.  And no, the query wasn’t timing out.

The answer for what happened, and why you really do need to handle continuation tokens in your code is a bit subtle in the Query Timeout and Pagination MSDN documentation (emphasis added):

A query against the Table service may return a maximum of 1,000 items (tables or entities) for a single request. If there are additional items to be returned, the response to the query request includes custom headers containing a set of continuation tokens. The continuation token headers may be sent to the server with a subsequent request to resume the execution of the query.

The Table service times out when a query operation does not complete within 60 seconds. If the query has returned results up to that point, these results are returned in the response. Continuation tokens are also returned in this case, so that the client may make another request for the remainder of the data.

Continuation tokens may also be returned when a query crosses the partition boundary.

That last line is incredibly important, because whether or not a query crosses a partition boundary is not always in your control.

(Note that the documentation quoted above is referring to an older version of storage… the query timeout is 30 seconds now.)

What’s a partition boundary?

A partition boundary is what it sounds like… it’s the logical barrier between two partitions in the storage service.  In Windows Azure Tables, it’s all about partitions.  Each of your tables is divided into partitions according to the partition key you chose.  Physically, in the data center, each partition might be on a separate server, or they may be on the same server (if they’re small or infrequently accessed).  Partitions are the way that tables scale, and they’re also the way the system responds to heavy load (by isolating “hot” partitions on dedicated servers).

When you perform a query that doesn’t specify a partition key, this is a cross-partition query.  If all the partitions happen to be on the same server, you might get all the results at once.  If the partitions are physically separated in the data center, you’ll get partial results and a continuation token.

What happened on my blog?

In the case of my broken blog query, I was querying for the top five posts without specifying a partition key.  The entries table for my blog only has one partition, so this might seem reasonable.

Without a partition key, though, my query could land on a server that didn’t have the right data.  This hadn’t happened for the entire lifetime of my blog, but a few weeks ago, things moved around in the storage service (which happens all the time as we balance for load or change hardware).  Now my query without a specified partition was initially landing on a server that didn’t actually have my data!  All that server could do was return an empty set of results and a continuation token that made sure my next query would land in my first (and only) partition.

This “empty” server was there to handle additional partitions, if I chose to create them.  This can happen if there used to be more partitions but they’ve been deleted, or if the storage service has allocated future placement there as part of automatic load balancing.  In the future, we may be able to optimize this path to avoid the initial empty round trip.

Because I wasn’t properly dealing with continuation tokens in my code, my blog happily showed an empty page, thinking there were no blog posts at all.

What can be done?

There are a few things you should do to deal with continuation tokens in your code:

1. If you know it, always specify a partition key in your queries.  This is the primary way I fixed my blog code.
2. Even if you do (1), your code should properly respond to a response that includes a continuation token.

The simplest way by far to do (2) is to make use of the ExecuteAll(...) and ExecuteAllWithRetries(...) on the TableStorageDataServiceQuery<T> class in the sample storage client library in the SDK.  These methods automatically repeat the query as needed, following the chain of continuation tokens until all results have been retrieved.  From the code comments:

/// <summary>
/// Returns all results of the query and hides the complexity of continuation if 
/// this is desired by a user. Users should be aware that this operation can return 
/// many objects. Uses no retries.
/// Important: this function does not call Execute immediately. Instead, it calls Execute() on 
/// the query only when the result is enumerated. This is a difference to the normal 
/// Execute() and Execute() with retry method.
/// </summary>
/// <returns>An IEnumerable representing the results of the query.</returns>

The bottom line

The bottom line is that any query that doesn’t specify both a partition key and a row key can return a continuation token.  [UPDATE 10/28: More precisely, to avoid continuation tokens, a query must address a single entity completely.]  Fortunately, it’s not hard to deal with this in your code.  You just need to understand it and remember to do it, even if your code is running fine today.

If you want to understand continuation tokens better, I’d recommend reading the Query Timeout and Pagination MSDN documentation as well as my earlier blog post about Paging Over Data in Windows Azure Tables.