blog.smarx.com - cloud development blog

Using the New Windows Azure CDN with a Custom Domain

2009-11-06T00:09:02Z

Today we announced a new service in Windows Azure: the Windows Azure Content Delivery Network (CDN). You can read about the details over on the Windows Azure blog. In short, you can now tap into our global CDN to cache your content close to your users. The Windows Azure CDN is a free preview for now, and we’ll announce pricing information in the future.

As we all know, my blog is an international sensation, and I sometimes include images in my posts. Now I have the opportunity to boost the performance of my blog by serving those images from the Windows Azure CDN. In this post, I’ll show you how I enabled the CDN under a custom domain name for images on my blog.

Step 1: Enable the CDN

As of today, there’s a new section in the Windows Azure portal when you view one of your storage accounts. Click the “Enable CDN” button to turn on the CDN. Note that even though you get immediate confirmation that the CDN is enabled, it may take quite some time (expect 60 minutes) for the change to propagate throughout the network.

Once you’ve completed this step, you should see something like the following, which tells you the URL at which your blobs are now available via the CDN. Note that only blobs in a public container will be cached and served by the CDN

That’s all you have to do to get the CDN functionality, but as you’ll notice, the URL for the CDN is a bit ugly. In the next step, we’ll get a cleaner URL by mapping a custom domain.

Step 2: Map a Custom Domain

Along with the Windows Azure CDN, today we’ve introduced functionality to map custom domains to either CDN URLs or storage accounts directly. I don’t want the image URLs on my blog to have the domain az1022.vo.msecnd.net in them, so let’s map the custom domain cdn.blog.smarx.com and use that instead.

Below the CDN section in the portal, you should see another new section called “Custom Domains.” It shows your normal blob endpoint as well as your CDN endpoint. When you click the “Manage” button, you get to choose the domain you’re going to map (via a CNAME record) to the endpoint. In my case, I chose cdn.blog.smarx.com. Before the CDN will start serving requests from this custom domain, you need to prove that you actually own it. When you click the “Generate Key,” you’ll get instructions for what to change in your DNS settings to validate your ownership of the domain.

Follow the instructions to prove ownership of the domain. If you’ve never set up a CNAME record before, you might want to read my previous blog post about using custom domains in Windows Azure, which includes a screenshot of what it looks like to do this with GoDaddy.

After mapping the GUID subdomain to domainnameverification.windows.azure.com, you can click the “Validate” button. Assuming your CNAME record was properly set up, you should be taken back to the storage account page.

Now you’ll see that your custom domain is mapped to the CDN. To finish the process, go back to your registrar and set up the real CNAME record (in my case, cdn.blog mapping to az1022.vo.msecnd.net). Now you should be able to use your custom domain name and the domain provided by the CDN interchangeably.

You can follow the same mapping and validation steps to use a custom domain to point directly to your blob storage endpoint.

Success (and proof)!

I did all the above steps, and now the blobs I use for my blog are available under:

http://smarxblogstorage.blob.core.windows.net/
http://az1022.vo.msecnd.net/
http://cdn.blog.smarx.com/

As proof, check out the properties of any of the screenshots in this post. They’re being served from the CDN, and if you click them, you can browse directly to a higher-resolution version.

Windows Azure at PDC 2009

2009-10-29T19:13:59Z

[UPDATE 11/24]: PDC is over! I’ve updated the below to include the sessions that were previously kept secret. Now all of the links point to videos of the sessions. Enjoy!

PDC 2009 is just around the corner, and Windows Azure has a lot of coverage throughout the event. Over the past two weeks, I’ve been lucky enough to see previews of every session presented by a Windows Azure team member. I can tell you there’s a lot of great content you won’t want to miss.

If you want to view all the published sessions relating to Windows Azure, you can search for “Windows Azure” on microsoftpdc.com. Below, I’ve listed only the sessions about Windows Azure presented by Windows Azure team members (and my commentary on them).

Windows Azure Present and Future (Manuvir Das)
Even if you’ve already seen Manuvir present an overview of the platform (at PDC08 or at MIX09), you’ll want to hear from him again. He’s going to give an overview of the new features and a peak at what’s coming next year. If you see only one Windows Azure talk at PDC, this is the one to see.
Introduction to Building Applications with Windows Azure (David Lemphers)
If you want to understand the developer perspective on Windows Azure, you should attend David’s talk. You’ll get to learn the high-level concepts you need as well as see hands-on demos, all from someone with a cool Australian accent.
Developing Advanced Applications with Windows Azure (Steve Marx)
This is my session. I’ve graduated from being the king of “Hello, World!” and will show you some advanced techniques using the new features we’re announcing at PDC. This will be one of those build-up-an-app-throughout-the-session talks.
Building Java Applications with Windows Azure (Steve Marx)
I have another session which hasn’t been published yet, because it would spoil a bit of the news at PDC. This will also be a code-heavy session, but you’ll have to wait until PDC to find out exactly what it’s all about. I dusted off my Java and Eclipse skills and wrote some Java code to run in the cloud, including using the new Java storage client library.
Patterns for Building Scalable and Reliable Applications with Windows Azure (Brad Calder)
Come prepared to think, because Brad’s taking on some of the more complex topics in building scalable apps. The content in this session is deep, but really important if you’re serious about designing for scale on Windows Azure.
Windows Azure Blob and Drive Deep Dive (Brad Calder)
Brad is the storage guy. He runs the storage team and knows every detail of the design and implementation. If for no other reason than that, you should be at his session if you care about storage. In addition, Brad will be covering a few new features that I can’t tell you about yet…
Windows Azure Tables and Queues Deep Dive (Jai Haridas)
Jai, who works on the storage team, eases you into this topic by giving a good overview of the features, but things get really interesting when he starts talking about common pitfalls and tips for getting the most out of the storage service.
Windows Azure Monitoring, Logging, and Management APIs (Matthew Kerner)
Matt works on the Fabric team, and he’s going to show you the new diagnostic APIs and what they can do. Want to have a live view of what your servers are doing? Matt will show you how to do it.
Automating the Application Lifecycle with Windows Azure (Sriram Krishnan)
Sriram will cover the new Service Management API and upgrade options, including how you can automate things like deploying new bits to the cloud.
Developing PHP and MySQL Applications with Windows Azure (Tushar Shanbhag and Mohit Srivastava)
~~This is another session that hasn’t been published yet. It’s a talk that may surprise you, but you’ll have to wait until PDC to find out what it is.~~ Tushar and Mohit showed cool things like running MediaWiki using PHP, MySQL, and memcached in Windows Azure! A must-see for people interested in these technologies.
The Business of Windows Azure: What You Should Know About Windows Azure Platform Pricing and SLAs (Dianne O’Brien)
If you’re interested in the business side of the platform, or if you’re a developer who wants to understand how pricing should influence your application design, this talk is for you. Unlike the talks above, this talk is about the full Windows Azure platform (including SQL Azure and .NET Services too).

There’s no real substitute for being at PDC in person, but if you’re unable to make it, all the sessions should be posted online a day or two after they’re presented.

If you are attending PDC, be sure to stop by and say hello. Lots of Windows Azure team members will be around and are eager to talk to you, so don’t be shy!

Windows Azure Tables: Expect Continuation Tokens, Seriously!

2009-10-28T16:37:11Z

A few weeks ago, my blog appeared suddenly empty. More recently, a customer reported that all of his data had seemingly disappeared. In his case and in mine, we were forgetting an important part of using Windows Azure Tables: continuation tokens. Read on to understand why continuation tokens are so important to handle properly in your application.

What are continuation tokens?

I’ve talked before about continuation tokens in Windows Azure Tables, in the context of paging over data. Basically, continuation tokens are the way you can pick up a query where it left off. In the paging example, you’re explicitly querying for a subset of the results (using the $top parameter in the REST API, or .Take(n) syntax in LINQ). The storage service returns those results, along with a continuation token. When you’re ready for the next page, you query again, passing in the continuation token you received from the first query. This gives you the next set of results and the next continuation token.

Another common place you’ll see continuation tokens is when the results of a query exceed 1000 entities. Windows Azure Tables returns up to a maximum of 1000 entities in a single request and returns a continuation token when more results are available.

The final place most people expect to see a continuation token is when the request to the server timed out. In practice, I have yet to see a query exceed the 30-second timeout, but it could, particularly in the case of a scan (non-indexed query) over a large set of entities.

What if you’re not doing those things?

Well, you still need to be ready for continuation tokens. When my blog appeared empty, I was querying for the top five posts (or in some queries, one post), and I was receiving no results at all. And no, the query wasn’t timing out.

The answer for what happened, and why you really do need to handle continuation tokens in your code is a bit subtle in the Query Timeout and Pagination MSDN documentation (emphasis added):

A query against the Table service may return a maximum of 1,000 items (tables or entities) for a single request. If there are additional items to be returned, the response to the query request includes custom headers containing a set of continuation tokens. The continuation token headers may be sent to the server with a subsequent request to resume the execution of the query.

The Table service times out when a query operation does not complete within 60 seconds. If the query has returned results up to that point, these results are returned in the response. Continuation tokens are also returned in this case, so that the client may make another request for the remainder of the data.

Continuation tokens may also be returned when a query crosses the partition boundary.

That last line is incredibly important, because whether or not a query crosses a partition boundary is not always in your control.

(Note that the documentation quoted above is referring to an older version of storage… the query timeout is 30 seconds now.)

What’s a partition boundary?

A partition boundary is what it sounds like… it’s the logical barrier between two partitions in the storage service. In Windows Azure Tables, it’s all about partitions. Each of your tables is divided into partitions according to the partition key you chose. Physically, in the data center, each partition might be on a separate server, or they may be on the same server (if they’re small or infrequently accessed). Partitions are the way that tables scale, and they’re also the way the system responds to heavy load (by isolating “hot” partitions on dedicated servers).

When you perform a query that doesn’t specify a partition key, this is a cross-partition query. If all the partitions happen to be on the same server, you might get all the results at once. If the partitions are physically separated in the data center, you’ll get partial results and a continuation token.

What happened on my blog?

In the case of my broken blog query, I was querying for the top five posts without specifying a partition key. The entries table for my blog only has one partition, so this might seem reasonable.

Without a partition key, though, my query could land on a server that didn’t have the right data. This hadn’t happened for the entire lifetime of my blog, but a few weeks ago, things moved around in the storage service (which happens all the time as we balance for load or change hardware). Now my query without a specified partition was initially landing on a server that didn’t actually have my data! All that server could do was return an empty set of results and a continuation token that made sure my next query would land in my first (and only) partition.

This “empty” server was there to handle additional partitions, if I chose to create them. This can happen if there used to be more partitions but they’ve been deleted, or if the storage service has allocated future placement there as part of automatic load balancing. In the future, we may be able to optimize this path to avoid the initial empty round trip.

Because I wasn’t properly dealing with continuation tokens in my code, my blog happily showed an empty page, thinking there were no blog posts at all.

What can be done?

There are a few things you should do to deal with continuation tokens in your code:

If you know it, always specify a partition key in your queries. This is the primary way I fixed my blog code.
Even if you do (1), your code should properly respond to a response that includes a continuation token.

The simplest way by far to do (2) is to make use of the ExecuteAll(...) and ExecuteAllWithRetries(...) on the TableStorageDataServiceQuery<T> class in the sample storage client library in the SDK. These methods automatically repeat the query as needed, following the chain of continuation tokens until all results have been retrieved. From the code comments:

/// <summary>
/// Returns all results of the query and hides the complexity of continuation if 
/// this is desired by a user. Users should be aware that this operation can return 
/// many objects. Uses no retries.
/// Important: this function does not call Execute immediately. Instead, it calls Execute() on 
/// the query only when the result is enumerated. This is a difference to the normal 
/// Execute() and Execute() with retry method.
/// </summary>
/// <returns>An IEnumerable representing the results of the query.</returns>

The bottom line

The bottom line is that any query that doesn’t specify both a partition key and a row key can return a continuation token. [UPDATE 10/28: More precisely, to avoid continuation tokens, a query must address a single entity completely.] Fortunately, it’s not hard to deal with this in your code. You just need to understand it and remember to do it, even if your code is running fine today.

If you want to understand continuation tokens better, I’d recommend reading the Query Timeout and Pagination MSDN documentation as well as my earlier blog post about Paging Over Data in Windows Azure Tables.

Manage Your Windows Azure Application From PowerShell

2009-10-27T05:11:15Z

Are you reading Ryan Dunn’s blog? If not, go subscribe to his RSS feed now. Ryan’s the Technical Evangelist for Windows Azure. Among other things, he’s constantly releasing useful tools for Windows Azure developers. Last week it was myazurestorage.com, a tool for viewing your tables in the browser. This week, it’s PowerShell cmdlets for managing your Windows Azure applications.

I haven’t gotten to play with them yet, but I’m eager to. From Ryan’s post:

As of today, you can download some Powershell cmdlets that wrap this API and make managing your Windows Azure applications simple from script. With these cmdlets, you can script your deploys, upgrades, and scaling operations very easily.

You can download the cmdlets from MSDN Code Gallery page.

What Makes a Great Developer Website? Help Me Improve http://dev.windowsazure.com!

2009-10-22T18:35:55Z

Did you see the new http://windowsazure.com? I think the site has a much cleaner look and makes it easier to find what you’re looking for.

http://dev.windowsazure.com takes you directly to the Windows Azure developer site, which is the place developers go for information about Windows Azure.

My responsibility is to play curator for that site. I’m going to try to find the best content available and organize it in a way that developers can find the information they need as quickly and easily as possible. I’d like your help doing that.

Three questions every product website should answer

When I’m investigating a new development technology, these are the questions I immediately ask (and the time I’m willing to spend answering each):

What is this? (30 seconds)
Is it useful to me? (5 minutes)
How can I get started? (15 minutes)

I’d like http://dev.windowsazure.com to deliver great answers to these questions for a developer audience and then provide organized, deeper content for people who decide to invest their time learning the platform.

I need your help!

I’ve set up a Writeboard page (basically a one-page wiki) with the password “azure” where you can contribute to this effort by helping me collect the right content and organize it. There’s also a section on the page for you to tell me about the best developer websites you’ve seen and why you like them.

Here’s a screenshot of the page at the time of this writing:

To contribute

Browse to http://123.writeboard.com/4b893890aa0ca3a65 and enter the password “azure”.

Transcript from Today’s Windows Azure Lounge Chat

2009-09-05T00:44:57Z

This afternoon I spent three hours hanging out in the Windows Azure Lounge. With a longer period of time to chat and less hype, we had a more relaxed and social chat than the first time. I like that, and that’s part of the reason I want to do these more regularly.

(Interesting stat: this chat had 530 messages over three hours. The first chat had 572 messages in just one hour!)

As came up again in this chat, time zones are a pain, so I vow to do one of these at night (for me) so the rest of the world has a chance to hang out during normal waking hours.

For those who missed out on the fun (or were there and want to relive it!), the transcript is available:

The word cloud on the right is courtesy of wordle.net and accurately reflects the meandering nature of the conversation. :)

Chat About Windows Azure Tomorrow

2009-09-03T19:55:04Z

Remember the Windows Azure Lounge? I’ll be hanging out there (maybe with a few coworkers) tomorrow, September 4th, from 12:00 PM to 3:00 PM PDT. Feel free to stop by and chat about whatever’s on your mind. Unlike the first chat, this one will be a little less structured. Just show up and hang out, any time noon to three.

I look forward to chatting with you tomorrow!

Simple Example of Twilio and Windows Azure

2009-09-02T19:04:22Z

As you may have already seen, Twilio chose Windows Azure as this week’s category for their developer contest. (UPDATE: Now more like two weeks!) In addition to Twilio’s normal prize of a netbook, we’ve kicked in a couple additional prizes for this week: a copy of Windows 7 and $500 of free Windows Azure use after our commercial launch in November.

To help give everyone a head start on getting their contest entry ready, I thought I’d put together a simple example of using Twilio from a Windows Azure application. While not quite as cool as my earlier experiment with Twilio (The CIA Pickup), this example should serve as a basic, how-to-make-it-work sample.

The Application

Early in the days of the Windows Azure CTP, Tobias Zimmergren recorded a song called “I Can (Windows) Azure You.” It’s kind of catchy.

My sample Twilio + Windows Azure application calls you at a phone number you specify and plays the song to you over the phone. Play with it at http://wazsong.cloudapp.net and download the full source.

What You’ll Need

A Windows Azure account (Register for your invitation code, and get it instantly!)
A ~~paid~~ Twilio account with one Outgoing CallerID Number (This sample makes outbound phone calls, ~~which~~ ~~free trial accounts~~ ~~can’t do~~.) UPDATE [9/2/2009 @ 5:15PM]: I got this wrong initially. Trial accounts can do this as long as you put in a valid Caller ID Number of your own. See Dennis’s comment below.
The Windows Azure SDK and tools (These require Windows Server 2008, Vista, or Windows 7.)

The Code

There’s exactly one page to this application. The markup isn’t terribly interesting; it’s basically just a text box and a submit button. The code-behind handles making the call to Twilio (using Twilio’s C# helper library). Here’s the full code-behind (default.aspx.cs):

using System;
using System.Collections;
using Microsoft.ServiceHosting.ServiceRuntime;
using TwilioRest;

namespace WindowsAzureSong_WebRole
{
    public partial class _Default : System.Web.UI.Page
    {
        protected void singButton_Click(object sender, EventArgs e)
        {
            var id = RoleManager.GetConfigurationSetting("TwilioID");
            var token = RoleManager.GetConfigurationSetting("TwilioToken");
            var account = new Account(id, token);
            var vars = new Hashtable();
            vars["Caller"] = RoleManager.GetConfigurationSetting("OutboundNumber");
            vars["Called"] = phoneNumber.Text;
            vars["Url"] = string.Format("http://{0}{1}", Request.Url.Host, ResolveUrl("~/callscript.xml"));
            vars["Method"] = "GET";
            account.request(string.Format("2008-08-01/Accounts/{0}/Calls", id), "POST", vars);
            submittedLabel.Visible = true;
        }
    }
}

As you can see, it generates a reference to callscript.xml, which just looks like this:

<?xml version="1.0" encoding="UTF-8" ?>
<Response>
  <Play>http://smarxblogstorage.blob.core.windows.net/files/WindowsAzureYou.mp3</Play>
</Response>

That URL is where I’ve stored the song (in a public container in Windows Azure blob storage), but you could put it anywhere (including in your app).

Links

Play with the sample at http://wazsong.cloudapp.net and download the full source code. Enjoy, and good luck if you enter the contest!

No More Waiting: Use Windows Azure Right Now!

2009-09-02T05:41:53Z

Until this week, using the Windows Azure CTP meant signing up and then waiting a couple of days for an invitation code to arrive by email. No more. You can now register for access and receive an invitation code right there on the spot. No email, no waiting, no excuses.

Go register now and build something cool!

Using Container-Level Access Policies in Windows Azure Storage

2009-08-22T19:13:02Z

Last week, I blogged about shared access signatures, one of the new blob storage features introduced in July. This feature lets you embed signatures in URLs to grant granular access to containers and blobs. In the approach I took in the blog post, you embed the access policy directly in the URL, which means there’s no way for you to modify or revoke permission after I’ve given out the URL. To limit the scope of these unrevokable privileges, explicit access policies in the URL are limited to granting permissions for up to only one hour.

To grant longer-term permissions or to retain the ability to modify or revoke permissions after handing them out, you can use another new feature called container-level access policies (MSDN documentation). These are named access policies that take the place of explicit policies in the URL. In this blog post, I’ll walk you through a simple example of using a container-level access policy.

Creating the Container-Level Access Policy

The first step is to create the container-level access policy. This is done using the “set container ACL” method (MSDN documentation). The following code from the sample creates an access policy that has an explicit start time, expiry time, and permission set.

var req = (HttpWebRequest)(WebRequest.Create(
    string.Format("http://{0}.blob.core.windows.net/{1}?restype=container&comp=acl", account,
        containername)));
req.Method = "PUT";
req.Headers.Add("x-ms-version", "2009-07-17");
req.Headers.Add("x-ms-date", DateTime.UtcNow.ToString("R"));
req.Headers.Add("x-ms-prop-publicaccess", "false");
var body = string.Format(@"<?xml version=""1.0"" encoding=""utf-8""?>
                            <SignedIdentifiers>
                                <SignedIdentifier>
                                    <Id>{0}</Id>
                                    <AccessPolicy>
                                        <Start>{1}</Start>
                                        <Expiry>{2}</Expiry>
                                        <Permission>{3}</Permission>
                                    </AccessPolicy>
                                </SignedIdentifier>
                            </SignedIdentifiers>", policyname, yesterday, tomorrow, permissions);
req.ContentLength = body.Length;
skc.SignRequest(req, new ResourceUriComponents(account, containername, "comp=acl"));
using (var stream = new StreamWriter(req.GetRequestStream()))
{
    stream.Write(body);
    stream.Close();
}
req.GetResponse();

Building the Query String

Now that we have created an access policy, we need to construct a URL that makes use of it. The following code creates a web request that uses our new access policy. Note that because we’ve specified an explicit start time, expiry time, and permission set, we don’t specify them in the URL (as we did when we weren’t using a named access policy). This tripped me up at first. The start time, expiry time, and permissions are all optional when creating the container-level access policy, and any which are specified there should not be specified in the URL.

var sig = MakePolicySignature(account, key, containername, policyname);
string text = null;
using (var reader = 
    new StreamReader(
        WebRequest.Create(
            string.Format("http://{0}.blob.core.windows.net/{1}/{2}?sr=c&si={3}&sig={4}",
                account,
                containername,
                blobname,
                policyname,
                Uri.EscapeDataString(sig)
            )
        ).GetResponse().GetResponseStream()))
{
    text = reader.ReadToEnd();
}

Signing the Request

In the previous code snippet, we used a method called MakePolicySignature to create the signature for our request. Here’s the source code for that method. See the shared access signature MSDN documentation for details on what should go in the “StringToSign.”

public static string MakePolicySignature(string account, string key, string container, string policyId)
{
    string stringtosign = string.Format("\n\n\n/{0}/{1}\n{2}", account, container, policyId);
    using (var hmac = new HMACSHA256(Convert.FromBase64String(key)))
    {
        return Convert.ToBase64String(hmac.ComputeHash(Encoding.UTF8.GetBytes(stringtosign)));
    }
}

When to Use Container-Level Access Policies

Shared access signatures give you the ability to create granular permission sets with or without the use of a container-level access policy, so when should you use them? Let’s take a look at some advantages and disadvantages to using container-level access policies.

Advantages:

Container-level access policies let you grant permissions that last more than an hour.
You can modify or revoke container-level access policies at any time.

Disadvantages:

Container-level access policies are, as the name states, at the level of the container. You can’t use them to create permissions for a specific blob (except by putting it by itself in a container).
There’s a limit of up to five access policies per container. This is a problem if you want to grant different permissions to many different users.

Because they give a greater level of control, I think container-level access policies will be my default way of granting permissions in URLs. I’ll forgo them for short-term access, like giving a user permission to upload a blob or embedding image tags on a web page.

Download the Sample Code

The above code snippets are part of a full command-line sample. Download the full source here.