blog.smarx.com - cloud development blog

Adaptive Streaming with Windows Azure Blobs and CDN

2010-07-29T22:22:40Z

In this post, I’ll show you how to use Windows Azure Blobs and the Windows Azure CDN to deliver adaptive streaming video content to your users in a format compatible with Silverlight’s Smooth Streaming player. For those who just want to try it out, head over to the Adaptive Streaming with Windows Azure Blobs Uploader project on Code Gallery. The instructions there will get you going.

Understanding Smooth Streaming

Before we get into the details of how adaptive streaming works on top of Windows Azure Blobs, it’s necessary to understand what Smooth Streaming is and how it works.

Smooth Streaming is Microsoft’s HTTP-based adaptive streaming protocol. As Alex Zambelli’s wrote in his excellent “Smooth Streaming Technical Overview”:

Adaptive streaming is a hybrid delivery method that acts like streaming but is based on HTTP progressive download. It's an advanced concept that uses HTTP rather than a new protocol.

…

In a typical adaptive streaming implementation, the video/audio source is cut into many short segments ("chunks") and encoded to the desired delivery format… The encoded chunks are hosted on a HTTP Web server. A client requests the chunks from the Web server in a linear fashion and downloads them using plain HTTP progressive download.

…

The "adaptive" part of the solution comes into play when the video/audio source is encoded at multiple bit rates, generating multiple chunks of various sizes for each 2-to-4-seconds of video. The client can now choose between chunks of different sizes. Because Web servers usually deliver data as fast as network bandwidth allows them to, the client can easily estimate user bandwidth and decide to download larger or smaller chunks ahead of time. The size of the playback/download buffer is fully customizable.

In other words, HTTP-based adaptive streaming is about taking a source video, encoding it into lots of small chunks at various bitrates, and then letting the client play back the most appropriate chunks (based on available bandwidth).

If you’ve ever looked at IIS Smooth Streaming content, though, you’ll notice that there aren’t lots of tiny chunks. There are a few, fairly large video files. Alex explains this too:

IIS Smooth Streaming uses the MPEG-4 Part 14 (ISO/IEC 14496-12) file format as its disk (storage) and wire (transport) format. Specifically, the Smooth Streaming specification defines each chunk/GOP as an MPEG-4 Movie Fragment and stores it within a contiguous MP4 file for easy random access. One MP4 file is expected for each bit rate. When a client requests a specific source time segment from the IIS Web server, the server dynamically finds the appropriate Movie Fragment box within the contiguous MP4 file and sends it over the wire as a standalone file, thus ensuring full cacheability downstream.

Smooth Streaming files are quite literally all those little chunks concatenated together. I encourage you to read Alex’s entire article to understand the exact file format and wire format.

The key insight for our purpose is that to the client, Smooth Streaming content is just many small video chunks. The beauty of this model is that Smooth Streaming works great with CDNs and caches in between the client and the server. To the client, all that matters is that small chunks of video are being served from the appropriate URLs.

Using Windows Azure Blobs as an Adaptive Streaming Host

Windows Azure Blobs can serve specific content at configurable URLs, which as we’ve seen is the only requirement to provide clients with an adaptive streaming experience. To sweeten the deal, there’s built-in integration between Windows Azure Blobs and the Windows Azure CDN.

All that’s left for us to do is to figure out the set of URLs a Smooth Streaming client might request and store the appropriate video chunks at those URLs. There are two files that will help us do that:

The server manifest (*.ism) – This is a SMIL file that maps video and audio tracks to the file that contains them and the bitrates at which they were encoded.
The client manifest (*.ismc) – This is an XML file that specifies to the client which bitrates and timestamps are available. It also specifies the URL template clients should use to request chunks.

The combination of these two files tells us everything we need to know to extract the video chunks and store them in Widows Azure Blobs.

The Adaptive Streaming with Windows Azure Blobs Uploader code first reads the server manifest and keeps track of the mapping of bitrate and content type (video or audio) to tracks within files. Then it reads the client manifest and generates all the permutations of bitrate, content type, and timestamp. For each of these, it looks up the appropriate track of the appropriate file, extracts that chunk from the file, and stores it in blob storage according to the URL template in the client manifest.

The code’s not too complicated, and you can find it in the Code Gallery project in SmoothStreamingAzure.cs.

Prior Work

After I patted myself on the back for coming up with this brilliant scheme, it was pointed out to me that Alden Torres blogged about this back in December 2009. He used a tool on Codeplex called MP4 Explorer, which has a feature that allows uploading to blob storage. That tool reads the source MP4 files themselves and derives the chunks from there (as opposed to my approach, which reads the client manifest).

The two big reasons I decided to write my own code for this were that I wanted a command-line tool and that I wanted to upload the blobs in parallel. I was able to cut down the upload time for the Big Buck Bunny video from around three hours (as Alden mentions in his post) to around thirty minutes simply by doing the uploads in parallel.

Shortcomings of This Approach

To the client doing simple playback, there’s no difference between IIS Smooth Streaming (hosted by IIS Media Services) or Adaptive Streaming with Windows Azure Blobs. However, to the content owner and to the server, there are significant differences:

With IIS on the server, scenarios that require server intelligence are possible (like real-time transcoding or encryption).
There are fewer files to manage with IIS (since it keeps all the content in a small number of files). This makes copying files around and renaming them much simpler.
As future features (like fast-forward and new targets like the iPad) come out, all you need to do is update IIS Media Services to get the new functionality. With a solution like the one described in this post, you’ll need to reprocess existing content.
Because the manifest formats for IIS Smooth Streaming are actively evolving, there’s no guarantee that my code will work correctly with future Smooth Streaming clients and content.

Specifically, there are a few features of IIS Smooth Streaming that my code doesn’t handle today:

Trick play (fast-forward and rewind). This is supported under IIS by extracting keyframes from the video. My code doesn’t support extracting these keyframes.
Live Smooth Streaming. Handling a live event (where the manifest is changing and the chunks include extra hints about future chunks) isn’t supported in my code.

The Windows Azure team is still committed to running full IIS Media Services within Windows Azure web roles in the future.

Get the Tool

If you’d like to host Smooth Streaming content in Windows Azure Blobs, please check out the Adaptive Streaming with Windows Azure Blobs Uploader project on Code Gallery, where you can download the command-line tool as well as the full source code.

Sample

Here's Big Buck Bunny served from cdn.blog.smarx.com:

[UPDATE 11:29pm] The name of the tool has been changed to “Adaptive Streaming with Windows Azure Blobs Uploader”.

Deleting Windows Azure Queue Messages: Handling Exceptions

2010-07-02T04:11:53Z

This week’s episode of Cloud Cover (scheduled to go live tomorrow morning) is all about Windows Azure queues. It’s a bit of a long episode, but there’s a lot of interesting technical content in there. During the show, a detail came up as Ryan and I were discussing queues and concurrency. At the time, I wasn’t sure exactly what guidance to give, so I committed to following up before the show went live.

To understand the situation, remember that Windows Azure employs reliable queueing, meaning that it guarantees no message is lost without being handled by a consumer. That means that consuming queue messages is a two step process. First, the consumer dequeues the message, specifying a visibility timeout. At this point, the message is invisible and can’t be retrieved by other consumers. When the consumer is finished with the message, it deletes it. If, however, the consumer is unable to finish processing the message, the visibility timeout will expire, and the message will reappear on the queue. This is what guarantees the message will eventually be handled.

This leads us to the situation Ryan and I were discussing on the show. The scenario is as follows:

Instance #1 dequeues a message and starts working on it.
The message’s visibility timeout expires, making it visible again on the queue.
Instance #2 dequeues the message and starts working on it.
Instance #1 finishes working on the message and tries to delete it.

At that last step, an error will be returned from the queue service. This is because the first instance no longer “owns” the message; it’s already been delivered to another instance. (The underlying mechanism is a pop receipt which is invalidated by the second dequeueing of the message.)

The question I couldn’t answer on the fly during the show was how to accurately detect this error and handle it in code. After some discussion with the storage team, this is the .NET code I’m recommending people use to identify this error:

try
{
    q.DeleteMessage(msg);
}
catch (StorageClientException ex)
{
    if (ex.ExtendedErrorInformation.ErrorCode == "MessageNotFound")
    {
        // pop receipt must be invalid
        // ignore or log (so we can tune the visibility timeout)
    }
    else
    {
        // not the error we were expecting
        throw;
    }
}

It would be nice if the storage client library included a constant for “MessageNotFound,” as it does for a number of other common error codes, but we can be confident that’s the right string by consulting the documentation on Queue Service Error Codes.

Note that I’m not just checking for the HTTP 404 status code, because that could mean some other things (like an incorrect queue name). Looking for the “MessageNotFound” error code is more specific and thus better to use.

Now go watch the latest Cloud Cover episode!

Pivot, OData, and Windows Azure: Visual Netflix Browsing

2010-06-29T17:19:42Z

The PivotViewer Silverlight control shipped this morning, which means you can now embed a Pivot collection (with great UI) directly in a web page. Pivot is fantastic for sorting, filtering, and browsing large numbers of items.

I’ve put together my own example of using the new PivotViewer control at http://netflixpivot.cloudapp.net. It lets you browse the top ~3,000 movies that Netflix has available to stream online. I really encourage you to click through to the demo… it’s a fantastic way to find a movie to watch.

Technical Overview

The demo is built on Windows Azure and consists of a web role (which serves the web page itself), a worker role (which creates the Pivot collection once every hour or so), and blob storage, which hosts the collection and the Silverlight control (all behind the Windows Azure CDN). The data comes from Netflix’s OData feed.

I only had to write about 500 lines of code to make this all happen, and I suspect that number would go down if I used the Pauthor library (which I didn’t have access to when I wrote this demo).

Creating the Pivot Collection

The Pivot collection is created by a worker role that only has a single instance. It takes more than an hour to process the latest Netflix feed into the form needed for Pivot. I could have parallelized some of this and spread the load across multiple instances, but the feed changes infrequently, so I’m not in any particular rush to get the work done. Using a single instance makes the code very simple, because everything happens locally on a single disk, but I have also built Pivot collections in the past using a large number of instances.

The collection is created on the local disk in NetflixPivotCreator.cs. The first step is loading all the available titles from the OData feed. To generate the NetflixCatalog class, I just right-clicked on the worker role’s references and added a service reference to http://odata.netflix.com/Catalog.

var context = new NetflixCatalog(new Uri("http://odata.netflix.com/Catalog"));
DataServiceQueryContinuation<Title> token = null;
var response = ((from title in context.Titles 
                 where title.Instant.Available && title.Type == "Movie"
                 orderby title.AverageRating descending select title)
                as DataServiceQuery<Title>)
               .Expand("Genres,Cast,Directors")
               .Execute() as QueryOperationResponse<Title>;
int count = 0;
var ids = new HashSet<string>();
do
{
    if (token != null)
    {
        response = context.Execute<Title>(token);
    }
    foreach (var title in response)
    {
        if (ids.Add(title.Id))
        {
            if (count < howMany)
            {
                yield return title;
            }
            count++;
        }
    }
    token = response.GetContinuation();
}
while (token != null && count < howMany);

The next step is to download each title’s box art and create a Deep Zoom images out of it. This is a simplified version of that code:

Parallel.ForEach(GetTopInstantWatchTitles(3000),
    new ParallelOptions { MaxDegreeOfParallelism = 16 },
    (title) =>
{
    var boxArtUrl = title.BoxArt.HighDefinitionUrl ?? title.BoxArt.LargeUrl;
    var imagePath = string.Format(@"{0}\images\{1}.jpg", outputDirectory, title.Id.ToHex());
    new WebClient().DownloadFile(boxArtUrl, imagePath);
    new ImageCreator().Create(imagePath, string.Format(@"{0}\output\{1}.xml", outputDirectory, title.Id));
});

Note the use of the Task Parallel Library, which is an awesome way to make multi-threaded programming easy.

From there, there are just one more line to create the full Deep Zoom collection:

new CollectionCreator().Create(
    titles.Select(t => string.Format(@"{0}\output\{1}.xml", outputDirectory, t.Id.ToHex())).ToList(),
    string.Format(@"{0}\output\collection-{1}.dzc", outputDirectory, suffix));

At this point, I’m ready to create the actual Pivot collection (a .cxml file that contains all the details about the movies). Check out the source code in the method CreateCxml to see how this is done. (It’s just XML generation, probably made much simpler if I use the Pauthor library.

Storing the Collection in Blob Storage

Once the collection has been created, the worker role uploads it to blob storage, using some rather mundane code. I’m including it here because it demonstrates a few important details: parallelizing uploads for performance, setting the correct content type on blobs, and setting the cache control header when using the CDN. Note also that the main .cxml file is uploaded last, to ensure that it’s not served to users before all the supporting files have been uploaded.

private void UploadDirectoryRecursive(string path, CloudBlobContainer container)
{
    string cxmlPath = null;

    // use 16 threads to upload
    Parallel.ForEach(EnumerateDirectoryRecursive(path),
        new ParallelOptions { MaxDegreeOfParallelism = 16 },
        (file) =>
    {
        // save collection-#####.cxml for last
        if (Path.GetFileName(file).StartsWith("collection-") && Path.GetExtension(file) == ".cxml")
        {
            cxmlPath = file;
        }
        else
        {
            // upload each file, using the relative path as a blob name
            UploadFile(file, container.GetBlobReference(Path.GetFullPath(file).Substring(path.Length)));
        }
    });

    // finish up with the cxml itself
    if (cxmlPath != null)
    {
        UploadFile(cxmlPath, container.GetBlobReference(Path.GetFullPath(cxmlPath).Substring(path.Length)));
    }
}

private IEnumerable<string> EnumerateDirectoryRecursive(string root)
{
    foreach (var file in Directory.GetFiles(root))
        yield return file;
    foreach (var subdir in Directory.GetDirectories(root))
        foreach (var file in EnumerateDirectoryRecursive(subdir))
            yield return file;
}

private void UploadFile(string filename, CloudBlob blob)
{
    var extension = Path.GetExtension(filename).ToLower();
    if (extension == ".cxml")
    {
        // cache CXML for 30 minutes
        blob.Properties.CacheControl = "max-age=1800";
    }
    else
    {
        // cache everything else (images) for 2 hours
        blob.Properties.CacheControl = "max-age=7200";
    }
    switch (extension)
        {
            case ".xml":
            case ".cxml":
            case ".dzc":
                blob.Properties.ContentType = "application/xml";
                break;
            case ".jpg":
                blob.Properties.ContentType = "image/jpeg";
                break;
        }
    blob.UploadFile(filename);
}

Serving the Collection

Once the collection is done, there’s very little left to do. I subclassed PivotViewer to handle users clicking through to the movie listing on Netflix (either by clicking “View on Netflix” or by double-clicking an item).

public class NetflixPivotControl : PivotViewer
{
    public NetflixPivotControl()
    {
        ItemActionExecuted += new EventHandler<ItemActionEventArgs>(NetflixPivotViewer_ItemActionExecuted);
        ItemDoubleClicked += new EventHandler<ItemEventArgs>(NetflixPivotViewer_ItemDoubleClicked);
    }

    private void BrowseTo(string itemId)
    {
        HtmlPage.Window.Navigate(new Uri(GetItem(itemId).Href));
    }

    private void NetflixPivotViewer_ItemDoubleClicked(object sender, ItemEventArgs e)
    {
        BrowseTo(e.ItemId);
    }

    private void NetflixPivotViewer_ItemActionExecuted(object sender, ItemActionEventArgs e)
    {
        BrowseTo(e.ItemId);
    }

    protected override List<CustomAction> GetCustomActionsForItem(string itemId)
    {
        var list = new List<CustomAction>();
        list.Add(new CustomAction("View on Netflix", null, "View this movie at Netflix", "view"));
        return list;
    }
}

Finally, I wrote an ASP.NET MVC web role that serves up a web page with the Silverlight control embedded. The actual Silverlight application (.xap file) is stored in blob storage just like the rest of the content, so the interesting part of the controller is constructing the proper URL to the CDN version of the blob.

We have to be careful not to get a mismatched collection due to CDN caching (for example, an updated collection with new movies mismatched with cached Deep Zoom images). To avoid this situation, every time a new collection is created, all files involved are suffixed with a reversed timestamp. The ASP.NET MVC controller below references the latest .cxml file (which in turn references the matching Deep Zoom images).

private Uri GetBlobOrCdnUri(CloudBlob blob, string cdnHost)
{
    // always use HTTP to avoid Silverlight cross-protocol issues
    var ub = new UriBuilder(blob.Uri)
    {
        Scheme = "http",
        Port = 80
    };
    if (!string.IsNullOrEmpty(cdnHost))
    {
        ub.Host = cdnHost;
    }
    return ub.Uri;
}

public ActionResult Index()
{
    var blobs = CloudStorageAccount.Parse(RoleEnvironment.GetConfigurationSettingValue("DataConnectionString"))
        .CreateCloudBlobClient();
    var cdnHost = RoleEnvironment.GetConfigurationSettingValue("CdnHost");

    var controlBlob = blobs.GetBlobReference("control/NetflixPivotViewer.xap");
    var collectionBlob = blobs.ListBlobsWithPrefix("collection/collection-").OfType<CloudBlob>()
        .Where(b => b.Uri.AbsolutePath.EndsWith(".cxml")).First();

    ViewData["xapUrl"] = GetBlobOrCdnUri(controlBlob, cdnHost).AbsoluteUri;
    ViewData["collectionUrl"] = GetBlobOrCdnUri(collectionBlob, cdnHost).AbsoluteUri;
    return View();
}

Download the Code

You’ve now seen nearly all of the code involved, but you can download the full Visual Studio 2010 solution at http://cdn.blog.smarx.com/files/NetflixPivot_source_updated2.zip.

If you want to run it, you’ll also need:

And note that the collection takes quite some time to create, so expect to run this for at least an hour before you can see anything.

[UPDATE 2:58pm] The first revision of this code had a couple bugs, notably around the use of the CDN. (I originally didn’t create new blob names for each update to the collection, so mismatches due to caching were possible.) I’ve updated a couple code snippets in the text of this post, and I’ve posted a new version of the source code.

[UPDATE 6/30/2010 4:28pm] I made another small revision to the code to make sure the .cxml file is uploaded last. I updated one code snippet above, and I’ve posted a new version of the source code.

Computing the Total Size of Your Blobs

2010-06-01T23:48:41Z

A question came up on the Windows Azure MSDN forum recently about how to find the total number of bytes used by a blobs in a particular container. There’s no API that retrieves that information at the container level, but you can compute it by enumerating the blobs, as in the following one-liner:

var totalBytes = (from CloudBlob blob in
                  container.ListBlobs(new BlobRequestOptions() { UseFlatBlobListing = true })
                  select blob.Properties.Length
                 ).Sum();

To go one step further, we can enumerate all the containers too. Here’s how to sum the sizes of all the blobs in a particular account (still a one-liner):

var totalBytes = (from container in blobClient.ListContainers()
                  select
                  (from CloudBlob blob in
                   container.ListBlobs(new BlobRequestOptions() { UseFlatBlobListing = true })
                   select blob.Properties.Length
                  ).Sum()
                 ).Sum();

Note that this does not reflect the number of bytes you’re billed for. Things like empty pages in page blobs, uncommitted blocks in block blobs, snapshots, metadata, etc. all affect the total storage used in your account. The code snippets above simply sums the “sizes” (if you were to download them, for example) of all the blobs.

Making Songs Swing with Windows Azure, Python, and the Echo Nest API

2010-05-27T23:01:21Z

I’ve put together a sample application at http://swingify.cloudapp.net that lets you upload a song as an MP3 and produces a “swing time” version of it. It’s easier to explain by example, so here’s the Tetris theme song as converted by Swingify.

Background

The app makes use of the Echo Nest API and a sample developed by Tristan Jehan that converts any straight-time song to swing time by extended the first half of each beat and compressing the second half. I first saw the story over on the Music Machinery blog and then later in the week on Engadget.

I immediately wanted to try this with some songs of my own, and I thought others would want to do the same, so I thought I’d create a Windows Azure application to do this in the cloud.

How it Works

We covered this application on the latest episode of the Cloud Cover show on Channel 9 (to go live tomorrow morning – watch the teaser now). In short, the application consists of an ASP.NET MVC web role and a worker role that is mostly a thin wrapper around a Python script.

The ASP.NET MVC web role accepts an MP3 upload, stores the file in blob storage, and enqueues the name of the blob:

[HttpPost]
public ActionResult Create()
{
    var guid = Guid.NewGuid().ToString();
    var file = Request.Files[0];
    var account = CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
    var blob = account.CreateCloudBlobClient().GetContainerReference("incoming").GetBlobReference(guid);
    blob.UploadFromStream(file.InputStream);
    account.CreateCloudQueueClient().GetQueueReference("incoming").AddMessage(new CloudQueueMessage(guid));
    return RedirectToAction("Result", new { id = guid });
}

The worker role mounts a Windows Azure drive in OnStart(). Here I used the same tools and initialization code as I developed for my blog post “Serving Your Website From a Windows Azure Drive.” In OnStart():

var cache = RoleEnvironment.GetLocalResource("DriveCache");
CloudDrive.InitializeCache(cache.RootPath.TrimEnd('\\'), cache.MaximumSizeInMegabytes);

drive = CloudStorageAccount.FromConfigurationSetting("DataConnectionString")
    .CreateCloudDrive(RoleEnvironment.GetConfigurationSettingValue("DriveSnapshotUrl"));
drive.Mount(cache.MaximumSizeInMegabytes, DriveMountOptions.None);

Then there’s a simple loop in Run():

while (true)
{
    var msg = q.GetMessage(TimeSpan.FromMinutes(5));
    if (msg != null)
    {
        SwingifyBlob(msg.AsString);
        q.DeleteMessage(msg);
    }
    else
    {
        Thread.Sleep(TimeSpan.FromSeconds(5));
    }
}

The meat of the application is in the implementation of SwingifyBlob(), which calls out to python.exe on the mounted Windows Azure drive:

public void SwingifyBlob(string guid)
{
    var account = CloudStorageAccount.FromConfigurationSetting("DataConnectionString");
    var blobs = account.CreateCloudBlobClient();
    var blob = blobs.GetContainerReference("incoming").GetBlobReference(guid);

    var tempdir = RoleEnvironment.GetLocalResource("temp").RootPath;
    string filepath = Path.Combine(tempdir, guid + ".mp3");
    blob.DownloadToFile(filepath);

    var localPath = drive.LocalPath + @"\";

    var process = new Process()
    {
        StartInfo = new ProcessStartInfo(
            Path.Combine(localPath, @"python\python.exe"),
            string.Format(@"swinger.py ""{0}""", filepath))
        {
            RedirectStandardOutput = true,
            RedirectStandardError = true,
            UseShellExecute = false,
            WorkingDirectory = localPath
        }
    };

    process.StartInfo.EnvironmentVariables["PATH"] = Path.Combine(localPath, @"python");
    process.StartInfo.EnvironmentVariables["TEMP"] = tempdir;
    process.StartInfo.EnvironmentVariables["ECHO_NEST_API_KEY"] =
        RoleEnvironment.GetConfigurationSettingValue("EchoNestApiKey");

    DataReceivedEventHandler logit = (s, e) =>
        {
            if (e.Data != null)
            {
                Debug.WriteLine(e.Data);
            }
        };

    process.OutputDataReceived += logit;
    process.ErrorDataReceived += logit;

    process.Start();
    process.BeginOutputReadLine();
    process.BeginErrorReadLine();
    process.WaitForExit();

    var outfilepath = Path.Combine(tempdir, guid + "_swing+33.mp3");
    var outblob = blobs.GetContainerReference("output").GetBlobReference(guid);
    if (File.Exists(outfilepath))
    {
        outblob.Properties.ContentType = "audio/mp3";
        using (var reader = File.OpenText(Path.Combine(tempdir, guid + "_metadata.txt")))
        {
            var title = reader.ReadLine().Trim();
            if (!string.IsNullOrEmpty(title))
            {
                outblob.Metadata["title"] = title;
            }
            var artist = reader.ReadLine().Trim();
            if (!string.IsNullOrEmpty(artist))
            {
                outblob.Metadata["artist"] = artist;
            }
        }
        outblob.UploadFile(outfilepath);
    }
    else
    {
        outblob.Properties.ContentType = "text/plain";
        outblob.UploadText("Sorry... we failed to swingify this song.  Try a different one?");
    }

    foreach (var filename in Directory.GetFiles(tempdir))
    {
        File.Delete(filename);
    }
}

The first part of this code sets up the call to the python.exe process by downloading the blob to local storage, making sure the path environment variable is correct, the Echo Nest API key is set, and that the working directory is right.

The second part of the method uploads the processed MP3 and parses out the title and artist written to disk by the Python script.

Running Python in Windows Azure

For the most part, all I need to do to run Python in Windows Azure was copy my local Python directory (c:\python26) to a Windows Azure drive and mount that in my worker role. There was one gotcha, however, which stumped me for almost a day. Python is actually not copy-deployable. I needed to copy python26.dll onto the drive as well (in the Python folder, which you’ll note I added to the path). On my local machine, this DLL was in %windir%\system32.

You might also check out the Portable Python project, which seems like an easier (and better supported) way to make sure your Python distribution can actually run in Windows Azure.

Other Swingified Songs

If you wanted more examples of what you can make with http://swingify.cloudapp.net, here are some that I’ve made:

Also be sure to check out the original post on the Music Machinery blog for a few more examples.

Make Some Music

Go to http://swingify.cloudapp.net and make the songs in your own MP3 collection swing, and please send me a tweet (@smarx) with the good ones you come up with!

EmailTheInternet.com: Sending and Receiving Email in Windows Azure

2010-05-20T17:33:19Z

Running right now at http://emailtheinternet.com is my latest demo, which lets you send anything via email and posts it to a public URL on the web.

This sample, which you can download here, does three things of technical interest:

It uses a third party service (SendGrid) to send email from inside Windows Azure.
It uses a worker role with an input endpoint to listen for SMTP traffic on port 25.
It uses a custom domain name on a CDN endpoint to cache blobs.

Background: email and Windows Azure

Sending email is more complicated than you might think. Jeff Atwood has a blog post called “So You’d Like to Send Some Email (Through Code)” that sums up some of the complexities.

Sending email directly from a cloud like Windows Azure presents further challenges, because you don’t have a dedicated IP address, and it’s quite likely that spammers will use Windows Azure (if they haven’t already) to send truckloads of spam. Once that happens, spam blacklists will quickly flag the IP range of Windows Azure data centers as sources of spam. That means your legitimate email will stop getting through.

The best solution to these challenges is to not send email dircetly from Windows Azure. Instead, relay all email through a third-party SMTP service (like SendGrid or AuthSMTP) with strict anti-spam rules and perhaps dedicated IP addresses.

Note that receiving email is a completely different story. As long as people are willing to send email to your domain, you can receive it in Windows Azure by just listening on port 25 for SMTP traffic.

Sending email using a third-party service

The first thing I did was sign up for a free account with SendGrid. The free tier allows me to send up to 200 emails per day, but it doesn’t give me access to some of the advanced features. I’d recommend using at least the Silver tier if you’re serious about your email being delivered and looking correct for all users.

Once I completed the quick signup, I took down all the details, which I added as configuration settings to my project:

Actually sending email is incredibly easy using the System.Net.Mail namespace. Here’s the code that sends email replies for EmailTheInternet.com:

var reply = new MailMessage(RoleEnvironment.GetConfigurationSettingValue("EmailAddress"),
    msg.FromAddress)
{
    Subject = msg.Subject.StartsWith("RE:", StringComparison.InvariantCultureIgnoreCase)
                                            ? msg.Subject : string.Format("RE: " + msg.Subject),
    Body = body,
    IsBodyHtml = msg.HasHtmlBody // send HTML if we got HTML
};
if (!reply.IsBodyHtml) reply.BodyEncoding = Encoding.UTF8;
// make it a proper reply
reply.Headers["References"] = msg.MessageID;
reply.Headers["In-Reply-To"] = msg.MessageID;
// use our SMTP server, port, username, and password to send the mail
(new SmtpClient(RoleEnvironment.GetConfigurationSettingValue("SmtpServer"),
    int.Parse(RoleEnvironment.GetConfigurationSettingValue("SmtpPort")))
{
    Credentials = new NetworkCredential(RoleEnvironment.GetConfigurationSettingValue("SmtpUsername"),
        RoleEnvironment.GetConfigurationSettingValue("SmtpPassword"))
}).Send(reply);

Yes, it’s really that easy. If you take a look at the code, you’ll see that there’s much more work involved in constructing the HTML body of the email than in sending it via SMTP.

Note that for EmailTheInternet.com, I’m using SendGrid’s free pricing tier, which means I get a limited number of emails per day, and I don’t get a dedicated IP address or whitelabeling. Because of this, my emails may not all make it through spam filters, and some email clients will show the emails as coming from sendgrid “on behalf of post@emailtheinternet.com.” This is simply because I haven’t paid for those services, not because of some limitation of this approach.

Receiving email using a worker role

I was surprised how hard it was to find good, free libraries for receiving email in C#. I settled on Eric Daugherty’s C# Email Server (CSES) to receive the email, and then I added SharpMimeTools to handle the complex task of decoding MIME emails with attachments.

You can read the code for the gory details, but I essentially do two things:

Start a TcpListener in OnStart() to listen on the appropriate port.
Start a loop in Run() that handles each incoming email by saving everything to blobs and replying via email.

Here’s the code that initializes our SMTP handler (part of CSES) and starts a TcpListener listening on the right port (called from OnStart()):

listener = new TcpListener(IPAddress.Any,
    RoleEnvironment.CurrentRoleInstance.InstanceEndpoints["SmtpIn"].IPEndpoint.Port);
processor = new SMTPProcessor(RoleEnvironment.GetConfigurationSettingValue("DomainName"),
    new RecipientFilter(), new MessageSpool());
listener.Start();

Note that I’m using the service runtime API to determine the correct port.

Here’s the simple asynchronous handling of incoming TCP connections (called from Run()):

var mutex = new ManualResetEvent(false);
while (true)
{
    mutex.Reset();
    listener.BeginAcceptSocket((ar) =>
        {
            mutex.Set();
            processor.ProcessConnection(listener.EndAcceptSocket(ar));
        }, null);
    mutex.WaitOne();
}

Finally, here’s the code that handles an incoming email:

// make a container, with public access to blobs
var id = Guid.NewGuid().ToString().Replace("-", null);
var container = account.CreateCloudBlobClient().GetContainerReference(id);
container.Create();
container.SetPermissions(new BlobContainerPermissions() { PublicAccess=BlobContainerPublicAccessType.Blob });

// parse the message
var msg = new SharpMessage(new MemoryStream(Encoding.ASCII.GetBytes(message.Data)),
    SharpDecodeOptions.AllowAttachments | SharpDecodeOptions.AllowHtml | SharpDecodeOptions.DecodeTnef);

// create a permalink-style name for the blob
var permalink = Regex.Replace(Regex.Replace(msg.Subject.ToLower(), @"[^a-z0-9]", "-"), "--+", "-").Trim('-');
if (string.IsNullOrEmpty(permalink))
{
    // in case there's no subject
    permalink = "message";
}
var bodyBlob = container.GetBlobReference(permalink);
// set the CDN to cache the object for 2 hours
bodyBlob.Properties.CacheControl = "max-age=7200";

// replaces references to attachments with the URL of where we'll put them
msg.SetUrlBase(Utility.GetCdnUrlForUri(bodyBlob.Uri) + "/[Name]");

// save each attachment in a blob, setting the appropriate content type
foreach (SharpAttachment attachment in msg.Attachments)
{
    var blob = container.GetBlobReference(permalink + "/" + attachment.Name);
    blob.Properties.ContentType = attachment.MimeTopLevelMediaType + "/" + attachment.MimeMediaSubType;
    blob.Properties.CacheControl = "max-age=7200";
    attachment.Stream.Position = 0;
    blob.UploadFromStream(attachment.Stream);
}
// add the footer and save the body to the blob
SaveBody(msg, bodyBlob, message, container, permalink);

From there, it’s just a matter of sending the reply, which we’ve already seen.

Custom domains and CDN

Using a custom domain name is a must for your email-enabled application if you want to take advantage of email verification mechanisms like SendGrid’s whitelabel features (including Domain Keys). This is because those features require that you’re able to change DNS settings for your domain. I’m not taking advantage of those features, but I decided to use a custom domain name anyway.

I’ve covered the Windows Azure CDN before, but I thought it’s worth going over again. The Windows Azure CDN is perfect for an application where this, where (in my imagination) things that get published this way will potentially be viewed by many people around the world. I decided to turn on the CDN for this demo’s blob storage account. I also decided to use custom domain names for the CDN endpoint (http://content.emailtheinternet.com).

Step 1: Set up the CNAME record for EmailTheInternet.com

Setting up a custom domain with a Windows Azure application is easy. I just went to my domain registrar (GoDaddy) and set up domain forwarding from emailtheinternet.com to www.emailtheinternet.com:

Then I added a CNAME record that maps “www” to emailtheinternet.cloudapp.net (the domain I got from Windows Azure):

Finally, because I want email to get routed correctly, I needed to add an MX record also maping to emailtheinternet.cloudapp.net:

Step 2: Enable the CDN for my storage account

Adding a CDN endpoint to your storage account is as easy as clicking one button. Once I clicked “Enable CDN,” I got a generic CDN endpoint (az2919.vo.smecnd.net):

Step 3: Add a custom domain name for the CDN endpoint

Adding a custom domain name for a storage account or CDN endpoint consists of the same process. First I clicked “Manage” next to the CDN endpoint and entered a custom domain name:

I then clicked the “Generate Key” button, which is the first step towards validating that I own that custom domain:

Following the instructions, I added a CNAME record mapping the generated name to verify.azure.com.

Then I want back to the portal and clicked through to validate the domain. Here’s the screen showing the custom domain name validated.

Once the domain name was validated, I went back to my domain registrar and mapped the “content” CNAME to the CDN endpoint:

The whole process took only a few minutes.

You may have noticed in the code snippets above that I’m calling a method called Utility.GetCdnUrlForUri(). This method transforms blob URIs to use the CDN host, which I store as a configuration setting:

public static string GetCdnUrlForUri(Uri uri)
{
    var builder = new UriBuilder(uri);
    builder.Host = RoleEnvironment.GetConfigurationSettingValue("CdnHostName");
    return builder.Uri.AbsoluteUri;
}

Download the code

You can download the Visual Studio 2010 solution here, but note that I have not included the dependencies. Here’s where you can download them:

Thanks, Twitter followers!

A special thanks to my followers on Twitter who helped me test this (and fix a few bugs) before I released the code here. I really appreciate the help!

Serving Your Website From a Windows Azure Drive

2010-05-14T20:23:59Z

For this week’s episode of Cloud Cover, Ryan and I showed how to mount a Windows Azure drive and serve your website from it using my Hosted Web Core Worker Role. I’ve released a new Visual Studio 2010 solution called HWCWorker_Drive_source.zip that mounts a snapshot of a Windows Azure Drive and points Hosted Web Core at it. You can download it over on the Hosted Web Core Worker Role project on Code Gallery.

To use the project, you need to store your web content in a VHD, upload it to the cloud, take a snapshot of the VHD, and then configure the project to use that snapshot. To make changes, just upload the new version of the VHD, take a new snapshot, and change the configuration setting in the project.

In the rest of this post, I’ll walk through how the various steps work.

Creating a VHD on Windows 7 or Windows Server

Yeah, you could be a wuss and use the GUI, but for those of you who love the command-line (as I do), I’ve included three scripts along with the code that create, mount, and unmount VHDs.

The first script, createvhd.cmd, creates a VHD of the specified size and assigns it a drive letter:

@echo off
if "%2"=="" goto usage
set driveletter=%3
if driveletter=="" set driveletter=v
echo create vdisk file="%~f1" maximum=%2 type=fixed > "%TEMP%\script.txt"
echo select vdisk file="%~f1" >> "%TEMP%\script.txt"
echo attach vdisk >> "%TEMP%\script.txt"
echo create partition primary >> "%TEMP%\script.txt"
echo assign letter=%driveletter% >> "%TEMP%\script.txt"
echo format fs=ntfs label=vhd quick >> "%TEMP%\script.txt"
diskpart -s %TEMP%\script.txt
echo VHD should be available shortly at "%driveletter%:"
goto exit
:usage
echo USAGE: createvhd.cmd <path-to-vhd> <size-in-megabytes>
:exit

This also mounts the drive, so you can go copy in your web content. To unmount the drive (which you must do before uploading it to the cloud), you can use unmountvhd.cmd:

@echo off
if "%1"=="" goto usage
echo select vdisk file="%~f1" > "%TEMP%\script.txt"
echo detach vdisk >> "%TEMP%\script.txt"
diskpart -s %TEMP%\script.txt
goto exit
:usage
echo USAGE: unmountvhd.cmd <path-to-vhd>
:exit

And finally, to mount it again (to make changes), you can use mountvhd.cmd:

@echo off
if "%1"=="" goto usage
echo select vdisk file="%~f1" > "%TEMP%\script.txt"
echo attach vdisk >> "%TEMP%\script.txt"
diskpart -s %TEMP%\script.txt
echo VHD should be available shortly at the drive letter you assigned when you created it.
goto exit
:usage
echo USAGE: mountvhd.cmd <path-to-vhd>
:exit

Uploading and Snapshotting the VHD

One of the projects included in the HWCWorker_Drive_source.zip solution is a command-line tool called UploadAndSnapshot.exe. It does exactly what the name suggests: it uploads a VHD (or really any file) as a page blob in Windows Azure storage, takes a snapshot of that blob, and then returns the full URL to the snapshot. You can take the output of this command and put it directly into the DriveSnapshotUrl configuration setting of the cloud project. This is the snapshot that the application will mount and serve your website from.

UploadAndSnapshot uses the technique described on the Windows Azure storage team blog to efficiently upload the VHD by including only the non-zero pages. This is important both for efficiency reasons (less bytes to upload) and for cost reasons (only charged for the pages uploaded).

The usage of UploadAndSnapshot is simple. Run it without any parameters to see the usage:


  
    Usage UploadAndSnapshot.exe <connectionstring> <path-to-local-file> <url-to-uploaded-blob>
e.g. UploadAndSnapshot.exe UseDefaultEndpointsProtocol=http;AccountName=myaccount;AccountKey=1F84D...
    c:\mydrive.vhd drives/mydrive.vhd

Mounting and Using the VHD

To use a Windows Azure drive, you just need to create a local cache and then mount the drive. The following code handles this in HWCWorker_Drive in OnStart():

var localCache = RoleEnvironment.GetLocalResource("DriveCache");
CloudDrive.InitializeCache(localCache.RootPath.TrimEnd('\\'), localCache.MaximumSizeInMegabytes);

var cloudAccount = CloudStorageAccount.Parse(
    RoleEnvironment.GetConfigurationSettingValue("DriveConnectionString"));
drive = cloudAccount.CreateCloudDrive(RoleEnvironment.GetConfigurationSettingValue("DriveSnapshotUrl"));

var drivePath = drive.Mount(localCache.MaximumSizeInMegabytes, DriveMountOptions.None);

Then in OnStop() I call drive.Unmount().

Using the Development Fabric

Windows Azure drives work completely differently in the development fabric. Instead of mounting an actual VHD, development storage simulates it by creating a directory on the local filesystem, and it can not be used to mount a VHD located in the cloud. I haven’t played around with doing this on development storage enough to talk through it yet, so I’ll probably follow up with a post in the future about how to do this.

For now, my solution only works in the cloud.

Known Issues

After deploying this solution to the cloud, changing the snapshot configuration setting seems to work inconsistently. Sometimes it works fine, but sometimes a role instance will cycle, continually restarting. You can, of course, deploy the application again with the new snapshot to staging, and then swap it into production.

~~I need to add some logging to debug this further. If I figure out the mystery, I’ll update this post, but I wanted to get the code out now.~~

[UPDATE 5/14/2010] After a number of attempts to reproduce the error, I think it may have simply been caused by a simple error on my part specifying a bogus snapshot. If someone sees this problem, please let me know so I can investigate.

More Information

If you’re interested in learning more about Windows Azure drives, I recommend reading the following blog posts on the Windows Azure storage team blog:

You should also read the Windows Azure Drives whitepaper.

Get the Code

You can download the HWCWorker_drive_source.zip package over on the Hosted Web Core Worker Role project on Code Gallery.

Shared Access Signatures Are Easy These Days

2010-05-02T16:22:34Z

I wrote a blog post back when Shared Access Signatures were first released called “New Storage Feature: Shared Access Signatures,” which gave some sample code to use what was then a brand new feature in Windows Azure storage (and not supported by the storage client library).

These days, using Shared Access Signatures is much simpler. I just wrote some .NET code that uses the Microsoft.WindowsAzure.StorageClient library to do the following:

Create a blob.
Generate a Shared Access Signature (SAS) for that blob that allows read and write access.
Display a working URL to the blob.
Modify and read back the blob using only the SAS for authorization.

Here’s the code:

// regular old blob storage
var account = CloudStorageAccount.DevelopmentStorageAccount; // or your cloud account
var container = account
    .CreateCloudBlobClient()
    .GetContainerReference("testcontainer");
container.CreateIfNotExist();
var blob = container.GetBlobReference("test.txt");
blob.Properties.ContentType = "text/plain";
blob.UploadText("Hello, World!");

// create a shared access signature (looks like a query param: ?se=...)
var sas = blob.GetSharedAccessSignature(new SharedAccessPolicy()
    {
        Permissions = SharedAccessPermissions.Read
                        |SharedAccessPermissions.Write,
        SharedAccessExpiryTime = DateTime.UtcNow + TimeSpan.FromMinutes(5)
    });
Console.WriteLine("This link should work for the next five minutes:");
Console.WriteLine(blob.Uri.AbsoluteUri + sas);

// now just use the SAS to do blob operations
var sasCreds = new StorageCredentialsSharedAccessSignature(sas);
// new client using the same endpoint (including account name),
//   but using the SAS as the credentials
var sasBlob = new CloudBlobClient(account.BlobEndpoint, sasCreds)
    .GetBlobReference("testcontainer/test.txt");
sasBlob.UploadText("Hello again!");
Console.WriteLine(sasBlob.DownloadText());

There’s nothing more to it than that! For more details about Shared Access Signatures, see “Cloud Cover Episode 8: Shared Access Signatures” or the MSDN documentation on the details of signature itself.

[UPDATE 6/4/2010] I didn’t show how to use Signed Identifiers the first time around, but never fear! It’s easy too. Here’s how to add an access policy to a container and use that in a Shared Access Signature:

var permissions = container.GetPermissions();
permissions.SharedAccessPolicies.Remove("readonly");
permissions.SharedAccessPolicies.Add("readonly", new SharedAccessPolicy()
    {
        Permissions = SharedAccessPermissions.Read
    });
container.SetPermissions(permissions, new BlobRequestOptions()
{
    // fail if someone else has already changed the container before we do
    AccessCondition = AccessCondition.IfMatch(container.Properties.ETag)
});

var sasWithIdentifier = blob.GetSharedAccessSignature(new SharedAccessPolicy()
    {
        SharedAccessExpiryTime = DateTime.UtcNow + TimeSpan.FromDays(7)
    }, "readonly");

Console.WriteLine("This link should work for the next seven days:");
Console.WriteLine(blob.Uri.AbsoluteUri + sasWithIdentifier);

Update Your Windows Azure Website in Just Seconds by Syncing with Blob Storage

2010-04-29T17:41:57Z

One of the coolest uses I’ve found for my Windows Azure Hosted Web Core Worker Role is to sync my website with blob storage, letting me change files at will and immediately see the results in the cloud. You can grab the code over on Code Gallery, or a prebuilt package that you can deploy right away.

Here’s a 30-second video showing it in action:

How It Works

In a typical Windows Azure web role, IIS is configured to host the website contained in your application package. As you may know, this content is in a read-only directory on the virtual machine. That means that to change anything in your website, you need to redeploy your application.

With the Hosted Web Core Worker Role, I can point Hosted Web Core (HWC) at any local directory by just changing applicationHost.config. This means I can have a writable local directory (using local storage resources), and I can populate that directory with the website content I fetch from blob storage. Then it’s just a matter of continually syncing those blobs as they change.

The class that does all the work is called OneWayBlobSync. It’s used in WorkerRole.cs like this in OnStart():

sync = new OneWayBlobSync(
    container,
    RoleEnvironment.GetLocalResource("Websites").RootPath,
    TimeSpan.FromSeconds(int.Parse(
        RoleEnvironment.GetConfigurationSettingValue("SyncIntervalInSeconds"))));

and then in Run():

sync.Start();

Under the hood, this class does the following:

List all the blobs in the source container.
Delete any local blobs that are no longer in the source container.
Compare the blobs’ ETags with the corresponding local files, and update any changed blobs.
Repeat.

That’s it!

Drawbacks

There are a few drawbacks to using this approach:

Updating the website happens at roughly the same time on all instances. No rolling upgrade means potential downtime for your application.
Copying over files that are in use might have unintended consequences (or might not work at all). For most ASP.NET content (including web.config), this seems to work fine, but I wouldn’t be surprised to learn that some files can’t be modified this way. My code doesn’t handle any sort of concurrency errors.
There are no atomic updates. During a sync, some files might be the old versions and others the new versions. There’s no way with this method to make sure everything changes at once.

The bottom line is that I’d hesitate to use this in production. But it’s awesome as a development tool.

Get the Code

You can download the full source code or a prebuilt package over at the Windows Azure Hosted Web Core Worker Role on Code Gallery. With the prebuilt package, you can simply update the configuration file (ServiceConfiguration.cscfg) and deploy. Then you can copy the website to blob storage whenever you’re ready.

IIS Compression in Windows Azure

2010-04-22T17:12:25Z

One change you may have noticed in the latest operating system release in Windows Azure is that the dynamic compression module has been turned on in IIS. This means that without doing anything, you should now see the default dynamic compression settings take effect.

Changing the Defaults

Compression settings are primarily controlled by two configuration elements: <urlCompression> and <httpCompression>.

<urlCompression> can be configured at the application level in web.config, and it lets you turn on and off dynamic and static compression. By default, dynamic compression is turned off, so you may want to add the following line to your web.config file (but see the word of caution at the end of this post):

<urlCompression doDynamicCompression="true" dynamicCompressionBeforeCache="true" />

<httpCompression> can only be configured at the level of applicationHost.config, so with today’s web role, you’ll get configuration that looks like the following (though the directory attribute will be different):

<httpCompression directory="%SystemDrive%\inetpub\temp\IIS Temporary Compressed Files"
    lockAttributes="directory">
  <scheme name="gzip" dll="%Windir%\system32\inetsrv\gzip.dll" />
  <dynamicTypes>
    <add mimeType="text/*" enabled="true" />
    <add mimeType="message/*" enabled="true" />
    <add mimeType="application/x-javascript" enabled="true" />
    <add mimeType="*/*" enabled="false" />
  </dynamicTypes>
  <staticTypes>
    <add mimeType="text/*" enabled="true" />
    <add mimeType="message/*" enabled="true" />
    <add mimeType="application/javascript" enabled="true" />
    <add mimeType="*/*" enabled="false" />
  </staticTypes>
</httpCompression>

Unfortunately, you can’t change applicationHost.config settings in today’s Windows Azure web role. However, you can edit this section in applicationHost.config using the Hosted Web Core Worker Role project.

A Word of Caution

Compression settings are tricky, and adding more compression will not necessarily increase the performance of your application. (Sometimes it will do the exact opposite!) Be sure to do your research first, and then test your new settings to make sure they’re having the effect you expected.