Consuming Feeds And Web Services In Razor Web Pages

The proliferation of Web Service APIs over that last 5 years or so has been immense, as more and more businesses expose their inner workings through formal contracts. You can find an API for almost any informational need; from weather forecasts to stock prices, email validation services to parcel tracking and much more. There are nearly 9000 of them listed at programmableweb.com. These services are all based on the provision of data via HTTP. They are provided via a variety of protocols, with REST being the most popular. For that reason, I shall cover REST based services first.

RESTful Web Services

A key characteristic of REST services is the explicit use of HTTP methods: GET, POST, PUT, DELETE etc to execute data operations semantically. GET is used to obtain data. POST is used to create new instances of data (like an SQL INSERT operation). PUT is the equivalent to SQL's UPDATE, and DELETE means the same thing in both paradigms. Since the scope of this article is the consumption of services, I will look only at the GET operation. Data formats vary. The most popular formats are JSON and XML (increasingly, ATOM) for representing structured data, although these services can return simple strings if that's all that is needed.

One of the main characteristics of REST-based services and a key driver behind their growing popularity is simplicity. An endpoint or resource is simply represented by a URL. Data operations are executed via a GET request and can accept parameters which are passed via the query string.

JSON

Javascript Object Notation (JSON) is a lightweight text-based format for data exchange. Because it is structured, it is relatively easy to work with in both server-side and client-side code. The following code shows how to use the WebClient class to make an HTTP GET request to the Twitter search service for tweets that include "webmatrix":

var client = new WebClient();
var json = client.DownloadString("http://search.twitter.com/search.json?q=webmatrix");

The image below shows the JSON that's returned as captured by Fiddler:

A good API will offer documentation on the structure of the data it returns so that you know how to work with it, and Twitter is no exception (https://dev.twitter.com/docs/api/1/get/search) although this particular API has been deprecated and may no longer work. The structure consists of a single object that has a number of properties: completed_in, max_id, query, and a collection of objects in another property called results. Each of these objects have a created_at property as well as others including from_user and text.

If you want to work with the data in your client side code, you can use the above code in a separate file that is called via AJAX and that outputs the value of the json variable. Your AJAX call can then make use of the data obtained in a callback. If you want to work with JSON in server-side code, you need to be able to deserialise it to something that C# (or VB) understands. Some APIs provide libraries that you can use in your application to simplify this process and allow you to work with strongly typed objects. However, the Web Pages framework provides a helper for deserialising JSON - the JSON helper. So long as the JSON that you receive from the external source is valid in structure, the JSON helper will be able to convert it to a dynamic object, or if you have a class definition that matches the JSON structure, it can deserialize the JSON into strongly typed objects. In this case, dynamic will do:

@{
    var client = new WebClient();
    var json = client.DownloadString("http://search.twitter.com/search.json?q=webmatrix");
    var search = Json.Decode(json);
}

@foreach(var result in search.results){
    <p><strong>@result.from_user (@result.from_user_name)</strong><br />
        @result.text<br />
        <em>@DateTime.Parse(result.created_at).ToString("F")</em>
    </p>
}

ATOM

The deprecated Twitter API also offers an ATOM feed. ATOM is an XML format used primarily for syndication feeds in place of RSS, but it is also used for providing a much wider range of data over HTTP. There are a number of ways in which you can obtain an consume an ATOM feed from a RESTful service. Often, the method by which you obtain the data is tied to the library you use to consume the data. I'm going to look at two alternatives: LINQ to XML and the SyndicationFeed class.

LINQ to XML is an all-purpose API for working with XML documents in .NET. It doesn't care what type of XML it is asked to process so it will work with ATOM, RSS or indeed any XML based data. XML documents are represented by the XDocument class which can be instantiated in a number of ways. The easiest way to instantiate an XDocument when working with RESTful services is to pass the URL of the resource into the static Load method:

@using System.Xml.Linq
@{
    var feed = XDocument.Load("http://search.twitter.com/search.atom?q=webmatrix");
}

LINQ to XML resides in System.Xml.Linq, which is not one of the default set of namespaces referenced by the Web Pages framework. Therefore you need to add a using directive to reference the namespace at the top of the page when you want to work with it. Once you have your XDocument, you can query it to extract data. The following sample shows how to obtain the same data as the JSON example, transfer it from the ATOM feed into a collection of anonymous objects and then output it to the browser:

@using System.Xml.Linq
@{
    var feed = XDocument.Load("http://search.twitter.com/search.atom?q=webmatrix");
    XNamespace xmlns = "http://www.w3.org/2005/Atom";
    var items = feed.Descendants(xmlns + "entry").Select(item => new {
        Title = item.Element(xmlns + "title").Value,
        Published = item.Element(xmlns + "published").Value,
        Author = item.Element(xmlns + "author").Element(xmlns + "name").Value
    });
}

@foreach (var item in items) {
    <p><strong>@item.Author</strong><br />
        @item.Title<br />
        <em>@DateTime.Parse(item.Published).ToString("F"))</em>
    </p>
}

The SyndicationFeed class is part of the.NET framework and lives in System.ServiceModel.Syndication, which like LINQ to XML is not included in the default Web Pages namespaces, so you need a using directive to be able to work with it. It is designed to work solely with XML that follows the popular syndication formats: RSS and ATOM. The main advantage of using the SyndicationFeed class oer LINQ to XML is that you don't need to query the XML yourself to get values. Since ATOM and RSS are specifications, the structure of documents is known and the SyndicationFeed class parses the XML for you, leaving you with nice IntelliSense-friendly strongly typed objects to work with. It's main disadvantage is that it is very strict in terms of its application of RSS and ATOM rules, and will not parse a document if, for example, the date format differs from the specification. Yahoo's Weather RSS service will fail as the dates in it are currently not RFC 822 compliant.

The SyndicationFeed class also offers a static Load method, but it oddly requires an XmlReader object (from System.Xml) which means another using directive:

@using System.Xml
@using System.ServiceModel.Syndication;
@{
    var feed = SyndicationFeed.Load(new XmlTextReader("http://search.twitter.com/search.atom?q=webmatrix"));
}

Just that one line of code is required to enable you to render the contents:

@foreach (var item in feed.Items) {
    <p><strong>@item.Authors.First().Name</strong><br />
        @item.Title<br />
        <em>@item.PublishDate.ToString("F")</em>
    </p>
}

SOAP Web Services

SOAP is a more complex technology to work with than REST, which is one of the reasons it is losing popularity. However, it was strong in the Enterprise, and I am including the technology in this article in case you find that you are required to work with SOAP-based services that haven't been replaced by REST alternatives yet. Unfortunately, WebMatrix offers no tooling to work with SOAP-based services, so you can either use wsdl.exe (usually found in C:\Program Files (x86)\Microsoft SDKs\Windows\v8.0A\Bin\NETFX 4.0 Tools) to generate proxies based on the metadata included in the service description document (WSDL), or if command line tools and switches make your chest tighten, you can download the free Visual Studio Express For Web and use that instead. I will illustrate the saner latter approach by referencing the sample web service for converting Celsius to Fahrenheit (and vice versa) provided by W3Schools.

The first thing you need to do is locate the URL to the WSDL document for the service you want to reference. This is an XML document describing the operations available, the parameters required and the nature and format of the response. It is also known as the service description - WDSL being an acronym for Web Service Description Language. Next, either from the WEBSITE menu or by right clicking on the web site in the Solution Explorer, choose the Add Service Reference option:

Paste the URL to the WSDL file in the Address box and change the default value of the Namespace to a more meaningful one:

Click the Go button, and wait for the wizard to download the WSDL file and to read it. If the file is found, the available services are listed in the Services pane.

Since this is a .NET service (discernable from the .asmx extension) an HTTP Post method is also provided. We shall ignore that. We want to play with SOAP. Click the OK button. You should get an extra folder with some new files:

A SOAP message consists of a number of elements all represented as XML. The message has an envelope, header, body and a fault section. It must follow SOAP syntax rules, but you don't need to know the details of any of that because the wizard has taken care of generating the proxy classes to manage the construction and transmission of SOAP messages to invoke the web service methods, and to handle the response from the web service and to extract the returned data. It has also added some entries to the web.config file that detail the configuration of the client and endpoints of the service:

  <system.serviceModel>
    <bindings>
      <basicHttpBinding>
        <binding name="TempConvertSoap" />
      </basicHttpBinding>
      <customBinding>
        <binding name="TempConvertSoap12">
          <textMessageEncoding messageVersion="Soap12" />
          <httpTransport />
        </binding>
      </customBinding>
    </bindings>
    <client>
      <endpoint address="http://www.w3schools.com/webservices/tempconvert.asmx"
        binding="basicHttpBinding" bindingConfiguration="TempConvertSoap"
        contract="TempConvertService.TempConvertSoap" name="TempConvertSoap" />
      <endpoint address="http://www.w3schools.com/webservices/tempconvert.asmx"
        binding="customBinding" bindingConfiguration="TempConvertSoap12"
        contract="TempConvertService.TempConvertSoap" name="TempConvertSoap12" />
    </client>
  </system.serviceModel>

There are two endpoint configurations, both of which have the same address. One is the configuration for the basicHttpBinding (SOAP 1.1) option, and the other is for the SOAP 1.2 option. Since there are multiple configurations for the same endpoint, you have to specify which one to use when invoking the service by passing the configuration name into the proxy client's constructor:

@{
    var service = new TempConvertService.TempConvertSoapClient("TempConvertSoap");
}

Now that you have a reference to the actual service, you can invoke its operations:

@{
    var service = new TempConvertService.TempConvertSoapClient("TempConvertSoap");
    var result = service.CelsiusToFahrenheit("20");
}

This example is a simple one in that it returns a single value that results from the calculation as a string. More complex values can be returned in a number of formats. The sample site that accompanies this article includes a weather forecast service example that returns a more complex object. The proxy client that was generated through the Add Service Reference wizard understands from the infomraiton presented in the WSDL file that this object is of type ForecastReturn and has properties such as a City property and creates classes accordingly. You can see it in Intellisense:

Another example in the sample site - the SOAP - XML page features a stock price web service that returns a string. The string is actually a fragment of XML and contains a number of values. If you want to turn this fragment into an XDocument, you should use the XDocument.Parse method:

@using System.Xml.Linq;
@{
    Page.Title = "SOAP example with XML Fragment";
    var service = new StockQuoteService.StockQuoteSoapClient("StockQuoteSoap");
    var fragment = service.GetQuote("MSFT");
    var xml = XDocument.Parse(fragment);
    var stockQuote = xml.Descendants("StockQuotes").Descendants("Stock").Select(quote => new { 
        Symbol = quote.Element("Symbol").Value,
        Last = quote.Element("Last").Value,
        Time = quote.Element("Time").Value,
        Change = quote.Element("Change").Value,
        Open = quote.Element("Open").Value,
        Low = quote.Element("Low").Value,
        Volume = quote.Element("Volume").Value,
        MktCap = quote.Element("MktCap").Value,
        PreviousClose = quote.Element("PreviousClose").Value,
        PercentageChange = quote.Element("PercentageChange").Value,
        AnnRange = quote.Element("AnnRange").Value,
        Earns = quote.Element("Earns").Value,
        PE = quote.Element("P-E").Value,
        Name = quote.Element("Name").Value
    }).FirstOrDefault();
}
<h2>@stockQuote.Name (@stockQuote.Symbol)</h2> 
<p>
    Last: @stockQuote.Last<br />
    Change: @stockQuote.Change<br />
    Volume: @stockQuote.Volume
</p>

Summary

Data for your web application can come from a variety of sources and arrive in a variety of formats. This article covered the most common combinations and provides you with the tools to consume that data in your application.

A sample site containing the source code for the scenarios covered in this article is available as a free download.