Lessons learned from getting .NET to REST with Java

On a recent project I had to call Java REST services from a .NET Client. Several problems came up – ranging from authentication to hidden performance issues. I want to share my lessons learned and encourage you to share your own opinions and experiences on this topic.

The Context: REST to automate analysis processes in Continuous Integration

Let’s start by giving you some context on my project. I am using dynaTrace and it’s PurePath technology for performance analysis and architecture validation of software applications. Instead of doing this manually I want to automate this process by integrating dynaTrace into my Continuous Integration process so that every build that I run is automatically verified in terms of performance regressions and architectural shortcomings.

dynaTrace exposes several REST services – both on the dynaTrace Server as well as on the dynaTrace Client side. It uses Jetty to host these services. These services enable automation of capturing and analysing diagnostics data that is captured by dynaTrace for every transaction that is executed in my CI environment, e.g.: by my Unit- and Integration Tests.

In order to extend my build process I start with automatically recording the performance traces (PurePath’s) for every single test case and letting dynaTrace store it for me for later analysis. Therefore – just before running my NUnit tests - a custom NANT task calls the dynaTrace REST service to start a new recording. Another custom NANT task stops the recording after unit test execution is completed. As a final step an analysis NANT task calls a dynaTrace REST service to query the captured diagnostics metrics of the latest test run recording as well as the metrics from our currently accepted baseline recording (from the previous milestone build). Comparing metrics from both recordings automatically highlights regressions introduced between the current build and the milestone build.

With these simple steps I just extended my CI process to automatically perform performance regression analysis by leveraging my existing unit tests.

As a final step I publish the comparison results along with the unit test results to give every developer easy access to it. The following screenshot shows how such a comparision report would look like – giving the developer insight into how the performance characteristics of their code has changed from build to build:

Regression Analysis Report Showing Database and Web Service Problems

The Task: Letting .NET REST with Java

Before creating the NANT tasks I started with a Console Application to see how to implement the communication layer to the REST service. With .NET it’s pretty straight forward to send a REST request. With System.Net.HttpWebRequest you can create a connection to a URI – specify user credentials if you need to authenticate on the server – specify the HTTP Method and can write data that is passed as HTTP Body for HTTP POST/PUT requests.

Calling via HTTP GET

I started my experimental phase with a simple service call. one of the REST services exposed by the dynaTrace Server allows querying configuration information. The service is exposed via HTTP GET on the URI /rest/management/profiles. The response will be an XML document containing a list of dynaTrace System Profile’s that are configured on the server. The service requires user authentication. The .NET Code for calling this service looks like this:

HttpWebRequest webRequest = (HttpWebRequest)HttpWebRequest.Create("http://localhost:8020/rest/management/profiles");

webRequest.Credentials = new NetworkCredentials(myUsername, myPassword);

webRequest.CookieContainer = new CookieContainer();

webRequest.Method = "GET";

HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse();

if(webResponse.StatusCode != 200) Console.WriteLine("Return Status Code not 200-OK: " + webResponse.StatusCode);

StreamReader sReader = new StreamReader(webResponse.GetResponseStream());

Console.WriteLine("Service Response: " + sReader.ReadToEnd());

sReader.Close();

Hidden Performance Problem with Http Authentication

The code above worked fine. I was however curious about the actual HTTP interactions. So I analyzed the traffic with a network sniffer. When analyzing the actual send HTTP Requests from my .NET Application to the Java Web Service we actually see two requests being sent.

Request 1:

GET /rest/management/profiles/? HTTP/1.1

Connection: Keep-Alive

Host: localhost:8020

Response 1:

HTTP/1.1 401 Unauthorized

WWW-Authenticate: BASIC realm="dynaTrace Server"

Server: Jetty/5.1.x (Windows XP/5.1 x86 java/1.6.0_06

Request 2:

GET /rest/management/profiles/? HTTP/1.1

Authorization: Basic YWRtaW46YWRtaW4=

Host: localhost:8020

Response 2:

HTTP/1.1 200 OK

Set-Cookie: JSESSIONID=xnuozfyensgg;Path=/

Server: Jetty/5.1.x (Windows XP/5.1 x86 java/1.6.0_06

Content-Length: 949

Here comes the HTTP Body Content

The .NET Framework first sends a request without any authentication credentials. It expects a 401 with the information about the authentication type. After that it responds with the same request including the authentication header.

There are two performance penalties here:

• an extra roundtrip to the server for every request to the server that is done on a new connection

• doubled network utilization. Especially for HTTP POST requests with large content this can easily become a big problem

Calling via HTTP POST

The next service I called required HTTP POST. It is a service that allows starting a new session recording. From a REST definition perspective this call should probably be a HTTP GET as we are not posting new resources to the server – we are “just” triggering an action. Certain browsers and web servers have a maximum length of the URL. In order to overcome this potential problem it was decided to use HTTP POST.

In order to invoke an HTTP POST you simply set the value “POST” to the Method property of the web request object.

webRequest.Method = "POST";

The following code shows how to write the parameters to the output stream:

StreamWriter sWriter = new StreamWriter(webRequest.GetRequestStream());

sWriter.Write(myPostParameters);

sWriter.Close();

Hidden Interoperability Problem with Http Authentication

After modifying my code to call the /rest/management/profiles/GoSpace/startrecording REST interface on the dynaTrace Server and writing all parameters to the request stream I expected that everything works as expected. I was surprised to get a response exception HTTP 405 Method not allowed.

I went analyzing the HTTP Requests that were sent between my .NET Client and the dynaTrace Server:

Request 1:

POST /rest/management/profiles/GoSpace/startrecording HTTP/1.1

Content-Length: 82

Host: localhost:8020

Expect: 100-continue

Connection: Keep-Alive

HTTP BODY comes here

Response 1:

HTTP/1.1 401 Unauthorized

WWW-Authenticate: BASIC realm="dynaTrace Server"

Server: Jetty/5.1.x (Windows XP/5.1 x86 java/1.6.0_06

Request 2:

POST /rest/management/profiles/GoSpace/startrecording HTTP/1.1

Authorization: Basic YWRtaW46YWRtaW4=

Host: localhost:8020

Content-Length: 82

HTTP BODY comes here again

Response 2:

HTTP/1.1 405 Method Not Allowed

Allow: POST,OPTIONS

Set-Cookie: JSESSIONID=gb70vl7hao65;Path=/

Server: Jetty/5.1.x (Windows XP/5.1 x86 java/1.6.0_06

Even though the request looks alright the web server cannot correctly handle the HTTP POST event. I tried to debug through the Java side but couldn’t figure out the exact problem which I believe must be an implementation issue on the used Java Web Server or an interoperability issue between the two used Web Service Stacks.

There is one solution though that solved both highlighted problems: the performance related (two roundtrips per request) and this interoperability problem.

How to solve the Performance and Interoperability Problem

Running into both problems made me search the web to find a solution for it. I found the following blog post by Ian Dykes whodescribes how to manually set the authentication header on the first request in order to avoid the extra roundtrip. As it turns out this change also solved the interoperability issue. With the suggestions in the blog I added the following code to my implementation when building the HttpWebRequest:

string authInfo = myDomainName != null ? myDomainName + @"\" : string.Empty + myUsername + ":" + myPassword;

authInfo = Convert.ToBase64String(System.Text.Encoding.Default.GetBytes(authInfo));

webRequest.Headers["Authorization"] = "Basic " + authInfo;

Lets have a look at the HTTP POST request now with these changes. The request that failed in my above described scenario.

Request 1:

POST /rest/management/profiles/GoSpace/startrecording HTTP/1.1

Authorization: Basic YWRtaW46YWRtaW4=

Connection: Keep-Alive

Expect: 100-continue

Host: localhost:8020

Content-Length: 82

HTTP BODY comes here

Response 1:

HTTP/1.1 200 OK

Set-Cookie: JSESSIONID=1s3b2p5exe5yu;Path=/

Content-Length: 102

Content-Type: text/xml

Server: Jetty/5.1.x (Windows XP/5.1 x86 java/1.6.0_06

HTTP BODY comes here

This approach of course only works if you know the authentication type that allows you to specify the authentication header.

Lessons learned

Analyze the actual network traffic between your client and your server is always a good thing – don’t always blindely trust the frameworks you are using. These types of problems – that can be identified and fixed early on – usually don’t manifest themselves as huge problems on the local developer box can end up to be very expensive later down the road.

Your Experience and Feedback

I encourage you to provide your own feedback and experience on the described problem and solution. I would also be interested if other frameworks have similar issues that we also have to be aware of.

Final remarks and follow up readings

My original project was automating performance analysis in a Continuous Integration environment. If you are interested in this topic please check out the blog postings about Performance Management in Continuous Integration and Do more with Functional Testing. Also check out our White Papers about Continuous Performance Management.