HTTP: Communicating with Web Servers
4.2 HTTP
4.2.3 MIME types
Multipart Internet mail extensions (MIME) types are a means of describing the type of data, such that another computer will know how to handle the data and how to display it effectively to the user.
To illustrate the example, if you changed the extension of a JPEG image (.JPG) to .TXT, and clicked on it, you would see a jumble of strange char-acters, not the image. This is because Windows contains a mapping from file extension to file type, and .JPG and .TXT are mapped to different file types: image/jpeg for .JPG and text/plain for .TXT.
To find an MIME type for a particular file, such as .mp3, you can open the registry editor by clicking on Start > Run, then typing REGEDIT. Then click on HKEY_CLASSES_ROOT, scroll down to .mp3, and the MIME type is written next to Content Type.
Note: Not all file types have a MIME type (e.g., .hlp help files).
4.2.4 System.Web
One of the most common uses of HTTP within applications is the ability to download the HTML content of a page into a string. The following application demonstrates this concept.
It is certainly possible to implement HTTP at the socket level, but there is a wealth of objects ready for use in HTTP client applications, and it
400–499 Redirection: Further action must be taken in order to complete the request.
500-599 Server error: The server failed to fulfill an apparently valid request.
Table 4.3 HTTP response codes (continued).
HTTP response
code range Meaning
makes little sense to reinvent the wheel. The HTTP server in the next sec-tion is implemented using HTTPWebReqest.
Start a new project in Visual Studio .NET, and drag on two textboxes,
tbResult and tbUrl. TbResults should be set with multiline=true. A button, btnCapture, should also be added. Click on the Capture button, and enter the following code:
C#
private void btnCapture_Click(object sender, System.EventArgs e)
{
tbResult.Text = getHTTP(tbUrl.Text);
}
VB.NET
Private Sub btnCapture_Click(ByVal sender As Object, _ ByVal e As System.EventArgs) Handles btnCapture.Click tbResult.Text = getHTTP(tbUrl.Text)
End Sub
Then implement the getHTTP function:
C#
public string getHTTP(string szURL) {
HttpWebRequest httpRequest;
HttpWebResponse httpResponse;
string bodyText = "";
Stream responseStream;
Byte[] RecvBytes = new Byte[Byte.MaxValue];
Int32 bytes;
httpRequest = (HttpWebRequest) WebRequest.Create(szURL);
httpResponse = (HttpWebResponse) httpRequest.GetResponse();
responseStream = httpResponse.GetResponseStream();
while(true) {
bytes = responseStream.Read(RecvBytes, 0,RecvBytes.Length);
if (bytes<=0) break;
bodyText += System.Text.Encoding.UTF8.GetString(RecvBytes, 0, bytes);
}
return bodyText;
}
VB.NET
Public Function getHTTP(ByVal szURL As String) As String Dim httprequest As HttpWebRequest
Dim httpresponse As HttpWebResponse Dim bodytext As String = ""
Dim responsestream As Stream Dim bytes As Int32
Dim RecvBytes(Byte.MaxValue) As Byte
httprequest = CType(WebRequest.Create(szURL), _ HttpWebRequest)
httpresponse = CType(httprequest.GetResponse(), _ HttpWebResponse)
responsestream = httpresponse.GetResponseStream() Do While (True)
bytes = responsestream.Read(RecvBytes, 0, _ RecvBytes.Length)
If bytes <= 0 Then Exit Do
bodytext += System.Text.Encoding.UTF8.GetString _ (RecvBytes, 0, bytes)
Loop
Return bodytext End Function
Taking a closer look at this code, it should be relatively easy to identify how it operates. The first action taken as this code is executed is that a static method on the WebRequest class is called and passed the string szURL as a parameter. This creates a webRequest object that can be cast to an
HttpWe-bRequest object, which will handle outgoing HTTP connections.
Once we have an HttpWebRequest object, we can then send the HTTP request to the server and start receiving data back from the server by calling the GetResponse method. The return value is then cast to an
HttpWebResponse object, which is then held in the httPresponse variable.
A response from a Web server is asynchronous by nature, so it is natural to create a stream from this returning data and read it in as it becomes avail-able. To do this, we can create a stream by calling the GetResponseStream
method. Once the stream is obtained, we can read bytes from it in chunks
of 256 bytes (byte.Max). Reading data in chunks improves performance.
The chunk size can be arbitrarily chosen, but 256 is efficient.
The code sits in an infinite loop until all of the incoming data is received. In a production environment, therefore, this type of action should be contained within a separate thread. Once we have a string containing all of the HTML, we can simply dump it to screen. No other processing is required. You will also need some extra namespaces:
C#
using System.Net;
using System.IO;
VB.NET
Imports System.Net Imports System.IO
To test the application, run it from Visual Studio, type in a Web site address (not forgetting the http:// prefix), and press Capture. The HTML source will appear in the body (Figure 4.1).
This is a very simple HTTP client, with no error handling, and is single threaded; however, it should suffice for simpler applications.
Figure 4.1 HTTP client application.
Table 4.4 shows the significant methods of HttpWebResponse. 4.2.5 Posting data
Many dynamic Web sites contain forms for login details, search criteria, or similar data. These forms are usually submitted via the POST method. This poses a problem, however, for any application that needs to query a page that lies behind such a form because you cannot specify posted data in the URL line.
Table 4.4 Significant members of the HttpWebResponse class.
Method or property Meaning
ContentEncoding Gets the method used to encode the body of the.
response. Returns String.
ContentLength Gets the length of the content returned by the request.
Returns Long.
ContentType Gets the content type of the response. Returns String.
Cookies Gets or sets the cookies associated with this request.
May be used thus:
Cookies[“name”].ToString().
Headers Gets the headers associated with this response from the server. May be invoked thus:
Headers[“Content-Type”].ToString().
ResponseUri Gets the URI of the Internet resource that responded to the request. May be invoked thus:
RequestURI.ToString().
Server Gets the name of the server that sent the response.
Returns String.
StatusCode Gets the status of the response. Returns the HttpStatusCode enumerated type. The StatusDescription returns a descriptive String.
GetResponseHeader Gets the specified header contents that were returned with the response. Returns String.
GetResponseStream Gets the stream used to read the body of the response.
No asynchronous variant. Returns stream.
First, prepare a page that handles POST requests. In this case, type the fol-lowing lines into a file called postTest.aspx in c:\inetpub\wwwroot (your HTTP root):
ASP.NET
<%@ Page language="c#" Debug="true"%>
<script language="C#" runat="server">
public void Page_Load(Object sender, EventArgs E) {
if (Request.Form["tbPost"]!=null) {
Response.Write(Request.Form["tbPost"].ToString());
} }
</script>
<form method="post">
<input type="text" name="tbpost">
<input type="submit">
</form>
ASP.NET is a vast subject that lies outside the scope of this book; how-ever, for the sake of explaining the above example, a quick introduction is necessary. ASP.NET is an extension to IIS that enables .NET code to be executed on receipt of requests for Web pages. This also provides means for .NET code to dynamically generate responses to clients in the form of HTML, viewable on Web browsers.
Incoming requests and outgoing data are mapped to objects in .NET, which can easily be read and manipulated. The most fundamental of these objects are the Request and Response objects. The Request object encapsu-lates the data sent from the Web browser to the server; of its properties, two of the most important are the Form and QueryString collections. The Form
collection reads data sent from the client via the POST method, whereas the
QueryString collection reads data sent from the client via the GET method.
The Response object places data on the outgoing HTTP stream to be sent to the client. One of its most important methods is Write. This method is passed a string that will be rendered as HTML on the client.
One of the features that makes ASP.NET more powerful than its predeces-sor, classic ASP, is its ability to model HTML elements as objects, not merely
as input and output streams. For example, an input box would be typically written in ASP.NET as <ASP:TEXTBOX id=”tbText” runat=”server”/>, and the properties of this textbox could then be modified from code by accessing the tbText object. In classic ASP, the only way to achieve such an effect would be to include code within the textbox declaration, such as <input
type=”text” <%=someCode%>>, which is less desirable because functional code is intermixed with HTML.
ASP.NET provides better performance than classic ASP because it is compiled on first access (in-line model) or precompiled (code-behind model). It also leverages the .NET framework, which is much richer than the scripting languages available to ASP.
The example above is appropriate for demonstrating the posting method. Every Web scripting language handles posted data in much the same way, so the technique is applicable to interfacing with any Web form.
Web scripting languages share a common feature: some sections of the page are rendered on the browser screen as HTML, and some are processed by the server and not displayed on the client. In the example, anything marked runat=”server” or prefixed <% will be processed by the server.
When the user presses the submit button (<input type=”submit”>), the browser packages any user-entered data that was contained within the
<form> tags and passes it back to the server as a POST request.
The server parses out the data in the POST request once it is received.
The server-side script can retrieve this data by accessing the Request.Form
collection. The Response.Write command prints this data back out to the browser.
To try the page out, open a browser and point it at http://localhost/post-Test.aspx; type something into the textbox, and press Submit. Then you will see the page refresh, and the text you typed appears above the form.
Reopen the previous example and add a new textbox named tbPost. Click on the Capture button and modify the code as follows:
C#
private void btnCapture_Click(object sender, System.EventArgs e)
{
tbPost.Text = HttpUtility.UrlEncode(tbPost.Text);
tbResult.Text =
getHTTP(tbUrl.Text,"tbPost="+tbPost.Text);
}
VB.NET
Private Sub btnCapture_Click(ByVal sender As Object, _ ByVal e As System.EventArgs) Handles btnCapture.Click tbPost.Text = HttpUtility.UrlEncode(tbPost.Text)
tbResult.Text = getHTTP(tbUrl.Text,"tbPost="+tbPost.Text) End Sub
The reason for the call to HttpUtility.UrlEncode is to convert the text entered by the user into a string that is safe for transport by HTTP. This means the removal of white space (spaces are converted to “+”) and the con-version of nonalphanumeric characters, which is a requirement of the HTTP protocol.
Once the data to post is encoded, it can be passed to the getHTTP func-tion, which is described below. It is a modified version of the code previ-ously listed.
C#
public string getHTTP(string szURL,string szPost) {
HttpWebRequest httprequest;
HttpWebResponse httpresponse;
StreamReader bodyreader;
string bodytext = "";
Stream responsestream;
Stream requestStream;
httprequest = (HttpWebRequest) WebRequest.Create(szURL);
httprequest.Method = "POST";
httprequest.ContentType =
"application/x-www-form-urlencoded";
httprequest.ContentLength = szPost.Length;
requestStream = httprequest.GetRequestStream();
requestStream.Write(Encoding.ASCII.GetBytes(szPost),0, szPost.Length);
requestStream.Close();
httpresponse = (HttpWebResponse) httprequest.GetResponse();
responsestream = httpresponse.GetResponseStream();
bodyreader = new StreamReader(responsestream);
bodytext = bodyreader.ReadToEnd();
return bodytext;
}
VB.NET
Public Function getHTTP(ByVal szURL As String, _ ByVal szPost As String) As String
Dim httprequest As HttpWebRequest Dim httpresponse As HttpWebResponse Dim bodyreader As StreamReader Dim bodytext As String = ""
Dim responsestream As Stream Dim requestStream As Stream
httprequest = CType(WebRequest.Create(szURL), _ HttpWebRequest)
httprequest.Method = "POST"
httprequest.ContentType = _
"application/x-www-form-urlencoded"
httprequest.ContentLength = szPost.Length requestStream = httprequest.GetRequestStream()
requestStream.Write(Encoding.ASCII.GetBytes(szPost), _ 0,szPost.Length)
requestStream.Close()
httpresponse = CType(httprequest.GetResponse(), _ HttpWebResponse)
responsestream = httpresponse.GetResponseStream() bodyreader = New StreamReader(responsestream) bodytext = bodyreader.ReadToEnd()
Return bodytext End Function
This differs from the code to simply retrieve a Web page in that once the
HttpWebRequest has been created, several parameters are set such that the request also includes the posted data. The chunked reader loop is also replaced with the ReadToEnd() method of StreamReader. This method may be elegant, but it is not compatible with binary data.
The three settings that need to be changed are the request method, con-tent type, and concon-tent length. The request method is usually GET but now must be set to POST. The content type should be set to the MIME type
application/x-www-form-urlencoded, although this is not strictly neces-sary. The content length is simply the length of the data being posted, including the variable names, and after URL encoding.
The data to be posted must then be sent to the server using the Write
method on the request stream. Once the request has been created, it is sim-ply a matter of receiving the stream from the remote server and reading to the end of the stream.
Finally, we need namespaces for the HttpUtility and Encoding objects.
You will need to make a reference to System.Web.dll by selecting Project→→→→ Add Reference, as shown in Figure 4.2.
C#
using System.Web;
using System.Text;
using System.IO;
using System.Net;
VB.NET
Imports System.Web Imports System.Text Imports System.IO Imports System.Net Figure 4.2
Visual Studio .NET, Add Reference dialog.
To test the application, run it through Visual Studio .NET, enter http://
localhost/postTest.aspx into the URL textbox, and add some other text into the POST textbox. When you press Capture, you will see that the posted text appears as part of the Web page (Figure 4.3).
Table 4.5 shows the significant members of HttpWebRequest. Figure 4.3
HTTP client application with POST facility.
Table 4.5 Significant members of HttpWebRequest . Method or Property Meaning
Accept Gets or sets the value of the Accept HTTP header.
Returns String.
AllowAutoRedirect Gets or sets a Boolean value that indicates whether the request should follow redirection (3xx) responses.
ContentLength Gets or sets the Content-length HTTP header.
ContentType Gets or sets the value of the Content-type HTTP header.
CookieContainer Gets or sets the cookies associated with the request.
May be invoked thus:
CookieContainer.getCookies[“name”].ToS tring().