Download Data from URL - FORM POST

Recently, I was working on a requirement where we download data from site which requires login before that.

Site was using FORM POST. I've tried many ways like WebClient etc but in vain. Every time  I get login page html as a output rather than actual data.

Finally got a way as follows.

It is simple three step process:

1. Hit the login URL and post login information after encoding.
2. Get response and save it to CookieContainer
3. Hit resource to be downloaded URL directly and pass this Cookie information
4. Get response stream. Once you get stream, you can either save to hard disk or play with it.

Here is the sample code:

This method returns Byte[] of requested resource. You can further play with it.

public byte[] GetFileStreamFromServer(string objectURL)

        {
            byte[] byteArray = null;

            string responseData = "";

            string userName = "<userName>";
            string password = "<Password>";
            string url = "<Login URL>"

            string postData = "Username={0}&Password={1}";

            string resourceUrl = objectURL;

            postData = String.Format(postData, userName, password);
           
             // Use UTF8Encoding instead of ASCIIEncoding for XML requests:
            System.Text.ASCIIEncoding encoding = new System.Text.ASCIIEncoding();

            System.Net.HttpWebRequest hwrequest = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(url);

            System.Net.HttpWebResponse hwresponse = null;            

            CookieContainer cookies;

            System.IO.Stream resourceResponseStream = null;

            try
            {
                //Creating Cookies
                hwrequest.Accept = "*/*";
                hwrequest.AllowAutoRedirect = true;
                hwrequest.UserAgent = "Mozilla/5.0 (Windows NT 5.2) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17";
                hwrequest.Timeout = 60000;
                hwrequest.Method = "POST";
                cookies = new CookieContainer();

                if (hwrequest.Method == "POST")
                {
                    hwrequest.ContentType = "application/x-www-form-urlencoded";
                    hwrequest.CookieContainer = cookies;
                    byte[] postByteArray = encoding.GetBytes(postData);
                    hwrequest.ContentLength = postByteArray.Length;
                    System.IO.Stream postStream = hwrequest.GetRequestStream();
                    postStream.Write(postByteArray, 0, postByteArray.Length);
                    postStream.Close();
                }

                hwresponse = (System.Net.HttpWebResponse)hwrequest.GetResponse();

                if (hwresponse.StatusCode == System.Net.HttpStatusCode.OK)
                {
                    System.IO.Stream responseStream = hwresponse.GetResponseStream();
                    System.IO.StreamReader myStreamReader = new System.IO.StreamReader(responseStream);
                    responseData = myStreamReader.ReadToEnd();
                }

                hwresponse.Close();

                // DOWNLOAD PAGE
                hwrequest = (HttpWebRequest)WebRequest.Create(resourceUrl);
                hwrequest.CookieContainer = cookies;
                hwresponse = (HttpWebResponse)hwrequest.GetResponse();
                resourceResponseStream = hwresponse.GetResponseStream();
                
                byteArray = StreamToByteArray(resourceResponseStream);

                resourceResponseStream.Close();
                hwresponse.Close();
                hwresponse.Close();          
            }
            catch (Exception e)
            {
                responseData = "An error occurred: " + e.Message;
            }

            return byteArray;
        }

Hope this helps of people who are in need of similar type of requirements.

Thanks!

Tarun