Not understanding why WinHTTP does NOT authenticate certain HTTPS resource

Question

I'd be extremely grateful for any kind of help that may help me resolving the problem.

From Excel VBA code I need to download & parse CSV file from HTTPS site https://redmine.itransition.com/. I try to use WinHTTP to get the file. However, I can't understand why authentication does not work. Here is the piece of related code:

TargetURL = "https://redmine.itransition.com/projects/pmct/time_entries.csv"
Set HTTPReq = CreateObject("WinHttp.WinHttpRequest.5.1")
HTTPReq.Option(4) = 13056 ' WinHttpRequestOption_SslErrorIgnoreFlags 13056: ignore all err, 0: accept no err
HTTPReq.Open "GET", TargetURL, False
HTTPReq.SetCredentials "UN", "PW", 0
HTTPReq.send

returns the following response (only certain strings are listed):

Content-Type: text/html; charset=utf-8
Status: 406
X-Runtime: 5

However, if I send "Cookie" string from Firefox cookie after successful manual authentication using

HTTPReq.setRequestHeader "Cookie", SetCookieString
HTTPReq.send

I easily get the expected file. Of course I'm not happy with such solution, and want to perform true WinHTTP authentication. However, I can't understand what's wrong or what I miss in my code. Most likely I have to use .SetClientCertificate method, but this is unclear for me - which cert is required?

Or, being more general: which WinHTTP methods or functions I should use for debugging to find out which step is blocking / incorrect and prevents me from correct authentication? I spent 2 weeks seeking through MSDN and various resources, but still have no solution.

Thanks in advance for your suggestions!

Answer 1

The above @Alex K. response was exactly what I was looking for soooo long! With the help of Firebug and MSDN I finished with 3 requests:

• GET request to collect authenticity_token data from login page using RegEx
• POST request to authenticate & collect required Cookie string from response
• GET request to finally obtain my beloved CSV

The following piece of code which is working as expected:

Set RegX_AuthToken = CreateObject("VBScript.RegExp")
' Below Pattern w/o double-quotes encoded: (?:input name="authenticity_token" type="hidden" value=")(.*)(?:")
RegX_AuthToken.Pattern = "(?:input name=" & Chr(34) & "authenticity_token" & Chr(34) & " type=" & Chr(34) & "hidden" & Chr(34) & " value=" & Chr(34) & ")(.*)(?:" & Chr(34) & ")"
RegX_AuthToken.IgnoreCase = True
RegX_AuthToken.Global = True

TargetURL = "https://redmine.itransition.com/login"

Set HTTPReq = CreateObject("WinHttp.WinHttpRequest.5.1")
HTTPReq.Open "GET", TargetURL, False
HTTPReq.Send

Set Token_Match = RegX_AuthToken.Execute(HTTPReq.ResponseText)
AuthToken = Token_Match.Item(0).SubMatches.Item(0)

PostData = "authenticity_token=" & AuthToken & "&back_url=https://redmine.itransition.com/" & "&username=" & UN & "&password=" & PW & "&login=Login »"

HTTPReq.Open "POST", TargetURL, False
HTTPReq.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
HTTPReq.Send (PostData)

SetCookieString = HTTPReq.GetResponseHeader("Set-Cookie")

TargetURL = "https://redmine.itransition.com/projects/pmct/time_entries.csv"
HTTPReq.Open "GET", TargetURL, False
HTTPReq.setRequestHeader "Cookie", SetCookieString
HTTPReq.Send

The following URL was helpful in building POST request: http://tkang.blogspot.com/2010/09/sending-http-post-request-with-vba.html

You need to load that page with no credentials, grab what looks like the volatile field authenticity_token from the generated form & post that along with username & password to /login.

Alex K. - thanks again for the brilliant suggestion! (: