Re: [basex-talk] http:send-request - problem with response
Hi Bogdan (cc to the list), Thanks for digging deeper. I noticed that the standard Java function that we used returned only one value per header field, and dropped the others [1]. I managed to fix this in the latest stable snapshot; could you give it a try [2]? BaseX 9.3 will be released end of November. Cheers, Christian [1] http://files.basex.org/releases/latest/ [2] https://github.com/BaseXdb/basex/issues/1751 On Wed, Nov 13, 2019 at 4:26 PM Bogdan Bogucki <bbogucki@hussar.pl> wrote:
Hi Christian, Lack of many redirection is not a big problem. I can handle it manually. Problem is not complete response header form first request.
Browser make 3 redirection I am talking about first.
Please take a look.
Response header from browser (first request) - contains three fields Set Cookies:
set-cookie __cfduid=daf54110b2a87d66c2a53…; domain=.pracuj.pl; HttpOnly
set-cookie _yaic=13; expires=Fri, 31-Dec-9999 23:59:59 GMT; path=/
set-cookie _urnadiam=A; domain=.pracuj.pl…3:00:00 GMT; path=/; HttpOnly
but response header form http:send-request contains only last one:
<http:header name="Set-Cookie" value="_urnadiam=B; domain=.pracuj.pl; expires=Tue, 31-Dec-2019 23:00:00 GMT; path=/; HttpOnly"/>
I need information from previous set-cookie fields to make manual redirection passing information about sessions to next requests.
Regards
Bogdan Bogucki
W dniu 29.10.2019 o 03:06, Christian Grün pisze:
Hi Bogdan,
The current http:send-request implementation is based on the default Java HttpURLConnection, which does not resolve redirects that use different protocols [1]. This is the reason why your request will not be fully processed (as it e.g. happens when you use the browser).
It seems that your initial request to the https protocol returns a 302 redirect to a (now unsafe) http URL, which returns another redirect to https. I don’t know who maintains the discussed web site, but it could be worth contacting the admins and asking them if they could update and simplify their redirect policy.
Hope this helps, Christian
[1] https://stackoverflow.com/a/1884427/1018790
On Fri, Oct 25, 2019 at 1:52 PM Bogdan Bogucki <bbogucki@hussar.pl> wrote:
Hello,
I encounter problem with http:send-request function. I need handle multiple forward requests (3) with cookies.
First request returns cookies information which are required with next requests. Output from http:send-request dosen't contains all fields which are returned by server.
Response from browser is flowing: cache-control private cf-cache-status DYNAMIC cf-ray 52b3cedc5b9bcc9f-WAW content-type text/html; charset=utf-8 date Fri, 25 Oct 2019 11:21:37 GMT expect-ct max-age=604800, report-uri="ht….com/cdn-cgi/beacon/expect-ct" location http://www.pracuj.pl/praca/jav…eloper-warszawa,oferta,7171988 server cloudflare set-cookie __cfduid=daf54110b2a87d66c2a53…; domain=.pracuj.pl; HttpOnly set-cookie _yaic=13; expires=Fri, 31-Dec-9999 23:59:59 GMT; path=/ set-cookie _urnadiam=A; domain=.pracuj.pl…3:00:00 GMT; path=/; HttpOnly x-aspnet-version 4.0.30319 x-aspnetmvc-version 5.2 X-Firefox-Spdy h2 x-powered-by ASP.NET x-ua-compatible IE=edge
Request:
http:send-request( <http:request method='get' follow-redirect='false'> <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0"/> <http:header name="Accept" value="text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"/> <http:header name="Accept-Languag" value="pl,en-US;q=0.7,en;q=0.3"/> <http:header name="DNT" value="1"/> <http:header name="Connection" value="keep-alive"/> <http:header name="Upgrade-Insecure-Requests" value="1"/> <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'/> </http:request>, 'https://www.pracuj.pl/praca/java-developer-warszawa,oferta,7171988')
Response form http:send-request is:
<http:response xmlns:http="http://expath.org/ns/http-client" status="302" message="Found"> <http:header name="Transfer-Encoding" value="chunked"/> <http:header name="CF-RAY" value="52b3d0de097fb839-WAW"/> <http:header name="Server" value="cloudflare"/> <http:header name="Connection" value="keep-alive"/> <http:header name="Date" value="Fri, 25 Oct 2019 11:23:00 GMT"/> <http:header name="X-AspNetMvc-Version" value="5.2"/> <http:header name="X-UA-Compatible" value="IE=edge"/> <http:header name="CF-Cache-Status" value="DYNAMIC"/> <http:header name="Cache-Control" value="private"/> <http:header name="X-AspNet-Version" value="4.0.30319"/> <http:header name="Set-Cookie" value="_urnadiam=B; domain=.pracuj.pl; expires=Tue, 31-Dec-2019 23:00:00 GMT; path=/; HttpOnly"/> <http:header name="Expect-CT" value="max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct""/> <http:header name="X-Powered-By" value="ASP.NET"/> <http:header name="Content-Type" value="text/html; charset=utf-8"/> <http:body media-type="text/html"/> </http:response>
Two set-cookie fields is missing.
How should I invoke http:send-request to receive proper result ?
Regards
Bogdan
Hi Christian, It works. Thank you for quick fix. Best Regards Bogdan Bogucki W dniu 11/13/2019 o 17:42, Christian Grün pisze:
Hi Bogdan (cc to the list),
Thanks for digging deeper. I noticed that the standard Java function that we used returned only one value per header field, and dropped the others [1].
I managed to fix this in the latest stable snapshot; could you give it a try [2]?
BaseX 9.3 will be released end of November.
Cheers, Christian
[1] http://files.basex.org/releases/latest/ [2] https://github.com/BaseXdb/basex/issues/1751
On Wed, Nov 13, 2019 at 4:26 PM Bogdan Bogucki <bbogucki@hussar.pl> wrote:
Hi Christian, Lack of many redirection is not a big problem. I can handle it manually. Problem is not complete response header form first request.
Browser make 3 redirection I am talking about first.
Please take a look.
Response header from browser (first request) - contains three fields Set Cookies:
set-cookie __cfduid=daf54110b2a87d66c2a53…; domain=.pracuj.pl; HttpOnly
set-cookie _yaic=13; expires=Fri, 31-Dec-9999 23:59:59 GMT; path=/
set-cookie _urnadiam=A; domain=.pracuj.pl…3:00:00 GMT; path=/; HttpOnly
but response header form http:send-request contains only last one:
<http:header name="Set-Cookie" value="_urnadiam=B; domain=.pracuj.pl; expires=Tue, 31-Dec-2019 23:00:00 GMT; path=/; HttpOnly"/>
I need information from previous set-cookie fields to make manual redirection passing information about sessions to next requests.
Regards
Bogdan Bogucki
W dniu 29.10.2019 o 03:06, Christian Grün pisze:
Hi Bogdan,
The current http:send-request implementation is based on the default Java HttpURLConnection, which does not resolve redirects that use different protocols [1]. This is the reason why your request will not be fully processed (as it e.g. happens when you use the browser).
It seems that your initial request to the https protocol returns a 302 redirect to a (now unsafe) http URL, which returns another redirect to https. I don’t know who maintains the discussed web site, but it could be worth contacting the admins and asking them if they could update and simplify their redirect policy.
Hope this helps, Christian
[1] https://stackoverflow.com/a/1884427/1018790
On Fri, Oct 25, 2019 at 1:52 PM Bogdan Bogucki <bbogucki@hussar.pl> wrote:
Hello,
I encounter problem with http:send-request function. I need handle multiple forward requests (3) with cookies.
First request returns cookies information which are required with next requests. Output from http:send-request dosen't contains all fields which are returned by server.
Response from browser is flowing: cache-control private cf-cache-status DYNAMIC cf-ray 52b3cedc5b9bcc9f-WAW content-type text/html; charset=utf-8 date Fri, 25 Oct 2019 11:21:37 GMT expect-ct max-age=604800, report-uri="ht….com/cdn-cgi/beacon/expect-ct" location http://www.pracuj.pl/praca/jav…eloper-warszawa,oferta,7171988 server cloudflare set-cookie __cfduid=daf54110b2a87d66c2a53…; domain=.pracuj.pl; HttpOnly set-cookie _yaic=13; expires=Fri, 31-Dec-9999 23:59:59 GMT; path=/ set-cookie _urnadiam=A; domain=.pracuj.pl…3:00:00 GMT; path=/; HttpOnly x-aspnet-version 4.0.30319 x-aspnetmvc-version 5.2 X-Firefox-Spdy h2 x-powered-by ASP.NET x-ua-compatible IE=edge
Request:
http:send-request( <http:request method='get' follow-redirect='false'> <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0"/> <http:header name="Accept" value="text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"/> <http:header name="Accept-Languag" value="pl,en-US;q=0.7,en;q=0.3"/> <http:header name="DNT" value="1"/> <http:header name="Connection" value="keep-alive"/> <http:header name="Upgrade-Insecure-Requests" value="1"/> <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'/> </http:request>, 'https://www.pracuj.pl/praca/java-developer-warszawa,oferta,7171988')
Response form http:send-request is:
<http:response xmlns:http="http://expath.org/ns/http-client" status="302" message="Found"> <http:header name="Transfer-Encoding" value="chunked"/> <http:header name="CF-RAY" value="52b3d0de097fb839-WAW"/> <http:header name="Server" value="cloudflare"/> <http:header name="Connection" value="keep-alive"/> <http:header name="Date" value="Fri, 25 Oct 2019 11:23:00 GMT"/> <http:header name="X-AspNetMvc-Version" value="5.2"/> <http:header name="X-UA-Compatible" value="IE=edge"/> <http:header name="CF-Cache-Status" value="DYNAMIC"/> <http:header name="Cache-Control" value="private"/> <http:header name="X-AspNet-Version" value="4.0.30319"/> <http:header name="Set-Cookie" value="_urnadiam=B; domain=.pracuj.pl; expires=Tue, 31-Dec-2019 23:00:00 GMT; path=/; HttpOnly"/> <http:header name="Expect-CT" value="max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct""/> <http:header name="X-Powered-By" value="ASP.NET"/> <http:header name="Content-Type" value="text/html; charset=utf-8"/> <http:body media-type="text/html"/> </http:response>
Two set-cookie fields is missing.
How should I invoke http:send-request to receive proper result ?
Regards
Bogdan
participants (2)
-
Bogdan Bogucki -
Christian Grün