In case it helps: I had a similar problem in eXist some years ago. My fix was to put the attribute `http-version="1.0"` on the request element. The HTTP client defaults to HTTP v1.1, but for whatever reason, only HTTP 1.0 worked for me. ~Ash ________________________________ From: BaseX-Talk <basex-talk-bounces@mailman.uni-konstanz.de> on behalf of Tim Thompson <timathom@gmail.com> Sent: Wednesday, February 2, 2022 1:59 PM To: Christian Grün <christian.gruen@gmail.com>; BaseX <basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Issue with HTTP client and authentication Ah! Thanks for clarifying that. So, now it is working for GET requests in BaseX, but still not for POST. So, this works: http:send-request( <http:request method="GET" href="{$endpoint}?query={encode-for-uri('select * where {?s ?p ?o} limit 1')}" username="{$user}" password="{$pass}" auth-method="Digest"> </http:request> ) But when I try to POST a request body with Digest auth, it is still failing (httpbin doesn't seem to support testing Digest with POST, unfortunately). The request hangs for a while, then fails with 401 Unauthorized. Wireshark reveals that the first network response from the server is actually a 502 Bad Gateway. Attached are two screenshots from Wireshark: first, the request/response sequence with curl, then with BaseX. Here's what Wireshark shows for the first request via curl: Hypertext Transfer Protocol POST /v1/graphs/sparql HTTP/1.1\r\n [Expert Info (Chat/Sequence): POST /v1/graphs/sparql HTTP/1.1\r\n] [POST /v1/graphs/sparql HTTP/1.1\r\n] [Severity level: Chat] [Group: Sequence] Request Method: POST Request URI: /v1/graphs/sparql Request Version: HTTP/1.1 Host: example.org<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2F&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=j3Wipf2PLgF3njCsb6BTN1J%2BwrOOfe1%2Fcm%2F%2BUiGTMQk%3D&reserved=0>\r\n User-Agent: curl/7.71.1\r\n Accept: */*\r\n Content-type: application/sparql-query\r\n Content-Length: 0\r\n [Content length: 0] \r\n [Full request URI: example.org/v1/graphs/sparql<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2Fv1%2Fgraphs%2Fsparql&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PeCGztKkndWthFeKyUrYUljM2oq4ivkyP1ugAYL64QU%3D&reserved=0>] Then via BaseX: Hypertext Transfer Protocol POST /v1/graphs/sparql HTTP/1.1\r\n [Expert Info (Chat/Sequence): POST /v1/graphs/sparql HTTP/1.1\r\n] [POST /v1/graphs/sparql HTTP/1.1\r\n] [Severity level: Chat] [Group: Sequence] Request Method: POST Request URI: /v1/graphs/sparql Request Version: HTTP/1.1 Content-type: application/sparql-query\r\n Authorization: Digest\r\n User-Agent: Java/1.8.0_121\r\n Host: example.org<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2F&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=j3Wipf2PLgF3njCsb6BTN1J%2BwrOOfe1%2Fcm%2F%2BUiGTMQk%3D&reserved=0>\r\n Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2\r\n Connection: keep-alive\r\n \r\n [Full request URI: example.org/v1/graphs/sparql<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2Fv1%2Fgraphs%2Fsparql&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PeCGztKkndWthFeKyUrYUljM2oq4ivkyP1ugAYL64QU%3D&reserved=0>] I notice that curl doesn't include the Authorization header on the first request, but I am out of my depth here. Best, Tim -- Tim A. Thompson Metadata Librarian Yale University Library On Wed, Feb 2, 2022 at 12:40 PM Christian Grün <christian.gruen@gmail.com<mailto:christian.gruen@gmail.com>> wrote: Hi Tim, Your request should be successful if you omit the send-authorization attribute: “If send-authorization is true (default value is false) and the authentication method supports generating the header Authorization without challenge, the request contains this header. The default value is to send a non-authenticated request, and if the response is an authentication challenge, then only send the credentials in a second message.” [1] If you use Basic authentication, you can reduce the communication to a single request by sending the authorization with the first request. With Digest authentication, there will always be two requests. With the (still to be finalized) HTTP Client module 2.0, send-authorization will disappear, and it will be up to the implementation to decide how many requests will be sent [2]. Hope this helps, Christian [1] http://expath.org/spec/http-client#d2e430<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexpath.org%2Fspec%2Fhttp-client%23d2e430&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=RREH6BeyWOV8juK%2Brf0UuofvBiC2JOv7B1%2BWXLYCjUQ%3D&reserved=0> [2] https://expath.github.io/expath-cg/specs/http-client-2/<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fexpath.github.io%2Fexpath-cg%2Fspecs%2Fhttp-client-2%2F&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=owfYuuSXSpkkPTnQhBIEJ1wBcKPCs4oFnaaVEvndvMA%3D&reserved=0> On Wed, Feb 2, 2022 at 6:19 PM Tim Thompson <timathom@gmail.com<mailto:timathom@gmail.com>> wrote:
Thanks, Andy! That's very handy. So, when I test against httpbin, the same thing happens.
curl request: curl --digest --user user:pw -X GET 'http://httpbin.org/digest-auth/auth/user/pw<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttpbin.org%2Fdigest-auth%2Fauth%2Fuser%2Fpw&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=SWqtnqsC98VlfYr7TeIUggFdoR7dJDZtkNJxz9051jw%3D&reserved=0>'
response: { "authenticated": true, "user": "user" }
let $response := ( http:send-request( <http:request method="GET" href="{$endpoint}" username="user" password="pw" auth-method="Digest" send-authorization="true"> <http:header name="Content-Type" value="{$type}; charset=utf-8"/> </http:request> ) ) return $response
response: <http:response xmlns:http="http://expath.org/ns/http-client<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexpath.org%2Fns%2Fhttp-client&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FNS%2BIceZOx6yP%2Fp4ikxkNqAhYq44F49nUaE3u8iijr0%3D&reserved=0>" status="401" message="UNAUTHORIZED"> <http:header name="Server" value="gunicorn/19.9.0"/> <http:header name="Access-Control-Allow-Origin" value="*"/> <http:header name="Access-Control-Allow-Credentials" value="true"/> <http:header name="WWW-Authenticate" value="Digest realm="me@kennethreitz.com<mailto:quot%3Bme@kennethreitz.com>", nonce="36d59385e96e1689595f4f14fa2921ca", qop="auth", opaque="a21df3287b5cb9cbf8b6a68b2c15fa1d", algorithm=MD5, stale=FALSE"/> <http:header name="Connection" value="keep-alive"/> <http:header name="Set-Cookie" value="fake=fake_value; Path=/"/> <http:header name="Set-Cookie" value="stale_after=never; Path=/"/> <http:header name="Content-Length" value="0"/> <http:header name="Date" value="Wed, 02 Feb 2022 17:14:13 GMT"/> <http:header name="Content-Type" value="text/html; charset=utf-8"/> </http:response>
-- Tim A. Thompson Metadata Librarian Yale University Library
On Wed, Feb 2, 2022 at 12:06 PM Andy Bunce <bunce.andy@gmail.com<mailto:bunce.andy@gmail.com>> wrote:
I find httpbin.org<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttpbin.org%2F&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=5ZA6A7rJL%2B2YD%2BPvtA9wtLUjJVf3EAZw3Fb5ifo4H%2BU%3D&reserved=0> a useful resource to test this kind of thing [1] and [2] /Andy
[1] http://httpbin.org/#/Auth/get_basic_auth__user___passwd_<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttpbin.org%2F%23%2FAuth%2Fget_basic_auth__user___passwd_&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=TU9Itw0Bs4Wbk51bnStj2oFqWMq%2BrdIOxgUwq5fFFrM%3D&reserved=0> [2] http://httpbin.org/basic-auth/user1/mypass<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttpbin.org%2Fbasic-auth%2Fuser1%2Fmypass&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=lH8HeRKkVu%2Fev4Er8lokTCZ0xqyS3HOYtBBqgTSw0PY%3D&reserved=0>
On Wed, 2 Feb 2022 at 16:58, Christian Grün <christian.gruen@gmail.com<mailto:christian.gruen@gmail.com>> wrote:
Good to hear; so at least Digest is detected at this stage ;)
Tim Thompson <timathom@gmail.com<mailto:timathom@gmail.com>> schrieb am Mi., 2. Feb. 2022, 17:54:
With "digest" I get: "[experr:HC0004] Invalid authentication method: digest."
-- Tim A. Thompson Metadata Librarian Yale University Library
On Wed, Feb 2, 2022 at 11:52 AM Christian Grün <christian.gruen@gmail.com<mailto:christian.gruen@gmail.com>> wrote:
Yes, it should still be supported. I assume it doesn't make a difference if you use "Digest" or "digest"?
Tim Thompson <timathom@gmail.com<mailto:timathom@gmail.com>> schrieb am Mi., 2. Feb. 2022, 17:48:
Thanks, Christian. Right, that wasn't a working example. The server I'm querying is behind a firewall, so I can't really reproduce the issue unfortunately.
The BaseX documentation states that the HTTP client does support Digest authentication. However, I did some poking around using Wireshark and limited knowledge of network traffic. When I run the request in curl, the expected protocol is played out, and I see the evidence in Wireshark:
(1) The server responds with "401 Unauthorized" and provides the nonce value. (2) The client (curl) then does its business (supplies the username and password, resends the request, etc.)
But when I submit the request via BaseX, it never gets past step (1). It never seems to send the username and password at all. Is Digest authentication still supported?
Best, Tim
-- Tim A. Thompson Metadata Librarian Yale University Library
On Wed, Feb 2, 2022 at 10:13 AM Christian Grün <christian.gruen@gmail.com<mailto:christian.gruen@gmail.com>> wrote: > > Hi Tim, > > Difficult to tell; both the curl and the XQuery variants give me 404. > Do you think you could provide us with an example that works out of > the box? > > Best, > Christian > > > > On Wed, Feb 2, 2022 at 12:54 AM Tim Thompson <timathom@gmail.com<mailto:timathom@gmail.com>> wrote: > > > > Hello, > > > > I'm trying to post a SPARQL query to an endpoint using Digest authentication with the HTTP client. > > > > The query works fine using curl: > > > > curl --digest --user user:pass -X POST -d@'test.rq' \ > > -H "Content-type: application/sparql-query" \ > > 'http://example.org/sparql<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2Fsparql&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=8uhboyxkhGWHk8HbZRKgDXOAbDRa2UwIigI1H5vWxt0%3D&reserved=0>' > > > > But the equivalent request in BaseX fails with 401 Unauthorized: > > > > let $endpoint := "http://example.org/sparql<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2Fsparql&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=8uhboyxkhGWHk8HbZRKgDXOAbDRa2UwIigI1H5vWxt0%3D&reserved=0>" > > let $user := "user" > > let $pass := "pass" > > let $type := "application/sparql-query" > > > > let $response := ( > > http:send-request( > > <http:request > > method="POST" > > href="{$endpoint}" > > username="{$user}" > > password="{$pass}" > > auth-method="Digest" > > send-authorization="true"> > > <http:header > > name="Content-Type" > > value="{$type}; charset=utf-8"/> > > <http:body > > media-type="{$type}">{ > > ``[ > > select * where {?s ?p ?o} limit 1 > > ]`` > > }</http:body> > > </http:request> > > ) > > ) > > return > > $response > > > > Any ideas about what might be causing the BaseX HTTP client to be denied here? > > > > Thanks in advance, > > Tim > > > > > > -- > > Tim A. Thompson > > Metadata Librarian > > Yale University Library > >