Thanks, Ash! That attribute doesn't seem to be available in BaseX, as far as I can tell.
-- Tim A. Thompson Metadata Librarian Yale University Library
On Wed, Feb 2, 2022 at 2:21 PM Clark, Ash as.clark@northeastern.edu wrote:
In case it helps: I had a similar problem in eXist some years ago. My fix was to put the attribute `http-version="1.0"` on the request element. The HTTP client defaults to HTTP v1.1, but for whatever reason, only HTTP 1.0 worked for me.
~Ash
*From:* BaseX-Talk basex-talk-bounces@mailman.uni-konstanz.de on behalf of Tim Thompson timathom@gmail.com *Sent:* Wednesday, February 2, 2022 1:59 PM *To:* Christian Grün christian.gruen@gmail.com; BaseX < basex-talk@mailman.uni-konstanz.de> *Subject:* Re: [basex-talk] Issue with HTTP client and authentication
Ah! Thanks for clarifying that. So, now it is working for GET requests in BaseX, but still not for POST.
So, this works: http:send-request( <http:request method="GET" href="{$endpoint}?query={encode-for-uri('select * where {?s ?p ?o} limit 1')}" username="{$user}" password="{$pass}" auth-method="Digest"> </http:request> )
But when I try to POST a request body with Digest auth, it is still failing (httpbin doesn't seem to support testing Digest with POST, unfortunately). The request hangs for a while, then fails with 401 Unauthorized. Wireshark reveals that the first network response from the server is actually a 502 Bad Gateway.
Attached are two screenshots from Wireshark: first, the request/response sequence with curl, then with BaseX.
Here's what Wireshark shows for the first request via curl:
Hypertext Transfer Protocol POST /v1/graphs/sparql HTTP/1.1\r\n [Expert Info (Chat/Sequence): POST /v1/graphs/sparql HTTP/1.1\r\n] [POST /v1/graphs/sparql HTTP/1.1\r\n] [Severity level: Chat] [Group: Sequence] Request Method: POST Request URI: /v1/graphs/sparql Request Version: HTTP/1.1 Host: example.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2F&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=j3Wipf2PLgF3njCsb6BTN1J%2BwrOOfe1%2Fcm%2F%2BUiGTMQk%3D&reserved=0 \r\n User-Agent: curl/7.71.1\r\n Accept: */*\r\n Content-type: application/sparql-query\r\n Content-Length: 0\r\n [Content length: 0] \r\n [Full request URI: example.org/v1/graphs/sparql https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2Fv1%2Fgraphs%2Fsparql&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PeCGztKkndWthFeKyUrYUljM2oq4ivkyP1ugAYL64QU%3D&reserved=0 ]
Then via BaseX:
Hypertext Transfer Protocol POST /v1/graphs/sparql HTTP/1.1\r\n [Expert Info (Chat/Sequence): POST /v1/graphs/sparql HTTP/1.1\r\n] [POST /v1/graphs/sparql HTTP/1.1\r\n] [Severity level: Chat] [Group: Sequence] Request Method: POST Request URI: /v1/graphs/sparql Request Version: HTTP/1.1 Content-type: application/sparql-query\r\n Authorization: Digest\r\n User-Agent: Java/1.8.0_121\r\n Host: example.org https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2F&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=j3Wipf2PLgF3njCsb6BTN1J%2BwrOOfe1%2Fcm%2F%2BUiGTMQk%3D&reserved=0 \r\n Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2\r\n Connection: keep-alive\r\n \r\n [Full request URI: example.org/v1/graphs/sparql https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2Fv1%2Fgraphs%2Fsparql&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PeCGztKkndWthFeKyUrYUljM2oq4ivkyP1ugAYL64QU%3D&reserved=0 ]
I notice that curl doesn't include the Authorization header on the first request, but I am out of my depth here.
Best, Tim
-- Tim A. Thompson Metadata Librarian Yale University Library
On Wed, Feb 2, 2022 at 12:40 PM Christian Grün christian.gruen@gmail.com wrote:
Hi Tim,
Your request should be successful if you omit the send-authorization attribute:
“If send-authorization is true (default value is false) and the authentication method supports generating the header Authorization without challenge, the request contains this header. The default value is to send a non-authenticated request, and if the response is an authentication challenge, then only send the credentials in a second message.” [1]
If you use Basic authentication, you can reduce the communication to a single request by sending the authorization with the first request. With Digest authentication, there will always be two requests.
With the (still to be finalized) HTTP Client module 2.0, send-authorization will disappear, and it will be up to the implementation to decide how many requests will be sent [2].
Hope this helps, Christian
[1] http://expath.org/spec/http-client#d2e430 https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexpath.org%2Fspec%2Fhttp-client%23d2e430&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=RREH6BeyWOV8juK%2Brf0UuofvBiC2JOv7B1%2BWXLYCjUQ%3D&reserved=0 [2] https://expath.github.io/expath-cg/specs/http-client-2/ https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fexpath.github.io%2Fexpath-cg%2Fspecs%2Fhttp-client-2%2F&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=owfYuuSXSpkkPTnQhBIEJ1wBcKPCs4oFnaaVEvndvMA%3D&reserved=0
On Wed, Feb 2, 2022 at 6:19 PM Tim Thompson timathom@gmail.com wrote:
Thanks, Andy! That's very handy. So, when I test against httpbin, the
same thing happens.
curl request: curl --digest --user user:pw -X GET '
http://httpbin.org/digest-auth/auth/user/pw https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttpbin.org%2Fdigest-auth%2Fauth%2Fuser%2Fpw&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=SWqtnqsC98VlfYr7TeIUggFdoR7dJDZtkNJxz9051jw%3D&reserved=0 '
response: { "authenticated": true, "user": "user" }
BaseX request: let $endpoint := "http://httpbin.org/digest-auth/auth/user/pw
let $response := ( http:send-request( <http:request method="GET" href="{$endpoint}" username="user" password="pw" auth-method="Digest" send-authorization="true"> <http:header name="Content-Type" value="{$type}; charset=utf-8"/> </http:request> ) ) return $response
response: <http:response xmlns:http="http://expath.org/ns/http-client
<http:header name="Server" value="gunicorn/19.9.0"/> <http:header name="Access-Control-Allow-Origin" value="*"/> <http:header name="Access-Control-Allow-Credentials" value="true"/> <http:header name="WWW-Authenticate" value="Digest realm=&
quot;me@kennethreitz.com", nonce="36d59385e96e1689595f4f14fa2921ca", qop="auth", opaque="a21df3287b5cb9cbf8b6a68b2c15fa1d", algorithm=MD5, stale=FALSE"/>
<http:header name="Connection" value="keep-alive"/> <http:header name="Set-Cookie" value="fake=fake_value; Path=/"/> <http:header name="Set-Cookie" value="stale_after=never; Path=/"/> <http:header name="Content-Length" value="0"/> <http:header name="Date" value="Wed, 02 Feb 2022 17:14:13 GMT"/> <http:header name="Content-Type" value="text/html; charset=utf-8"/> </http:response>
-- Tim A. Thompson Metadata Librarian Yale University Library
On Wed, Feb 2, 2022 at 12:06 PM Andy Bunce bunce.andy@gmail.com wrote:
I find httpbin.org
https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttpbin.org%2F&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=5ZA6A7rJL%2B2YD%2BPvtA9wtLUjJVf3EAZw3Fb5ifo4H%2BU%3D&reserved=0 a useful resource to test this kind of thing [1] and [2]
/Andy
[1] http://httpbin.org/#/Auth/get_basic_auth__user___passwd_
On Wed, 2 Feb 2022 at 16:58, Christian Grün christian.gruen@gmail.com
wrote:
Good to hear; so at least Digest is detected at this stage ;)
Tim Thompson timathom@gmail.com schrieb am Mi., 2. Feb. 2022, 17:54:
With "digest" I get: "[experr:HC0004] Invalid authentication method:
digest."
-- Tim A. Thompson Metadata Librarian Yale University Library
On Wed, Feb 2, 2022 at 11:52 AM Christian Grün <
christian.gruen@gmail.com> wrote:
Yes, it should still be supported. I assume it doesn't make a
difference if you use "Digest" or "digest"?
Tim Thompson timathom@gmail.com schrieb am Mi., 2. Feb. 2022,
17:48:
> > Thanks, Christian. Right, that wasn't a working example. The server
I'm querying is behind a firewall, so I can't really reproduce the issue unfortunately.
> > The BaseX documentation states that the HTTP client does support
Digest authentication. However, I did some poking around using Wireshark and limited knowledge of network traffic. When I run the request in curl, the expected protocol is played out, and I see the evidence in Wireshark:
> > (1) The server responds with "401 Unauthorized" and provides the
nonce value.
> (2) The client (curl) then does its business (supplies the username
and password, resends the request, etc.)
> > But when I submit the request via BaseX, it never gets past step
(1). It never seems to send the username and password at all. Is Digest authentication still supported?
> > Best, > Tim > > > -- > Tim A. Thompson > Metadata Librarian > Yale University Library > > > On Wed, Feb 2, 2022 at 10:13 AM Christian Grün <
christian.gruen@gmail.com> wrote:
>> >> Hi Tim, >> >> Difficult to tell; both the curl and the XQuery variants give me
>> Do you think you could provide us with an example that works out of >> the box? >> >> Best, >> Christian >> >> >> >> On Wed, Feb 2, 2022 at 12:54 AM Tim Thompson timathom@gmail.com
wrote:
>> > >> > Hello, >> > >> > I'm trying to post a SPARQL query to an endpoint using Digest
authentication with the HTTP client.
>> > >> > The query works fine using curl: >> > >> > curl --digest --user user:pass -X POST -d@'test.rq' \ >> > -H "Content-type: application/sparql-query" \ >> > 'http://example.org/sparql
>> > >> > But the equivalent request in BaseX fails with 401 Unauthorized: >> > >> > let $endpoint := "http://example.org/sparql
>> > let $user := "user" >> > let $pass := "pass" >> > let $type := "application/sparql-query" >> > >> > let $response := ( >> > http:send-request( >> > http:request > > method="POST" >> > href="{$endpoint}" >> > username="{$user}" >> > password="{$pass}" >> > auth-method="Digest" >> > send-authorization="true"> >> > http:header > > name="Content-Type" >> > value="{$type}; charset=utf-8"/> >> > http:body > > media-type="{$type}">{ >> > ``[ >> > select * where {?s ?p ?o} limit 1 >> > ]`` >> > }</http:body> >> > </http:request> >> > ) >> > ) >> > return >> > $response >> > >> > Any ideas about what might be causing the BaseX HTTP client to
be denied here?
>> > >> > Thanks in advance, >> > Tim >> > >> > >> > -- >> > Tim A. Thompson >> > Metadata Librarian >> > Yale University Library >> >