We have been using BaseX for about 10 years. We were stuck at 7.9 because the performance of 8 was too restrictive. We tried 9.4 as a last ditch effort to correct some issues and the performance was just as good as 7.9. You made mention that many tcp/ip issues were resolved. We are using BaseX in a multithreaded client server configuration. I have architected a singleton as the server, allowing synchronized clients access to it. Everything is fine until we are under heavy load. We get the following error. IO exception Software caused connection abort: socket write error in mapperProcess query: declare option output:item-separator " "; distinct-values(for $h in //Concept[@NameSpace='Pilotfish.MergeLevel'] for $d in //Concept[@NameSpace=$h/AssociatedConcept/@NameSpace and .=$h/AssociatedConcept/@Concept] where $h/@Value='ByProductCodeDaily' return $d/@Value) retries: 9 We try 8 times, with a 100 msec wait between each attempt, and then generate an exception on the 9th. We have added tcp/ip registry fixes (MaxUserPort, TcpTimedWaitDelay) to the Windows server as a recommended fix for these types of errors. Do you have any other suggestions? Thanks for a great product! Carl R Bondeson IT Analyst 3 Department of Public Health Operation & Support Services Information Technology Phone: 860-509-7434 Carl.Bondeson@ct.gov<mailto:Carl.Bondeson@ct.gov> [cid:image001.jpg@01D73B38.801C1390][cid:image002.png@01D73B38.801C1390]
Hi Carl, It’s difficult to guess what might to on in your code. Do you think you can manage to provide us with a self-contained and minimalized code snippet that allows us to reproduce the problem, or parts of it? Best, Christian On Tue, Apr 27, 2021 at 1:38 PM Bondeson, Carl <Carl.Bondeson@ct.gov> wrote:
We have been using BaseX for about 10 years. We were stuck at 7.9 because the performance of 8 was too restrictive. We tried 9.4 as a last ditch effort to correct some issues and the performance was just as good as 7.9. You made mention that many tcp/ip issues were resolved. We are using BaseX in a multithreaded client server configuration. I have architected a singleton as the server, allowing synchronized clients access to it. Everything is fine until we are under heavy load. We get the following error.
*IO exception Software caused connection abort*: *socket write error* in mapperProcess query: declare option output:item-separator " "; distinct-values(for $h in //Concept[@NameSpace='Pilotfish.MergeLevel'] for $d in //Concept[@NameSpace=$h/AssociatedConcept/@NameSpace and .=$h/AssociatedConcept/@Concept] where $h/@Value='ByProductCodeDaily' return $d/@Value) retries: 9
We try 8 times, with a 100 msec wait between each attempt, and then generate an exception on the 9th. We have added tcp/ip registry fixes (MaxUserPort, TcpTimedWaitDelay) to the Windows server as a recommended fix for these types of errors. Do you have any other suggestions?
Thanks for a great product!
Carl R Bondeson
IT Analyst 3
Department of Public Health
Operation & Support Services
Information Technology
Phone: 860-509-7434
Carl.Bondeson@ct.gov
I wish I could. This is an enterprise implementation handling the processing of 100,000 lab results per day. For each result there are probably 1000 individual queries which handle all aspects of this processing. I am using BaseX to drive a rules engine. Much less complex than Drools, but meets our needs. Each of these queries resides in its own class, which has its own “connection”, to the single server (singleton). This singleton was created since the size of the “database” is prohibitive in loading them per class. This application runs under JBoss, and utilizes Hibernate as the ORM. It interfaces to 4 separate data tiers to collect data, and monitor the operations. I was wondering what in the code can constitute a failure of this type. I have looked in stackoverflow, which only elicits the usual fact that there is a network issue. Where is this exception triggered in the BaseX paradigm? Is this just the usual stack trace from Java which normally has little value in determining a causation? Thanks once again! Carl R Bondeson IT Analyst 3 Department of Public Health Operation & Support Services Information Technology Phone: 860-509-7434 Carl.Bondeson@ct.gov<mailto:Carl.Bondeson@ct.gov> [cid:image001.jpg@01D73B4C.A7D2C3C0][cid:image002.png@01D73B4C.A7D2C3C0] From: Christian Grün <christian.gruen@gmail.com> Sent: Tuesday, April 27, 2021 8:41 AM To: Bondeson, Carl <Carl.Bondeson@ct.gov> Cc: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] 7.9 to 9.4 upgrade EXTERNAL EMAIL: This email originated from outside of the organization. Do not click any links or open any attachments unless you trust the sender and know the content is safe. Hi Carl, It’s difficult to guess what might to on in your code. Do you think you can manage to provide us with a self-contained and minimalized code snippet that allows us to reproduce the problem, or parts of it? Best, Christian On Tue, Apr 27, 2021 at 1:38 PM Bondeson, Carl <Carl.Bondeson@ct.gov<mailto:Carl.Bondeson@ct.gov>> wrote: We have been using BaseX for about 10 years. We were stuck at 7.9 because the performance of 8 was too restrictive. We tried 9.4 as a last ditch effort to correct some issues and the performance was just as good as 7.9. You made mention that many tcp/ip issues were resolved. We are using BaseX in a multithreaded client server configuration. I have architected a singleton as the server, allowing synchronized clients access to it. Everything is fine until we are under heavy load. We get the following error. IO exception Software caused connection abort: socket write error in mapperProcess query: declare option output:item-separator " "; distinct-values(for $h in //Concept[@NameSpace='Pilotfish.MergeLevel'] for $d in //Concept[@NameSpace=$h/AssociatedConcept/@NameSpace and .=$h/AssociatedConcept/@Concept<mailto:.=$h/AssociatedConcept/@Concept>] where $h/@Value='ByProductCodeDaily' return $d/@Value) retries: 9 We try 8 times, with a 100 msec wait between each attempt, and then generate an exception on the 9th. We have added tcp/ip registry fixes (MaxUserPort, TcpTimedWaitDelay) to the Windows server as a recommended fix for these types of errors. Do you have any other suggestions? Thanks for a great product! Carl R Bondeson IT Analyst 3 Department of Public Health Operation & Support Services Information Technology Phone: 860-509-7434 Carl.Bondeson@ct.gov<mailto:Carl.Bondeson@ct.gov> [cid:image001.jpg@01D73B4C.A7D2C3C0][cid:image002.png@01D73B4C.A7D2C3C0]
Where is this exception triggered in the BaseX paradigm? Is this just the usual stack trace from Java which normally has little value in determining a causation?
…close to impossible for me to say more about it, I’m sorry. You could launch your BaseX instance(s) with debugging enabled [1] and share the full stack traces with us, maybe that gives us more hints. Best, Christian [1] https://docs.basex.org/wiki/Options#DEBUG
On Tue, 2021-04-27 at 14:07 +0000, Bondeson, Carl wrote:
Where is this exception triggered in the BaseX paradigm?
The message comes from the Windows-native socket implementation. Some things to try - * a server on the same machine as the test client * a server on Linux instead of Windows * run the client on Linux instead of Windows any of these may still fail but with different error messages. If everything works on the local machine it might be a problem with a firewall, or a virus checker, especially if the problem goes away on Linux. It's also possible to trace network traffic to try & work out if the problem is in the client or in the server. Obvious thing to chek - make sure you're using the right classes in the client to communicate with BaseX. Which JVM version exactly are you using? Liam -- Liam Quin, https://www.delightfulcomputing.com/ Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: http://www.fromoldbooks.org
Thanks for your valuable input I would have gone to Linux a long time ago but our infrastructure won't support it. We are currently using RedHat Java 11.05 Java, running under JBoss 7.2 Carl R Bondeson IT Analyst 3 Department of Public Health Operation & Support Services Information Technology Phone: 860-509-7434 Carl.Bondeson@ct.gov -----Original Message----- From: Liam R. E. Quin <liam@fromoldbooks.org> Sent: Tuesday, April 27, 2021 11:24 AM To: Bondeson, Carl <Carl.Bondeson@ct.gov> Cc: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] 7.9 to 9.4 upgrade EXTERNAL EMAIL: This email originated from outside of the organization. Do not click any links or open any attachments unless you trust the sender and know the content is safe. On Tue, 2021-04-27 at 14:07 +0000, Bondeson, Carl wrote:
Where is this exception triggered in the BaseX paradigm?
The message comes from the Windows-native socket implementation. Some things to try - * a server on the same machine as the test client * a server on Linux instead of Windows * run the client on Linux instead of Windows any of these may still fail but with different error messages. If everything works on the local machine it might be a problem with a firewall, or a virus checker, especially if the problem goes away on Linux. It's also possible to trace network traffic to try & work out if the problem is in the client or in the server. Obvious thing to chek - make sure you're using the right classes in the client to communicate with BaseX. Which JVM version exactly are you using? Liam -- Liam Quin, https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.delightfulcomputing.com%2F&data=04%7C01%7CCarl.Bondeson%40ct.gov%7C2d3cda6fa61646db0fc708d909908ada%7C118b7cfaa3dd48b9b02631ff69bb738b%7C0%7C0%7C637551338694018172%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=pJjRteuU5wAu3hOyj9BQ2sQcZwH0RgMKLSB2GSBctMc%3D&reserved=0 Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.fromoldbooks.org%2F&data=04%7C01%7CCarl.Bondeson%40ct.gov%7C2d3cda6fa61646db0fc708d909908ada%7C118b7cfaa3dd48b9b02631ff69bb738b%7C0%7C0%7C637551338694028130%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=mpB1tFgTMBQjhu5YyLS%2FRLvM68iXQkpoy%2BAa1cLTYHU%3D&reserved=0
On Tue, 2021-04-27 at 15:29 +0000, Bondeson, Carl wrote:
Thanks for your valuable input
I would have gone to Linux a long time ago but our infrastructure won't support it.
Well, i was talking about setting up a test system. -- Liam Quin, https://www.delightfulcomputing.com/ Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: http://www.fromoldbooks.org
Tech support won't allow that. We are managed in a VmWare VSphere environment. They don't even have the ability to clone a Linux distro. They only have Windows servers as available distributions. I have asked over and over again. No Unix knowledge in our tech support group. Carl R Bondeson IT Analyst 3 Department of Public Health Operation & Support Services Information Technology Phone: 860-509-7434 Carl.Bondeson@ct.gov -----Original Message----- From: Liam R. E. Quin <liam@fromoldbooks.org> Sent: Tuesday, April 27, 2021 12:06 PM To: Bondeson, Carl <Carl.Bondeson@ct.gov> Cc: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] 7.9 to 9.4 upgrade EXTERNAL EMAIL: This email originated from outside of the organization. Do not click any links or open any attachments unless you trust the sender and know the content is safe. On Tue, 2021-04-27 at 15:29 +0000, Bondeson, Carl wrote:
Thanks for your valuable input
I would have gone to Linux a long time ago but our infrastructure won't support it.
Well, i was talking about setting up a test system. -- Liam Quin, https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.delightfulcomputing.com%2F&data=04%7C01%7CCarl.Bondeson%40ct.gov%7C7694a88991424031bfb408d909968d67%7C118b7cfaa3dd48b9b02631ff69bb738b%7C0%7C0%7C637551364492087526%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=CK3kSn%2FiuulXddYOWfdwvVvZ80%2B93rjRptAnGjIIN6s%3D&reserved=0 Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.fromoldbooks.org%2F&data=04%7C01%7CCarl.Bondeson%40ct.gov%7C7694a88991424031bfb408d909968d67%7C118b7cfaa3dd48b9b02631ff69bb738b%7C0%7C0%7C637551364492087526%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=LY%2F1MVq4gvrFcphuxh6zi9Lgakie%2FXEAFArR3Yv62fE%3D&reserved=0
Modern windows contains a ubuntu linux kernel right out of the box. You just have to enable it. -----Original Message----- From: BaseX-Talk <basex-talk-bounces@mailman.uni-konstanz.de> On Behalf Of Bondeson, Carl Sent: Tuesday, April 27, 2021 9:49 AM To: Liam R. E. Quin <liam@fromoldbooks.org> Cc: basex-talk@mailman.uni-konstanz.de Subject: [EXTERNAL] Re: [basex-talk] 7.9 to 9.4 upgrade Tech support won't allow that. We are managed in a VmWare VSphere environment. They don't even have the ability to clone a Linux distro. They only have Windows servers as available distributions. I have asked over and over again. No Unix knowledge in our tech support group. Carl R Bondeson IT Analyst 3 Department of Public Health Operation & Support Services Information Technology Phone: 860-509-7434 Carl.Bondeson@ct.gov -----Original Message----- From: Liam R. E. Quin <liam@fromoldbooks.org> Sent: Tuesday, April 27, 2021 12:06 PM To: Bondeson, Carl <Carl.Bondeson@ct.gov> Cc: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] 7.9 to 9.4 upgrade EXTERNAL EMAIL: This email originated from outside of the organization. Do not click any links or open any attachments unless you trust the sender and know the content is safe. On Tue, 2021-04-27 at 15:29 +0000, Bondeson, Carl wrote:
Thanks for your valuable input
I would have gone to Linux a long time ago but our infrastructure won't support it.
Well, i was talking about setting up a test system. -- Liam Quin, https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.delightfulcomputing.com%2F&data=04%7C01%7Cpeter.villadsen%40microsoft.com%7C212c312bd4934267cfe408d9099c658a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637551391050001559%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=s0CBulac%2B0hxOdun9ENGNcmgwT6NCs4vv3oeofcgqFo%3D&reserved=0 Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.fromoldbooks.org%2F&data=04%7C01%7Cpeter.villadsen%40microsoft.com%7C212c312bd4934267cfe408d9099c658a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637551391050001559%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hL%2FibqBZ89QnirTrqIsml1mOnh0VMQYboHVoRvBAVqg%3D&reserved=0
For best Linux on Windows experience, I can recommend Multipass.run (from Ubuntu) Peter Villadsen <Peter.Villadsen@microsoft.com> schrieb am Di., 27. Apr. 2021, 19:37:
Modern windows contains a ubuntu linux kernel right out of the box. You just have to enable it.
-----Original Message----- From: BaseX-Talk <basex-talk-bounces@mailman.uni-konstanz.de> On Behalf Of Bondeson, Carl Sent: Tuesday, April 27, 2021 9:49 AM To: Liam R. E. Quin <liam@fromoldbooks.org> Cc: basex-talk@mailman.uni-konstanz.de Subject: [EXTERNAL] Re: [basex-talk] 7.9 to 9.4 upgrade
Tech support won't allow that. We are managed in a VmWare VSphere environment. They don't even have the ability to clone a Linux distro. They only have Windows servers as available distributions. I have asked over and over again. No Unix knowledge in our tech support group.
Carl R Bondeson IT Analyst 3 Department of Public Health Operation & Support Services Information Technology Phone: 860-509-7434 Carl.Bondeson@ct.gov
-----Original Message----- From: Liam R. E. Quin <liam@fromoldbooks.org> Sent: Tuesday, April 27, 2021 12:06 PM To: Bondeson, Carl <Carl.Bondeson@ct.gov> Cc: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] 7.9 to 9.4 upgrade
EXTERNAL EMAIL: This email originated from outside of the organization. Do not click any links or open any attachments unless you trust the sender and know the content is safe.
On Tue, 2021-04-27 at 15:29 +0000, Bondeson, Carl wrote:
Thanks for your valuable input
I would have gone to Linux a long time ago but our infrastructure won't support it.
Well, i was talking about setting up a test system.
-- Liam Quin, https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.delightfulcomputing.com%2F&data=04%7C01%7Cpeter.villadsen%40microsoft.com%7C212c312bd4934267cfe408d9099c658a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637551391050001559%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=s0CBulac%2B0hxOdun9ENGNcmgwT6NCs4vv3oeofcgqFo%3D&reserved=0 Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.fromoldbooks.org%2F&data=04%7C01%7Cpeter.villadsen%40microsoft.com%7C212c312bd4934267cfe408d9099c658a%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637551391050001559%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hL%2FibqBZ89QnirTrqIsml1mOnh0VMQYboHVoRvBAVqg%3D&reserved=0
Thanks for all the ideas. The solution I was able to implement was a throttling mechanism to limit the amount of data per unit time. If you have 1000 things to process in 1 minute, don't feed it all 1000 in 1 chunk. Split them into 80 chunks and feed them every 12 seconds. This allows the system to operate more quiescently. Carl R Bondeson IT Analyst 3 Department of Public Health Operation & Support Services Information Technology Phone: 860-509-7434 Carl.Bondeson@ct.gov -----Original Message----- From: BaseX-Talk <basex-talk-bounces@mailman.uni-konstanz.de> On Behalf Of Bondeson, Carl Sent: Tuesday, April 27, 2021 12:49 PM To: Liam R. E. Quin <liam@fromoldbooks.org> Cc: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] 7.9 to 9.4 upgrade Tech support won't allow that. We are managed in a VmWare VSphere environment. They don't even have the ability to clone a Linux distro. They only have Windows servers as available distributions. I have asked over and over again. No Unix knowledge in our tech support group. Carl R Bondeson IT Analyst 3 Department of Public Health Operation & Support Services Information Technology Phone: 860-509-7434 Carl.Bondeson@ct.gov -----Original Message----- From: Liam R. E. Quin <liam@fromoldbooks.org> Sent: Tuesday, April 27, 2021 12:06 PM To: Bondeson, Carl <Carl.Bondeson@ct.gov> Cc: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] 7.9 to 9.4 upgrade EXTERNAL EMAIL: This email originated from outside of the organization. Do not click any links or open any attachments unless you trust the sender and know the content is safe. On Tue, 2021-04-27 at 15:29 +0000, Bondeson, Carl wrote:
Thanks for your valuable input
I would have gone to Linux a long time ago but our infrastructure won't support it.
Well, i was talking about setting up a test system. -- Liam Quin, https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.delightfulcomputing.com%2F&data=04%7C01%7CCarl.Bondeson%40ct.gov%7C40b3c804f117442caade08d9099c683b%7C118b7cfaa3dd48b9b02631ff69bb738b%7C0%7C0%7C637551389651154964%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=vvBXWZo0NMZldD62iN3xH1RFxHkzTZOdytDoS1Tv%2Bzg%3D&reserved=0 Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.fromoldbooks.org%2F&data=04%7C01%7CCarl.Bondeson%40ct.gov%7C40b3c804f117442caade08d9099c683b%7C118b7cfaa3dd48b9b02631ff69bb738b%7C0%7C0%7C637551389651164917%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3EydQ2M3VmIzCgtwD80rq6QTMfrWYSIVzP%2BMPnYdMoM%3D&reserved=0
participants (5)
-
Bondeson, Carl -
Christian Grün -
Liam R. E. Quin -
Maximilian Gärber -
Peter Villadsen