- BaseX-Talk - mailman.uni-konstanz.de

Fwd: Storing sequences of atomic type into BaseX ?
by jean-marc Mercier 08 Jan '15

08 Jan '15

Christian, Excellent ! thanks a lot ! prof:time(local:savebin((1 to 10000000),$binfile)) ,prof:time(local:loadbin($binfile)) output is now 930.71 ms 907.04 ms 2015-01-08 22:11 GMT+01:00 Christian Grün <christian.gruen(a)gmail.com>: > Hi Jean-Marc, > > > 2) However unserialization seems to perform awfully, or I do not know > how to > > do it properly. > > I haven't tried the query, but my guess is that the binary data is > streamed again and again. The stream:materialize function should help > you [1]: > > let $data := stream:materialize(file:read-binary($file)) > ... > > Christian > > [1] http://docs.basex.org/wiki/Streaming_Module > > > > > declare function local:savebin($seq,$file as xs:string) { > > file:write-binary($file,bin:join( (bin:pack-integer(count($seq),4) > ,$seq > > ! bin:pack-integer(.,4)))) > > }; > > declare function local:loadbin($file as xs:string) { > > let $data := file:read-binary($file) > > let $size:= bin:unpack-integer($data,0,4) > > let $seq := for $i in (1 to ($size)) return > > bin:unpack-integer($data,$i*4,4) > > return count($seq) > > }; > > > > prof:time(local:savebin((1 to 100000),"Bin.dat")) > > ,prof:time(local:loadbin("Bin.dat")) > > > > output : > > > > 46.38 ms > > 10775.12 ms > > 100000 > > > > To compare, unserializing a sequence (1 to 10 000 000) stored in a file > as a > > big string using fn:tokenize takes about 10 sec (100 x faster). Did I > > mistake something ? > > > > > > 2015-01-08 16:44 GMT+01:00 Christian Grün <christian.gruen(a)gmail.com>: > >> > >> > This way of doing stores integers as string, then call a cast string / > >> > integer to unserialize it. For large integer list (I am dealing with > >> > lists > >> > of size 134 Mo), it is quite time and mem consuming. > >> > > >> > I was wondering if there exists a more efficient way to store and > >> > retrieve > >> > atomic list into BaseX ? > >> > >> One alternative is to store the integers in a binary file: > >> > >> let $size := 4 > >> let $data := bin:join( > >> for $n in 1 to 100 > >> return bin:pack-integer($n, $size) > >> ) > >> return db:store('db', 'integers.bin', $data) > >> > >> This way, every integer will occupy the supplied number of bytes > >> (here: 4, allowing you to address 2^32 integers). > > > > >

2 1

Storing sequences of atomic type into BaseX ?
by jean-marc Mercier 08 Jan '15

08 Jan '15

Hello, I am currently trying to store precomputed maps. The basic issue is to store a sequence $seq of integers (that are indeed element id, i.e. retrieved using db:node-pre). Actually, I am storing it using db:replace($db,"map",<map>{fn:serialize($seq)}</map>) and retrieve the sequence using using fn:tokenize(db:open($db,"map")/map," ") ! xs:integer(.). This way of doing stores integers as string, then call a cast string / integer to unserialize it. For large integer list (I am dealing with lists of size 134 Mo), it is quite time and mem consuming. I was wondering if there exists a more efficient way to store and retrieve atomic list into BaseX ?

2 3

Reg : Setting Time Out
by Chandrasekhar 08 Jan '15

08 Jan '15

Dear Christian We are optimizing the collection database(5 GB DB SIZE) .After some time(30 minutes) we are getting the below message. org.basex.core.BaseXException: Timeout exceeded. We are not able to setting time out option.We are setting the time out as below mentioned Context context = new Context(); context.globalopts.set(GlobalOptions.TIMEOUT,10); context.globalopts.set(GlobalOptions.SERVERHOST, "localhost"); context.globalopts.set(GlobalOptions.SERVERPORT, 1984); ClientSession ls = new ClientSession(context, "admin", "admin"); Suggest me the right way to setting the time out. Thanks&Regards Chandra The information contained in this e-mail and any accompanying documents may contain information that is confidential or otherwise protected from disclosure. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message, including any attachments. Any dissemination, distribution or other use of the contents of this message by anyone other than the intended recipient is strictly prohibited.

2 1

data protection at rest
by Mansi Sheth 08 Jan '15

08 Jan '15

Hello, I am thinking about my options of protecting BaseXDB's data at rest. We are storing sensitive client data which we need to protect. We would be migrating BaseXDB on an amazon instance, so protection at rest would be a primary concern. What I wish to do is, while inserting data into the database, it should be encrypted and while using XQUERY, we should have a mechanism to decrypt the data and retrieve the information needed. I am aware of the performance hit here. Would evaluate if its acceptable after I could collect some statistics. I looked at the docs: http://docs.basex.org/wiki/Cryptographic_Module#Encryption_.26_Decryption But, I didn't completely understand a use case for this example. Or if it would solve my purpose. I am currently using some Java code to insert files into the database. Has anyone done something on this line ? Please share some use cases. - Mansi

2 1

Database corrupted while inserting big string
by jean-marc Mercier 08 Jan '15

08 Jan '15

Hello, I am trying to insert big string into BaseX, and ended corrupting the whole database. Here a test code , can you reproduce it ? declare updating function local:dbtest($db as xs:string) { db:replace($db,"map",<map>{fn:serialize(1 to 10000000)}</map>) }; local:dbtest("MyDataBase") output : Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 8.0 beta 769b53b Java: Oracle Corporation, 1.8.0_25 OS: Windows 7, amd64 Stack Trace: java.lang.IndexOutOfBoundsException at java.io.RandomAccessFile.writeBytes(Native Method) at java.io.RandomAccessFile.write(Unknown Source) at org.basex.io.random.DataAccess.writeBlock(DataAccess.java:448) at org.basex.io.random.DataAccess.flush(DataAccess.java:49) at org.basex.io.random.DataAccess.close(DataAccess.java:61) at org.basex.data.DiskData.close(DiskData.java:151) at org.basex.core.cmd.Close.close(Close.java:45) at org.basex.query.QueryResources.close(QueryResources.java:110) at org.basex.query.QueryContext.close(QueryContext.java:564) at org.basex.query.QueryProcessor.close(QueryProcessor.java:235) at org.basex.core.cmd.AQuery.query(AQuery.java:129) at org.basex.core.cmd.XQuery.run(XQuery.java:22) at org.basex.core.Command.run(Command.java:379) at org.basex.core.Command.execute(Command.java:95) at org.basex.api.client.LocalSession.execute(LocalSession.java:132) at org.basex.api.client.Session.execute(Session.java:36) at org.basex.core.CLI.execute(CLI.java:106) at org.basex.BaseX.<init>(BaseX.java:124) at org.basex.BaseX.main(BaseX.java:42)

2 1

Silly XQUERY exception
by Mansi Sheth 08 Jan '15

08 Jan '15

Hello, I feel very stupid and frustrated, not able to fix this error: Below, is a my query code, which I am trying to run. I am passing the value for "contains" clause thru command line and I expect to receive # of xml files matching $cmd xpath. I always get Stopped at /veracode/msheth/BaseXWeb/get_prevalence.xq, 16/12: *[XPST0003] Expecting ':=', found ':'.* When ran thru REST: *[XPST0003] Expecting valid step, found 'd'.* I think, its the way $cmd is being set. I have tried, simple string concat using ||, using concat command, using html entities etc. declare variable $n as xs:string external; declare option output:item-separator "
"; (: let $cmd := concat("/A/*/descendant::C/descandant::*[contains(@name,",$singlequote,$n,$singlequote,")]") :) *let $cmd := "/A/*/descendant::C/descandant::*[contains(@name,'"|| $n ||"')]"* let $aPath := for $db in db:list() let $query := "declare variable $db external; " || "db:open($db)" || $cmd *return xquery:eval($query,* * map { 'db': $db, 'query': $cmd })* let $clients := for $elem in $aPath return db:path($elem) return $n , distinct-values(count($clients)) Lines of code, which are the culprit, are marked in bold above. Any and all suggestions are greatly appreciated. - Mansi

2 2

prof:time and updating expression
by jean-marc Mercier 08 Jan '15

08 Jan '15

Hi all, I was trying to profile updating expression, and the result is [XUST0001] prof:time(value[,cache[,label]]): no updating expression allowed. Is this a desired behavior ? Cheers, Jean-Marc

2 1

Linux lessons
by Wendell Piez 08 Jan '15

08 Jan '15

Dear BaseX people, I am a hopeless newbie when it comes to Linux. Seriously, my ignorance is astonishing. I am now running Linux Mint. It's a huge amount of fun but I'm left guessing about many simply things. I am running the current BaseX 7.9. I am using it to provide a pipeline architecture calling lots of XSLT2, producing HTML and SVG. It works well for this. To my RESTXQPATH I wish to append to - but not override (if possible) - a path. By editing my .basex configuration I have managed to reset it, but not simply to append to it. That is, RESTXQPATH = /home/wendell/basex/webapp:/home/wendell/XQuery-project/restxq With this, the server skips the basex/webapp directory. If I remove the second segment and restart, it goes live again. What is the simplest way for me to set up my BaseX so that I can maintain my project xqm files (including a project's RestXQ), in the project, but keep BaseX otherwise (including its data directories) separate? Over the longer term I can see several such directories where I'd like BaseX to look. On another operating system I used to start up using different shell scripts; is that the best way here (and if so does my shell script have to call Java or can I use the scripts in the basex/bin and pass in a value somehow)? In general, what recommendations can you offer for no-fuss configuration of RestXQ for separate projects? Thanks! Wendell -- Wendell Piez | http://www.wendellpiez.com XML | XSLT | electronic publishing Eat Your Vegetables _____oo_________o_o___ooooo____ooooooo_^

2 2

BaseX 8.0 snapshot on OpenShift
by Andy Bunce 08 Jan '15

08 Jan '15

Hi, I am having difficulty getting this to work. I had something working with 7.9 (see https://github.com/Quodatum/openshift-basex-quick-start/tree/releases/0.2 ) Using a recent snapshot. I am running ${OPENSHIFT_DATA_DIR}/basex/bin/basexhttp -n ${OPENSHIFT_DIY_IP} -h ${OPENSHIFT_DIY_PORT} -p ${PORT_DATABASE} -e ${PORT_EVENT} -s ${PORT_STOP} -U ${BASEX_USER} -P ${BASEX_PASSWORD} The debug o/p is STOPPORT: 15007 PASSWORD: admin USER: admin EVENTPORT: 15006 DEBUG: true PORT: 15005 SERVERPORT: 15005 HOST: 127.13.28.129 STOPPORT: 15007 PASSWORD: admin USER: admin EVENTPORT: 15006 DEBUG: true PORT: 15005 SERVERPORT: 15005 HOST: 127.13.28.129 WEBPATH: /var/lib/openshift/549d9d794382ec39b8000105/app-root/data/basex/webapp HTTP Server was started (port: 8080) java.net.BindException: Permission denied at java.net.PlainSocketImpl.socketBind(Native Method) at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376) at java.net.ServerSocket.bind(ServerSocket.java:376) at java.net.ServerSocket.bind(ServerSocket.java:330) at org.basex.BaseXHTTP$StopServer.<init>(BaseXHTTP.java:403) at org.basex.BaseXHTTP.<init>(BaseXHTTP.java:139) at org.basex.BaseXHTTP.main(BaseXHTTP.java:50) Server is running or permission was denied. https://github.com/BaseXdb/basex/blob/0af4ead914d3d1946e5a8ce9672811a34f933… I wonder if SERVERHOST should be set here? As with SERVER PORT.. https://github.com/BaseXdb/basex/blob/0af4ead914d3d1946e5a8ce9672811a34f933… /Andy

2 2

BaseX Concurrency
by Bondeson, Carl 07 Jan '15

07 Jan '15

I have a multithreaded web application running under JBoss 6.2. It seems that 2 threads are attempting access concurrently. Any suggestions. 01/07/15 10:07:14 WARN [server ]: Error creating connection to BaseX server - attempt 0 java.io.IOException: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 7.6 Java: Oracle Corporation, 1.7.0_71 OS: Windows Server 2008 R2, amd64 Stack Trace: java.nio.channels.OverlappingFileLockException sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255) sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152) sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1056) org.basex.io.random.TableDiskAccess.lck(TableDiskAccess.java:183) org.basex.io.random.TableDiskAccess.sharedLock(TableDiskAccess.java:170) org.basex.io.random.TableDiskAccess.<init>(TableDiskAccess.java:83) org.basex.data.DiskData.init(DiskData.java:122) org.basex.data.DiskData.<init>(DiskData.java:92) org.basex.core.cmd.Open.open(Open.java:70) org.basex.core.cmd.Open.run(Open.java:36) org.basex.core.Command.run(Command.java:342) org.basex.core.Command.run(Command.java:117) org.basex.core.cmd.Check.run(Check.java:38) org.basex.core.Command.run(Command.java:342) org.basex.core.Command.exec(Command.java:321) org.basex.core.Command.execute(Command.java:78) org.basex.server.ClientListener.run(ClientListener.java:145) at com.pilotfish.eip.basex.BaseXClient.execute(BaseXClient.java:79) at com.pilotfish.eip.basex.BaseXClient.execute(BaseXClient.java:91) at com.pilotfish.eip.basex.txlog.BaseXSubmitter.createBaseXClient(BaseXSubmitter.java:54) at com.pilotfish.eip.basex.txlog.BaseXSubmitter.getBaseXClient(BaseXSubmitter.java:45) at com.pilotfish.eip.basex.txlog.BaseXSubmitter.execute(BaseXSubmitter.java:111) at com.pilotfish.eip.rest.impl.TransactionLogServiceImpl.constructHealthComponentSettings(TransactionLogServiceImpl.java:726) at com.pilotfish.eip.rest.impl.TransactionLogServiceImpl.setHealthComponentConfiguration(TransactionLogServiceImpl.java:696) at com.pilotfish.eip.rest.impl.TransactionLogServiceImpl.setHealthComponentConfiguration(TransactionLogServiceImpl.java:700) at com.pilotfish.eip.rest.impl.TransactionLogServiceImpl.setHealthComponentConfiguration(TransactionLogServiceImpl.java:700) at com.pilotfish.eip.rest.impl.TransactionLogServiceImpl.getHealthComponentConfiguration(TransactionLogServiceImpl.java:607) at com.pilotfish.eip.rest.services.TransactionLogResource.getHealthComponentConfiguration(TransactionLogResource.java:247) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:353) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:343) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) at org.glassfish.jersey.internal.Errors.process(Errors.java:315) at org.glassfish.jersey.internal.Errors.process(Errors.java:297) at org.glassfish.jersey.internal.Errors.process(Errors.java:267) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:318) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:983) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:359) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:372) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:335) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:218) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) at java.lang.Thread.run(Thread.java:745) Carl R Bondeson Systems Developer Department of Public Health Data Processing 410 Capitol Ave Hartford, CT 06134 Phone: 860-509-7434 carl.bondeson(a)ct.gov<mailto:carl.bondeson@ct.gov> [cid:image001.jpg@01D02A62.E91E67F0]<http://www.ct.gov/dph>

2 2