Hi All,
Because it got rejected sending this email again.
[Sriram]
We have an application where we need to process huge XML(300-500 XML document per day -of 15-20GB) documents need to be inserted and updated by more than 200 people. Can we use Basex for this purpose. Here is the application requirement.
1. We get 300-500 xml documents everyday
2. It should be saved in Basex server
3. Every day around 200 users would edit those document using front end application created using ASP.Net. At a time maximum 75 people edit those xml documents.
I could able to insert the 3Gb of XML data in to Basex server without any issue. Which is good.
I am looking for logging feature when these XML documents were edited by different users. Is this feature available in Basex Server.
Is Basex is suitable for content management application?
Appreciate your help this regard.
Regards
Sriram V
.Net Tech Architect
GPO | 732 North Capitol Street, NW, Washington, DC 20401
Find Government information http://www.fdsys.gov/ http://www.fdsys.gov | http://bookstore.gpo.gov/ http://bookstore.gpo.gov | http://govbooktalk.gpo.gov/ http://govbooktalk.gpo.gov
Dear Sriram V,
We have an application where we need to process huge XML(300-500 XML document per day -of 15-20GB) documents need to be inserted and updated by more than 200 people.
Does every document really have 15-20 GB, or is this the total amount of daily XML input data?
In the second case, each document would be sized around 60 MB, and the total size would be 5-7 TB per year. To deal with that amount of data, you will either need to work with large hard disk arrays, or you could distribute your data by using one central BaseX master server, which delegates your documents to other BaseX slave servers.
I am looking for logging feature when these XML documents were edited bydifferent users. Is this feature available in Basex Server.
The existing locking mechanism is restricted to WebDAV. We have already got requests to extend this to arbitrary database resources (see [1]), so I will count you in here and rank this higher. – Apart from our internal features, it’s always possible to write some RESTXQ application logic that takes care of the user locks (but this requires some basic knowledge of XQuery).
One of the important features of BaseX is support for the XQuery language, which allows you to query your stored resources. If you only want to store and retrieve large XML resources, and if you do not need all the other features, one of the many NoSQL systems that have no rich query languages but built-in support for distribution may be a better choice.
Is Basex is suitable for content management application?
Absolutely. For example, XML Director [2] is a CMS solution that works with BaseX as storage back end.
Hope this helps, Christian
[1] https://github.com/BaseXdb/basex/issues/1083 [2] https://xml-director.info/
Dear Christian,
I appreciate your quick response. Here is my clarification.
1. 15 to 20Gb size of data is for a given day. This is maximum size. Some time we could get only few GB. Once the data is loaded in to database, users would edit it, end of the day I would take an archive. Everyday this process would continue. At any point of time current database (where user would edit) would not be more than 20GB. Here is the flow. a. we get data (XML document) for the given day. b. we save it in BaseX c. Users (around 200 employee, who works in different shift time) work on it and complete the work. d. end of the day all the documents would be approved. e. archive it database (could be any Database)
next day same process continues. Hope this explains the application flow better.
My main question is can I use BaseX for such database intensive application. Also I would like to know is there any build feature like audit log. Because any point of time I would like to know who edited or deleted the contents of the XML document. In worst case I could rollback their changes.
FYI, I downloaded the BaseX and learning using Xquery using BaseX GUI. Product looks promising. I will read more documentation to understand it better. Have a nice weekend my friend.
Thanks Sriram.
-----Original Message----- From: Christian Grün [mailto:christian.gruen@gmail.com] Sent: Saturday, February 4, 2017 4:09 AM To: Srirama V sriramav@yakshna.com Cc: BaseX basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Suitability
Dear Sriram V,
We have an application where we need to process huge XML(300-500 XML document per day -of 15-20GB) documents need to be inserted and updated by more than 200 people.
Does every document really have 15-20 GB, or is this the total amount of daily XML input data?
In the second case, each document would be sized around 60 MB, and the total size would be 5-7 TB per year. To deal with that amount of data, you will either need to work with large hard disk arrays, or you could distribute your data by using one central BaseX master server, which delegates your documents to other BaseX slave servers.
I am looking for logging feature when these XML documents wereedited by different users. Is this feature available in Basex Server.
The existing locking mechanism is restricted to WebDAV. We have already got requests to extend this to arbitrary database resources (see [1]), so I will count you in here and rank this higher. – Apart from our internal features, it’s always possible to write some RESTXQ application logic that takes care of the user locks (but this requires some basic knowledge of XQuery).
One of the important features of BaseX is support for the XQuery language, which allows you to query your stored resources. If you only want to store and retrieve large XML resources, and if you do not need all the other features, one of the many NoSQL systems that have no rich query languages but built-in support for distribution may be a better choice.
Is Basex is suitable for content management application?
Absolutely. For example, XML Director [2] is a CMS solution that works with BaseX as storage back end.
Hope this helps, Christian
[1] https://github.com/BaseXdb/basex/issues/1083 [2] https://xml-director.info/
Hi Srirama,
This is definitely something that can be done with BaseX.
You indicated that the day’s work will be archived, so this means you don’t want to have terabytes of XML data available at the same time, right?
There is no built-in “audit log” feature, but you could surely realize this with XQuery. Uploading 60 MB may take some time, so I assume that the 200 users won’t do this every minute? Can you already guess how many updates operations there will be every day?
It’s good to hear you already have gathered some experience with XQuery. RESTXQ [1] can be used to write all the server-side business logic.
Hope this helps, Christian
[1] http://docs.basex.org/wiki/RESTXQ
-----Original Message----- From: Christian Grün [mailto:christian.gruen@gmail.com] Sent: Saturday, February 4, 2017 4:09 AM To: Srirama V sriramav@yakshna.com Cc: BaseX basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Suitability
Dear Sriram V,
We have an application where we need to process huge XML(300-500 XML document per day -of 15-20GB) documents need to be inserted and updated by more than 200 people.
Does every document really have 15-20 GB, or is this the total amount of daily XML input data?
In the second case, each document would be sized around 60 MB, and the total size would be 5-7 TB per year. To deal with that amount of data, you will either need to work with large hard disk arrays, or you could distribute your data by using one central BaseX master server, which delegates your documents to other BaseX slave servers.
I am looking for logging feature when these XML documents wereedited by different users. Is this feature available in Basex Server.
The existing locking mechanism is restricted to WebDAV. We have already got requests to extend this to arbitrary database resources (see [1]), so I will count you in here and rank this higher. – Apart from our internal features, it’s always possible to write some RESTXQ application logic that takes care of the user locks (but this requires some basic knowledge of XQuery).
One of the important features of BaseX is support for the XQuery language, which allows you to query your stored resources. If you only want to store and retrieve large XML resources, and if you do not need all the other features, one of the many NoSQL systems that have no rich query languages but built-in support for distribution may be a better choice.
Is Basex is suitable for content management application?Absolutely. For example, XML Director [2] is a CMS solution that works with BaseX as storage back end.
Hope this helps, Christian
[1] https://github.com/BaseXdb/basex/issues/1083 [2] https://xml-director.info/
Chris,
Nice to hear from you. Here is my answer. 1. You indicated that the day’s work will be archived, so this means you don’t want to have terabytes of XML data available at the same time, right? We have not yet decided, we could use BaseX for archiving too.
2. There is no built-in “audit log” feature, but you could surely realize this with XQuery. Uploading 60 MB may take some time, so I assume that the 200 users won’t do this every minute? Can you already guess how many updates operations there will be every day? As I mentioned total 200 employees (approximately 65 employee per shift) would be working on it. We can assume there would be 65 concurrent users working on one day XML documents. We are still collecting the information about how many updates could be possible.
3. One more question, does Basex works well in Microsoft IIS server and dot net environment?
Appreciate your help. Have a good day.
Thanks Sriram.
-----Original Message----- From: Christian Grün [mailto:christian.gruen@gmail.com] Sent: Saturday, February 4, 2017 11:54 AM To: Srirama V sriramav@yakshna.com Cc: BaseX basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Suitability
Hi Srirama,
This is definitely something that can be done with BaseX.
You indicated that the day’s work will be archived, so this means you don’t want to have terabytes of XML data available at the same time, right?
There is no built-in “audit log” feature, but you could surely realize this with XQuery. Uploading 60 MB may take some time, so I assume that the 200 users won’t do this every minute? Can you already guess how many updates operations there will be every day?
It’s good to hear you already have gathered some experience with XQuery. RESTXQ [1] can be used to write all the server-side business logic.
Hope this helps, Christian
[1] http://docs.basex.org/wiki/RESTXQ
-----Original Message----- From: Christian Grün [mailto:christian.gruen@gmail.com] Sent: Saturday, February 4, 2017 4:09 AM To: Srirama V sriramav@yakshna.com Cc: BaseX basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Suitability
Dear Sriram V,
We have an application where we need to process huge XML(300-500 XML document per day -of 15-20GB) documents need to be inserted and updated by more than 200 people.
Does every document really have 15-20 GB, or is this the total amount of daily XML input data?
In the second case, each document would be sized around 60 MB, and the total size would be 5-7 TB per year. To deal with that amount of data, you will either need to work with large hard disk arrays, or you could distribute your data by using one central BaseX master server, which delegates your documents to other BaseX slave servers.
I am looking for logging feature when these XML documents wereedited by different users. Is this feature available in Basex Server.
The existing locking mechanism is restricted to WebDAV. We have already got requests to extend this to arbitrary database resources (see [1]), so I will count you in here and rank this higher. – Apart from our internal features, it’s always possible to write some RESTXQ application logic that takes care of the user locks (but this requires some basic knowledge of XQuery).
One of the important features of BaseX is support for the XQuery language, which allows you to query your stored resources. If you only want to store and retrieve large XML resources, and if you do not need all the other features, one of the many NoSQL systems that have no rich query languages but built-in support for distribution may be a better choice.
Is Basex is suitable for content management application?Absolutely. For example, XML Director [2] is a CMS solution that works with BaseX as storage back end.
Hope this helps, Christian
[1] https://github.com/BaseXdb/basex/issues/1083 [2] https://xml-director.info/
Hi Srirama,
- One more question, does Basex works well in Microsoft IIS server and dot net environment?
BaseX is a Java application. As such, it works on all platforms that provide a Java runtime environment. You could check out [1] to see how Java servlet apps can be deployed on IIS. Apart from that, a cross-compiled IKVM version of BaseX has been made available (called Nxdb), but I think the project was discontinued some years ago.
Hope this helps, Christian
[1] http://www.helicontech.com/articles/deploying-java-servlet-applications-on-w...
-----Original Message----- From: Christian Grün [mailto:christian.gruen@gmail.com] Sent: Saturday, February 4, 2017 11:54 AM To: Srirama V sriramav@yakshna.com Cc: BaseX basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Suitability
Hi Srirama,
This is definitely something that can be done with BaseX.
You indicated that the day’s work will be archived, so this means you don’t want to have terabytes of XML data available at the same time, right?
There is no built-in “audit log” feature, but you could surely realize this with XQuery. Uploading 60 MB may take some time, so I assume that the 200 users won’t do this every minute? Can you already guess how many updates operations there will be every day?
It’s good to hear you already have gathered some experience with XQuery. RESTXQ [1] can be used to write all the server-side business logic.
Hope this helps, Christian
[1] http://docs.basex.org/wiki/RESTXQ
-----Original Message----- From: Christian Grün [mailto:christian.gruen@gmail.com] Sent: Saturday, February 4, 2017 4:09 AM To: Srirama V sriramav@yakshna.com Cc: BaseX basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Suitability
Dear Sriram V,
We have an application where we need to process huge XML(300-500 XML document per day -of 15-20GB) documents need to be inserted and updated by more than 200 people.
Does every document really have 15-20 GB, or is this the total amount of daily XML input data?
In the second case, each document would be sized around 60 MB, and the total size would be 5-7 TB per year. To deal with that amount of data, you will either need to work with large hard disk arrays, or you could distribute your data by using one central BaseX master server, which delegates your documents to other BaseX slave servers.
I am looking for logging feature when these XML documents wereedited by different users. Is this feature available in Basex Server.
The existing locking mechanism is restricted to WebDAV. We have already got requests to extend this to arbitrary database resources (see [1]), so I will count you in here and rank this higher. – Apart from our internal features, it’s always possible to write some RESTXQ application logic that takes care of the user locks (but this requires some basic knowledge of XQuery).
One of the important features of BaseX is support for the XQuery language, which allows you to query your stored resources. If you only want to store and retrieve large XML resources, and if you do not need all the other features, one of the many NoSQL systems that have no rich query languages but built-in support for distribution may be a better choice.
Is Basex is suitable for content management application?Absolutely. For example, XML Director [2] is a CMS solution that works with BaseX as storage back end.
Hope this helps, Christian
[1] https://github.com/BaseXdb/basex/issues/1083 [2] https://xml-director.info/
Hi Christian,
Greetings.
In BaseX, How could validate XML documents using Schema.
Appreciate your help.
Regards Sriram.
On Sun, Feb 5, 2017 at 12:19 PM, Christian Grün christian.gruen@gmail.com wrote:
Hi Srirama,
- One more question, does Basex works well in Microsoft IIS server and
dot net environment?
BaseX is a Java application. As such, it works on all platforms that provide a Java runtime environment. You could check out [1] to see how Java servlet apps can be deployed on IIS. Apart from that, a cross-compiled IKVM version of BaseX has been made available (called Nxdb), but I think the project was discontinued some years ago.
Hope this helps, Christian
[1] http://www.helicontech.com/articles/deploying-java- servlet-applications-on-windows-with-iis/
-----Original Message----- From: Christian Grün [mailto:christian.gruen@gmail.com] Sent: Saturday, February 4, 2017 11:54 AM To: Srirama V sriramav@yakshna.com Cc: BaseX basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Suitability
Hi Srirama,
This is definitely something that can be done with BaseX.
You indicated that the day’s work will be archived, so this means you
don’t want to have terabytes of XML data available at the same time, right?
There is no built-in “audit log” feature, but you could surely realize
this with XQuery. Uploading 60 MB may take some time, so I assume that the 200 users won’t do this every minute? Can you already guess how many updates operations there will be every day?
It’s good to hear you already have gathered some experience with XQuery.
RESTXQ [1] can be used to write all the server-side business logic.
Hope this helps, Christian
[1] http://docs.basex.org/wiki/RESTXQ
-----Original Message----- From: Christian Grün [mailto:christian.gruen@gmail.com] Sent: Saturday, February 4, 2017 4:09 AM To: Srirama V sriramav@yakshna.com Cc: BaseX basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Suitability
Dear Sriram V,
We have an application where we need to process huge XML(300-500 XML document per day -of 15-20GB) documents need to be inserted and updated by more than 200 people.
Does every document really have 15-20 GB, or is this the total amount
of daily XML input data?
In the second case, each document would be sized around 60 MB, and the
total size would be 5-7 TB per year. To deal with that amount of data, you will either need to work with large hard disk arrays, or you could distribute your data by using one central BaseX master server, which delegates your documents to other BaseX slave servers.
I am looking for logging feature when these XML documents wereedited by different users. Is this feature available in Basex Server.
The existing locking mechanism is restricted to WebDAV. We have already
got requests to extend this to arbitrary database resources (see [1]), so I will count you in here and rank this higher. – Apart from our internal features, it’s always possible to write some RESTXQ application logic that takes care of the user locks (but this requires some basic knowledge of XQuery).
One of the important features of BaseX is support for the XQuery
language, which allows you to query your stored resources. If you only want to store and retrieve large XML resources, and if you do not need all the other features, one of the many NoSQL systems that have no rich query languages but built-in support for distribution may be a better choice.
Is Basex is suitable for content management application?Absolutely. For example, XML Director [2] is a CMS solution that works
with BaseX as storage back end.
Hope this helps, Christian
[1] https://github.com/BaseXdb/basex/issues/1083 [2] https://xml-director.info/
Hi Srirama,
In BaseX, How could validate XML documents using Schema.
Please have a look at our documentation [1].
Hope this helps, Christian
basex-talk@mailman.uni-konstanz.de
-
Christian Grün -
Srirama V -
Srirama Veeraraghavan