Hi everyone, I'm wondering if someone could provide what I think is a brief script for a scientific project to do the following. The goal is to identify XML documents from a very large collection that would be too big to load into a database all at once. Here is how I see the functions provided by the code. 1. In the script, the user could enter the path of the target folder (with millions of XML documents). 2. In the script, the user would enter the number of documents to load into a database at a given time (i =. 1,000) depending on memory limitations. 3. The code would then create a temporary database from the first (i) xml files in the target folder. 4. The code would then search the 1000 xml documents in the database for a pre-defined text string. 5. If hits exist for the text string, the code would write those documents to a unique XML file. 6. Clear the database. 7. Read in the next 1000 files (or remaining files in the folder). 8. Return to #4. There would be no need to append XML files in step 5. The resulting XML files could be concatenated afterwards. Thank you in advance. If you have any questions, please feel free to email me here. Greg *************************************************** Greg Kawchuk BSC, DC, MSc, PhD. Professor, Faculty of Rehabilitation Medicine University of Alberta greg.kawchuk@ualberta.ca 780-492-6891
Am 05.08.2019 um 08:41 schrieb Greg Kawchuk:
Hi everyone, I'm wondering if someone could provide what I think is a brief script for a scientific project to do the following. The goal is to identify XML documents from a very large collection that would be too big to load into a database all at once.
Here is how I see the functions provided by the code. 1. In the script, the user could enter the path of the target folder (with millions of XML documents). 2. In the script, the user would enter the number of documents to load into a database at a given time (i =. 1,000) depending on memory limitations. 3. The code would then create a temporary database from the first (i) xml files in the target folder. 4. The code would then search the 1000 xml documents in the database for a pre-defined text string.
What kind of search is that exactly? Does it depend on any database related features at all or can't you just use BaseX as a standalone XQuery processor?
5. If hits exist for the text string, the code would write those documents to a unique XML file.
What kind of structure would that unique file have, simply <root>{collection('foo')[1 to 1000][condition]}</root>
6. Clear the database. 7. Read in the next 1000 files (or remaining files in the folder). 8. Return to #4.
There would be no need to append XML files in step 5. The resulting XML files could be concatenated afterwards. Thank you in advance. If you have any questions, please feel free to email me here.
Hi Greg, So, to be clear and succinct, the goal is to create a single XML file containing all XML files that have a predefined text string match in them, yes? If so, I'm wondering if creating any database is necessary. A single pass through all the files, searching for the text string, and appending matched files as you go seems sufficient. R On Mon, Aug 5, 2019, 08:42 Greg Kawchuk <greg.kawchuk@ualberta.ca> wrote:
Hi everyone, I'm wondering if someone could provide what I think is a brief script for a scientific project to do the following. The goal is to identify XML documents from a very large collection that would be too big to load into a database all at once.
Here is how I see the functions provided by the code. 1. In the script, the user could enter the path of the target folder (with millions of XML documents). 2. In the script, the user would enter the number of documents to load into a database at a given time (i =. 1,000) depending on memory limitations. 3. The code would then create a temporary database from the first (i) xml files in the target folder. 4. The code would then search the 1000 xml documents in the database for a pre-defined text string. 5. If hits exist for the text string, the code would write those documents to a unique XML file. 6. Clear the database. 7. Read in the next 1000 files (or remaining files in the folder). 8. Return to #4.
There would be no need to append XML files in step 5. The resulting XML files could be concatenated afterwards. Thank you in advance. If you have any questions, please feel free to email me here. Greg
*************************************************** Greg Kawchuk BSC, DC, MSc, PhD. Professor, Faculty of Rehabilitation Medicine University of Alberta greg.kawchuk@ualberta.ca 780-492-6891
Creating a bunch of temporary databases that you’re going to delete doesn’t sound like the most efficient way to process this data. But it hard to tell what alternative to recommend without more info about what your desired end result is. Is this something that you’re going to do once, or do repeatedly for different strings ? Taking your description literally, I would use ‘grep -l ‘ to generate a list of files with the specific string, and either feed that list into ‘cat’ or else use it to build a database of a subset of files for further investigation. There are also other ways to filter the data through a stream to select a subset if that is what you want to do. But if you’re going to do this repeatedly for different subsets, then it might make more sense to try to get everything parsed and indexed into the database once. If it really is too large for a single database after adjusting the java memory parameters in the basex scripts, you could try sharding the data into several databases, repeat the search on each collection, and concatenate the results.
On Aug 5, 2019, at 2:41 AM, Greg Kawchuk <greg.kawchuk@ualberta.ca> wrote:
Hi everyone, I'm wondering if someone could provide what I think is a brief script for a scientific project to do the following. The goal is to identify XML documents from a very large collection that would be too big to load into a database all at once.
Here is how I see the functions provided by the code. 1. In the script, the user could enter the path of the target folder (with millions of XML documents). 2. In the script, the user would enter the number of documents to load into a database at a given time (i =. 1,000) depending on memory limitations. 3. The code would then create a temporary database from the first (i) xml files in the target folder. 4. The code would then search the 1000 xml documents in the database for a pre-defined text string. 5. If hits exist for the text string, the code would write those documents to a unique XML file. 6. Clear the database. 7. Read in the next 1000 files (or remaining files in the folder). 8. Return to #4.
There would be no need to append XML files in step 5. The resulting XML files could be concatenated afterwards. Thank you in advance. If you have any questions, please feel free to email me here. Greg
*************************************************** Greg Kawchuk BSC, DC, MSc, PhD. Professor, Faculty of Rehabilitation Medicine University of Alberta greg.kawchuk@ualberta.ca <mailto:greg.kawchuk@ualberta.ca> 780-492-6891
For this type of project, I regard into the solution with a file:list and a stream use of XSLT, or a use of the saxon:discard-document or the use of collection as in https://ajwelch.blogspot.com/2006/11/using-collection-and-saxondiscard.html If the query to select file doesn'y need Xquery but XPath, it can be done like that. If the collection is to big you can do more than one pass with the file selector (files started by a, next b, etc....) Marc Le 06/08/2019 à 18:12, Majewski, Steven Dennis (sdm7g) a écrit :
Creating a bunch of temporary databases that you’re going to delete doesn’t sound like the most efficient way to process this data. But it hard to tell what alternative to recommend without more info about what your desired end result is.
Is this something that you’re going to do once, or do repeatedly for different strings ?
Taking your description literally, I would use ‘grep -l ‘ to generate a list of files with the specific string, and either feed that list into ‘cat’ or else use it to build a database of a subset of files for further investigation.
There are also other ways to filter the data through a stream to select a subset if that is what you want to do.
But if you’re going to do this repeatedly for different subsets, then it might make more sense to try to get everything parsed and indexed into the database once. If it really is too large for a single database after adjusting the java memory parameters in the basex scripts, you could try sharding the data into several databases, repeat the search on each collection, and concatenate the results.
On Aug 5, 2019, at 2:41 AM, Greg Kawchuk <greg.kawchuk@ualberta.ca <mailto:greg.kawchuk@ualberta.ca>> wrote:
Hi everyone, I'm wondering if someone could provide what I think is a brief script for a scientific project to do the following. The goal is to identify XML documents from a very large collection that would be too big to load into a database all at once.
Here is how I see the functions provided by the code. 1. In the script, the user could enter the path of the target folder (with millions of XML documents). 2. In the script, the user would enter the number of documents to load into a database at a given time (i =. 1,000) depending on memory limitations. 3. The code would then create a temporary database from the first (i) xml files in the target folder. 4. The code would then search the 1000 xml documents in the database for a pre-defined text string. 5. If hits exist for the text string, the code would write those documents to a unique XML file. 6. Clear the database. 7. Read in the next 1000 files (or remaining files in the folder). 8. Return to #4.
There would be no need to append XML files in step 5. The resulting XML files could be concatenated afterwards. Thank you in advance. If you have any questions, please feel free to email me here. Greg
*************************************************** Greg Kawchuk BSC, DC, MSc, PhD. Professor, Faculty of Rehabilitation Medicine University of Alberta greg.kawchuk@ualberta.ca <mailto:greg.kawchuk@ualberta.ca> 780-492-6891
participants (5)
-
Greg Kawchuk -
Majewski, Steven Dennis (sdm7g) -
Marc -
Martin Honnen -
Rick Graham