Hi,
We run batch processes on our files in order to send them out to translation in xliff format (xml format for translations).
I run a piece of code that: 1. applies transformations 2. file:writes the resulting files (for debug purposes) 3. and then db:replaces the resulting files.
If I send 5 files into the process, it takes a few seconds before the files appear on the file system. Yet full process with the db:replace takes 2 minutes to complete.
When I send 60 files... the process never ends. All files are available on the file system and are valid, which means I am fairly certain that the freeze doesn't occur because of an error in one of the files.
There is no error message in the terminal window. Is possible that something was changed after 7.7 that would affect performance? On our production server, which is on 7.7 we can run 100+ files in just a few minutes. Maybe there are new server settings that we missed?
For reference, here is the code. I understand that updates are all performed at the end of the process, which is why we get the debug files first, but the lag time between the time where we get all files on the file system and the db:replace seems huge.
*declare* *updating* *function* *translation:transform-to-xliff*(*$menu-id* *as **xs:string*, *$lang* *as **xs:string**, *$package-number* *as **xs:string* ){
*let **$menu* := *menu:xml-get-menu*(*$menu-id*, 'en-us')
(: Need to translate menu and its topics without getting duplicated topics twice:) (: Always send the reusable strings from writers and from dev... doesn't cost anything if no new string anyway :) *let* *$file-ids* := (*$menu-id*, *for **$id* *in **distinct-values*( *$menu*/descendant::*[*name*()='topicref']/*data*(*@id-ref*))[1] *return * *$id*, 'stringreferences', 'stringreferencestransform')
*return *(*for **$file-id* *in **distinct-values*(*$file-ids*) *let* *$file* := *db:open*('en-us')/*[*@id*=*$file-id*] *let* *$skl-href* := 'skeletons/' || *$package-number* || '/'|| *substring-after*(*$file*/*base-uri*(), 'en-us/') *let* *$file-with-preserve* := *copy* *$copy* := *$file* *modify*(*if* (*exists*(*$copy*/*@xml:space*)) *then *() *else **insert* *node* *attribute* {'xml:space' }{'preserve'} *into* *$copy*) *return **$copy* *let* *$all-trans-files* := *xliff-transformer:create-translation-files*(*$file-with-preserve*, *$package-number*, *$menu-id*, *$lang*, *$skl-href*)
*let* *$skeleton* := *$all-trans-files*[1]
*let* *$xliffs* := *for **$xliff* *in **$all-trans-files*[ *position*() != 1] *return **$xliff*
*let* *$debug* := *file:write*('Debug-xliff/skls/' || *tokenize*(*$skl-href*, '/')[*last*()], *$skeleton*)
*return *(*db:replace*('Translation-management', *$skl-href*, *$skeleton*), *for **$xliff* *in **$xliffs* *let* *$target-lang* := *$xliff*/descendant::*[*name* ()='file']/*data*(*@target-language*) *let* *$lang-pack-name* := *concat*(*$target-lang*,'_', *$package-number*) *let* *$file-loc-end* := *$menu-id* || '/' || *$lang-pack-name* ||'/' || *substring-after*(*$file*/*base-uri*(), 'en-us/') (:let $debug := file:append('debug-ids.xml', ' ' || $file-loc-end):) *let* *$file-loc* := *if* (*exists*(*$xliff*/ descendant::*[*@translate*='yes'])) *then *'xliff-to-translate/' || *$file-loc-end* *else *'xliff-no-new-segment/' || *$file-loc-end* *let* *$debug* := *file:write*('Debug-xliff/' || *tokenize*(*$file-loc*, '/')[*last*()], *$xliff*) *return **db:replace*('Translation-management', *$file-loc*, *$xliff*) ) ) };
Hi France,
sorry, I know that an answer to your e-mail is still pending. If you should manage to build a self-contained example, it could help us to find the culprit.
Best, Christian
On Wed, Mar 19, 2014 at 6:59 PM, France Baril france.baril@architextus.com wrote:
We may have found a first evidence for the slow down. Do your documents use namespaces?
On Tue, Mar 25, 2014 at 2:41 PM, Christian Grün christian.gruen@gmail.com wrote:
Yes, we do use namespaces.
On Tue, Mar 25, 2014 at 2:29 PM, Christian Grün christian.gruen@gmail.comwrote:
Any news on this issue?
On Tue, Mar 25, 2014 at 2:30 PM, France Baril france.baril@architextus.comwrote:
Hi France,
sorry for lettting you wait. We've already solved part of the problem, but we would first like to shed some more light on another related issue before relasing a new patch version.
Christian ____________________________________
On Tue, Apr 1, 2014 at 5:26 PM, France Baril france.baril@architextus.com wrote:
Ok, thanks for the update.
We are dependent on this to make the next release to our production server, so if you have an idea on timing, it would really help us adapt.
France
On Tue, Apr 1, 2014 at 11:43 AM, Christian Grün christian.gruen@gmail.comwrote:
basex-talk@mailman.uni-konstanz.de
-
Christian Grün -
France Baril