Java XPath (Apache JAXP implementation) performance2019 Community Moderator ElectionJava ServiceLoader with multiple ClassloadersXPath.evaluate performance slows down (absurdly) over multiple callsA nice Java XML DOM utilityOptimizing DOM and XPath Java codeXpath memory leak?Java xpath memory leak?Xalan DTMManager.newInstance performance issuetomcat 8 upgrade from tomcat 6 - classloading performance issuexPath multiple xml files from different url very slowIs there any range limit for XSL:VARIABLEIs Java “pass-by-reference” or “pass-by-value”?Does a finally block always get executed in Java?What is the difference between public, protected, package-private and private in Java?How do I generate random integers within a specific range in Java?“implements Runnable” vs “extends Thread” in JavaImprove INSERT-per-second performance of SQLite?How do I convert a String to an int in Java?Creating a memory leak with JavaWhat exactly is Apache Camel?Why does changing 0.1f to 0 slow down performance by 10x?

Java XPath (Apache JAXP implementation) performance



2019 Community Moderator ElectionJava ServiceLoader with multiple ClassloadersXPath.evaluate performance slows down (absurdly) over multiple callsA nice Java XML DOM utilityOptimizing DOM and XPath Java codeXpath memory leak?Java xpath memory leak?Xalan DTMManager.newInstance performance issuetomcat 8 upgrade from tomcat 6 - classloading performance issuexPath multiple xml files from different url very slowIs there any range limit for XSL:VARIABLEIs Java “pass-by-reference” or “pass-by-value”?Does a finally block always get executed in Java?What is the difference between public, protected, package-private and private in Java?How do I generate random integers within a specific range in Java?“implements Runnable” vs “extends Thread” in JavaImprove INSERT-per-second performance of SQLite?How do I convert a String to an int in Java?Creating a memory leak with JavaWhat exactly is Apache Camel?Why does changing 0.1f to 0 slow down performance by 10x?










53
















NOTE: If you experience this issue as well, please upvote it on Apache JIRA:



https://issues.apache.org/jira/browse/XALANJ-2540




I have come to an astonishing conclusion that this:



Element e = (Element) document.getElementsByTagName("SomeElementName").item(0);
String result = ((Element) e).getTextContent();


Seems to be an incredible 100x faster than this:



// Accounts for 30%, can be cached
XPathFactory factory = XPathFactory.newInstance();

// Negligible
XPath xpath = factory.newXPath();

// Negligible
XPathExpression expression = xpath.compile("//SomeElementName");

// Accounts for 70%
String result = (String) expression.evaluate(document, XPathConstants.STRING);


I'm using the JVM's default implementation of JAXP:



org.apache.xpath.jaxp.XPathFactoryImpl
org.apache.xpath.jaxp.XPathImpl


I'm really confused, because it's easy to see how JAXP could optimise the above XPath query to actually execute a simple getElementsByTagName() instead. But it doesn't seem to do that. This problem is limited to around 5-6 frequently used XPath calls, that are abstracted and hidden by an API. Those queries involve simple paths (e.g. /a/b/c, no variables, conditions) against an always available DOM Document only. So, if an optimisation can be done, it will be quite easy to achieve.



My question: Is XPath's slowness an accepted fact, or am I overlooking something? Is there a better (faster) implementation? Or should I just avoid XPath altogether, for simple queries?










share|improve this question



















  • 3





    Have you tried comparing your results with that of a compiled, reusable XPathExpression?

    – McDowell
    Jun 14 '11 at 8:18











  • Is it problematic slow? One possibility is to evaluate another library, such as jaxen.

    – Johan Sjöberg
    Jun 14 '11 at 8:19











  • @McDowell, nope. I'll try that, thanks

    – Lukas Eder
    Jun 14 '11 at 8:23











  • @Johan: It is. Typically, those queries tend to take 2-5ms a piece, or 10% of a user's request time. We think this starts being problematic, as we're going to use more XPath in the future. jaxen might indeed be an option in the future...

    – Lukas Eder
    Jun 14 '11 at 8:25











  • @McDowell: Caching the expression is negligible, especially compared to caching of the factory. But after some re-tests, I have to correct the times. Caching the factory will accelerate roughly by 30%

    – Lukas Eder
    Jun 14 '11 at 8:28
















53
















NOTE: If you experience this issue as well, please upvote it on Apache JIRA:



https://issues.apache.org/jira/browse/XALANJ-2540




I have come to an astonishing conclusion that this:



Element e = (Element) document.getElementsByTagName("SomeElementName").item(0);
String result = ((Element) e).getTextContent();


Seems to be an incredible 100x faster than this:



// Accounts for 30%, can be cached
XPathFactory factory = XPathFactory.newInstance();

// Negligible
XPath xpath = factory.newXPath();

// Negligible
XPathExpression expression = xpath.compile("//SomeElementName");

// Accounts for 70%
String result = (String) expression.evaluate(document, XPathConstants.STRING);


I'm using the JVM's default implementation of JAXP:



org.apache.xpath.jaxp.XPathFactoryImpl
org.apache.xpath.jaxp.XPathImpl


I'm really confused, because it's easy to see how JAXP could optimise the above XPath query to actually execute a simple getElementsByTagName() instead. But it doesn't seem to do that. This problem is limited to around 5-6 frequently used XPath calls, that are abstracted and hidden by an API. Those queries involve simple paths (e.g. /a/b/c, no variables, conditions) against an always available DOM Document only. So, if an optimisation can be done, it will be quite easy to achieve.



My question: Is XPath's slowness an accepted fact, or am I overlooking something? Is there a better (faster) implementation? Or should I just avoid XPath altogether, for simple queries?










share|improve this question



















  • 3





    Have you tried comparing your results with that of a compiled, reusable XPathExpression?

    – McDowell
    Jun 14 '11 at 8:18











  • Is it problematic slow? One possibility is to evaluate another library, such as jaxen.

    – Johan Sjöberg
    Jun 14 '11 at 8:19











  • @McDowell, nope. I'll try that, thanks

    – Lukas Eder
    Jun 14 '11 at 8:23











  • @Johan: It is. Typically, those queries tend to take 2-5ms a piece, or 10% of a user's request time. We think this starts being problematic, as we're going to use more XPath in the future. jaxen might indeed be an option in the future...

    – Lukas Eder
    Jun 14 '11 at 8:25











  • @McDowell: Caching the expression is negligible, especially compared to caching of the factory. But after some re-tests, I have to correct the times. Caching the factory will accelerate roughly by 30%

    – Lukas Eder
    Jun 14 '11 at 8:28














53












53








53


31







NOTE: If you experience this issue as well, please upvote it on Apache JIRA:



https://issues.apache.org/jira/browse/XALANJ-2540




I have come to an astonishing conclusion that this:



Element e = (Element) document.getElementsByTagName("SomeElementName").item(0);
String result = ((Element) e).getTextContent();


Seems to be an incredible 100x faster than this:



// Accounts for 30%, can be cached
XPathFactory factory = XPathFactory.newInstance();

// Negligible
XPath xpath = factory.newXPath();

// Negligible
XPathExpression expression = xpath.compile("//SomeElementName");

// Accounts for 70%
String result = (String) expression.evaluate(document, XPathConstants.STRING);


I'm using the JVM's default implementation of JAXP:



org.apache.xpath.jaxp.XPathFactoryImpl
org.apache.xpath.jaxp.XPathImpl


I'm really confused, because it's easy to see how JAXP could optimise the above XPath query to actually execute a simple getElementsByTagName() instead. But it doesn't seem to do that. This problem is limited to around 5-6 frequently used XPath calls, that are abstracted and hidden by an API. Those queries involve simple paths (e.g. /a/b/c, no variables, conditions) against an always available DOM Document only. So, if an optimisation can be done, it will be quite easy to achieve.



My question: Is XPath's slowness an accepted fact, or am I overlooking something? Is there a better (faster) implementation? Or should I just avoid XPath altogether, for simple queries?










share|improve this question

















NOTE: If you experience this issue as well, please upvote it on Apache JIRA:



https://issues.apache.org/jira/browse/XALANJ-2540




I have come to an astonishing conclusion that this:



Element e = (Element) document.getElementsByTagName("SomeElementName").item(0);
String result = ((Element) e).getTextContent();


Seems to be an incredible 100x faster than this:



// Accounts for 30%, can be cached
XPathFactory factory = XPathFactory.newInstance();

// Negligible
XPath xpath = factory.newXPath();

// Negligible
XPathExpression expression = xpath.compile("//SomeElementName");

// Accounts for 70%
String result = (String) expression.evaluate(document, XPathConstants.STRING);


I'm using the JVM's default implementation of JAXP:



org.apache.xpath.jaxp.XPathFactoryImpl
org.apache.xpath.jaxp.XPathImpl


I'm really confused, because it's easy to see how JAXP could optimise the above XPath query to actually execute a simple getElementsByTagName() instead. But it doesn't seem to do that. This problem is limited to around 5-6 frequently used XPath calls, that are abstracted and hidden by an API. Those queries involve simple paths (e.g. /a/b/c, no variables, conditions) against an always available DOM Document only. So, if an optimisation can be done, it will be quite easy to achieve.



My question: Is XPath's slowness an accepted fact, or am I overlooking something? Is there a better (faster) implementation? Or should I just avoid XPath altogether, for simple queries?







java performance apache xpath jaxp






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Aug 7 '11 at 14:18







Lukas Eder

















asked Jun 14 '11 at 8:14









Lukas EderLukas Eder

135k72441972




135k72441972







  • 3





    Have you tried comparing your results with that of a compiled, reusable XPathExpression?

    – McDowell
    Jun 14 '11 at 8:18











  • Is it problematic slow? One possibility is to evaluate another library, such as jaxen.

    – Johan Sjöberg
    Jun 14 '11 at 8:19











  • @McDowell, nope. I'll try that, thanks

    – Lukas Eder
    Jun 14 '11 at 8:23











  • @Johan: It is. Typically, those queries tend to take 2-5ms a piece, or 10% of a user's request time. We think this starts being problematic, as we're going to use more XPath in the future. jaxen might indeed be an option in the future...

    – Lukas Eder
    Jun 14 '11 at 8:25











  • @McDowell: Caching the expression is negligible, especially compared to caching of the factory. But after some re-tests, I have to correct the times. Caching the factory will accelerate roughly by 30%

    – Lukas Eder
    Jun 14 '11 at 8:28













  • 3





    Have you tried comparing your results with that of a compiled, reusable XPathExpression?

    – McDowell
    Jun 14 '11 at 8:18











  • Is it problematic slow? One possibility is to evaluate another library, such as jaxen.

    – Johan Sjöberg
    Jun 14 '11 at 8:19











  • @McDowell, nope. I'll try that, thanks

    – Lukas Eder
    Jun 14 '11 at 8:23











  • @Johan: It is. Typically, those queries tend to take 2-5ms a piece, or 10% of a user's request time. We think this starts being problematic, as we're going to use more XPath in the future. jaxen might indeed be an option in the future...

    – Lukas Eder
    Jun 14 '11 at 8:25











  • @McDowell: Caching the expression is negligible, especially compared to caching of the factory. But after some re-tests, I have to correct the times. Caching the factory will accelerate roughly by 30%

    – Lukas Eder
    Jun 14 '11 at 8:28








3




3





Have you tried comparing your results with that of a compiled, reusable XPathExpression?

– McDowell
Jun 14 '11 at 8:18





Have you tried comparing your results with that of a compiled, reusable XPathExpression?

– McDowell
Jun 14 '11 at 8:18













Is it problematic slow? One possibility is to evaluate another library, such as jaxen.

– Johan Sjöberg
Jun 14 '11 at 8:19





Is it problematic slow? One possibility is to evaluate another library, such as jaxen.

– Johan Sjöberg
Jun 14 '11 at 8:19













@McDowell, nope. I'll try that, thanks

– Lukas Eder
Jun 14 '11 at 8:23





@McDowell, nope. I'll try that, thanks

– Lukas Eder
Jun 14 '11 at 8:23













@Johan: It is. Typically, those queries tend to take 2-5ms a piece, or 10% of a user's request time. We think this starts being problematic, as we're going to use more XPath in the future. jaxen might indeed be an option in the future...

– Lukas Eder
Jun 14 '11 at 8:25





@Johan: It is. Typically, those queries tend to take 2-5ms a piece, or 10% of a user's request time. We think this starts being problematic, as we're going to use more XPath in the future. jaxen might indeed be an option in the future...

– Lukas Eder
Jun 14 '11 at 8:25













@McDowell: Caching the expression is negligible, especially compared to caching of the factory. But after some re-tests, I have to correct the times. Caching the factory will accelerate roughly by 30%

– Lukas Eder
Jun 14 '11 at 8:28






@McDowell: Caching the expression is negligible, especially compared to caching of the factory. But after some re-tests, I have to correct the times. Caching the factory will accelerate roughly by 30%

– Lukas Eder
Jun 14 '11 at 8:28













3 Answers
3






active

oldest

votes


















60














I have debugged and profiled my test-case and Xalan/JAXP in general. I managed to identify the big major problem in



org.apache.xml.dtm.ObjectFactory.lookUpFactoryClassName()


It can be seen that every one of the 10k test XPath evaluations led to the classloader trying to lookup the DTMManager instance in some sort of default configuration. This configuration is not loaded into memory but accessed every time. Furthermore, this access seems to be protected by a lock on the ObjectFactory.class itself. When the access fails (by default), then the configuration is loaded from the xalan.jar file's



META-INF/service/org.apache.xml.dtm.DTMManager


configuration file. Every time!:



JProfiler profiling results



Fortunately, this behaviour can be overridden by specifying a JVM parameter like this:



-Dorg.apache.xml.dtm.DTMManager=
org.apache.xml.dtm.ref.DTMManagerDefault


or



-Dcom.sun.org.apache.xml.internal.dtm.DTMManager=
com.sun.org.apache.xml.internal.dtm.ref.DTMManagerDefault


The above works, as this will allow to bypass the expensive work in lookUpFactoryClassName() if the factory class name is the default anyway:



// Code from com.sun.org.apache.xml.internal.dtm.ObjectFactory
static String lookUpFactoryClassName(String factoryId,
String propertiesFilename,
String fallbackClassName) {
SecuritySupport ss = SecuritySupport.getInstance();

try
String systemProp = ss.getSystemProperty(factoryId);
if (systemProp != null)

// Return early from the method
return systemProp;

catch (SecurityException se)


// [...] "Heavy" operations later


So here's a performance improvement overview for 10k consecutive XPath evaluations of //SomeNodeName against a 90k XML file (measured with System.nanoTime():



measured library : Xalan 2.7.0 | Xalan 2.7.1 | Saxon-HE 9.3 | jaxen 1.1.3
--------------------------------------------------------------------------------
without optimisation : 10400ms | 4717ms | | 25500ms
reusing XPathFactory : 5995ms | 2829ms | |
reusing XPath : 5900ms | 2890ms | |
reusing XPathExpression : 5800ms | 2915ms | 16000ms | 25000ms
adding the JVM param : 1163ms | 761ms | n/a |


note that the benchmark was a very primitive one. it may well be that your own benchmark will show that saxon outperforms xalan



I have filed this as a bug to the Xalan guys at Apache:



https://issues.apache.org/jira/browse/XALANJ-2540






share|improve this answer

























  • Very interesting. I would be fascinated to see what numbers you get if you substitute the Saxon implementation of XPathFactory. I know there is a big overhead for searching the classpath for the XPath factory, and I suspect there's a significant overhead for parsing the XPath expression, but I think the actual execution should be pretty fast.

    – Michael Kay
    Jun 14 '11 at 10:10











  • @Michael: You're affiliated with Saxon, I see? I downloaded the home edition and ran my benchmarks against it. Where Xalan had 5800ms, Saxon took 16000ms...

    – Lukas Eder
    Jun 14 '11 at 10:34











  • A great, low cost optimisation - Thanks Lukas! I note that re-using XpathFactory, Xpath, XpathExpression is not possible across multiple threads, all three are unfortunately not thread-safe. Did you find any further optimisation of quicker alternative libraries??

    – joelittlejohn
    Jun 17 '11 at 16:31











  • @joelittlejohn, nope. Only what I've mentioned. 1) transform simple XPath expressions into DOM API calls, 2) add the JVM parameter and maybe I'm going to add 3) a shared pool of XPathFactory to thread-safely reuse them. With these measurements, I expect XPath queries to take only about 5% of today's XPath CPU time.

    – Lukas Eder
    Jun 17 '11 at 18:31






  • 1





    See also the 2009 blog post scn.sap.com/community/java/blog/2009/12/04/…

    – JasonPlutext
    Nov 13 '13 at 22:37


















6














Not a solution, but a pointer to the main problem:
The slowest part of the process for evaluating an xpath in relation to an arbitrary node is the time it takes the DTM manager to find the node handle:



http://javasourcecode.org/html/open-source/jdk/jdk-6u23/com/sun/org/apache/xml/internal/dtm/ref/dom2dtm/DOM2DTM.html#getHandleOfNode%28org.w3c.dom.Node%29



If the node in question is at the end of the Document, it can end up walking the entire tree to find the node in question, for each and every query.



This explains why the hack to orphan out the target node works.
There should be a way to cache these lookups, but at this point I can't see how.






share|improve this answer























  • Hmm, that's interesting input. I have to profile again, though. It hasn't occurred to me so far that this method had any relevant impact on the overall performance.

    – Lukas Eder
    Dec 23 '11 at 9:55






  • 1





    I ran into this recently, and discovered the ridiculously inefficient DTM handle lookup by stepping through it in the debugger. Who would've thought that passing a context node, and an expression that selects a direct child of the context node, would involve walking the entire DOM tree? I rewrote my loading code using StAX instead of XPath and my run time went from 5 hours to half a second.

    – Wyzard
    Mar 27 '12 at 1:16


















0














To answer your question, vtd-xml is way faster than Jaxen or Xalan) (I would say on average
10x, and 60x has been reported...






share|improve this answer























  • Interesting. Can you provide a bit more details to back your claims? While this won't be applicable to my original problem anymore, I'll certainly follow up on your tool!

    – Lukas Eder
    Jul 29 '13 at 8:46











  • Well, vtd-xml is a simple tool to download, it takes 10 min to test, go to vtd-xml web site and there are lots of relevant info including benchmark tests comparing to Jaxen, and xalan. Our internal results actually show that jaxen's performance has deteriorated considerably over the past several releases..

    – vtd-xml-author
    Jul 29 '13 at 17:03










Your Answer






StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f6340802%2fjava-xpath-apache-jaxp-implementation-performance%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























3 Answers
3






active

oldest

votes








3 Answers
3






active

oldest

votes









active

oldest

votes






active

oldest

votes









60














I have debugged and profiled my test-case and Xalan/JAXP in general. I managed to identify the big major problem in



org.apache.xml.dtm.ObjectFactory.lookUpFactoryClassName()


It can be seen that every one of the 10k test XPath evaluations led to the classloader trying to lookup the DTMManager instance in some sort of default configuration. This configuration is not loaded into memory but accessed every time. Furthermore, this access seems to be protected by a lock on the ObjectFactory.class itself. When the access fails (by default), then the configuration is loaded from the xalan.jar file's



META-INF/service/org.apache.xml.dtm.DTMManager


configuration file. Every time!:



JProfiler profiling results



Fortunately, this behaviour can be overridden by specifying a JVM parameter like this:



-Dorg.apache.xml.dtm.DTMManager=
org.apache.xml.dtm.ref.DTMManagerDefault


or



-Dcom.sun.org.apache.xml.internal.dtm.DTMManager=
com.sun.org.apache.xml.internal.dtm.ref.DTMManagerDefault


The above works, as this will allow to bypass the expensive work in lookUpFactoryClassName() if the factory class name is the default anyway:



// Code from com.sun.org.apache.xml.internal.dtm.ObjectFactory
static String lookUpFactoryClassName(String factoryId,
String propertiesFilename,
String fallbackClassName) {
SecuritySupport ss = SecuritySupport.getInstance();

try
String systemProp = ss.getSystemProperty(factoryId);
if (systemProp != null)

// Return early from the method
return systemProp;

catch (SecurityException se)


// [...] "Heavy" operations later


So here's a performance improvement overview for 10k consecutive XPath evaluations of //SomeNodeName against a 90k XML file (measured with System.nanoTime():



measured library : Xalan 2.7.0 | Xalan 2.7.1 | Saxon-HE 9.3 | jaxen 1.1.3
--------------------------------------------------------------------------------
without optimisation : 10400ms | 4717ms | | 25500ms
reusing XPathFactory : 5995ms | 2829ms | |
reusing XPath : 5900ms | 2890ms | |
reusing XPathExpression : 5800ms | 2915ms | 16000ms | 25000ms
adding the JVM param : 1163ms | 761ms | n/a |


note that the benchmark was a very primitive one. it may well be that your own benchmark will show that saxon outperforms xalan



I have filed this as a bug to the Xalan guys at Apache:



https://issues.apache.org/jira/browse/XALANJ-2540






share|improve this answer

























  • Very interesting. I would be fascinated to see what numbers you get if you substitute the Saxon implementation of XPathFactory. I know there is a big overhead for searching the classpath for the XPath factory, and I suspect there's a significant overhead for parsing the XPath expression, but I think the actual execution should be pretty fast.

    – Michael Kay
    Jun 14 '11 at 10:10











  • @Michael: You're affiliated with Saxon, I see? I downloaded the home edition and ran my benchmarks against it. Where Xalan had 5800ms, Saxon took 16000ms...

    – Lukas Eder
    Jun 14 '11 at 10:34











  • A great, low cost optimisation - Thanks Lukas! I note that re-using XpathFactory, Xpath, XpathExpression is not possible across multiple threads, all three are unfortunately not thread-safe. Did you find any further optimisation of quicker alternative libraries??

    – joelittlejohn
    Jun 17 '11 at 16:31











  • @joelittlejohn, nope. Only what I've mentioned. 1) transform simple XPath expressions into DOM API calls, 2) add the JVM parameter and maybe I'm going to add 3) a shared pool of XPathFactory to thread-safely reuse them. With these measurements, I expect XPath queries to take only about 5% of today's XPath CPU time.

    – Lukas Eder
    Jun 17 '11 at 18:31






  • 1





    See also the 2009 blog post scn.sap.com/community/java/blog/2009/12/04/…

    – JasonPlutext
    Nov 13 '13 at 22:37















60














I have debugged and profiled my test-case and Xalan/JAXP in general. I managed to identify the big major problem in



org.apache.xml.dtm.ObjectFactory.lookUpFactoryClassName()


It can be seen that every one of the 10k test XPath evaluations led to the classloader trying to lookup the DTMManager instance in some sort of default configuration. This configuration is not loaded into memory but accessed every time. Furthermore, this access seems to be protected by a lock on the ObjectFactory.class itself. When the access fails (by default), then the configuration is loaded from the xalan.jar file's



META-INF/service/org.apache.xml.dtm.DTMManager


configuration file. Every time!:



JProfiler profiling results



Fortunately, this behaviour can be overridden by specifying a JVM parameter like this:



-Dorg.apache.xml.dtm.DTMManager=
org.apache.xml.dtm.ref.DTMManagerDefault


or



-Dcom.sun.org.apache.xml.internal.dtm.DTMManager=
com.sun.org.apache.xml.internal.dtm.ref.DTMManagerDefault


The above works, as this will allow to bypass the expensive work in lookUpFactoryClassName() if the factory class name is the default anyway:



// Code from com.sun.org.apache.xml.internal.dtm.ObjectFactory
static String lookUpFactoryClassName(String factoryId,
String propertiesFilename,
String fallbackClassName) {
SecuritySupport ss = SecuritySupport.getInstance();

try
String systemProp = ss.getSystemProperty(factoryId);
if (systemProp != null)

// Return early from the method
return systemProp;

catch (SecurityException se)


// [...] "Heavy" operations later


So here's a performance improvement overview for 10k consecutive XPath evaluations of //SomeNodeName against a 90k XML file (measured with System.nanoTime():



measured library : Xalan 2.7.0 | Xalan 2.7.1 | Saxon-HE 9.3 | jaxen 1.1.3
--------------------------------------------------------------------------------
without optimisation : 10400ms | 4717ms | | 25500ms
reusing XPathFactory : 5995ms | 2829ms | |
reusing XPath : 5900ms | 2890ms | |
reusing XPathExpression : 5800ms | 2915ms | 16000ms | 25000ms
adding the JVM param : 1163ms | 761ms | n/a |


note that the benchmark was a very primitive one. it may well be that your own benchmark will show that saxon outperforms xalan



I have filed this as a bug to the Xalan guys at Apache:



https://issues.apache.org/jira/browse/XALANJ-2540






share|improve this answer

























  • Very interesting. I would be fascinated to see what numbers you get if you substitute the Saxon implementation of XPathFactory. I know there is a big overhead for searching the classpath for the XPath factory, and I suspect there's a significant overhead for parsing the XPath expression, but I think the actual execution should be pretty fast.

    – Michael Kay
    Jun 14 '11 at 10:10











  • @Michael: You're affiliated with Saxon, I see? I downloaded the home edition and ran my benchmarks against it. Where Xalan had 5800ms, Saxon took 16000ms...

    – Lukas Eder
    Jun 14 '11 at 10:34











  • A great, low cost optimisation - Thanks Lukas! I note that re-using XpathFactory, Xpath, XpathExpression is not possible across multiple threads, all three are unfortunately not thread-safe. Did you find any further optimisation of quicker alternative libraries??

    – joelittlejohn
    Jun 17 '11 at 16:31











  • @joelittlejohn, nope. Only what I've mentioned. 1) transform simple XPath expressions into DOM API calls, 2) add the JVM parameter and maybe I'm going to add 3) a shared pool of XPathFactory to thread-safely reuse them. With these measurements, I expect XPath queries to take only about 5% of today's XPath CPU time.

    – Lukas Eder
    Jun 17 '11 at 18:31






  • 1





    See also the 2009 blog post scn.sap.com/community/java/blog/2009/12/04/…

    – JasonPlutext
    Nov 13 '13 at 22:37













60












60








60







I have debugged and profiled my test-case and Xalan/JAXP in general. I managed to identify the big major problem in



org.apache.xml.dtm.ObjectFactory.lookUpFactoryClassName()


It can be seen that every one of the 10k test XPath evaluations led to the classloader trying to lookup the DTMManager instance in some sort of default configuration. This configuration is not loaded into memory but accessed every time. Furthermore, this access seems to be protected by a lock on the ObjectFactory.class itself. When the access fails (by default), then the configuration is loaded from the xalan.jar file's



META-INF/service/org.apache.xml.dtm.DTMManager


configuration file. Every time!:



JProfiler profiling results



Fortunately, this behaviour can be overridden by specifying a JVM parameter like this:



-Dorg.apache.xml.dtm.DTMManager=
org.apache.xml.dtm.ref.DTMManagerDefault


or



-Dcom.sun.org.apache.xml.internal.dtm.DTMManager=
com.sun.org.apache.xml.internal.dtm.ref.DTMManagerDefault


The above works, as this will allow to bypass the expensive work in lookUpFactoryClassName() if the factory class name is the default anyway:



// Code from com.sun.org.apache.xml.internal.dtm.ObjectFactory
static String lookUpFactoryClassName(String factoryId,
String propertiesFilename,
String fallbackClassName) {
SecuritySupport ss = SecuritySupport.getInstance();

try
String systemProp = ss.getSystemProperty(factoryId);
if (systemProp != null)

// Return early from the method
return systemProp;

catch (SecurityException se)


// [...] "Heavy" operations later


So here's a performance improvement overview for 10k consecutive XPath evaluations of //SomeNodeName against a 90k XML file (measured with System.nanoTime():



measured library : Xalan 2.7.0 | Xalan 2.7.1 | Saxon-HE 9.3 | jaxen 1.1.3
--------------------------------------------------------------------------------
without optimisation : 10400ms | 4717ms | | 25500ms
reusing XPathFactory : 5995ms | 2829ms | |
reusing XPath : 5900ms | 2890ms | |
reusing XPathExpression : 5800ms | 2915ms | 16000ms | 25000ms
adding the JVM param : 1163ms | 761ms | n/a |


note that the benchmark was a very primitive one. it may well be that your own benchmark will show that saxon outperforms xalan



I have filed this as a bug to the Xalan guys at Apache:



https://issues.apache.org/jira/browse/XALANJ-2540






share|improve this answer















I have debugged and profiled my test-case and Xalan/JAXP in general. I managed to identify the big major problem in



org.apache.xml.dtm.ObjectFactory.lookUpFactoryClassName()


It can be seen that every one of the 10k test XPath evaluations led to the classloader trying to lookup the DTMManager instance in some sort of default configuration. This configuration is not loaded into memory but accessed every time. Furthermore, this access seems to be protected by a lock on the ObjectFactory.class itself. When the access fails (by default), then the configuration is loaded from the xalan.jar file's



META-INF/service/org.apache.xml.dtm.DTMManager


configuration file. Every time!:



JProfiler profiling results



Fortunately, this behaviour can be overridden by specifying a JVM parameter like this:



-Dorg.apache.xml.dtm.DTMManager=
org.apache.xml.dtm.ref.DTMManagerDefault


or



-Dcom.sun.org.apache.xml.internal.dtm.DTMManager=
com.sun.org.apache.xml.internal.dtm.ref.DTMManagerDefault


The above works, as this will allow to bypass the expensive work in lookUpFactoryClassName() if the factory class name is the default anyway:



// Code from com.sun.org.apache.xml.internal.dtm.ObjectFactory
static String lookUpFactoryClassName(String factoryId,
String propertiesFilename,
String fallbackClassName) {
SecuritySupport ss = SecuritySupport.getInstance();

try
String systemProp = ss.getSystemProperty(factoryId);
if (systemProp != null)

// Return early from the method
return systemProp;

catch (SecurityException se)


// [...] "Heavy" operations later


So here's a performance improvement overview for 10k consecutive XPath evaluations of //SomeNodeName against a 90k XML file (measured with System.nanoTime():



measured library : Xalan 2.7.0 | Xalan 2.7.1 | Saxon-HE 9.3 | jaxen 1.1.3
--------------------------------------------------------------------------------
without optimisation : 10400ms | 4717ms | | 25500ms
reusing XPathFactory : 5995ms | 2829ms | |
reusing XPath : 5900ms | 2890ms | |
reusing XPathExpression : 5800ms | 2915ms | 16000ms | 25000ms
adding the JVM param : 1163ms | 761ms | n/a |


note that the benchmark was a very primitive one. it may well be that your own benchmark will show that saxon outperforms xalan



I have filed this as a bug to the Xalan guys at Apache:



https://issues.apache.org/jira/browse/XALANJ-2540







share|improve this answer














share|improve this answer



share|improve this answer








edited Sep 18 '14 at 15:45

























answered Jun 14 '11 at 9:45









Lukas EderLukas Eder

135k72441972




135k72441972












  • Very interesting. I would be fascinated to see what numbers you get if you substitute the Saxon implementation of XPathFactory. I know there is a big overhead for searching the classpath for the XPath factory, and I suspect there's a significant overhead for parsing the XPath expression, but I think the actual execution should be pretty fast.

    – Michael Kay
    Jun 14 '11 at 10:10











  • @Michael: You're affiliated with Saxon, I see? I downloaded the home edition and ran my benchmarks against it. Where Xalan had 5800ms, Saxon took 16000ms...

    – Lukas Eder
    Jun 14 '11 at 10:34











  • A great, low cost optimisation - Thanks Lukas! I note that re-using XpathFactory, Xpath, XpathExpression is not possible across multiple threads, all three are unfortunately not thread-safe. Did you find any further optimisation of quicker alternative libraries??

    – joelittlejohn
    Jun 17 '11 at 16:31











  • @joelittlejohn, nope. Only what I've mentioned. 1) transform simple XPath expressions into DOM API calls, 2) add the JVM parameter and maybe I'm going to add 3) a shared pool of XPathFactory to thread-safely reuse them. With these measurements, I expect XPath queries to take only about 5% of today's XPath CPU time.

    – Lukas Eder
    Jun 17 '11 at 18:31






  • 1





    See also the 2009 blog post scn.sap.com/community/java/blog/2009/12/04/…

    – JasonPlutext
    Nov 13 '13 at 22:37

















  • Very interesting. I would be fascinated to see what numbers you get if you substitute the Saxon implementation of XPathFactory. I know there is a big overhead for searching the classpath for the XPath factory, and I suspect there's a significant overhead for parsing the XPath expression, but I think the actual execution should be pretty fast.

    – Michael Kay
    Jun 14 '11 at 10:10











  • @Michael: You're affiliated with Saxon, I see? I downloaded the home edition and ran my benchmarks against it. Where Xalan had 5800ms, Saxon took 16000ms...

    – Lukas Eder
    Jun 14 '11 at 10:34











  • A great, low cost optimisation - Thanks Lukas! I note that re-using XpathFactory, Xpath, XpathExpression is not possible across multiple threads, all three are unfortunately not thread-safe. Did you find any further optimisation of quicker alternative libraries??

    – joelittlejohn
    Jun 17 '11 at 16:31











  • @joelittlejohn, nope. Only what I've mentioned. 1) transform simple XPath expressions into DOM API calls, 2) add the JVM parameter and maybe I'm going to add 3) a shared pool of XPathFactory to thread-safely reuse them. With these measurements, I expect XPath queries to take only about 5% of today's XPath CPU time.

    – Lukas Eder
    Jun 17 '11 at 18:31






  • 1





    See also the 2009 blog post scn.sap.com/community/java/blog/2009/12/04/…

    – JasonPlutext
    Nov 13 '13 at 22:37
















Very interesting. I would be fascinated to see what numbers you get if you substitute the Saxon implementation of XPathFactory. I know there is a big overhead for searching the classpath for the XPath factory, and I suspect there's a significant overhead for parsing the XPath expression, but I think the actual execution should be pretty fast.

– Michael Kay
Jun 14 '11 at 10:10





Very interesting. I would be fascinated to see what numbers you get if you substitute the Saxon implementation of XPathFactory. I know there is a big overhead for searching the classpath for the XPath factory, and I suspect there's a significant overhead for parsing the XPath expression, but I think the actual execution should be pretty fast.

– Michael Kay
Jun 14 '11 at 10:10













@Michael: You're affiliated with Saxon, I see? I downloaded the home edition and ran my benchmarks against it. Where Xalan had 5800ms, Saxon took 16000ms...

– Lukas Eder
Jun 14 '11 at 10:34





@Michael: You're affiliated with Saxon, I see? I downloaded the home edition and ran my benchmarks against it. Where Xalan had 5800ms, Saxon took 16000ms...

– Lukas Eder
Jun 14 '11 at 10:34













A great, low cost optimisation - Thanks Lukas! I note that re-using XpathFactory, Xpath, XpathExpression is not possible across multiple threads, all three are unfortunately not thread-safe. Did you find any further optimisation of quicker alternative libraries??

– joelittlejohn
Jun 17 '11 at 16:31





A great, low cost optimisation - Thanks Lukas! I note that re-using XpathFactory, Xpath, XpathExpression is not possible across multiple threads, all three are unfortunately not thread-safe. Did you find any further optimisation of quicker alternative libraries??

– joelittlejohn
Jun 17 '11 at 16:31













@joelittlejohn, nope. Only what I've mentioned. 1) transform simple XPath expressions into DOM API calls, 2) add the JVM parameter and maybe I'm going to add 3) a shared pool of XPathFactory to thread-safely reuse them. With these measurements, I expect XPath queries to take only about 5% of today's XPath CPU time.

– Lukas Eder
Jun 17 '11 at 18:31





@joelittlejohn, nope. Only what I've mentioned. 1) transform simple XPath expressions into DOM API calls, 2) add the JVM parameter and maybe I'm going to add 3) a shared pool of XPathFactory to thread-safely reuse them. With these measurements, I expect XPath queries to take only about 5% of today's XPath CPU time.

– Lukas Eder
Jun 17 '11 at 18:31




1




1





See also the 2009 blog post scn.sap.com/community/java/blog/2009/12/04/…

– JasonPlutext
Nov 13 '13 at 22:37





See also the 2009 blog post scn.sap.com/community/java/blog/2009/12/04/…

– JasonPlutext
Nov 13 '13 at 22:37













6














Not a solution, but a pointer to the main problem:
The slowest part of the process for evaluating an xpath in relation to an arbitrary node is the time it takes the DTM manager to find the node handle:



http://javasourcecode.org/html/open-source/jdk/jdk-6u23/com/sun/org/apache/xml/internal/dtm/ref/dom2dtm/DOM2DTM.html#getHandleOfNode%28org.w3c.dom.Node%29



If the node in question is at the end of the Document, it can end up walking the entire tree to find the node in question, for each and every query.



This explains why the hack to orphan out the target node works.
There should be a way to cache these lookups, but at this point I can't see how.






share|improve this answer























  • Hmm, that's interesting input. I have to profile again, though. It hasn't occurred to me so far that this method had any relevant impact on the overall performance.

    – Lukas Eder
    Dec 23 '11 at 9:55






  • 1





    I ran into this recently, and discovered the ridiculously inefficient DTM handle lookup by stepping through it in the debugger. Who would've thought that passing a context node, and an expression that selects a direct child of the context node, would involve walking the entire DOM tree? I rewrote my loading code using StAX instead of XPath and my run time went from 5 hours to half a second.

    – Wyzard
    Mar 27 '12 at 1:16















6














Not a solution, but a pointer to the main problem:
The slowest part of the process for evaluating an xpath in relation to an arbitrary node is the time it takes the DTM manager to find the node handle:



http://javasourcecode.org/html/open-source/jdk/jdk-6u23/com/sun/org/apache/xml/internal/dtm/ref/dom2dtm/DOM2DTM.html#getHandleOfNode%28org.w3c.dom.Node%29



If the node in question is at the end of the Document, it can end up walking the entire tree to find the node in question, for each and every query.



This explains why the hack to orphan out the target node works.
There should be a way to cache these lookups, but at this point I can't see how.






share|improve this answer























  • Hmm, that's interesting input. I have to profile again, though. It hasn't occurred to me so far that this method had any relevant impact on the overall performance.

    – Lukas Eder
    Dec 23 '11 at 9:55






  • 1





    I ran into this recently, and discovered the ridiculously inefficient DTM handle lookup by stepping through it in the debugger. Who would've thought that passing a context node, and an expression that selects a direct child of the context node, would involve walking the entire DOM tree? I rewrote my loading code using StAX instead of XPath and my run time went from 5 hours to half a second.

    – Wyzard
    Mar 27 '12 at 1:16













6












6








6







Not a solution, but a pointer to the main problem:
The slowest part of the process for evaluating an xpath in relation to an arbitrary node is the time it takes the DTM manager to find the node handle:



http://javasourcecode.org/html/open-source/jdk/jdk-6u23/com/sun/org/apache/xml/internal/dtm/ref/dom2dtm/DOM2DTM.html#getHandleOfNode%28org.w3c.dom.Node%29



If the node in question is at the end of the Document, it can end up walking the entire tree to find the node in question, for each and every query.



This explains why the hack to orphan out the target node works.
There should be a way to cache these lookups, but at this point I can't see how.






share|improve this answer













Not a solution, but a pointer to the main problem:
The slowest part of the process for evaluating an xpath in relation to an arbitrary node is the time it takes the DTM manager to find the node handle:



http://javasourcecode.org/html/open-source/jdk/jdk-6u23/com/sun/org/apache/xml/internal/dtm/ref/dom2dtm/DOM2DTM.html#getHandleOfNode%28org.w3c.dom.Node%29



If the node in question is at the end of the Document, it can end up walking the entire tree to find the node in question, for each and every query.



This explains why the hack to orphan out the target node works.
There should be a way to cache these lookups, but at this point I can't see how.







share|improve this answer












share|improve this answer



share|improve this answer










answered Dec 23 '11 at 7:32









Robbie MatthewsRobbie Matthews

605512




605512












  • Hmm, that's interesting input. I have to profile again, though. It hasn't occurred to me so far that this method had any relevant impact on the overall performance.

    – Lukas Eder
    Dec 23 '11 at 9:55






  • 1





    I ran into this recently, and discovered the ridiculously inefficient DTM handle lookup by stepping through it in the debugger. Who would've thought that passing a context node, and an expression that selects a direct child of the context node, would involve walking the entire DOM tree? I rewrote my loading code using StAX instead of XPath and my run time went from 5 hours to half a second.

    – Wyzard
    Mar 27 '12 at 1:16

















  • Hmm, that's interesting input. I have to profile again, though. It hasn't occurred to me so far that this method had any relevant impact on the overall performance.

    – Lukas Eder
    Dec 23 '11 at 9:55






  • 1





    I ran into this recently, and discovered the ridiculously inefficient DTM handle lookup by stepping through it in the debugger. Who would've thought that passing a context node, and an expression that selects a direct child of the context node, would involve walking the entire DOM tree? I rewrote my loading code using StAX instead of XPath and my run time went from 5 hours to half a second.

    – Wyzard
    Mar 27 '12 at 1:16
















Hmm, that's interesting input. I have to profile again, though. It hasn't occurred to me so far that this method had any relevant impact on the overall performance.

– Lukas Eder
Dec 23 '11 at 9:55





Hmm, that's interesting input. I have to profile again, though. It hasn't occurred to me so far that this method had any relevant impact on the overall performance.

– Lukas Eder
Dec 23 '11 at 9:55




1




1





I ran into this recently, and discovered the ridiculously inefficient DTM handle lookup by stepping through it in the debugger. Who would've thought that passing a context node, and an expression that selects a direct child of the context node, would involve walking the entire DOM tree? I rewrote my loading code using StAX instead of XPath and my run time went from 5 hours to half a second.

– Wyzard
Mar 27 '12 at 1:16





I ran into this recently, and discovered the ridiculously inefficient DTM handle lookup by stepping through it in the debugger. Who would've thought that passing a context node, and an expression that selects a direct child of the context node, would involve walking the entire DOM tree? I rewrote my loading code using StAX instead of XPath and my run time went from 5 hours to half a second.

– Wyzard
Mar 27 '12 at 1:16











0














To answer your question, vtd-xml is way faster than Jaxen or Xalan) (I would say on average
10x, and 60x has been reported...






share|improve this answer























  • Interesting. Can you provide a bit more details to back your claims? While this won't be applicable to my original problem anymore, I'll certainly follow up on your tool!

    – Lukas Eder
    Jul 29 '13 at 8:46











  • Well, vtd-xml is a simple tool to download, it takes 10 min to test, go to vtd-xml web site and there are lots of relevant info including benchmark tests comparing to Jaxen, and xalan. Our internal results actually show that jaxen's performance has deteriorated considerably over the past several releases..

    – vtd-xml-author
    Jul 29 '13 at 17:03















0














To answer your question, vtd-xml is way faster than Jaxen or Xalan) (I would say on average
10x, and 60x has been reported...






share|improve this answer























  • Interesting. Can you provide a bit more details to back your claims? While this won't be applicable to my original problem anymore, I'll certainly follow up on your tool!

    – Lukas Eder
    Jul 29 '13 at 8:46











  • Well, vtd-xml is a simple tool to download, it takes 10 min to test, go to vtd-xml web site and there are lots of relevant info including benchmark tests comparing to Jaxen, and xalan. Our internal results actually show that jaxen's performance has deteriorated considerably over the past several releases..

    – vtd-xml-author
    Jul 29 '13 at 17:03













0












0








0







To answer your question, vtd-xml is way faster than Jaxen or Xalan) (I would say on average
10x, and 60x has been reported...






share|improve this answer













To answer your question, vtd-xml is way faster than Jaxen or Xalan) (I would say on average
10x, and 60x has been reported...







share|improve this answer












share|improve this answer



share|improve this answer










answered Jul 29 '13 at 8:14









vtd-xml-authorvtd-xml-author

2,86141727




2,86141727












  • Interesting. Can you provide a bit more details to back your claims? While this won't be applicable to my original problem anymore, I'll certainly follow up on your tool!

    – Lukas Eder
    Jul 29 '13 at 8:46











  • Well, vtd-xml is a simple tool to download, it takes 10 min to test, go to vtd-xml web site and there are lots of relevant info including benchmark tests comparing to Jaxen, and xalan. Our internal results actually show that jaxen's performance has deteriorated considerably over the past several releases..

    – vtd-xml-author
    Jul 29 '13 at 17:03

















  • Interesting. Can you provide a bit more details to back your claims? While this won't be applicable to my original problem anymore, I'll certainly follow up on your tool!

    – Lukas Eder
    Jul 29 '13 at 8:46











  • Well, vtd-xml is a simple tool to download, it takes 10 min to test, go to vtd-xml web site and there are lots of relevant info including benchmark tests comparing to Jaxen, and xalan. Our internal results actually show that jaxen's performance has deteriorated considerably over the past several releases..

    – vtd-xml-author
    Jul 29 '13 at 17:03
















Interesting. Can you provide a bit more details to back your claims? While this won't be applicable to my original problem anymore, I'll certainly follow up on your tool!

– Lukas Eder
Jul 29 '13 at 8:46





Interesting. Can you provide a bit more details to back your claims? While this won't be applicable to my original problem anymore, I'll certainly follow up on your tool!

– Lukas Eder
Jul 29 '13 at 8:46













Well, vtd-xml is a simple tool to download, it takes 10 min to test, go to vtd-xml web site and there are lots of relevant info including benchmark tests comparing to Jaxen, and xalan. Our internal results actually show that jaxen's performance has deteriorated considerably over the past several releases..

– vtd-xml-author
Jul 29 '13 at 17:03





Well, vtd-xml is a simple tool to download, it takes 10 min to test, go to vtd-xml web site and there are lots of relevant info including benchmark tests comparing to Jaxen, and xalan. Our internal results actually show that jaxen's performance has deteriorated considerably over the past several releases..

– vtd-xml-author
Jul 29 '13 at 17:03

















draft saved

draft discarded
















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f6340802%2fjava-xpath-apache-jaxp-implementation-performance%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

How to get text form Clipboard with JavaScript in Firefox 56?How to validate an email address in JavaScript?How do JavaScript closures work?How do I remove a property from a JavaScript object?How do you get a timestamp in JavaScript?How do I copy to the clipboard in JavaScript?How do I include a JavaScript file in another JavaScript file?Get the current URL with JavaScript?How to replace all occurrences of a string in JavaScriptHow to check whether a string contains a substring in JavaScript?How do I remove a particular element from an array in JavaScript?

Can't initialize raids on a new ASUS Prime B360M-A motherboard2019 Community Moderator ElectionSimilar to RAID config yet more like mirroring solution?Can't get motherboard serial numberWhy does the BIOS entry point start with a WBINVD instruction?UEFI performance Asus Maximus V Extreme

List of MPs elected to the English parliament in 1640 (April) Contents List of constituencies and members See also Notes References Navigation menueNational Archives – The Glynde Place ArchivesCobbett's Parliamentary history of England, from the Norman Conquest in 1066 to the year 1803'Aldermen in Parliament', The Aldermen of the City of London: Temp. Henry III – 1912onepage&q&f&#61, false 229