This means that the port has now achieved complete parity with the official project, and further adds full support for GenXDM for all public APIs.

The 1.4.5 release of Santuario introduced some additional test cases, and as before, I copied those test cases and ported them to use GenXDM. This means that all existing test cases continue to run unmodified, and there are yet more test cases that confirm GenXDM based functionality.

Particularly if you only use the library via the javax.xml.crypto APIs, this library should now be a complete drop-in replacement for existing uses, and enables the use of GenXDM whenever you want it….

Please feel free to kick the tires, and let us know about issues you find with either the port, or GenXDM itself!

As with the port of the rest of the library, I preserved the existing API while rewriting the existing DOM-based code to use GenXDM constructs. For example, in places where a DOMStructure gets passed the implementation observes that it is an instance of that class, and provides an appropriate instance of Model and DocumentHandlerFactory to code that does the actual work. Of course, clients need to be able to pass their own instances of Model and DocumentHandlerFactory. To that end, I added a GenXDMStructure class, which is analogous to the DOMStructure class, and adds instances of the Model and DocumentHandlerFactory.

Following the pattern of DOMStructure/GenXDMStructure, for four other classes in the original API (DOMCryptoContext, DOMSignContext, DOMURIReference, and DOMValidateContext), I created new classes for GenXDM constructs. Thus, a client wishing to use the GenXDM form of the implementation merely needs to instantiate these new constructs, rather than the old DOM-specific ones.

Since the port modifies the existing DOM code to use GenXDM, this means that all of the current test cases help to validate the quality of the port. To complete the picture, though, I provided additional test cases that use the APIs with non-DOM underlying trees. Since many of the existing test cases rely on a “SignatureValidator” class, and don’t have any DOM specific code, I updated the “SignatureValidator” class to enable switching amongst different underlying tree models. Fortunately, that left only a small number of additional test case files that I needed to copy and rewrite to use GenXDM constructs.

Please check out the 1.4.4-r02 release, and let us know how it works for you. It should be functionally equivalent to the Santuario 1.4.4 release for DOM. Of course, since it uses GenXDM, it supports non-DOM XML trees for which bridges exist – that we know of, this includes Axiom & our “Cx” reference implementation.

The next release will address issue 14: porting changes made in the 1.4.5 release.

This release is supposed to be functionally equivalent to the xmlsec-1.4.4 release. I’ve attempted to keep API compatibility, but only for those methods actually being used. For example, all of the official test cases for the library pass unmodified. However, if you use some of the less frequent entry points, not covered by the test cases, then they might not be there. If that’s the case, let me know, and we can restore them!

Please feel free to take it for a spin!

For details about the project, check the project page.

Of course, you could search the issues list, but you might miss some of the subtlety, so let me see if I can summarize the key points:

• 34 & 16 & 19: We changed the mutable API to match, as closely as possible, the XQuery Update Facility specification.
• 48: Reorganized core API packages. We wanted to get this out of the way so that we can consider the API stable.
• 9 & 11: Significant overhaul to the XPath API. See the previous post on this.
• 35: Switched to using JUnit 4. OK, well maybe clients of the code won’t care, but we do.
• 41: Clarified meaning of node identity
• 12 & 42 and various commits: DOM bridge improvements & bug fixes.
• A completely overhauled test suite that does a much better job of validating the implementation of a bridge. Also fixed the DOM & Cx bridges to match the test suite.
• A completely overhauled, and almost complete implementation of a bridge for Axiom. Note that we since fixed a bug related to the axiom bridge that didn’t make it in this release.

The bridge implementations now get tested far more fully, and we’re far more confident of their completeness, now. Please dive in and let us know what you think of the latest release!

One of our recent key changes dealt with a long-standing bugaboo of mine. Specifically, I filed bugs 9 and 11. Issues of the sort represented by this pair will cause us headaches further on if we don’t deal with them now. That’s because we don’t want to change the API more than we have to, but we definitely don’t want to change it after we reach a “1.0″ milestone. Issue 11, in particular, noted the presence of a lot of concrete classes in an apparent API package.

Taking the approach of diving in and seeing what happens, I started replacing all the references to concrete classes with references to interfaces. The intermediate results caused some angst (“IConvertible____” interfaces) suggesting that the abstractions were not quite right. After a brief email exchange, we figured out how to address that problem, and turned forty-one class files as part of an API into a much smaller collection of what is now just twenty-two, and also reduced the package count down to just two.

As a nice side benefit, I can describe the purpose of the new packages much more readily. One package (org.genxdm.xpath.v10) is there for mere clients of XPath. The other package (org.genxdm.xpath.v10.extend) is there if you need to add a function to the XPath engine. What makes this change even cooler – I had to implement an extension function (“here”) for the port of the XML Security library. With the newly restructured API, I was able to implement the function purely with the new API, rather than resorting to extending the implementation we have.

One final note. Now that we have this simplified API, it looks quite possible to create a javax.xml.xpath wrapper around our API. Such an API would give people who want to experiment with GenXDM a way to do so while still using what they’re familiar with. This wrapper won’t replace the API we have for at least two reasons. Our API:

• uses generics and types, producing more type-safety than does the javax.xml.xpath API
• defines how to add extension functions

Well, in any case, an implementation using javax.xml.xpath APIs is possible, but we don’t have any demand for it yet….

The new release has some other cool stuff in it, but that’s for another blog post.

We think this port shows important aspects of GenXDM. It shows you can take a sophisticated code base, that cares about many details of XML, and convert it to work using GenXDM. It shows that you can build such ports in a way that provides backwards compatibility with existing DOM APIs (or other tree APIs, for that matter). It validates the mutation part of the GenXDM API, something we introduced to the code-base only after we decided to release the technology as open-source. And it shows that you can do side-by-side comparisons of XML tree technologies, and choose one at runtime that best fits your use case.

We recognize that there’s still more work to do on this port. However, we pass all the existing test cases, so most of what’s to be done really falls into the category of enhancements.

If you’re at all interested, please check out the project, and let us know what you think. If you give it a try, let us know what you find out!

We’ve made a critical improvement (well, in the source, we’ve not done a release with it yet).  Our original drop included a mutation API.  We added this API to facilitate the conversion of existing XML tools that actually need to mutate existing XML trees.  When we added the mutation APIs, we mirrored the functionality on what we were trying to replace: DOM.  Among the unfortunately aspects, this gave us bug 16, that of baking a notion of “owner” into the mutable API.

A key trouble is that DOM conflates the “root” of a document with that of a “factory” for new elements.  For all other XML tree APIs we’ve looked at, the APIs separate these two concepts.  GenXDM already separated these two notions by introducing a “NodeFactory”, but that still required the use of an “owner” parameter.  What to do?

XQuery Update Facility

Along our journey, we noticed XQuery Update Facility (XQUF), and consequently filed bug 19.  We brought together the two bugs – the annoying DOM notion of “ownership” and our failure to follow the XQUF APIs.  Fortunately, the XQUF APIs don’t have a notion of ownership, so perhaps we could kill two birds with one stone.  We converted our mutable API to follow the pattern of XQUF.  To prove it out, we also converted the one project that extensively uses the mutable APIs – XML Security.  This ended up being a fair chunk of work, in part because there was no way to incrementally port XML Security to the new APIs, the whole library had to be converted all at once.  (I’m using the combination of all of these changes plus holidays and such as part of my lame excuse for not posting more frequently.)

Very much like with the core GenXDM APIs, we’re not able to exactly follow the XQUF specification.  Notably, XQUF has a notion of pending changes – all the APIs we have make immediate changes.  This means that some of the methods, like “mergeUpdates”, and “applyUpdates” don’t make sense, and “propogateNamespaces” sort of makes sense, but we added “insertNamespace” instead.  For the APIs that do make sense, we’ve done our best to follow XQUF.

For now, we’re thinking that a notion of pending changes will have to wait until we see a need for it, and then we can make it a part of interfaces that extend the ones we just built.

Overall, I’m happy with the changes, because it actually simplified some of the XML Security code changes.  In places where we needed to pass along an instance of “NodeFactory”, an “owner”, and the node to be updated, we now only need to pass along the “NodeFactory” and the node to be updated.  Of course, this implies that NodeFactory actually has state, and can only be used with a single DOM Document (if that’s your underlying model), but that’s pretty easy to deal with.

Conclusion

Our mutable API now has the benefit of actually being more closely tied to the XQuery standard that we’re trying to follow.  Also simplified some of the code that actually mutates XML trees.  Overall, a significant improvement.

• Why doesn’t it have a better name?
• Why isn’t there a better web site, source code repository, and all the other infrastructure?

Yay, us. We’ve changed the name, in response to comments. After quite a bit of discussion (and running off in all directions), we ended up settling on GenXDM, which amuses those of us who are gen-X-ers (which is all of the current committers). What do you think? Better?

We have also had to concede that we don’t yet have the community around our code that would convince folks at Apache to sponsor the project. So we’ve created a repository, and can generate a website, and have a good mailing list, and … well, there’s this blog, as well.

That’s what we’ve done. There are also two questions I’d like to address now (without making this post too chunky).

• Has the opportunity for an in-memory API other than the DOM already passed?
• Why not use an existing in-memory tree model API, then?

Now, a discussion of the DOM is liable to become rather heated. Note that I need to restrict the discussion to Java alone. In my opinion, the opportunity for DOM to achieve credibility passed about five years ago—which doesn’t prevent it from being the dominant model. The problem is that there is almost zero possibility of displacing an API with so broad and deep a network effect all at once. The DOM is too many places, in too much code. If you have to hold your nose and use it, then why make things harder by using some other tree model as well?

That’s the problem that all of the alternative XML tree model APIs in Java have faced, for the last ten years. There are already better tree model APIs out there. But if the developers of an XML processor want to reach an audience, to build a community, to see their code widely adopted, then they have to acknowledge that the DOM is dominant (except in certain niches). In the development of GenXDM, we also faced and acknowledged the problem. The GenXDM solution is deliberately built in a fashion that allows the DOM to continue in use. We don’t like it, but we build on top of it—and also on top of alternative tree models. It’s basically the same pattern that you’ll see in a number of applications/processors, where the XML tree model layer has been abstracted, so that the code can run over multiple different models. As a rule, this has been done as a tangent, a side effort, and as a consequence tends to be the least common functionality across models. GenXDM differs significantly: we’re providing an XDM API, which defines the utility that must be made available. It may mean enhancing the tree models inside the bridge code, in one way or another. Supporting multiple tree models is one of our central goals, not a tangent. We’re doing it well.

In other words, we are using existing tree model APIs for XML—not “an” existing model, but “any” existing model (for which a bridge has been written). In doing so, we expect not only to not fight DOM’s network effect, but to make use of it.

Suppose you have an idea for an application or processor for XML. If you write it for the DOM, you get to use the DOM. Lucky you. You also get to integrate with other DOM-targeted processors. Write it for GenXDM, and you get to use a much less odious API, and you get to integrate with other DOM-targeted processors. Just inject a DOM bridge. When operating in isolation, inject whatever bridge you prefer. Down the line, when other processors also target GenXDM, inject whatever bridge you prefer.

It will take us some time to convince folks that this approach will work, but it has the advantage of not forcing massive disruption of existing code.

Enough proselytizing for now, I think. Take a look at the code, won’t you? Maybe you’ll find it useful.

Here’s the rest of the list of questions; I will get to them, in time.

• Why should I implement now?
• What’s the advantage to using gXML?
• What is this “bridge” or “handle/body” pattern and why should I care?
• What can it do right now, today?
• Can it really grow to do all the things the developers say it could do?
• Give me some numbers!

We announced it on our new mailing list, and we’re posting the site generated by Maven.

In other words, lots of fun to be had.  Please jump in and “kick the tires,” and let us know what you think!

In the previous discussion about migration to gXML, I explored the conversion of one simple stateless method. That post explored some of the general patterns of migrating code to use gXML, by way of deprecating old methods and adding new ones.

With this discussion, I want to take up a much more tangled scenario – a class that holds state, and how to migrate that.

Parallel Tracks

I had the most trouble with the ElementProxy class.  This class has a protected field in it of type Element.  A search of the existing source quickly revealed quite a number of subclasses that directly refer to this field. To change the class to work with gXML, though I needed to make sure that the class ElementProxy ended up being parameterized by <N>, and that a field corresponding to this existing field had to be of type N, rather than of type Element.  Remember, though, that my goal is to migrate the library incrementally – existing classes have to operate unmodified, at least until I have a chance to convert them.  So I took an approach of having parallel tracks.  Where I started with this:

protected Element _constructionElement;

… I ended up with this:

/** @deprecated **/
protected Element _constructionElement;

private N _wrappedElement;

private N _lastSyncedElement;

public void setElementNode(N elem) { ... }

public N getElementNode() { ... }

Before explaining this further, perhaps I have to explain the two scenarios more fully. In the easy case, a given subclass has been fully modified to use the new gXML-based approach, and that class uses new getter/setter methods I defined. This easy scenario works, because all access goes through the the get and set methods.  In the harder case, existing clients directly set and get the old protected member that I want to replace.  Unless I wanted to convert the entire class hierarchy stemming from this one class all at the same time, I needed a way to support a partially ported class hierarchy stemming from this one class.

In my first attempt at this, I didn’t have the “_lastSyncedElement” member. This didn’t work, though.  Instead, I found code directly setting the protected member, and yet I had already rewritten the parent class to use the new private member via the getter method.  Since Java doesn’t give me any way to detect when a protected member is altered, the value returned by the get method didn’t match the value directly set into the protected method, and tests failed.  The solution, simple enough, was to add a third copy of the field, as show above, the “lastSyncedElement”.  The third copy kept track of the last value set using the new API.  Now, the “get” method, whenever it went to get the new private value, it would first check to see if the “last sync value” matched the value of the old protected field.  If it didn’t, it would copy the value from the protected field into the new private member.

This change meant that all the as-yet-unmodified DOM-based code could run unmodified by directly accessing the “_constructionElement” field.  All the new code would call the “set” method to set the value, and the object being set was of type Element, I would also set the original protected value, and its shadow – “_lastSyncedElement”.  The new set  and get methods look like this:

protected void setElementNode(N elem) {
    _constructionElement = (elem instanceof Element) ? (Element) elem : null;
    _lastSyncedElement = _constructionElement;
    _wrappedElement = elem;
}

public N getElementNode() {
    if (_constructionElement != _lastSyncedElement) {
        _lastSyncedElement = _constructionElement;
        _wrappedElement = (N) _constructionElement;
    }

    return _wrappedElement;
}

In this way, any direct changes to the old protected member would be detected and used by new code.

Tricks For Constructors

ElementProxy gave me another problem, too.  It had existing constructors that took Element object as parameters.  In the gXML world, that had to be turned into an element of type N, and an XmlContext<N> ctx, so that it would be possible to do something with the type N parameter.  So similar to the approach used with the purely static methods I discussed in my previous writeup, I deprecated the existing constructor, and had it call a new constructor with a new DOM-specific context object.

That solved one problem, but another cropped up.  Some subclasses of ElementProxy had a pair of constructors, each with a single parameter, but one taking Document, and one taking Element.  In gXML, both Document and Element collapse to type N.  This meant that a new constructor method taking a context and an N could no longer distinguish these two cases.  As a somewhat silly solution, I changed it so that the “Document” form of the constructors took an extra “do nothing” boolean parameter (which I named “unusedDiscriminator”).

Perhaps A Different Approach?

I might have followed an alternate course.  I could have wrapped the _constructionElement member with methods to access it.  That is, I might have swept through the code base once to add calls to get/set methods that still used the Element type.  This would have let me directly intercept all attempts to set the value.  However, as I went through and converted code, I would have had to replace these method calls with ones that returned and set values of type <N>.  I think this means that I would have needed to touch each source file twice – once to convert it to use the method wrapping access to an Element, and then to convert those calls to something wrapping <N>.  In the end, I think the approach I took probably worked just as well, if not slightly better.

End Result

The end result of this careful backwards compatibility left me with unnecessary code.  Once I converted all of the code – all of the subclasses of this class, and all of the direct uses of the protected field, I was left with no direct access to the protected field, and I have found I can now remove the parallel tracks.  But the intermediate solution did let me worry about migrating code a file at a time, rather than a class hierarchy at a time.  I think that made the extra complexity worth it.