The XML C parser and toolkit of Gnome

Table of Content:

1. General overview
2. Setting libxml2 set of memory routines
3. Cleaning up after parsing
4. Debugging routines
5. General memory requirements

General overview

The module xmlmemory.hprovidesthe interfaces to the libxml2 memory system:

• libxml2 does not use the libc memory allocator directly butxmlFree(),xmlMalloc() and xmlRealloc()
• those routines can be reallocated to a specific set of routine,bydefault the libc ones i.e. free(), malloc() and realloc()
• the xmlmemory.c module includes a set of debugging routine

Setting libxml2 set of memory routines

It is sometimes useful to not use the default memory allocator, eitherfordebugging, analysis or to implement a specific behaviour on memorymanagement(like on embedded systems). Two function calls are available to doso:

• xmlMemGet()whichreturn the current set of functions in use by the parser
• xmlMemSetup()whichallow to set up a new set of memory allocation functions

Of course a call to xmlMemSetup() should probably be done beforecallingany other libxml2 routines (unless you are sure your allocationsroutines arecompatibles).

Cleaning up after parsing

Libxml2 is not stateless, there is a few set of memory structuresneedingallocation before the parser is fully functional (some encodingstructuresfor example). This also mean that once parsing is finished there isa tinyamount of memory (a few hundred bytes) which can be recollected if youdon'treuse the parser immediately:

• xmlCleanupParser()isa centralized routine to free the parsing states. Note that itwon'tdeallocate any produced tree if any (use the xmlFreeDoc() andrelatedroutines for this).
• xmlInitParser()isthe dual routine allowing to preallocate the parsing statewhich can beuseful for example to avoid initialization reentrancyproblems when usinglibxml2 in multithreaded applications

Generally xmlCleanupParser() is safe, if needed the state will berebuildat the next invocation of parser routines, but be careful of theconsequencesin multithreaded applications.

Debugging routines

When configured using --with-mem-debug flag (off by default), libxml2usesa set of memory allocation debugging routines keeping track of allallocatedblocks and the location in the code where the routine was called. Acouple ofother debugging routines allow to dump the memory allocated infos toa fileor call a specific routine when a given block number is allocated:

• xmlMallocLoc()xmlReallocLoc()andxmlMemStrdupLoc()arethe memory debugging replacement allocation routines
• xmlMemoryDump()dumpsall the informations about the allocated memory block leftsin the.memdumpfile

When developing libxml2 memory debug is enabled, the tests programscallxmlMemoryDump () and the "make test" regression tests will check foranymemory leak during the full regression test sequence, this helps alotensuring that libxml2 does not leak memory and bullet proofmemoryallocations use (some libc implementations are known to be far toopermissiveresulting in major portability problems!).

If the .memdump reports a leak, it displays the allocation functionandalso tries to give some informations about the content and structure oftheallocated blocks left. This is sufficient in most cases to find theculprit,but not always. Assuming the allocation problem is reproducible, itispossible to find more easily:

1. write down the block number xxxx not allocated
2. export the environment variable XML_MEM_BREAKPOINT=xxxx , theeasiestwhen using GDB is to simply give the command

   set environment XML_MEM_BREAKPOINT xxxx

   before running the program.

3. run the program under a debugger and set a breakpointonxmlMallocBreakpoint() a specific function called when this preciseblockis allocated
4. when the breakpoint is reached you can then do a fine analysis oftheallocation an step to see the condition resulting in themissingdeallocation.

I used to use a commercial tool to debug libxml2 memory problems butafternoticing that it was not detecting memory leaks that simple mechanismwasused and proved extremely efficient until now. Lately I have also used valgrindwith quite somesuccess,it is tied to the i386 architecture since it works by emulating theprocessorand instruction set, it is slow but extremely efficient, i.e. itspot memoryusage errors in a very precise way.

General memory requirements

How much libxml2 memory require ? It's hard to tell in average itdependsof a number of things:

• the parser itself should work in a fixed amount of memory, exceptforinformation maintained about the stacks of names and entitieslocations.The I/O and encoding handlers will probably account for a fewKBytes.This is true for both the XML and HTML parser (though the HTMLparserneed more state).
• If you are generating the DOM tree then memory requirements willgrownearly linear with the size of the data. In general for abalancedtextual document the internal memory requirement is about 4 timesthesize of the UTF8 serialization of this document (example theXML-1.0recommendation is a bit more of 150KBytes and takes 650KBytes ofmainmemory when parsed). Validation will add a amount of memory requiredformaintaining the external Dtd state which should be linear withthecomplexity of the content model defined by the Dtd
• If you need to work with fixed memory requirements or don't needthefull DOM tree then using the xmlReaderinterfaceis probably the best way toproceed, it still allows tovalidate or operate on subset of the tree ifneeded.
• If you don't care about the advanced features of libxml2likevalidation, DOM, XPath or XPointer, don't use entities, need to workwithfixed memory requirements, and try to get the fastest parsingpossiblethen the SAX interface should be used, but it has knownrestrictions.

Daniel Veillard