From 2ee13d9e464a1f5daccaff58f5d09d36b7c4f667 Mon Sep 17 00:00:00 2001 From: Aron Xu Date: Mon, 21 Sep 2015 22:58:06 +0800 Subject: Revert "Imported Upstream version 2.9.1+dfsg1" This reverts commit 7300193becde71a344c8ac0973dc290fa24d800d. --- doc/encoding.html | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) (limited to 'doc/encoding.html') diff --git a/doc/encoding.html b/doc/encoding.html index 93de5bf..7c7953f 100644 --- a/doc/encoding.html +++ b/doc/encoding.html @@ -13,7 +13,8 @@ by Tim Bray on Unicode and why you should care about it.

If you don't unde without knowing what encoding it uses, then as Joel Spolsky said please do not write another line of code until you finish reading that article.. It is a prerequisite to understand this page, and avoid a lot of problems with -libxml2, XML or text processing in general.

Table of Content:

  1. What does internationalization support +libxml2, XML or text processing in general.

    Table of Content:

      +
    1. What does internationalization support mean ?
    2. The internal encoding, how and why
      3. @@ -33,7 +34,8 @@ allows the document to be encoded in other encodings at the condition that they are clearly labeled as such. For example the following is a wellformed XML document encoded in ISO-8859-1 and using accentuated letters that we French like for both markup and content:

      <?xml version="1.0" encoding="ISO-8859-1"?>
-<très>là </très>

      Having internationalization support in libxml2 means the following:

      • the document is properly parsed
        • +<très>là </très>

        Having internationalization support in libxml2 means the following:

          +
        • the document is properly parsed
        • information about it's encoding is saved
        • it can be modified
        • it can be saved in its original encoding
          • @@ -54,7 +56,8 @@ an internationalized fashion by libxml2 too:

          <!DOCTYPE HTML PUBLIC "-
 <p>W3C crée des standards pour le Web.</body>
 </html>

          The internal encoding, how and why

          One of the core decisions was to force all documents to be converted to a default internal encoding, and that encoding to be UTF-8, here are the -rationales for those choices:

          • keeping the native encoding in the internal form would force the libxml +rationales for those choices:

              +
            • keeping the native encoding in the internal form would force the libxml users (or the code associated) to be fully aware of the encoding of the original document, for examples when adding a text node to a document, the content would have to be provided in the document encoding, i.e. the @@ -67,7 +70,8 @@ rationales for those choices:

              • keeping the native encoding in the inte considered an intelligent choice too since it's a direct Unicode mapping support. I selected UTF-8 on the basis of efficiency and compatibility with surrounding software: -
                • UTF-8 while a bit more complex to convert from/to (i.e. slightly +
                    +
                  • UTF-8 while a bit more complex to convert from/to (i.e. slightly more costly to import and export CPU wise) is also far more compact than UTF-16 (and UCS-4) for a majority of the documents I see it used for right now (RPM RDF catalogs, advogato data, various configuration @@ -86,8 +90,10 @@ rationales for those choices:

                    • keeping the native encoding in the inte upcoming Gnome text widget, and a lot of Unix code (yet another place where Unix programmer base takes a different approach from Microsoft - they are using UTF-16)
                      • -
                    • -

                  What does this mean in practice for the libxml2 user:

                  • xmlChar, the libxml2 data type is a byte, those bytes must be assembled +
                  +
                  • +

                What does this mean in practice for the libxml2 user:

                  +
                • xmlChar, the libxml2 data type is a byte, those bytes must be assembled as UTF-8 valid strings. The proper way to terminate an xmlChar * string is simply to append 0 byte, as usual.
                • One just need to make sure that when using chars outside the ASCII set, @@ -95,7 +101,8 @@ rationales for those choices:

                  • keeping the native encoding in the inte

                  How is it implemented ?

                  Let's describe how all this works within libxml, basically the I18N (internationalization) support get triggered only during I/O operation, i.e. when reading a document or saving one. Let's look first at the reading -sequence:

                  1. when a document is processed, we usually don't know the encoding, a +sequence:

                      +
                    1. when a document is processed, we usually don't know the encoding, a simple heuristic allows to detect UTF-16 and UCS-4 from encodings where the ASCII range (0-0x7F) maps with ASCII
                    2. the xml declaration if available is parsed, including the encoding @@ -136,7 +143,8 @@ err2.xml:1: error: Unsupported encoding UnsupportedEnc collected/built an xmlDoc DOM like structure) ? It depends on the function called, xmlSaveFile() will just try to save in the original encoding, while xmlSaveFileTo() and xmlSaveFileEnc() can optionally save to a given -encoding:

                      1. if no encoding is given, libxml2 will look for an encoding value +encoding:

                          +
                        1. if no encoding is given, libxml2 will look for an encoding value associated to the document and if it exists will try to save to that encoding,

                          otherwise everything is written in the internal form, i.e. UTF-8

                          @@ -175,7 +183,8 @@ so a couple of functions htmlGetMetaEncoding() and htmlSetMetaEncoding() have been provided. The parser also attempts to switch encoding on the fly when detecting such a tag on input. Except for that the processing is the same (and again reuses the same code).

                          Default supported encodings

                          libxml2 has a set of default converters for the following encodings -(located in encoding.c):

                          1. UTF-8 is supported by default (null handlers)
                            2. +(located in encoding.c):

                              +
                            1. UTF-8 is supported by default (null handlers)
                            2. UTF-16, both little and big endian
                            3. ISO-Latin-1 (ISO-8859-1) covering most western languages
                            4. ASCII, useful mostly for saving
                              5. @@ -193,7 +202,8 @@ goal is to be able to parse document whose encoding is supported but where the name differs (for example from the default set of names accepted by iconv). The following functions allow to register and handle new aliases for existing encodings. Once registered libxml2 will automatically lookup the -aliases when handling a document:

                              • int xmlAddEncodingAlias(const char *name, const char *alias);
                                • +aliases when handling a document:

                                  +
                                • int xmlAddEncodingAlias(const char *name, const char *alias);
                                • int xmlDelEncodingAlias(const char *alias);
                                • const char * xmlGetEncodingAlias(const char *alias);
                                • void xmlCleanupEncodingAliases(void);
                                  • -- cgit v1.2.3