#LyX 2.0 created this file. For more info see http://www.lyx.org/ \lyxformat 413 \begin_document \begin_header \textclass report \begin_preamble \usepackage{hyperref} \end_preamble \use_default_options false \maintain_unincluded_children false \language english \language_package default \inputencoding auto \fontencoding global \font_roman default \font_sans default \font_typewriter default \font_default_family default \use_non_tex_fonts false \font_sc false \font_osf false \font_sf_scale 100 \font_tt_scale 100 \graphics default \default_output_format default \output_sync 0 \bibtex_command default \index_command default \paperfontsize default \spacing single \use_hyperref false \papersize a4paper \use_geometry false \use_amsmath 1 \use_esint 1 \use_mhchem 1 \use_mathdots 1 \cite_engine basic \use_bibtopic false \use_indices false \paperorientation portrait \suppress_date false \use_refstyle 0 \index Index \shortcut idx \color #008000 \end_index \secnumdepth 3 \tocdepth 3 \paragraph_separation skip \defskip medskip \quotes_language english \papercolumns 1 \papersides 1 \paperpagestyle default \tracking_changes false \output_changes false \html_math_output 0 \html_css_as_file 0 \html_be_strict false \end_header \begin_body \begin_layout Title The new string unicode type \end_layout \begin_layout Author Marco van de Voort \end_layout \begin_layout Standard Version: 0.07 \begin_inset Graphics filename unicode_small.jpg \end_inset \end_layout \begin_layout Standard \begin_inset CommandInset toc LatexCommand tableofcontents \end_inset \end_layout \begin_layout Section Introduction \end_layout \begin_layout Standard Lately there has been some discussion about a new unicode type, mostly due to a request from the Lazarus team to support unicode in file operations (for filenames, not handling of unicode files). A few proposals were made on the fpc-pascal maillist, abd some discussion followed, but it died out, and there a lot of details of the proposals were only discussed on subthreads. \end_layout \begin_layout Standard I decided to try to summarize all positions and requirements, at least as I saw them as a kind of a discussion document. During the discussions I also detailed the requirements I had in mind a bit more, so I decided to write them down too. \end_layout \begin_layout Standard Versioning: \end_layout \begin_layout Itemize First version mostly my own writeup. Was originally meant to highlight the flaws that I saw in Florian's original proposal. There might still be some of the negative sentiment left, please skip it. \end_layout \begin_layout Itemize Second version mostly Florian's feedback which I commented on \end_layout \begin_layout Itemize Third version vastly expanded the Tiburón paragraph when CG lifted the veil a bit late July/early August, and the hybrid model. \end_layout \begin_layout Itemize Fourth version added the \begin_inset Quotes eld \end_inset economics \begin_inset Quotes erd \end_inset paragraph, expanded the hybrid model and mentions Yury's proposal and wiki page. \end_layout \begin_layout Itemize Fifth version(sept 2010) is mostly due to the new requirements now that FPC support compatible to Delphi/Unicode is becoming a possibility (cpnewstr). Some other details (OEMSTRING) were also added. \end_layout \begin_layout Itemize Sixth (0.6) (dec 2010) version adds some minor completion of unfinished sentences and other minor clarifications \end_layout \begin_layout Itemize Seventh (0.7) (nov 2011) adds some new insights of the last year. Specially the \begin_inset Quotes eld \end_inset \series bold default string \series default \begin_inset Quotes erd \end_inset ( \begin_inset CommandInset ref LatexCommand ref reference "sub:The-base-string" \end_inset ) discussion raged on the maillists, since it is the main issue left to discuss, now that the base support has been merged back to trunk. \end_layout \begin_layout Subsection Tiburón \end_layout \begin_layout Standard Tiburón is the codename for what is supposed to be the next version of Delphi ( version 2008?), and is supposed to have unicode. While we currently do not follow Delphi compatibility slavishly, it should only be broken if there are good reasons. A main reason for this is to not make life too hard on Delphi open source projects that also want to support FPC/Lazarus. Slowly details about Tiburón are starting to appear in CG oriented blogs. (e.g. Andreas Bauer's) \end_layout \begin_layout Itemize A new utf-16 ref counted unicode stringtype is added. \end_layout \begin_deeper \begin_layout Itemize s[x] doesn't take care of surrogates. \end_layout \begin_layout Itemize It is not yet clear if and how it supports endianness. \end_layout \end_deeper \begin_layout Itemize Ansistring becomes a basetype for all 1 byte based encodings (ansi, codepages,UT F-8), based on the fact that for internal windows functions, UTF-8 is treated as a codepage. \end_layout \begin_deeper \begin_layout Itemize To define a stringtype for a certain (Windows) codepage enumeration value, type mycodepagestring = type ansistring (1251); \end_layout \begin_layout Itemize Conversions that write a non UTF-8 codepage can be lossy. \end_layout \begin_layout Itemize UTF-8 is codepage 65001 (ident CP_UTF8) \end_layout \begin_layout Itemize probably value codepage $0 is used for the old ansistring. The conversions to and from this type (which codepage?) are not clear. \end_layout \begin_layout Itemize It seems that the typing of ansistring has become stronger, and honor TYPE (as in something = TYPE ansistring) is now really an incompatible type. \end_layout \begin_layout Itemize Conversions are done over UTF-16, but this might be a Windows implementation detail. (IOW on Unix use UTF-8) \end_layout \begin_layout Itemize Windows has a separate codepage (OEM) for console. So in fact there are three encodings (OEM, Ansi and UTF16) in a Windows enviroment. There is a separate tag (CP_OEMCP) for the default OEM page. I suggest we predefine OEMSTRING = ansistring(CP_OEMCP); \end_layout \begin_layout Standard Standard these two types share the same record (tansirec). There is a kind of open array type called RawByteString that accepts both types of strings. This is mainly used to simplify RTL helpers, it is too limited for application development. \end_layout \end_deeper \begin_layout Standard This quick summary has four aspects I don't like for porting to FPC: \end_layout \begin_layout Enumerate The use of windows specific codepage enumeration values in language syntax. However maybe they are really serious about the constants use, and this is livable. In my opinion it is the VCLs job to encapsulate the winapi gore, and if it can't be avoided, at least encourage a clean use. Daniel notes that there are not much platform independent choices to begin with. \end_layout \begin_layout Enumerate The fact that conversions between codepages are automated and can fail. (see also the discussion about codepages in the critique of Florian's proposal) This means that if you use codepage strings, you must be very careful with your codepaths, so that you can be pretty sure that there aren't alternate paths that mutilate data. This probably means that Codegear (not surprisingly) assumes that the bulk of all code is unicode or will be upgraded to it. \end_layout \begin_layout Enumerate UTF-8 and UTF-16 are scattered over two different types. This solution is non-orthogonal. \end_layout \begin_layout Enumerate The big one, the compatibility break between Delphi2007- and 2009+. FPC can avoid this on a compiler level the same way they fixed the string=short string to string=ansistring move, but the library level is more difficult. D2007- expects e.g. Windows API functions to call the -A versions, while 2009+ calls the -W versions \end_layout \begin_layout Standard They probably had the same as we problem for multiple-encodings types (see \begin_inset CommandInset ref LatexCommand ref reference "par:granularity" \end_inset ) , but chose to keep this compiletime by dividing the types according to 1 or 2 byte granularity. Maybe this also has some advantages in the compiler (being able to treat tunicodestring and twidestring the same here and there). And they don't support UTF-32, probably because windows doesn't (or it isn't used) \end_layout \begin_layout Standard One of the problems of a fully polymorphic (both 1 and 2-byte) string type is that it is incompatible with either Delphi/ansi or Delphi/unicode code, since such code will assume either 1 or 2-byte granularity. \end_layout \begin_layout Standard Another question mark is the fact that a lot of new ansistring variants are introduced that are apparantly type safe. The question begs what stringtype is in e.g. variant (my guess: all non ansi ansistrings are converted to either widestring or a new tunicodestring field) \end_layout \begin_layout Subsection The encodings \end_layout \begin_layout Standard The three main encodings are UTF-8, UTF-16 and UTF-32. An important property of these is that they are basically different ways to describe the same, so they can be convererted to eachother pretty easily and safely. Note that the multi byte encodings (16 and 32 (?)) also have big endian and little endian variants. \end_layout \begin_layout Standard However for now I'm going to \series bold forget the big endian and little endianess \series default . This kind of cross-platform compability is fairly rarely a problem. Only with wire-protocols and files that are shared between different architectu res need to insert conversions, and this can be better done manually. The same goes for arbitrary other sources that might have a different encoding. The difference is also only important on the perimeter of the system (when you load external data), since the system will mostly be in the same endianess \end_layout \begin_layout Standard Besides these three main encodings, conversions of the string type to and from the older codepages could be useful too, because the world won't become unicode instantly, and ansistrings are here to stay for a while. Most notably Florian's proposal has some (potential) support for other codepages too, though not many details. \end_layout \begin_layout Subsection Economics of the encodings \end_layout \begin_layout Standard In one of the unicode discussions Daniel posted this link: http://unicode.org/no tes/tn12/ \begin_inset Flex URL status collapsed \begin_layout Plain Layout http://unicode.org/notes/tn12/ \end_layout \end_inset . I just had some discussion about this in a different maillist on the subject of which is the ideal encoding, and here is my opinion some comments about encoding enconomics. Note that not all points are meant as arguments in favour of UTF-8 per se, just observations. \end_layout \begin_layout Itemize First and for all, the question is mostly irrelevant since the choice of primary encoding (and endianness if>8) for a platform/target has been made already by the OS and the general ABI. Deviating from this to simply possible multiplatform programmers at the expensive of people programming for the platform natively is IMHO not an option. \end_layout \begin_layout Itemize An often misinformed statement is that everything but ansi is worse in utf-8. This is not true, everything up from ascii to codepoint $0800 is equal in size between UTF-8 and UTF-16. This plane contains Cyrillic as well as several popular languages from the Semetic group like Hebrew and Arabic. \end_layout \begin_layout Itemize The simplicity of UTF-16 is quoted in a lot of place, the above link inclusive. While some may see it acceptable to cut corners in applications, it is IMHO not acceptable to break full unicode compliance in a serious library, and most of all, a RTL. This means that most speeddependant routines in an app must be able to handle UTF-16 surrogates and maybe also endianness. I personally think that serious applications shouldn't cut corners either. Note though that surrogates don't hinder all string routines. \end_layout \begin_layout Itemize Routines that don't need to process UTF-8 surrogates and encounter mostly Latin scripts are faster in UTF-8. (less bytes to move) \end_layout \begin_layout Standard Btw I use http://www.unicode.org/roadmaps/bmp/ \begin_inset Flex URL status collapsed \begin_layout Plain Layout http://www.unicode.org/roadmaps/bmp/ \end_layout \end_inset to quickly see what language groups are where in the BMP. \end_layout \begin_layout Subsection Granularity of [] \begin_inset CommandInset label LatexCommand label name "par:granularity" \end_inset \begin_inset Index idx status collapsed \begin_layout Plain Layout granularity \end_layout \end_inset \end_layout \begin_layout Standard One of the benefits of the discussion was that it called some attention to the s[] operator. First because it was a possible weakness of Florian's proposal (that got remedied later), but the more important one from a design perspective is what c:=s[5]; is supposed to mean with (s in [UTF8,UTF16,UTF32]). \end_layout \begin_layout Standard Let's take utf16 for a moment, and assume we have 10 codepoints, and every second is a surrogate. Then there are three possible meanings: \end_layout \begin_layout Subsubsection Meaning 1: index means codepoints \end_layout \begin_layout Standard In this meaning, a string is (a view on) an array of codepoints. So \end_layout \begin_layout Standard c:=s[5]; means the 5th codepoint. A codepoint can be >2 bytes, so type of \begin_inset Quotes eld \end_inset c \begin_inset Quotes erd \end_inset must be able to contain a 32-bit value. The first 5 codepoints have two with surrogates so the address of the first char is @s[1]+5*2 + 2*2=@s[1]+14 (all in bytes) \end_layout \begin_layout Standard Writing a character (s[5]:=c) is an even worse problem, since a codepoint written might not have the same size as the codepoint current;y at the at s[5], needing costly (O(n)) insertion routines. \end_layout \begin_layout Subsubsection Meaning II: index means granularity of the encoding \end_layout \begin_layout Standard In this meaning the string is (a view on) an array with the ganularity of the encoding. So 1 in the case of UTF-8, 2 in the case of UTF-16 etc. \end_layout \begin_layout Standard c:=s[5]; in UTF-16 means s[1]+5*2 =@s[1]+10 (all in bytes) \end_layout \begin_layout Standard Writing a character (s[5]:=c) pretty much remains the same everything has the granularity of the encoding. \end_layout \begin_layout Subsubsection Meaning III: index means character \end_layout \begin_layout Standard This is nasty, even UTF32 has the granularity of a codepoint. However printable characters may be composed out of multiple codepoints. This basically means the end of \begin_inset Quotes eld \end_inset char \begin_inset Quotes erd \end_inset as a separate type. Everything is a variable length string, and basic string operations have to be code on a lower level. \end_layout \begin_layout Subsubsection Granularity conclusion \end_layout \begin_layout Standard (Note that the same problem also goes for Length(s). codepoints or elements in the granularity of the encoding?) \end_layout \begin_layout Standard The problem with the array of codepoints is that typical code like \end_layout \begin_layout LyX-Code for i:=1 to length(s) do \end_layout \begin_layout LyX-Code s[i]:=' '; \end_layout \begin_layout Standard is very expensive since \end_layout \begin_layout Itemize the address of s[x] depends on all codepoints before codepoints x. This can make the above loop quadratic in the number of codepoints jumps ( on average (n^2)/2). Most platforms also use a procedure to iterate over codepoints. \end_layout \begin_layout Itemize each codepoint assignment can possibly be an insertion or deletion of bytes, since the assigned codepoint can be smaller or larger than the codepoint already in place. \end_layout \begin_layout Standard IMHO this opens a can of worms where we don't want to go \begin_inset Foot status collapsed \begin_layout Plain Layout Since we don't have any optimizations that optimize loops in an advance way, I don't think it is acceptable to waive this point in the hope that future optimizations will solve this. \end_layout \end_inset . However it might be an argument to (also) support UTF-32, since that does allow fairly easy char manipulation, with minimal limitations: If it is a routine that is not really much used, the simplest way to convert would be to do something like \end_layout \begin_layout LyX-Code procedure dosomething (var s:utf16string); \end_layout \begin_layout LyX-Code var internals: utf32string; \end_layout \begin_layout LyX-Code begin \end_layout \begin_layout LyX-Code internals:=s; // force conversion to utf32. \end_layout \begin_layout LyX-Code <> \end_layout \begin_layout LyX-Code s:=internals; // convert back. \end_layout \begin_layout LyX-Code end; \end_layout \begin_layout Standard Of course this is not perfect (e.g. charsets won't work because even a charset for the defined codepoints would be in the magnitude of 125k), but it is easy, and avoids messing too much with working code. \end_layout \begin_layout Standard To state the obvious: to go there, we would have to forgo using the basic string types in standard routines, and code every reusable string routine on an assembler or pointer level. \end_layout \begin_layout Section Requirements \end_layout \begin_layout Standard The requirements are a bit of a problem because there are several factors that are not compatible to each other (e.g. speed and ease of use), and tradeoffs vary. Anyway the main requirements in a very broad definition are: \end_layout \begin_layout Itemize Ease of use \end_layout \begin_layout Itemize Reasonable to good performance should be possible without having to convert whole codebases to the pointer level. \end_layout \begin_layout Itemize Compatibility \end_layout \begin_deeper \begin_layout Itemize with Delphi/ansi \end_layout \begin_layout Itemize with Tiburón (Delphi/Unicode) \end_layout \begin_layout Itemize with existing FPC code. \end_layout \end_deeper \begin_layout Itemize Multi platform aspects. \end_layout \begin_layout Itemize Respect certain FPC traditions, most notably \end_layout \begin_deeper \begin_layout Itemize the need to combine code from different origins/styles into one program. (e.g shortstring TP and ansistring Delphi code) code are currently combinable in one program, and a single directive controls the meaning of the \begin_inset Quotes eld \end_inset string \begin_inset Quotes erd \end_inset type to make it compatible on a per unit basis with both) \end_layout \begin_layout Itemize the fact that the entire RTL is mostly implemented in (FPC's) Pascal. \end_layout \begin_layout Itemize FPC being portable means nativeness on every platform. Not carrying conventions from other operating systems to operating systems where they are alien (e.g. POSIX on Windows) \end_layout \end_deeper \begin_layout Standard Note: Most of the unices, but not all, use UTF-8, Windows use UTF-16. \end_layout \begin_layout Standard Since there have been debates about what compatibility means, my definition of compatibility is: \end_layout \begin_layout Itemize Compatibility mostly means that I must set some directives and update the uses clause on a per module (unit) basis, based on the sourcecodes origin (e.g. FPC mode X, or Delphi/ansi, or Delphi/unicode) \end_layout \begin_layout Itemize In very extreme circumstances global replace is acceptable too, but only if it is totally automatable. \end_layout \begin_layout Standard These reasons are founded on practical experience with 3rd party Delphi code, as well as with my own codebases. It is about as far as they will go. First and for all, \series bold total code audits should be avoided at near all costs. \end_layout \begin_layout Standard Here I define a global code audit as a search in all code for an occurance of code that could potentially break(behaviour that will be broken by the new implementation). Porters might not have written or understand the code they are porting, and codebases can be big. \end_layout \begin_layout Standard Keep in mind that \begin_inset Quotes eld \end_inset almost compatible \begin_inset Quotes erd \end_inset in practice means \begin_inset Quotes eld \end_inset not compatible \begin_inset Quotes erd \end_inset . I make an exception for the uses clause, because Delphi has there internal broken compatibility in the past. \end_layout \begin_layout Subsection Problems with the requirements \end_layout \begin_layout Standard The above requirements are conflicting. The two big problems are: \end_layout \begin_layout Itemize Delphi 2009+ breaks compatibility on this level with versions before it. Supporting Delphi before and after 2009 is therefore not trivial. \end_layout \begin_layout Itemize Delphi 2009+ chose a certain encoding to be the center of the universe, while it is mostly only a Windows convention. \end_layout \begin_layout Subsection Required \begin_inset Quotes eld \end_inset new \begin_inset Quotes erd \end_inset primitives in the RTL \end_layout \begin_layout Enumerate Regardless which choice is made for the default (see \begin_inset CommandInset ref LatexCommand vref reference "par:granularity" \end_inset ), Length(s) should be available in both meanings: length in codepoints and in granularity length. Probably length in codepoints (Delphi compat) \end_layout \begin_layout Enumerate charat(n) - returns codepoint [n]... assuming we chose the encoding granularity. \end_layout \begin_layout Enumerate charnext (strnext out of delphi compat?) \end_layout \begin_layout Standard How much of these will/should be (partially) inlinable? Is it worth it? It seems that most libc's use functions, not macro's, which might be an indicator that procedural overhead is less than the actual operation. \end_layout \begin_layout Subsection The Windows \begin_inset Quotes eld \end_inset W \begin_inset Quotes erd \end_inset problem \end_layout \begin_layout Standard Sideways related is the windows problem that on NT special functions must be called for unicode strings, all these functions end on -W instead of -A. Also all these symbols (and their record definitions) are typically organized in the windows header source as \end_layout \begin_layout LyX-Code {$ifdef unicode} \end_layout \begin_layout LyX-Code procedure xxx; (arguments);stdcall; external 'kernel32.dll' name 'xxxW'; \end_layout \begin_layout LyX-Code {$else} \end_layout \begin_layout LyX-Code procedure xxx; (arguments);stdcall; external 'kernel32.dll' name 'xxxA'; \end_layout \begin_layout LyX-Code {$endif} \end_layout \begin_layout Standard The actual problem is that these (W) calls don't exist (or work) on windows 9x. There are several solutions for this problem: \end_layout \begin_layout Enumerate A combination of runtime OS detection and loading. Problem is that the windows header sets are huge, and there is a great potential for error. \end_layout \begin_layout Enumerate Splitting the win32 target over unicode support.. So the current implementation is parameterized and move to a shared dir, and win9x target sets some types and defines, and imports these includefiles, as well as the NT-unicode target that defines UNICODE. \end_layout \begin_layout Standard Personally I like the splitting. Note that the \begin_inset Quotes eld \end_inset win9x \begin_inset Quotes erd \end_inset target will still work on win NT/2k/XP, and is in fact a \begin_inset Quotes eld \end_inset real \begin_inset Quotes erd \end_inset win32. Note that the target names were picked in a hurry, maybe \begin_inset Quotes eld \end_inset win32 \begin_inset Quotes erd \end_inset and \begin_inset Quotes eld \end_inset winnt \begin_inset Quotes erd \end_inset are better target names. \end_layout \begin_layout Standard Some people keep banging on about using UTF8 on Windows, but afaik only some console functions take utf8. The rest is all default codepage (some ansi codepage like windows-1250) for the -A APIs, and UTF16 for the -W. \end_layout \begin_layout Subsection The base string type problem \begin_inset CommandInset label LatexCommand label name "sub:The-base-string" \end_inset \end_layout \begin_layout Standard Now that the base language support of Delphi/Unicode has been merged into trunk, the logical question is how to roll it out throughout the rest of the codebase, what functionality of the past is deprecated, and what is kept. \end_layout \begin_layout Standard The first big problem is how to handle the \emph on string \emph default type. In FPC, string is considered to be a type alias to the core \emph on string \emph default type to be used. In the past it has been either \emph on shortstring \emph default or \emph on ansistring \emph default , but with Delphi/unicode, \emph on unicodestring \emph default has to be added. \end_layout \begin_layout Standard Currently, using the {$H+/-} parameter and $mode settings that preset {$H}, \emph on string \emph default can already be aliased to \emph on shortstring \emph default and \emph on ansistring \emph default for resp Turbo Pascal and Delphi like modes. At the time that ansistring was added, the following modifications were needed: \end_layout \begin_layout Enumerate The length needed to be set with setlength(s,newlength); instead of s[0]:=newlen gth \end_layout \begin_layout Enumerate Access to chars >length was disallowed, as ansistring only allocated as many bytes as needed. \end_layout \begin_layout Enumerate Of course various library procedure routines needed to be overloaded, to avoid excessive conversions. But this was pretty much only unit \begin_inset Quotes eld \end_inset system \begin_inset Quotes erd \end_inset \end_layout \begin_layout Standard However the situation back then is different from now for several reasons: \end_layout \begin_layout Enumerate With the short->ansistring conversion, old TP units for the better part remained shortstring. Delphi related and new libraries were set up using ansistring. \end_layout \begin_layout Enumerate The codebases outside of the RTL were relatively small. \end_layout \begin_layout Enumerate The amount of OOP code was relative small. Lazarus didn't exist, and when it emerged it was nearly directly Delphi dialect only. \end_layout \begin_layout Standard Contrary to then, this time not all old code is deprecated, and there is no new start with a near blank slate. Some operating systems are 1-byte (UTF8) oriented, some are 2-byte oriented, and this difference might last forever. The differences between the various 1-byte encodings can be massaged away by having some variable that defines the default (1-byte) encoding, and setting that depending on runtime library detection or compiler commandline parameters. \end_layout \begin_layout Standard New this time also is the large amount of OOP code which has virtual methods with \begin_inset Quotes eld \end_inset string \begin_inset Quotes erd \end_inset arguments. One can't override a virtual method defined with string=ansistring by a method that is string=unicodestring (and even if it could, one would have to be terribly careful to avoid extreme amounts of conversion. \end_layout \begin_layout Subsection Delphi/Unicode lacking variable encoding - Databases \end_layout \begin_layout Standard Embarcadero clearly decided all \end_layout \begin_layout Section The proposals \end_layout \begin_layout Standard In the maillist discussion there were 3 proposals that I'll summarize shortly below. \end_layout \begin_layout Subsection Felipe's proposal. \end_layout \begin_layout Standard Felipe's proposal was the first, and was mostly still oriented towards the direct File I/O problem. He proposed to use UTF-16 exclusively. Period. \end_layout \begin_layout Standard Advantages \end_layout \begin_layout Enumerate Simplicity \end_layout \begin_layout Enumerate Carries Delphi compatibility to the extreme, introducing Delphi/Unicode UTF16 assumption on all platforms. Even if UTF16 is not the native unicode encoding. \end_layout \begin_layout Standard Disadvantages \end_layout \begin_layout Enumerate No way to support UTF-8, this means that all dealing with UTF-8 (the main encoding on Unix) must be manual on p(ansi)char level or through careful use of ansistring workarounds, or face heavy repeated conversion penalties. This also means code must be written to pass a readonly unicode string to a library on unix, instead of simply passing pwidechar(s). It is a windows centric proposition \end_layout \begin_layout Enumerate No utf-32, so also no simple way \end_layout \begin_layout Standard Keep in mind that this also means some complications for e.g. standard file I/O, that must change from UTF-8 to UTF-16. \end_layout \begin_layout Subsection Marco's proposal \end_layout \begin_layout Standard This proposal was more in line with earlier discussions on core, simply have three separate types for the three encodings, that autoconvert reasonably, and the implementation is nearly the same. To keep RTL size down, most system calls would only accept strings in the system encoding, except for VAR parameters that need to be wrapped or double implemented. \end_layout \begin_layout Standard So for clarity: an utf8string, utf16string and a utf32string type. \end_layout \begin_layout Standard Advantages \end_layout \begin_layout Enumerate The string types that a routine use signal the encodings it accepts/returns. \end_layout \begin_layout Enumerate Maximum speed for code that uses only one encoding, no conversion, no runtime behaviour. \end_layout \begin_layout Enumerate The fact that the types have exactly the same content in a different representat ion (4 types, together with UTF-32 and the COM widestring) made me hope that the implementation would not be that much more complicated than one + a bunch of special options and directives. \end_layout \begin_layout Enumerate Interfacing with systems with a different encoding is simple. Convert to correct type if not already, and then typecast. \end_layout \begin_layout Enumerate Tiburón code could simply use UTF16 string everywhere (a simple {$H like directive), and be very to totally compatible, and yet mixable. \end_layout \begin_layout Standard Disadvantage \end_layout \begin_layout Enumerate Most new types, thus also the most conversions. \end_layout \begin_layout Enumerate Separate types, so one can't pass UTF-8 string to a procedure with a var or out parameter of UTF-16 type. \end_layout \begin_layout Enumerate Only overloading and conversion as instrument for routines that must accept multiple encodings. Not unlike ansistring and shortstring IOW with the same problems. \end_layout \begin_layout Enumerate More types also means a lot more vt constants in tvarrecs, variants etc. \end_layout \begin_layout Enumerate Prefix records of types can't be Tiburón compatible \end_layout \begin_layout Subsubsection Aliases \end_layout \begin_layout Standard To make this work properly, there will be some additional aliases: \end_layout \begin_layout Itemize An alias to a type that always is the same as the system encoding. If you use this you are always safe performance wise. \end_layout \begin_layout Itemize An alias to utf16string of whatever identifier Tiburón uses for \end_layout \begin_layout Standard This also means that encoding agnostic code should use the system encoding, since the average string will be probably in the system encoding \end_layout \begin_layout Subsection Florian's proposal. \end_layout \begin_layout Standard Florian proposed to have a single unicode type that can represent the three encodings (UTF-x), and maybe others too (the old ascii codepages as well as LE vs BE). The principle is the same as ansistring, additional needed info is prefixed at addresses before s[1]. Currently it is only the encoding type, but it could be expanded. \end_layout \begin_layout Standard There are a lot more implementation details to be resolved in this proposal. \end_layout \begin_layout Standard Florian says the following about the granularity of the type. \end_layout \begin_layout Quote to overcome the indexing problem efficiently when using an encoding field (this is not about surrogates), we could do the following: introduce a compiler switch {$unicodestringindex default,byte,word,dword}. In default mode the compiler gets a shifting value from the encoding field (this is 4 bytes anyways and could be split into 1 byte shifting, 2 bytes encoding, 1 bytes reserved). In the other modes the compiler uses the given size when indexing. For example, a Tiberion (or how is it called?) switch could set this to word. \end_layout \begin_layout Standard Later however he says (in response to the below granularity challenge) \end_layout \begin_layout Quote I described this already in detail in my first mail: just in one of the four bytes available for storing the encoding. \end_layout \begin_layout Standard Now I'm confused :) \end_layout \begin_layout Standard Anyway about the performance he says: \end_layout \begin_layout Quote The approach has the big advantage, that you really need all procedures only once if desired. For example e.g. linux would get only utf-8 routines by default, utf-16 is converted to utf-8 at the entry of the helper procedures if needed. Usually, no conversion would be necessary because you see seldomly utf-16 in linux applications so only the check if the input strings are really utf-8 is necessary, this is very cheap because the data is anyways already in a cache line. \end_layout \begin_layout Standard He also says \end_layout \begin_layout Quote Keep in mind in your response, that we want also handle other formats than utf-8 or utf-16 if needed :) \end_layout \begin_layout Standard Michael says: \end_layout \begin_layout Quote For the LCL/fpGUI/MSEGui programmers, nothing changes, > you can even throw away your own conversion routines. > You need only a single call just prior to passing a string > to the OS/GUI system: ForceEncoding(). No ifdefs needed, > all is transparant. \end_layout \begin_layout Standard The type is a bit too complex to have a series of simple advantages and disadvantages, so I just going to describe some of the problems, and ask for clarification. \end_layout \begin_layout Subsubsection The biggest problem: can't declare what type to expect. \end_layout \begin_layout Standard My initial reaction was \begin_inset Quotes eld \end_inset oh my, a runtime type in Pascal, what about performance? It will be pretty much likes variants, and they are known to be slow. We will become Perl/Python \begin_inset Quotes erd \end_inset \end_layout \begin_layout Standard However while I still have serious doubts about performance, that's not the bigger problem. Since with pretty much any solution you can always isolate the speed dependant part, force the encoding to be constant (preferably the system encoding), and be done with it. Moreover, there is much to say for having only one string type, even if it is polymorphic internally. \end_layout \begin_layout Standard The \emph on bigger \emph default problem however is that you don't declare the type of the encoding anymore in parameters, local variables and return type. This means manual insertion of Michael's Enforceencoding calls everywhere, also in existing Tiburón code. It invalidates my own (but agreed: not Florian's requirements) that existing code remains running with only some global mode settings. (assuming Tiburón is \begin_inset Quotes eld \end_inset existing code \begin_inset Quotes erd \end_inset ). Or, generalized: if you synthesize an application using code from various sources you'll need \end_layout \begin_layout Standard I can illustrate that with two examples or thought experiments: \end_layout \begin_layout Subsubsection Existing code \end_layout \begin_layout Standard Assume I have a unit with UTF-16 Tiburón code. And and some unit with UTF-8 code of Lazarus descent where I globally replaced \begin_inset Quotes eld \end_inset ansistring \begin_inset Quotes erd \end_inset by unicodestring (or whatever identifier for the native type) to upgrade it to \begin_inset Quotes eld \end_inset native \begin_inset Quotes erd \end_inset unicode on an Unix target. \end_layout \begin_layout Standard Now we want these to work call eachother, and neither of these is prepared for the polymorphic type to contain the wrong encoding. Worse, literals in the Tiburón code will probably be created in the native (UTF-16) encoding. In turn, the UTF-8 routines might receive occasionally a string that has passed the Tiburón code and contains code that assumes UTF-16 encoding. The only solution is to audit the _entire_ source code for all these points, and insert ForceEncodings() statements for all parameters and after assignment of a literal. Here another potential problem surfaces, an empty string might not be forcable. \end_layout \begin_layout Standard This is an extremely hard sell to Delphi users, and IMHO not necessary anyway. Something will have to be done about this. A solution would be the hybrid proposal, see the separate paragraph further down. It is more or less the declarative behaviour of my proposal combined with the implementation of Florian's. \end_layout \begin_layout Subsubsection The granularity \end_layout \begin_layout Standard The problem with the granularity lies a bit in the same region as the last: if you have a procedure you must be prepared to handle all types. Now assume I honour that, and I am trying to make a procedure that understands both encodings, e.g. a dual encoding version of the \begin_inset Quotes eld \end_inset granularity \begin_inset Quotes erd \end_inset problem above. Then according to Florian's first quote above \emph on I only have one compiletime granularity while the type of my unicodestring is defined runtime ! \emph default \end_layout \begin_layout LyX-Code {$unicodestringindex } \end_layout \begin_layout LyX-Code \end_layout \begin_layout LyX-Code procedure myuniversalstringroutine(s:tunicodestring); \end_layout \begin_layout LyX-Code begin \end_layout \begin_layout LyX-Code if encodingof(s)=utf_8 Then \end_layout \begin_layout LyX-Code begin \end_layout \begin_layout LyX-Code for i:=1 to length(s) do // s in single bytes \end_layout \begin_layout LyX-Code s[i]:='a'; // s[i] in single byte values. type of literal? \end_layout \begin_layout LyX-Code end \end_layout \begin_layout LyX-Code else \end_layout \begin_layout LyX-Code begin // utf 16 \end_layout \begin_layout LyX-Code for i:=1 to length(s) do // length(s) in 2 byte values \end_layout \begin_layout LyX-Code s[i]:='a'; // s[i] in two byte values. type of literal? \end_layout \begin_layout LyX-Code end \end_layout \begin_layout LyX-Code \end_layout \begin_layout LyX-Code end; \end_layout \begin_layout LyX-Code \end_layout \begin_layout LyX-Code begin \end_layout \begin_layout LyX-Code myunversialstringroutine(getutf16stringroutine); \end_layout \begin_layout LyX-Code myunversialstringroutine(getutf8stringroutine); \end_layout \begin_layout LyX-Code end; \end_layout \begin_layout LyX-Code \end_layout \begin_layout Standard The conclusion of this is IMHO that shift size should be part of the runtime string too, iow a value of 1,2,4 somewhere at negative offset of the pointer. This is a performance penalty, since s[4] is then a more runtime construct. \end_layout \begin_layout Subsubsection Performance \end_layout \begin_layout Standard A runtime solution is always slower as a compiletime one. While performance isn't my biggest gripe, the problem is that I only see a small advantage in return: working VAR parameters and a lower need for overloading. For that we see a lot more checks done (because the encoding check must be after the nil check which will complicate codegeneration). \end_layout \begin_layout Standard Florian claims to partially earn this back with less conversions in all, but I don't buy that, except maybe in cases like (mostly GUI) apps with QT on *nix (where widgetset (UTF16) and system encoding (UTF8) are different). Simply having an type-alias for whatever encoding is the system encoding will achieve the same. Moreover, the decision which type to convert lies with the compiler which has generally more information at its disposal than the runtime library. Take for instance the following example: \end_layout \begin_layout LyX-Code var s1: utf8string; // utf-8 is the system encoding, we're on unix \end_layout \begin_layout LyX-Code s2: utf16string; \end_layout \begin_layout LyX-Code s1:=someinit8(); \end_layout \begin_layout LyX-Code s2:=someinit16(); \end_layout \begin_layout LyX-Code s1:=s2+s1; \end_layout \begin_layout LyX-Code utf8routine(s1); \end_layout \begin_layout Standard (note that for Florian's example, all string types are the same, in his case, read the declarations of s1 and s2 as \begin_inset Quotes eld \end_inset strings initialised filled with a utf-8/16 value) \end_layout \begin_layout Standard Now the runtime libs can probably not exploit the fact that the system encoding is more useful, and s1:=s2+s1; might end up converting the utf-8 type to utf-16, and storing the utf-16 result in s1. And then the check in utf8routine() will have to change the encoding again. \end_layout \begin_layout Standard Also the \begin_inset Quotes eld \end_inset leaf out routines \begin_inset Quotes erd \end_inset argument is IMHO bogus, since if the types of my proposal autoconvert (not unlike uniquestring()), the more complex routines like the bulky floating point and datetimeformatters could also be available only in the system encoding (which is most likely to happen), give or take a few small wrappers to work around VAR parameter problems. \end_layout \begin_layout Subsubsection Alternate encodings. \end_layout \begin_layout Standard (this paragraph is academic since we need to support other encodings because of Delphi, it was written before this was known though) \end_layout \begin_layout Standard Florian also mentioned an interest in supporting the old codepages as part of the requirements. I don't know if that was only a teaser because his proposal had more leeway for that or because he \emph on really \emph default saw a case and a need for that. \end_layout \begin_layout Standard However while I entertained the idea as interesting for a while, I'm not so convinced this is doable for two main reasons, \end_layout \begin_layout Itemize the UTF-x to UTF-y conversions are guaranteed to work if not corrupt, and if there are corner cases, they are far and few. But the codepages only accept a real small set of the possible codepoint set of the UTF-encodings and also eachother. The errorhandling is IMHO a problem. \end_layout \begin_layout Itemize Because the type of the polymorphic doesn't change unless forced, these strange encodings could penetrate everwhere in your codebase when simply strings are passed on unmodified. The amount of exceptions of unexpected encodings, and conversion failures all over your (till now working) code is confusing, unless you want to manually try except all string code in case some conversion goes wrong. \end_layout \begin_layout Subsubsection Florian's response \end_layout \begin_layout Standard The discussion about this article doesn't seem to have changed much about each parties viewpoint. Except maybe the \begin_inset Quotes eld \end_inset existing code \begin_inset Quotes erd \end_inset problem, \begin_inset Foot status collapsed \begin_layout Plain Layout Note that existing code is not only code that is \begin_inset Quotes eld \end_inset old \begin_inset Quotes erd \end_inset or \begin_inset Quotes eld \end_inset Tiburón \begin_inset Quotes erd \end_inset but in general all code that can only accept one encoding. \end_layout \end_inset \end_layout \begin_layout Standard (quote Florian) \end_layout \begin_layout Standard Indeed, it requires some work but there are several possibilities: \end_layout \begin_layout Enumerate add a switch for runtime checks about string encoding \end_layout \begin_layout Enumerate add a switch to enforce encoding at procedure entries and for function results \end_layout \begin_layout Standard The code needs to be reworked anyways. \end_layout \begin_layout Standard (...end quote..) \end_layout \begin_layout Standard I think this is butt ugly, and overly complicated, but at least it fixes my most major problem. Maybe if we can predeclare a lot of these as types, we can actually confine the clutter. \end_layout \begin_layout Standard \series bold note: \series default see also the \begin_inset CommandInset ref LatexCommand vref reference "sub:Problems-of-hybrid:" \end_inset paragraph, and the hybrid paragraph in general. There are complications. \end_layout \begin_layout Subsubsection The good points \end_layout \begin_layout Standard In some ways this proposal was better than the Windows centric Tiburon implement ation, in the sense that it unifies UTF8 and UTf16 in one stringtype. Apparantly Codegear even put more stress on cheapness of [] than I did. In theory a stringtype with a granularity field (in the TAnsiRec), could host both UTF8 and UTF16. This might cause Codegear pain when going multiplatform. If they persist in UTF16. Of course we'll never know for sure if this was a lost chance or not. This is likely, since for them multiplatform is mostly only a \begin_inset Quotes eld \end_inset feature \begin_inset Quotes erd \end_inset in addition to the core win32/64 product. \end_layout \begin_layout Subsection Yury's proposal \end_layout \begin_layout Standard Yury wrote something up independantly at FPC wiki about FPC Unicode support \begin_inset Flex URL status collapsed \begin_layout Plain Layout http://wiki.freepascal.org/FPC_Unicode_support \end_layout \end_inset . It is the same basic idea as Florian's: encodingtype and granularity-of-encodin g in the prefix of the string. He goes a step further and also seems to hint on reimplementing existing types on this scheme. (which is not realistic for shortstring, and maybe widestring). \end_layout \begin_layout Standard What I like in Yury's proposal is that he combines the implementation from Florian with the declaration that shows real types that I favour, in short, essentially it is the hybrid detail of the next paragraph in the rough. The hybrid model does divide some of the types over two types, the new unicodestring and ansistring (the codepage stuff, if we do that, there is no need to be Tiburón incompatible) \end_layout \begin_layout Subsection The hybrid model \end_layout \begin_layout Standard This is just a short thought experiment, this part hasn't been discussed with Florian and Michael much yet (though Yuri seems to come up with it independantly). The main reason is that the typing is my main grudge against Florian's proposal, and the performance less. It builds a bit on Florian's willingness to tackle some of those with directive s. If that gives enough leeway to define types, Florian's proposal morphs into this hybrid model. \end_layout \begin_layout Standard So assume we combine Florian's and some of the requirements (but not implementat ion) that are the basis for Marco's example. This means one base unicode type that can be parameterized to four types for declaration purposes (a single implementation of generic runtime dependant unicodestring as per Florian's proposal, but separate (sub)types per encoding (TUtf8string,TUtfstring16 and TUtfstring32)). These latter might be not real (compiler) types, but defined like below. \end_layout \begin_layout LyX-Code Type \end_layout \begin_layout LyX-Code tutf8string = type tunicodestring(Mandatory_UTF8); // or however we style the modifier. \end_layout \begin_layout LyX-Code \end_layout \begin_layout LyX-Code alternate syntax (?), more in Florian's style with directives \end_layout \begin_layout LyX-Code \end_layout \begin_layout LyX-Code type \end_layout \begin_layout LyX-Code {$unicodetype mandatory_utf8} \end_layout \begin_layout LyX-Code tutf8string = tunicodestring; \end_layout \begin_layout LyX-Code {$unicodetype general} \end_layout \begin_layout LyX-Code \end_layout \begin_layout Standard However because these forced types are 100% compatible with the full type, there is less of a multitude of overloads for VAR or overloading of helpers (for e.g. variant which only contains the general type). \end_layout \begin_layout Itemize the desired compiletime declarative behaviour, to be able to declare when a certain routine only accepts/expects a certain encoding. \end_layout \begin_layout Itemize the ability to have compiletime type knowledge to rearrange expressions to prefer a certain encoding result (see the performance paragraph) by using a different declaration (much like the Tiburón ansistring), if all components are typed. \end_layout \begin_layout Itemize In Tiburón mode, the string type is equal to TUTF16string, but can be mixed with any of the other types. \end_layout \begin_layout Itemize On implementation level, a single runtime implementation. No 3 ways of overloading. \end_layout \begin_layout Itemize The whole situation is then a bit analog to shortstring vs shortstring[] (from a typing point of view). All RTL routines are var shortstring, and accept all. However if you want to only support a certain size (like extensions), you can declare it using var s:shortstring[2]. But the unicode equivalent would be expected encoding, not size. Also e.g. variant would hold an FPC unicodestring, which is compatible without conversion to utf8string, utf16string,utf32string \end_layout \begin_layout Standard The main advantage of would be keeping the number of type dependant (not the more general routines) down, but to be able to retain the compiletime typed behaviour. Slowly I'm convinced this might be a doable way, but I need Florian's input for that. \end_layout \begin_layout Standard As a bonus, expanding this hybrid with Tiburón functionality is also possible, with quite high Tiburón compat, at the expense of having two UTF-8 types: \end_layout \begin_layout Itemize Implement the hybrid type as above. Only TUnicodestring only has the base three encodings. \end_layout \begin_layout Itemize Implement the Tiburón ansistring. utf-8 inclusive. This also includes the codepage support then. \end_layout \begin_layout Itemize In Delphi (Tiburón?) mode, the default unicodestring is an alias for TFlorianStr ing(talwaysutf16). \end_layout \begin_layout Standard This trick allows to simply add Tiburón code under the relative IFDEFS, and keep it working. And to gain maximum performance (avoid too much conversions in one codepage) on Unix utf-8 people would could remove the Tiburón flags on a per unit basis after inspecting the encoding state of an unit. \end_layout \begin_layout Standard The problems that I can think of, is that there is still a VAR problem, and the type and conversion situation in the compiler might get complicated (a lot of combinations), even though the number of overloads might be less. \end_layout \begin_layout Subsubsection Problems of hybrid: var \begin_inset CommandInset label LatexCommand label name "sub:Problems-of-hybrid:" \end_inset \end_layout \begin_layout Itemize VAR remains a problem, but afaik it is fixable. \end_layout \begin_layout Standard Assume we have RTL routine that does \end_layout \begin_layout LyX-Code procedure stringroutine (var s:TUNICODESTRING); \end_layout \begin_layout LyX-Code begin \end_layout \begin_layout LyX-Code forceencoding(s,utf16); // code only can deal with utf16 \end_layout \begin_layout LyX-Code process; // the utf16 processing code. \end_layout \begin_layout LyX-Code end; \end_layout \begin_layout LyX-Code \end_layout \begin_layout LyX-Code and \end_layout \begin_layout LyX-Code \end_layout \begin_layout LyX-Code var n : tUTF8String; \end_layout \begin_layout LyX-Code begin \end_layout \begin_layout LyX-Code {assign n} \end_layout \begin_layout LyX-Code stringroutine(n); // we can pass, since this is not a fully different type, but a TUNICODESTRING with a bit of afinity \end_layout \begin_layout LyX-Code // BUT: here n would be UTF16, a violation of the type declaration. \end_layout \begin_layout LyX-Code end \end_layout \begin_layout Standard This means that the compiler should insert a forceencoding after passing a string with encoding affinity to a generic VAR parameter. I hope that is doable. \end_layout \begin_layout Itemize As in Florian's original proposal the [] operator gets more expensive in generic routines. In non generic (roughly equivalent to Rawbytestring in Tiburón terms, but then completely implemented, not as just an open array solution for deep RTL routines) it is less of a problem, since the typing fixates the granularity ? This could offer a nice solution in the sense that high speed routines could be overloaded with typed equivalents for speed, while not so interesting routines could rely on the \begin_inset Quotes eld \end_inset general \begin_inset Quotes erd \end_inset type with runtime granularity. \end_layout \begin_layout LyX-Code \end_layout \end_body \end_document