Chapter 30. Unicode/Charsets

+ Every industry eventually matures. One of the great areas of maturation is in the focus that has been given over the past decade to make it possible for anyone anywhere to use a computer. It has not always been that way. In fact, not so long @@ -11,19 +11,19 @@ language support for all computer users, the efforts of the Openi18n organization is deserving of special mention.

- + Samba-2.x supported a single locale through a mechanism called codepages. Samba-3 is destined to become a truly transglobal file- and printer-sharing platform. -

What Are Charsets and Unicode?

- +

What Are Charsets and Unicode?

+ Computers communicate in numbers. In texts, each number is translated to a corresponding letter. The meaning that will be assigned to a certain number depends on the character set (charset) that is used.

- - + + A charset can be seen as a table that is used to translate numbers to letters. Not all computers use the same charset (there are charsets with German umlauts, Japanese characters, and so on). The American Standard Code @@ -31,36 +31,36 @@ for Information Interchange (ASCII) encoding system has been the normative chara encoding scheme used by computers to date. This employs a charset that contains 256 characters. Using this mode of encoding, each character takes exactly one byte.

- - + + There are also charsets that support extended characters, but those need at least twice as much storage space as does ASCII encoding. Such charsets can contain 256 * 256 = 65536 characters, which is more than all possible characters one could think of. They are called multibyte charsets because they use more then one byte to store one character.

- + One standardized multibyte charset encoding scheme is known as unicode. A big advantage of using a multibyte charset is that you only need one. There is no need to make sure two computers use the same charset when they are communicating.

- - - + + + Old Windows clients use single-byte charsets, named codepages, by Microsoft. However, there is no support for negotiating the charset to be used in the SMB/CIFS protocol. Thus, you have to make sure you are using the same charset when talking to an older client. Newer clients (Windows NT, 200x, XP) talk Unicode over the wire. -

Samba and Charsets

- - +

Samba and Charsets

+ + As of Samba-3, Samba can (and will) talk Unicode over the wire. Internally, Samba knows of three kinds of character sets:

unix charset: - - + + This is the charset used internally by your operating system. The default is UTF-8, which is fine for most systems and covers all characters in all languages. The default @@ -73,26 +73,26 @@ Samba knows of three kinds of character sets: The default depends on the charsets you have installed on your system. Run testparm -v | grep "dos charset" to see what the default is on your system. -

Conversion from Old Names

- +

Conversion from Old Names

+ Because previous Samba versions did not do any charset conversion, characters in filenames are usually not correct in the UNIX charset but only for the local charset used by the DOS/Windows clients.

Bjoern Jacke has written a utility named convmv that can convert whole directory structures to different charsets with one single command. -

Japanese Charsets

Setting up Japanese charsets is quite difficult. This is mainly because:

- + The Windows character set is extended from the original legacy Japanese standard (JIS X 0208) and is not standardized. This means that the strictly standardized implementation cannot support the full Windows character set.
- - - - - + + + + + Mainly for historical reasons, there are several encoding methods in Japanese, which are not fully compatible with each other. There are two major encoding methods. One is the Shift_JIS series used in Windows @@ -112,24 +112,24 @@ Setting up Japanese charsets is quite difficult. This is mainly because: names may be different on different systems. This means that the value of the charset parameters depends on the implementation of iconv() you are using.
- - - - + + + + Though 2-byte fixed UCS-2 encoding is used in Windows internally, Shift_JIS series encoding is usually used in Japanese environments as ASCII encoding is in English environments. -

Basic Parameter Setting

- +

Basic Parameter Setting

+ The dos charset and display charset should be set to the locale compatible with the character set and encoding method used on Windows. This is usually CP932 but sometimes has a different name.

- - - + + + The unix charset can be either Shift_JIS series, EUC-JP series, or UTF-8. UTF-8 is always available, but the availability of other locales and the name itself depends on the system. @@ -166,8 +166,8 @@ Setting up Japanese charsets is quite difficult. This is mainly because: only. It is good practice to verify that the Japanized free software can work with Shift_JIS.

EUC-JP series

- - + + EUC-JP series means a locale that is equivalent to the industry standard called EUC-JP, widely used in Japanese UNIX (although EUC contains specifications for languages other than Japanese, such as @@ -176,16 +176,16 @@ Setting up Japanese charsets is quite difficult. This is mainly because: Windows on Samba, the filename on UNIX becomes 0xb6a6, 0xcdad, “.txt” (an 8-byte BINARY string).

- - - - - - - - - - + + + + + + + + + + Since EUC-JP is usually used on open source UNIX, Linux, and FreeBSD, and on commercial-based UNIX, Solaris, IRIX, and Tru64 UNIX as Japanese locale (however, it is also possible on Solaris to use Shift_JIS and UTF-8, and on Tru64 UNIX it is possible to use Shift_JIS). To use EUC-JP series, most Japanese filenames created from @@ -198,7 +198,7 @@ Setting up Japanese charsets is quite difficult. This is mainly because: commands that cannot handle non-ASCII filenames may be aborted during parsing filenames.

- + Moreover, if you built Samba using differently installed libiconv, the eucJP-ms locale included in libiconv and EUC-JP series locale included in the operating system may not be compatible. In this case, you may need to @@ -223,9 +223,9 @@ Setting up Japanese charsets is quite difficult. This is mainly because: must be handled carefully, so you had better not touch filenames written from Windows on UNIX.

- - - + + + In addition, although it is not directly concerned with Samba, since there is a delicate difference between the iconv() function, which is generally used on UNIX, and the functions used on other platforms, @@ -233,14 +233,14 @@ Setting up Japanese charsets is quite difficult. This is mainly because: Shift_JIS and Unicode UTF-8 must be done with care and recognition of the limitations involved in the process.

- + Although Mac OS X uses UTF-8 as its encoding method for filenames, it uses an extended UTF-8 specification that Samba cannot handle, so UTF-8 locale is not available for Mac OS X.

Shift_JIS series + vfs_cap (CAP encoding)

- - - + + + CAP encoding means a specification used in CAP and NetAtalk, file server software for Macintosh. In the case of CAP encoding, for example, if a Japanese filename consists of 0x8ba4 and 0x974c, and @@ -269,14 +269,14 @@ Setting up Japanese charsets is quite difficult. This is mainly because:

To use CAP encoding on Samba-3, you should use the unix charset parameter and VFS as in the VFS CAP smb.conf file. -

Example 30.1. VFS CAP

[global]

# the locale name "CP932" may be different

dos charset = CP932

unix charset = CP932

[cap-share]

vfs option = cap

- - - - +

Example 30.1. VFS CAP

[global]

# the locale name "CP932" may be different

dos charset = CP932

unix charset = CP932

[cap-share]

vfs option = cap

+ + + + You should set CP932 if using GNU libiconv for unix charset. With this setting, filenames in the “cap-share” share are written with CAP encoding. -

Individual Implementations

Here is some additional information regarding individual implementations:

GNU libiconv

To handle Japanese correctly, you should apply the patch @@ -299,15 +299,15 @@ display charset = CP932 to glibc-2.2.5/2.3.1/2.3.2 or should use the patch-merged versions, glibc-2.3.3 or later.

Using the above glibc, these setting are available: -

dos charset = CP932

unix charset = CP932 / eucJP-ms / UTF-8

display charset = CP932

dos charset = CP932

unix charset = CP932 / eucJP-ms / UTF-8

display charset = CP932

Other Japanese locales (for example, Shift_JIS and EUC-JP) should not be used because of the lack of the compatibility with Windows. -

Samba-2.2 Coding System	Samba-3 unix charset
SJIS	Shift_JIS series
EUC	EUC-JP series
EUC3^[a]	EUC-JP series
CAP	Shift_JIS series + VFS
HEX	currently none
UTF8	UTF-8
UTF8-Mac^[b]	currently none
others	none
^[a]Only exists in Japanese Samba version ^[b]Only exists in Japanese Samba version

Samba-2.2 Coding System	Samba-3 unix charset
SJIS	Shift_JIS series
EUC	EUC-JP series
EUC3^[a]	EUC-JP series
CAP	Shift_JIS series + VFS
HEX	currently none
UTF8	UTF-8
UTF8-Mac^[b]	currently none
others	none
^[a]Only exists in Japanese Samba version ^[b]Only exists in Japanese Samba version