Appendix 6 – Character Encoding

A6.1 Introduction

Encoding refers to the details of how the characters in the source file for a web page are coded for transmission over the web. For the most part the author can leave all the details to be handled by KompoZer or the browser. This leaves a few options available to the author these generally provide a means of optimising a file only in rare cases do they affect functionality.

A6.2 Quick start

KompoZer defaults to using ISO-8859-1 encoding with the following settings. You may check or reset this as follows.

• Use Format > Page Title and Properties and in the Internationalization area see that the 'Character set' box reads 'ISO-8859-1'. To change it using the drop down menu ' Choose a charset'.
• Use Tools > Preferences> Advanced and in the 'Special characters' area check that 'Output the following characters as entities' is set to 'Only & < > and non-breakable whitespace'.
• For the two check boxes below this:
  Clear 'Don't encode > outside of attribute values'
  Check 'Don't encode special characters in attribute values'

These settings are completely suitable for pages using English and adequate, though not necessarily optimised, for most other languages used in Western and Northern Europe.

If using an other European language and some other languages a different selection from ISO-8859 may be preferable. KompoZer offers the full available range. Wikipedia [Ref 14] has a useful article detailing the coverage.

A6.3 Encoding text

A6.3.1 ASCII and Latin 1 characters

Early computers used the ASCII (American Standard Code for Information Interchange) which provides a set of 95 printable characters dating from the teleprinter era. An eight bit byte however allows a doubling of this number (while reserving a number of codes for control purposes) and gives rise to the Latin-1 set illustrated in table A6.3-1. The row and column headings indicate the more and less significant parts of the code (in hexadecimal) corresponding to each character. For instance, the code for character 'A' is 41.

Latin-1 corresponds to the ISO-8859-1 set which is sufficient for web pages in English and many other western European languages. Include the appropriate code in a file and the corresponding character will appear.

A6.3.2 ISO-8859

The needs of many languages, European and other, can be satisfied by similar sets of characters, all share the ASCII characters and substitute some in other positions. This give rise to 15 standards in the ISO-8859 series. You can find which language, along with the list of characters, supported by each encoding in the Wikipedia article [Ref 9] referred to above.

To implement this it is clear that more than 256 characters are needed although only 256 locations (less control positions) are available to address them. The characters required to satisfy all in the series are drawn from a much larger set.

A6.3.3 Universal Character Set

The Unicode Consortium [Ref 17] have standardised a universal character set (UCS), i.e. a standard that defines, in one place, all the characters needed for writing the majority of living languages in use on computers. It aims to be, and to a large extent already is, a superset of all other character sets that have been encoded.

Unicode (as the UCS is commonly referred to) can access over a million characters of which about 100,000 have already been defined. These include characters for all the world's main languages along with a selection of symbols for various purposes.

HTML specifies a Document Character Set which is a list of the character repertoire available along with the corresponding code points (sometimes referred to as code positions). For HTML (and XHTML) the Document Character Set is identical to the UCS which means that, in principle, any character in the UCS may be used in any HTML document. In practice support for the complete character range is uneconomic and systems provide support for subsets only.

A6.3.4 Character Encoding

Character Encoding, at its simplest, refers to the process whereby the codes for the characters are mapped to the code points for the Unicode characters appropriate to the language in use. In the case of ISO-8859-1 the character codes are mapped to identical Unicode code points. (The first 256 Unicode characters being the same as the Latin-1 set.) As another example, ISO-8859-7 encodes Greek characters displacing many from the Latin-1 set to make room. (Compare table A6.3-2 to table A6.3-1.) In this case the code EA instead of being mapped to Unicode code point EA (giving e circumflex ê) is mapped to code point 03BA which returns a small kappa κ. In fact ISO-8859-7 does not include the ê character.

All ISO-8859 encodings retain the ASCII characters at the original positions.

This document uses ISO-8859-1 encoding but, in spite of this, has no difficulty in representing the full repertoire of the Greek characters covered by ISO-8859-7 as can be seen in the table. How this is achieved is explained in the next section.

Authors should note that every page uses one character encoding, and one only, irrespective of the number or range of languages encountered on a page.

In HTML pages character encoding is specified using the 'charset' parameter in the head area for each page. Several options are permissible but KompoZer always uses the form .

Note 'charset', in spite of its name, does not specify a character set. The character set for HTML documents is always the UCS. 'charset' specifies the encoding.

A6.3.5 Character references

ISO-8859 uses a single byte per character to represent all the characters commonly expected in a language but clearly there may be a need to represent uncommon characters. HTML provides two mechanisms – Character entity references (entities) and numeric character references. Using these methods any character in the UCS may be reached by using a sequence of ASCII characters to point to the required character. Entities take the form € and numeric references the form € or € all representing the euro symbol. The 8464 and x20AC represent the Unicode code point for the symbol in decimal and hexadecimal notation.

These methods free the author to employ Unicode characters, irrespective of the encoding in use, at the expense of increasing file size. Where such use is limited this is inconsequential.

The list of entities is included at section 24 of the HTML specification [Ref 16]. About 250 are defined, numeric character references must be used for characters outside this range. Characters do not have to be out of range of the encoding for entity references to be provided as is clear from Table A6.3-3 which lists some of the most frequently used including some in the ASCII set.

Note Entities are case sensitive thus É represents upper case E with an acute accent (É) while é represents the corresponding lower case letter (é). &EacutE; does not represent anything (&EacutE;). (The error just gets printed out.)

Note Irrespective of the ISO-8859 encoding employed the entity or numeric reference to be input remains the same. So, although in ISO-8859-7 the euro symbol is represented as byte A4, entering the code ¤ will generate a ¤ symbol not a euro symbol. The code to be input is the entity or numeric character reference for the character required.

A6.3.6 UTF encodings

ISO-8859 is fine when using one language at a time but becomes clumsy and slow when languages are mixed. UTF coding releases us from this restriction and provides a mechanism for addressing the full range of Unicode characters quickly. KompoZer allows coding in either UTF-8, UTF-16 or UTF-32 formats which are based on units of 8, 16 or 32 bits respectively. UTF-32 is not usually used for coding web pages.

UTF-8 uses 1 to 4 bytes to represent a character. It uses 1 byte to represent characters in the ASCII set, two bytes for the next 1920 characters (including the Latin alphabet characters with diacritics, Greek, Cyrillic, Coptic, Armenian, Hebrew, and Arabic characters) and three bytes for the rest of 65,000 characters in the Basic Multilingual plane (BMP). Supplementary characters use 4 bytes.

UTF-16 uses 2 bytes for any character in the BMP, and 4 bytes for supplementary characters.

UTF-32 uses 4 bytes for all characters

KompoZer offers additional practical advantages when UTF-8 encoding is used for pages with multiple languages. Irrespective of the encoding in use in Normal or Preview mode KompoZer depicts all characters correctly using the corresponding 'glyph'. In source view encodings which require characters to be represented as character references show the character reference. This reduces readability significantly. With UTF encoding all characters can be represented as glyphs so the problem is avoided.

A6.3.7 Symbols

Traditionally computers have relied on special fonts like 'Symbol' or 'Wingdings' to produce symbols. This is not necessary on web pages. Since such fonts do not support Unicode any attempt to use them will yield unreliable results which may vary from browser to browser.

Fortunately Unicode supports a large range of symbols which fulfills many needs.

Inputting special symbols

There are several ways of inserting symbols into a page using KompoZer.

1. Via the Insert menu. Using Insert > Characters and Symbols. Covers accented characters and a number of other common symbols
2. Using Windows Character Map. Using the standard font in use find the symbol required, select and copy it, then in KompoZer paste it as required. Recent versions of Character map allow you to group characters by Unicode subrange which makes it easier to find a particular symbol
3. If you use a particular symbol frequently it may be easier to insert it by keystroke. Several characters permit this and, where it can be done, the keystroke is shown at the bottom right corner of Character Map. The euro symbol, €, for instance, may be inserted using ALT+0128
4. AllChars [Ref21]http://allchars.zwolnet.com/ is a useful utility that allows any program to insert any Windows-1282 (see below) character using a few easily discovered keystrokes
5. In KompoZer Source view any character may be inserted using the Numeric character reference. In the bottom left corner of Character Map this is given in hex format. Thus for euro you insert €
   Note The code is shown in Character map as U+20AC This is the conventional way of representing Unicode characters i.e. by 'U+' followed by the hexadecimal code point
6. In source view, entity references may be inserted similarly. Many are easy to memorise e.g. € > <

The first four methods can be used in normal or preview mode, the last two in source view.

The keen-eyed may observe that the key codes are neither the character codes nor the Unicode code points for the character required. In fact they are the (decimal) character codes derived from Windows-1252 encoding. KompoZer will convert these to whatever code is appropriate depending on the encoding selected and the character involved. (So the key code never appears in the source.) Incidentally encoding is a possible alternative to ISO-8859-1 suitable for western languages. It increases the number of available character codes to 218 characters by re-allocating some of the codes in the range 80 to 9F which are normally unused.

Alan Wood's website [Ref 2] is a useful resource listing entities (where defined) and Numeric character references for a large number of characters from the Monotype Typography Symbol font (as on Windows XP) including Greek, Mathematical and Punctuation [Ref 6] and also the Microsoft Wingdings font [Ref 7]. (For Windings in several cases there is no Unicode equivalent.)

A6.3.8 Unicode support

A6.3.8.1 Fonts

Although Unicode offers tremendous potential the usual caveats apply when choosing fonts. See for instance section 4.2.1.1 but, in this case, it is important to check that all fonts in the list include the characters required. No font covers the full range of Unicode, or even a small single digit percentage of it. To check the supported Unicode ranges of a font Microsoft supply an for Windows Explorer. With it installed, right-click any TrueType (TTF) font file in Windows Explorer and select the Properties tab. Particular characters can be searched for using Character map.

Checking for support is more than usually difficult if unusual characters are required. Compatible fonts must be installed on any visitor's computer and, where in a style sheet the font-family is specified as a prioritised list of font family names (as it should be), ideally all fonts in the list should be checked.

Note A font list lists fonts the first of which will be used if available. It does not check that the character required is supported by the font. So even if support is provided by a font lower in the list that may not be accessed.

Check the rendering of a page on as many different browsers as possible. Mozilla browsers do authors only a partial service since, if a character is encountered which is not included in the font(s) listed they will make an attempt to find the character on other fonts installed on the machine. Authors should however check all pages using MSIE 6 which does not offer this capability. (MSIE7 will substitute for a few characters.) Visitors may finish up looking at square boxes instead of the character required.

Alan Wood offers several pages which are extremely useful in this respect. Using special characters from Windows Glyph List 4 (WGL4) in HTML [Ref 3] lists characters in the WGL4 set and which are likely to be widely available. Unicode fonts for Windows computers [Ref 4] lists which fonts carry specific ranges of Unicode characters and more interestingly shows distribution of the fonts so that authors may check likely availability to visitors. Those wishing to use a rarer character may check which fonts include them at Unicode character ranges and the Unicode fonts that support them [Ref 5].

A6.3.8.2 Examples

While preparing this page, for instance, Table A6.3-1 displayed correctly in Firefox and KompoZer but in MSIE the arrows originally appeared as squares. The issue is reproduced in the box on the right. The arrows use comparatively rare characters that do not appear in the Tahoma font used but, on the writer's machine at least, the Gecko engine was able to retrieve them, possibly from Lucida Sans Unicode.

The result is that visitors using MSIE see boxes instead of arrows but those using Firefox or Opera may see the arrows if Lucida Sans Unicode or some other font with the characters is installed on their machine.

A work-around this issue is possible, as also shown in the box. The list specifying the font is modified so that the first in the list becomes 'Lucida Sans Unicode'. If this is available it will be used, otherwise the choice passes down the list. Alan Wood shows that this font is supplied with Windows XP and Windows 2000 which cover 90% of installations (mid 2007).

This is a moderately, but not very, robust solution. Had the availability of the arrows been critical to understanding the table it would have been necessary to change the design.

While the arrows may be considered rare and unusual characters even characters covered by some ISO-8859 options may not be reliable. In viewing Table A6.3-2, depending on the browser in use and fonts installed there are two characters, Drachma sign (Code A5) and Greek ypogegrammeni (Code AA), which may not display correctly. In cases like these checking the WGL4 list may provide a warning because neither of the characters is listed.

A6.4 Special characters

A6.4.1 Entities or numeric character references

A6.4.1.1 Output the following characters

The way in which characters are coded in the source for the page may be altered in KompoZer using Tools > Preferences > Advanced. In the Special Characters area there are four options under 'Output the following characters as entities'

• Only & < > and non-breakable whitespace
• The above and Latin-1 letters
• HTML 4 special characters
• Use &#…; notation for all non-ASCII characters

The options refer to characters typed onto the page which will be readable by a visitor to a site. Irrespective of the option set the visual appearance on screen will remain the same.

• First option

Only & < > and non-breakable whitespace

Note In normal practice the character referred to as 'non-breakable whitespace' is called 'no-break space' (entity  ).

The section on 'Preferences' recommends this as the preferred option.

This is the minimal setting. The characters listed must always be encoded whatever option is selected. With this selection the encoding will be as entities. Since the character < occurs in HTML code to mark the start of an element, if it is included in the page text the browser would expect to to start a new element and the page would become corrupted. It must always be encoded. The > character marks the end of an element and should be safe to use but W3C recommend that it also be encoded since it may confuse older browsers See section 5.3.2 of the HTML Specification [Ref 16]. If you wish to override this check the box 'Don't encode > outside of attribute values'.

Since entities and numeric character references start with an ampersand (&) a similar problem occurs with this character.

Outputing the no-break space as an entity is convenient since it would otherwise look like a normal space in a listing.

With this option the output will be an entity where specified, else for encodable characters, it will be the code for the character else the character will be output as a decimal numeric character code.

Before publishing a page always select this option since it will result in the smallest file size.

• Second option

The option 'The above and Latin-1 letters' Strictly should read 'The above and Latin-1 characters except ASCII characters'. That refers to characters in the Latin-1 set with codes in the range A0 to FF.

The output code for ASCII characters is the character code, for the remaining Latin-1 characters is the entity, else is a decimal numeric character reference.

• Third option

The option 'HTML 4 special characters' refers to all characters for which The HTML 4 specification [Ref 16] (Section 24) provides an entity reference.

The output code for ASCII characters is the character code, for all characters for which an entity exists is the entity else it is a decimal numeric character reference.

• Final option

The final option should output decimal numeric character reference for all non-ASCII characters but has a bug so that it uses a mixture of character codes (for ASCII characters) and entities or numeric references for others.

Note The output will always produce valid and operating code for the corresponding character so files run correctly. With Nvu 1.0 and KompoZer 0.7.10, for all options, there are a few detailed non-compliances with the above description.

The options provided may be useful during development if you work in source code and may make it easier for you to read. They may also be required if transferring source code to some other applications. If, for instance, your file was encoded as UTF-8 and you wanted to transfer the content to an application which could accept only text the fourth option would be useful.

This ensures that only ASCII characters appear in the file. Unfortunately this is one option that does not work in Nvu 1.0 and KompoZer 0.7.10

Note. In source view you may enter characters using any of these systems irrespective of the option selected. After leaving and re-entering source view, the display will conform to the option selected and not to the form in which it was entered.

In Normal and Preview mode KompoZer will attempt to render all characters correctly, irrespective of the option selected, subject to the limitations described under 'Unicode support' above.

If you need an ascii only file an alternative way to obtain one is via the File > Save and change Character Encoding menu item. Check the 'Export to text' box and save the file as text. The original file remains intact. Again with UTF encoding this appears to be faulty but if the encoding is temporally switched to ISO-8859-1 it works.

A6.4.1.2 Don't encode > outside of attribute values

This is explained under the first option above.

The section on 'Preferences' suggests that this be left in the default, cleared, state but this is optional.

A6.4.2 Encoding in attribute values

One final option in Tools > Preferences > Advanced provides the option 'Don't encode special characters in Attribute values'. That is not a very precise definition of what happens because the whole page must be encoded in the same way. What it changes is the way in which character references are used in attribute values.

When you check the box special characters appearing in the value of any attribute will be left untouched and not encoded as entities or numerical character references.

A6.4.2.1 Don't encode special characters in attribute values

An example will serve to illustrate.

The table element has an attribute 'summary' and might appear as:

The attribute is 'summary' and the value is 'Resumé of results'. If the characters were encoded as entities the source would read

Even worse if the characters were encoded as numeric character references the source would read

Any readout on the screen, whether through a properties inspector or at the bottom of the browser window, will be normal but those doing lot of work in source view might prefer not to encode in this situation.

You are recommended to check the box 'Don't encode special characters in attribute values' unless you have good reason to do otherwise.

Where the attribute value is a URL a different encoding method is covered in the following section.

Table A6.5-1

Characters permissible in a URL
	x0	x1	x2	x3	x4	x5	x6	x7	x8	x9	xA	xB	xC	xD	xE	xF
0x																	⇑ A S C I I ⇓	⇑ L A T I N 1 ⇓
1x
2x	sp	!	"	#	$	%	&	'	(	)	*	+	,	-	.	/
3x	0	1	2	3	4	5	6	7	8	9	:	;	<	=	>	?
4x	@	A	B	C	D	E	F	G	H	I	J	K	L	M	N	O
5x	P	Q	R	S	T	U	V	W	X	Y	Z	[	\	]	^	_
6x	`	a	b	c	d	e	f	g	h	i	j	k	l	m	n	o
7x	p	q	r	s	t	u	v	w	x	y	z	{	|	}	~
8x
9x
Ax	nbsp	!	¢	£	¤	¥	¦	§	¨	©	ª	«	¬	shy	®	¯
Bx	°	±	²	³	´	µ	¶	·	¸	¹	º	»	¼	½	¾	¿
Cx	À	Á	Â	Ã	Ä	Å	Æ	Ç	È	É	Ê	Ë	Ì	Í	Î	Ï
Dx	Ð	Ñ	Ò	Ó	Ê	Ë	Ö	×	Ø	Ù	Ú	Û	Ü	Ý	Þ	ß
Ex	à	á	â	ã	ä	å	æ	ç	è	é	ê	ë	ì	í	î	ï
Fx	ð	ñ	ò	ó	ô	õ	ö	÷	ø	ù	ú	û	ü	ý	þ	ÿ

Previous

Next

Top of page