Showing posts with label unicode. Show all posts
Showing posts with label unicode. Show all posts

2008-05-06

Mr Donald, Please Correct the Alphabet First!

I have already replied to Mr Donald Gaminitillake's mudslinging campaign against Sinhala Unicode, which he wields through akuru.org web site and by hijacking discussions on various blogs and forums.

Mr Donald's motives are quite clear. He claims that every Sinhala character shape needs an individual "code point", and has applied for a patent for this "invention". With Sinhala Unicode becoming mainstream, avenues for making money with his pending patent are going thin.

So he is doing what any desperate human being (or animal for that matter) would do; try everything to remove the "opponent".

One of the examples Mr Donald always uses is the absence of character "du" in the Sinhala Unicode codepage.

Of course he conveniently forgets to mention that "da" and "papilla" are in fact available. Well, it requires a bit of brains to put them together. ;-)

Mr Donald, there are lots of missing characters in the Sinhala Hodiya (alphabet), including your infamous "du", let alone "yansaya" and "rakaransaya". If you love the Sinhala language so much as you claim, please start a campaign to "fix" Hodiya!

I have previously pointed out this similarity between Hodiya and Sinhala Unicode, and why "du" + "papilla" is as good as "du". This blog post discusses technicalities in detail including the matter of "yansaya" and "rakaransaya".

Unfortunately for Mr Donald, his "opponent", namely Sinhala Unicode, is growing stronger day by day. Implementations are maturing, more standards compliant fonts are beginning to appear, and as I wrote earlier, more web sites and blogs are now Unicode compliant (e.g.: Sinhala Bloggers, Sinhala Wikipedia, Sinhala Blogs and of course our own Sinhala GNU/Linux).

Read more...

2008-04-20

LaTeX and Sinhala Unicode

When we met at Excel World on last 17th, Bud, Srimal and myself started talking about using Sinhala Unicode in TeX / LaTeX.

It didn't occur to me that Chamath, who also created one of the first Sinhala FOSS keyboard drivers, has already created a preprocessor for LaTeX called sintex which reads Sinhala files in Unicode/UTF-8. In fact, not only had I replied to his announcement, but also sent a patch to Debianize it! Life is too complex, and I am too human to keep track of all these.

But that forgetfulness turned out to be a lucky incident, as our pursuit lead to something more useful!

So we started creating a preprocessor for Vasantha Saparamadu's Sinhala TeX package which uses Samanala transliteration scheme.

However, Bud pointed out that the generated PDF files will have ASCII characters instead of Unicode, making it a problem for search engines that index them, and convert them for "HTML view" pages.

After some research, we found XeTeX, a Unicode enabled version of LaTeX.

XeTeX uses ICU for text layout, and ICU versions after 3.6 supports Sinhala out of the box. However, latest stable version 0.996 of XeTeX uses statically linked ICU 3.4. I managed to patch the "tetex-xetex" package that comes with Debian and make it recognize Sinhala. The patches were also submitted to Debian.

XeTeX font changes are always manual, which made the source look ugly. After a bit of research, I found zhspacing package, which among other things automatically sets fonts for Chinese characters. But it is a complicated package, but I managed to get an idea of how it uses character class feature in the latest XeTeX version 0.997.

Downloading the latest version of XeTeX from SVN repository and building for Debian was not difficult, except I had to edit debian/control files to replace tetex-base and tetex-bin dependency to their texlive counterparts. I had to first get xdvipdfmx. Here is a rough sketch of the work.

% mkdir xdvipdfmx
% cd xdvipdfmx
% svn co http://scripts.sil.org/svn-view/xdvipdfmx/TRUNK
% cd TRUNK
% chmod +x debian/rules
# dpkg-buildpackage -b
# cd ..
# dpkg --purge dvipdfmx
# dpkg -i xdvipdfmx...deb
% cd ..

% mkdir xetex
% cd xetex
% svn co http://scripts.sil.org/svn-view/xetex/TRUNK
% cd TRUNK
% vi debian/control
% chmod +x debian/rules
# dpkg-buildpackage -b
# cd ..
# dpkg --purge texlive-xetex
# dpkg -i xetex...deb

As the XeTeX web site had warned, the Debian build files provided by vanilla XeTeX were not up to date. After installing I had to create a /etc/texmf/fmt.d/10local.cnf with the following two lines:

xetex   xetex  -             *xetex.ini
xelatex xetex  language.dat  *xelatex.ini

and then run the following commands:

# update-fmutil
# fmutil-sys --enablefmt xetex
# fmutil-sys --enablefmt xelatex

to make "xelatex" command to work properly.

After getting latest version of XeTeX working, the last remaining step was to create a small style file, which I called "sinhala.sty", to make automatic font switching for Sinhala.

% sinhala.sty version 20080420
% Typesetting mixed Sinhala documents in XeTeX
%
% Copyright (C) 2008 by Anuradha Ratnaweera
%
\ifx\XeTeXrevision\@undefined
  \errmessage{XeTeX is required to use sinhala}
\fi
\ifx\XeTeXinterchartokenstate\@undefined
  \errmessage{XeTeX 0.997 or above required to use sinhala}
\fi
\ProvidesPackage{sinhala}[2008/04/20]
\RequirePackage{fontspec}
\newfontinstance{\sifont}[Script=Sinhala]{LKLUG}
\newcommand\latinfont{\fontfamily{lmr}\selectfont}
\XeTeXinterchartokenstate = 1
\newcount\cnt\cnt="0D80
\loop
  \XeTeXcharclass\cnt=10 \ifnum\cnt<"0DFF \advance\cnt1
\repeat
\XeTeXcharclass "200C = 10
\XeTeXcharclass "200D = 10
\XeTeXinterchartoks 0 10 = {\sifont}
\XeTeXinterchartoks 255 10 = {\sifont}
\XeTeXinterchartoks 10 0 = {\latinfont}
\XeTeXinterchartoks 10 255 = {\latinfont}

So, all you need is XeTeX 0.997 and sinhala.sty to write LaTeX files using Sinhala Unicode.

Read more...

2008-03-07

More Sinhala Web Sites Using Unicode

Use of of Sinhala Unicode on the Web is slowly, but steadily, growing.

Two years ago, our good friend Niranjan Meegammana started the Sinhala Unicode group to bring together all those who are involved in the development of Sinhala Unicode to communicate, and to use Sinhala itself to do so. Today the group has become one of the very active forums on the topic, and provides a living example for the success of the standard.

Although some newspapers are still reluctant to move from legacy font tricks to Sinhala Unicode, Dinamina and e-thalaya online newspapers made a bold move by using only Unicode in their web sites, providing fine examples that the implementations are mature.

This Sinhala blog sindicator aggregates dozens of blogs written in Sinhala.

My previous posts on this topic are here, here and here.

Read more...

2007-05-29

Unicode and Sinhala Alphabet

There is a great deal of similarity between Sinhala Unicode (~ SLS 1134) and Sinhala Hodiya (alphabet).

Sidath Sangarawa, one of the oldest texts on Sinhala grammar written over 2000 years ago, lists 10 vowels and 20 consonants (see footnote 1), but the book also uses two unlisted vowels ඇ and ඈ (see footnote 2).

Sanskrit influence increased the number of characters to over 50.

Actual number of shapes, known as "glyphs" in modern typographic terminology, needed to write in Sinhala is in the range of thousands, due to derived and joint forms of basic characters.

Listing all these thousands of glyphs was never a popular practice. Students learn basic characters and modifiers, and common sense takes care of generating the thousands of other shapes. For example, after learning "ispilla", you can add it to basic consonants and generate all the "i" forms such as "ki", "gi", "ji" etc.

Hodiya doesn't have any of these extra characters such as "ki" or "du". Hodiya doesn't have rakaransaya nor yansaya. But nobody complained. Everybody knew, and still know, that the Hodiya is only a basic guide to generate more complex glypls.

However, this didn't work when Sinhala texts started to be printed on printing machines. These machines don't have brains and couldn't learn how to "generate". Therefore every possible glyph had to be given.

Walk into an old press to see a large "matrix" or such glyphs.

Then came the age of computer based typography. Computers can be taught to do things, and that is exactly how standards like Unicode and SLS 1134 generate shapes. We can teach computers to generate thousands of glyphs using less than a hundred of basic shapes. For example, we can generate "du" by adding "da" and "papilla", so a seperate "du" is not necessary.

How about "yansaya" and "rakaransaya"? They are generated by sequences including the zero-width joiner (ZWJ). For example, "pra" is represented as "pa", "hal kireema", ZWJ and "ra". ZWJ also is used to represent joint and touching letters.

Gone are the days of brainless matrix-based printing machines.

We need two things to view Sinhala on a computer. A font containing Sinhala glyphs, and the computer programs should knows how to generate glyphs using sequences of basic characters. Let me explain using an example.

Step 0. Here is how a sample web page looks on a browser when it cannot find a Sinhala Unicode font. The "boxes" indicate unavailable character numbers:

Step 1. After installing a font, the browser will show some Sinhala, but if it hasn't "learned" how to generate glyphs, only basic characters and modifiers are shown independently:

Step 2. Now I have enabled the "shaper" in the browser, which is the part that knows how to generate Sinhala glyphs using basic characters:

All is well!

So where exactly is the similarity?

Students learn less than 100 basic characters in the alphabet and modifiers, and use their brains with some support from teachers to generate the rest of the 1000+ shapes.

Computers can be programmed - and some computers have already been programmed - to generate 1000+ Sinhala glyphs using less than 100 basic shapes in Sinhala Unicode / SLS 1134 standard.

As the standard is platform independent, we use it to communicate with people using diverse platforms in the Sinhala Unicode Group among others.

Footnote 1. පණකුරු පසෙක් එද ලුහු ගුරු බෙයින් දසවේ, ගතකුරුද වේ විස්සෙක්, වහරට යුහු හෙළ බස

Footnote 2. Notice the use of ඇ and ඈ, both independently and in consonants: පසැස් ඈ සරලොප් නැතද සර ගතට පැමිණවූ බැවින් සර සඳ නම්.

Read more...