intertwingly

“Incompetent?”  What is it with XML gurus and the name-calling?

Seriously though, Korean text is usually in EUC-KR, not UTF-8, and it’s one of the easiest languages/encodings to auto-detect accurately, using a multibyte prober and some frequency distribution analysis.  And Korean (or anything else) in UTF-8 can be detected with a regular expression.  I’m not sure what the chances are that a random binary file would match, but I’m sure the chances go down as the file size goes up.

(Yah, those quotes were intentional.  Taken from here.  Savor the irony.)

Random Thoughts, in no particular order:

• HTTP’s Content-Type headers assumes that anything served as text/* is an incompetent format, including HTML.  That hasn’t worked out too well either.
• Requiring people to declare their encoding only works if one can assume that authors understand character encoding.  Experience with XML on the web suggests otherwise.
• Sniffing content type is generally a bad idea.  Sniffing character encoding generally is better than trusting authors to get it right.  Funny that.
• UTF-8 is indeed highly sniffable, primarily due to it’s rather unforgiving design.  One that is often implemented in a draconian manner.
• Don’t get me started on normalization forms and Supplementary Multilingual Planes.
• Un nuevo principio.  Savor the irony indeed.  :-)
• “Just” use HTTP

PEP 263: This PEP proposes to introduce a syntax to declare the encoding of a Python source file. The encoding information is then used by the Python parser to interpret the file using the given encoding. I wonder if this will work: #!/usr/bin/env...

A post from Sam Ruby on encoding issues, quoting a post by Rick Jelliffe on the same issues, prompted an interesting response by Lenny Dommitser . For those that didn’t follow the link, Lenny’s post includes an excellent example of the use of rot13...