Efficient way to ASCII encode UTF-8

Posted by Andreas Gohr on Stack Overflow See other posts from Stack Overflow or by Andreas Gohr
Published on 2010-04-02T14:59:00Z Indexed on 2010/04/02 15:03 UTC
Read the original article Hit count: 307

Filed under:

utf-8

|

punycode

|

encoding

|

ascii

I'm looking for a simple and efficient way to store UTF-8 strings in ASCII-7. With efficient I mean the following:

all ASCII chars in the input should stay ASCII chars in the output
the resulting string should be as short as possible
the operation needs to be reversable without any data loss
there should be no restriction on the input length
the whole UTF-8 range should be allowed

My first idea was to use Punycode (IDNA) as it fits the first three requirements, but it fails at the last two.

Can anyone recommend an alternative encoding scheme? Even better if there's some code available to look at.

© Stack Overflow or respective owner

Related posts about utf-8

Why can't I change the AU_AU locale to en_US?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
/bin/bash: warning: setlocale: LC_ALL: cannot change locale ( (unset)) Generating locales... en_US.ISO-8859-1... /usr/sbin/locale-gen: line 177: warning: setlocale: LC_ALL: cannot change locale ( (unset)) done Generation complete. ganesha@ubuntu:~$ sudo update_locale LANG=en_US sudo: update_locale:… >>> More
Reading a plist utf-8 value as utf-16

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm working on an iphone app that needs to display superscripts and subscripts. I'm using a picker to read in data from a plist but the unicode values aren't being displayed corretly in the pickerview. Subscripts and superscripts are not being recognized. I'm assuming this is due to the encoding… >>> More
Forcing a mixed ISO-8859-1 and UTF-8 multi-line string into UTF-8

as seen on Stack Overflow - Search for 'Stack Overflow'
Consider the following problem: A multi-line string $junk contains some lines which are encoded in UTF-8 and some in ISO-8859-1. I don't know a priori which lines are in which encoding, so heuristics will be needed. I want to turn $junk into pure UTF-8 with proper re-encoding of the ISO-8859-1… >>> More
How can I tell if a CSV is in UTF-7 or UTF-8

as seen on Stack Overflow - Search for 'Stack Overflow'
Excel seems to save CSV files in (what I think is) UTF-7, despite the fact that most information I have read suggest that in general, you should not UTF-7. Indeed, other applications (Text pad, which lets me choose) save things in UTF-8 (or Unicode etc, but UTF-7 is not even an option). Using .NET… >>> More
Confused about C++'s std::wstring, UTF-16, UTF-8 and displaying strings in a windows GUI

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm working on a english only C++ program for Windows where we were told "always use std::wstring", but it seems like nobody on the team really has much of an understanding beyond that. I already read the question titled "std::wstring VS std::string. It was very helpful, but I still don't quite… >>> More

Related posts about punycode

Complications registering a punycode domain name

as seen on Server Fault - Search for 'Server Fault'
Not sure if any of you have experience with this, but I am trying to include the anchor (?) in my domain name (using the appropriate punycode to allow it) but upon registering it I encounter the error that the symbol is not supported by the language I have chosen. Does anyone know what language… >>> More
Trouble registering punycode domain!

as seen on Super User - Search for 'Super User'
Not sure if any of you have experience with this, but I am trying to include the anchor (?) in my domain name (using the appropriate punycode to allow it) but upon registering it I encounter the error that the symbol is not supported by the language I have chosen. Does anyone know what language would… >>> More
Efficient way to ASCII encode UTF-8

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm looking for a simple and efficient way to store UTF-8 strings in ASCII-7. With efficient I mean the following: all ASCII chars in the input should stay ASCII chars in the output the resulting string should be as short as possible the operation needs to be reversable without any data loss there… >>> More
Allowed unicode characters in IDN host labels

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi all, Im currently working on a "proper" URI validator and currently it all comes down to hostname validation, the rest isnt that tricky. Im stuck at IDN hostname labels (e.g. containing unicode; possible punycode encoded strings have been decoded at this point). My first idea was basicly a regex… >>> More
IDN aware tools to encode/decode human readable IRI to/from valid URI

as seen on Stack Overflow - Search for 'Stack Overflow'
Let's assume a user enter address of some resource and we need to translate it to: <a href="valid URI here">human readable form</a> HTML4 specification refers to RFC 3986 which allows only ASCII alphanumeric characters and dash in host part and all non-ASCII character in other parts… >>> More