Can GNU sed (for Windows) handle Unicode? If so, is it a code-page/locale issue, or a switch?

Posted by Peter.O on Super User See other posts from Super User or by Peter.O
Published on 2010-08-04T20:57:34Z Indexed on 2012/06/15 21:19 UTC
Read the original article Hit count: 217

Filed under:

Windows

|

encoding

|

unicode

|

sed

|

gnu

I've been using GNU SED on and off for a couple of years now. It spins me out a bit sometimes, but it does a good job... for single-byte char sets!
I now and then notice references to GNU SED being Unicode-aware, but the closest I've seen of this is its "binary" mode.. and binary is not Unicode.
Can GSED process a Unicode text file at CodePoint resolution, including and especially \r\n (Windows)... and if it can, does it expect UTF-8, UTF-16, or what? and how does SED detect the encoding?

© Super User or respective owner

Related posts about Windows

Server 2012 DFS New Member Issue

as seen on Server Fault - Search for 'Server Fault'
I am trying to add a new member to our DFS topology. We have 3 DCs (VMs - VMware) running Windows server 2012, two servers are located in or Primary site and the third at our DR site. Currently the two servers at our primary site are currently replicating DFS (full mesh) and are working fine. I have… >>> More
error in IIS7 but not on IIS6

as seen on Server Fault - Search for 'Server Fault'
I have a website that is we are now deploying to windows 2008 servers that has worked in the past on IIS6 without a problem. It is using .net 2 framework. Most of the website works. Just when we create a screen report over a certain size on the server we get this error. Event code: 3005 Event… >>> More
error in IIS7 but not on IIS6

as seen on Server Fault - Search for 'Server Fault'
I have a website that is we are now deploying to windows 2008 servers that has worked in the past on IIS6 without a problem. It is using .net 2 framework. Most of the website works. Just when we create a screen report over a certain size on the server we get this error. Event code: 3005 Event… >>> More
Microsoft Management Console stops working when I add snap-in to it

as seen on Super User - Search for 'Super User'
I have Windows 7 Ultimate OS. I'm opening mmc.exe as administrator and trying add Certificates or any other snap-in, then while loading that snap-in MMC breaks and displays following message and after that it closes automatically once I click on close button on that message. What could be the problem… >>> More
Handling keyboard and mouse input (Win API)

as seen on Game Development - Search for 'Game Development'
There is a number of ways to catch mouse or keyboard under Windows. So I tried some of them, but every of them has some advantages and drawbacks. I want to ask you: Which method do use? I've tried these: WM_KEYDOWN/WM_KEYUP - Main disadvantage is that, I can't distinguish between left and right-handed… >>> More

Related posts about encoding

<?xml version=“1.0” encoding=“UTF-8”?> not <?xml version='1.0' encoding='UTF-8'?>

as seen on Stack Overflow - Search for 'Stack Overflow'
I am using lxml with tree.write(xmlFileOut, pretty_print = True, xml_declaration = True, encoding='UTF-8' to write out my opened and edited xml file, but I absolutely need to have the xml declaration as <?xml version=“1.0” encoding=“UTF-8”?> and NOT <?xml version='1.0' encoding='UTF-8'… >>> More
Ivar definitions show 'long' type encoding as 'long long' type encoding

as seen on Stack Overflow - Search for 'Stack Overflow'
I've found what I think may be a bug with Ivar and Objective-C runtime. I'm using XCode 3.2.1 and associated libraries, developing a 64 bit app on X86_64 (MacBook Pro). Where I would expect the type encoding for the following "longVal" to be 'l', the Ivar encoding is showing a 'q' (which is a 'long… >>> More
How to avoid encoding the key of request parameters being encoding

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm trying to send a http request using WS.url() with a action receive a custom class parameter like public static void add(@Valid MyPage info) {...} There is a Map in MyPage @Required public Map<String, String> content = new HashMap<String, String>(); But When I try to send a request… >>> More
C# Check if character exists in encoding

as seen on Stack Overflow - Search for 'Stack Overflow'
I am writing a program that a part renders a bitmap font in CP437. In a function that renders the text with I want to be able to check whether a char is available in CP437 before the encoding conversion, like: public static void DrawCharacter(this Graphics g, char c) { if (char_exist_in_encoding(Encoding… >>> More
How to detect the character encoding of a text file?

as seen on Stack Overflow - Search for 'Stack Overflow'
I try to detect which character encoding is used in my file. I try with this code to get the standard encoding public static Encoding GetFileEncoding(string srcFile) { // *** Use Default of Encoding.Default (Ansi CodePage) Encoding enc = Encoding.Default; // *** Detect byte… >>> More