String chunking algorithm with natural language context

Posted by Chris Ballance on Stack Overflow See other posts from Stack Overflow or by Chris Ballance
Published on 2010-03-22T18:37:50Z Indexed on 2010/03/22 18:41 UTC
Read the original article Hit count: 568

Filed under:

string-manipulation

|

c#

|

language-processing

I have a arbitrarily large string of text from the user that needs to be split into 10k chunks (potentially adjustable value) and sent off to another system for processing.

Chunks cannot be longer than 10k (or other arbitrary value)
Text should be broken with natural language context in mind
- split on punctuation when possible
- split on spaces if no punction exists
- break a word as a last resort

I'm trying not to re-invent the wheel with this, any suggestions before I roll this from scratch?

Using C#.

© Stack Overflow or respective owner

Related posts about string-manipulation

String manipulation in C#

as seen on Stack Overflow - Search for 'Stack Overflow'
I have the following text, I need to extract the exception name and the continuing sentence from the file, but the file has continuous sentences without a space. ??????>????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ??????????… >>> More
Regarding String manipulation

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a String str which can have list of values like below. I want the first letter in the string to be uppercase and if underscore appears in the string then i need to remove it and need to make the letter after it as upper case. The rest all letter i want it to be lower case. "" "abc" "abc_def" "Abc_def_Ghi12_abd" "abc__de" "_" Output: "" "Abc" "AbcDef" "AbcDefGhi12Abd" "AbcDe" "… >>> More
String manipulation in C# ( Brainstroming ;-) )

as seen on Stack Overflow - Search for 'Stack Overflow'
I have the following text , i need to extract the exception name and the continuing sentence from the file... but the file has continues sentence without a space. ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????… >>> More
php string-manipulation (clear the "_" and the timestamp)

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi! I cant find the way how to strip the useless section of my string (read from SQL) I've tried the strreplace and truncate but those was not good at all. I've got a string variable called $stuff if $stuff = 145_timestamp i want to clear the _ and the chars after it. so i want to $stuff be 145… >>> More
c# string manipulation

as seen on Stack Overflow - Search for 'Stack Overflow'
i have string of 9 letters. string myString = "123987898"; I want to retrieve frist 3 letter "123" then 2 more leters "98" and then 4 more letters "7898". Which c# string function support this functionality. >>> More

Related posts about c#

.NET WebRequest.PreAuthenticate not quite what it sounds like

as seen on West-Wind - Search for 'West-Wind'
I’ve run into the problem a few times now: How to pre-authenticate .NET WebRequest calls doing an HTTP call to the server – essentially send authentication credentials on the very first request instead of waiting for a server challenge first? At first glance this sound like it should be easy:… >>> More
HttpWebRequest and Ignoring SSL Certificate Errors

as seen on West-Wind - Search for 'West-Wind'
Man I can't believe this. I'm still mucking around with OFX servers and it drives me absolutely crazy how some these servers are just so unbelievably misconfigured. I've recently hit three different 3 major brokerages which fail HTTP validation with bad or corrupt certificates at least according to… >>> More
The dynamic Type in C# Simplifies COM Member Access from Visual FoxPro

as seen on West-Wind - Search for 'West-Wind'
I’ve written quite a bit about Visual FoxPro interoperating with .NET in the past both for ASP.NET interacting with Visual FoxPro COM objects as well as Visual FoxPro calling into .NET code via COM Interop. COM Interop with Visual FoxPro has a number of problems but one of them at least got a lot… >>> More
Dynamic Type to do away with Reflection

as seen on West-Wind - Search for 'West-Wind'
The dynamic type in C# 4.0 is a welcome addition to the language. One thing I’ve been doing a lot with it is to remove explicit Reflection code that’s often necessary when you ‘dynamically’ need to walk and object hierarchy. In the past I’ve had a number of ReflectionUtils that used string based expressions… >>> More
Finding a Relative Path in .NET

as seen on West-Wind - Search for 'West-Wind'
Here’s a nice and simple path utility that I’ve needed in a number of applications: I need to find a relative path based on a base path. So if I’m working in a folder called c:\temp\templates\ and I want to find a relative path for c:\temp\templates\subdir\test.txt I want to receive back subdir\test… >>> More