Data clean up: are there libraries of common permutations that we can use? Or is there a better appr

Posted by anyaelena on Stack Overflow See other posts from Stack Overflow or by anyaelena
Published on 2010-03-17T04:54:43Z Indexed on 2010/03/17 5:01 UTC
Read the original article Hit count: 223

Filed under:

natural-language

|

data

|

match

We are working on clean-up and analysis of a lot of human-entered customer data. We need to decide programmatically whether 2 addresses (for example) are the same, even though the data was entered with slight variations.

Right now we run each address through fairly simplistic string replacement (replacing avenue with ave, for example), concatenate the fields and compare the results. We are doing something similar with names.

At the very least, it seems like our list of search-replace values should already exist somewhere.

Or perhaps you can suggest a totally different and superior way to detect matches?

© Stack Overflow or respective owner

Related posts about natural-language

details on the following Natural Language Processing terms ?

as seen on Stack Overflow - Search for 'Stack Overflow'
Named Entity Extraction (extract ppl, cities, organizations) Content Tagging (extract topic tags by scanning doc) Structured Data Extraction Topic Categorization (taxonomy classification by scanning doc....bayesian ) Text extraction (HTML page cleaning) are there libraries that i can use to do any… >>> More
Natural Language parsing of an appointment?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm looking for a Java library to help parse user entered text that represents an 'appointment' for a calendar application. For instance: Lunch with Mike at 11:30 on Tuesday or 5pm Happy hour on Friday I've found some promising leads like https://jchronic.dev.java.net/ and http://www.datejs… >>> More
Natural language processing - Ideas for beginner's projects

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi guys, I am a beginner in NLP and NLTK. I am very interested in NLP and hence joined a weekend course on AI in some local institution, which requires me to do a project for completion of the course, and I decided to do it in NLP. The problem is,the instructor is not good at all for this course… >>> More
String chunking algorithm with natural language context

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a arbitrarily large string of text from the user that needs to be split into 10k chunks (potentially adjustable value) and sent off to another system for processing. Chunks cannot be longer than 10k (or other arbitrary value) Text should be broken with natural language context in mind split… >>> More
oppertunities in the area of natural language processing

as seen on Stack Overflow - Search for 'Stack Overflow'
i worked on indian language telugu using python... and now i am interested to work in any company which works on natural language processing.if any oppertunities please tell me >>> More

Related posts about data

timetable in a jTable

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to create a timetable in a jTable. For the top row it will display from monday to sunday and the left colume will display the time of the day with 2h interval e.g 1st colume (0000 - 0200), 2nd colume (0200 - 0400) .... And if i click a button the timing will change from 2h interval to 1h interval… >>> More
Reading data from an Entity Framework data model through a WCF Data Service

as seen on ASP.net Weblogs - Search for 'ASP.net Weblogs'
This is going to be the fourth post of a series of posts regarding ASP.Net and the Entity Framework and how we can use Entity Framework to access our datastore. You can find the first one here , the second one here and the third one here . I have a post regarding ASP.Net and EntityDataSource. You… >>> More
SQL SERVER – Advanced Data Quality Services with Melissa Data – Azure Data Market

as seen on SQL Authority - Search for 'SQL Authority'
There has been much fanfare over the new SQL Server 2012, and especially around its new companion product Data Quality Services (DQS). Among the many new features is the addition of this integrated knowledge-driven product that enables data stewards everywhere to profile, match, and cleanse data.… >>> More
Modifying a HTML page to fix several "bugs" add a function to next/previous on a option dropdown

as seen on Stack Overflow - Search for 'Stack Overflow'
SOF, I've got a few problems plaguing me at the moment and am wondering if anyone could assist me with them. I'm trying to get Next Class | Previous Class to act as buttons so that when Next Class is clicked it will go to the next item in the dropdown list and for previous it would go to back one… >>> More
Shrinking TCP Window Size to 0 on Cisco ASA

as seen on Server Fault - Search for 'Server Fault'
Having an issue with any large file transfer that crosses our Cisco ASA unit come to an eventual pause. Setup Test1: Server A, FileZilla Client <- 1GBPS - Cisco ASA <- 1 GBPS - Server B, FileZilla Server TCP Window size on large transfers will drop to 0 after around 30 seconds of a large… >>> More