Searching through large data set

Posted by calccrypto on Stack Overflow See other posts from Stack Overflow or by calccrypto
Published on 2010-05-17T21:23:01Z Indexed on 2010/05/17 22:50 UTC
Read the original article Hit count: 288

Filed under:

python

|

arrays

|

strings

|

search

|

duplicate-entry

how would i search through a list with ~5 mil 128bit (or 256, depending on how you look at it) strings quickly and find the duplicates (in python)? i can turn the strings into numbers, but i don't think that's going to help much. since i haven't learned much information theory, is there anything about this in information theory?

and since these are hashes already, there's no point in hashing them again

© Stack Overflow or respective owner

Related posts about python

unmet dependencies in Ubuntu 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried today to install a dvb-card on my Ubuntu 12.04 (Linux blauhai-linux 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ). The installation failed with an error. After that, i tried to install python (it was already installed but i got this error): linux:~$… >>> More
How can I get sikuli-ide to work?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I installed sikuli-ide with sudo apt-get install sikuli-ide Everything was fine until I tried to start it from the terminal. I typed sikuli-ide But the only response I got was [info] locale: en_US The application was not started, furthermore there is no desktop file and sikuli-ide does not… >>> More
Getting PATH right for python after MacPorts install

as seen on Super User - Search for 'Super User'
I can't import some python libraries (PIL, psycopg2) that I just installed with MacPorts. I looked through these forums, and tried to adjust my PATH variable in $HOME/.bash_profile in order to fix this but it did not work. I added the location of PIL and psycopg2 to PATH. I know that Terminal is… >>> More
call python with system() in R to run a python script emulating the python console

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to pass a chunk of Python code to Python in R with something like system('python ...'), and I'm wondering if there is an easy way to emulate the python console in this case. For example, suppose the code is "print 'hello world'", how can I get the output like this in R? >>> print… >>> More
Python - Calling a non python program from python?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am currently struggling to call a non python program from a python script. I have a ~1000 files that when passed through this C++ program will generate ~1000 outputs. Each output file must have a distinct name. The command I wish to run is of the form: program_name -input -output -o1 -o2… >>> More

Related posts about arrays

Arrays in php, Compare 2 arrays and keep duplicate values

as seen on Stack Overflow - Search for 'Stack Overflow'
The following is 2 different definitions of the problem: How can I process 2 arrays so that I can keep the data of duplicate array[key] values. I have arrays A and B. I want to create array C with the children that their id/key values exist in both A and B arrays. Thank you >>> More
commands&creating pointer [closed]

as seen on Stack Overflow - Search for 'Stack Overflow'
input 23 3 4 4 42 n 23 0 9 9 n n n 3 9 9 x //according to input,i should create int pointer arrays. pointer arrays // starting from 1 (that is initial arrays is arrays[1].when program sees n ,it // must be jumb to arrays 2 // the first int input 23 is num_arrays which used… >>> More
n & x commands&creating pointer&with using malloc [closed]

as seen on Stack Overflow - Search for 'Stack Overflow'
input 23 3 4 4 42 n 23 0 9 9 n n n 3 9 9 x //according to input,i should create int pointer arrays. pointer arrays // starting from 1 (that is initial arrays is arrays[1].when program sees n ,it // must be jumb to arrays 2 // the first int input 23 is num_arrays which… >>> More
reopen or read and say why not reopened [closed]

as seen on Stack Overflow - Search for 'Stack Overflow'
input 23 3 4 4 42 n 23 0 9 9 n n n 3 9 9 x //according to input,i should create int pointer arrays. pointer arrays // starting from 1 (that is initial arrays is arrays[1].when program sees n ,it // must be jumb to arrays 2 // the first int input 23 is num_arrays which used… >>> More
Fortran arrays and subroutines (sub arrays)

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm going through a Fortran code, and one bit has me a little puzzled. There is a subroutine, say SUBROUTINE SSUB(X,...) REAL*8 X(0:N1,1:N2,0:N3-1),... ... RETURN END Which is called in another subroutine by: CALL SSUB(W(0,1,0,1),...) where W is a 'working array'. It appears that a specific… >>> More