Reading and conditionally updating N rows, where N > 100,000 for DNA Sequence processing

Posted by makerofthings7 on Programmers See other posts from Programmers or by makerofthings7
Published on 2012-09-12T03:13:26Z Indexed on 2012/09/12 3:48 UTC
Read the original article Hit count: 545

Filed under:

c#

|

multithreading

|

Azure

|

locks

I have a proof of concept application that uses Azure tables to associate DNA sequences to "something".

Table 1 is the master table. It uniquely lists every DNA sequence. The PK is a load balanced hash of the RK. The RK is the unique encoded value of the DNA sequence.

Additional tables are created per subject. Each subject has a list of N DNA sequences that have one reference in the Master table, where N is > 100,000.

It is possible for many tables to reference the same DNA sequence, but in this case only one entry will be present in the Master table.

My Azure dilemma:

I need to lock the reference in the Master table as I work with the data. I need to handle timeouts, and prevent other threads from overwriting my data as one C# thread is working with the information. Other threads need to realise that this is locked, and move onto other unlocked records and do the work.

Ideally I'd like to get some progress report of how my computation is going, and have the option to cancel the process (and unwind the locks).

Question

What is the best approach for this?

I'm looking at these code snippets for inspiration:

http://blogs.msdn.com/b/jimoneil/archive/2010/10/05/azure-home-part-7-asynchronous-table-storage-pagination.aspx

http://stackoverflow.com/q/4535740/328397

© Programmers or respective owner

Related posts about c#

.NET WebRequest.PreAuthenticate not quite what it sounds like

as seen on West-Wind - Search for 'West-Wind'
I’ve run into the problem a few times now: How to pre-authenticate .NET WebRequest calls doing an HTTP call to the server – essentially send authentication credentials on the very first request instead of waiting for a server challenge first? At first glance this sound like it should be easy:… >>> More
HttpWebRequest and Ignoring SSL Certificate Errors

as seen on West-Wind - Search for 'West-Wind'
Man I can't believe this. I'm still mucking around with OFX servers and it drives me absolutely crazy how some these servers are just so unbelievably misconfigured. I've recently hit three different 3 major brokerages which fail HTTP validation with bad or corrupt certificates at least according to… >>> More
The dynamic Type in C# Simplifies COM Member Access from Visual FoxPro

as seen on West-Wind - Search for 'West-Wind'
I’ve written quite a bit about Visual FoxPro interoperating with .NET in the past both for ASP.NET interacting with Visual FoxPro COM objects as well as Visual FoxPro calling into .NET code via COM Interop. COM Interop with Visual FoxPro has a number of problems but one of them at least got a lot… >>> More
Dynamic Type to do away with Reflection

as seen on West-Wind - Search for 'West-Wind'
The dynamic type in C# 4.0 is a welcome addition to the language. One thing I’ve been doing a lot with it is to remove explicit Reflection code that’s often necessary when you ‘dynamically’ need to walk and object hierarchy. In the past I’ve had a number of ReflectionUtils that used string based expressions… >>> More
Finding a Relative Path in .NET

as seen on West-Wind - Search for 'West-Wind'
Here’s a nice and simple path utility that I’ve needed in a number of applications: I need to find a relative path based on a base path. So if I’m working in a folder called c:\temp\templates\ and I want to find a relative path for c:\temp\templates\subdir\test.txt I want to receive back subdir\test… >>> More

Related posts about multithreading

C++ Multithreading on Unix

as seen on Programmers - Search for 'Programmers'
I have two related questions: 1) Are there any good books for multithreading in C++, especially now that C++11 contains multithreading in the standard library? 2) I have the Wrox Programming on Unix book (1000 pages fat red one) and within it, it uses the Unix Thread class. How does this code relate… >>> More
what are the difficulties of operating system multithreading?

as seen on Stack Overflow - Search for 'Stack Overflow'
I am reading a book that compares two ways of implementing threads, Middleware Threads and OS Threads. I have a question about these sentences: "A difficulty of operating system multithreading, however, is performance overhead. Since it is the operating system that is involved in switching threads… >>> More
TCP multicast and multithreading

as seen on Stack Overflow - Search for 'Stack Overflow'
I need to come up with clients that can multicast to other clients reliably. That implies I'll be using TCP to connect reliably between clients within a multicast group. Doesn't that come up to n^2 number of connections? That seems a little silly to me. Wouldn't/shouldn't there be a way to more easily… >>> More
Multithreading recommendation based on program description

as seen on Stack Overflow - Search for 'Stack Overflow'
I would like to describe some specifics of my program and get feedback on what the best multithreading model to use would be most applicable. I've spent a lot of time now reading on ThreadPool, Threads, Producer/Consumer, etc. and have yet to come to solid conclusions. I have a list of files (all… >>> More
How to achieve multithreading using JavaScript with IE6?

as seen on Stack Overflow - Search for 'Stack Overflow'
How to achieve multithreading using JavaScript with IE6? Is there a third party library to do this? >>> More