Get info from multiple files, match it and then display to end user, what is fastest?

Posted by Patrick on Stack Overflow See other posts from Stack Overflow or by Patrick
Published on 2010-05-11T14:59:34Z Indexed on 2010/05/11 15:04 UTC
Read the original article Hit count: 190

Filed under:

multiple

|

data

|

database

Hi, I need to build a website where we display data that is refreshed every 5minutes in a text file with a | separator.

I currently use Java to do this.

What I do now: I grab the textfile for every request through the website and process it and then display the data to the end user, this works fine, since Java can go through like 5000 lines of data fast, and when I filter it it is still extremely fast.

However now the management wants the following: They added 3 textfiles with the | separator to it, and now want me to also read those files and match the information on certain fields, and if there is a match also display that information to the end user.

I think soon enough, although Java is fast, I will run into trouble when 10 people want that information and I have to run through 4 total files matching the information.

What can I do to make this process super fast?

My Creative solutions so far: -Leave it this way, since Java is fast and end users can wait (probably less then 1second) -Have a background process that dumps new data into a MySQL database every 5minutes, since databases are extremely good at getting the same data from multiple tables.

Thank you!

© Stack Overflow or respective owner

Related posts about multiple

Multiple Base Addresses and Multiple Endpoints in WCF

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm using two bindings TCP and HTTP. I want to give mex data on both bindings. What I want is that the mexHttpBinding only exposes the HTTP services while the mexTcpBinding exposes TCP services only. Or is this possible that I access stats service only from HTTP binding and the eventLogging service… >>> More
LINQ Query using Multiple From and Multiple Collections

as seen on Microsoft .NET Support Team - Search for 'Microsoft .NET Support Team'
1: using System; 2: using System.Collections.Generic; 3: using System.Linq; 4: using System.Text; 5: 6: namespace ConsoleApplication2 7: { 8: class Program 9: { 10: static void Main(string[] args) 11: { 12:… >>> More
How to avoid LinearAlloc Exceeded Capacity error android

as seen on Stack Overflow - Search for 'Stack Overflow'
The application gets crashing every-time, when am running eclipse saying LinearAlloc exceeded capacity (5242880), last=208 This is happening, when am creating AsyncTask, thats strange this is happening everytime . when am commenting and running its running. Logcat is: 02-09 04:02:23.374:… >>> More
Run MySQL INSERT Query multiple times (insert values into multiple tables)

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, basically, I have 3 tables; users and projects (which is a many-to-many relationship), then I have 'usersprojects' to allow the one-to-many formation. When a user adds a project, I need the project information stored and then the 'userid' and 'projectid' stored in the usersprojects table. It sounds… >>> More
Selecting multiple fields into multiple variables in a MySQL stored procedure

as seen on Stack Overflow - Search for 'Stack Overflow'
I am a little new to store procedures in MySQL and was wondering if I can SELECT multiple columns into multiple variables within the same select query. for example (iName is the input to the function): DECLARE iId INT(20); DECLARE dCreate DATETIME; SELECT Id INTO iId, dateCreated INTO dCreate… >>> More

Related posts about data

timetable in a jTable

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to create a timetable in a jTable. For the top row it will display from monday to sunday and the left colume will display the time of the day with 2h interval e.g 1st colume (0000 - 0200), 2nd colume (0200 - 0400) .... And if i click a button the timing will change from 2h interval to 1h interval… >>> More
Reading data from an Entity Framework data model through a WCF Data Service

as seen on ASP.net Weblogs - Search for 'ASP.net Weblogs'
This is going to be the fourth post of a series of posts regarding ASP.Net and the Entity Framework and how we can use Entity Framework to access our datastore. You can find the first one here , the second one here and the third one here . I have a post regarding ASP.Net and EntityDataSource. You… >>> More
SQL SERVER – Advanced Data Quality Services with Melissa Data – Azure Data Market

as seen on SQL Authority - Search for 'SQL Authority'
There has been much fanfare over the new SQL Server 2012, and especially around its new companion product Data Quality Services (DQS). Among the many new features is the addition of this integrated knowledge-driven product that enables data stewards everywhere to profile, match, and cleanse data.… >>> More
Modifying a HTML page to fix several "bugs" add a function to next/previous on a option dropdown

as seen on Stack Overflow - Search for 'Stack Overflow'
SOF, I've got a few problems plaguing me at the moment and am wondering if anyone could assist me with them. I'm trying to get Next Class | Previous Class to act as buttons so that when Next Class is clicked it will go to the next item in the dropdown list and for previous it would go to back one… >>> More
Shrinking TCP Window Size to 0 on Cisco ASA

as seen on Server Fault - Search for 'Server Fault'
Having an issue with any large file transfer that crosses our Cisco ASA unit come to an eventual pause. Setup Test1: Server A, FileZilla Client <- 1GBPS - Cisco ASA <- 1 GBPS - Server B, FileZilla Server TCP Window size on large transfers will drop to 0 after around 30 seconds of a large… >>> More