Search Results

Search found 3618 results on 145 pages for 'huge'.

Page 7/145 | < Previous Page | 3 4 5 6 7 8 9 10 11 12 13 14  | Next Page >

  • JBoss envers and huge audit tables

    - by LeChe
    All, I am auditing my JEE application with JBoss Evers and the nature of my application causes the audit table to grow very fast. The historic data is queried infrequently and access time is not really an issue, apart from the data from the last week. This data IS queried frequently and access needs to be fast. Ideally, I would split the data and distribute it over two tables, with the older data in compressed format. Unfortunately, Envers does not allow spreading data over multiple tables as far as I can tell from the docs. Does somebody have any idea what would be the best way to achieve this (if possible while still using Envers)? Cheers, LeChe

    Read the article

  • Read random lines from huge CSV file in Python

    - by jbssm
    I have this quite big CSV file (15 Gb) and I need to read about 1 million random lines from it. As far as I can see - and implement - the CSV utility in Python only allows to iterate sequentially in the file. It's very memory consuming to read the all file into memory to use some random choosing and it's very time consuming to go trough all the file and discard some values and choose others, so, is there anyway to choose some random line from the CSV file and read only that line? I tried without success: import csv with open('linear_e_LAN2A_F_0_435keV.csv') as file: reader = csv.reader(file) print reader[someRandomInteger] A sample of the CSV file: 331.093,329.735 251.188,249.994 374.468,373.782 295.643,295.159 83.9058,0 380.709,116.221 352.238,351.891 183.809,182.615 257.277,201.302 61.4598,40.7106

    Read the article

  • Cloud HUGE data storage options?

    - by ToughPal
    Hi, Does anyone have a good suggestion on how to do video recording? We have a camera that can record and then stream live video to a server. So this means we can have 1000's of cameras sending data 24X7 for recording. We will store data for over 7 / 14 / 30 days depending on the package. Per day if a camera is sending data to the server then it will store 1.5GB. So that means there is a traffic of 1.5GB / day / camera Total monthly 45GB / month / camera (Data + bandwidth for one camera) Please let me know the most cost effective way to get this data stored? Thanks!

    Read the article

  • querying huge database table takes too much of time in mysql

    - by Vijay
    Hi all, I am running sql queries on a mysql db table that has 110Mn+ unique records for whole day. Problem: Whenever I run any query with "where" clause it takes at least 30-40 mins. Since I want to generate most of data on the next day, I need access to whole db table. Could you please guide me to optimize / restructure the deployment model? Site description: mysql Ver 14.12 Distrib 5.0.24, for pc-linux-gnu (i686) using readline 5.0 4 GB RAM, Dual Core dual CPU 3GHz RHEL 3 my.cnf contents : [root@reports root]# cat /etc/my.cnf [mysqld] datadir=/data/mysql/data/ socket=/tmp/mysql.sock sort_buffer_size = 2000000 table_cache = 1024 key_buffer = 128M myisam_sort_buffer_size = 64M # Default to using old password format for compatibility with mysql 3.x # clients (those using the mysqlclient10 compatibility package). old_passwords=1 [mysql.server] user=mysql basedir=/data/mysql/data/ [mysqld_safe] err-log=/data/mysql/data/mysqld.log pid-file=/data/mysql/data/mysqld.pid [root@reports root]# DB table details: CREATE TABLE `RAW_LOG_20100504` ( `DT` date default NULL, `GATEWAY` varchar(15) default NULL, `USER` bigint(12) default NULL, `CACHE` varchar(12) default NULL, `TIMESTAMP` varchar(30) default NULL, `URL` varchar(60) default NULL, `VERSION` varchar(6) default NULL, `PROTOCOL` varchar(6) default NULL, `WEB_STATUS` int(5) default NULL, `BYTES_RETURNED` int(10) default NULL, `RTT` int(5) default NULL, `UA` varchar(100) default NULL, `REQ_SIZE` int(6) default NULL, `CONTENT_TYPE` varchar(50) default NULL, `CUST_TYPE` int(1) default NULL, `DEL_STATUS_DEVICE` int(1) default NULL, `IP` varchar(16) default NULL, `CP_FLAG` int(1) default NULL, `USER_LOCATE` bigint(15) default NULL ) ENGINE=MyISAM DEFAULT CHARSET=latin1 MAX_ROWS=200000000; Thanks in advance! Regards,

    Read the article

  • Help me for creating huge database in Mysql

    - by user90552
    We are building a website for business on global wise, for every country major cities are covered in this concept. I need some suggestions from PHP Mysql People. Can i create single databse for all cities or multiple databases. Because in this system contains some relations between cities ,every chamber need nearly 50 tables for networking and some other tables. If I can create separate databases for every chamber there would be nearly 50*1000 tables need because we have 1000 cities. So Please give suggestions how can i build database for my system. Thank you Ravi

    Read the article

  • MVC framework for huge JEE application

    - by chaKa
    Which MVC-framework is the best option (performance/ease of development) for a web application, that will have + 2 million visits per week. Basically the site is a search engine,but also there will be large amounts of xml parsing, and high db traffic. We are using Java, over Jboss 4.2.3x, with PG as DB, and Solr for the searches. We were thinking on code JSPs with taglibs, and Servlets, but we were feeling like there would be a better alternative, which don't know yet, as we are starting on the Java Web applications world. Any opinions, and shares of your experience will be appreciated! Thanks in advance!

    Read the article

  • How to search for duplicate values in a huge text file having around Half Million records

    - by Shibu
    I have an input txt file which has data in the form of records (each row is a record and represents more or less like a DB table) and I need to find for duplicate values. For example: Rec1: ACCOUNT_NBR_1*NAME_1*VALUE_1 Rec2: ACCOUNT_NBR_2*NAME_2*VALUE_2 Rec3: ACCOUNT_NBR_1*NAME_3*VALUE_3 In the above set, the Rec1 and Rec2 are considered to be duplicates as the ACCOUNT NUMBERS are same(ACCOUNT_NBR1). Note: The input file shown above is a delimiter type file (the delimiter being *) however the file type can also be a fixed length file where each column starts and ends with a specified positions. I am currently doing this with the following logic: Loop thru each ACCOUNT NUMBER Loop thru each line of the txt file and record and check if this is repeated. If repeated record the same in a hashtable. End End And I am using 'Pattern' & 'BufferedReader' java API's to perform the above task. But since it is taking a long time, I would like to know a better way of handling it. Thanks, Shibu

    Read the article

  • UITableView: Need a liveUpdate (Huge amount of cells)

    - by antihero
    Hey all, I have a UITableView which shows some information. The informations are from a XMLFile. The Xmlfile is pretty big, about 500 entries. I don't want that the user has to wait until the parsing has finished, i want to add cell by cell to the TableView. Here's my code: -(void) onFinishedPartDownload:(id)item { if(odvTableViewDelegateController.odvs == nil) odvTableViewDelegateController.odvs = [NSMutableArray new]; if([odvTableViewDelegateController.odvs count]==0) { genericDelegate.tableView.delegate = odvTableViewDelegateController; genericDelegate.tableView.dataSource = odvTableViewDelegateController; } [odvTableViewDelegateController.odvs addObject:item]; [genericDelegate.tableView reloadData]; } This method is called everytime an item has parsed, but it reloads the tableView all at once at the end. I don't have my expected liveUpdate which i'm searching for. The intervall the method is called is about 0.001 to 0.0025 seconds. Any ideas to fix that? Sebastian

    Read the article

  • Loading Huge Image

    - by japs
    Hi, I Want to load Image size 2550X3300 (i.e 1.7 Mb size), i have loaded the image into UIImageView and application gets crash due to low memory, Now i have loaded into uiWebview it works fine but i have to save this image into an PDF file in local resource. While iam saving UIImage in background same app gets crash due to low memory. Anyone has some suggestion or help to solve this issue. Thank You.

    Read the article

  • How can I sqldump a huge database?

    - by meder
    SELECT count(*) from table gives me 3296869 rows. The table only contains 4 columns, storing dropped domains. I tried to dump the sql through: $backupFile = $dbname . date("Y-m-d-H-i-s") . '.gz'; $command = "mysqldump --opt -h $dbhost -u $dbuser -p $dbpass $dbname | gzip > $backupFile"; However, this just dumps an empty 20 KB gzipped file. My client is using shared hosting so the server specs and resource usage aren't top of the line. I'm not even given ssh access or access directly to the database so I have to make queries through PHP scripts I upload via FTP ( SFTP isn't an option, again ). Is there some way I can perhaps sequentially download portions of it, or pass an argument to mysqldump that will optimize it? I came across http://jeremy.zawodny.com/blog/archives/000690.html which mentions the -q flag and tried that but it didn't seem to do anything differently.

    Read the article

  • How do you handle huge if-conditions?

    - by Teifion
    It's something that's bugged me in every language I've used, I have an if statement but the conditional part has so many checks that I have to split it over multiple lines, use a nested if statement or just accept that it's ugly and move on with my life. Are there any other methods that you've found that might be of use to me and anybody else that's hit the same problem? Example, all on one line: if (var1 = true && var2 = true && var2 = true && var3 = true && var4 = true && var5 = true && var6 = true){ Example, multi-line: if (var1 = true && var2 = true && var2 = true && var3 = true && var4 = true && var5 = true && var6 = true){ Example-nested: if (var1 = true && var2 = true && var2 = true && var3 = true){     if (var4 = true && var5 = true && var6 = true)     {

    Read the article

  • How to handle huge table ?

    - by misha-moroshko
    I would like to display to user a table which contains ~500,000 rows. The table should be calculated based on a file (i.e. the table is known at the beginning, and does not change). I guess that building the HTML text of this table is not a good idea in terms of memory performance. So how can I build such table ? I would like user to be able to scroll the table using a vertical scroll bar. Is it possible to build on the fly only the visible part of the table ? I'm afraid to see delays because of this. Is it a better idea to use server side programming rather than Javascript ? Any advise would be appreciated.

    Read the article

  • Database design: one huge table or separate tables?

    - by littlegreen
    Currently I am designing a database for use in our company. We are using SQL Server 2008. The database will hold data gathered from several customers. The goal of the database is to acquire aggregate benchmark numbers over several customers. Recently, I have become worried with the fact that one table in particular will be getting very big. Each customer has approximately 20.000.000 rows of data, and there will soon be 30 customers in the database (if not more). A lot of queries will be done on this table. I am already noticing performance issues and users being temporarily locked out. My question, will we be able to handle this table in the future, or is it better to split this table up into smaller tables for each customer?

    Read the article

  • Creating and parsing huge strings with javascript?

    - by user246114
    Hi, I have a simple piece of data that I'm storing on a server, as a plain string. It is kind of ridiculous, but it looks like this: name|date|grade|description|name|date|grade|description|repeat for a long time this string can be up to 1.4mb in size. The idea is that it's a bunch of student records, just strung together with a simple pipe delimeter. It's a very poor serialization method. Once this massive string is pushed to the client, it is split along the pipes into student records again, using javascript. I've been timing how long it takes to create, and split, these strings on the client side. The times are actually quite good, the slowest run I've seen on a few different machines is 0.2 seconds for 10,000 'student records', which has a final string size of ~1.4mb. I realize this is quite bizarre, just wondering if there are any inherent problems with creating and splitting such large strings using javascript? I don't know how different browsers implement their javascript engines. I've tried this on the 'major' browsers, but don't know how this would perform on earlier versions of each. Yeah looking for any comments on this, this is more for fun than anything else! Thanks

    Read the article

  • Mysql console slow on import of huge sql files

    - by Kennethvr
    My import of sql via the mysql console is rather slow and as our sql file is increasing every day I would like to know if there are any alternatives on how to import a sql file faster. Changing to oracle or other systems is no option, the configuration has to stay the same. Currently the sql file is: 1.5 Gb I'm on Wamp with Apache 2.2.14, PHP 5.2.11 and MySQL 5.1.41. Any suggestions?

    Read the article

  • Why does GC.GetTotalMemory() report huge memory allocations?

    - by Seventh Element
    I have been playing around with GC.GetTotalMemory(). When I create a local variable of type Titles in the example below, the consumed amount of memory increases by 6276 bytes. What's going on here? class Program { enum Titles { Mr, Ms, Mrs, Dr }; static void Main(string[] args) { GetTotalMemory(); Titles t = Titles.Dr; GetTotalMemory(); } static void GetTotalMemory() { long bytes = GC.GetTotalMemory(true); Console.WriteLine("{0}", bytes); } }

    Read the article

  • Parsing Huge XML Files in PHP

    - by Ian
    I'm trying to parse the dmoz content/structures xml files into mysql, but all existing scripts to do this are very old and don't work well. How can I go about opening a large (+1GB) xml file in php for parsing?

    Read the article

  • Getting one line in a huge file with PHP

    - by JavaRocky
    How can i get a particular line in a 3 gig text file. The lines are delimited by \n. And i need to be able to get any line on demand. How can this be done? Only one line need be returned. And i would not like to use any system calls. Note: There is the same question elsewhere regarding how to do this in bash. I would like to compare it with the PHP equiv.

    Read the article

  • Java-Counting occurrence of word from huge textfile

    - by Naveen
    I have a text file of size 115MB. It consists of about 20 million words. I have to use the file as a word collection, and use it to search the occurrence of each user-given words from the collection. I am using this process as a small part in my project. I need a method for finding out the number of occurrence of given words in a faster and correct manner since i may use it in iterations. I am in need of suggestion about any API that i can make use or some other way that performs the task in a quicker manner. Any recommendations are appreciated.

    Read the article

  • Drawing of a huge model - How to regain performance?

    - by marc wellman
    I have a huge model I want to draw in my XNA application but because of its size I am experiencing a tremendous loss of performance. The model has about ~50 000 000 edges and has a size on disk of 205 MB in DirectX Format. Please don't ask whether this model has to be that big - yes it has! Is there a way to transfer the model directly to my GPU in order to let the GPU do the drawing like when transferring a VertexBuffer like this: graphicsDevice.Vertices[1].SetSource(_instanceBuffers[i], 0, _sizeofMatrix); because when I try to fill a vertexBuffer with all the vertices I am getting a OutOfMemoryException.

    Read the article

  • Box2D blocky map. Body, Fixtures a huge map and performance

    - by Solom
    Right now I'm still in the planning phase of a my very first game. I'm creating a "Minecraft"-like game in 2D that features blocks that can be destroyed as well as players moving around the map. For creating the map I chose a 2D-Array of Integers that represent the Block ID. For testing purposes I created a huge map (16348 * 256) and in my prototype that didn't use Box2D everything worked like a charm. I only rendered those blocks that where within the bounds of my camera and got 60 fps straight. The problem started when I decided to use an existing physics-solution rather than implementing my own one. What I had was basically simple hitboxes around the blocks and then I had to manually check if the player collided with any of those in his neighborhood. For more advanced physics as well as the collision detection I want to switch over to Box2D. The problem I have right now is ... how to go about the bodies? I mean, the blocks are of a static bodytype. They don't move on their own, they just are there to be collided with. But as far as I can see it, every block needs his own body with a rectangular fixture attached to it, so as to be destroyable. But for a huge map such as mine, this turns out to be a real performance bottle-neck. (In fact even a rather small map [compared to the other] of 1024*256 is unplayable.) I mean I create thousands of thousands of blocks. Even if I just render those that are in my immediate neighborhood there are hundreds of them and (at least with the debugRenderer) I drop to 1 fps really quickly (on my own "monster machine"). I thought about strategies like creating just one body, attaching multiple fixtures and only if a fixture got hit, separate it from the body, create a new one and destroy it, but this didn't turn out quite as successful as hoped. (In fact the core just dumps. Ah hello C! I really missed you :X) Here is the code: public class Box2DGameScreen implements Screen { private World world; private Box2DDebugRenderer debugRenderer; private OrthographicCamera camera; private final float TIMESTEP = 1 / 60f; // 1/60 of a second -> 1 frame per second private final int VELOCITYITERATIONS = 8; private final int POSITIONITERATIONS = 3; private Map map; private BodyDef blockBodyDef; private FixtureDef blockFixtureDef; private BodyDef groundDef; private Body ground; private PolygonShape rectangleShape; @Override public void show() { world = new World(new Vector2(0, -9.81f), true); debugRenderer = new Box2DDebugRenderer(); camera = new OrthographicCamera(); // Pixel:Meter = 16:1 // Body definition BodyDef ballDef = new BodyDef(); ballDef.type = BodyDef.BodyType.DynamicBody; ballDef.position.set(0, 1); // Fixture definition FixtureDef ballFixtureDef = new FixtureDef(); ballFixtureDef.shape = new CircleShape(); ballFixtureDef.shape.setRadius(.5f); // 0,5 meter ballFixtureDef.restitution = 0.75f; // between 0 (not jumping up at all) and 1 (jumping up the same amount as it fell down) ballFixtureDef.density = 2.5f; // kg / m² ballFixtureDef.friction = 0.25f; // between 0 (sliding like ice) and 1 (not sliding) // world.createBody(ballDef).createFixture(ballFixtureDef); groundDef = new BodyDef(); groundDef.type = BodyDef.BodyType.StaticBody; groundDef.position.set(0, 0); ground = world.createBody(groundDef); this.map = new Map(20, 20); rectangleShape = new PolygonShape(); // rectangleShape.setAsBox(1, 1); blockFixtureDef = new FixtureDef(); // blockFixtureDef.shape = rectangleShape; blockFixtureDef.restitution = 0.1f; blockFixtureDef.density = 10f; blockFixtureDef.friction = 0.9f; } @Override public void render(float delta) { Gdx.gl.glClearColor(1, 1, 1, 1); Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT); debugRenderer.render(world, camera.combined); drawMap(); world.step(TIMESTEP, VELOCITYITERATIONS, POSITIONITERATIONS); } private void drawMap() { for(int a = 0; a < map.getHeight(); a++) { /* if(camera.position.y - (camera.viewportHeight/2) > a) continue; if(camera.position.y - (camera.viewportHeight/2) < a) break; */ for(int b = 0; b < map.getWidth(); b++) { /* if(camera.position.x - (camera.viewportWidth/2) > b) continue; if(camera.position.x - (camera.viewportWidth/2) < b) break; */ /* blockBodyDef = new BodyDef(); blockBodyDef.type = BodyDef.BodyType.StaticBody; blockBodyDef.position.set(b, a); world.createBody(blockBodyDef).createFixture(blockFixtureDef); */ PolygonShape rectangleShape = new PolygonShape(); rectangleShape.setAsBox(1, 1, new Vector2(b, a), 0); blockFixtureDef.shape = rectangleShape; ground.createFixture(blockFixtureDef); rectangleShape.dispose(); } } } @Override public void resize(int width, int height) { camera.viewportWidth = width / 16; camera.viewportHeight = height / 16; camera.update(); } @Override public void hide() { dispose(); } @Override public void pause() { } @Override public void resume() { } @Override public void dispose() { world.dispose(); debugRenderer.dispose(); } } As you can see I'm facing multiple problems here. I'm not quite sure how to check for the bounds but also if the map is bigger than 24*24 like 1024*256 Java just crashes -.-. And with 24*24 I get like 9 fps. So I'm doing something really terrible here, it seems and I assume that there most be a (much more performant) way, even with Box2D's awesome physics. Any other ideas? Thanks in advance!

    Read the article

  • Getting out of web-development before I make a huge investment? [closed]

    - by zhenka
    I am still in college. I've been doing web-app development for about a year now and I'm growing to hate it more and more. The whole thing feels like a huge hack and I am loosing my interest in programming because of it. Too much time is spent on learning tricks and libraries in javascript/css/html and battling the statelessness of it all. I don't so much mind back-end development, I just hate ALL of the frontend technology stack. What attracted me to programming was software architecture. I love design patterns, clean code, etc... I just feel like there is a lot more to play with in that regard in other forms of development. Moreover I feel like by becoming a Java or .Net expert I will be able to do A LOT more in terms of career choices. I would be able to do anything from server-side to desktop to mobile, but ruby, javascript, php, css etc... makes me completely unemployable in any other sub-domain of SE. Plus most of the learning on web seems to be technology tricks rather then becoming a better developer and expanding one's mind. Should I get out of it and start coding side mobile projects before I invest too much into it? Does anyone have any advice or perhaps share this feeling and moved out of it successfully? Thanks!

    Read the article

  • One huge drive (network share) from many computers, with folder priority redundancy.

    - by Exception Duck
    Not sure if this exists, but I have a huge amount of data to store (about 5-50mb files) and as it is now, I have 5 computers, each with raid 5 providing about 6TB hard drive each. This is causing some problems with the software I am using (something home made) so I'm wondering, is there some software that I can install on all those computers that will mask it as one huge drive... Running windows on those computers, from Xp 64 bit to windows server 2008 I would also like to set a priority on each folder on the redundancy it has, some folders I can live without no online backup (I have a backup in a safe of that data) but some I need full online backup system if one hard drive fails. Something open source, as I try to use that as much as I can, but all ideas welcome.

    Read the article

< Previous Page | 3 4 5 6 7 8 9 10 11 12 13 14  | Next Page >