Search Results

Search found 6392 results on 256 pages for 'reduce duplicate'.

Page 209/256 | < Previous Page | 205 206 207 208 209 210 211 212 213 214 215 216  | Next Page >

• Altruistic network connection bandwidth estimation

  - by datenwolf

  Assume two peers Alice and Bob connected over a IP network. Alice and Bob are exchanging packets of lossy compressed data which are generated and to be consumes in real time (think a VoIP or video chat application). The service is designed to cope with as little bandwidth available, but relies on low latencies. Alice and Bob would mark their connection with an apropriate QoS profile. Alice and Bob want use a variable bitrate compression and would like to consume all of the leftover bandwidth available for the connection between them, but would voluntarily reduce the consumed bitrate depending on the state of the network. However they'd like to retain a stable link, i.e. avoid interruptions in their decoded data stream caused by congestion and the delay until the bandwidth got adjusted. However it is perfectly possible for them to loose a few packets. TL;DR: Alice and Bob want to implement a VoIP protocol from scratch, and are curious about bandwidth and congestion control. What papers and resources do you suggest for Alice and Bob to read? Mainly in the area of bandwidth estimation and congestion control.

  Read the article

• Which java technology should i use , if i want to build website with thousands of users

  - by Mirage

  Sorry for asking this question but i searched all java realted question but i got more confused. I am still not clear what should i start with 1)My main is build website in Java because someone told me that there are some machine learning or AI libraries available in java which i can use in java. so i decided to use that as it can reduce my work. Now i don't know java at all some people say following are used to build websites like 1)Servelts alone can build webiste 2)JSP alone can also build webiste 3)Struts 4)Spring with Hibernate 5)SEAM 6)JEE also for webistes Now guys i am confused where should i start from. Where does core JAVA fits in here. i was thinking of perhaps learning python as i know i have to learn Python only not its variations SO please guide me one thing which can solve my basic purpose of using ready made libraries of AI If i can do that with JSP then i will start with that. But i need to learn all of them then its better if can start learning python. I have 5 months to finish webiste. I really don't know why there are many braches of java for doing one thing Please help me.

  Read the article

• iPhone static library Clang/LLVM error: non_lazy_symbol_pointers

  - by Bekenn

  After several hours of experimentation, I've managed to reduce the problem to the following example (C++): extern "C" void foo(); struct test { ~test() { } }; void doTest() { test t; // 1 foo(); // 2 } This is being compiled for iOS devices in XCode 4.2, using the provided Clang compiler (Apple LLVM compiler 3.0) and the iOS 5.0 SDK. The project is configured as a Cocoa Touch Static Library, and "Enable Linking With Shared Libraries" is set to No because I'm building an AIR native extension. The function foo is defined in another external library. (In my actual project, this would be any of the C API functions defined by Adobe for use in AIR native extensions.) When attempting to compile this code, I get back the error: FATAL:incompatible feature used: section type non_lazy_symbol_pointers (must specify "-dynamic" to be used) clang: error: assembler command failed with exit code 1 (use -v to see invocation) The error goes away if I comment out either of the lines marked 1 or 2 above, or if I change the build setting "Enable Linking With Shared Libraries" to Yes. (However, if I change the build setting, then I get multiple ld warning: unexpected srelocation type 9 warnings when linking the library into the final project, and the application crashes when running on the device.) The build error also goes away if I remove the destructor from test. So: Is this a bug in Clang? Am I missing some all-important and undocumented build setting? The interaction between an externally-provided function and a struct with a destructor is very peculiar, to say the least.

  Read the article

• XSL Template - reducing replication

  - by Chris

  Hey Sorry about the extremely vague question title (any suggestions for improvements welcome) I have an XSL document that, currently, has lots of replication that I want to reduce. Here is the following XML snippet I am working with <Unit Status="alive"> I am currently using the following XSL to show images based on the status of the Unit <xsl:choose> <xsl:when test="@Status = 'alive'"> <img src="/web/resources/graphics/accept.png" /> </xsl:when> <xsl:when test="@Status = 'missingUnit'"> <img src="/web/resources/graphics/error.png" /> </xsl:when> <xsl:when test="@Status = 'missingNode'"> <img src="/web/resources/graphics/exclamation.png" /> </xsl:when> <xsl:when test="@Status = 'unexpectedUnit'"> <img src="/web/resources/graphics/exclamation_blue.png" /> </xsl:when> <xsl:otherwise> <!-- Should never get here --> <img src="/web/resources/graphics/delete.png" /> </xsl:otherwise> </xsl:choose> How do I put this code in a template or stylesheet that will allow me to stop copying / pasting this everywhere. Thanks

  Read the article

• How to terminate a particular Azure worker role instance

  - by Oliver Bock

  Background I am trying to work out the best structure for an Azure application. Each of my worker roles will spin up multiple long-running jobs. Over time I can transfer jobs from one instance to another by switching them to a readonly mode on the source instance, spinning them up on the target instance, and then spinning the original down on the source instance. If I have too many jobs then I can tell Azure to spin up extra role instance, and use them for new jobs. Conversely if my load drops (e.g. during the night) then I can consolidate outstanding jobs to a few machines and tell Azure to give me fewer instances. The trouble is that (as I understand it) Azure provides no mechanism to allow me to decide which instance to stop. Thus I cannot know which servers to consolidate onto, and some of my jobs will die when their instance stops, causing delays for users while I restart those jobs on surviving instances. Idea 1: I decide which instance to stop, and return from its Run(). I then tell Azure to reduce my instance count by one, and hope it concludes that the broken instance is a good candidate. Has anyone tried anything like this? Idea 2: I predefine a whole bunch of different worker roles, with identical contents. I can individually stop and start them by switching their instance count from zero to one, and back again. I think this idea would work, but I don't like it because it seems to go against the natural Azure way of doing things, and because it involves me in a lot of extra bookkeeping to manage the extra worker roles. Idea 3: Live with it. Any better ideas?

  Read the article

• Codeigniter common templates

  - by Darthg8r

  Let's say that I have a website that has 100 different pages. Each page uses a common header and footer. Inside the header is some dynamic content that comes from a database. I'd like to avoid having to have code in every single controller and action that passes this common code into the view. function index() { // It sucks to have to include this on every controller action. data['title'] = "This is the index page"; data['currentUserName'] = "John Smith"; $this->load->view("main_view", data); } function comments() { // It sucks to have to include this on every controller action. data['title'] = "Comment list"; data['currentUserName'] = "John Smith"; $this->load->view("comment_view", data); } I realize that I could refactor the code so that the common parts are in a single function and the function is called by the action. Doing so would reduce SOME of the pain, but it still doesn't feel right since I'd still have to make a call to that function every time. What's the correct way to handle this?

  Read the article

• Asynchronous event loop design and issues.

  - by Artyom

  Hello, I'm designing event loop for asynchronous socket IO using epoll/devpoll/kqueue/poll/select (including windows-select). I have two options of performing, IO operation: Non-blocking mode, poll on EAGAIN Set socket to non-blocking mode. Read/Write to socket. If operation succeeds, post completion notification to event loop. If I get EAGAIN, add socket to "select list" and poll socket. Polling mode: poll and then execute Add socket to select list and poll it. Wait for notification that it is readable writable read/write Post completion notification to event loop of sucseeds To me it looks like first would require less system calls when using in normal mode, especially for writing to socket (buffers are quite big). Also it looks like that it would be possible to reduce the overhead over number of "select" executions, especially it is nice when you do not have something that scales well as epoll/devpoll/kqueue. Questions: Are there any advantages of the second approach? Are there any portability issues with non-blocking operations on sockets/file descriptors over numerous operating systems: Linux, FreeBSD, Solaris, MacOSX, Windows. Notes: Please do not suggest using existing event-loop/socket-api implementations

  Read the article

• Reducing moire when downsampling halftone comic images.

  - by drawnonward

  How can I reduce moire effects when downsampling halftone comic book images during live zoom on an iPhone or iPad? I am writing a comic book viewer. It would be nice to provide higher resolution images and allow the user to zoom in while reading the comic book. However, my client is averse to moire effects and will not allow this feature if there are noticeable moire artifacts while zooming, which of course there are. Modifying the images to be less susceptible to moire would only work if the modifications were not perceptible. Blur was specifically prohibited, as is anything that removes the beloved halftone dots. The images are black and white halftone and line art. The originals are 600 dpi but what we ship with the application will be half that at best, so probably 2500 pixels or less tall. So what are my options? If I write a custom downsampling algorithm would it be fast enough for real time on these devices? Are there other tricks I can do? Would it work to just avoid the size ratios that have the most visual moire effects? As you zoom in an out, there are definitely peaks where the moire effects are worst. Is there a way to calculate what those points are and just zoom to a nearby scale that is not as bad? Any suggestions are welcome. I have very little experience with image and signal processing, but am enjoying the opportunity to learn. I know nothing of wavelets and acutance and other jargon, so please be verbose.

  Read the article

• Prevent "hg status" from showing everything under untracked directories

  - by Wei Hu

  I find the output of hg status too verbose for untracked directories. Suppose I have an empty repository that's managed by both git and hg. So there would be two directories, .git and .hg. The output of git status is: # Untracked files: # (use "git add <file>..." to include in what will be committed) # # .hg/ The output of hg status is: ? .git/HEAD ? .git/config ? .git/description ? .git/hooks/applypatch-msg.sample ? .git/hooks/commit-msg.sample ? .git/hooks/post-commit.sample ? .git/hooks/post-receive.sample ? .git/hooks/post-update.sample ? .git/hooks/pre-applypatch.sample ? .git/hooks/pre-commit.sample ? .git/hooks/pre-rebase.sample ? .git/hooks/prepare-commit-msg.sample ? .git/hooks/update.sample ? .git/info/exclude Is there a way to reduce its output to something like the following line? ? .git/

  Read the article

• Reducing Time Complexity in Java

  - by Koeneuze

  Right, this is from an older exam which i'm using to prepare my own exam in january. We are given the following method: public static void Oorspronkelijk() { String bs = "Dit is een boodschap aan de wereld"; int max = -1; char let = '*'; for (int i=0;i<bs.length();i++) { int tel = 1; for (int j=i+1;j<bs.length();j++) { if (bs.charAt(j) == bs.charAt(i)) tel++; } if (tel > max) { max = tel; let = bs.charAt(i); } } System.out.println(max + " keer " + let); } The questions are: what is the output? - Since the code is just an algorithm to determine the most occuring character, the output is "6 keer " (6 times space) What is the time complexity of this code? Fairly sure it's O(n²), unless someone thinks otherwise? Can you reduce the time complexity, and if so, how? Well, you can. I've received some help already and managed to get the following code: public static void Nieuw() { String bs = "Dit is een boodschap aan de wereld"; HashMap<Character, Integer> letters = new HashMap<Character, Integer>(); char max = bs.charAt(0); for (int i=0;i<bs.length();i++) { char let = bs.charAt(i); if(!letters.containsKey(let)) { letters.put(let,0); } int tel = letters.get(let)+1; letters.put(let,tel); if(letters.get(max)<tel) { max = let; } } System.out.println(letters.get(max) + " keer " + max); } However, I'm uncertain of the time complexity of this new code: Is it O(n) because you only use one for-loop, or does the fact we require the use of the HashMap's get methods make it O(n log n) ? And if someone knows an even better way of reducing the time complexity, please do tell! :)

  Read the article

• Is it legal to have different SOAP namespaces/versions between the request and response?

  - by Lord Torgamus

  THIRD EDIT: I now believe that this problem is due to a SOAP version mismatch (1.1 request, 1.2 response) masquerading as a namespace problem. Is it illegal to mix versions, or just bad style? Am I completely out of luck if I can't change my SOAP version or the service's? SECOND EDIT: Clarified error message, and tried to reduce "tl;dr"-ness. EDIT: [Link deleted, not related] Using soapUI, I'm sending a request that starts with: <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" ... and getting a response that starts with: <soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope" ... I know the service is getting the info, because processes down the line are working. However, my soapUI teststep fails. It has two active assertions: "SOAP Response" and "Not SOAP Fault." The failure marker is next to "SOAP Response," with the following message: line -1: Element Envelope@http://www.w3.org/2003/05/soap-envelope is not a valid Envelope@http://schemas.xmlsoap.org/soap/envelope/ document or a valid substitution. I have tried mixing and matching the namespace prefixes and schema URLs. Changing prefixes seems to have no effect; changing URLs causes a VersionMismatch error. I have also tried to use a substitution group, but that doesn't seem to be legal.

  Read the article

• What java web application framework to use?

  - by frohiky

  One of the main products of my company is an Oracle Forms (and Reports) based application, that "needs" to be re-written in another technology. Why? Users want a more rich interface experience, and we want, preferably, to reduce costs with an open source application server. For this (HUGE) project, we intend to use a java web application framework, keep these points in mind: We have: hundreds of tables on our database (the ORM must be as flexible as possible); some logic which is (and will still be) based on PL/SQL procedures/functions/packages; a lot of CRUDs (the application itself is of an considerable size); a demand to work with/generate documents and workflows; an intranet based user environment; We want: to offer a RIA interface experience; use (if possible) an open source app server; a rapid (as possible) development framework; a somewhat mature framework with a "wise" roadmap (and a considerable community support); a MVC approach combined with JS or GWT widgets (e.g. Vaadin or SmartGWT); Well, in the past weeks I've read a lot of posts, Q&As on stackoverflow, and much more: Wicket, JSF, Tapestry, Grails, GWT, Struts2, Play, Spring, Seam, Echo, .... the list goes on! I've even researched about Alfresco..! The obvious question: Which one to use? At this time, any insight, recommendation, shared experience, advice will be more then welcome!

  Read the article

• Python - wxPython custom button -> unbound method __init__()? what?

  - by Wallter

  After looking at questions like this it doesn't make sense that my __init__(self, parrent, id) would be throwing a unbound error? help? main.py import wx from customButton import customButton from wxPython.wx import * class MyFrame(wx.Frame): def __init__(self, parent, ID, title): wxFrame.__init__(self, parent, ID, title, wxDefaultPosition, wxSize(400, 400)) # Non-important code here... # This is the first declaration of the Button1 # This is also where the ERROR is thrown. # Omitting this line causes the window to execute # flawlessly. self.Button1 = customButton.__init__(self, parent, -1) # ... finishes in a basic wx.program style... customButton.py # I've included all of the code in the file # because have no idea where the bug/error happens import wx from wxPython.wx import * class Custom_Button(wx.PyControl): # The BMP's Over_bmp = None #wxEmptyBitmap(1,1,1) # When the mouse is over Norm_bmp = None #wxEmptyBitmap(1,1,1) # The normal BMP Push_bmp = None #wxEmptyBitmap(1,1,1) # The down BMP def __init__(self, parent, id, **kwargs): wx.PyControl.__init__(self,parent, id, **kwargs) # Set the BMP's to the ones given in the constructor #self.Over_bmp = wx.Bitmap(wx.Image(MOUSE_OVER_BMP, wx.BITMAP_TYPE_ANY).ConvertToBitmap()) #self.Norm_bmp = wx.Bitmap(wx.Image(NORM_BMP, wx.BITMAP_TYPE_ANY).ConvertToBitmap()) #self.Push_bmp = wx.Bitmap(wx.Image(PUSH_BMP, wx.BITMAP_TYPE_ANY).ConvertToBitmap()) #self.Pos_bmp = self.pos self.Bind(wx.EVT_LEFT_DOWN, self._onMouseDown) self.Bind(wx.EVT_LEFT_UP, self._onMouseUp) self.Bind(wx.EVT_LEAVE_WINDOW, self._onMouseLeave) self.Bind(wx.EVT_ENTER_WINDOW, self._onMouseEnter) self.Bind(wx.EVT_ERASE_BACKGROUND,self._onEraseBackground) self.Bind(wx.EVT_PAINT,self._onPaint) self._mouseIn = self._mouseDown = False def _onMouseEnter(self, event): self._mouseIn = True def _onMouseLeave(self, event): self._mouseIn = False def _onMouseDown(self, event): self._mouseDown = True def _onMouseUp(self, event): self._mouseDown = False self.sendButtonEvent() def sendButtonEvent(self): event = wx.CommandEvent(wx.wxEVT_COMMAND_BUTTON_CLICKED, self.GetId()) event.SetInt(0) event.SetEventObject(self) self.GetEventHandler().ProcessEvent(event) def _onEraseBackground(self,event): # reduce flicker pass def _onPaint(self, event): dc = wx.BufferedPaintDC(self) dc.SetFont(self.GetFont()) dc.SetBackground(wx.Brush(self.GetBackgroundColour())) dc.Clear() dc.DrawBitmap(self.Norm_bmp) # draw whatever you want to draw # draw glossy bitmaps e.g. dc.DrawBitmap if self._mouseIn: # If the Mouse is over the button dc.DrawBitmap(self, self.Mouse_over_bmp, self.Pos_bmp, useMask=False) if self._mouseDown: # If the Mouse clicks the button dc.DrawBitmap(self, self.Push_bmp, self.Pos_bmp, useMask=False)

  Read the article

• Wizard style navigation, dismissing a view and showing another one

  - by Robin Jamieson

  I'm making a set of screens similar to a wizard and I'd like to know how to make a view dismiss itself and its parent view and immediately show a 'DoneScreen' without worrying about resource leaks. My views look like the following: Base -> Level1 -> DoneScreen -> Level2 -> DoneScreen The Level1 controller is a navigation controller created with a view.xib and shown with [self presentModalViewController ...] by the Base controller. The Level1 controller is also responsible for creating the 'DoneScreen' which may be shown instead of the Level2 Screen based on a certain criteria. When the user taps a button on the screen, the Level1 controller instantiates the the Level2 controller and it displays it via [self.navigationController pushViewController ..] and Level2 controller's view has a 'Next' button. When the use hits the 'Next' button in the Level2 screen, I need to dismiss the current Level2's view as well as the Level1's view and display the 'DoneScreen', which would have been created and passed in to the Level2 controller from Level1. (partly to reduce code duplication, and partly to separate responsibilities among the controllers) In the Level2 controller, if I show the 'DoneScreen' first and dismiss itself with [self.navigationController popViewControllerAnimated:YES]; then the Level1 controller's modal view is still present above the 'Base' but under the Done screen. What's a good way to clear out all of these views except the Base and then show the 'DoneScreen'? Any good suggestions on how to get this done in a simple but elegant manner?

  Read the article

• Caching Authentication Data

  - by PartlyCloudy

  Hi, I'm currently implementing a REST web service using CouchDB and RESTlet. The RESTlet layer is mainly for authentication and some minor filtering of the JSON data served by CouchDB: Clients <= HTTP = [ RESTlet <= HTTP = CouchDB ] I'm using CouchDB also to store user login data, because I don't want to add an additional database server for that purpose. Thus, each request to my service causes two CouchDB requests conducted by RESTlet (auth data + "real" request). In order to keep the service as efficent as possible, I want to reduce the number of requests, in this case redundant requests for login data. My idea now is to provide a cache (i.e.LRU-Cache via LinkedHashMap) within my RESTlet application that caches login data, because HTTP caching will probabily not be enough. But how do I invalidate the cache data, once a user changes the password, for instance. Thanks to REST, the application might run on several servers in parallel, and I don't want to create a central instance just to cache login data. Currently, I save requested auth data in the cache and try to auth new requests by using them. If a authentication fails or there is now entry available, I'll dispatch a GET request to my CouchDB storage in order to obtain the actual auth data. So in a worst case, users that have changed their data will perhaps still be able to login with their old credentials. How can I deal with that? Or what is a good strategy to keep the cache(s) up-to-date in general? Thanks in advance.

  Read the article

• Condensing repeating JQuery code

  - by Craig Ward

  I have a page that has a large image on it with a number of thumbnails. When you mouse over a thumbnail the main image changes to the image you have just rolled your mouse over. The problem is the more thumbnails I have, the more repeated code I have. How could I reduce it? The Jquery code is as follows. <script type="text/javascript"> $('#thumb1') .mouseover(function(){ $('#big_img').fadeOut('slow', function(){ $('#big_img').attr('src', '0001.jpg'); $('#big_img').fadeIn('slow'); }); }); $('#thumb2') .mouseover(function(){ $('#big_img').fadeOut('slow', function(){ $('#big_img').attr('src', 'p_0002.jpg'); $('#big_img').fadeIn('slow'); }); }); $('#thumb3') .mouseover(function(){ $('#big_img').fadeOut('slow', function(){ $('#big_img').attr('src', '_img/p_0003.jpg'); $('#big_img').fadeIn('slow'); }); }); $('#thumb4') .mouseover(function(){ $('#big_img').fadeOut('slow', function(){ $('#big_img').attr('src', '0004.jpg'); $('#big_img').fadeIn('slow'); }); }); </script> #big_img = the ID of the full size image #thumb1, #thumb2, #thumb3, #thumb4 = The ID's of the thumbnails The main code for the page is PHP if that helps.

  Read the article

• sql-server performance optimization by removing print statements

  - by AG

  We're going through a round of sql-server stored procedure optimizations. The one recommendation we've found that clearly applies for us is 'SET NOCOUNT ON' at the top of each procedure. (Yes, I've seen the posts that point out issues with this depending on what client objects you run the stored procedures from but these are not issues for us.) So now I'm just trying to add in a bit of common sense. If the benefit of SET NOCOUNT ON is simply to reduce network traffic by some small amount every time, wouldn't it also make sense to turn off all the PRINT statements we have in the stored procedures that we only use for debugging? I can't see how it can hurt performance. OTOH, it's a bit of a hassle to implement due to the fact that some of the print statements are the only thing within else clauses, so you can't just always comment out the one line and be done. The change carries some amount of risk so I don't want to do it if it isn't going to actually help. But I don't see eliminating print statements mentioned anywhere in articles on optimization. Is that because it is so obvious no one bothers to mention it?

  Read the article

• Resizing image to fit its container

  - by jack moore

  #foo {width: 300px; height: 400px; overflow: hidden;} <div id="foo"></div> this.someimage = randomImageUrl; $("foo").innerHTML = "<img src='"+this.someimage+"' class='fooimage' />"; Now, the picture could be 200x200 or 1100x400 ... it's totally random. I can just stretch it (or reduce its size) by using: .fooimage {width: 300px; height: 400px;} $("foo").innerHTML = "<img src='"+this.someimage+"' class='fooimage' />"; Or I could test its size: imgHeight = newImg.height; imgWidth = newImg.width; ... and maybe something like this: if(imgHeight >400){ $("foo").innerHTML = "<img src='"+this.someimage+"' height='400' />"; } and browsers will do the rest. But there must be something better. Any ideas? Thanks! :)

  Read the article

• Java: Efficiency of the readLine method of the BufferedReader and possible alternatives

  - by Luhar

  We are working to reduce the latency and increase the performance of a process written in Java that consumes data (xml strings) from a socket via the readLine() method of the BufferedReader class. The data is delimited by the end of line separater (\n), and each line can be of a variable length (6KBits - 32KBits). Our code looks like: Socket sock = connection; InputStream in = sock.getInputStream(); BufferedReader inputReader = new BufferedReader(new InputStreamReader(in)); ... do { String input = inputReader.readLine(); // Executor call to parse the input thread in a seperate thread }while(true) So I have a couple of questions: Will the inputReader.readLine() method return as soon as it hits the \n character or will it wait till the buffer is full? Is there a faster of picking up data from the socket than using a BufferedReader? What happens when the size of the input string is smaller than the size of the Socket's receive buffer? What happens when the size of the input string is bigger than the size of the Socket's receive buffer? I am getting to grips (slowly) with Java's IO libraries, so any pointers are much appreciated. Thank you!

  Read the article

• Is it wise to use temporary tables?

  - by Industrial

  Hi guys, We have a mySQL database table for products. We are utilizing a cache layer to reduce database load, but we think that it's a good idea to minimize the actual data needed to be stored in the cache layer to speed up the application further. All the products in the database, that is visible to visitors have a price attached to them: The prices are stored in a different table, called prices . There are multiple price categories depending on which discount level each visitor (customer) applies to. From time to time, there are campaigns which means that a special price for each product is available. The special prices are stored in a table called specials. Is it a bad to make a temp table that binds the tables together? It would only have the neccessary information and would ofcourse be cached. -------------|-------------|------------ | productId | hasPrice | hasSpecial -------------|-------------|------------ 1 | 1 | 0 2 | 1 | 1 By doing such, it would be super easy to know if the specific product really has a price, without having to iterate through the complete prices or specials table each time a product should be listed or presented. Are temp tables a common thing for web applications or is it just bad design?

  Read the article

• File Storage for Web Applications: Filesystem vs DB vs NoSQL engines

  - by El Yobo

  I have a web application that stores a lot of user generated files. Currently these are all stored on the server filesystem, which has several downsides for me. When we move "folders" (as defined by our application) we also have to move the files on disk (although this is more due to strange design decisions on the part of the original developers than a requirement of storing things on the filesystem). It's hard to write tests for file system actions; I have a mock filesystem class that logs actions like move, delete etc, without performing them, which more or less does the job, but I don't have 100% confidence in the tests. I will be adding some other jobs which need to access the files from other service to perform additional tasks (e.g. indexing in Solr, generating thumbnails, movie format conversion), so I need to get at the files remotely. Doing this over network shares seems dodgy... Dealing with permissions on the filesystem as sometimes given us problems in the past, although now that we've moved to a pure Linux environment this should be less of an issue. What are the downsides of storing files as BLOBs in MySQL? I guess that it would massively increase the database size and reduce the effectiveness of caches, but are there other problems? Do the same problems exist with NoSQL systems like Cassandra? Does anyone have any other suggestions that might be appropriate?

  Read the article

• HDFS some datanodes of cluster are suddenly disconnected while reducers are running

  - by user1429825

  I have 8 slave computers and 1 master computer for running Hadoop (ver 0.21) some datanodes of cluster are suddenly disconnected while I was running MapReduce code on 10GB data After all mappers finished and around 80% of reducers was processed, randomly one or more datanode disconned from network. and then the other datanodes start to disappear from network even if I killed the MapReduce job when I found some datanode was disconnected. I've tried to change dfs.datanode.max.xcievers to 4096, turned off fire-walls of all computing node, disabled selinux and increased the number of file open limit to 20000 but they didn't work at all... anyone have a idea to solve this problem? followings are error log from mapreduce 12/06/01 12:31:29 INFO mapreduce.Job: Task Id : attempt_201206011227_0001_r_000006_0, Status : FAILED java.io.IOException: Bad connect ack with firstBadLink as ***.***.***.148:20010 at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:889) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:820) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:427) and followings are logs from datanode 2012-06-01 13:01:01,118 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-5549263231281364844_3453 src: /*.*.*.147:56205 dest: /*.*.*.142:20010 2012-06-01 13:01:01,136 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(*.*.*.142:20010, storageID=DS-1534489105-*.*.*.142-20010-1337757934836, infoPort=20075, ipcPort=20020) Starting thread to transfer block blk_-3849519151985279385_5906 to *.*.*.147:20010 2012-06-01 13:01:19,135 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(*.*.*.142:20010, storageID=DS-1534489105-*.*.*.142-20010-1337757934836, infoPort=20075, ipcPort=20020):Failed to transfer blk_-5797481564121417802_3453 to *.*.*.146:20010 got java.net.ConnectException: > Connection timed out at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:373) at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1257) at java.lang.Thread.run(Thread.java:722) 2012-06-01 13:06:20,342 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification succeeded for blk_6674438989226364081_3453 2012-06-01 13:09:01,781 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(*.*.*.142:20010, storageID=DS-1534489105-*.*.*.142-20010-1337757934836, infoPort=20075, ipcPort=20020):Failed to transfer blk_-3849519151985279385_5906 to *.*.*.147:20010 got java.net.SocketTimeoutException: 480000 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/*.*.*.142:60057 remote=/*.*.*.147:20010] at org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246) at org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:164) at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:203) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:388) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:476) at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1284) at java.lang.Thread.run(Thread.java:722) hdfs-site.xml <configuration> <property> <name>dfs.name.dir</name> <value>/home/hadoop/data/name</value> </property> <property> <name>dfs.data.dir</name> <value>/home/hadoop/data/hdfs1,/home/hadoop/data/hdfs2,/home/hadoop/data/hdfs3,/home/hadoop/data/hdfs4,/home/hadoop/data/hdfs5</value> </property> <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.datanode.max.xcievers</name> <value>4096</value> </property> <property> <name>dfs.http.address</name> <value>0.0.0.0:20070</value> <description>50070 The address and the base port where the dfs namenode web ui will listen on. If the port is 0 then the server will start on a free port. </description> </property> <property> <name>dfs.datanode.http.address</name> <value>0.0.0.0:20075</value> <description>50075 The datanode http server address and port. If the port is 0 then the server will start on a free port. </description> </property> <property> <name>dfs.secondary.http.address</name> <value>0.0.0.0:20090</value> <description>50090 The secondary namenode http server address and port. If the port is 0 then the server will start on a free port. </description> </property> <property> <name>dfs.datanode.address</name> <value>0.0.0.0:20010</value> <description>50010 The address where the datanode server will listen to. If the port is 0 then the server will start on a free port. </description> <property> <name>dfs.datanode.ipc.address</name> <value>0.0.0.0:20020</value> <description>50020 The datanode ipc server address and port. If the port is 0 then the server will start on a free port. </description> </property> <property> <name>dfs.datanode.https.address</name> <value>0.0.0.0:20475</value> </property> <property> <name>dfs.https.address</name> <value>0.0.0.0:20470</value> </property> </configuration> mapred-site.xml <configuration> <property> <name>mapred.job.tracker</name> <value>masternode:29001</value> </property> <property> <name>mapred.system.dir</name> <value>/home/hadoop/data/mapreduce/system</value> </property> <property> <name>mapred.local.dir</name> <value>/home/hadoop/data/mapreduce/local</value> </property> <property> <name>mapred.map.tasks</name> <value>32</value> <description> default number of map tasks per job.</description> </property> <property> <name>mapred.tasktracker.map.tasks.maximum</name> <value>4</value> </property> <property> <name>mapred.reduce.tasks</name> <value>8</value> <description> default number of reduce tasks per job.</description> </property> <property> <name>mapred.map.child.java.opts</name> <value>-Xmx2048M</value> </property> <property> <name>io.sort.mb</name> <value>500</value> </property> <property> <name>mapred.task.timeout</name> <value>1800000</value> <!-- 30 minutes --> </property> <property> <name>mapred.job.tracker.http.address</name> <value>0.0.0.0:20030</value> <description> 50030 The job tracker http server address and port the server will listen on. If the port is 0 then the server will start on a free port. </description> </property> <property> <name>mapred.task.tracker.http.address</name> <value>0.0.0.0:20060</value> <description> 50060 </property> </configuration>

  Read the article

• How to set ReceiveBufferSize for UDPClient? or Does it make sense to set? C#

  - by Jack

  Hello all. I am implementing a UDP data transfer thing. I have several questions about UDP buffer. I am using UDPClient to do the UDP send / receive. and my broadband bandwidth is 150KB/s (bytes/s, not bps). I send out a 500B datagram out to 27 hosts 27 hosts send back 10KB datagram back if they receive. So, I should receive 27 responses, right? however, I only get averagely 8 - 12 instead. I then tried to reduce the size of the response down to 500B, yes, I receive all. A thought of mine is that if all 27 hosts send back 10KB response at almost same time, the incoming traffic will be 270KB/s (likely), that exceeds my incoming bandwidth so loss happens. Am I right? But I think even if the incoming traffic exceeds the bandwidth, is the Windows supposed to put the datagram in the buffer and wait for receive? I then suspect that maybe the ReceiveBufferSize of my UdpClient is too small? by default, it is 8092B?? I don't know whether I am all right at these points. Please give me some help.

  Read the article

• Winsock WSAAsyncSelect sending without an infinite buffer

  - by Xexr

  Hi, This is more of a design question than a specific code question, I'm sure I am missing the obvious, I just need another set of eyes. I am writing a multi-client server based on WSAAsyncSelect, each connection is made into an object of a connection class I have written which contains associated settings and buffers etc. My question concerns FD_WRITE, I understand how it operates: One FD_WRITE is sent immediately after a connection is established. Thereafter, you should send until WSAEWOULDBLOCK is received at which point you store what is left to send in a buffer, and wait to be told that it is ok to send again. This is where I have a problem, how large do I make this holding buffer within each connections object? The amount of time until a new FD_WRITE is received is unknown, I could be attempting to send a lot of stuff during this period, all the time adding to my outgoing buffer. If I make the buffer dynamic, memory usage could spiral out of control if for whatever reason, I am unable to send() and reduce the buffer. So my question is how do you generally handle this situation? Note I am not talking about the network buffer itself which winsock uses, but one of my own creation used to "queue" up sends. Hope I explained that well enough, thanks all!

  Read the article

• Recycle Freed Objects

  - by uray

  suppose I need to allocate and delete object on heap frequently (of arbitrary size), is there any performance benefit if instead of deleting those objects, I will return it back to some "pool" to be reused later? would it give benefit by reduce heap allocation/deallocation?, or it will be slower compared to memory allocator performance, since the "pool" need to manage a dynamic collection of pointers. my use case: suppose I create a queue container based on linked list, and each node of that list are allocated on the heap, so every call to push() and pop() will allocate and deallocate that node: ` template <typename T> struct QueueNode { QueueNode<T>* next; T object; } template <typename T> class Queue { void push(T object) { QueueNode<T>* newNode = QueueNodePool<T>::get(); //get recycled node if(!newNode) { newNode = new QueueNode<T>(object); } // push newNode routine here.. } T pop() { //pop routine here... QueueNodePool<T>::store(unusedNode); //recycle node return unusedNode->object; } } `

  Read the article

< Previous Page | 205 206 207 208 209 210 211 212 213 214 215 216  | Next Page >