Search Results

Search found 13664 results on 547 pages for 'storage engine'.

Page 57/547 | < Previous Page | 53 54 55 56 57 58 59 60 61 62 63 64  | Next Page >

  • Recommendations for distributed processing/distributed storage systems

    - by Eddie
    At my organization we have a processing and storage system spread across two dozen linux machines that handles over a petabyte of data. The system right now is very ad-hoc; processing automation and data management is handled by a collection of large perl programs on independent machines. I am looking at distributed processing and storage systems to make it easier to maintain, evenly distribute load and data with replication, and grow in disk space and compute power. The system needs to be able to handle millions of files, varying in size between 50 megabytes to 50 gigabytes. Once created, the files will not be appended to, only replaced completely if need be. The files need to be accessible via HTTP for customer download. Right now, processing is automated by perl scripts (that I have complete control over) which call a series of other programs (that I don't have control over because they are closed source) that essentially transforms one data set into another. No data mining happening here. Here is a quick list of things I am looking for: Reliability: These data must be accessible over HTTP about 99% of the time so I need something that does data replication across the cluster. Scalability: I want to be able to add more processing power and storage easily and rebalance the data on across the cluster. Distributed processing: Easy and automatic job scheduling and load balancing that fits with processing workflow I briefly described above. Data location awareness: Not strictly required but desirable. Since data and processing will be on the same set of nodes I would like the job scheduler to schedule jobs on or close to the node that the data is actually on to cut down on network traffic. Here is what I've looked at so far: Storage Management: GlusterFS: Looks really nice and easy to use but doesn't seem to have a way to figure out what node(s) a file actually resides on to supply as a hint to the job scheduler. GPFS: Seems like the gold standard of clustered filesystems. Meets most of my requirements except, like glusterfs, data location awareness. Ceph: Seems way to immature right now. Distributed processing: Sun Grid Engine: I have a lot of experience with this and it's relatively easy to use (once it is configured properly that is). But Oracle got its icy grip around it and it no longer seems very desirable. Both: Hadoop/HDFS: At first glance it looked like hadoop was perfect for my situation. Distributed storage and job scheduling and it was the only thing I found that would give me the data location awareness that I wanted. But I don't like the namename being a single point of failure. Also, I'm not really sure if the MapReduce paradigm fits the type of processing workflow that I have. It seems like you need to write all your software specifically for MapReduce instead of just using Hadoop as a generic job scheduler. OpenStack: I've done some reading on this but I'm having trouble deciding if it fits well with my problem or not. Does anyone have opinions or recommendations for technologies that would fit my problem well? Any suggestions or advise would be greatly appreciated. Thanks!

    Read the article

  • How can I set up a load balancer to direct all Search Engine Bot traffic to one server?

    - by Ryan
    We have a simple load balancer set up on Rackspace to 3 web server nodes. After reviewing our traffic and expenses, the largest bandwidth hog is Google Bot. Since on Rackspace we pay for bandwidth by the byte, we'd like to direct all traffic from GoogleBot to another host (MediaTemple) with unlimited bandwidth. We think this would cut our hosting bill several thousand dollars a month. Is this possible? Advisable?

    Read the article

  • SQL SERVER – Subquery or Join – Various Options – SQL Server Engine Knows the Best – Part 2

    - by pinaldave
    This blog post is part 2 of the earlier written article SQL SERVER – Subquery or Join – Various Options – SQL Server Engine knows the Best by Paulo R. Pereira. Paulo has left excellent comment to earlier article once again proving the point that SQL Server Engine is smart enough to figure out the best plan itself and uses the same for the query. Let us go over his comment as he has posted. “I think IN or EXISTS is the best choice, because there is a little difference between ‘Merge Join’ of query with JOIN (Inner Join) and the others options (Left Semi Join), and JOIN can give more results than IN or EXISTS if the relationship is 1:0..N and not 1:0..1. And if I try use NOT IN and NOT EXISTS the query plan is different from LEFT JOIN too (Left Anti Semi Join vs. Left Outer Join + Filter). So, I found a case where EXISTS has a different query plan than IN or ANY/SOME:” USE AdventureWorks GO -- use of SOME SELECT * FROM HumanResources.Employee E WHERE E.EmployeeID = SOME ( SELECT EA.EmployeeID FROM HumanResources.EmployeeAddress EA UNION ALL SELECT EA.EmployeeID FROM HumanResources.EmployeeDepartmentHistory EA ) -- use of IN SELECT * FROM HumanResources.Employee E WHERE E.EmployeeID IN ( SELECT EA.EmployeeID FROM HumanResources.EmployeeAddress EA UNION ALL SELECT EA.EmployeeID FROM HumanResources.EmployeeDepartmentHistory EA ) -- use of EXISTS SELECT * FROM HumanResources.Employee E WHERE EXISTS ( SELECT EA.EmployeeID FROM HumanResources.EmployeeAddress EA UNION ALL SELECT EA.EmployeeID FROM HumanResources.EmployeeDepartmentHistory EA ) When looked into execution plan of the queries listed above indeed we do get different plans for queries and SQL Server Engines creates the best (least cost) plan for each query. Click on image to see larger images. Thanks Paulo for your wonderful contribution. Reference : Pinal Dave (http://blog.SQLAuthority.com) Filed under: Pinal Dave, Readers Contribution, SQL, SQL Authority, SQL Joins, SQL Optimization, SQL Performance, SQL Query, SQL Scripts, SQL Server, SQL Tips and Tricks, T SQL, Technology

    Read the article

  • Using Oracle Data in the Business Rules Engine

    - by Christopher House
    Yesterday I started working on some new functionality that I had planned to implement using the Business Rules Engine.  As I got further into it, I realized that some of my rules were going to need to reference some data that resides in an Oracle database.  I knew the Business Rules Composer supports using DataConnections and TypedDataTables, but I’d never used this functionality myself, so I wasn’t so sure how it would work with Oracle.  As it turns out, it’s very do-able, there’s just little hoop you need to jump through. I fired up BRC and my suspicions were quickly confirmed.  BRC only recognizes SQL Server databases when it comes to editing rules.  Not letting that deter me, I decided to see if I could “trick” BRE into using Oracle data. On my local SQL server, I created a new database and in that database, created a table that matched the schema of the table I wanted to use in the Oracle database.  I then set about creating my rules, referencing the new SQL Server database everywhere I wanted to use Oracle data.  Finally, I created a new class library and added a class that implements Microsoft.RuleEngine.IFactRetriever.  In that class, I added the necessary code to get a DataSet from the Oracle server, wrap it in a TypedDataTable and assert it into the rule engine.  It’s worth pointing out that in my IFactRetriever class, I made sure to set my DataSet name to the name of the database I’d referenced in the BRC and the DataTable’s name to the name of the table that I’d referenced in the BRC. After gac’ing the new class library and deploying my policy, I tested and everything worked as expected.

    Read the article

  • Just released: a new SEO extension for the ASP.NET MVC routing engine

    - by efran.cobisi
    Dear users,after several months of hard work, we are proud to announce to the world that Cobisi's new SEO routing engine for ASP.NET MVC has been officially released! We even provide a free edition which comes at no cost, so this is something you can't really miss if you are a serious ASP.NET developer. ;)SEO routes for ASP.NET MVCCobisi SEO Extensions - this is the name of the product - is an advanced tool for software developers that allows to optimize ASP.NET MVC web applications and sites for search engines. It comes with a powerful routing engine, which extends the standard ASP.NET routing module to provide a much more flexible way to define search optimized routes, and a complete set of classes that make customizing the entire routing infrastructure very easy and cool.In its simplest form, defining a route for an MVC action is just a matter of decorating the method with the [Route("...")] attribute and specifying the desired URL. The library will take care of the rest and set up the route accordingly; while coding routes this way, Cobisi SEO Extensions also shows how the final routes will be, without leaving the Visual Studio IDE!Manage MVC routes with easeIn fact, Cobisi SEO Extensions integrates with the Visual Studio IDE to offer a large set of time-saving improvements targeted at ASP.NET developers. A new tool window, for example, allows to easily browse among the routes exposed by your applications, being them standard ASP.NET routes, MVC specific routes or SEO routes. The routes can be easily filtered on the fly, to ease finding the ones you are interested in. Double clicking a SEO route will even open the related ASP.NET MVC controller, at the beginning of the specified action method.In addition to that, Cobisi SEO Extensions allows to easily understand how each SEO route is composed by showing the routing model details directly in the IDE, beneath each MVC action route.Furthermore, Cobisi SEO Extensions helps developers to easily recognize which class is an MVC controller and which methods is an MVC action by drawing a special dashed underline mark under each items of these categories.Developers, developers, developers, ...We are really eager to receive your feedback and suggestions - please feel free to ping us with your comments! Thank you! Cheers! -- Efran Cobisi Cobisi lead developer Microsoft MVP, MCSD, MCAD, MCTS: SQL Server 2005, MCP

    Read the article

  • Continuous Physics Engine's Collision Detection Techniques

    - by Griffin
    I'm working on a purely continuous physics engine, and I need to choose algorithms for broad and narrow phase collision detection. "Purely continuous" means I never do intersection tests, but instead want to find ways to catch every collision before it happens, and put each into "planned collisions" stack that is ordered by TOI. Broad Phase The only continuous broad-phase method I can think of is encasing each body in a circle and testing if each circle will ever overlap another. This seems horribly inefficient however, and lacks any culling. I have no idea what continuous analogs might exist for today's discrete collision culling methods such as quad-trees either. How might I go about preventing inappropriate and pointless broad test's such as a discrete engine does? Narrow Phase I've managed to adapt the narrow SAT to a continuous check rather than discrete, but I'm sure there's other better algorithms out there in papers or sites you guys might have come across. What various fast or accurate algorithm's do you suggest I use and what are the advantages / disatvantages of each? Final Note: I say techniques and not algorithms because I have not yet decided on how I will store different polygons which might be concave, convex, round, or even have holes. I plan to make a decision on this based on what the algorithm requires (for instance if I choose an algorithm that breaks down a polygon into triangles or convex shapes I will simply store the polygon data in this form).

    Read the article

  • How does a segment based rendering engine work?

    - by Calmarius
    As far as I know Descent was one of the first games that featured a fully 3D environment, and it used a segment based rendering engine. Its levels are built from cubic segments (these cubes may be deformed as long as it remains convex and sides remain roughly flat). These cubes are connected by their sides. The connected sides are traversable (maybe doors or grids can be placed on these sides), while the unconnected sides are not traversable walls. So the game is played inside of this complex. Descent was software rendered and it had to be very fast, to be playable on those 10-100MHz processors of that age. Some latter levels of the game are huge and contain thousands of segments, but these levels are still rendered reasonably fast. So I think they tried to minimize the amount of cubes rendered somehow. How to choose which cubes to render for a given location? As far as I know they used a kind of portal rendering, but I couldn't find what was the technique used in this particular kind of engine. I think the fact that the levels are built from convex quadrilateral hexahedrons can be exploited.

    Read the article

  • Engine for 2D Top-Down Physics-Based Skeletal Animation

    - by RylandAlmanza
    I just watched at the Sui Generis video, and was completely amazed. Specifically, the part where the big troll thing is beating up the player with his flail. This got me really excited, and I would like to try implementing something like this in a 2D Top-Down format. Something like this. That atloria example seems simple enough, but it's not exactly what I'm looking to make. I think atloria is using predefined animations, where as I would like to make something more physics-based like the Sui Generis engine does. So, I'm wondering what physics engines might work for something like this, and if I'd need to implement my own skeletal system, or if I could just use "joints" and such from the engine. The only experience I have in terms of physics engines is Box2D, which I've heard shouldn't be used for top-down settings, and I can think of a few reasons it wouldn't work out well. One of those reasons being gravity. In box 2D, gravity pulls towards a side of the screen (usually the bottom.) I wouldn't want my player's forearms constantly being pulled to one side. :) Also should mention that the programming language doesn't matter all that much to me. I'm currently playing with HTML5 stuff, though. :) Thanks in advance!

    Read the article

  • XML Rules Engine and Validation Tutorial with NIEM

    - by drrwebber
    Our new XML Validation Framework tutorial video is now available. See how to easily integrate code-free adaptive XML validation services into your web services using the Java CAMV validation engine. CAMV allows you to build fault tolerant content checking with XPath that optionally use SQL data lookups. This can provide warnings as well as error conditions to tailor your validation layer to exactly meet your business application needs. Also available is developing test suites using Apache ANT scripting of validations.  This allows a community to share sets of conformance checking test and tools . On the technical XML side the video introduces XPath validation rules and illustrates and the concepts of XML content and structure validation. CAM validation templates allow contextual parameter driven dynamic validation services to be implemented compared to using a static and brittle XSD schema approach.The SQL table lookup and code list validation are discussed and examples presented.Features are highlighted along with a demonstration of the interactive generation of actual live XML data from a SQL data store and then validation processing complete with errors and warnings detection.The presentation provides a primer for developing web service XML validation and integration into a SOA approach along with examples and resources. Also alignment with the NIEM IEPD process for interoperable information exchanges is discussed along with NIEM rules services.The CAMV engine is a high performance scalable Java component for rapidly implementing code-free validation services and methods. CAMV is a next generation WYSIWYG approach that builds from older Schematron coding based interpretative runtime tools and provides a simpler declarative metaphor for rules definition. See: http://www.youtube.com/user/TheCAMeditor

    Read the article

  • Using IsolatedStorage on a IIS server

    - by JoeBilly
    I'am a bit confusing about the use of Isolated Storage on an IIS server. I understand the goal of Isolated Storage : provides a safe place to store data with no worry about how and where is this place. Since Isolated Storage has a by-user and by-assembly approach, I'am not to wild about using it on a IIS server where applications have almost their own identity. I haven't really seen the interest of impersonating a web application and almost never seen impersonated web applications myself but this is my point of view. Using Isolated Storage on a server mean : Using Isolated stores in \Documents and Settings\<user>\ Which mean \Documents and Settings\Default User\ when the application pool is owned by Local System or Network Services I guess Which also mean Write rights on this folder for Local System or Network Services Using of impersonation Regarding a web application (logic), these ideas are confusing me... Document and Settings ? Default User ? Enable impersonation just for storage ? No control about storage on server ? Uh ? And then I'am a front of a dilema : use System.IO.Packaging (with Isolated Storage inside) on web applications or find an alternative ? Am I wrong in my approach ? Did I miss something ? Any point of view is appreciated and an explanation about the Isolated Storage with IIS philosophy could be an anwser. Thanks !

    Read the article

  • Initial Review - Mastering Unreal Technology I: Introduction to Level Design with Unreal Engine 3

    - by Matt Christian
    Recently I purchased 3 large volumes on using the Unreal 3 Engine to create levels and custom games.  This past weekend I cracked the spine of the first and started reading.  Here are my early impressions (I'm ~250 pages into it, with appendices it's about 900). Pros Interestingly, the book starts with an overview of the Unreal engines leading up to Unreal 3 (including Gears of War) and follows with some discussion on planning a mod and what goes into the game development process.  This is nice for an intro to the book and is much preferred rather than a simple chapter detailing what is on the included CD, how to install and setup UnrealEd, etc...  While the chapter on Unreal history and planning can be considered 'fluff', it's much less 'fluffy' than most books provide. I need to mention one thing here that is pretty crucial in the way I'm going to continue reviewing this book.  Most technical books like this are used as a shelf reference; as a thick volume you use for looking up techniques every now and again.  Even so, I prefer reading from cover to cover, including chapters I may already be knowledgable on (I'm sure this is typical for most people).  If there was a chapter on installing UnrealEd (the previously mentioned 'fluff'), I would probably force myself to read it, even though I've installed the game and engine multiple times on different systems. Chapter 3 is where we really get to the introduction piece of UnrealEd, creating your first basic level.  This large chapter details creating two small rooms, adding static meshes, adding lighting, creating and adding particle emitters, creating a door that animates with Unreal Matinee and Kismet, static meshes with physics, and other little additions to make your level look less beginner.  This really is a chapter that overviews the entire rest of the book, as each chapter following details the creation and intermediate usages of Static Meshes, Materials, Lighting, etc... One other very nice part to this book is the way the tutorials are setup.  Each tutorial builds off the previous and all are step-by-step.  If you haven't completed one yet, you can find all the starting files on the CD that comes with the book. Cons While the description of the overview chapter (Chapter 3) is fresh in your mind, let me start the cons by saying this chapter is setup extremely confusing for the noob.  At one point, you end up creating a door mesh and setting it up as a InteropMesh so that it is ready to be animated, only to switch to particles and spend a good portion of time working on a different piece of the level.  Yes, this is actually how I develop my levels (jumping back and forth), though it's very odd for a book to jump out of sequence. The next item might be a positive or a negative depending on your skill level with UnrealEd.  Most of the introduction to the editor layout is found in one of the Appendices instead of before Chapter 3.  For new readers, this might lead to confusion as Appendix A would typically be read between Chapter 2 and 3.  However, this is a positive for those with some experience in UnrealEd as they don't have to force themselves through a 'learn every editor button' chapter.  I'm listing this in the Cons section as the book is 'Introduction to...' and is probably going to be directed toward a lot of very beginner developers. Finally, there's a lack of general description to a lot of the underlying engine and what each piece in UnrealEd is or does.  Sometimes you'll be performing Tutorial after Tutorial with barely a paragraph in between describing ANY of what you've just done.  Tutorial 1.1 Step 6 says to press Button X, so you do.  But why?  This is in part a problem with the structure of the tutorials rather than the content of the book.  Since the tutorials are so focused on a step-by-step (or procedural) description of a process, you learn the process and not why you're doing that.  For example, you might learn how to size a material to a surface, but will only learn what buttons to press and not what each one does. This becomes extremely apparent in the chapter on Static Meshes as most of the chapter is spent in 3D Studio Max.  Since there are books on 3DSM and modelling, the book really only tells you the steps and says to go grab a book on modelling if you're really interested in 3DSM.  Again, I've learned the process to develop my own meshes in 3DSM, but I don't know the why behind the steps. Conclusion So far the book is very good though I would have a hard time recommending it to a complete beginner.  I would suggest anyone looking at this book (obviously including the other 2, more advanced volumes) to pick up a copy of UDK or Unreal 3 (available online or via download services such as Steam) and watch some online tutorials and play with it first.  You'll find plenty of online videos available that were created by the authors and may suit as a better introduction to the editor.

    Read the article

  • What is the performance hit of enabling sessions on Google App Engine?

    - by Spines
    What is the performance hit of enabling sessions on the Google App Engine? I just turned on <sessions-enabled>true</sessions-enabled> in my Google App Engine app and now my requests are consistently using 100 more ms of CPU time than before I enabled it. It also makes the user wait an additional 100ms for the server to respond on each request. This seems to be quite a significant cost, I'm not even calling getSession or using it in any way yet and it still adds this extra latency. Is there something I can do to speed this up?

    Read the article

  • Spark View Engine: How to set default master page name?

    - by Dave
    I use Spark View Engine with nested master pages. I have Application.spark which defines the basic layout of the website. Then there are several other masters which themselves use Application.spark as master page (Default.spark, SinlgeColumn.spark, Gallery.spark, ...) If no master page is specified in a view file, then automatically Application.spark is choosen by the Spark View Engine. Since almost all my pages use "Default.spark" as master, is there a way to configure this globally? The other possibilities would be: Set the master in each spark file individually <use master="Default" />. But that's really annoying. Rename my master files (Default.spark <- Application.spark) but that really doesn't make any sense in naming.

    Read the article

  • Google Custom Search Engine not giving the expected search result.

    - by iecut
    Hi, I have been trying to create a new google custom search engine, but when I try some query,the search engine it is not giving me the expected search result.On some queries it is working fine, but on other queries, it says"no result". I tried adding the URL of the website that I wanted to search for, but there are certain pages and keywords that are not coming up in the search result when I try to search for the keyword of that page. I tired adding both the main page URL and the URL of the sub page that I want to search for, but nothing is working. There are some sub pages to the main URL that are coming in the search result.

    Read the article

  • How do I change the default format of log messages in python app engine?

    - by dazed-n-confused
    I would like to log the module and classname by default in log messages from my request handlers. The usual way to do this seems to be to set a custom format string by calling logging.basicConfig, but this can only be called once and has already been called by the time my code runs. Another method is to create a new log Handler which can be passed a new log Formatter, but this doesn't seem right as I want to use the existing log handler that App Engine has installed. What is the right way to have extra information added to all log messages in python App Engine, but otherwise use the existing log format and sink?

    Read the article

  • Adding a subdomain to my google app engine project?

    - by user246114
    Hi, I created a google app engine project. I just successfully mapped it to a new domain. The name of my project is "grape". So by default, it is published at http://www.grape.appspot.com. I mapped it to http://www.grape.com, which is terrific. Now I'd like to create a new app engine project, and have it mapped to: http://api.grape.com how do I go about doing this? I think it is possible, I'm just not sure where I would do this mapping? Since I own grape.com, I am hoping I can map a new project to t. The basic idea was to have one project which is responsible for the UI stuff, then a second project responsible just for a public api, which would be great, Thanks

    Read the article

  • How to get a physics engine like Nape working?

    - by Glacius
    Introduction: I think Nape is a relatively new engine so some of you may not know it. It's supposedly faster than box2d and I like that there is decent documentation. Here's the site: http://code.google.com/p/nape/ I'm relatively new to programming. I am decent at AS3's basic functionality, but every time I try to implement some kind of engine or framework I can't even seem to get it to work. With Nape I feel I got a little further than before but I still got stuck. My problem: I'm using Adobe CS5, I managed to import the SWC file like described here. Next I tried to copy the source of one of the demo's like this one and get it to work but I keep getting errors. I made a new class file, copied the demo source to it, and tried to add it to the stage. My stage code basically looks like this: import flash.Boot; // these 2 lines are as described in the tutorial new Boot(); var demo = new Main(); // these 2 are me guessing what I'm supposed to do addChild(demo); Well, it seems the source code is not even being recognized by flash as a valid class file. I tried editing it, but even if I get it recognized (give a package name and add curly brackets) but I still get a bunch of errors. Is it psuedo code or something? What is going on? My goal: I can imagine I'm going about this the wrong way. So let me explain what I'm trying to achieve. I basically want to learn how to use the engine by starting from a simple basic example that I can edit and mess around with. If I can't even get a working example then I'm unable to learn anything. Preferably I don't want to start using something like FlashDevelop (as I'd have to learn how to use the program) but if it can't be helped then I can give it a try. Thank you.

    Read the article

  • Comments on this assumption about running on dev server vs a real instance in app engine (python)?

    - by Jacob Oscarson
    Hello app engineers! I'm on an app engine project where I'd like to put in a link to a Javascript test runner that I'd like to only exist when running the development server. I've made some experiments on a local shell with configuration loaded using the technique found in NoseGAE versus live on the 'App Engine Console' [1] and it looks to me like a distinction btw real instance and dev server is the presence of the module google.appengine.tools. Which lead me to this utility function: def is_dev(): """ Tells us if we're running under the development server or not. :return: ``True`` if the code is running under the development server. """ try: from google.appengine import tools return True except ImportError: return False The question (finally!) would be: is this a bad idea? And in that case, can anyone suggest a better approach? [1] http://con.appspot.com/console/ (try it! very handy indeed)

    Read the article

  • Google App Engine says "Must authenticate first." while trying to deploy any app

    - by Oleksandr Bolotov
    Google App Engine says "Must authenticate first." while trying to deploy any app: me@myhost /opt/google_appengine $ python appcfg.py update ~/sda2/workspace/lyapapam/ Application: lyapapam; version: 1. Server: appengine.google.com. Scanning files on local disk. Scanned 500 files. Scanned 1000 files. Initiating update. Email: <my_email_was_here>@gmail.com Password for <my_email_was_here>@gmail.com: Error 401: --- begin server output --- Must authenticate first. --- end server output --- We are getting this message with any appliation and under any developer account avialable to us That's what we have installed: App Engine SDK - 1.3.2 PIL - 1.1.7 Python - 2.5.5 pip - 0.6.3 ssl - 1.15 wsgiref - 0.1.2 How can I fix it? Is it well known problem?

    Read the article

  • How do I dynamically import a module in App Engine?

    - by Scott Ferguson
    I'm trying to dynamically load a class from a specific module (called 'commands') and the code runs totally cool on my local setup running from a local Django server. This bombs out though when I deploy to Google App Engine. I've tried adding the commands module's parent module to the import as well with no avail (on either setup in that case). Here's the code: mod = __import__('commands.%s' % command, globals(), locals(), [command]) return getattr(mod, command) App Engine just throws an ImportError whenever it hits this. And the clarify, it doesn't bomb out on the commands module. If I have a command like 'commands.cat' it can't find 'cat'.

    Read the article

< Previous Page | 53 54 55 56 57 58 59 60 61 62 63 64  | Next Page >