Search Results

Search found 94348 results on 3774 pages for 'system data oracleclient'.

Page 11/3774 | < Previous Page | 7 8 9 10 11 12 13 14 15 16 17 18  | Next Page >

• Implementing a generic repository for WCF data services

  - by cibrax

  The repository implementation I am going to discuss here is not exactly what someone would call repository in terms of DDD, but it is an abstraction layer that becomes handy at the moment of unit testing the code around this repository. In other words, you can easily create a mock to replace the real repository implementation. The WCF Data Services update for .NET 3.5 introduced a nice feature to support two way data bindings, which is very helpful for developing WPF or Silverlight based application but also for implementing the repository I am going to talk about. As part of this feature, the WCF Data Services Client library introduced a new collection DataServiceCollection<T> that implements INotifyPropertyChanged to notify the data context (DataServiceContext) about any change in the association links. This means that it is not longer necessary to manually set or remove the links in the data context when an item is added or removed from a collection. Before having this new collection, you basically used the following code to add a new item to a collection. Order order = new Order {   Name = "Foo" }; OrderItem item = new OrderItem {   Name = "bar",   UnitPrice = 10,   Qty = 1 }; var context = new OrderContext(); context.AddToOrders(order); context.AddToOrderItems(item); context.SetLink(item, "Order", order); context.SaveChanges(); Now, thanks to this new collection, everything is much simpler and similar to what you have in other ORMs like Entity Framework or L2S. Order order = new Order {   Name = "Foo" }; OrderItem item = new OrderItem {   Name = "bar",   UnitPrice = 10,   Qty = 1 }; order.Items.Add(item); var context = new OrderContext(); context.AddToOrders(order); context.SaveChanges(); In order to use this new feature, you first need to enable V2 in the data service, and then use some specific arguments in the datasvcutil tool (You can find more information about this new feature and how to use it in this post). DataSvcUtil /uri:"http://localhost:3655/MyDataService.svc/" /out:Reference.cs /dataservicecollection /version:2.0 Once you use those two arguments, the generated proxy classes will use DataServiceCollection<T> rather than a simple ObjectCollection<T>, which was the default collection in V1. There are some aspects that you need to know to use this feature correctly. 1. All the entities retrieved directly from the data context with a query track the changes and report those to the data context automatically. 2. A entity created with “new” does not track any change in the properties or associations. In order to enable change tracking in this entity, you need to do the following trick. public Order CreateOrder() {   var collection = new DataServiceCollection<Order>(this.context);   var order = new Order();   collection.Add(order);   return order; } You basically need to create a collection, and add the entity to that collection with the “Add” method to enable change tracking on that entity. 3. If you need to attach an existing entity (For example, if you created the entity with the “new” operator rather than retrieving it from the data context with a query) to a data context for tracking changes, you can use the “Load” method in the DataServiceCollection. var order = new Order {   Id = 1 }; var collection = new DataServiceCollection<Order>(this.context); collection.Load(order); In this case, the order with Id = 1 must exist on the data source exposed by the Data service. Otherwise, you will get an error because the entity did not exist. These cool extensions methods discussed by Stuart Leeks in this post to replace all the magic strings in the “Expand” operation with Expression Trees represent another feature I am going to use to implement this generic repository. Thanks to these extension methods, you could replace the following query with magic strings by a piece of code that only uses expressions. Magic strings, var customers = dataContext.Customers .Expand("Orders")         .Expand("Orders/Items") Expressions, var customers = dataContext.Customers .Expand(c => c.Orders.SubExpand(o => o.Items)) That query basically returns all the customers with their orders and order items. Ok, now that we have the automatic change tracking support and the expression support for explicitly loading entity associations, we are ready to create the repository. The interface for this repository looks like this,public interface IRepository { T Create<T>() where T : new(); void Update<T>(T entity); void Delete<T>(T entity); IQueryable<T> RetrieveAll<T>(params Expression<Func<T, object>>[] eagerProperties); IQueryable<T> Retrieve<T>(Expression<Func<T, bool>> predicate, params Expression<Func<T, object>>[] eagerProperties); void Attach<T>(T entity); void SaveChanges(); } The Retrieve and RetrieveAll methods are used to execute queries against the data service context. While both methods receive an array of expressions to load associations explicitly, only the Retrieve method receives a predicate representing the “where” clause. The following code represents the final implementation of this repository.public class DataServiceRepository: IRepository { ResourceRepositoryContext context; public DataServiceRepository() : this (new DataServiceContext()) { } public DataServiceRepository(DataServiceContext context) { this.context = context; } private static string ResolveEntitySet(Type type) { var entitySetAttribute = (EntitySetAttribute)type.GetCustomAttributes(typeof(EntitySetAttribute), true).FirstOrDefault(); if (entitySetAttribute != null) return entitySetAttribute.EntitySet; return null; } public T Create<T>() where T : new() { var collection = new DataServiceCollection<T>(this.context); var entity = new T(); collection.Add(entity); return entity; } public void Update<T>(T entity) { this.context.UpdateObject(entity); } public void Delete<T>(T entity) { this.context.DeleteObject(entity); } public void Attach<T>(T entity) { var collection = new DataServiceCollection<T>(this.context); collection.Load(entity); } public IQueryable<T> Retrieve<T>(Expression<Func<T, bool>> predicate, params Expression<Func<T, object>>[] eagerProperties) { var entitySet = ResolveEntitySet(typeof(T)); var query = context.CreateQuery<T>(entitySet); foreach (var e in eagerProperties) { query = query.Expand(e); } return query.Where(predicate); } public IQueryable<T> RetrieveAll<T>(params Expression<Func<T, object>>[] eagerProperties) { var entitySet = ResolveEntitySet(typeof(T)); var query = context.CreateQuery<T>(entitySet); foreach (var e in eagerProperties) { query = query.Expand(e); } return query; } public void SaveChanges() { this.context.SaveChanges(SaveChangesOptions.Batch); } } For instance, you can use the following code to retrieve customers with First name equal to “John”, and all their orders in a single call. repository.Retrieve<Customer>(    c => c.FirstName == “John”, //Where    c => c.Orders.SubExpand(o => o.Items)); In case, you want to have some pre-defined queries that you are going to use across several places, you can put them in an specific class. public static class CustomerQueries {   public static Expression<Func<Customer, bool>> LastNameEqualsTo(string lastName)   {     return c => c.LastName == lastName;   } } And then, use it with the repository. repository.Retrieve<Customer>(    CustomerQueries.LastNameEqualsTo("foo"),    c => c.Orders.SubExpand(o => o.Items));

  Read the article

• data structure for counting frequencies in a database table-like format

  - by user373312

  i was wondering if there is a data structure optimized to count frequencies against data that is stored in a database table-like format. for example, the data comes in a (comma) delimited format below. col1, col2, col3 x, a, green x, b, blue ... y, c, green now i simply want to count the frequency of col1=x or col1=x and col2=green. i have been storing the data in a database table, but in my profiling and from empirical observation, database connection is the bottle-neck. i have tried using in-memory database solutions too, and that works quite well; the only problem is memory requirements and quirky init/destroy calls. also, i work mainly with java, but have experience with .net, and was wondering if there was any api to work with "tabular" data in a linq way using java. any help is appreciated.

  Read the article

• Oracle Big Data Software Downloads

  - by Mike.Hallett(at)Oracle-BI&EPM

  Companies have been making business decisions for decades based on transactional data stored in relational databases. Beyond that critical data, is a potential treasure trove of less structured data: weblogs, social media, email, sensors, and photographs that can be mined for useful information. Oracle offers a broad integrated portfolio of products to help you acquire and organize these diverse data sources and analyze them alongside your existing data to find new insights and capitalize on hidden relationships. Oracle Big Data Connectors Downloads here, includes: Oracle SQL Connector for Hadoop Distributed File System Release 2.1.0 Oracle Loader for Hadoop Release 2.1.0 Oracle Data Integrator Companion 11g Oracle R Connector for Hadoop v 2.1 Oracle Big Data Documentation The Oracle Big Data solution offers an integrated portfolio of products to help you organize and analyze your diverse data sources alongside your existing data to find new insights and capitalize on hidden relationships. Oracle Big Data, Release 2.2.0 - E41604_01 zip (27.4 MB) Integrated Software and Big Data Connectors User's Guide HTML PDF Oracle Data Integrator (ODI) Application Adapter for Hadoop Apache Hadoop is designed to handle and process data that is typically from data sources that are non-relational and data volumes that are beyond what is handled by relational databases. Typical processing in Hadoop includes data validation and transformations that are programmed as MapReduce jobs. Designing and implementing a MapReduce job usually requires expert programming knowledge. However, when you use Oracle Data Integrator with the Application Adapter for Hadoop, you do not need to write MapReduce jobs. Oracle Data Integrator uses Hive and the Hive Query Language (HiveQL), a SQL-like language for implementing MapReduce jobs. Employing familiar and easy-to-use tools and pre-configured knowledge modules (KMs), the application adapter provides the following capabilities: Loading data into Hadoop from the local file system and HDFS Performing validation and transformation of data within Hadoop Loading processed data from Hadoop to an Oracle database for further processing and generating reports Oracle Database Loader for Hadoop Oracle Loader for Hadoop is an efficient and high-performance loader for fast movement of data from a Hadoop cluster into a table in an Oracle database. It pre-partitions the data if necessary and transforms it into a database-ready format. Oracle Loader for Hadoop is a Java MapReduce application that balances the data across reducers to help maximize performance. Oracle R Connector for Hadoop Oracle R Connector for Hadoop is a collection of R packages that provide: Interfaces to work with Hive tables, the Apache Hadoop compute infrastructure, the local R environment, and Oracle database tables Predictive analytic techniques, written in R or Java as Hadoop MapReduce jobs, that can be applied to data in HDFS files You install and load this package as you would any other R package. Using simple R functions, you can perform tasks such as: Access and transform HDFS data using a Hive-enabled transparency layer Use the R language for writing mappers and reducers Copy data between R memory, the local file system, HDFS, Hive, and Oracle databases Schedule R programs to execute as Hadoop MapReduce jobs and return the results to any of those locations Oracle SQL Connector for Hadoop Distributed File System Using Oracle SQL Connector for HDFS, you can use an Oracle Database to access and analyze data residing in Hadoop in these formats: Data Pump files in HDFS Delimited text files in HDFS Hive tables For other file formats, such as JSON files, you can stage the input in Hive tables before using Oracle SQL Connector for HDFS. Oracle SQL Connector for HDFS uses external tables to provide Oracle Database with read access to Hive tables, and to delimited text files and Data Pump files in HDFS. Related Documentation Cloudera's Distribution Including Apache Hadoop Library HTML Oracle R Enterprise HTML Oracle NoSQL Database HTML Recent Blog Posts Big Data Appliance vs. DIY Price Comparison Big Data: Architecture Overview Big Data: Achieve the Impossible in Real-Time Big Data: Vertical Behavioral Analytics Big Data: In-Memory MapReduce Flume and Hive for Log Analytics Building Workflows in Oozie

  Read the article

• Why does Windows 7 have three system partitions?

  - by Ben

  I am using Windows 7, and I wanted to make a System image (using Windows 7), but Windows 7 checked three partitions as System (100 MB + C (install partition) + D (my partition for my files, all programs are installed at C)). I don't want to backup my D partition, but that is not really the point. I don't want Windows messing with my other partitions and making them system. Is there a way to limit Windows 7 just to partition C (install partition)? If there is no way to stop Windows from making other partitions system, can I at least delete the files that make partition D system? PS: All these three partitions are on one physical disk, partitions from other disks aren't treated as System. FACTS: desktop PC, no OEM partitions, I personally have installed Windows 7 (many times) on the C partition. Why is my D partition checked as System partition when I try to create a System Image (using Windows 7 Ultimate built in tool), even though Windows (and all the software) are installed on the C partition? Is there a way to make D "normal" or non-system partition? Here is a picture of how it looks like if I try to create a system image. Once again, why is D also a system partition?

  Read the article

• Box2D Joints in entity components system

  - by Johnmph

  I search a way to have Box2D joints in an entity component system, here is what i found : 1) Having the joints in Box2D/Body component as parameters, we have a joint array with an ID by joint and having in the other body component the same joint ID, like in this example : Entity1 - Box2D/Body component { Body => (body parameters), Joints => { Joint1 => (joint parameters), others joints... } } // Joint ID = Joint1 Entity2 - Box2D/Body component { Body => (body parameters), Joints => { Joint1 => (joint parameters), others joints... } } // Same joint ID than in Entity1 There are 3 problems with this solution : The first problem is the implementation of this solution, we must manage the joints ID to create joints and to know between which bodies they are connected. The second problem is the parameters of joint, where are they got ? on the Entity1 or Entity2 ? If they are the same parameters for the joint, there is no problem but if they are differents ? The third problem is that we can't limit number of bodies to 2 by joint (which is mandatory), a joint can only link 2 bodies, in this solution, nothing prevents to create more than 2 entities with for each a body component with the same joint ID, in this case, how we know the 2 bodies to joint and what to do with others bodies ? 2) Same solution than the first solution but by having entities ID instead of Joint ID, like in this example : Entity1 - Box2D/Body component { Body => (body parameters), Joints => { Entity2 => (joint parameters), others joints... } } Entity2 - Box2D/Body component { Body => (body parameters), Joints => { Entity1 => (joint parameters), others joints... } } With this solution, we fix the first problem of the first solution but we have always the two others problems. 3) Having a Box2D/Joint component which is inserted in the entities which contains the bodies to joint (we share the same joint component between entities with bodies to joint), like in this example : Entity1 - Box2D/Body component { Body => (body parameters) } - Box2D/Joint component { Joint => (Joint parameters) } // Shared, same as in Entity2 Entity2 - Box2D/Body component { Body => (body parameters) } - Box2D/Joint component { Joint => (joint parameters) } // Shared, same as in Entity1 There are 2 problems with this solution : The first problem is the same problem than in solution 1 and 2 : We can't limit number of bodies to 2 by joint (which is mandatory), a joint can only link 2 bodies, in this solution, nothing prevents to create more than 2 entities with for each a body component and the shared joint component, in this case, how we know the 2 bodies to joint and what to do with others bodies ? The second problem is that we can have only one joint by body because entity components system allows to have only one component of same type in an entity. So we can't put two Joint components in the same entity. 4) Having a Box2D/Joint component which is inserted in the entity which contains the first body component to joint and which has an entity ID parameter (this entity contains the second body to joint), like in this example : Entity1 - Box2D/Body component { Body => (body parameters) } - Box2D/Joint component { Entity2 => (Joint parameters) } // Entity2 is the entity ID which contains the other body to joint, the first body being in this entity Entity2 - Box2D/Body component { Body => (body parameters) } There are exactly the same problems that in the third solution, the only difference is that we can have two differents joints by entity instead of one (by putting one joint component in an entity and another joint component in another entity, each joint referencing to the other entity). 5) Having a Box2D/Joint component which take in parameter the two entities ID which contains the bodies to joint, this component can be inserted in any entity, like in this example : Entity1 - Box2D/Body component { Body => (body parameters) } Entity2 - Box2D/Body component { Body => (body parameters) } Entity3 - Box2D/Joint component { Joint => (Body1 => Entity1, Body2 => Entity2, others parameters of joint) } // Entity1 is the ID of the entity which have the first body to joint and Entity2 is the ID of the entity which have the second body to joint (This component can be in any entity, that doesn't matter) With this solution, we fix the problem of the body limitation by joint, we can only have two bodies per joint, which is correct. And we are not limited by number of joints per body, because we can create an another Box2D/Joint component, referencing to Entity1 and Entity2 and put this component in a new entity. The problem of this solution is : What happens if we change the Body1 or Body2 parameter of Joint component at runtime ? We need to add code to sync the Body1/Body2 parameters changes with the real joint object. 6) Same as solution 3 but in a better way : Having a Box2D/Joint component Box2D/Joint which is inserted in the entities which contains the bodies to joint, we share the same joint component between these entities BUT the difference is that we create a new entity to link the body component with the joint component, like in this example : Entity1 - Box2D/Body component { Body => (body parameters) } // Shared, same as in Entity3 Entity2 - Box2D/Body component { Body => (body parameters) } // Shared, same as in Entity4 Entity3 - Box2D/Body component { Body => (body parameters) } // Shared, same as in Entity1 - Box2D/Joint component { Joint => (joint parameters) } // Shared, same as in Entity4 Entity4 - Box2D/Body component { Body => (body parameters) } // Shared, same as in Entity2 - Box2D/Joint component { Joint => (joint parameters) } // Shared, same as in Entity3 With this solution, we fix the second problem of the solution 3, because we can create an Entity5 which will have the shared body component of Entity1 and an another joint component so we are no longer limited in the joint number per body. But the first problem of solution 3 remains, because we can't limit the number of entities which have the shared joint component. To resolve this problem, we can add a way to limit the number of share of a component, so for the Joint component, we limit the number of share to 2, because we can only joint 2 bodies per joint. This solution would be perfect because there is no need to add code to sync changes like in the solution 5 because we are notified by the entity components system when components / entities are added to/removed from the system. But there is a conception problem : How to know easily and quickly between which bodies the joint operates ? Because, there is no way to find easily an entity with a component instance. My question is : Which solution is the best ? Is there any other better solutions ? Sorry for the long text and my bad english.

  Read the article

• The Ins and Outs of Effective Smart Grid Data Management

  - by caroline.yu

  Oracle Utilities and Accenture recently sponsored a one-hour Web cast entitled, "The Ins and Outs of Effective Smart Grid Data Management." Oracle and Accenture created this Web cast to help utilities better understand the types of data collected over smart grid networks and the issues associated with mapping out a coherent information management strategy. The Web cast also addressed important points that utilities must consider with the imminent flood of data that both present and next-generation smart grid components will generate. The three speakers, including Oracle Utilities' Brad Williams, focused on the key factors associated with taking the millions of data points captured in real time and implementing the strategies, frameworks and technologies that enable utilities to process, store, analyze, visualize, integrate, transport and transform data into the information required to deliver targeted business benefits. The Web cast replay is available here. The Web cast slides are available here.

  Read the article

• What Works in Data Integration?

  - by dain.hansen

  TDWI just recently put out this paper on "What Works in Data Integration". I invite you especially to take a look at the section on "Accelerating your Business with Real-time Data Integration" and the DIRECTV case study. The article discusses some of the technology considerations for BI/DW and how data integration plays a role to deliver timely, accessible, and high-quality data. It goes on to outline the three key requirements for how to deliver high performance, low impact, and reliability and how that can translate to faster results. The DIRECTV webinar is something you definitely want to take a look at, you'll hear how DIRECTV successfully transformed their data warehouse investments into a competitive advantage with Oracle GoldenGate.

  Read the article

• Are there sources of email marketing data available?

  - by Gortron

  Are sources of email marketing data available to the public? I would like to see email marketing data to see what kind of content a business sends out, the frequency of sending, the number of people emailed, especially the resulting open rates and click through rates. Are businesses willing to share data on their previous email marketing campaigns without divulging their contact list? I would like to use this data to create an application to help businesses create better newsletters by using this data as a benchmark, basically sharing what works and what doesn't for each industry.

  Read the article

• How to Achieve Real-Time Data Protection and Availabilty....For Real

  - by JoeMeeks

  There is a class of business and mission critical applications where downtime or data loss have substantial negative impact on revenue, customer service, reputation, cost, etc. Because the Oracle Database is used extensively to provide reliable performance and availability for this class of application, it also provides an integrated set of capabilities for real-time data protection and availability. Active Data Guard, depicted in the figure below, is the cornerstone for accomplishing these objectives because it provides the absolute best real-time data protection and availability for the Oracle Database. This is a bold statement, but it is supported by the facts. It isn’t so much that alternative solutions are bad, it’s just that their architectures prevent them from achieving the same levels of data protection, availability, simplicity, and asset utilization provided by Active Data Guard. Let’s explore further. Backups are the most popular method used to protect data and are an essential best practice for every database. Not surprisingly, Oracle Recovery Manager (RMAN) is one of the most commonly used features of the Oracle Database. But comparing Active Data Guard to backups is like comparing apples to motorcycles. Active Data Guard uses a hot (open read-only), synchronized copy of the production database to provide real-time data protection and HA. In contrast, a restore from backup takes time and often has many moving parts - people, processes, software and systems – that can create a level of uncertainty during an outage that critical applications can’t afford. This is why backups play a secondary role for your most critical databases by complementing real-time solutions that can provide both data protection and availability. Before Data Guard, enterprises used storage remote-mirroring for real-time data protection and availability. Remote-mirroring is a sophisticated storage technology promoted as a generic infrastructure solution that makes a simple promise – whatever is written to a primary volume will also be written to the mirrored volume at a remote site. Keeping this promise is also what causes data loss and downtime when the data written to primary volumes is corrupt – the same corruption is faithfully mirrored to the remote volume making both copies unusable. This happens because remote-mirroring is a generic process. It has no  intrinsic knowledge of Oracle data structures to enable advanced protection, nor can it perform independent Oracle validation BEFORE changes are applied to the remote copy. There is also nothing to prevent human error (e.g. a storage admin accidentally deleting critical files) from also impacting the remote mirrored copy. Remote-mirroring tricks users by creating a false impression that there are two separate copies of the Oracle Database. In truth; while remote-mirroring maintains two copies of the data on different volumes, both are part of a single closely coupled system. Not only will remote-mirroring propagate corruptions and administrative errors, but the changes applied to the mirrored volume are a result of the same Oracle code path that applied the change to the source volume. There is no isolation, either from a storage mirroring perspective or from an Oracle software perspective.  Bottom line, storage remote-mirroring lacks both the smarts and isolation level necessary to provide true data protection. Active Data Guard offers much more than storage remote-mirroring when your objective is protecting your enterprise from downtime and data loss. Like remote-mirroring, an Active Data Guard replica is an exact block for block copy of the primary. Unlike remote-mirroring, an Active Data Guard replica is NOT a tightly coupled copy of the source volumes - it is a completely independent Oracle Database. Active Data Guard’s inherent knowledge of Oracle data block and redo structures enables a separate Oracle Database using a different Oracle code path than the primary to use the full complement of Oracle data validation methods before changes are applied to the synchronized copy. These include: physical check sum, logical intra-block checking, lost write validation, and automatic block repair. The figure below illustrates the stark difference between the knowledge that remote-mirroring can discern from an Oracle data block and what Active Data Guard can discern. An Active Data Guard standby also provides a range of additional services enabled by the fact that it is a running Oracle Database - not just a mirrored copy of data files. An Active Data Guard standby database can be open read-only while it is synchronizing with the primary. This enables read-only workloads to be offloaded from the primary system and run on the active standby - boosting performance by utilizing all assets. An Active Data Guard standby can also be used to implement many types of system and database maintenance in rolling fashion. Maintenance and upgrades are first implemented on the standby while production runs unaffected at the primary. After the primary and standby are synchronized and all changes have been validated, the production workload is quickly switched to the standby. The only downtime is the time required for user connections to transfer from one system to the next. These capabilities further expand the expectations of availability offered by a data protection solution beyond what is possible to do using storage remote-mirroring. So don’t be fooled by appearances.  Storage remote-mirroring and Active Data Guard replication may look similar on the surface - but the devil is in the details. Only Active Data Guard has the smarts, the isolation, and the simplicity, to provide the best data protection and availability for the Oracle Database. Stay tuned for future blog posts that dive into the many differences between storage remote-mirroring and Active Data Guard along the dimensions of data protection, data availability, cost, asset utilization and return on investment. For additional information on Active Data Guard, see: Active Data Guard Technical White Paper Active Data Guard vs Storage Remote-Mirroring Active Data Guard Home Page on the Oracle Technology Network

  Read the article

• Extending SSIS with custom Data Flow components (Presentation)

  Download the slides and sample code from my Extending SSIS with custom Data Flow components presentation, first presented at the SQLBits II (The SQL) Community Conference. Abstract Get some real-world insights into developing data flow components for SSIS. This starts with an introduction to the data flow pipeline engine, and explains the real differences between adapters and the three sub-types of transformation. Understanding how the different types of component behave and manage data is key to writing components of your own, and probably should but be required knowledge for anyone building packages at all. Using sample code throughout, I will show you how to write components, as well as highlighting best practice and lessons learned. The sample code includes fully working example projects for source, destination and transformation components. Presentation & Samples (358KB) Extending SSIS with custom Data Flow components.zip

  Read the article

• Entity Component System for HUD and GUI

  - by Jason L.

  This is a very rough sketch of how I currently have things designed. It should, at least, give an idea of how my ECS is currently designed. If you notice in that diagram, I have basically split the HUD out of the ECS. They have their own set of things (HudLayer, HudComponent, etc) and are handled differently. This is where I'm struggling, though. There are many different instances in which the HUD will need to know about entities. Not just data changing (I have an event dispatcher for that), but the actual entity and all it encompasses. There are also situations where entities will need to be able to query the HUD for data. Let's take a couple examples: First, my equipment screen. On here I can change the equipment on a character (Entity). In order for this to happen, I need to know about the entity. At least I think I do? How can I handle this? The second scenario involves my Systems needing to query a HudComponent for data. A specific example would be my battle system. Each "team" is given a 3x3 grid they can move around in. See here: Skills target these cells, and not the player, so I would need a way for my systems to determine which cells are occupied and which are not. Basically I need a way for two way communication between Systems and my HUD. I know it's recommended (by some people, anyways) to take your HUD out of the ECS. Is that appropriate in my case?

  Read the article

• How to use OO for data analysis? [closed]

  - by Konsta

  In which ways could object-orientation (OO) make my data analysis more efficient and let me reuse more of my code? The data analysis can be broken up into get data (from db or csv or similar) transform data (filter, group/pivot, ...) display/plot (graph timeseries, create tables, etc.) I mostly use Python and its Pandas and Matplotlib packages for this besides some DB connectivity (SQL). Almost all of my code is a functional/procedural mix. While I have started to create a data object for a certain collection of time series, I wonder if there are OO design patterns/approaches for other parts of the process that might increase efficiency?

  Read the article

• Winner of the 2012 Government Big Data Solutions Award

  - by Jean-Pierre Dijcks

  Hot off the press: The winner of the 2012 Government Big Data Solutions Aware is the National Cancer Institute!! Read all the details on CTOLabs.com. A short excerpt to wet your appetite: "... This solution, based on the Oracle Big Data Appliance with the Cloudera Distribution of Apache Hadoop (CDH), leverages capabilities available from the Big Data community today in pioneering ways that can serve a broad range of researchers. The promising approach of this solution is repeatable across many other Big Data challenges for bioinfomatics, making this approach worthy of its selection as the 2012 Government Big Data Solution Award." Read the entire post. Congrats to the entire team!!

  Read the article

• Markup format or script for data files?

  - by Aaron

  The game I'm designing will be mainly written in a high level scripting language (leaning towards either Lua or Squirrel) with a C++ core. In addition to scripts I'm also going to need different data files. Many data files will be for static information such as graphical assets and monster types. I'd also want to create and update data files at runtime for user information like option settings and game saves. Can I get away with using plain script files (i.e. .lua or .nut files) for my data files, or is it better to use dedicated markup formats like XML or YAML? If I use script files, loaded separately from my true scripts, then I wouldn't need an extra library to read those files. Scripting languages like Lua also have table syntax that lend themselves towards data definition. On the other hand I'd have to write my own schema check code. These languages also don't seem to support serialization "out of the box" like the markup format libraries do.

  Read the article

• SQL Server and the XML Data Type : Data Manipulation

  The introduction of the xml data type, with its own set of methods for processing xml data, made it possible for SQL Server developers to create columns and variables of the type xml. Deanna Dicken examines the modify() method, which provides for data manipulation of the XML data stored in the xml data type via XML DML statements. Too many SQL Servers to keep up with?Download a free trial of SQL Response to monitor your SQL Servers in just one intuitive interface."The monitoringin SQL Response is excellent." Mike Towery.

  Read the article

• Getting data from a webpage in a stable and efficient way

  - by Mike Heremans

  Recently I've learned that using a regex to parse the HTML of a website to get the data you need isn't the best course of action. So my question is simple: What then, is the best / most efficient and a generally stable way to get this data? I should note that: There are no API's There is no other source where I can get the data from (no databases, feeds and such) There is no access to the source files. (Data from public websites) Let's say the data is normal text, displayed in a table in a html page I'm currently using python for my project but a language independent solution/tips would be nice. As a side question: How would you go about it when the webpage is constructed by Ajax calls?

  Read the article

• ClickOnce manifest problem

  - by TWith2Sugars

  We are currently deploying a WPF 4 app via click once and there is a scenario when the installation fails. If the user does not have .Net 4.0 Full install and attempts to install our app the framework installs fine but the app fails to install. If we re-run the installation again the app installs fine. Here is a copy of the log: PLATFORM VERSION INFO Windows : 6.1.7600.0 (Win32NT) Common Language Runtime : 2.0.50727.4927 System.Deployment.dll : 2.0.50727.4927 (NetFXspW7.050727-4900) mscorwks.dll : 2.0.50727.4927 (NetFXspW7.050727-4900) dfdll.dll : 2.0.50727.4927 (NetFXspW7.050727-4900) dfshim.dll : 4.0.31106.0 (Main.031106-0000) SOURCES Deployment url : [URL REMOVED] Server : Apache/2.0.54 Application url : [URL REMOVED] Server : Apache/2.0.54 IDENTITIES Deployment Identity : Graphicly.App.application, Version=0.3.2.0, Culture=neutral, PublicKeyToken=c982228345371fbc, processorArchitecture=msil Application Identity : Graphicly.App.exe, Version=0.3.2.0, Culture=neutral, PublicKeyToken=c982228345371fbc, processorArchitecture=msil, type=win32 APPLICATION SUMMARY * Installable application. ERROR SUMMARY Below is a summary of the errors, details of these errors are listed later in the log. * Dependency Graphicly.WCFClient.dll cannot be processed for patching. Following failure messages were detected: + Exception occurred loading manifest from file Graphicly.WCFClient.dll: the manifest may not be valid or the file could not be opened. + Cannot load internal manifest from component file. * Dependency Microsoft.Surface.Presentation.Design.dll cannot be processed for patching. Following failure messages were detected: + Exception occurred loading manifest from file Microsoft.Surface.Presentation.Design.dll: the manifest may not be valid or the file could not be opened. + Cannot load internal manifest from component file. * Dependency GalaSoft.MvvmLight.WPF4.dll cannot be processed for patching. Following failure messages were detected: + Exception occurred loading manifest from file GalaSoft.MvvmLight.WPF4.dll: the manifest may not be valid or the file could not be opened. + Cannot load internal manifest from component file. * Dependency Graphicly.Infrastructure.dll cannot be processed for patching. Following failure messages were detected: + Exception occurred loading manifest from file Graphicly.Infrastructure.dll: the manifest may not be valid or the file could not be opened. + Cannot load internal manifest from component file. * Dependency Graphicly.AutoUpdater.dll cannot be processed for patching. Following failure messages were detected: + Exception occurred loading manifest from file Graphicly.AutoUpdater.dll: the manifest may not be valid or the file could not be opened. + Cannot load internal manifest from component file. * Dependency System.Windows.Interactivity.dll cannot be processed for patching. Following failure messages were detected: + Exception occurred loading manifest from file System.Windows.Interactivity.dll: the manifest may not be valid or the file could not be opened. + Cannot load internal manifest from component file. * Dependency Microsoft.Surface.Presentation.dll cannot be processed for patching. Following failure messages were detected: + Exception occurred loading manifest from file Microsoft.Surface.Presentation.dll: the manifest may not be valid or the file could not be opened. + Cannot load internal manifest from component file. * Dependency Graphicly.Fonts.dll cannot be processed for patching. Following failure messages were detected: + Exception occurred loading manifest from file Graphicly.Fonts.dll: the manifest may not be valid or the file could not be opened. + Cannot load internal manifest from component file. * Dependency Graphicly.Reader.dll cannot be processed for patching. Following failure messages were detected: + Exception occurred loading manifest from file Graphicly.Reader.dll: the manifest may not be valid or the file could not be opened. + Cannot load internal manifest from component file. * Dependency Microsoft.Surface.Presentation.Generic.dll cannot be processed for patching. Following failure messages were detected: + Exception occurred loading manifest from file Microsoft.Surface.Presentation.Generic.dll: the manifest may not be valid or the file could not be opened. + Cannot load internal manifest from component file. * Dependency Graphicly.Controls.dll cannot be processed for patching. Following failure messages were detected: + Exception occurred loading manifest from file Graphicly.Controls.dll: the manifest may not be valid or the file could not be opened. + Cannot load internal manifest from component file. * Dependency Graphicly.SocialNetwork.dll cannot be processed for patching. Following failure messages were detected: + Exception occurred loading manifest from file Graphicly.SocialNetwork.dll: the manifest may not be valid or the file could not be opened. + Cannot load internal manifest from component file. * Dependency Graphicly.Archive.dll cannot be processed for patching. Following failure messages were detected: + Exception occurred loading manifest from file Graphicly.Archive.dll: the manifest may not be valid or the file could not be opened. + Cannot load internal manifest from component file. * Dependency Graphicly.App.exe cannot be processed for patching. Following failure messages were detected: + Exception occurred loading manifest from file Graphicly.App.exe: the manifest may not be valid or the file could not be opened. + Cannot load internal manifest from component file. * Dependency GalaSoft.MvvmLight.Extras.WPF4.dll cannot be processed for patching. Following failure messages were detected: + Exception occurred loading manifest from file GalaSoft.MvvmLight.Extras.WPF4.dll: the manifest may not be valid or the file could not be opened. + Cannot load internal manifest from component file. * Activation of [URL REMOVED] resulted in exception. Following failure messages were detected: + Exception occurred loading manifest from file GalaSoft.MvvmLight.Extras.WPF4.dll: the manifest may not be valid or the file could not be opened. + Cannot load internal manifest from component file. COMPONENT STORE TRANSACTION FAILURE SUMMARY No transaction error was detected. WARNINGS * The file named Microsoft.Windows.Design.Extensibility.dll does not have a hash specified in the manifest. Hash validation will be ignored. * The file named Ionic.Zip.Reduced.dll does not have a hash specified in the manifest. Hash validation will be ignored. * The file named Newtonsoft.Json.dll does not have a hash specified in the manifest. Hash validation will be ignored. * The file named Microsoft.WindowsAzure.StorageClient.dll does not have a hash specified in the manifest. Hash validation will be ignored. * The file named Dimebrain.TweetSharp.dll does not have a hash specified in the manifest. Hash validation will be ignored. * The file named Microsoft.Windows.Design.Interaction.dll does not have a hash specified in the manifest. Hash validation will be ignored. * The file named HtmlAgilityPack.dll does not have a hash specified in the manifest. Hash validation will be ignored. * The file named Facebook.dll does not have a hash specified in the manifest. Hash validation will be ignored. OPERATION PROGRESS STATUS * [20/05/2010 09:17:33] : Activation of [URL REMOVED] has started. * [20/05/2010 09:17:38] : Processing of deployment manifest has successfully completed. * [20/05/2010 09:17:38] : Installation of the application has started. * [20/05/2010 09:17:39] : Processing of application manifest has successfully completed. * [20/05/2010 09:17:40] : Request of trust and detection of platform is complete. ERROR DETAILS Following errors were detected during this operation. * [20/05/2010 09:17:40] System.Deployment.Application.InvalidDeploymentException (ManifestLoad) - Exception occurred loading manifest from file Graphicly.WCFClient.dll: the manifest may not be valid or the file could not be opened. - Source: System.Deployment - Stack trace: at System.Deployment.Application.Manifest.AssemblyManifest.ManifestLoadExceptionHelper(Exception exception, String filePath) at System.Deployment.Application.Manifest.AssemblyManifest.LoadFromInternalManifestFile(String filePath) at System.Deployment.Application.FileDownloader.AddFilesInHashtable(Hashtable hashtable, AssemblyManifest applicationManifest, String applicationFolder) --- Inner Exception --- System.Deployment.Application.DeploymentException (InvalidManifest) - Cannot load internal manifest from component file. - Source: - Stack trace: * [20/05/2010 09:17:40] System.Deployment.Application.InvalidDeploymentException (ManifestLoad) - Exception occurred loading manifest from file Microsoft.Surface.Presentation.Design.dll: the manifest may not be valid or the file could not be opened. - Source: System.Deployment - Stack trace: at System.Deployment.Application.Manifest.AssemblyManifest.ManifestLoadExceptionHelper(Exception exception, String filePath) at System.Deployment.Application.Manifest.AssemblyManifest.LoadFromInternalManifestFile(String filePath) at System.Deployment.Application.FileDownloader.AddFilesInHashtable(Hashtable hashtable, AssemblyManifest applicationManifest, String applicationFolder) --- Inner Exception --- System.Deployment.Application.DeploymentException (InvalidManifest) - Cannot load internal manifest from component file. - Source: - Stack trace: * [20/05/2010 09:17:40] System.Deployment.Application.InvalidDeploymentException (ManifestLoad) - Exception occurred loading manifest from file GalaSoft.MvvmLight.WPF4.dll: the manifest may not be valid or the file could not be opened. - Source: System.Deployment - Stack trace: at System.Deployment.Application.Manifest.AssemblyManifest.ManifestLoadExceptionHelper(Exception exception, String filePath) at System.Deployment.Application.Manifest.AssemblyManifest.LoadFromInternalManifestFile(String filePath) at System.Deployment.Application.FileDownloader.AddFilesInHashtable(Hashtable hashtable, AssemblyManifest applicationManifest, String applicationFolder) --- Inner Exception --- System.Deployment.Application.DeploymentException (InvalidManifest) - Cannot load internal manifest from component file. - Source: - Stack trace: * [20/05/2010 09:17:40] System.Deployment.Application.InvalidDeploymentException (ManifestLoad) - Exception occurred loading manifest from file Graphicly.Infrastructure.dll: the manifest may not be valid or the file could not be opened. - Source: System.Deployment - Stack trace: at System.Deployment.Application.Manifest.AssemblyManifest.ManifestLoadExceptionHelper(Exception exception, String filePath) at System.Deployment.Application.Manifest.AssemblyManifest.LoadFromInternalManifestFile(String filePath) at System.Deployment.Application.FileDownloader.AddFilesInHashtable(Hashtable hashtable, AssemblyManifest applicationManifest, String applicationFolder) --- Inner Exception --- System.Deployment.Application.DeploymentException (InvalidManifest) - Cannot load internal manifest from component file. - Source: - Stack trace: * [20/05/2010 09:17:40] System.Deployment.Application.InvalidDeploymentException (ManifestLoad) - Exception occurred loading manifest from file Graphicly.AutoUpdater.dll: the manifest may not be valid or the file could not be opened. - Source: System.Deployment - Stack trace: at System.Deployment.Application.Manifest.AssemblyManifest.ManifestLoadExceptionHelper(Exception exception, String filePath) at System.Deployment.Application.Manifest.AssemblyManifest.LoadFromInternalManifestFile(String filePath) at System.Deployment.Application.FileDownloader.AddFilesInHashtable(Hashtable hashtable, AssemblyManifest applicationManifest, String applicationFolder) --- Inner Exception --- System.Deployment.Application.DeploymentException (InvalidManifest) - Cannot load internal manifest from component file. - Source: - Stack trace: * [20/05/2010 09:17:40] System.Deployment.Application.InvalidDeploymentException (ManifestLoad) - Exception occurred loading manifest from file System.Windows.Interactivity.dll: the manifest may not be valid or the file could not be opened. - Source: System.Deployment - Stack trace: at System.Deployment.Application.Manifest.AssemblyManifest.ManifestLoadExceptionHelper(Exception exception, String filePath) at System.Deployment.Application.Manifest.AssemblyManifest.LoadFromInternalManifestFile(String filePath) at System.Deployment.Application.FileDownloader.AddFilesInHashtable(Hashtable hashtable, AssemblyManifest applicationManifest, String applicationFolder) --- Inner Exception --- System.Deployment.Application.DeploymentException (InvalidManifest) - Cannot load internal manifest from component file. - Source: - Stack trace: * [20/05/2010 09:17:40] System.Deployment.Application.InvalidDeploymentException (ManifestLoad) - Exception occurred loading manifest from file Microsoft.Surface.Presentation.dll: the manifest may not be valid or the file could not be opened. - Source: System.Deployment - Stack trace: at System.Deployment.Application.Manifest.AssemblyManifest.ManifestLoadExceptionHelper(Exception exception, String filePath) at System.Deployment.Application.Manifest.AssemblyManifest.LoadFromInternalManifestFile(String filePath) at System.Deployment.Application.FileDownloader.AddFilesInHashtable(Hashtable hashtable, AssemblyManifest applicationManifest, String applicationFolder) --- Inner Exception --- System.Deployment.Application.DeploymentException (InvalidManifest) - Cannot load internal manifest from component file. - Source: - Stack trace: * [20/05/2010 09:17:40] System.Deployment.Application.InvalidDeploymentException (ManifestLoad) - Exception occurred loading manifest from file Graphicly.Fonts.dll: the manifest may not be valid or the file could not be opened. - Source: System.Deployment - Stack trace: at System.Deployment.Application.Manifest.AssemblyManifest.ManifestLoadExceptionHelper(Exception exception, String filePath) at System.Deployment.Application.Manifest.AssemblyManifest.LoadFromInternalManifestFile(String filePath) at System.Deployment.Application.FileDownloader.AddFilesInHashtable(Hashtable hashtable, AssemblyManifest applicationManifest, String applicationFolder) --- Inner Exception --- System.Deployment.Application.DeploymentException (InvalidManifest) - Cannot load internal manifest from component file. - Source: - Stack trace: * [20/05/2010 09:17:40] System.Deployment.Application.InvalidDeploymentException (ManifestLoad) - Exception occurred loading manifest from file Graphicly.Reader.dll: the manifest may not be valid or the file could not be opened. - Source: System.Deployment - Stack trace: at System.Deployment.Application.Manifest.AssemblyManifest.ManifestLoadExceptionHelper(Exception exception, String filePath) at System.Deployment.Application.Manifest.AssemblyManifest.LoadFromInternalManifestFile(String filePath) at System.Deployment.Application.FileDownloader.AddFilesInHashtable(Hashtable hashtable, AssemblyManifest applicationManifest, String applicationFolder) --- Inner Exception --- System.Deployment.Application.DeploymentException (InvalidManifest) - Cannot load internal manifest from component file. - Source: - Stack trace: * [20/05/2010 09:17:40] System.Deployment.Application.InvalidDeploymentException (ManifestLoad) - Exception occurred loading manifest from file Microsoft.Surface.Presentation.Generic.dll: the manifest may not be valid or the file could not be opened. - Source: System.Deployment - Stack trace: at System.Deployment.Application.Manifest.AssemblyManifest.ManifestLoadExceptionHelper(Exception exception, String filePath) at System.Deployment.Application.Manifest.AssemblyManifest.LoadFromInternalManifestFile(String filePath) at System.Deployment.Application.FileDownloader.AddFilesInHashtable(Hashtable hashtable, AssemblyManifest applicationManifest, String applicationFolder) --- Inner Exception --- System.Deployment.Application.DeploymentException (InvalidManifest) - Cannot load internal manifest from component file. - Source: - Stack trace: * [20/05/2010 09:17:41] System.Deployment.Application.InvalidDeploymentException (ManifestLoad) - Exception occurred loading manifest from file Graphicly.Controls.dll: the manifest may not be valid or the file could not be opened. - Source: System.Deployment - Stack trace: at System.Deployment.Application.Manifest.AssemblyManifest.ManifestLoadExceptionHelper(Exception exception, String filePath) at System.Deployment.Application.Manifest.AssemblyManifest.LoadFromInternalManifestFile(String filePath) at System.Deployment.Application.FileDownloader.AddFilesInHashtable(Hashtable hashtable, AssemblyManifest applicationManifest, String applicationFolder) --- Inner Exception --- System.Deployment.Application.DeploymentException (InvalidManifest) - Cannot load internal manifest from component file. - Source: - Stack trace: * [20/05/2010 09:17:41] System.Deployment.Application.InvalidDeploymentException (ManifestLoad) - Exception occurred loading manifest from file Graphicly.SocialNetwork.dll: the manifest may not be valid or the file could not be opened. - Source: System.Deployment - Stack trace: at System.Deployment.Application.Manifest.AssemblyManifest.ManifestLoadExceptionHelper(Exception exception, String filePath) at System.Deployment.Application.Manifest.AssemblyManifest.LoadFromInternalManifestFile(String filePath) at System.Deployment.Application.FileDownloader.AddFilesInHashtable(Hashtable hashtable, AssemblyManifest applicationManifest, String applicationFolder) --- Inner Exception --- System.Deployment.Application.DeploymentException (InvalidManifest) - Cannot load internal manifest from component file. - Source: - Stack trace: * [20/05/2010 09:17:41] System.Deployment.Application.InvalidDeploymentException (ManifestLoad) - Exception occurred loading manifest from file Graphicly.Archive.dll: the manifest may not be valid or the file could not be opened. - Source: System.Deployment - Stack trace: at System.Deployment.Application.Manifest.AssemblyManifest.ManifestLoadExceptionHelper(Exception exception, String filePath) at System.Deployment.Application.Manifest.AssemblyManifest.LoadFromInternalManifestFile(String filePath) at System.Deployment.Application.FileDownloader.AddFilesInHashtable(Hashtable hashtable, AssemblyManifest applicationManifest, String applicationFolder) --- Inner Exception --- System.Deployment.Application.DeploymentException (InvalidManifest) - Cannot load internal manifest from component file. - Source: - Stack trace: * [20/05/2010 09:17:41] System.Deployment.Application.InvalidDeploymentException (ManifestLoad) - Exception occurred loading manifest from file Graphicly.App.exe: the manifest may not be valid or the file could not be opened. - Source: System.Deployment - Stack trace: at System.Deployment.Application.Manifest.AssemblyManifest.ManifestLoadExceptionHelper(Exception exception, String filePath) at System.Deployment.Application.Manifest.AssemblyManifest.LoadFromInternalManifestFile(String filePath) at System.Deployment.Application.FileDownloader.AddFilesInHashtable(Hashtable hashtable, AssemblyManifest applicationManifest, String applicationFolder) --- Inner Exception --- System.Deployment.Application.DeploymentException (InvalidManifest) - Cannot load internal manifest from component file. - Source: - Stack trace: * [20/05/2010 09:17:41] System.Deployment.Application.InvalidDeploymentException (ManifestLoad) - Exception occurred loading manifest from file GalaSoft.MvvmLight.Extras.WPF4.dll: the manifest may not be valid or the file could not be opened. - Source: System.Deployment - Stack trace: at System.Deployment.Application.Manifest.AssemblyManifest.ManifestLoadExceptionHelper(Exception exception, String filePath) at System.Deployment.Application.Manifest.AssemblyManifest.LoadFromInternalManifestFile(String filePath) at System.Deployment.Application.FileDownloader.AddFilesInHashtable(Hashtable hashtable, AssemblyManifest applicationManifest, String applicationFolder) --- Inner Exception --- System.Deployment.Application.DeploymentException (InvalidManifest) - Cannot load internal manifest from component file. - Source: - Stack trace: * [20/05/2010 09:17:41] System.Deployment.Application.InvalidDeploymentException (ManifestLoad) - Exception occurred loading manifest from file GalaSoft.MvvmLight.Extras.WPF4.dll: the manifest may not be valid or the file could not be opened. - Source: System.Deployment - Stack trace: at System.Deployment.Application.Manifest.AssemblyManifest.ManifestLoadExceptionHelper(Exception exception, String filePath) at System.Deployment.Application.Manifest.AssemblyManifest.LoadFromInternalManifestFile(String filePath) at System.Deployment.Application.DownloadManager.ProcessDownloadedFile(Object sender, DownloadEventArgs e) at System.Deployment.Application.FileDownloader.DownloadModifiedEventHandler.Invoke(Object sender, DownloadEventArgs e) at System.Deployment.Application.FileDownloader.PatchSingleFile(DownloadQueueItem item, Hashtable dependencyTable) at System.Deployment.Application.FileDownloader.PatchFiles(SubscriptionState subState) at System.Deployment.Application.FileDownloader.Download(SubscriptionState subState) at System.Deployment.Application.DownloadManager.DownloadDependencies(SubscriptionState subState, AssemblyManifest deployManifest, AssemblyManifest appManifest, Uri sourceUriBase, String targetDirectory, String group, IDownloadNotification notification, DownloadOptions options) at System.Deployment.Application.ApplicationActivator.DownloadApplication(SubscriptionState subState, ActivationDescription actDesc, Int64 transactionId, TempDirectory& downloadTemp) at System.Deployment.Application.ApplicationActivator.InstallApplication(SubscriptionState& subState, ActivationDescription actDesc) at System.Deployment.Application.ApplicationActivator.PerformDeploymentActivation(Uri activationUri, Boolean isShortcut, String textualSubId, String deploymentProviderUrlFromExtension, BrowserSettings browserSettings, String& errorPageUrl) at System.Deployment.Application.ApplicationActivator.ActivateDeploymentWorker(Object state) --- Inner Exception --- System.Deployment.Application.DeploymentException (InvalidManifest) - Cannot load internal manifest from component file. - Source: - Stack trace: COMPONENT STORE TRANSACTION DETAILS No transaction information is available. I'm baffled. Any ideas what this could be? Cheers Tony

  Read the article

• What data structure to use / data persistence

  - by Dave

  I have an app where I need one table of information with the following fields: field 1 - int or char field 2 - string (max 10 char) field 3 - string (max 20 char) field 4 - float I need the program to filter on field 1 based upon a segmented control and select a field 2 from a picker. From this data I need to look up field 4 to use in a calculation. Total records will be about 200. I never see it go above 400 - 500. I am going to use a singleton which I am able to do, I just need help with the structure for this with data persistence. What type of data structure should I use for this and should I use NSNumber, NSString, etc. or old data types like float, Char, etc. I thought about a struct put into an array but there is probably a better way. This is new to me so any help or reference to examples would be great. I also thought about a plist or dictionary but it looks like it is just a lookup and a field which obviously won't work. Core data looked like overkill to me. Also, with any recommendation how should I get initial data into it? I want the user to be able to edit and add to the database. Sorry for the old terms, you can see what generation I am from... Thanks in advance!!!!

  Read the article

• Using a "white list" for extracting terms for Text Mining

  - by [email protected]

  In Part 1 of my post on "Generating cluster names from a document clustering model" (part 1, part 2, part 3), I showed how to build a clustering model from text documents using Oracle Data Miner, which automates preparing data for text mining. In this process we specified a custom stoplist and lexer and relied on Oracle Text to identify important terms.  However, there is an alternative approach, the white list, which uses a thesaurus object with the Oracle Text CTXRULE index to allow you to specify the important terms. INTRODUCTIONA stoplist is used to exclude, i.e., black list, specific words in your documents from being indexed. For example, words like a, if, and, or, and but normally add no value when text mining. Other words can also be excluded if they do not help to differentiate documents, e.g., the word Oracle is ubiquitous in the Oracle product literature. One problem with stoplists is determining which words to specify. This usually requires inspecting the terms that are extracted, manually identifying which ones you don't want, and then re-indexing the documents to determine if you missed any. Since a corpus of documents could contain thousands of words, this could be a tedious exercise. Moreover, since every word is considered as an individual token, a term excluded in one context may be needed to help identify a term in another context. For example, in our Oracle product literature example, the words "Oracle Data Mining" taken individually are not particular helpful. The term "Oracle" may be found in nearly all documents, as with the term "Data." The term "Mining" is more unique, but could also refer to the Mining industry. If we exclude "Oracle" and "Data" by specifying them in the stoplist, we lose valuable information. But it we include them, they may introduce too much noise. Still, when you have a broad vocabulary or don't have a list of specific terms of interest, you rely on the text engine to identify important terms, often by computing the term frequency - inverse document frequency metric. (This is effectively a weight associated with each term indicating its relative importance in a document within a collection of documents. We'll revisit this later.) The results using this technique is often quite valuable. As noted above, an alternative to the subtractive nature of the stoplist is to specify a white list, or a list of terms--perhaps multi-word--that we want to extract and use for data mining. The obvious downside to this approach is the need to specify the set of terms of interest. However, this may not be as daunting a task as it seems. For example, in a given domain (Oracle product literature), there is often a recognized glossary, or a list of keywords and phrases (Oracle product names, industry names, product categories, etc.). Being able to identify multi-word terms, e.g., "Oracle Data Mining" or "Customer Relationship Management" as a single token can greatly increase the quality of the data mining results. The remainder of this post and subsequent posts will focus on how to produce a dataset that contains white list terms, suitable for mining. CREATING A WHITE LIST We'll leverage the thesaurus capability of Oracle Text. Using a thesaurus, we create a set of rules that are in effect our mapping from single and multi-word terms to the tokens used to represent those terms. For example, "Oracle Data Mining" becomes "ORACLEDATAMINING." First, we'll create and populate a mapping table called my_term_token_map. All text has been converted to upper case and values in the TERM column are intended to be mapped to the token in the TOKEN column. TERM                                TOKEN DATA MINING                         DATAMINING ORACLE DATA MINING                  ORACLEDATAMINING 11G                                 ORACLE11G JAVA                                JAVA CRM                                 CRM CUSTOMER RELATIONSHIP MANAGEMENT    CRM ... Next, we'll create a thesaurus object my_thesaurus and a rules table my_thesaurus_rules: CTX_THES.CREATE_THESAURUS('my_thesaurus', FALSE); CREATE TABLE my_thesaurus_rules (main_term     VARCHAR2(100),                                  query_string  VARCHAR2(400)); We next populate the thesaurus object and rules table using the term token map. A cursor is defined over my_term_token_map. As we iterate over  the rows, we insert a synonym relationship 'SYN' into the thesaurus. We also insert into the table my_thesaurus_rules the main term, and the corresponding query string, which specifies synonyms for the token in the thesaurus. DECLARE   cursor c2 is     select token, term     from my_term_token_map; BEGIN   for r_c2 in c2 loop     CTX_THES.CREATE_RELATION('my_thesaurus',r_c2.token,'SYN',r_c2.term);     EXECUTE IMMEDIATE 'insert into my_thesaurus_rules values                        (:1,''SYN(' || r_c2.token || ', my_thesaurus)'')'     using r_c2.token;   end loop; END; We are effectively inserting the token to return and the corresponding query that will look up synonyms in our thesaurus into the my_thesaurus_rules table, for example:     'ORACLEDATAMINING'        SYN ('ORACLEDATAMINING', my_thesaurus)At this point, we create a CTXRULE index on the my_thesaurus_rules table: create index my_thesaurus_rules_idx on        my_thesaurus_rules(query_string)        indextype is ctxsys.ctxrule; In my next post, this index will be used to extract the tokens that match each of the rules specified. We'll then compute the tf-idf weights for each of the terms and create a nested table suitable for mining.

  Read the article

• Lot's of errors by insserv on apt-get operations after trying to install tomcat

  - by yankee

  I wanted to install tomcat on my Debian 6.0.4 machine. I tried apt-get install tomcat6-user which worked fine. But then I changed my mind about the user installation and wanted to install the package tomcat6. This resulted in a bunch of errors (see below). Now whatever I try to do with apt-get or with aptitude (trying to remove tomcat6-user, trying to remove tomcat6, trying to perform an apt-get upgrade,...) just results in the same list of errors. How did I manage that? And how can I fix it? # apt-get install tomcat6 Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: authbind Suggested packages: tomcat6-docs tomcat6-admin tomcat6-examples libtcnative-1 The following NEW packages will be installed: authbind tomcat6 0 upgraded, 2 newly installed, 0 to remove and 32 not upgraded. Need to get 56.6 kB of archives. After this operation, 442 kB of additional disk space will be used. Do you want to continue [Y/n]? Get:1 http://mirror.hetzner.de/debian/packages/ squeeze/main authbind amd64 1.2.0 [17.3 kB] Get:2 http://mirror.hetzner.de/debian/security/ squeeze/updates/main tomcat6 all 6.0.35-1+squeeze2 [39.3 kB] Fetched 56.6 kB in 0s (441 kB/s) Preconfiguring packages ... Selecting previously deselected package authbind. (Reading database ... 34717 files and directories currently installed.) Unpacking authbind (from .../authbind_1.2.0_amd64.deb) ... Selecting previously deselected package tomcat6. Unpacking tomcat6 (from .../tomcat6_6.0.35-1+squeeze2_all.deb) ... Processing triggers for man-db ... Setting up authbind (1.2.0) ... Setting up tomcat6 (6.0.35-1+squeeze2) ... Creating config file /etc/default/tomcat6 with new version Adding system user `tomcat6' (UID 108) ... Adding new user `tomcat6' (UID 108) with group `tomcat6' ... Not creating home directory `/usr/share/tomcat6'. insserv: warning: script 'S99iptables-custom' missing LSB tags and overrides insserv: warning: script 'iptables-custom' missing LSB tags and overrides insserv: There is a loop at service iptables-custom if started insserv: There is a loop between service rmnologin and mountnfs if started insserv: loop involving service mountnfs at depth 6 insserv: loop involving service networking at depth 5 insserv: loop involving service kbd at depth 9 insserv: There is a loop between service rmnologin and mountall-bootclean if started insserv: loop involving service mountall-bootclean at depth 5 insserv: loop involving service mountall at depth 4 insserv: There is a loop between service iptables-custom and lvm2 if started insserv: loop involving service lvm2 at depth 2 insserv: loop involving service udev at depth 1 insserv: There is a loop at service rmnologin if started insserv: There is a loop between service iptables-custom and checkroot if started insserv: loop involving service checkroot at depth 2 insserv: loop involving service keyboard-setup at depth 1 insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Max recursions depth 99 reached insserv: loop involving service courier-imap-ssl at depth 1 insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: loop involving service hwclockfirst at depth 2 insserv: loop involving service mountoverflowtmp at depth 9 insserv: loop involving service checkfs at depth 6 insserv: loop involving service mdadm-raid at depth 4 insserv: loop involving service hostname at depth 3 insserv: There is a loop between service iptables-custom and ifupdown-clean if started insserv: loop involving service ifupdown-clean at depth 5 insserv: There is a loop between service rmnologin and mountall if started insserv: There is a loop between service iptables-custom and mountdevsubfs if started insserv: loop involving service mountdevsubfs at depth 1 insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: loop involving service mtab at depth 6 insserv: There is a loop between service rmnologin and mountoverflowtmp if started insserv: Starting iptables-custom depends on rmnologin and therefore on system facility `$all' which can not be true! insserv: exiting now without changing boot order! update-rc.d: error: insserv rejected the script header dpkg: error processing tomcat6 (--configure): subprocess installed post-installation script returned error exit status 1 configured to not write apport reports Errors were encountered while processing: tomcat6 E: Sub-process /usr/bin/dpkg returned an error code (1)

  Read the article

• ADO.NET (WCF) Data Services Query Interceptor Hangs IIS

  - by PreMagination

  I have an ADO.NET Data Service that's supposed to provide read-only access to a somewhat complex database. Logically I have table-per-type (TPT) inheritance in my data model but the EDM doesn't implement inheritance. (Limitation of EF and navigation properties on derived types. STILL not fixed in EF4!) I can query my EDM directly (using a separate project) using a copy of the query I'm trying to run against the web service, results are returned within 10 seconds. Disabling the query interceptors I'm able to make the same query against the web service, results are returned similarly quickly. I can enable some of the query interceptors and the results are returned slowly, up to a minute or so later. Alternatively, I can enable all the query interceptors, expand less of the properties on the main object I'm querying, and results are returned in a similar period of time. (I've increased some of the timeout periods) Up til this point Sql Profiler indicates the slow-down is the database. (That's a post for a different day) But when I enable all my query interceptors and expand all the properties I'd like to have the IIS worker process pegs the CPU for 20 minutes and a query is never even made against the database. This implies to me that yes, my implementation probably sucks but regardless the Data Services "tier" is having an issue it shouldn't. WCF tracing didn't reveal anything interesting to my untrained eye. Details: Data model: Agent-Person-Student Student has a collection of referrals Students and referrals are private, queries against the web service should only return "your" students and referrals. This means Person and Agent need to be filtered too. Other entities (Agent-Organization-School) can be accessed by anyone who has authenticated. The existing security model is poorly suited to perform this type of filtering for this type of data access, the query interceptors are complicated and cause EF to generate some entertaining sql queries. Sample Interceptor [QueryInterceptor("Agents")] public Expression<Func<Agent, Boolean>> OnQueryAgents() { //Agent is a Person(1), Educator(2), Student(3), or Other Person(13); allow if scope permissions exist return ag => (ag.AgentType.AgentTypeId == 1 || ag.AgentType.AgentTypeId == 2 || ag.AgentType.AgentTypeId == 3 || ag.AgentType.AgentTypeId == 13) && ag.Person.OrganizationPersons.Count<OrganizationPerson>(op => op.Organization.ScopePermissions.Any<ScopePermission> (p => p.ApplicationRoleAccount.Account.UserName == HttpContext.Current.User.Identity.Name && p.ApplicationRoleAccount.Application.ApplicationId == 124) || op.Organization.HierarchyDescendents.Any<OrganizationsHierarchy>(oh => oh.AncestorOrganization.ScopePermissions.Any<ScopePermission> (p => p.ApplicationRoleAccount.Account.UserName == HttpContext.Current.User.Identity.Name && p.ApplicationRoleAccount.Application.ApplicationId == 124))) > 0; } The query interceptors for Person, Student, Referral are all very similar, ie they traverse multiple same/similar tables to look for ScopePermissions as above. Sample Query var referrals = (from r in service.Referrals .Expand("Organization/ParentOrganization") .Expand("Educator/Person/Agent") .Expand("Student/Person/Agent") .Expand("Student") .Expand("Grade") .Expand("ProblemBehavior") .Expand("Location") .Expand("Motivation") .Expand("AdminDecision") .Expand("OthersInvolved") where r.DateCreated >= coupledays && r.DateDeleted == null select r); Any suggestions or tips would be greatly associated, for fixing my current implementation or in developing a new one, with the caveat that the database can't be changed and that ultimately I need to expose a large portion of the database via a web service that limits data access to the data authorized for, for the purpose of data integration with multiple outside parties. THANK YOU!!!

  Read the article

• Texture switching with a entity system

  - by GameDev-er

  I'm using thinking of using an entity system in my game. So far I've been using Artemis with success. However, I have a question about texture switching. I read that switching textures too often is bad. So I load all the textures when the game loads like so: import org.newdawn.slick.opengl.TextureLoader; ... public HashMap<String, Texture> Textures; ... Then for each texture I do this: Texture tex = TextureLoader.getTexture("PNG", this.getClass().getResourceAsStream(texturePath)); Textures.put(textureName, tex); Then when drawing entities I do this: drawEntity() { glBindTexture(GL_TEXTURE_2D, Textures.get(entityTexture).getTextureID()); ... } Say I have 50 entities, using 10 different 3D models, each with their own texture. When the drawEntity system runs, it doesn't group by which entities use which texture. So I could be switching textures before drawing each entity! Is there a more efficient way to switch textures between entities? Or is glBindTexture() a good option?

  Read the article

• Oracle Insurance Gets Innovative with Insurance Business Intelligence

  - by nicole.bruns(at)oracle.com

  Oracle Insurance announced yesterday the availability of Oracle Insurance Insight 7.0, an insurance-specific data warehouse and business intelligence (BI) system that transforms the traditional approach to BI by involving business users in the creation and maintenance."Rapid access to business intelligence is essential to compete and thrive in today's insurance industry," said Srini Venkatasantham, vice president, Product Strategy, Oracle Insurance. "The adaptive data modeling approach of Oracle Insurance Insight 7.0, combined with the insurance-specific data model, offers global insurance companies a faster, easier way to get the intelligence they need to make better-informed business decisions." New Features in Oracle Insurance 7.0 include:"Adaptive Data Modeling" via the new warehouse palette: Gives business users the power to configure lines of business via an easy-to-use warehouse palette tool. Oracle Insurance Insight then automatically creates data warehouse elements - such as line-specific database structures and extract-transform-load (ETL) processes -speeding up time-to-value for BI initiatives. Out-of-the-box insurance models or create-from-scratch option: Includes pre-built content and interfaces for six Property and Casualty (P&C) lines. Additionally, insurers can use the warehouse palette to deploy any and all P&C or General Insurance lines of business from scratch, helping insurers support operations in any country.Leverages Oracle technologies: In addition to Oracle Business Intelligence Enterprise Edition, the solution includes Oracle Database 11g as well as Oracle Data Integrator Enterprise Edition 11g, which delivers Extract, Load and Transform (E-L-T) architecture and eliminates the need for a separate transformation server. Additionally, the expanded Oracle technology infrastructure enables support for Oracle Exadata. Martina Conlon, a Principal with Novarica's Insurance practice, and author of Business Intelligence in Insurance: Current State, Challenges, and Expectations says, "The need for continued investment by insurers in business intelligence capabilities is widely understood, and the industry is acting. Arming the business intelligence implementation with predefined insurance specific content, and flexible and configurable technology will get these projects up and running faster."Learn moreTo see a demo of the Oracle Insurance Insight system, click hereTo read the press announcement, click here

  Read the article

• make-like build tools for data?

  - by miku

  Make is a standard tools for building software. But make decides whether a target needs to be regenerated by comparing file modification times. Are there any proven, preferably small tools that handle builds not for software but for data? Something that regenerates targets not only on mod times but on certain other properties (e.g. completeness). (Or alternatively some paper that describes such a tool.) As illustration: I'd like to automate the following process: get data (e.g. a tarball) from some regularly updated source copy somewhere if it's not there (based e.g. on some filename-scheme) convert the files to different format (but only if there aren't successfully converted ones there - e.g. from a previous attempt - custom comparison routine) for each file find a certain data element and fetch some additional file from say an URL, but only if that hasn't been downloaded yet (decide on existence of file and file "freshness") finally compute something (e.g. word count for something identifiable and store it in the database, but only if the DB does not have an entry for that exact ID yet) Observations: there are different stages each stage is usually simple to compute or implement in isolation each stage may be simple, but the data volume may be large each stage may produce a few errors each stage may have different signals, on when (re)processing is needed Requirements: builds should be interruptable and idempotent (== robust) when interrupted, already processed objects should be reused to speedup the next run data paths should be easy to adjust (simple syntax, nothing new to learn, internal dsl would be ok) some form of dependency graph, that describes the process would be nice for later visualizations should leverage existing programs, if possible I've done some research on make alternatives like rake and have worked a lot with ant and maven in the past. All these tools naturally focus on code and software build, not on data builds. A system we have in place now for a task similar to the above is pretty much just shell scripts, which are compact (and are a ok glue for a variety of other programs written in other languages), so I wonder if worse is better?

  Read the article

• SQL SERVER – Installing Data Quality Services (DQS) on SQL Server 2012

  - by pinaldave

  Data Quality Services is very interesting enhancements in SQL Server 2012. My friend and SQL Server Expert Govind Kanshi have written an excellent article on this subject earlier on his blog. Yesterday I stumbled upon his blog one more time and decided to experiment myself with DQS. I have basic understanding of DQS and MDS so I knew I need to start with DQS Client. However, when I tried to find DQS Client I was not able to find it under SQL Server 2012 installation. I quickly realized that I needed to separately install the DQS client. You will find the DQS installer under SQL Server 2012 >> Data Quality Services directory. The pre-requisite of DQS is Master Data Services (MDS) and IIS. If you have not installed IIS, you can follow the simple steps and install IIS in your machine. Once the pre-requisites are installed, click on MDS installer once again and it will install DQS just fine. Be patient with the installer as it can take a bit longer time if your machine is low on configurations. Once the installation is over you will be able to expand SQL Server 2012 >> Data Quality Services directory and you will notice that it will have a new item called Data Quality Client.  Click on it and it will open the client. Well, in future blog post we will go over more details about DQS and detailed practical examples. Reference: Pinal Dave (http://blog.sqlauthority.com) Filed under: PostADay, SQL, SQL Authority, SQL Query, SQL Server, SQL Tips and Tricks, SQL Utility, T SQL, Technology Tagged: Data Quality Services

  Read the article

< Previous Page | 7 8 9 10 11 12 13 14 15 16 17 18  | Next Page >