Search Results

Search found 58774 results on 2351 pages for 'ssis data tranformations'.

Page 15/2351 | < Previous Page | 11 12 13 14 15 16 17 18 19 20 21 22  | Next Page >

  • I need some help creating a non-binary tree (or some other data structure that will better solve my problem)

    - by EDO
    I have about ten lists of numbers and some strings. Each list has about <= 30K lines. Each line on a list has a distinct number. I need to build an efficient way of finding all the lines in each list that has the same 'control' number (or key for dB guys) and comparing what is in their string parts. I am writing this in Java. I have thought about using trees but my brain cells are about burnt now. I need some help.

    Read the article

  • replacing data.frame element-wise operations with data.table (that used rowname)

    - by Harold
    So lets say I have the following data.frames: df1 <- data.frame(y = 1:10, z = rnorm(10), row.names = letters[1:10]) df2 <- data.frame(y = c(rep(2, 5), rep(5, 5)), z = rnorm(10), row.names = letters[1:10]) And perhaps the "equivalent" data.tables: dt1 <- data.table(x = rownames(df1), df1, key = 'x') dt2 <- data.table(x = rownames(df2), df2, key = 'x') If I want to do element-wise operations between df1 and df2, they look something like dfRes <- df1 / df2 And rownames() is preserved: R> head(dfRes) y z a 0.5 3.1405463 b 1.0 1.2925200 c 1.5 1.4137930 d 2.0 -0.5532855 e 2.5 -0.0998303 f 1.2 -1.6236294 My poor understanding of data.table says the same operation should look like this: dtRes <- dt1[, !'x', with = F] / dt2[, !'x', with = F] dtRes[, x := dt1[,x,]] setkey(dtRes, x) (setkey optional) Is there a more data.table-esque way of doing this? As a slightly related aside, more generally, I would have other columns such as factors in each data.table and I would like to omit those columns while doing the element-wise operations, but still have them in the result. Does this make sense? Thanks!

    Read the article

  • Reduce Multiple Errors logging in sysssislog

    - by Akshay
    Need help. I am trying to automate error notifications to be sent in mailers. For that I am querying the sysssislog table. I have pasted an "Execute SQl task" on the package event handler "On error". For testing purpose, I am deliberately trying to load duplicate keys in a table which consists of a Primary key column(so as to get an error). But instead of having just one error, "Violation of primary key constraint", SSIS records 3 in the table. PFA the screenshot as well. How can i restrict the tool to log only one error and not multiple ??? Package Structure. Package ("On error Event handler") - DFT - Oledb Source - Oledb Destination SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005. An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "The statement has been terminated.". An OLE DB record is available. Source: "Microsoft SQL Server Native Client 10.0" Hresult: 0x80004005 Description: "Violation of PRIMARY KEY constraint 'PK_SalesPerson_SalesPersonID'. Cannot insert duplicate key in object 'dbo.SalesPerson'.". SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "input "OLE DB Destination Input" (56)" failed because error code 0xC020907B occurred, and the error row disposition on "input "OLE DB Destination Input" (56)" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure. SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "OLE DB Destination" (43) failed with error code 0xC0209029 while processing input "OLE DB Destination Input" (56). The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure. Please guide me. Your help is very much appreciated. Thanks

    Read the article

  • PHP - post data ends when '&' is in data.

    - by Phil Jackson
    Hi all, im posting data using jquery/ajax and PHP at the backend. Problem being, when I input something like 'Jack & Jill went up the hill' im only recieving 'Jack' when it gets to the backend. I have thrown an error at the frontend before that data is sent which alerts 'Jack & Jill went up the hill'. When I put die(print_r($_POST)); at the very top of my index page im only getting [key] => Jack how can I be loosing the data? I thought It may have been my filter; <?php function filter( $data ) { $data = trim( htmlentities( strip_tags( mb_convert_encoding( $data, 'HTML-ENTITIES', "UTF-8") ) ) ); if ( get_magic_quotes_gpc() ) { $data = stripslashes( $data ); } //$data = mysql_real_escape_string( $data ); return $data; } echo "<xmp>" . filter("you & me") . "</xmp>"; ?> but that returns fine in the test above you &amp; me which is in place after I added die(print_r($_POST));. Can anyone think of how and why this is happening? Any help much appreciated. Regards, Phil.

    Read the article

  • Handling primary key duplicates in a data warehouse load

    - by Meff
    I'm currently building an ETL system to load a data warehouse from a transactional system. The grain of my fact table is the transaction level. In order to ensure I don't load duplicate rows I've put a primary key on the fact table, which is the transaction ID. I've encountered a problem with transactions being reversed - In the transactional database this is done via a status, which I pick up and I can work out if the transaction is being done, or rolled back so I can load a reversal row in the warehouse. However, the reversal row will have the same transaction ID and so I get a primary key violation. I've solved this for now by negating the primary key, so transaction ID 1 would be a payment, and transaction ID -1 (In the warehouse only) would be the reversal. I have considered an alternative of generating a BIT column, where 0 is normal and 1 is reversal, then making the PK the transaction ID and the BIT column. My question is, is this a good practice, and has anyone else encountered anything like this? For reference, this is a payment processing system, so values will not be modified, so there will only ever be transactions and reversals.

    Read the article

  • SSIS 2008 Rows per batch and Maximum insert commit size

    - by Nissan Fan
    I've got about 100 million rows that I'm moving in SSIS 2008 via a Data Flow Task. It's pretty much a straight table data copy using a Multicast. My question is this: Using the OLE DB Destination Editor I have two options: Rows per batch and Maximum insert commit size. What are good settings for this? I've only been able to find that you are recommended to set Maximum insert commit size to 2147483647 instead of 0, but then tweak both these settings based on testing. I'm curious to find out if anyone has discovered anything useful in their own management of these values.

    Read the article

  • SSIS Configuration error: Cannot retrieve configuration table schema

    - by Glenn M
    I'm trying to add a simple configuration to a SSIS package, of type SQL Server, so stored in a table. At the end of the wizard, when it goes to try and write a new row to the nominated table to store the configuration it fails with the error: TITLE: Microsoft Visual Studio Could not complete wizard actions. Cannot retrieve configuration table schema. (Microsoft.DataTransformationServices.Wizards) I can't seem to resolve this. The configuration connection has full permissions on the table, and it sees it and can read from it as it reports there is no current data for the filter I provide. It just wont write to it. A Google search of the error message above in quotes returns literally no hits! Any suggestions? Glenn

    Read the article

  • Using npgsql with SSIS

    - by Flash
    Hi, I am using npgsql as the ADO.NET provider in SSIS to connect to Postgres and create some workflows. I am able to connect to postgres but am unable to use the "Table or view" data access mode to list the tables and views. I have to resort to using "SQL command" data access mode. For the "table or view" access mode, the log shows that the last call is GetDataSourceInformation and it returns successfully. After that there is a close connection. Has anyone successfully used "Table or view" data access mode using npgsql? Thanks, Flash

    Read the article

  • Modify the Event Log Source name for an SSIS package

    - by Paul Kohler
    I have an SQL Server integration Services (SSIS) package using the standard Event Log provider (yes, the event log! I know we can use SQL etc...) The default "Source" of the log events is "SQLISPackage100" but I want it to be something like "AppName" so that the errors are more visible between the different packages when viewing the event log (also for MOM use). Event Type: Error Event Source: SQLISPackage100 Event Category: None ... Description: Package "Foo" failed. I hope the answer is simple, I simply cannot find it, but does anyone know how to change the text of the Event Log Source name to something more meaningful? PK :-)

    Read the article

  • SSIS - Wizard vs manual vs programming

    - by alchemical
    I'd like to move 26 tables from one DB to another. I see I can do this in the SSIS Import and Export Wizard. I believe the other approach would be to select tools from the toolbar in Data Flow and then configure them all. When is it better to use the wizard and when is it best to create the package manually (with the visual tools) or programmatically? One thing I noticed with the Wizard is that it lets me select multiple tables at once, but I could not find a way to get back to that screen once the package is created, so that I could edit the various tables all in one place.

    Read the article

  • SSIS DTSX File Repair Tool

    - by Eric Ness
    I'm working with an SSIS 2005 file that crashes Visual Studio 2005 on my workstation. This happens when I open the data flow diagram and Visual Studio attempts to validate the package. I can open it successfully on another computer though. The package itself is fairly simple and only has two control flow tasks and maybe ten tasks in the data flow. I'm wondering if there is a tool that goes through the XML in the dtsx file and repairs any issues or if this is even necessary. The dtsx file is about 171 kB and it seems like there's a lot in it considering what a simple package it is.

    Read the article

  • Extracting data from multiple servers SQL 2005 SSIS

    - by Raj
    I have created an SSIS package to connect to multiple SQL servers, create a database, a table and a stored procedure. The package also creates a job and schedules it to run every 5 minutes. The requirement is to collect performance metrics. I am using an ado object variable to get the server names and all the above tasks are in a for each loop and everything works fine. Now the problem: I need to create a data flow task, which will connect to each of these servers in turn, copy the performance metrics data over to a central server and purge the source table. I am unable to get this task to work. This task fails with "Unable to obtain Connection" error. Any help will be greatly appreciated. SQL Server Version : 2005 Thanks, Raj

    Read the article

  • SSIS(sql server integration service) xml data flow

    - by swapna
    Hi, I have an xml file the content which i have to write to a Database table using ssis pacakge. I am using xml source nad oledb destination My issue now is this xml file generate multiple outputs .(event,produt,offer,form) etc. But i need to write all in one data row(more than one if 2 products are there for the event) in the database. But i do not know how to use this multiple outputs and make a single row for a event. I hav read numerous articles about this subject but not able to take a decision.what is the right way of doing this. 1) xml source ? (if i use this how do i merge the multiple outputs) 2) or a script task using xml objects read and write to the DB. or anything new ? Please provide me some solutions xml sample file * - ABc. 2009-06-07 2010-04-30 region test 1 contact - offertest product1 product1 187 * Thanks SNA

    Read the article

  • Nonstandard SSIS lookup

    - by Stefan
    I have a situation where I am trying to lookup a value in one table based on values in another table, using a BETWEEN operator and not an = operator. In one table, I have a value "EffectiveDate". I want to get a Weight number from another table, but the other table has two fields: "Inception" and "Termination". What I want to do is extract the Weight from that table for use where the EffectiveDate is between Inception and Termination. SSIS doesn't seem to provide a way to do this. It's good at matching one column to another column, but doesn't seem to allow one to many-column comparison/operations. Am I missing anything? Is this possible to do somehow?

    Read the article

  • ssis Connection manager password ("DontSaveSensitive")

    - by swapna
    Hi, I have a SSIS package used the protection level as "DontSaveSensitive" So initially i have saved the oledb connection manager password as in a XML config file.And this worked fine for me. Now for business requiremnt i have stored the configuration in sql server. Including password. Initially it worked fine.But suddenly connection manager is not taking password from sqlDB. If i set an environment variable for the "password" it's working. My question is. 1) what is the solution for this.( i can only save the configurations in sql) 2) initially how it worked.I mean initailly from sql server the password was taking. Please give some clue.I am doing a lot of trial and error. Thanks SNA

    Read the article

  • SSIS - Update flag of selected rows from more than one table

    - by Rob Bowman
    Hi I have a SSIS package that copies data from table A to table B and sets a flag in table A so that the same data is not copied subsequently. This works great by using the following as the SQL command text on the ADO Net Source object: update transfer set ProcessDateTimeStamp = GetDate(), LastUpdatedBy = 'legacy processed' output inserted.* where LastUpdatedBy = 'legacy' and ProcessDateTimeStamp is not null The problem I have is that I need to run a similar data copy but from two sources table, joined on a primary / foreign key - select from table A join table B update flag in table A. I don't think I can use the technique above because I don't know where I'd put the join! Is there another way around this problem? Thanks Rob.

    Read the article

  • SSIS Custom Control Task Debugging UI in BIDS and VS

    - by zeencat
    I've created a SSIS Custom Task in C# and I'm currently developing the UI. I was wondering if there is a better way of debugging the UI instead of compiling the project, copying the DLL's into the appropriate DTS folder and then opening the test Package within BIDS and then attaching the process to Visual Studio. This part I'm not bothered about but once you've tested the UI and made changes to UI within Visual Studio. I've got to recomplile the DLL's and then repeat the entire process. I've got to close BIDS and VS because they don't release the DLL's before I have to start the entire process over again. Does anyone have any tips to speed up this process. It's just so frustrating having to do this everytime.

    Read the article

  • discovering files in the FileSystem, through SSIS

    - by cometbill
    I have a folder where files are going to be dropped for importing into my data warehouse. \\server\share\loading_area I have the following (inherited) code that uses xp_cmdshell shivers to call out to the command shell to run the DIR command and insert the resulting filenames into a table in SQL Server. I would like to 'go native' and reproduce this functionality in SSIS. Thanks in advance guys and girls. Here's the code USE MyDatabase GO declare @CMD varchar(500) declare @EXTRACT_PATH varchar(255) set @EXTRACT_PATH = '\\server\share\folder\' create table tmp_FILELIST([FILENUM] int identity(1,1), [FNAME] varchar(100), [FILE_STATUS] varchar(20) NULL CONSTRAINT [DF_FILELIST_FILE_STATUS] DEFAULT ('PENDING')) set @CMD = 'dir ' + @EXTRACT_PATH + '*.* /b /on' insert tmp_FILELIST([FNAME]) exec master..xp_cmdshell @CMD --remove the DOS reply when the folder is empty delete tmp_FILELIST where [FNAME] is null or [FNAME] = 'File Not Found' --Remove my administrative and default/common, files not for importing, such as readme.txt delete tmp_FILELIST where [FNAME] is null or [FNAME] = 'readme.txt'

    Read the article

  • SSIS Script Component, Allow Null values

    - by user2471943
    I have a SSIS package that I am programming and my script component won't allow null column inputs. I have checked the box to keep nulls in the flat file source component. My program is running well until my script component where I get the error "The column has a null value" (super vague, I know). The column currently throwing the error is an "int" valued column and is used for aggregations in my script. I could make the null values 0s or to say "NULL" but I'd prefer to just leave them blank. Any advice on how to handle this problem would be greatly appreciated! Thanks in advance! I am using SQL Server BIDS 2008.

    Read the article

  • SSIS - Limiting Concurrent Connections

    - by Bigtoe
    Hi Folks, I am using SSIS to connect to a legecy mainframe database and this allows only 5 concurrent connections at a time. I have a dataflow task with many tables to transfer and it kicks outs because of this limitation. I have split up the Data Flow task into seperate data flows and this is working for the moment, but it is not optiomal as they need to be sequenced and 1 large transfer in a flow is holding up subsequent transfers. Anyone any idea of how to limit the number of connections in a single data flow, I had a look at using the Engine Threads but this did not make any difference. Any help much appericated.

    Read the article

  • SSIS web service task producing "object reference" error

    - by gfrizzle
    Our care management system uses a web service to import data. I've successfully executed one of its methods using soapUI, and now I want to replicate this with an SSIS 2008 web service task, but I'm running into a problem. I've created the "HTTP Connection Manager" successfully, and specified the location of the "WSDLFile", but when I go to the Input tab and select the Service from the dropdown (there is only one), I get an "Object reference not set to an instance of an object" error, and and the Method dropdown is empty. Any idea what this is trying to tell me?

    Read the article

  • SSIS - Connection Management Within a Loop

    - by Rob Bowman
    Hi I have the following SSIS package: The problem is that within the Foreach loop a connection is opened and closed for each iteration. On running SQL Profiler I see a series of: Audit Login RPC:Completed Audit Lout The duration for the login and the RPC that actually does the work is minimal. However, the duration for the logout is significant, running into several seconds each. This causes the JOB to run very slowly - taking many hours. I get the same problem when running either on a test server or stand-alone laptop. Could anyone please suggest how I may change the package to improve performance? Also, I have noticed that when running the package from Visual Studio, it looks as though it continues to run with the component blocks going amber then green but actually all the processing has been completed and SQL profiler has dropped silent? Thanks, Rob.

    Read the article

  • Microsoft SQL Server 2005/2008 SSIS are oversized

    - by Ice
    In this case i'm old style and loved 'my fathers DTS' from SQL 2000. Most of the cases i have to import a flatfile into a table. In a second step i use some procedures (with the new MERGE-Statement) to process the imported content. For Export, i define a export-table and populate it with a store proc (containing a MERGE-Statement) and in a second step the content will be exported to a flat file. In some cases there is no flat file because there is annother sql-server or in rare cases an ODBC-Connection to a sybase or similar. What do you think? When it comes to complex ETL-Stuff the SSIS may be the right tool...but i haven't seen such a case yet.

    Read the article

  • How to prevent CAST errors on SSIS ?

    - by manitra
    Hello, The question Is it possible to ask SSIS to cast a value and return NULL in case the cast is not allowed instead of throwing an error ? My environment I'm using Visual Studio 2005 and Sql Server 2005 on Windows Server 2003. The general context Just in case you're curious, here is my use case. I have to store data coming from somewhere in a generic table (key/value structure with history) witch contains some sort of value that can be strings, numbers or dates. The structure is something like this : table Values { Id int, Date datetime, -- for history Key nvarchar(50) not null, Value nvarchar(50), DateValue datetime, NumberValue numeric(19,9) } I want to put the raw value in the Value column and try to put the same value in the DateValue column when i'm able to cast it to Datetime in the NumberValue column when i'm able to cast it to a number Those two typed columns would make all sort of aggregation and manipulation much easier and faster later. That's it, now you know why i'm asking this strange question. ============ Thanks in advance for your help.

    Read the article

  • check if lookup yields any valid rows for insertion before clearing table using ssis

    - by Chris
    SSIS ignoramus needing help! the situation: a temp table is populated from an excel file, which has been known to change formats at random times, that is owned by a different group. a lookup need to be performed on the temp table, tableA, to populate tableB with valid data. if the lookup results in 0 rows being returned, an email should be sent and the existing data in tableB should remain untouched. If the lookup results in a number of valid rows 0, tableB should have all rows deleted and the new records from the lookup on tableA inserted. question: what would be the best way to check if there are any valid rows and perform the appropriate action(s), depending on my results? Thanks!

    Read the article

< Previous Page | 11 12 13 14 15 16 17 18 19 20 21 22  | Next Page >