Search Results

Search found 1221 results on 49 pages for 'ssis pivot'.

Page 28/49 | < Previous Page | 24 25 26 27 28 29 30 31 32 33 34 35  | Next Page >

  • SQL Aggregate all Purchases for a certain product with same rebatecode

    - by debuggerlikeanother
    Hi SO, i would like to aggregate all purchases for a certain product that used the same rebatecode (using SQL Server 2005) Assume we have the following table: ID ProductID Product RebateCode Amount 1 123 7HM ABC 1 2 123 7HM XYZ 2 3 124 7HM ABC 10 4 124 7HM XYZ 20 5 125 2EB LOI 4 6 126 2EB LOI 40 CREATE TABLE #ProductSales(ID SMALLINT, ProductID int, Product varchar(6), RebateCode varchar(4), Amount int) GO INSERT INTO #ProductSales select 1, 123, '7HM', 'A', 1 union all select 2, 123, '7HM', 'B', 2 union all select 3, 124, '7HM', 'A', 10 union all select 4, 124, '7HM', 'B', 20 union all select 5, 125, '7HM', 'A', 100 union all select 6, 125, '7HM', 'B', 200 union all select 7, 125, '7HM', 'C', 3 union all select 8, 126, '2EA', 'E', 4 union all select 8, 127, '2EA', 'E', 40 union all select 9, 128, '2EB', 'F', 5 union all select 9, 129, '2EB', 'F', 50 union all select 10, 130, '2EB', 'F', 500 GO SELECT * FROM #ProductSales GO /* And i would like to have the following result Product nrOfProducts CombinationRebateCode SumAmount ABC LOI XYZ 7HM 2 ABC, XYZ 33 11 0 22 2EB 2 LOI 44 0 44 0 .. */ CREATE TABLE #ProductRebateCode(Product varchar(6), nrOfProducts int, sumAmountRebateCombo int, rebateCodeCombination varchar(80), A int, B int, C int, E int, F int) Go INSERT INTO #ProductRebateCode select '7HM', 2, 33, 'A, B', 2, 2, 0, 0, 0 union all select '7HM', 1, 303, 'A, B, C', 1, 1, 1, 0, 0 union all select '2EA', 2, 44, 'E', 0, 0, 0, 2, 0 union all select '2EB', 3, 555, 'E', 0, 0, 0, 0, 2 Select * from #ProductRebateCode -- Drop Table #ProductSales IF EXISTS ( SELECT * FROM tempdb.dbo.sysobjects WHERE name LIKE '#ProductSales%') DROP TABLE #ProductSales -- Drop Table #ProductRebateCode IF EXISTS ( SELECT * FROM tempdb.dbo.sysobjects WHERE name LIKE '#ProductRebateCode%') DROP TABLE #ProductRebateCode I would like to have the result like in the example (see second select (#ProductRebateCode). I tried to achieve it with the crosstab from this post: http://www.sqlteam.com/forums/topic.asp?TOPIC_ID=6216&whichpage=6. exec CrossTab2b @SQL = 'SELECT [ProductID], Product, RebateCode, Amount FROM #ProductSales' ,@PivotCol = 'RebateCode' ,@Summaries = 'Sum(Amount ELSE 0)[_Sum], Count([ProductID])[_nrOfProducts]' /* SUM(Amount ELSE 0)[Amount], COUNT(Amount)[Qty] */ ,@GroupBy = 'RebateCode, Product' ,@OtherFields = 'Product' I believe that this could work, but i am unable to solve it. Do you believe that it is possible to achieve what i am trying without MDX or the other fancy ?DX-Stuff? Best regards And Thanks a lot debugger the other

    Read the article

  • SQL Server pivots? some way to set column names to values within a row

    - by ccsimpson3
    I am building a system of multiple trackers that are going to use a lot of the same columns so there is a table for the trackers, the tracker columns, then a cross reference for which columns go with which tracker, when a user inserts a tracker row the different column values are stored in multiple rows that share the same record id and store both the value and the name of the particular column. I need to find a way to dynamically change the column name of the value to be the column name that is stored in the same row. i.e. id | value | name ------------------ 23 | red | color 23 | fast | speed needs to look like this. id | color | speed ------------------ 23 | red | fast Any help is greatly appreciated, thank you.

    Read the article

  • How to convert rows returned by a query into columns in oracle?

    - by Piyush Lohana
    I have to display the results of the below query as columns. select to_char(sysdate + 1 - rownum,'MON-YYYY') as d from all_objects where trunc(sysdate + 1 - rownum,'MM') = trunc(to_date(:from_date,'MON-YYYY'),'MM') minus select to_char(sysdate + 1 - rownum,'MON-YYYY') as d from all_objects where trunc(sysdate + 1 - rownum,'MM') trunc(to_date(':to_date','MON-YYYY'),'MM') Please help me in figuring that out.

    Read the article

  • How can I "merge" or "flatten" results from a query which returns multiple rows into a single result

    - by dsm
    I have a simple query over a table, which returns results like the following: id id_type id_ref 2702 5 31 2702 16 14 2702 17 3 2702 40 1 2702 26 4 And I would like to merge the results into a single row, for instance: id concatenation 2702 5,16,17,40,26:31,14,3,1,4 Is there any way to do this within a trigger? NB: I know I can use a cursor, but I would really prefer not to unless there is no better way.

    Read the article

  • In Excel 2010, how can I show a count of occurrences on a specific date within multiple time ranges?

    - by Justin
    Here's what I'm trying to do. I have three columns of data. ID, Date(MM/DD/YY), Time(00:00). I need to create a chart or table that shows the number of occurrences on, say, 12/10/2010 between 00:00 and 00:59, 1:00 and 1:59, etc, for each hour of the day. I can do countif and get results for the date, but I cannot figure out how to show a summary of the count of occurrences per hour for the 24 hour period. I have months of data and many times each day. Example of data set is below. Any help is greatly ID Date Time 221 12/10/2010 00:01 223 12/10/2010 00:45 227 12/10/2010 01:13 334 12/11/2010 14:45 I would like the results to read: Date Time Count 12/10/2010 00:00AM - 00:59AM 2 12/10/2010 01:00AM - 01:59AM 1 12/10/2010 02:00AM - 02:59AM 0 ......(continues for every hour of the day) 12/11/2010 00:00AM - 00:59AM 0 ......... 12/11/2010 14:00PM - 14:59PM 1 And so on. Sorry for the length but I wanted to be clear. EDIT Here is a sample spreadsheet. Very little data, but I couldn't figure out a better way without having a huge file. Tested in notepad for formatting and worked ok on import as csv. PID,Date,Time 2888759,12/10/2010,0:10 2888760,12/10/2010,0:10 2888761,12/10/2010,0:10 2888762,12/10/2010,0:11 2889078,12/10/2010,15:45 2889079,12/10/2010,15:57 2889080,12/10/2010,15:57 2889081,12/10/2010,15:58 2889082,12/10/2010,16:10 2889083,12/10/2010,16:11 2889084,12/10/2010,16:11 2889085,12/10/2010,16:12 2889086,12/10/2010,16:12 2889087,12/10/2010,16:12 2889088,12/10/2010,16:13 2891529,12/14/2010,16:21

    Read the article

  • join select from multiple row values?

    - by user1869132
    Two tables 1) product -------------------- id | Name | price 1 | p1 | 10 2 | p2 | 20 3 | p3 | 30 2) product_attributes: --------------------------------------------------- id | product_id | attribute_name | attribute_value --------------------------------------------------- 1 | 1 | size | 10 2 | 1 | colour | red 3 | 2 | size | 20 I need to join these two tables. In the where clause I need to match both the two rows attribute values. Is it possible to get the result based on two rows value. Here if size=10 and colour=red. Output should be 1 | p1 | 10 It could be greatly helpful to get a query for this.

    Read the article

  • Side-by-side comparison of data by month in SQL

    - by ScottR
    I have table similar to the following: Year | Product | Value 2006 A 10 2006 B 20 2006 C 30 2007 A 40 2007 B 50 2007 C 60 I would like a query that would return the following comparison Product | 2006 Value | 2007 Value A 10 40 B 20 50 C 30 60 What are the options to do so? Can it be done without joins? I'm working with DB2, but answers in all SQL types would be helpful.

    Read the article

  • "Unable to read data from the transport connection: net_io_connectionclosed." - Windows Vista Busine

    - by John DaCosta
    Unable to test sending email from .NET code in Windows Vista Business. I am writing code which I will migrate to an SSIS Package once it its proven. The code is to send an error message via email to a list of recipients. The code is below, however I am getting an exception when I execute the code. I created a simple class to do the mailing... the design could be better, I am testing functionality before implementing more robust functionality, methods, etc. namespace LabDemos { class Program { static void Main(string[] args) { Mailer m = new Mailer(); m.test(); } } } namespace LabDemos { class MyMailer { List<string> _to = new List<string>(); List<string> _cc = new List<string>(); List<string> _bcc = new List<string>(); String _msgFrom = ""; String _msgSubject = ""; String _msgBody = ""; public void test(){ //create the mail message MailMessage mail = new MailMessage(); //set the addresses mail.From = new MailAddress("[email protected]"); //set the content mail.Subject = "This is an email"; mail.Body = "this is a sample body"; mail.IsBodyHtml = false; //send the message SmtpClient smtp = new SmtpClient(); smtp.Host = "emailservername"; smtp.Port = 25; smtp.UseDefaultCredentials = true; smtp.Send(mail); } } Exception Message Inner Exception {"Unable to read data from the transport connection: net_io_connectionclosed."} Stack Trace " at System.Net.Mail.SmtpReplyReaderFactory.ProcessRead(Byte[] buffer, Int32 offset, Int32 read, Boolean readLine)\r\n at System.Net.Mail.SmtpReplyReaderFactory.ReadLines(SmtpReplyReader caller, Boolean oneLine)\r\n at System.Net.Mail.SmtpReplyReaderFactory.ReadLine(SmtpReplyReader caller)\r\n at System.Net.Mail.SmtpConnection.GetConnection(String host, Int32 port)\r\n at System.Net.Mail.SmtpTransport.GetConnection(String host, Int32 port)\r\n at System.Net.Mail.SmtpClient.GetConnection()\r\n at System.Net.Mail.SmtpClient.Send(MailMessage message)" Outer Exception System.Net.Mail.SmtpException was unhandled Message="Failure sending mail." Source="System" StackTrace: at System.Net.Mail.SmtpClient.Send(MailMessage message) at LabDemos.Mailer.test() in C:\Users\username\Documents\Visual Studio 2008\Projects\LabDemos\LabDemos\Mailer.cs:line 40 at LabDemos.Program.Main(String[] args) in C:\Users\username\Documents\Visual Studio 2008\Projects\LabDemos\LabDemos\Program.cs:line 48 at System.AppDomain._nExecuteAssembly(Assembly assembly, String[] args) at System.AppDomain.nExecuteAssembly(Assembly assembly, String[] args) at System.Runtime.Hosting.ManifestRunner.Run(Boolean checkAptModel) at System.Runtime.Hosting.ManifestRunner.ExecuteAsAssembly() at System.Runtime.Hosting.ApplicationActivator.CreateInstance(ActivationContext activationContext, String[] activationCustomData) at System.Runtime.Hosting.ApplicationActivator.CreateInstance(ActivationContext activationContext) at System.Activator.CreateInstance(ActivationContext activationContext) at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssemblyDebugInZone() at System.Threading.ThreadHelper.ThreadStart_Context(Object state) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart() InnerException: System.IO.IOException Message="Unable to read data from the transport connection: net_io_connectionclosed." Source="System" StackTrace: at System.Net.Mail.SmtpReplyReaderFactory.ProcessRead(Byte[] buffer, Int32 offset, Int32 read, Boolean readLine) at System.Net.Mail.SmtpReplyReaderFactory.ReadLines(SmtpReplyReader caller, Boolean oneLine) at System.Net.Mail.SmtpReplyReaderFactory.ReadLine(SmtpReplyReader caller) at System.Net.Mail.SmtpConnection.GetConnection(String host, Int32 port) at System.Net.Mail.SmtpTransport.GetConnection(String host, Int32 port) at System.Net.Mail.SmtpClient.GetConnection() at System.Net.Mail.SmtpClient.Send(MailMessage message) InnerException:

    Read the article

  • Using Recursive SQL and XML trick to PIVOT(OK, concat) a "Document Folder Structure Relationship" table, works like MySQL GROUP_CONCAT

    - by Kevin Shyr
    I'm in the process of building out a Data Warehouse and encountered this issue along the way.In the environment, there is a table that stores all the folders with the individual level.  For example, if a document is created here:{App Path}\Level 1\Level 2\Level 3\{document}, then the DocumentFolder table would look like this:IDID_ParentFolderName1NULLLevel 121Level 232Level 3To my understanding, the table was built so that:Each proposal can have multiple documents stored at various locationsDifferent users working on the proposal will have different access level to the folder; if one user is assigned access to a folder level, she/he can see all the sub folders and their content.Now we understand from an application point of view why this table was built this way.  But you can quickly see the pain this causes the report writer to show a document link on the report.  I wasn't surprised to find the report query had 5 self outer joins, which is at the mercy of nobody creating a document that is buried 6 levels deep, and not to mention the degradation in performance.With the help of 2 posts (at the end of this post), I was able to come up with this solution:Use recursive SQL to build out the folder pathUse SQL XML trick to concat the strings.Code (a reminder, I built this code in a stored procedure.  If you copy the syntax into a simple query window and execute, you'll get an incorrect syntax error) Normal 0 false false false EN-US X-NONE X-NONE MicrosoftInternetExplorer4 /* Style Definitions */ table.MsoNormalTable {mso-style-name:"Table Normal"; mso-tstyle-rowband-size:0; mso-tstyle-colband-size:0; mso-style-noshow:yes; mso-style-priority:99; mso-style-qformat:yes; mso-style-parent:""; mso-padding-alt:0in 5.4pt 0in 5.4pt; mso-para-margin-top:0in; mso-para-margin-right:0in; mso-para-margin-bottom:10.0pt; mso-para-margin-left:0in; line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri","sans-serif"; mso-ascii-font-family:Calibri; mso-ascii-theme-font:minor-latin; mso-fareast-font-family:"Times New Roman"; mso-fareast-theme-font:minor-fareast; mso-hansi-font-family:Calibri; mso-hansi-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman"; mso-bidi-theme-font:minor-bidi;} -- Get all folders and group them by the original DocumentFolderID in PTSDocument table;WITH DocFoldersByDocFolderID(PTSDocumentFolderID_Original, PTSDocumentFolderID_Parent, sDocumentFolder, nLevel)AS (-- first member      SELECT 'PTSDocumentFolderID_Original' = d1.PTSDocumentFolderID            , PTSDocumentFolderID_Parent            , 'sDocumentFolder' = sName            , 'nLevel' = CONVERT(INT, 1000000)      FROM (SELECT DISTINCT PTSDocumentFolderID                  FROM dbo.PTSDocument_DY WITH(READPAST)            ) AS d1            INNER JOIN dbo.PTSDocumentFolder_DY AS df1 WITH(READPAST)                  ON d1.PTSDocumentFolderID = df1.PTSDocumentFolderID      UNION ALL      -- recursive      SELECT ddf1.PTSDocumentFolderID_Original            , df1.PTSDocumentFolderID_Parent            , 'sDocumentFolder' = df1.sName            , 'nLevel' = ddf1.nLevel - 1      FROM dbo.PTSDocumentFolder_DY AS df1 WITH(READPAST)            INNER JOIN DocFoldersByDocFolderID AS ddf1                  ON df1.PTSDocumentFolderID = ddf1.PTSDocumentFolderID_Parent)-- Flatten out folder path, DocFolderSingleByDocFolderID(PTSDocumentFolderID_Original, sDocumentFolder)AS (SELECT dfbdf.PTSDocumentFolderID_Original            , 'sDocumentFolder' = STUFF((SELECT '\' + sDocumentFolder                                         FROM DocFoldersByDocFolderID                                         WHERE (PTSDocumentFolderID_Original = dfbdf.PTSDocumentFolderID_Original)                                         ORDER BY PTSDocumentFolderID_Original, nLevel                                         FOR XML PATH ('')),1,1,'')      FROM DocFoldersByDocFolderID AS dfbdf      GROUP BY dfbdf.PTSDocumentFolderID_Original) And voila, I use the second CTE to join back to my original query (which is now a CTE for Source as we can now use MERGE to do INSERT and UPDATE at the same time).Each part of this solution would not solve the problem by itself because:If I don't use recursion, I cannot build out the path properly.  If I use the XML trick only, then I don't have the originating folder ID info that I need to link to the document.If I don't use the XML trick, then I don't have one row per document to show in the report.I could conceivably do this in the report function, but I'd rather not deal with the beginning or ending backslash and how to attach the document name.PIVOT doesn't do strings and UNPIVOT runs into the same problem as the above.I'm excited that each version of SQL server provides us new tools to solve old problems and/or enables us to solve problems in a more elegant wayThe 2 posts that helped me along:Recursive Queries Using Common Table ExpressionHow to use GROUP BY to concatenate strings in SQL server?

    Read the article

  • The blocking nature of aggregates

    - by Rob Farley
    I wrote a post recently about how query tuning isn’t just about how quickly the query runs – that if you have something (such as SSIS) that is consuming your data (and probably introducing a bottleneck), then it might be more important to have a query which focuses on getting the first bit of data out. You can read that post here.  In particular, we looked at two operators that could be used to ensure that a query returns only Distinct rows. and The Sort operator pulls in all the data, sorts it (discarding duplicates), and then pushes out the remaining rows. The Hash Match operator performs a Hashing function on each row as it comes in, and then looks to see if it’s created a Hash it’s seen before. If not, it pushes the row out. The Sort method is quicker, but has to wait until it’s gathered all the data before it can do the sort, and therefore blocks the data flow. But that was my last post. This one’s a bit different. This post is going to look at how Aggregate functions work, which ties nicely into this month’s T-SQL Tuesday. I’ve frequently explained about the fact that DISTINCT and GROUP BY are essentially the same function, although DISTINCT is the poorer cousin because you have less control over it, and you can’t apply aggregate functions. Just like the operators used for Distinct, there are different flavours of Aggregate operators – coming in blocking and non-blocking varieties. The example I like to use to explain this is a pile of playing cards. If I’m handed a pile of cards and asked to count how many cards there are in each suit, it’s going to help if the cards are already ordered. Suppose I’m playing a game of Bridge, I can easily glance at my hand and count how many there are in each suit, because I keep the pile of cards in order. Moving from left to right, I could tell you I have four Hearts in my hand, even before I’ve got to the end. By telling you that I have four Hearts as soon as I know, I demonstrate the principle of a non-blocking operation. This is known as a Stream Aggregate operation. It requires input which is sorted by whichever columns the grouping is on, and it will release a row as soon as the group changes – when I encounter a Spade, I know I don’t have any more Hearts in my hand. Alternatively, if the pile of cards are not sorted, I won’t know how many Hearts I have until I’ve looked through all the cards. In fact, to count them, I basically need to put them into little piles, and when I’ve finished making all those piles, I can count how many there are in each. Because I don’t know any of the final numbers until I’ve seen all the cards, this is blocking. This performs the aggregate function using a Hash Match. Observant readers will remember this from my Distinct example. You might remember that my earlier Hash Match operation – used for Distinct Flow – wasn’t blocking. But this one is. They’re essentially doing a similar operation, applying a Hash function to some data and seeing if the set of values have been seen before, but before, it needs more information than the mere existence of a new set of values, it needs to consider how many of them there are. A lot is dependent here on whether the data coming out of the source is sorted or not, and this is largely determined by the indexes that are being used. If you look in the Properties of an Index Scan, you’ll be able to see whether the order of the data is required by the plan. A property called Ordered will demonstrate this. In this particular example, the second plan is significantly faster, but is dependent on having ordered data. In fact, if I force a Stream Aggregate on unordered data (which I’m doing by telling it to use a different index), a Sort operation is needed, which makes my plan a lot slower. This is all very straight-forward stuff, and information that most people are fully aware of. I’m sure you’ve all read my good friend Paul White (@sql_kiwi)’s post on how the Query Optimizer chooses which type of aggregate function to apply. But let’s take a look at SQL Server Integration Services. SSIS gives us a Aggregate transformation for use in Data Flow Tasks, but it’s described as Blocking. The definitive article on Performance Tuning SSIS uses Sort and Aggregate as examples of Blocking Transformations. I’ve just shown you that Aggregate operations used by the Query Optimizer are not always blocking, but that the SSIS Aggregate component is an example of a blocking transformation. But is it always the case? After all, there are plenty of SSIS Performance Tuning talks out there that describe the value of sorted data in Data Flow Tasks, describing the IsSorted property that can be set through the Advanced Editor of your Source component. And so I set about testing the Aggregate transformation in SSIS, to prove for sure whether providing Sorted data would let the Aggregate transform behave like a Stream Aggregate. (Of course, I knew the answer already, but it helps to be able to demonstrate these things). A query that will produce a million rows in order was in order. Let me rephrase. I used a query which produced the numbers from 1 to 1000000, in a single field, ordered. The IsSorted flag was set on the source output, with the only column as SortKey 1. Performing an Aggregate function over this (counting the number of rows per distinct number) should produce an additional column with 1 in it. If this were being done in T-SQL, the ordered data would allow a Stream Aggregate to be used. In fact, if the Query Optimizer saw that the field had a Unique Index on it, it would be able to skip the Aggregate function completely, and just insert the value 1. This is a shortcut I wouldn’t be expecting from SSIS, but certainly the Stream behaviour would be nice. Unfortunately, it’s not the case. As you can see from the screenshots above, the data is pouring into the Aggregate function, and not being released until all million rows have been seen. It’s not doing a Stream Aggregate at all. This is expected behaviour. (I put that in bold, because I want you to realise this.) An SSIS transformation is a piece of code that runs. It’s a physical operation. When you write T-SQL and ask for an aggregation to be done, it’s a logical operation. The physical operation is either a Stream Aggregate or a Hash Match. In SSIS, you’re telling the system that you want a generic Aggregation, that will have to work with whatever data is passed in. I’m not saying that it wouldn’t be possible to make a sometimes-blocking aggregation component in SSIS. A Custom Component could be created which could detect whether the SortKeys columns of the input matched the Grouping columns of the Aggregation, and either call the blocking code or the non-blocking code as appropriate. One day I’ll make one of those, and publish it on my blog. I’ve done it before with a Script Component, but as Script components are single-use, I was able to handle the data knowing everything about my data flow already. As per my previous post – there are a lot of aspects in which tuning SSIS and tuning execution plans use similar concepts. In both situations, it really helps to have a feel for what’s going on behind the scenes. Considering whether an operation is blocking or not is extremely relevant to performance, and that it’s not always obvious from the surface. In a future post, I’ll show the impact of blocking v non-blocking and synchronous v asynchronous components in SSIS, using some of LobsterPot’s Script Components and Custom Components as examples. When I get that sorted, I’ll make a Stream Aggregate component available for download.

    Read the article

  • The blocking nature of aggregates

    - by Rob Farley
    I wrote a post recently about how query tuning isn’t just about how quickly the query runs – that if you have something (such as SSIS) that is consuming your data (and probably introducing a bottleneck), then it might be more important to have a query which focuses on getting the first bit of data out. You can read that post here.  In particular, we looked at two operators that could be used to ensure that a query returns only Distinct rows. and The Sort operator pulls in all the data, sorts it (discarding duplicates), and then pushes out the remaining rows. The Hash Match operator performs a Hashing function on each row as it comes in, and then looks to see if it’s created a Hash it’s seen before. If not, it pushes the row out. The Sort method is quicker, but has to wait until it’s gathered all the data before it can do the sort, and therefore blocks the data flow. But that was my last post. This one’s a bit different. This post is going to look at how Aggregate functions work, which ties nicely into this month’s T-SQL Tuesday. I’ve frequently explained about the fact that DISTINCT and GROUP BY are essentially the same function, although DISTINCT is the poorer cousin because you have less control over it, and you can’t apply aggregate functions. Just like the operators used for Distinct, there are different flavours of Aggregate operators – coming in blocking and non-blocking varieties. The example I like to use to explain this is a pile of playing cards. If I’m handed a pile of cards and asked to count how many cards there are in each suit, it’s going to help if the cards are already ordered. Suppose I’m playing a game of Bridge, I can easily glance at my hand and count how many there are in each suit, because I keep the pile of cards in order. Moving from left to right, I could tell you I have four Hearts in my hand, even before I’ve got to the end. By telling you that I have four Hearts as soon as I know, I demonstrate the principle of a non-blocking operation. This is known as a Stream Aggregate operation. It requires input which is sorted by whichever columns the grouping is on, and it will release a row as soon as the group changes – when I encounter a Spade, I know I don’t have any more Hearts in my hand. Alternatively, if the pile of cards are not sorted, I won’t know how many Hearts I have until I’ve looked through all the cards. In fact, to count them, I basically need to put them into little piles, and when I’ve finished making all those piles, I can count how many there are in each. Because I don’t know any of the final numbers until I’ve seen all the cards, this is blocking. This performs the aggregate function using a Hash Match. Observant readers will remember this from my Distinct example. You might remember that my earlier Hash Match operation – used for Distinct Flow – wasn’t blocking. But this one is. They’re essentially doing a similar operation, applying a Hash function to some data and seeing if the set of values have been seen before, but before, it needs more information than the mere existence of a new set of values, it needs to consider how many of them there are. A lot is dependent here on whether the data coming out of the source is sorted or not, and this is largely determined by the indexes that are being used. If you look in the Properties of an Index Scan, you’ll be able to see whether the order of the data is required by the plan. A property called Ordered will demonstrate this. In this particular example, the second plan is significantly faster, but is dependent on having ordered data. In fact, if I force a Stream Aggregate on unordered data (which I’m doing by telling it to use a different index), a Sort operation is needed, which makes my plan a lot slower. This is all very straight-forward stuff, and information that most people are fully aware of. I’m sure you’ve all read my good friend Paul White (@sql_kiwi)’s post on how the Query Optimizer chooses which type of aggregate function to apply. But let’s take a look at SQL Server Integration Services. SSIS gives us a Aggregate transformation for use in Data Flow Tasks, but it’s described as Blocking. The definitive article on Performance Tuning SSIS uses Sort and Aggregate as examples of Blocking Transformations. I’ve just shown you that Aggregate operations used by the Query Optimizer are not always blocking, but that the SSIS Aggregate component is an example of a blocking transformation. But is it always the case? After all, there are plenty of SSIS Performance Tuning talks out there that describe the value of sorted data in Data Flow Tasks, describing the IsSorted property that can be set through the Advanced Editor of your Source component. And so I set about testing the Aggregate transformation in SSIS, to prove for sure whether providing Sorted data would let the Aggregate transform behave like a Stream Aggregate. (Of course, I knew the answer already, but it helps to be able to demonstrate these things). A query that will produce a million rows in order was in order. Let me rephrase. I used a query which produced the numbers from 1 to 1000000, in a single field, ordered. The IsSorted flag was set on the source output, with the only column as SortKey 1. Performing an Aggregate function over this (counting the number of rows per distinct number) should produce an additional column with 1 in it. If this were being done in T-SQL, the ordered data would allow a Stream Aggregate to be used. In fact, if the Query Optimizer saw that the field had a Unique Index on it, it would be able to skip the Aggregate function completely, and just insert the value 1. This is a shortcut I wouldn’t be expecting from SSIS, but certainly the Stream behaviour would be nice. Unfortunately, it’s not the case. As you can see from the screenshots above, the data is pouring into the Aggregate function, and not being released until all million rows have been seen. It’s not doing a Stream Aggregate at all. This is expected behaviour. (I put that in bold, because I want you to realise this.) An SSIS transformation is a piece of code that runs. It’s a physical operation. When you write T-SQL and ask for an aggregation to be done, it’s a logical operation. The physical operation is either a Stream Aggregate or a Hash Match. In SSIS, you’re telling the system that you want a generic Aggregation, that will have to work with whatever data is passed in. I’m not saying that it wouldn’t be possible to make a sometimes-blocking aggregation component in SSIS. A Custom Component could be created which could detect whether the SortKeys columns of the input matched the Grouping columns of the Aggregation, and either call the blocking code or the non-blocking code as appropriate. One day I’ll make one of those, and publish it on my blog. I’ve done it before with a Script Component, but as Script components are single-use, I was able to handle the data knowing everything about my data flow already. As per my previous post – there are a lot of aspects in which tuning SSIS and tuning execution plans use similar concepts. In both situations, it really helps to have a feel for what’s going on behind the scenes. Considering whether an operation is blocking or not is extremely relevant to performance, and that it’s not always obvious from the surface. In a future post, I’ll show the impact of blocking v non-blocking and synchronous v asynchronous components in SSIS, using some of LobsterPot’s Script Components and Custom Components as examples. When I get that sorted, I’ll make a Stream Aggregate component available for download.

    Read the article

  • Why does Application_Deactivated not get called on a pivot page?

    - by willmel
    For my Windows Phone 7 app, I have a main panorama page which opens up into a pivot control. The main panorama page correctly calls Activated/Deactivated, and restores correctly. But after visiting the pivot page, pressing the Windows key doesn't call Deactivated. When the app is relaunched with the back button, it goes right to how the page looked before tombstoning, but Activated is not called, and the page is not usable, and the back key doesn't work. Has anyone else experienced this problem before?

    Read the article

  • what special issues are at play when loading a config file from the comand prompt with DTExec

    - by Ralph Shillington
    If I run a package from the Management Studio, and specify a configuration file, everything works as expected. However if I try and run the package from the command prompt with DTExec I get the error: Cannot load the XML configuration file. The XML configuration file may be malformed or not valid. The command I'm using to execute the package is: dtexec /conf ConfigurationDemo.dtsConfig /f Package.dtsx I am running the dtexec from the folder where these two files reside. Is there an addtional switch or something that must used to get dtexec to behave the same was at the management Stduio in launching a package?

    Read the article

  • Is it possible to force an error in an Integration Services data flow to demonstrate its rollback?

    - by Matt
    I have been tasked with demoing how Integration Services handles an error during a data flow to show that no data makes it into the destination. This is an existing package and I want to limit the code changes to the package as much as possible (since this is most likely a one time deal). The scenario that is trying to be understood is a "systemic" failure - the source file disappears midstream, or the file server loses power, etc. I know I can make this happen by having the Error Output of the source set to Failure and introducing bad data but I would like to do something lighter than that. I suppose I could add a Script Transform task and look for a certain value and throw an error but I was hoping someone has come up with something easier / more elegant. Thanks, Matt

    Read the article

  • Derived Column Editor

    - by Rob Bowman
    Hi I need to assign a formatted date to a column in a data flow. I have added a Derived Column editor and entered the following expression: "BBD" + SUBSTRING((DT_WSTR,4)DATEADD("Day",30,GETDATE()),1,4) + SUBSTRING((DT_WSTR,2)DATEADD("Day",30,GETDATE()),6,2) + SUBSTRING((DT_WSTR,2)DATEADD("Day",30,GETDATE()),9,2) The problem is that the "Derived Column Transformation Editor" automatically assigns a Data Type of "Unicode string[DT_WSTR]" and a length of "7". Howver, the length of a string is 11, therefore the following exception is thrown each time: [Best Before Date [112]] Error: The "component "Best Before Date" (112)" failed because truncation occurred, and the truncation row disposition on "output column "Comments" (132)" specifies failure on truncation. A truncation error occurred on the specified object of the specified component. Does anyone know why the edit is insisting on a length of 7? I don't seem to be able to change this. Many thanks, Rob.

    Read the article

  • how to get the sql connection

    - by sweetsecret
    FS_Setting is a VB class which has all the details of the connections ie: Public Class FS_Setting Public Function Get_RS_Connection() As SqlConnection Try Get_RS_Connection = New SqlConnection("Data Source=***********;User ID=sa;Password=*****;database=*********") Catch ex As System.Exception Throw New System.Exception("Get_RS_Connection Error:" + ex.Message) End Try End Function I need to call the function Get_RS_Connection() in a different class instead of getting the connection all the way again and hard coding.... I want to call the above class where the SQL connection is declared Namespace FS_Library Public Class FS_Errorlog Inherits FS_BaseClass Try **cn = New SqlConnection("Data Source=***********;UserID=sa;Password=*****;database=*********")** cmd = New SqlCommand("dbo.FS_ErrorLog_ADD", cn) cmd.CommandType = CommandType.StoredProcedure cmd.CommandTimeout = Convert.ToInt32(ConfigurationSettings.AppSettings("Command_Timeout")) Me.AddParameter(cmd, "@p_tableKey", SqlDbType.Int, tableKey) Me.AddParameter(cmd, "@p_FunctionCode", SqlDbType.Int, FunctionCode) Me.AddParameter(cmd, "@p_TableAlias", SqlDbType.VarChar, TableAlias) Me.AddParameter(cmd, "@p_ValidationCode", SqlDbType.Int, ValidationCode) If Filename = "" Then Filename = "N/A" End If Me.AddParameter(cmd, "@p_FileName", SqlDbType.VarChar, Filename) Me.AddParameter(cmd, "@p_Message", SqlDbType.VarChar, Message) Me.AddParameter(cmd, "@p_CreateUser", SqlDbType.VarChar, userID) Me.AddParameter(cmd, "@p_UserActionID", SqlDbType.UniqueIdentifier, UserActionID) cn.Open()

    Read the article

  • Newbie T-SQL dynamic stored procedure -- how can I improve it?

    - by Andy Jones
    I'm new to T-SQL; all my experience is in a completely different database environment (Openedge). I've learned enough to write the procedure below -- but also enough to know that I don't know enough! This routine will have to go into a live environment soon, and it works, but I'm quite certain there are a number of c**k-ups and gotchas in it that I know nothing about. The routine copies data from table A to table B, replacing the data in table B. The tables could be in any database. I plan to call this routine multiple times from another stored procedure. Permissions aren't a problem: the routine will be run by the dba as a timed job. Could I have your suggestions as to how to make it fit best-practice? To bullet-proof it? ALTER PROCEDURE [dbo].[copyTable2Table] @sdb varchar(30), @stable varchar(30), @tdb varchar(30), @ttable varchar(30), @raiseerror bit = 1, @debug bit = 0 as begin set nocount on declare @source varchar(65) declare @target varchar(65) declare @dropstmt varchar(100) declare @insstmt varchar(100) declare @ErrMsg nvarchar(4000) declare @ErrSeverity int set @source = '[' + @sdb + '].[dbo].[' + @stable + ']' set @target = '[' + @tdb + '].[dbo].[' + @ttable + ']' set @dropStmt = 'drop table ' + @target set @insStmt = 'select * into ' + @target + ' from ' + @source set @errMsg = '' set @errSeverity = 0 if @debug = 1 print('Drop:' + @dropStmt + ' Insert:' + @insStmt) -- drop the target table, copy the source table to the target begin try begin transaction exec(@dropStmt) exec(@insStmt) commit end try begin catch if @@trancount > 0 rollback select @errMsg = error_message(), @errSeverity = error_severity() end catch -- update the log table insert into HHG_system.dbo.copyaudit (copytime, copyuser, source, target, errmsg, errseverity) values( getdate(), user_name(user_id()), @source, @target, @errMsg, @errSeverity) if @debug = 1 print ( 'Message:' + @errMsg + ' Severity:' + convert(Char, @errSeverity) ) -- handle errors, return value if @errMsg <> '' begin if @raiseError = 1 raiserror(@errMsg, @errSeverity, 1) return 1 end return 0 END Thanks!

    Read the article

  • SQL Server - CAST AND DIVIDE

    - by rs
    DECLARE @table table(XYZ VARCHAR(8) , id int) INSERT INTO @table SELECT '4000', 1 UNION ALL SELECT '3.123', 2 UNION ALL SELECT '7.0', 3 UNION ALL SELECT '80000', 4 UNION ALL SELECT NULL, 5 SELECT CASE WHEN PATINDEX('^[0-9]{1,5}[\.][0-9]{1,3}$', XYZ) = 0 THEN XYZ WHEN PATINDEX('^[0-9]{1,8}$',XYZ) = 0 THEN CAST(XYZ AS decimal(18,3))/1000 ELSE NULL END FROM @table This part - CAST(XYZ AS decimal(18,3))/1000 doesn't divide value it gives me more number of zeros after decimal instead of dividing it. (I even enclosed that in brackets and tried but same result) Am i doing something wrong here?

    Read the article

  • SQL Server Agent 2005 job runs but no output

    - by alimack
    Essentially I have a job which runs in BIDS and as as a stand lone package and while it runs under the SQL Server Agent it doesn't complete properly (no error messages though). The job steps are: 1) Delete all rows from table; 2) Use For each loop to fill up table from Excel spreasheets; 3) Clean up table. I've tried this [MS page][1] (steps 1 & 2), didn't see any need to start changing from Server side security. Also SQLServerCentral.com for [this page][2], no resolution. How can I get error logging or a fix? Note I've reposted this from Server Fault as it's one of those questions that's not pure admin or programming. I have logged in as the proxy account I'm running this under, and the job runs stand alone but complains that the Excel tables are empty?

    Read the article

  • how to get external variable value in dtsx package.

    - by Rishabh
    Hi, I am executing .dtsx package from c#, it was executing fine, if i am passing one variable value from c# code then how can i get it on .dtsx package for my ole db source query. Here is my c# code. string file = @"D:\CYNCZFuzzy\CYNCZFuzzy\Contact.dtsx"; package = app.LoadPackage(file, null); Variables vars = package.Variables; vars["User::parentContactID"].Value = 1028203; pkgResults = package.Execute(); string result = pkgResults.ToString(); I need this 1028203 value on my ole db source query, here my query. select cr.MasterContactID as ParentContactID, c.ID,C.FirstName, C.MiddleName, c.LastName, c.ID as FieldID from Contact c inner join ContactRelation cr on cr.SlaveContactID = c.ID where RelationshipID = 1 AND cr.MasterContactID = ? what I should write on ? for getting 1028203 value from c# page. Thanks in advance...

    Read the article

< Previous Page | 24 25 26 27 28 29 30 31 32 33 34 35  | Next Page >