Search Results

Search found 27339 results on 1094 pages for 'sql dmv'.

Page 480/1094 | < Previous Page | 476 477 478 479 480 481 482 483 484 485 486 487  | Next Page >

  • Comparing Table Variables with Temporary Tables

    This articles brings a comparison of temporary tables with table variables from SQL Server author, Wayne Sheffield. In includes an in-depth look at the differences between them. SQL Server monitoring made easy "Keeping an eye on our many SQL Server instances is much easier with SQL Response." Mike Lile.Download a free trial of SQL Response now.

    Read the article

  • Getting started with StreamInsight

    A quick-start to SQL Server StreamInsight and Complex Event Processing with a step-by-step example on stock quotes. Too many SQL Servers to keep up with?Download a free trial of SQL Response to monitor your SQL Servers in just one intuitive interface."The monitoringin SQL Response is excellent." Mike Towery.

    Read the article

  • SSRS In a Flash

    Learn the basics of Reporting Services, what it is, and what it can do from you. From MVP Jessica Moss, we have a new series that can help you get started with this part of SQL Server. SQL Server monitoring made easy "Keeping an eye on our many SQL Server instances is much easier with SQL Response." Mike Lile.Download a free trial of SQL Response now.

    Read the article

  • But what version is the database now?

    - by BuckWoody
    When you upgrade your system to SQL Server 2008 R2, you’ll know that the instance is at that version by using the standard commands like SELECT @@VERSION or EXEC xp_msver. My system came back with this info when I typed those: Microsoft SQL Server 2008 R2 (RTM) - 10.50.1600.1 (Intel X86)   Apr  2 2010 15:53:02   Copyright (c) Microsoft Corporation  Developer Edition on Windows NT 6.0 <X86> (Build 6002: Service Pack 2) (Hypervisor) Index Name Internal_Value Character_Value 1 ProductName NULL Microsoft SQL Server 2 ProductVersion 655410 10.50.1600.1 3 Language 1033 English (United States) 4 Platform NULL NT INTEL X86 5 Comments NULL SQL 6 CompanyName NULL Microsoft Corporation 7 FileDescription NULL SQL Server Windows NT 8 FileVersion NULL 2009.0100.1600.01 ((KJ_RTM).100402-1540 ) 9 InternalName NULL SQLSERVR 10 LegalCopyright NULL Microsoft Corp. All rights reserved. 11 LegalTrademarks NULL Microsoft SQL Server is a registered trademark of Microsoft Corporation. 12 OriginalFilename NULL SQLSERVR.EXE 13 PrivateBuild NULL NULL 14 SpecialBuild 104857601 NULL 15 WindowsVersion 393347078 6.0 (6002) 16 ProcessorCount 1 1 17 ProcessorActiveMask 1 1 18 ProcessorType 586 PROCESSOR_INTEL_PENTIUM 19 PhysicalMemory 2047 2047 (2146934784) 20 Product ID NULL NULL   But a database properties are separate from the Instance. After an upgrade, you always want to make sure that the compatibility options (which have much to do with how NULLs and other objects are treated) is at what you expect. For the most part, as long as the application can handle it, I set my compatibility levels to the latest version. For SQL Server 2008, that was “10.0” or “10”. You can do this with the ALTER DATABASE command or you can just right-click the database and select “Properties” and then “Database Options” in SQL Server Management Studio. To check the database compatibility level, I use this query: SELECT name, cmptlevel FROM sys.sysdatabases When I did that this morning I saw that the databases (all of them) were at 10.0 – not 10.5 like the Instance. That’s expected – we didn’t revise the database format up with the Instance for this particular release. Didn’t want to catch you by surprise on that. While your databases should be at the “proper” level for your situation, you can’t rely on the compatibility level to indicate the Instance level. More info on the ALTER DATABASE command in SQL Server 2008 R2 is here: http://technet.microsoft.com/en-us/library/bb510680(SQL.105).aspx Share this post: email it! | bookmark it! | digg it! | reddit! | kick it! | live it!

    Read the article

  • Deleting Large Number of Records

    Often someone will try to perform a delete on a large number of records and run into a number of problems. Slow performance, log growth, and more. Lynn Pettis shows us how to better handle this situation in SQL Server 2000 and SQL Server 2005 The Future of SQL Server Monitoring "Being web-based, SQL Monitor 2.0 enables you to check on your servers from almost any location" Jonathan Allen.Try SQL Monitor now.

    Read the article

  • Free eBook - Control Your Transaction Log so it Doesn't Control You

    Download your free copy of SQL Server Transaction Log Management and see why understanding how log files work can make all the difference in a crisis. Want to work faster with SQL Server?If you want to work faster try out the SQL Toolbelt. "The SQL Toolbelt provides tools that database developers as well as DBAs should not live without." William Van Orden. Download the SQL Toolbelt here.

    Read the article

  • Master Immersion Events from SQLskills.com

    A new series of courses is available from SQLskills to help you better learn to manage and tune your SQL Server instances. These classes are designed to help you complete the MCM certification, but are also valuable for anyone that wants to become a better SQL Server DBA. NEW! SQL Monitor 2.0Monitor SQL Server Central's servers withRed Gate's new SQL Monitor.No installation required. Find out more.

    Read the article

  • Free eBook: Defensive Database Programming

    Resilient T-SQL code is code that is designed to last, and to be safely reused by others. The goal of defensive database programming, the goal of this book, is to help you to produce resilient T-SQL code that robustly and gracefully handles cases of unintended use, and is resilient to common changes to the database environment. 12 must-have SQL Server toolsThe award-winning SQL Developer Bundle contains 12 tools for faster, simpler SQL Server development. Download a free trial.

    Read the article

  • MSDTC attempts to enlist client machine in a distributed transaction

    - by Ken
    Hi there We're seeing the following intermittent warning logged by MSDTC: A caller has attempted to propagate a transaction to a remote system, but MSDTC network DTC access is currently disabled on machine 'X'. Please review the MS DTC configuration settings. However, MSDTC is disabled on machine X by design - it's a client machine, and has no business being enlisted in the transaction! Several windows service endpoints hosting WCF services over TCP Single SQL Server 2005 instance beneath Linq to Sql Remote client receives event callbacks over WCF/TCP The issue is tricky to reproduce - usually following restart of services. We suspect a callback to the client machine is occurring within the context of a transaction. Just wondering if anyone has seen similar issues?? Ken

    Read the article

  • Convert IEnumerable to EntitySet

    - by Gregorius
    Hey all, Hoping somebody can shed some light, and perhaps a possible solution to this issue I'm having... I have used LINQ to SQL to pull some data from a database into local entities. They are products from a shopping cart system. A product can contain a collection of KitGroups (which are stored in an EntitySet (System.Data.Linq.EntitySet). KitGroups contain collections of KitItems, and KitItems can contain Nested Products (which link back up to the original Product type - so its recursive). From these entities I'm building XML using LINQ to XML - all good here - my XML looks beautiful, calling a "GenerateProductElement" function, which calls itself recursively to generate the nested products. Wonderful stuff. However, here's where i'm stuck.. i'm now trying to deserialize that XML back to the original objects (all autogenerated by Linq to SQL)... and herein lies the problem. Linq tO Sql expects my collections to be EntitySet collections, however Linq to Xml (which i'm tyring to use to deserailise) is returning IEnumerable. I've experimented with a few ways of casting between the 2, but nothing seems to work... I'm starting to think that I should just deserialise manually (with some funky loops and conditionals to determine which KitGroup KitItems belong to, etc)... however its really quite tricky and that code is likely to be quite ugly, so I'd love to find a more elegant solution to this problem. Any suggestions? Here's a code snippet: private Product GenerateProductFromXML(XDocument inDoc) { var prod = from p in inDoc.Descendants("Product") select new Product { ProductID = (int)p.Attribute("ID"), ProductGUID = (Guid)p.Attribute("GUID"), Name = (string)p.Element("Name"), Summary = (string)p.Element("Summary"), Description = (string)p.Element("Description"), SEName = (string)p.Element("SEName"), SETitle = (string)p.Element("SETitle"), XmlPackage = (string)p.Element("XmlPackage"), IsAKit = (byte)(int)p.Element("IsAKit"), ExtensionData = (string)p.Element("ExtensionData"), }; //TODO: UUGGGGGGG Converting b/w IEnumerable & EntitySet var kitGroups = (from kg in inDoc.Descendants("KitGroups").Elements("KitGroup") select new KitGroup { KitGroupID = (int) kg.Attribute("ID"), KitGroupGUID = (Guid) kg.Attribute("GUID"), Name = (string) kg.Element("Name"), KitItems = // THIS IS WHERE IT FAILS - "Cannot convert source type IEnumerable to target type EntitySet..." (from ki in kg.Descendants("KitItems").Elements("KitItem") select new KitItem { KitItemID = (int) ki.Attribute("ID"), KitItemGUID = (Guid) ki.Attribute("GUID") }); }); Product ImportedProduct = prod.First(); ImportedProduct.KitGroups = new EntitySet<KitGroup>(); ImportedProduct.KitGroups.AddRange(kitGroups); return ImportedProduct; }

    Read the article

  • SSRS Report from Oracle DB - Use stored procedure

    - by Emtucifor
    I am developing a report in Sql Server Reporting Services 2005, connecting to an Oracle 11g database. As you post replies perhaps it will help to know that I'm skilled in MSSQL Server and inexperienced in Oracle. I have multiple nested subreports and need to use summary data in outer reports and the same data but in detail in the inner reports. In order to spare the DB server from multiple executions, I thought to populate some temp tables at the beginning and then query just them the multiple times in the report and the subreports. In SSRS, Datasets are evidently executed in the order they appear in the RDL file. And you can have a dataset that doesn't return a rowset. So I created a stored procedure to populate my four temp tables and made this the first Dataset in my report. This SP works when I run it from SQLDeveloper and I can query the data from the temp tables. However, this didn't appear to work out because SSRS was apparently not reusing the same session, so even though the global temporary tables were created with ON COMMIT PRESERVE ROWS my Datasets were empty. I switched to using "real" tables and am now passing in an additional parameter, a GUID in string form, uniquely generated on each new execution, that is part of the primary key of each table, so I can get back just the rows for this execution. Running this from Sql Developer works fine, example: DECLARE ActivityCode varchar2(15) := '1208-0916 '; ExecutionID varchar2(32) := SYS_GUID(); BEGIN CIPProjectBudget (ActivityCode, ExecutionID); END; Never mind that in this example I don't know the GUID, this simply proves it works because rows are inserted to my four tables. But in the SSRS report, I'm still getting no rows in my Datasets and SQL Developer confirms no rows are being inserted. So I'm thinking along the lines of: Oracle uses implicit transactions and my changes aren't getting committed? Even though I can prove that the non-rowset returning SP is executing (because if I leave out the parameter mapping it complains at report rendering time about not having enough parameters) perhaps it's not really executing. Somehow. Wrong execution order isn't the problem or rows would appear in the tables, and they aren't. I'm interested in any ideas about how to accomplish this (especially the part about not running the main queries multiple times). I'll redesign my whole report. I'll stop using a stored procedure. Suggest anything you like! I just need help getting this working and I am stuck. If you want more details, in my SSRS report I have a List object (it's a container that repeats once for each row in a Dataset) that has some header values and then contains a subreport. Eventually, there will be four total reports: one main report, with three nested subreports. Each subreport will be in a List on the parent report.

    Read the article

  • Adding Login does not work. Error: "Cannot open database \ requested by the login. The login failed.\r\nLogin failed for user"

    - by Jacob Pressures
    I'm receiving this error message: Cannot open database \"Northwind\" requested by the login. The login failed.\r\nLogin failed for user 'Mike\Mike Simpson' When i try to add the user, which is me. Then sql server says Msg 15063, Level 16, State 1, Line 1 The login already has an account under a different user name. HEre is my connection string: "Data Source=Localhost;Initial Catalog=Northwind;Integrated Security=True" I'm using SSMS not SQL Server express I don't know what else I should do. This application used to work before i zipped it up. Thanks!

    Read the article

  • FILESTREAM/FILETABLE Clarifications for Implementation

    - by user1209734
    Recently our team was looking at FILESTREAM to expand the capabilities of our proprietary application. The main purpose of this app is managing the various PDFS, Images and documents to all of the parts we manufacture. Our ASP application uses a few third party tools to allow viewing of these files. We currently have 980GB of data on the Fileserver. We have around 200GB of Binary data in SQL Server that we would like to extract since it is not performing well hence FILESTREAM seems to be a good compromise to the two major data storage/access issues. A few things are not exactly clear to us: FILESTREAM Can or Cannot store its data on a drive that is not locally attached. We already have a File Server with a RAID 10 (1.5TB drives). This server stores all of the documents right now, would we have to move these drives to the SQL Server for FILESTREAM? That would be a tough bullet to bite since the server also is doubling as the Application Server (Two VMs on one physical server). FILETABLE stores the common metadata about the files but where is the Full Text part of it stored to allow searching of files like doc/docx? Is this separate? Are you able to freely add criteria to this to search by? If so any links to clarify would be appreciated. Can FILETABLE be referenced in another table with a foreign key? Thank you in advance EDIT: For those having these questions this web video covered everything and more in terms of explaining filestream from 2008 to 2012 and the cavets to consider (I would seriously rep him if I could): http://channel9.msdn.com/Events/TechDays/Techdays-2012-the-Netherlands/2270 In conclusion we will not be using FILESTREAM as it would be way to huge of an upsurge to accommodate for investment. EDIT 2: Update to #1 - After carefully assessing FileTable in addition to FILESTREAM we got a winning combination. We did have to move the files over to the new server (wasn't to painful since they were on the same VM).It honestly took more time to write an extraction tool to dump the binary data within SQL to the File System. Update to #2 - This was seperate but again Bob had an excellent webinar explaining this: http://channel9.msdn.com/Events/TechEd/Europe/2012/DBI411 Update to #3 - Using TFT inheritance we recycled the Docs table we had (minus the huge binary blobs) which required very little changes in our legacy apps. This was a huge upshot for the developer team.

    Read the article

  • Timeout Expired error Using LINQ

    - by Refracted Paladin
    I am going to sum up my problem first and then offer massive details and what I have already tried. Summary: I have an internal winform app that uses Linq 2 Sql to connect to a local SQL Express database. Each user has there own DB and the DB stay in sync through Merge Replication with a Central DB. All DB's are SQL 2005(sp2or3). We have been using this app for over 5 months now but recently our users are getting a Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. Detailed: The strange part is they get that in two differnt locations(2 differnt LINQ Methods) and only the first time they fire in a given time period(~5mins). One LINQ method is pulling all records that match a FK ID and then Manipulating them to form a Heirarchy View for a TreeView. The second is pulling all records that match a FK ID and dumping them into a DataGridView. The only things I can find in common with the 2 are that the first IS an IEnumerable and the second converts itself from IQueryable - IEnumerable - DataTable... I looked at the query's in Profiler and they 'seemed' normal. They are not very complicated querys. They are only pulling back 10 - 90 records, from one table. Any thoughts, suggestions, hints whatever would be greatly appreciated. I am at my wit's end on this.... public IList<CaseNoteTreeItem> GetTreeViewDataAsList(int personID) { var myContext = MatrixDataContext.Create(); var caseNotesTree = from cn in myContext.tblCaseNotes where cn.PersonID == personID orderby cn.ContactDate descending, cn.InsertDate descending select new CaseNoteTreeItem { CaseNoteID = cn.CaseNoteID, NoteContactDate = Convert.ToDateTime(cn.ContactDate). ToShortDateString(), ParentNoteID = cn.ParentNote, InsertUser = cn.InsertUser, ContactDetailsPreview = cn.ContactDetails.Substring(0, 75) }; return caseNotesTree.ToList<CaseNoteTreeItem>(); } AND THIS ONE public static DataTable GetAllCNotes(int personID) { using (var context = MatrixDataContext.Create()) { var caseNotes = from cn in context.tblCaseNotes where cn.PersonID == personID orderby cn.ContactDate select new { cn.ContactDate, cn.ContactDetails, cn.TimeSpentUnits, cn.IsCaseLog, cn.IsPreEnrollment, cn.PresentAtContact, cn.InsertDate, cn.InsertUser, cn.CaseNoteID, cn.ParentNote }; return caseNotes.ToList().CopyLinqToDataTable(); } }

    Read the article

  • How to escape forward slash?

    - by AndrewB
    I have the following sql command through code and because the parameter contains a forward slash when I evaluate the sql row after the update the column is just empty. sqlCommand.CommandText = String.Format("update {0} set {1}='{2}'where id = @Id", tableName, ColumnName, forwardSlashText); sqlCommand.Parameters.Add("@Id", SqlDbType.UniqueIdentifier).Value = rowId; numRowsAffected = sqlCommand.ExecuteNonQuery(); adding a log.debug to this command i get the following output... update my_table_name set mime_type='application/pdf' where id = @Id So i would assume that the command is correct, but then looking at the row the mime_type column is empty.

    Read the article

  • Translate query to NHibernate

    - by Rob Walker
    I am trying to learn NHibernate, and am having difficulty translating a SQL query into one using the criteria API. The data model has tables: Part (Id, Name, ...), Order (Id, PartId, Qty), Shipment (Id, PartId, Qty) For all the parts I want to find the total quantity ordered and the total quantity shipped. In SQL I have: select shipment.part_id, sum(shipment.quantity), sum(order.quantity) from shipment cross join order on order.part_id = shipment.part_id group by shipment.part_id Alternatively: select id, (select sum(quantity) from shipment where part_id = part.id), (select sum(quantity) from order where part_id = part.id) from part But the latter query takes over twice as long to execute. Any suggestions on how to create these queries in (fluent) NHibernate? I have all the tables mapped and loading/saving/etc the entities works fine.

    Read the article

  • SQLPlus - spooling to multiple files from PL/SQL blocks

    - by FrustratedWithFormsDesigner
    I have a query that returns a lot of data into a CSV file. So much, in fact, that Excel can't open it - there are too many rows. Is there a way to control spool to spool to a new file everytime 65000 rows have been processed? Ideally, I'd like to have my output in files named in sequence, such as large_data_1.csv, large_data_2.csv, large_data_3.csv, etc... I could use dbms_output in a PL/SQL block to control how many rows are output, but then how would I switch files, as spool does not seem to be accessible from PL/SQL blocks? (Oracle 10g)

    Read the article

  • C# where does the dbml file come from?

    - by 5YrsLaterDBA
    Learning C# and learing Linq now. have lots of qestions about it. Basically I need a step by step tutorial. I suppose the dbml file is the configuration file of the database. I double click it and VS will open it with kind of design diagram. I can create/delete/modify table here? I can use add new item to add the Linq to SQL Classes to get a dbml file? what's next? generate tables in database? generate sql script? generate cs files? when? how?

    Read the article

  • Help combining these two queries

    - by Horace Loeb
    I need a SQL query that returns results matched by EITHER of the following SQL queries: Query 1: SELECT "annotations".* FROM "annotations" INNER JOIN "votes" ON "votes".voteable_id = "annotations".id AND "votes".voteable_type = 'Annotation' WHERE (votes.vote = 't' AND votes.voter_id = 78) Query 2: SELECT "annotations".* FROM "annotations" INNER JOIN "songs" ON "songs".id = "annotations".song_id INNER JOIN "songs" songs_annotations ON "songs_annotations".id = "annotations".song_id INNER JOIN "users" ON "users".id = "songs_annotations".state_last_updated_by_id WHERE (annotations.referent IS NOT NULL AND annotations.updated_at < '2010-04-05 01:51:24' AND (body = '?' OR body LIKE '%[?]%') AND ((users.id = songs.state_last_updated_by_id and users.needs_edit = 'f' and songs.state != 'work_in_progress') OR (songs.state = 'published')) Here's what I tried, but it doesn't work: SELECT "annotations".* FROM "annotations" INNER JOIN "songs" ON "songs".id = "annotations".song_id INNER JOIN "songs" songs_annotations ON "songs_annotations".id = "annotations".song_id INNER JOIN "users" ON "users".id = "songs_annotations".state_last_updated_by_id INNER JOIN "votes" ON "votes".voteable_id = "annotations".id AND "votes".voteable_type = 'Annotation' WHERE ((votes.vote = 't' and votes.voter_id = 78) OR (annotations.referent IS NOT NULL and annotations.updated_at < '2010-04-05 01:43:52' and (annotations.body = '?' OR annotations.body LIKE '%[?]%') and ((users.id = songs.state_last_updated_by_id and users.needs_edit = 'f') OR songs.state = 'published')))

    Read the article

  • SubSonic Alias/Where Clause

    - by JohnBob
    Hey, I want to convert the following SQL Query to a SubSonic Query. SELECT [dbo].[tbl_Agency].[ParentCompanyID] FROM [dbo].[tbl_Agency] WHERE REPLACE(PhoneNumber, ' ', '') LIKE REPLACE('%9481 1111%', ' ', '') I thought I would do it like below, but I just can't get it to produce valid SQL. //SubSonic string agencyPhoneNumber = "9481 1111"; SubSonic.SqlQuery subQueryagencyPhoneNumber = new SubSonic.Select(Agency.ParentCompanyIDColumn.ColumnName); subQueryagencyPhoneNumber.From(Agency.Schema.TableName); //WHERE subQueryagencyPhoneNumber.Where("REPLACE(" + Agency.PhoneNumberColumn.ColumnName + ", ' ', '')").Like("%" + agencyPhoneNumber + "%"); Does anyone out there know how to fix this - I'm using SubSonic 2.2. I feel like I'm taking crazy pills here - this should be straightforward, right? Cheers, JohnBob

    Read the article

  • getting count(*) using createSQLQuery in hibernate?

    - by JohnSmith
    I have several sql queries that I simply want to fire at the database. I am using hibernate throughout the whole application, so i would prefer to use hibernate to call this sql queries. In the example below i want to get count + name, but cant figure out how to get that info when i use createSQLQuery(). I have seen workarounds where people only need to get out a single "count()" from the result, but in this case I am using count() + a column as ouput SELECT count(*), a.name as count FROM user a WHERE a.user_id IN (SELECT b.user_id FROM user b) GROUP BY a.name HAVING COUNT(*) BETWEEN 2 AND 5; fyi, the above query would deliver a result like this if i call it directly on the database: 1, John 2, Donald 1, Ralph ...

    Read the article

  • RTF to TEXT in sql server

    - by user360109
    I have a RTF field in my SQL 2005 table, I need to convert it to Text and display it. After a quick research I got the following method... create function dbo.RTF2TXT(@in varchar(8000)) RETURNS varchar(8000) AS BEGIN DECLARE @object int DECLARE @hr int DECLARE @out varchar(8000) -- Create an object that points to the SQL Server EXEC @hr = sp_OACreate 'RICHTEXT.RichtextCtrl', @object OUT EXEC @hr = sp_OASetProperty @object, 'TextRTF', @in EXEC @hr = sp_OAGetProperty @object, 'Text', @out OUT EXEC @hr = sp_OADestroy @object return @out END GO select dbo.RTF2TXT('{\rtf1\ansi\ansicpg1252\uc1 aaa}') But Here I am getting only NULL as result... What could be the issue, please suggest Thanks

    Read the article

  • Oracle performance problem

    - by jreid42
    We are using an Oracle 11G machine that is very powerful; has redundant storage etc. It's a beast from what I have been told. We just got this DB for a tool that when I first came on as a coop had like 20 people using, now its upwards of 150 people. I am the only one working on it :( We currently have a system in place that distributes PERL scripts across our entire data center essentially giving us a sort of "grid" computing power. The Perl scripts run a sort of simulation and report back the results to the database. They do selects / inserts. The load is not very high for each script but it could be happening across 20-50 systems at the same time. We then have multiple data centers and users all hitting the same database with this same approach. Our main problem with this is that our database is getting overloaded with connections and having to drop some. We sometimes have upwards of 500 connections. These are old perl scripts and they do not handle this well. Essentially they fail and the results are lost. I would rather avoid having to rewrite a lot of these as they are poorly written, and are a headache to even look at. The database itself is not overloaded, just the connection overhead is too high. We open a connection, make a quick query and then drop the connection. Very short connections but many of them. The database team has basically said we need to lower the number of connections or they are going to ignore us. Because this is distributed across our farm we cant implement persistent connections. I do this with our webserver; but its on a fixed system. The other ones are perl scripts that get opened and closed by the distribution tool and thus arent always running. What would be my best approach to resolving this issue? The scripts themselves can wait for a connection to be open. They do not need to act immediately. Some sort of queing system? I've been suggested to set up a few instances of a tool called "SQL Relay". Maybe one in each data center. How reliable is this tool? How good is this approach? Would it work for what we need? We could have one for each data center and relay requests through it to our main database, keeping a pipeline of open persistent connections? Does this make sense? Is there any other suggestions you can make? Any ideas? Any help would be greatly appreciated. Sadly I am just a coop student working for a very big company and somehow all of this has landed all on my shoulders (there is literally nobody to ask for help; its a hardware company, everybody is hardware engineers, and the database team is useless and in India) and I am quite lost as what the best approach would be? I am extremely overworked and this problem is interfering with on going progress and basically needs to be resolved as quickly as possible; preferably without rewriting the whole system, purchasing hardware (not gonna happen), or shooting myself in the foot. HELP LOL!

    Read the article

  • How to optimize my PageRank calculation?

    - by asmaier
    In the book Programming Collective Intelligence I found the following function to compute the PageRank: def calculatepagerank(self,iterations=20): # clear out the current PageRank tables self.con.execute("drop table if exists pagerank") self.con.execute("create table pagerank(urlid primary key,score)") self.con.execute("create index prankidx on pagerank(urlid)") # initialize every url with a PageRank of 1.0 self.con.execute("insert into pagerank select rowid,1.0 from urllist") self.dbcommit() for i in range(iterations): print "Iteration %d" % i for (urlid,) in self.con.execute("select rowid from urllist"): pr=0.15 # Loop through all the pages that link to this one for (linker,) in self.con.execute("select distinct fromid from link where toid=%d" % urlid): # Get the PageRank of the linker linkingpr=self.con.execute("select score from pagerank where urlid=%d" % linker).fetchone()[0] # Get the total number of links from the linker linkingcount=self.con.execute("select count(*) from link where fromid=%d" % linker).fetchone()[0] pr+=0.85*(linkingpr/linkingcount) self.con.execute("update pagerank set score=%f where urlid=%d" % (pr,urlid)) self.dbcommit() However, this function is very slow, because of all the SQL queries in every iteration >>> import cProfile >>> cProfile.run("crawler.calculatepagerank()") 2262510 function calls in 136.006 CPU seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 136.006 136.006 <string>:1(<module>) 1 20.826 20.826 136.006 136.006 searchengine.py:179(calculatepagerank) 21 0.000 0.000 0.528 0.025 searchengine.py:27(dbcommit) 21 0.528 0.025 0.528 0.025 {method 'commit' of 'sqlite3.Connecti 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler 1339864 112.602 0.000 112.602 0.000 {method 'execute' of 'sqlite3.Connec 922600 2.050 0.000 2.050 0.000 {method 'fetchone' of 'sqlite3.Cursor' 1 0.000 0.000 0.000 0.000 {range} So I optimized the function and came up with this: def calculatepagerank2(self,iterations=20): # clear out the current PageRank tables self.con.execute("drop table if exists pagerank") self.con.execute("create table pagerank(urlid primary key,score)") self.con.execute("create index prankidx on pagerank(urlid)") # initialize every url with a PageRank of 1.0 self.con.execute("insert into pagerank select rowid,1.0 from urllist") self.dbcommit() inlinks={} numoutlinks={} pagerank={} for (urlid,) in self.con.execute("select rowid from urllist"): inlinks[urlid]=[] numoutlinks[urlid]=0 # Initialize pagerank vector with 1.0 pagerank[urlid]=1.0 # Loop through all the pages that link to this one for (inlink,) in self.con.execute("select distinct fromid from link where toid=%d" % urlid): inlinks[urlid].append(inlink) # get number of outgoing links from a page numoutlinks[urlid]=self.con.execute("select count(*) from link where fromid=%d" % urlid).fetchone()[0] for i in range(iterations): print "Iteration %d" % i for urlid in pagerank: pr=0.15 for link in inlinks[urlid]: linkpr=pagerank[link] linkcount=numoutlinks[link] pr+=0.85*(linkpr/linkcount) pagerank[urlid]=pr for urlid in pagerank: self.con.execute("update pagerank set score=%f where urlid=%d" % (pagerank[urlid],urlid)) self.dbcommit() This function is 20 times faster (but uses a lot more memory for all the temporary dictionaries) because it avoids the unnecessary SQL queries in every iteration: >>> cProfile.run("crawler.calculatepagerank2()") 64802 function calls in 6.950 CPU seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.004 0.004 6.950 6.950 <string>:1(<module>) 1 1.004 1.004 6.946 6.946 searchengine.py:207(calculatepagerank2 2 0.000 0.000 0.104 0.052 searchengine.py:27(dbcommit) 23065 0.012 0.000 0.012 0.000 {meth 'append' of 'list' objects} 2 0.104 0.052 0.104 0.052 {meth 'commit' of 'sqlite3.Connection 1 0.000 0.000 0.000 0.000 {meth 'disable' of '_lsprof.Profiler' 31298 5.809 0.000 5.809 0.000 {meth 'execute' of 'sqlite3.Connectio 10431 0.018 0.000 0.018 0.000 {method 'fetchone' of 'sqlite3.Cursor' 1 0.000 0.000 0.000 0.000 {range} But is it possible to further reduce the number of SQL queries to speed up the function even more?

    Read the article

< Previous Page | 476 477 478 479 480 481 482 483 484 485 486 487  | Next Page >