Search Results

Search found 14378 results on 576 pages for 'record count'.

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

Page 332/576 | < Previous Page | 328 329 330 331 332 333 334 335 336 337 338 339  | Next Page >

  • SQL SERVER – Weekly Series – Memory Lane – #031

    - by Pinal Dave
    Here is the list of selected articles of SQLAuthority.com across all these years. Instead of just listing all the articles I have selected a few of my most favorite articles and have listed them here with additional notes below it. Let me know which one of the following is your favorite article from memory lane. 2007 Find Table without Clustered Index – Find Table with no Primary Key Clustered index is very important concept for any table. They impact the performance very heavily. Here is a quick script to find tables without a clustered index. Replace TEXT with VARCHAR(MAX) – Stop using TEXT, NTEXT, IMAGE Data Types Question: “Is VARCHAR (MAX) big enough to store the TEXT field?” Answer: “Yes, VARCHAR(MAX) is big enough to accommodate TEXT field. TEXT, NTEXT and IMAGE data types of SQL Server 2000 will be deprecated in a future version of SQL Server, SQL Server 2005 provides backward compatibility to data types but it is recommended to use new data types which are VARHCAR (MAX), NVARCHAR (MAX) and VARBINARY (MAX).” Limiting Result Sets by Using TABLESAMPLE – Examples Introduced in SQL Server 2005, TABLESAMPLE allows you to extract a sampling of rows from a table in the FROM clause. The rows retrieved are random and they are are not in any order. This sampling can be based on a percentage of number of rows. You can use TABLESAMPLE when only a sampling of rows is necessary for the application instead of a full result set. User Defined Functions (UDF) Limitations UDF have its own advantage and usage but in this article we will see the limitation of UDF. Things UDF can not do and why Stored Procedure are considered as more flexible then UDFs. Stored Procedure are more flexibility then User Defined Functions(UDF). However, this blog post is a good read to know what are the limitations of UDF. Change Database Compatible Level – Backward Compatibility For a long time SQL Server stayed on the compatibility level of 80 which is of SQL Server 2000. However, as soon as SQL Server 2005 introduced the issue of compatibility was quite a major issue. Since that time MS has been releasing the versions at every 2-3 years, changing compatibility is a ever popular topic. In this blog post, we learn how we can do the same using T-SQL. We can also do the same using SSMS and here is the blog post for the same: Change Database Compatible Level – Backward Compatibility – Part 2 – Management Studio. Constraint on VARCHAR(MAX) Field To Limit It Certain Length How can I limit the VARCHAR(MAX) field with maximum length of 12500 characters only. His Question was valid as our application was allowed 12500 characters. First of all – this requirement is bit strange but if someone wants to do the same, they can do it as described in this blog post. 2008 UNPIVOT Table Example Understanding UNPIVOT can be very complicated at times. In this blog post, I have attempted to explain the same concept in very simple words. Create Default Constraint Over Table Column A simple straight to script blog post – I still use this blog quite many times for my own reference. UDF – Get the Day of the Week Function It took me 4 iteration to find this very simple function which can immediately get the day of the week in a single line. 2009 Find Hostname and Current Logged In User Name There are two tricks listed in this blog post where users can find out the hostname and current logged user name immediately and very easily. Interesting Observation of Logon Trigger On All Servers When I was doing a project, I made an interesting observation of executing a logon trigger multiple times. It was absolutely unexpected for me! As I was logging only once, naturally, I was expecting the entry only once. However, it did it multiple times on different threads – indeed an eccentric phenomenon at first sight! Difference Between Candidate Keys and Primary Key One needs to be very careful in selecting the Primary Key as an incorrect selection can adversely impact the database architect and future normalization. For a Candidate Key to qualify as a Primary Key, it should be Non-NULL and unique in any domain. I have observed quite often that Primary Keys are seldom changed. I would like to have your feedback on not changing a Primary Key. Create Multiple Filegroup For Single Database Why should one create multiple file group for any database and what are the advantages of the same. In this blog post, I explain the same in detail. List All Objects Created on All Filegroups in Database In this blog post we discuss the essential question – “How can I find which object belongs to which filegroup. Is there any way to know this?” 2010 DATE and TIME in SQL Server 2008 When DATE is converted to DATETIME it adds the of midnight. When TIME is converted to DATETIME it adds the date of 1900 and it is something one wants to consider if you are going to run scripts from SQL Server 2008 to earlier version with CONVERT. Disabled Index and Update Statistics If you do not need a nonclustered index, I suggest you to drop it as keeping them disabled is an overhead on your system. This is because every time the statistics are updated for system all the statistics for disabled indexes are also updated. Precision of SMALLDATETIME – A 1 Minute Precision The precision of the datatype SMALLDATETIME is 1 minute. It discards the seconds by rounding up or rounding down any seconds greater than zero. 2011 Getting Columns Headers without Result Data – SET FMTONLY ON SET FMTONLY ON returns only metadata to the client. It can be used to test the format of the response without actually running the query. When this setting is ON the resultset only have headers of the results but no data. Copy Database from Instance to Another Instance – Copy Paste in SQL Server SQL Server has a feature which copy database from one database to another database and it can be automated as well using SSIS. Make sure you have SQL Server Agent Turned on as this feature will create a job. Puzzle – SELECT * vs SELECT COUNT(*) If you have ever wondered SELECT * gives error when executed alone but SELECT COUNT(*) does not. Why? in that case, you should read this blog post. Creating All New Database with Full Recovery Model This blog post is very based on very interesting story where the user wants to do something by default for every single new database created. Model database is a secret weapon which should be used very carefully and with proper evalution. If used carefully this can be a very much beneficiary when we need a newly created database behave in certain fashion. 2012 In year 2012 I had two interesting series ran on the blog. If there is no fun in learning, the learning becomes a burden. For the same reason, I had decided to build a three part quiz around SEQUENCE. The quiz was to identify the next value of the sequence. I encourage all of you to take part in this fun quiz. Guess the Next Value – Puzzle 1 Guess the Next Value – Puzzle 2 Guess the Next Value – Puzzle 3 Can anyone remember their final day of schooling?  This is probably a silly question because – of course you can!  Many people mark this as the most exciting, happiest day of their life.  It marks the end of testing, the end of following rules set by teachers, and the beginning of finally being able to earn money and work in your chosen field. Read five part series on developer training subject Developer Training - Importance and Significance - Part 1 Developer Training – Employee Morals and Ethics – Part 2 Developer Training – Difficult Questions and Alternative Perspective - Part 3 Developer Training – Various Options for Developer Training – Part 4 Developer Training – A Conclusive Summary- Part 5 Reference: Pinal Dave (http://blog.sqlauthority.com) Filed under: Memory Lane, PostADay, SQL, SQL Authority, SQL Query, SQL Server, SQL Tips and Tricks, T SQL, Technology

    Read the article

  • Execution plan warnings–The final chapter

    - by Dave Ballantyne
    In my previous posts (here and here), I showed examples of some of the execution plan warnings that have been added to SQL Server 2012.  There is one other warning that is of interest to me : “Unmatched Indexes”. Firstly, how do I know this is the final one ?  The plan is an XML document, right ? So that means that it can have an accompanying XSD.  As an XSD is a schema definition, we can poke around inside it to find interesting things that *could* be in the final XML file. The showplan schema is stored in the folder Microsoft SQL Server\110\Tools\Binn\schemas\sqlserver\2004\07\showplan and by comparing schemas over releases you can get a really good idea of any new functionality that has been added. Here is the section of the Sql Server 2012 showplan schema that has been interesting me so far : <xsd:complexType name="AffectingConvertWarningType"> <xsd:annotation> <xsd:documentation>Warning information for plan-affecting type conversion</xsd:documentation> </xsd:annotation> <xsd:sequence> <!-- Additional information may go here when available --> </xsd:sequence> <xsd:attribute name="ConvertIssue" use="required"> <xsd:simpleType> <xsd:restriction base="xsd:string"> <xsd:enumeration value="Cardinality Estimate" /> <xsd:enumeration value="Seek Plan" /> <!-- to be extended here --> </xsd:restriction> </xsd:simpleType> </xsd:attribute> <xsd:attribute name="Expression" type ="xsd:string" use="required" /></xsd:complexType><xsd:complexType name="WarningsType"> <xsd:annotation> <xsd:documentation>List of all possible iterator or query specific warnings (e.g. hash spilling, no join predicate)</xsd:documentation> </xsd:annotation> <xsd:choice minOccurs="1" maxOccurs="unbounded"> <xsd:element name="ColumnsWithNoStatistics" type="shp:ColumnReferenceListType" minOccurs="0" maxOccurs="1" /> <xsd:element name="SpillToTempDb" type="shp:SpillToTempDbType" minOccurs="0" maxOccurs="unbounded" /> <xsd:element name="Wait" type="shp:WaitWarningType" minOccurs="0" maxOccurs="unbounded" /> <xsd:element name="PlanAffectingConvert" type="shp:AffectingConvertWarningType" minOccurs="0" maxOccurs="unbounded" /> </xsd:choice> <xsd:attribute name="NoJoinPredicate" type="xsd:boolean" use="optional" /> <xsd:attribute name="SpatialGuess" type="xsd:boolean" use="optional" /> <xsd:attribute name="UnmatchedIndexes" type="xsd:boolean" use="optional" /> <xsd:attribute name="FullUpdateForOnlineIndexBuild" type="xsd:boolean" use="optional" /></xsd:complexType> I especially like the “to be extended here” comment,  high hopes that we will see more of these in the future.   So “Unmatched Indexes” was a warning that I couldn’t get and many thanks must go to Fabiano Amorim (b|t) for showing me the way.   Filtered indexes were introduced in Sql Server 2008 and are really useful if you only need to index only a portion of the data within a table.  However,  if your SQL code uses a variable as a predicate on the filtered data that matches the filtered condition, then the filtered index cannot be used as, naturally,  the value in the variable may ( and probably will ) change and therefore will need to read data outside the index.  As an aside,  you could use option(recompile) here , in which case the optimizer will build a plan specific to the variable values and use the filtered index,  but that can bring about other problems.   To demonstrate this warning, we need to generate some test data :   DROP TABLE #TestTab1GOCREATE TABLE #TestTab1 (Col1 Int not null, Col2 Char(7500) not null, Quantity Int not null)GOINSERT INTO #TestTab1 VALUES (1,1,1),(1,2,5),(1,2,10),(1,3,20), (2,1,101),(2,2,105),(2,2,110),(2,3,120)GO and then add a filtered index CREATE INDEX ixFilter ON #TestTab1 (Col1)WHERE Quantity = 122 Now if we execute SELECT COUNT(*) FROM #TestTab1 WHERE Quantity = 122 We will see the filtered index being scanned But if we parameterize the query DECLARE @i INT = 122SELECT COUNT(*) FROM #TestTab1 WHERE Quantity = @i The plan is very different a table scan, as the value of the variable used in the predicate can change at run time, and also we see the familiar warning triangle. If we now look at the properties pane, we will see two pieces of information “Warnings” and “UnmatchedIndexes”. So, handily, we are being told which filtered index is not being used due to parameterization.

    Read the article

  • JBox2D Polygon Collisions Acting Strange

    - by andy
    I have been playing around with JBox2D and Slick2D and made a little demo with a ground object, a box object, and two different polygons. The problem I am facing is that the collision-detection for the polygons seems to be off (see picture below), but the box's collision works fine. My Code: Main Class package main; import org.jbox2d.common.Vec2; import org.jbox2d.dynamics.BodyType; import org.jbox2d.dynamics.World; import org.newdawn.slick.GameContainer; import org.newdawn.slick.Graphics; import org.newdawn.slick.SlickException; import org.newdawn.slick.state.BasicGameState; import org.newdawn.slick.state.StateBasedGame; import shapes.Box; import shapes.Polygon; public class State1 extends BasicGameState{ World world; int velocityIterations; int positionIterations; float pixelsPerMeter; int state; Box ground; Box box1; Polygon poly1; Polygon poly2; Renderer renderer; public State1(int state) { this.state = state; } @Override public void init(GameContainer gc, StateBasedGame game) throws SlickException { velocityIterations = 10; positionIterations = 10; pixelsPerMeter = 1f; world = new World(new Vec2(0.f, -9.8f)); renderer = new Renderer(gc, gc.getGraphics(), pixelsPerMeter, world); box1 = new Box(-100f, 200f, 40, 50, BodyType.DYNAMIC, world); ground = new Box(-14, -275, 50, 900, BodyType.STATIC, world); poly1 = new Polygon(50f, 10f, new Vec2[] { new Vec2(-6f, -14f), new Vec2(0f, -20f), new Vec2(6f, -14f), new Vec2(10f, 10f), new Vec2(-10f, 10f) }, BodyType.DYNAMIC, world); poly2 = new Polygon(0f, 10f, new Vec2[] { new Vec2(10f, 0f), new Vec2(20f, 0f), new Vec2(30f, 10f), new Vec2(30f, 20f), new Vec2(20f, 30f), new Vec2(10f, 30f), new Vec2(0f, 20f), new Vec2(0f, 10f) }, BodyType.DYNAMIC, world); } @Override public void update(GameContainer gc, StateBasedGame game, int delta) throws SlickException { world.step((float)delta / 180f, velocityIterations, positionIterations); } @Override public void render(GameContainer gc, StateBasedGame game, Graphics g) throws SlickException { renderer.render(); } @Override public int getID() { return this.state; } } Polygon Class package shapes; import org.jbox2d.collision.shapes.PolygonShape; import org.jbox2d.common.Vec2; import org.jbox2d.dynamics.Body; import org.jbox2d.dynamics.BodyDef; import org.jbox2d.dynamics.BodyType; import org.jbox2d.dynamics.FixtureDef; import org.jbox2d.dynamics.World; import org.newdawn.slick.Color; public class Polygon { public float x, y; public Color color; public BodyType bodyType; org.newdawn.slick.geom.Polygon poly; BodyDef def; PolygonShape ps; FixtureDef fd; Body body; World world; Vec2[] verts; public Polygon(float x, float y, Vec2[] verts, BodyType bodyType, World world) { this.verts = verts; this.x = x; this.y = y; this.bodyType = bodyType; this.world = world; init(); } public void init() { def = new BodyDef(); def.type = bodyType; def.position.set(x, y); ps = new PolygonShape(); ps.set(verts, verts.length); fd = new FixtureDef(); fd.shape = ps; fd.density = 2.0f; fd.friction = 0.7f; fd.restitution = 0.5f; body = world.createBody(def); body.createFixture(fd); } } Rendering Class package main; import org.jbox2d.collision.shapes.PolygonShape; import org.jbox2d.collision.shapes.ShapeType; import org.jbox2d.common.MathUtils; import org.jbox2d.common.Vec2; import org.jbox2d.dynamics.Body; import org.jbox2d.dynamics.Fixture; import org.jbox2d.dynamics.World; import org.newdawn.slick.Color; import org.newdawn.slick.GameContainer; import org.newdawn.slick.Graphics; import org.newdawn.slick.geom.Polygon; import org.newdawn.slick.geom.Transform; public class Renderer { World world; float pixelsPerMeter; GameContainer gc; Graphics g; public Renderer(GameContainer gc, Graphics g, float ppm, World world) { this.world = world; this.pixelsPerMeter = ppm; this.g = g; this.gc = gc; } public void render() { Body current = world.getBodyList(); Vec2 center = current.getLocalCenter(); while(current != null) { Vec2 pos = current.getPosition(); g.pushTransform(); g.translate(pos.x * pixelsPerMeter + (0.5f * gc.getWidth()), -pos.y * pixelsPerMeter + (0.5f * gc.getHeight())); Fixture f = current.getFixtureList(); while(f != null) { ShapeType type = f.getType(); g.setColor(getColor(current)); switch(type) { case POLYGON: { PolygonShape shape = (PolygonShape)f.getShape(); Vec2[] verts = shape.getVertices(); int count = shape.getVertexCount(); Polygon p = new Polygon(); for(int i = 0; i < count; i++) { p.addPoint(verts[i].x, verts[i].y); } p.setCenterX(center.x); p.setCenterY(center.y); p = (Polygon)p.transform(Transform.createRotateTransform(current.getAngle() + MathUtils.PI, center.x, center.y)); p = (Polygon)p.transform(Transform.createScaleTransform(pixelsPerMeter, pixelsPerMeter)); g.draw(p); break; } case CIRCLE: { f.getShape(); } default: } f = f.getNext(); } g.popTransform(); current = current.getNext(); } } public Color getColor(Body b) { Color c = new Color(1f, 1f, 1f); switch(b.m_type) { case DYNAMIC: if(b.isActive()) { c = new Color(255, 123, 0); } else { c = new Color(99, 99, 99); } break; case KINEMATIC: break; case STATIC: c = new Color(111, 111, 111); break; default: break; } return c; } } Any help with fixing the collisions would be greatly appreciated, and if you need any other code snippets I would be happy to provide them.

    Read the article

  • OData &ndash; The easiest service I can create: now with updates

    - by Jon Dalberg
    The other day I created a simple NastyWord service exposed via OData. It was read-only and used an in-memory backing store for the words. Today I’ll modify it to use a file instead of a list and I’ll accept new nasty words by implementing IUpdatable directly. The first thing to do is enable the service to accept new entries. This is done at configuration time by adding the “WriteAppend” access rule: 1: public class NastyWords : DataService<NastyWordsDataSource> 2: { 3: // This method is called only once to initialize service-wide policies. 4: public static void InitializeService(DataServiceConfiguration config) 5: { 6: config.SetEntitySetAccessRule("*", EntitySetRights.AllRead | EntitySetRights.WriteAppend); 7: config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2; 8: } 9: }   Next I placed a file, NastyWords.txt, in the “App_Data” folder and added a few *choice* words to start. This required one simple change to our NastyWordDataSource.cs file: 1: public NastyWordsDataSource() 2: { 3: UpdateFromSource(); 4: } 5:   6: private void UpdateFromSource() 7: { 8: var words = File.ReadAllLines(pathToFile); 9: NastyWords = (from w in words 10: select new NastyWord { Word = w }).AsQueryable(); 11: }   Nothing too shocking here, just reading each line from the NastyWords.txt file and exposing them. Next, I implemented IUpdatable which comes with a boat-load of methods. We don’t need all of them for now since we are only concerned with allowing new values. Here are the methods we must implement, all the others throw a NotImplementedException: 1: public object CreateResource(string containerName, string fullTypeName) 2: { 3: var nastyWord = new NastyWord(); 4: pendingUpdates.Add(nastyWord); 5: return nastyWord; 6: } 7:   8: public object ResolveResource(object resource) 9: { 10: return resource; 11: } 12:   13: public void SaveChanges() 14: { 15: var intersect = (from w in pendingUpdates 16: select w.Word).Intersect(from n in NastyWords 17: select n.Word); 18:   19: if (intersect.Count() > 0) 20: throw new DataServiceException(500, "duplicate entry"); 21:   22: var lines = from w in pendingUpdates 23: select w.Word; 24:   25: File.AppendAllLines(pathToFile, 26: lines, 27: Encoding.UTF8); 28:   29: pendingUpdates.Clear(); 30:   31: UpdateFromSource(); 32: } 33:   34: public void SetValue(object targetResource, string propertyName, object propertyValue) 35: { 36: targetResource.GetType().GetProperty(propertyName).SetValue(targetResource, propertyValue, null); 37: }   I use a simple list to contain the pending updates and only commit them when the “SaveChanges” method is called. Here’s the order these methods are called in our service during an insert: CreateResource – here we just instantiate a new NastyWord and stick a reference to it in our pending updates list. SetValue – this is where the “Word” property of the NastyWord instance is set. SaveChanges – get the list of pending updates, barfing on duplicates, write them to the file and clear our pending list. ResolveResource – the newly created resource will be returned directly here since we aren’t dealing with “handles” to objects but the actual objects themselves. Not too bad, eh? I didn’t find this documented anywhere but a little bit of digging in the OData spec and use of Fiddler made it pretty easy to figure out. Here is some client code which would add a new nasty word: 1: static void Main(string[] args) 2: { 3: var svc = new ServiceReference1.NastyWordsDataSource(new Uri("http://localhost.:60921/NastyWords.svc")); 4: svc.AddToNastyWords(new ServiceReference1.NastyWord() { Word = "shat" }); 5:   6: svc.SaveChanges(); 7: }   Here’s all of the code so far for to implement the service: 1: using System; 2: using System.Collections.Generic; 3: using System.Data.Services; 4: using System.Data.Services.Common; 5: using System.Linq; 6: using System.ServiceModel.Web; 7: using System.Web; 8: using System.IO; 9: using System.Text; 10:   11: namespace ONasty 12: { 13: [DataServiceKey("Word")] 14: public class NastyWord 15: { 16: public string Word { get; set; } 17: } 18:   19: public class NastyWordsDataSource : IUpdatable 20: { 21: private List<NastyWord> pendingUpdates = new List<NastyWord>(); 22: private string pathToFile = @"path to your\App_Data\NastyWords.txt"; 23:   24: public NastyWordsDataSource() 25: { 26: UpdateFromSource(); 27: } 28:   29: private void UpdateFromSource() 30: { 31: var words = File.ReadAllLines(pathToFile); 32: NastyWords = (from w in words 33: select new NastyWord { Word = w }).AsQueryable(); 34: } 35:   36: public IQueryable<NastyWord> NastyWords { get; private set; } 37:   38: public void AddReferenceToCollection(object targetResource, string propertyName, object resourceToBeAdded) 39: { 40: throw new NotImplementedException(); 41: } 42:   43: public void ClearChanges() 44: { 45: pendingUpdates.Clear(); 46: } 47:   48: public object CreateResource(string containerName, string fullTypeName) 49: { 50: var nastyWord = new NastyWord(); 51: pendingUpdates.Add(nastyWord); 52: return nastyWord; 53: } 54:   55: public void DeleteResource(object targetResource) 56: { 57: throw new NotImplementedException(); 58: } 59:   60: public object GetResource(IQueryable query, string fullTypeName) 61: { 62: throw new NotImplementedException(); 63: } 64:   65: public object GetValue(object targetResource, string propertyName) 66: { 67: throw new NotImplementedException(); 68: } 69:   70: public void RemoveReferenceFromCollection(object targetResource, string propertyName, object resourceToBeRemoved) 71: { 72: throw new NotImplementedException(); 73: } 74:   75: public object ResetResource(object resource) 76: { 77: throw new NotImplementedException(); 78: } 79:   80: public object ResolveResource(object resource) 81: { 82: return resource; 83: } 84:   85: public void SaveChanges() 86: { 87: var intersect = (from w in pendingUpdates 88: select w.Word).Intersect(from n in NastyWords 89: select n.Word); 90:   91: if (intersect.Count() > 0) 92: throw new DataServiceException(500, "duplicate entry"); 93:   94: var lines = from w in pendingUpdates 95: select w.Word; 96:   97: File.AppendAllLines(pathToFile, 98: lines, 99: Encoding.UTF8); 100:   101: pendingUpdates.Clear(); 102:   103: UpdateFromSource(); 104: } 105:   106: public void SetReference(object targetResource, string propertyName, object propertyValue) 107: { 108: throw new NotImplementedException(); 109: } 110:   111: public void SetValue(object targetResource, string propertyName, object propertyValue) 112: { 113: targetResource.GetType().GetProperty(propertyName).SetValue(targetResource, propertyValue, null); 114: } 115: } 116:   117: public class NastyWords : DataService<NastyWordsDataSource> 118: { 119: // This method is called only once to initialize service-wide policies. 120: public static void InitializeService(DataServiceConfiguration config) 121: { 122: config.SetEntitySetAccessRule("*", EntitySetRights.AllRead | EntitySetRights.WriteAppend); 123: config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2; 124: } 125: } 126: } Next time we’ll allow removing nasty words. Enjoy!

    Read the article

  • Restoring databases to a set drive and directory

    - by okeofs
     Restoring databases to a set drive and directory Introduction Often people say that necessity is the mother of invention. In this case I was faced with the dilemma of having to restore several databases, with multiple ‘ndf’ files, and having to restore them with different physical file names, drives and directories on servers other than the servers from which they originated. As most of us would do, I went to Google to see if I could find some code to achieve this task and found some interesting snippets on Pinal Dave’s website. Naturally, I had to take it further than the code snippet, HOWEVER it was a great place to start. Creating a temp table to hold database file details First off, I created a temp table which would hold the details of the individual data files within the database. Although there are a plethora of fields (within the temp table below), I utilize LogicalName only within this example. The temporary table structure may be seen below:   create table #tmp ( LogicalName nvarchar(128)  ,PhysicalName nvarchar(260)  ,Type char(1)  ,FileGroupName nvarchar(128)  ,Size numeric(20,0)  ,MaxSize numeric(20,0), Fileid tinyint, CreateLSN numeric(25,0), DropLSN numeric(25, 0), UniqueID uniqueidentifier, ReadOnlyLSN numeric(25,0), ReadWriteLSN numeric(25,0), BackupSizeInBytes bigint, SourceBlocSize int, FileGroupId int, LogGroupGUID uniqueidentifier, DifferentialBaseLSN numeric(25,0), DifferentialBaseGUID uniqueidentifier, IsReadOnly bit, IsPresent bit,  TDEThumbPrint varchar(50) )    We now declare and populate a variable(@path), setting the variable to the path to our SOURCE database backup. declare @path varchar(50) set @path = 'P:\DATA\MYDATABASE.bak'   From this point, we insert the file details of our database into the temp table. Note that we do so by utilizing a restore statement HOWEVER doing so in ‘filelistonly’ mode.   insert #tmp EXEC ('restore filelistonly from disk = ''' + @path + '''')   At this point, I depart from what I gleaned from Pinal Dave.   I now instantiate a few more local variables. The use of each variable will be evident within the cursor (which follows):   Declare @RestoreString as Varchar(max) Declare @NRestoreString as NVarchar(max) Declare @LogicalName  as varchar(75) Declare @counter as int Declare @rows as int set @counter = 1 select @rows = COUNT(*) from #tmp  -- Count the number of records in the temp                                    -- table   Declaring and populating the cursor At this point I do realize that many people are cringing about the use of a cursor. Being an Oracle professional as well, I have learnt that there is a time and place for cursors. I would remind the reader that the data that will be read into the cursor is from a local temp table and as such, any locking of the records (within the temp table) is not really an issue.   DECLARE MY_CURSOR Cursor  FOR  Select LogicalName  From #tmp   Parsing the logical names from within the cursor. A small caveat that works in our favour,  is that the first logical name (of our database) is the logical name of the primary data file (.mdf). Other files, except for the very last logical name, belong to secondary data files. The last logical name is that of our database log file.   I now open my cursor and populate the variable @RestoreString Open My_Cursor  set @RestoreString =  'RESTORE DATABASE [MYDATABASE] FROM DISK = N''P:\DATA\ MYDATABASE.bak''' + ' with  '   We now fetch the first record from the temp table.   Fetch NEXT FROM MY_Cursor INTO @LogicalName   While there are STILL records left within the cursor, we dynamically build our restore string. Note that we are using concatenation to create ‘one big restore executable string’.   Note also that the target physical file name is hardwired, as is the target directory.   While (@@FETCH_STATUS <> -1) BEGIN IF (@@FETCH_STATUS <> -2) -- As long as there are no rows missing select @RestoreString = case  when @counter = 1 then -- This is the mdf file    @RestoreString + 'move  N''' + @LogicalName + '''' + ' TO N’’X:\DATA1\'+ @LogicalName + '.mdf' + '''' + ', '   -- OK, if it passes through here we are dealing with an .ndf file -- Note that Counter must be greater than 1 and less than the number of rows.   when @counter > 1 and @counter < @rows then -- These are the ndf file(s)    @RestoreString + 'move  N''' + @LogicalName + '''' + ' TO N’’X:\DATA1\'+ @LogicalName + '.ndf' + '''' + ', '   -- OK, if it passes through here we are dealing with the log file When @LogicalName like '%log%' then    @RestoreString + 'move  N''' + @LogicalName + '''' + ' TO N’’X:\DATA1\'+ @LogicalName + '.ldf' +'''' end --Increment the counter   set @counter = @counter + 1 FETCH NEXT FROM MY_CURSOR INTO @LogicalName END   At this point we have populated the varchar(max) variable @RestoreString with a concatenation of all the necessary file names. What we now need to do is to run the sp_executesql stored procedure, to effect the restore.   First, we must place our ‘concatenated string’ into an nvarchar based variable. Obviously this will only work as long as the length of @RestoreString is less than varchar(max) / 2.   set @NRestoreString = @RestoreString EXEC sp_executesql @NRestoreString   Upon completion of this step, the database should be restored to the server. I now close and deallocate the cursor, and to be clean, I would also drop my temp table.   CLOSE MY_CURSOR DEALLOCATE MY_CURSOR GO   Conclusion Restoration of databases on different servers with different physical names and on different drives are a fact of life. Through the use of a few variables and a simple cursor, we may achieve an efficient and effective way to achieve this task.

    Read the article

  • top tweets WebLogic Partner Community – June 2012

    - by JuergenKress
    Send your tweets @wlscommunity #WebLogicCommunity and follow us at http://twitter.com/wlscommunity OTNArchBeat? Free Virtual Developer Day: Oracle ADF and Oracle Fusion Middleware Development http://bit.ly/MxuNAg AMIS, Oracle & Java? Checklist veearts nu ook op iPad. @amis_services Mobile integratie met Oracle Fusion Middleware http://dld.bz/buwsM #OSB #SOA WhitehorsesWhiteblog: Troubleshoot JVM crashes of Weblogic: CompilerThread (http://bit.ly/KcGzZK) Jon petter hjulstad E-vita is now Apps Grid Specialized! ODTUG Fusion Middleware Sessions RT @OTNArchBeat: ODTUG Kscope12 - June 24-28 - San Antonio, TX http://bit.ly/LlWkNV OTNArchBeat? Free Event: Modern #Java Development, in/outside the Enterprise - May 30 - Redwood Shores, CA http://bit.ly/LfB79a ADF Community DE? Oracle Advanced ADF 11g Partner Workshop Düsseldorf /Germany (english) June 26-29, click here to see Nicolas Lorain? Best Practices for #JavaFX 2 Enterprise Applications (Part Two) http://buff.ly/Lk1DBn by Jim Weaver shay shmeltzer? #Oracle Developers in #Israel - don't miss the free #ADF workshop July 2nd - get hands-on with Oracle ADF -here OTNArchBeat? Java at JAXconf | Tori Wieldt http://bit.ly/LdoLS2 Anand Akela? #Oracle Customers and Partners – Get your free pass to @CloudExpo in New York, June 11 to 14, http://goo.gl/RpYFT <- Stop by booth #511 OracleSupport_WLS? Did you know that since 3/15/12 #WebLogic Server 12.1.1.0 is certified for production with JDK 7? http://bit.ly/IYJE0L Sharat? Highly useful #JavaFX best practices blog by @JavaFXpert More details here ADF EMG How to set up a productive ADF Dev Env - discussion started by @baigsorcl. Click here to Read and comment. OracleSupport_WLS Upcoming #webcast: Diagnosing #weblogic performance issues through #java thread dumps http://bit.ly/M4O9qF My Oracle Support? New to Oracle Support? - Webcast on Support Basics webcast May 22 10:30 Central Europe. Register @ http://bit.ly/J8o0WG Mohamad Afshar? Cloud Expo – Oracle Customers and Partners – get your free pass to Cloud Expo in New York, June 11 to 14, http://goo.gl/RpYFT OTNArchBeat Oracle VM 3.1 is here | @Ronenkofman http://bit.ly/JriWTq Oracle Exalogic? RT @D0uglasPhillips: ExalogicTV New Video Introducing Oracle Secure Global Desktop for #Exalogic!! http://bit.ly/nwkrCu OracleBlogs? Java EE6 and WebLogic YouTube video channels http://ow.ly/1jVcYJ Oracle WebLogic RT @aleftik: Excited to spend some time today playing around with the WebSockets SDK http://bit.ly/NoTtri WebLogic Community Java EE6 and WebLogic YouTube video channels http://wp.me/p1LMIb-h0 OracleSupport_WLS New tutorial! How to use the #JMS #API to create a message producer with #GlassFish and #NetBeans http://bit.ly/Juqjn JDeveloper & ADF? Tip when installing JDeveloper 11.1.2.2.0 version http://dlvr.it/1b48s1 WebLogic Community Middleware Oracle Excellence Awards 2012 – HAPPY NEW YEAR! Click here to read WebLogicCommunity #opn #oracle#Specialization #opnaward Steven Davelaar? Improve performance of your ADF app using lazy, on-demand querying of detail view objects: Click here OracleBlogs? Middleware Oracle Excellence Awards 2012 & HAPPY NEW YEAR! http://ow.ly/1kahzZ OracleSupport_WLS Upgrading from #weblogic 9.2.x to 10.3.x? http://bit.ly/Kqzl9N AMIS, Oracle & Java “@JDeveloper: Logout from an ADF application http://dlvr.it/1fQBnm” WebLogic Community UK OUG call for papers–your middleware success! Click here #UKOUG #soacommunity #OPN Whitehorses Whiteblog: Enterprise Manager: Manage your Fusion Middleware logfiles (http://bit.ly/KQlZkR) WebLogic Community? @Jphjulstad HI Jon, should we send Pizza when you go in production with your WebLogic 12c project? Whish you success! #WebLogicCommunity Sabine Leitner ADF Einsteigerworkshops je 2 Tage im Juni in HAM, BLN, HANN #Oracle #WLS http://bit.ly/LcOIzB @OracleWebLogic @OracleAppGrid@soacommunity Andreas Koop new post Java Heap Monitor in JDeveloper http://bit.ly/LgSk85 Sabine Leitner? #Oracle Kundentag mit Vorträgen von Sparkasse, Schufa, LBBW, Allianz über FMW & Exa Lösungen! 21.06. FRA http://bit.ly/JtwE3v @wlscommunity NetBeans Team RT @chadlung: Installing and configuring #NetBeans 7.1.2 and the #Java JDK 1.7 on OS X: http://www.giantflyingsaucer.com/blog/p=3760 #osx WebLogic Community Happy New Year #WeblogicCommunity thanks for the business! Time for a drink http://pic.twitter.com/K34KFbvH WebLogic Community UK OUG call for papers&ndash;your middleware success! http://wp.me/p1LMIb-gU WebLogic Community? Middleware Oracle Excellence Awards 2012 - HAPPY NEW YEAR! http://wp.me/p1LMIb-h6 Oracle WebLogic? RT @wlscommunity: WebLogic World Record Two Processor Result with SPECjEnterprise2010 Benchmark Click here to read #weblogic #sunfire #li Marc? Relocate wlst script for all the logfiles in your domain @wlscommunity, http://tinyurl.com/btbjcco WebLogic Community WebLogic World Record Two Processor Result with SPECjEnterprise2010 Benchmark Click here #WebLogicCommunity #weblogic #sunfire Oracle WebLogic MIss a WebLogic Devcast webinar? Catch any of the replays in the series on-demand! #WebLogic #JavaEE #coherence http://bit.ly/LNGa4p JDeveloper & ADF? Bean DataControl - Edit table records http://dlvr.it/1ZWqCx Justin Kestelyn? Contents of "Virtual Developer Day: Java SE 7 and JavaFX 2.0" are now avail on demand; no reg http://tinyurl.com/78nxnyo Frank Nimphius? Preparing 12c new features for DOAG 2012 Development - June 14th in Bonn (http://development.doag.org) WebLogic Community? Middleware Oracle Excellence Awards 2012&ndash;HAPPY NEW YEAR! http://wp.me/p1LMIb-he JDeveloper & ADF Placeholder Watermarks with ADF 11.1.2 http://dlvr.it/1ZWDc9 Oracle ACE Program? May edition #ACE newsletter now available online. http://bit.ly/LKA2de chriscmuir New blog post: Which JDeveloper is right for me? http://bit.ly/J8sj9e GlassFish? Transactional Interceptors in Java EE 7 - Request for feedback: Linda described how EJB's container-managed tr http://bit.ly/KKuGNJ OracleEnterpriseMgr Oracle Application Testing Suite 12.1 Debuts at StarEast 2012 http://ow.ly/aXcv8 #em12c JAX London First set of speaker session announced for #JAXLondon see: http://bit.ly/L0HSME OTNArchBeat? Oracle Cloud Conference: dates and locations worldwide http://bit.ly/JgNeID NetBeans Team? Video: Create and debug a TestNG test class in #NetBeans IDE: http://ow.ly/b7NEW NetBeans Team #NetBeans tip: Code Template for #Kohana #PHP Framework: http://ow.ly/aWIvY Robin? Started to use the #Oracle #WebLogic Server #Maven Plugin. Really awesome to install a complete #WLS with "mvn wls:install" !@wlscommunity OTNArchBeat? Free Event: Modern #Java Development, in/outside the Enterprise - May 30 - Redwood Shores, CA http://bit.ly/JIN9tf OracleBlogs WebLogic Partner Community Newsletter May 2012 http://ow.ly/1k5TeG Java Certification? Java SE 7 Fundamentals course now available On Demand. Watch a preview now: http://ow.ly/aWYgD Whitehorses Whiteblog: Native IO in WebLogic on Solaris 11 X64 (http://bit.ly/KGM4mp) NetBeans Team? Quick video of FindBugs Integration in #NetBeans IDE 7.2: http://ow.ly/aNece NetBeans Team #JavaFX Scene Builder Docs Updated for 2.2 and #NetBeans 7.2 dev builds: http://ow.ly/b7Nie Duncan Mills? New blog posting on implementing input field watermarks with ADF Faces 11.1.2 Click here #adf WebLogic Community? WebLogic Partner Community Newsletter May 2012 http://wp.me/p1LMIb-h4 OracleBlogs? UK OUG call for papersyour middleware success! http://ow.ly/1jNs49 Nicolas Lorain? Java tip: Deploying #JavaFX apps to multiple environments - JavaWorld http://buff.ly/KDADvu Adam Bien? Java EE and How to Specify The Unconventional With Convention Over Configuration [Free Article]: The free http://bit.ly/JEUkUf Owen Hughes and team?#Oracle #Exalogic #Performance: What? How? Why? Click here GlassFish? SecuritEE in the Cloud: Java EE 7 and the Cloud theme continue to move full steam ahead. In a PaaS environment http://bit.ly/K2RPte JDeveloper & ADF? How to Align Managed Bean Scope and Bean Data Control in Oracle ADF http://dlvr.it/1dngxQ Andrejus Baranovskis Missing New Feature in JDev (11.1.2.2.0) - ADF Methods Security http://fb.me/1jQM1enls OracleSupport_WLS? Tutorial on managing #HTTP Sessions in a #Weblogic #Cluster http://bit.ly/JshESe Oracle WebLogic? ZeroTurnaround developer report: #Spring keeps getting heavier, and #Java EE keeps getting lighter http://bit.ly/JDmKy2 JDeveloper & ADF? How to Search in Views - Part 4 || Oracle ADF http://dlvr.it/1dpDjZ WebLogic Community Java Message Service with Java and Spring Framework on Oracle WebLogic; Webcast May 15th 2012 http://wp.me/p1LMIb-gS Andreas Koop? new post ADF Bug or Feature? Non-Breaking Space outside required icon style http://bit.ly/KDZnUo Oracle WebLogic? Don't miss this month's WebLogic DevCast: WebLogic JMS and Spring JMS http://bit.ly/J6g2ST Tuesday May 15th 10:00am PT JDeveloper & ADF How To Disable SELECT COUNT Execution for ADF Table Rendering http://dlvr.it/1dqKH6 OracleSupport_WLS? #SSL and security has its own Information Center, http://bit.ly/LP8Vil for troubleshooting, install, config and more NetBeans Team? Featured #NetBeans plugin is @Codename_One for creating native apps for major mobile platforms: http://plugins.netbeans.org/ JDeveloper & ADF? Using JDeveloper HTTP Analyser to intercept/forward requests http://dlvr.it/1Yzl4J Nicolas Lorain? Create native looks for JavaFX applications: JavaFX-CSS-Themes · http://buff.ly/M0jel0 by Gregg Setzer Devoxx? Want to make the world a better place? Then get involved in Random Hacks of Kindness on June 2 - 3 in Belgium @ http://www.rhok.be #RHoK WebLogic Community top tweets WebLogic Partner Community – May 2012 Click here #WebLogicCommunity Michel Schildmeijer Oracle Traffic Director 11g http://lnkd.in/-mm3Vy Andrejus Baranovskis? Proactively Monitoring JDeveloper 11g IDE Heap Memory http://fb.me/16YZErPrx Arun Gupta? 80+ attendees building a #javaee6 application using NetBeans/WebLogic at Java Day, Istanbul fun times! http://pic.twitter.com/odY19daW A. Chatziantoniou? Just registered for the Oracle FMW Summer Camp in Lisbon. Looking forward to learn, meet friends and try to buy ice cream on the beach OTNArchBeat Another Myth Debunked: 200 Continuous Redeployments with WebLogic|@munz http://bit.ly/JiPyM7 Oracle WebLogic? Need to learn more on #WebLogic Server #JVM performance tuning? http://bit.ly/MN UxHx GlassFish? Dukes Choice Awards 2012 Nominations Are Open: 2012 Duke's Choice Award are open for nominations. These awards http://bit.ly/Ksk4U3 Justin Kestelyn? Major cloud-related announcements from Larry Ellison and Mark Hurd on June 6 http://bit.ly/KTJiII Nicolas Lorain Transparent Windows (Stage) with #JavaFX 2 : Adam Bien's Weblog http://j.mp/INgq8K WebLogic Community Web Services with JAX and Spring on WebLogic–Webcast May 30th 2012 #WebLogicCommunity #weblogic #opn JDeveloper & ADF Oracle ADF - How to work with Dates http://dlvr.it/1Y70zw OracleBlogs Web Services with JAX and Spring on WebLogicWebcast May 30th 2012 http://ow.ly/1k2WtO Adam Bien? Summer Java EE Workshops: 23.05, Amsterdam Airport Java EE Hacking, Without Airport. The dutch version of Airport http://bit.ly/JeP6hV JDeveloper & ADF ADF 11g: BC4J or EJB3. http://bit.ly/JVVFZF ADF EMG? Great discussion with JSF guru Andy Schwartz on the forum - 38 posts! Check it out: here Devoxx? Oracle (http://www.oracle.com ) joins Devoxx 2012 as the first Premium partner, welcome aboard! Nicolas Lorain Developing a Simple Todo Application using #JavaFX, #Java and #MongoDB- Part-1JavaBeat http://j.mp/IDGxLA Nicolas Lorain Preview of JavaFX 2.2 canvas feature > Harmonic Code: Death bitmaps could be beautiful... Part I http://buff.ly/KyAXg5 #JavaFX OTNArchBeat?? New York Coherence Special Interest Group (NYCSIG) - May 24 - NYC http://bit.ly/JzJcbT WebLogic Community iAS upgrade to WebLogic watch #C2B2 online seminar http://youtu.be/5m2CNUjBIGQ #WebLogicCommunity Ruth Collett? Join Oracle in #Joburg on May 21 for OTN Developer Day - sessions on #Java #JavaEE 6/7 and much more! http://bit.ly/IENwnD WebLogic Community? Sending out invitations to our advanced Fusion Middleware Summer Camps! Want to learn more register for the community Ruth Collett? Join @ArunGupta in Istanbul this Monday to hear the latest on #JavaEE 6/7 http://bit.ly/Je63cc GlassFish? NetBeans 7.2 Beta - Built for Speed, Deploy Apps to Oracle Cloud: NetBeans 7.2 Beta is now available. The http://bit.ly/LxMMTK Lucas Jellema My latest SlideShare upload : Java ain't scary - introducing Java to PL/SQ. here via @slideshare JDeveloper & ADF? #Developer #free#ADF training in #Scotland - June 13. More information: http://bit.ly/LbPLlf AMIS, Oracle & Java? AMIS behaalt als eerste in Nedeland de Oracle ADF specialisatie - Channelworld nieuwsChannelconnect: http://bit.ly/JzAcB4 WebLogic Community Web Services with JAX and Spring on WebLogic&ndash;Webcast May 30th 2012 http://wp.me/p1LMIb-gX Nicolas Lorain?@ JavaFX-based SimpleDateFormat Demonstrator http://j.mp/KFCVOi #JavaFX via Dustin Marx Oracle Exalogic? Are you an Oracle partner? There's news on the Oracle Partner Network about #Exalogic specializations - http://bit.ly/Mt3ANY JDeveloper & ADF Shorter URL for your ADF application http://dlvr.it/1XqNLY OTNArchBeat? Bay Area Coherence Special Interest Group (BACSIG) Meeting June 7 http://bit.ly/JAa0Lx OTNArchBeat? Java EE 6 Sample Application on WebLogic 12c: Conference Planner | @arungupta http://bit.ly/LPvof4 JDeveloper & ADF? Excellent example of Oracle ADF - Google Maps/Earth integration http://dlvr.it/1cbc80 JDeveloper & ADF Setting Up JDeveloper's Embedded WLS for MySQL http://dlvr.it/1c4b8P JDeveloper & ADF? Solution for Sharing Global User Data in ADF BC http://dlvr.it/1cc7SJ Java? Java Magazine May/June #javaee #javafx #javame #openJDK #hotspot #wicket #lotsmore http://ow.ly/aX07v Oracle WebLogic? http://bit.ly/JxQsnS if you have trouble finding the right #patchset when doing an upgrade to your #weblogic server OracleEnterpriseMgr 15 minutes to go before we start our Application Testing Suite 12.1 webcast. http://bit.ly/JHyTEe Learn from the lead PM what's new. #em12c Sten Vesterli Eating your own dog food - Oracle support site finally in ADF: http://lnkd.in/s6hg_p Adam Bien Project: "Jenever" (=poison) checked-in with GIT:here CU at http://workshops.adam-bien.com. Thanks for attending! OTNArchBeat Web Service Development with NetBeans and Testing with WebLogic Admin Console | @munz http://bit.ly/JcWk34 Please feel free to send us your news! And add your blog to our SOA blog wiki

    Read the article

  • The Art of Productivity

    - by dwahlin
    Getting things done has always been a challenge regardless of gender, age, race, skill, or job position. No matter how hard some people try, they end up procrastinating tasks until the last minute. Some people simply focus better when they know they’re out of time and can’t procrastinate any longer. How many times have you put off working on a term paper in school until the very last minute? With only a few hours left your mental energy and focus seem to kick in to high gear especially as you realize that you either get the paper done now or risk failing. It’s amazing how a little pressure can turn into a motivator and allow our minds to focus on a given task. Some people seem to specialize in procrastinating just about everything they do while others tend to be the “doers” who get a lot done and ultimately rise up the ladder at work. What’s the difference between these types of people? Is it pure laziness or are other factors at play? I think that some people are certainly more motivated than others, but I also think a lot of it is based on the process that “doers” tend to follow - whether knowingly or unknowingly. While I’ve certainly fought battles with procrastination, I’ve always had a knack for being able to get a lot done in a relatively short amount of time. I think a lot of my “get it done” attitude goes back to the the strong work ethic my parents instilled in me at a young age. I remember my dad saying, “You need to learn to work hard!” when I was around 5 years old. I remember that moment specifically because I was on a tractor with him the first time I heard it while he was trying to move some large rocks into a pile. The tractor was big but so were the rocks and my dad had to balance the tractor perfectly so that it didn’t tip forward too far. It was challenging work and somewhat tedious but my dad finished the task and taught me a few important lessons along the way including persistence, the importance of having a skill, and getting the job done right without skimping along the way. In this post I’m going to list a few of the techniques and processes I follow that I hope may be beneficial to others. I blogged about the general concept back in 2009 but thought I’d share some updated information and lessons learned since then. Most of the ideas that follow came from learning and refining my daily work process over the years. However, since most of the ideas are common sense (at least in my opinion), I suspect they can be found in other productivity processes that are out there. Let’s start off with one of the most important yet simple tips: Start Each Day with a List. Start Each Day with a List What are you planning to get done today? Do you keep track of everything in your head or rely on your calendar? While most of us think that we’re pretty good at managing “to do” lists strictly in our head you might be surprised at how affective writing out lists can be. By writing out tasks you’re forced to focus on the most important tasks to accomplish that day, commit yourself to those tasks, and have an easy way to track what was supposed to get done and what actually got done. Start every morning by making a list of specific tasks that you want to accomplish throughout the day. I’ll even go so far as to fill in times when I’d like to work on tasks if I have a lot of meetings or other events tying up my calendar on a given day. I’m not a big fan of using paper since I type a lot faster than I write (plus I write like a 3rd grader according to my wife), so I use the Sticky Notes feature available in Windows. Here’s an example of yesterday’s sticky note: What do you add to your list? That’s the subject of the next tip. Focus on Small Tasks It’s no secret that focusing on small, manageable tasks is more effective than trying to focus on large and more vague tasks. When you make your list each morning only add tasks that you can accomplish within a given time period. For example, if I only have 30 minutes blocked out to work on an article I don’t list “Write Article”. If I do that I’ll end up wasting 30 minutes stressing about how I’m going to get the article done in 30 minutes and ultimately get nothing done. Instead, I’ll list something like “Write Introductory Paragraphs for Article”. The next day I may add, “Write first section of article” or something that’s small and manageable – something I’m confident that I can get done. You’ll find that once you’ve knocked out several smaller tasks it’s easy to continue completing others since you want to keep the momentum going. In addition to keeping my tasks focused and small, I also make a conscious effort to limit my list to 4 or 5 tasks initially. I’ve found that if I list more than 5 tasks I feel a bit overwhelmed which hurts my productivity. It’s easy to add additional tasks as you complete others and you get the added benefit of that confidence boost of knowing that you’re being productive and getting things done as you remove tasks and add others. Getting Started is the Hardest (Yet Easiest) Part I’ve always found that getting started is the hardest part and one of the biggest contributors to procrastination. Getting started working on tasks is a lot like getting a large rock pushed to the bottom of a hill. It’s difficult to get the rock rolling at first, but once you manage to get it rocking some it’s really easy to get it rolling on its way to the bottom. As an example, I’ve written 100s of articles for technical magazines over the years and have really struggled with the initial introductory paragraphs. Keep in mind that these are the paragraphs that don’t really add that much value (in my opinion anyway). They introduce the reader to the subject matter and nothing more. What a waste of time for me to sit there stressing about how to start the article. On more than one occasion I’ve spent more than an hour trying to come up with 2-3 paragraphs of text.  Talk about a productivity killer! Whether you’re struggling with a writing task, some code for a project, an email, or other tasks, jumping in without thinking too much is the best way to get started I’ve found. I’m not saying that you shouldn’t have an overall plan when jumping into a task, but on some occasions you’ll find that if you simply jump into the task and stop worrying about doing everything perfectly that things will flow more smoothly. For my introductory paragraph problem I give myself 5 minutes to write out some general concepts about what I know the article will cover and then spend another 10-15 minutes going back and refining that information. That way I actually have some ideas to work with rather than a blank sheet of paper. If I still find myself struggling I’ll write the rest of the article first and then circle back to the introductory paragraphs once I’m done. To sum this tip up: Jump into a task without thinking too hard about it. It’s better to to get the rock at the top of the hill rocking some than doing nothing at all. You can always go back and refine your work.   Learn a Productivity Technique and Stick to It There are a lot of different productivity programs and seminars out there being sold by companies. I’ve always laughed at how much money people spend on some of these motivational programs/seminars because I think that being productive isn’t that hard if you create a re-useable set of steps and processes to follow. That’s not to say that some of these programs/seminars aren’t worth the money of course because I know they’ve definitely benefited some people that have a hard time getting things done and staying focused. One of the best productivity techniques I’ve ever learned is called the “Pomodoro Technique” and it’s completely free. This technique is an extremely simple way to manage your time without having to remember a bunch of steps, color coding mechanisms, or other processes. The technique was originally developed by Francesco Cirillo in the 80s and can be implemented with a simple timer. In a nutshell here’s how the technique works: Pick a task to work on Set the timer to 25 minutes and work on the task Once the timer rings record your time Take a 5 minute break Repeat the process Here’s why the technique works well for me: It forces me to focus on a single task for 25 minutes. In the past I had no time goal in mind and just worked aimlessly on a task until I got interrupted or bored. 25 minutes is a small enough chunk of time for me to stay focused. Any distractions that may come up have to wait until after the timer goes off. If the distraction is really important then I stop the timer and record my time up to that point. When the timer is running I act as if I only have 25 minutes total for the task (like you’re down to the last 25 minutes before turning in your term paper….frantically working to get it done) which helps me stay focused and turns into a “beat the clock” type of game. It’s actually kind of fun if you treat it that way and really helps me focus on a the task at hand. I automatically know how much time I’m spending on a given task (more on this later) by using this technique. I know that I have 5 minutes after each pomodoro (the 25 minute sprint) to waste on anything I’d like including visiting a website, stepping away from the computer, etc. which also helps me stay focused when the 25 minute timer is counting down. I use this technique so much that I decided to build a program for Windows 8 called Pomodoro Focus (I plan to blog about how it was built in a later post). It’s a Windows Store application that allows people to track tasks, productive time spent on tasks, interruption time experienced while working on a given task, and the number of pomodoros completed. If a time estimate is given when the task is initially created, Pomodoro Focus will also show the task completion percentage. I like it because it allows me to track my tasks, time spent on tasks (very useful in the consulting world), and even how much time I wasted on tasks (pressing the pause button while working on a task starts the interruption timer). I recently added a new feature that charts productive and interruption time for tasks since I wanted to see how productive I was from week to week and month to month. A few screenshots from the Pomodoro Focus app are shown next, I had a lot of fun building it and use it myself to as I work on tasks.   There are certainly many other productivity techniques and processes out there (and a slew of books describing them), but the Pomodoro Technique has been the simplest and most effective technique I’ve ever come across for staying focused and getting things done.   Persistence is Key Getting things done is great but one of the biggest lessons I’ve learned in life is that persistence is key especially when you’re trying to get something done that at times seems insurmountable. Small tasks ultimately lead to larger tasks getting accomplished, however, it’s not all roses along the way as some of the smaller tasks may come with their own share of bumps and bruises that lead to discouragement about the end goal and whether or not it is worth achieving at all. I’ve been on several long-term projects over my career as a software developer (I have one personal project going right now that fits well here) and found that repeating, “Persistence is the key!” over and over to myself really helps. Not every project turns out to be successful, but if you don’t show persistence through the hard times you’ll never know if you succeeded or not. Likewise, if you don’t persistently stick to the process of creating a daily list, follow a productivity process, etc. then the odds of consistently staying productive aren’t good.   Track Your Time How much time do you actually spend working on various tasks? If you don’t currently track time spent answering emails, on phone calls, and working on various tasks then you might be surprised to find out that a task that you thought was going to take you 30 minutes ultimately ended up taking 2 hours. If you don’t track the time you spend working on tasks how can you expect to learn from your mistakes, optimize your time better, and become more productive? That’s another reason why I like the Pomodoro Technique – it makes it easy to stay focused on tasks while also tracking how much time I’m working on a given task.   Eliminate Distractions I blogged about this final tip several years ago but wanted to bring it up again. If you want to be productive (and ultimately successful at whatever you’re doing) then you can’t waste a lot of time playing games or on Twitter, Facebook, or other time sucking websites. If you see an article you’re interested in that has no relation at all to the tasks you’re trying to accomplish then bookmark it and read it when you have some spare time (such as during a pomodoro break). Fighting the temptation to check your friends’ status updates on Facebook? Resist the urge and realize how much those types of activities are hurting your productivity and taking away from your focus. I’ll admit that eliminating distractions is still tough for me personally and something I have to constantly battle. But, I’ve made a conscious decision to cut back on my visits and updates to Facebook, Twitter, Google+ and other sites. Sure, my Klout score has suffered as a result lately, but does anyone actually care about those types of scores aside from your online “friends” (few of whom you’ve actually met in person)? :-) Ultimately it comes down to self-discipline and how badly you want to be productive and successful in your career, life goals, hobbies, or whatever you’re working on. Rather than having your homepage take you to a time wasting news site, game site, social site, picture site, or others, how about adding something like the following as your homepage? Every time your browser opens you’ll see a personal message which helps keep you on the right track. You can download my ubber-sophisticated homepage here if interested. Summary Is there a single set of steps that if followed can ultimately lead to productivity? I don’t think so since one size has never fit all. Every person is different, works in their own unique way, and has their own set of motivators, distractions, and more. While I certainly don’t consider myself to be an expert on the subject of productivity, I do think that if you learn what steps work best for you and gradually refine them over time that you can come up with a personal productivity process that can serve you well. Productivity is definitely an “art” that anyone can learn with a little practice and persistence. You’ve seen some of the steps that I personally like to follow and I hope you find some of them useful in boosting your productivity. If you have others you use please leave a comment. I’m always looking for ways to improve.

    Read the article

  • CLSF & CLK 2013 Trip Report by Jeff Liu

    - by jamesmorris
    This is a contributed post from Jeff Liu, lead XFS developer for the Oracle mainline Linux kernel team. Recently, I attended both the China Linux Storage and Filesystem workshop (CLSF), and the China Linux Kernel conference (CLK), which were held in Shanghai. Here are the highlights for both events. CLSF - 17th October XFS update (led by Jeff Liu) XFS keeps rapid progress with a lot of changes, especially focused on the infrastructure/performance improvements as well as  new feature development.  This can be reflected with a sample statistics among XFS/Ext4+JBD2/Btrfs via: # git diff --stat --minimal -C -M v3.7..v3.12-rc4 -- fs/xfs|fs/ext4+fs/jbd2|fs/btrfs XFS: 141 files changed, 27598 insertions(+), 19113 deletions(-) Ext4+JBD2: 39 files changed, 10487 insertions(+), 5454 deletions(-) Btrfs: 70 files changed, 19875 insertions(+), 8130 deletions(-) What made up those changes in XFS? Self-describing metadata(CRC32c). This is a new feature and it contributed about 70% code changes, it can be enabled via `mkfs.xfs -m crc=1 /dev/xxx` for v5 superblock. Transaction log space reservation improvements. With this change, we can calculate the log space reservation at mount time rather than runtime to reduce the the CPU overhead. User namespace support. So both XFS and USERNS can be enabled on kernel configuration begin from Linux 3.10. Thanks Dwight Engen's efforts for this thing. Split project/group quota inodes. Originally, project quota can not be enabled with group quota at the same time because they were share the same quota file inode, now it works but only for v5 super block. i.e, CRC enabled. CONFIG_XFS_WARN, an new lightweight runtime debugger which can be deployed in production environment. Readahead log object recovery, this change can speed up the log replay progress significantly. Speculative preallocation inode tracking, clearing and throttling. The main purpose is to deal with inodes with post-EOF space due to speculative preallocation, support improved quota management to free up a significant amount of unwritten space when at or near EDQUOT. It support backgroup scanning which occurs on a longish interval(5 mins by default, tunable), and on-demand scanning/trimming via ioctl(2). Bitter arguments ensued from this session, especially for the comparison between Ext4 and Btrfs in different areas, I have to spent a whole morning of the 1st day answering those questions. We basically agreed on XFS is the best choice in Linux nowadays because: Stable, XFS has a good record in stability in the past 10 years. Fengguang Wu who lead the 0-day kernel test project also said that he has observed less error than other filesystems in the past 1+ years, I own it to the XFS upstream code reviewer, they always performing serious code review as well as testing. Good performance for large/small files, XFS does not works very well for small files has already been an old story for years. Best choice (maybe) for distributed PB filesystems. e.g, Ceph recommends delopy OSD daemon on XFS because Ext4 has limited xattr size. Best choice for large storage (>16TB). Ext4 does not support a single file more than around 15.95TB. Scalability, any objection to XFS is best in this point? :) XFS is better to deal with transaction concurrency than Ext4, why? The maximum size of the log in XFS is 2038MB compare to 128MB in Ext4. Misc. Ext4 is widely used and it has been proved fast/stable in various loads and scenarios, XFS just need more customers, and Btrfs is still on the road to be a manhood. Ceph Introduction (Led by Li Wang) This a hot topic.  Li gave us a nice introduction about the design as well as their current works. Actually, Ceph client has been included in Linux kernel since 2.6.34 and supported by Openstack since Folsom but it seems that it has not yet been widely deployment in production environment. Their major work is focus on the inline data support to separate the metadata and data storage, reduce the file access time, i.e, a file access need communication twice, fetch the metadata from MDS and then get data from OSD, and also, the small file access is limited by the network latency. The solution is, for the small files they would like to store the data at metadata so that when accessing a small file, the metadata server can push both metadata and data to the client at the same time. In this way, they can reduce the overhead of calculating the data offset and save the communication to OSD. For this feature, they have only run some small scale testing but really saw noticeable improvements. Test environment: Intel 2 CPU 12 Core, 64GB RAM, Ubuntu 12.04, Ceph 0.56.6 with 200GB SATA disk, 15 OSD, 1 MDS, 1 MON. The sequence read performance for 1K size files improved about 50%. I have asked Li and Zheng Yan (the core developer of Ceph, who also worked on Btrfs) whether Ceph is really stable and can be deployed at production environment for large scale PB level storage, but they can not give a positive answer, looks Ceph even does not spread over Dreamhost (subject to confirmation). From Li, they only deployed Ceph for a small scale storage(32 nodes) although they'd like to try 6000 nodes in the future. Improve Linux swap for Flash storage (led by Shaohua Li) Because of high density, low power and low price, flash storage (SSD) is a good candidate to partially replace DRAM. A quick answer for this is using SSD as swap. But Linux swap is designed for slow hard disk storage, so there are a lot of challenges to efficiently use SSD for swap. SWAPOUT swap_map scan swap_map is the in-memory data structure to track swap disk usage, but it is a slow linear scan. It will become a bottleneck while finding many adjacent pages in the use of SSD. Shaohua Li have changed it to a cluster(128K) list, resulting in O(1) algorithm. However, this apporoach needs restrictive cluster alignment and only enabled for SSD. IO pattern In most cases, the swap io is in interleaved pattern because of mutiple reclaimers or a free cluster is shared by all reclaimers. Even though block layer can merge interleaved IO to some extent, but we cannot count on it completely. Hence the per-cpu cluster is added base on the previous change, it can help reclaimer do sequential IO and the block layer will be easier to merge IO. TLB flush: If we're reclaiming one active page, we should first move the page from active lru list to inactive lru list, and then reclaim the page from inactive lru to swap it out. During the process, we need to clear PTE twice: first is 'A'(ACCESS) bit, second is 'P'(PRESENT) bit. Processors need to send lots of ipi which make the TLB flush really expensive. Some works have been done to improve this, including rework smp_call_functiom_many() or remove the first TLB flush in x86, but there still have some arguments here and only parts of works have been pushed to mainline. SWAPIN: Page fault does iodepth=1 sync io, but it's a little waste if only issue a page size's IO. The obvious solution is doing swap readahead. But the current in-kernel swap readahead is arbitary(always 8 pages), and it always doesn't perform well for both random and sequential access workload. Shaohua introduced a new flag for madvise(MADV_WILLNEED) to do swap prefetch, so the changes happen in userspace API and leave the in-kernel readahead unchanged(but I think some improvement can also be done here). SWAP discard As we know, discard is important for SSD write throughout, but the current swap discard implementation is synchronous. He changed it to async discard which allow discard and write run in the same time. Meanwhile, the unit of discard is also optimized to cluster. Misc: lock contention For many concurrent swapout and swapin , the lock contention such as anon_vma or swap_lock is high, so he changed the swap_lock to a per-swap lock. But there still have some lock contention in very high speed SSD because of swapcache address_space lock. Zproject (led by Bob Liu) Bob gave us a very nice introduction about the current memory compression status. Now there are 3 projects(zswap/zram/zcache) which all aim at smooth swap IO storm and promote performance, but they all have their own pros and cons. ZSWAP It is implemented based on frontswap API and it uses a dynamic allocater named Zbud to allocate free pages. Zbud means pairs of zpages are "buddied" and it can only store at most two compressed pages in one page frame, so the max compress ratio is 50%. Each page frame is lru-linked and can do shink in memory pressure. If the compressed memory pool reach its limitation, shink or reclaim happens. It decompress the page frame into two new allocated pages and then write them to real swap device, but it can fail when allocating the two pages. ZRAM Acts as a compressed ramdisk and used as swap device, and it use zsmalloc as its allocator which has high density but may have fragmentation issues. Besides, page reclaim is hard since it will need more pages to uncompress and free just one page. ZRAM is preferred by embedded system which may not have any real swap device. Now both ZRAM and ZSWAP are in driver/staging tree, and in the mm community there are some disscussions of merging ZRAM into ZSWAP or viceversa, but no agreement yet. ZCACHE Handles file page compression but it is removed out of staging recently. From industry (led by Tang Jie, LSI) An LSI engineer introduced several new produces to us. The first is raid5/6 cards that it use full stripe writes to improve performance. The 2nd one he introduced is SandForce flash controller, who can understand data file types (data entropy) to reduce write amplification (WA) for nearly all writes. It's called DuraWrite and typical WA is 0.5. What's more, if enable its Dynamic Logical Capacity function module, the controller can do data compression which is transparent to upper layer. LSI testing shows that with this virtual capacity enables 1x TB drive can support up to 2x TB capacity, but the application must monitor free flash space to maintain optimal performance and to guard against free flash space exhaustion. He said the most useful application is for datebase. Another thing I think it's worth to mention is that a NV-DRAM memory in NMR/Raptor which is directly exposed to host system. Applications can directly access the NV-DRAM via a memory address - using standard system call mmap(). He said that it is very useful for database logging now. This kind of NVM produces are beginning to appear in recent years, and it is said that Samsung is building a research center in China for related produces. IMHO, NVM will bring an effect to current os layer especially on file system, e.g. its journaling may need to redesign to fully utilize these nonvolatile memory. OCFS2 (led by Canquan Shen) Without a doubt, HuaWei is the biggest contributor to OCFS2 in the past two years. They have posted 46 upstream patches and 39 patches have been merged. Their current project is based on 32/64 nodes cluster, but they also tried 128 nodes at the experimental stage. The major work they are working is to support ATS (atomic test and set), it can be works with DLM at the same time. Looks this idea is inspired by the vmware VMFS locking, i.e, http://blogs.vmware.com/vsphere/2012/05/vmfs-locking-uncovered.html CLK - 18th October 2013 Improving Linux Development with Better Tools (Andi Kleen) This talk focused on how to find/solve bugs along with the Linux complexity growing. Generally, we can do this with the following kind of tools: Static code checkers tools. e.g, sparse, smatch, coccinelle, clang checker, checkpatch, gcc -W/LTO, stanse. This can help check a lot of things, simple mistakes, complex problems, but the challenges are: some are very slow, false positives, may need a concentrated effort to get false positives down. Especially, no static checker I found can follow indirect calls (“OO in C”, common in kernel): struct foo_ops { int (*do_foo)(struct foo *obj); } foo->do_foo(foo); Dynamic runtime checkers, e.g, thread checkers, kmemcheck, lockdep. Ideally all kernel code would come with a test suite, then someone could run all the dynamic checkers. Fuzzers/test suites. e.g, Trinity is a great tool, it finds many bugs, but needs manual model for each syscall. Modern fuzzers around using automatic feedback, but notfor kernel yet: http://taviso.decsystem.org/making_software_dumber.pdf Debuggers/Tracers to understand code, e.g, ftrace, can dump on events/oops/custom triggers, but still too much overhead in many cases to run always during debug. Tools to read/understand source, e.g, grep/cscope work great for many cases, but do not understand indirect pointers (OO in C model used in kernel), give us all “do_foo” instances: struct foo_ops { int (*do_foo)(struct foo *obj); } = { .do_foo = my_foo }; foo>do_foo(foo); That would be great to have a cscope like tool that understands this based on types/initializers XFS: The High Performance Enterprise File System (Jeff Liu) [slides] I gave a talk for introducing the disk layout, unique features, as well as the recent changes.   The slides include some charts to reflect the performances between XFS/Btrfs/Ext4 for small files. About a dozen users raised their hands when I asking who has experienced with XFS. I remembered that when I asked the same question in LinuxCon/Japan, only 3 people raised their hands, but they are Chris Mason, Ric Wheeler, and another attendee. The attendee questions were mainly focused on stability, and comparison with other file systems. Linux Containers (Feng Gao) The speaker introduced us that the purpose for those kind of namespaces, include mount/UTS/IPC/Network/Pid/User, as well as the system API/ABI. For the userspace tools, He mainly focus on the Libvirt LXC rather than us(LXC). Libvirt LXC is another userspace container management tool, implemented as one type of libvirt driver, it can manage containers, create namespace, create private filesystem layout for container, Create devices for container and setup resources controller via cgroup. In this talk, Feng also mentioned another two possible new namespaces in the future, the 1st is the audit, but not sure if it should be assigned to user namespace or not. Another is about syslog, but the question is do we really need it? In-memory Compression (Bob Liu) Same as CLSF, a nice introduction that I have already mentioned above. Misc There were some other talks related to ACPI based memory hotplug, smart wake-affinity in scheduler etc., but my head is not big enough to record all those things. -- Jeff Liu

    Read the article

  • SQL Server &ndash; Undelete a Table and Restore a Single Table from Backup

    - by Mladen Prajdic
    This post is part of the monthly community event called T-SQL Tuesday started by Adam Machanic (blog|twitter) and hosted by someone else each month. This month the host is Sankar Reddy (blog|twitter) and the topic is Misconceptions in SQL Server. You can follow posts for this theme on Twitter by looking at #TSQL2sDay hashtag. Let me start by saying: This code is a crazy hack that is to never be used unless you really, really have to. Really! And I don’t think there’s a time when you would really have to use it for real. Because it’s a hack there are number of things that can go wrong so play with it knowing that. I’ve managed to totally corrupt one database. :) Oh… and for those saying: yeah yeah.. you have a single table in a file group and you’re restoring that, I say “nay nay” to you. As we all know SQL Server can’t do single table restores from backup. This is kind of a obvious thing due to different relational integrity (RI) concerns. Since we have to maintain that we have to restore all tables represented in a RI graph. For this exercise i say BAH! to those concerns. Note that this method “works” only for simple tables that don’t have LOB and off rows data. The code can be expanded to include those but I’ve tried to leave things “simple”. Note that for this to work our table needs to be relatively static data-wise. This doesn’t work for OLTP table. Products are a perfect example of static data. They don’t change much between backups, pretty much everything depends on them and their table is one of those tables that are relatively easy to accidentally delete everything from. This only works if the database is in Full or Bulk-Logged recovery mode for tables where the contents have been deleted or truncated but NOT when a table was dropped. Everything we’ll talk about has to be done before the data pages are reused for other purposes. After deletion or truncation the pages are marked as reusable so you have to act fast. The best thing probably is to put the database into single user mode ASAP while you’re performing this procedure and return it to multi user after you’re done. How do we do it? We will be using an undocumented but known DBCC commands: DBCC PAGE, an undocumented function sys.fn_dblog and a little known DATABASE RESTORE PAGE option. All tests will be on a copy of Production.Product table in AdventureWorks database called Production.Product1 because the original table has FK constraints that prevent us from truncating it for testing. -- create a duplicate table. This doesn't preserve indexes!SELECT *INTO AdventureWorks.Production.Product1FROM AdventureWorks.Production.Product   After we run this code take a full back to perform further testing.   First let’s see what the difference between DELETE and TRUNCATE is when it comes to logging. With DELETE every row deletion is logged in the transaction log. With TRUNCATE only whole data page deallocations are logged in the transaction log. Getting deleted data pages is simple. All we have to look for is row delete entry in the sys.fn_dblog output. But getting data pages that were truncated from the transaction log presents a bit of an interesting problem. I will not go into depths of IAM(Index Allocation Map) and PFS (Page Free Space) pages but suffice to say that every IAM page has intervals that tell us which data pages are allocated for a table and which aren’t. If we deep dive into the sys.fn_dblog output we can see that once you truncate a table all the pages in all the intervals are deallocated and this is shown in the PFS page transaction log entry as deallocation of pages. For every 8 pages in the same extent there is one PFS page row in the transaction log. This row holds information about all 8 pages in CSV format which means we can get to this data with some parsing. A great help for parsing this stuff is Peter Debetta’s handy function dbo.HexStrToVarBin that converts hexadecimal string into a varbinary value that can be easily converted to integer tus giving us a readable page number. The shortened (columns removed) sys.fn_dblog output for a PFS page with CSV data for 1 extent (8 data pages) looks like this: -- [Page ID] is displayed in hex format. -- To convert it to readable int we'll use dbo.HexStrToVarBin function found at -- http://sqlblog.com/blogs/peter_debetta/archive/2007/03/09/t-sql-convert-hex-string-to-varbinary.aspx -- This function must be installed in the master databaseSELECT Context, AllocUnitName, [Page ID], DescriptionFROM sys.fn_dblog(NULL, NULL)WHERE [Current LSN] = '00000031:00000a46:007d' The pages at the end marked with 0x00—> are pages that are allocated in the extent but are not part of a table. We can inspect the raw content of each data page with a DBCC PAGE command: -- we need this trace flag to redirect output to the query window.DBCC TRACEON (3604); -- WITH TABLERESULTS gives us data in table format instead of message format-- we use format option 3 because it's the easiest to read and manipulate further onDBCC PAGE (AdventureWorks, 1, 613, 3) WITH TABLERESULTS   Since the DBACC PAGE output can be quite extensive I won’t put it here. You can see an example of it in the link at the beginning of this section. Getting deleted data back When we run a delete statement every row to be deleted is marked as a ghost record. A background process periodically cleans up those rows. A huge misconception is that the data is actually removed. It’s not. Only the pointers to the rows are removed while the data itself is still on the data page. We just can’t access it with normal means. To get those pointers back we need to restore every deleted page using the RESTORE PAGE option mentioned above. This restore must be done from a full backup, followed by any differential and log backups that you may have. This is necessary to bring the pages up to the same point in time as the rest of the data.  However the restore doesn’t magically connect the restored page back to the original table. It simply replaces the current page with the one from the backup. After the restore we use the DBCC PAGE to read data directly from all data pages and insert that data into a temporary table. To finish the RESTORE PAGE  procedure we finally have to take a tail log backup (simple backup of the transaction log) and restore it back. We can now insert data from the temporary table to our original table by hand. Getting truncated data back When we run a truncate the truncated data pages aren’t touched at all. Even the pointers to rows stay unchanged. Because of this getting data back from truncated table is simple. we just have to find out which pages belonged to our table and use DBCC PAGE to read data off of them. No restore is necessary. Turns out that the problems we had with finding the data pages is alleviated by not having to do a RESTORE PAGE procedure. Stop stalling… show me The Code! This is the code for getting back deleted and truncated data back. It’s commented in all the right places so don’t be afraid to take a closer look. Make sure you have a full backup before trying this out. Also I suggest that the last step of backing and restoring the tail log is performed by hand. USE masterGOIF OBJECT_ID('dbo.HexStrToVarBin') IS NULL RAISERROR ('No dbo.HexStrToVarBin installed. Go to http://sqlblog.com/blogs/peter_debetta/archive/2007/03/09/t-sql-convert-hex-string-to-varbinary.aspx and install it in master database' , 18, 1) SET NOCOUNT ONBEGIN TRY DECLARE @dbName VARCHAR(1000), @schemaName VARCHAR(1000), @tableName VARCHAR(1000), @fullBackupName VARCHAR(1000), @undeletedTableName VARCHAR(1000), @sql VARCHAR(MAX), @tableWasTruncated bit; /* THE FIRST LINE ARE OUR INPUT PARAMETERS In this case we're trying to recover Production.Product1 table in AdventureWorks database. My full backup of AdventureWorks database is at e:\AW.bak */ SELECT @dbName = 'AdventureWorks', @schemaName = 'Production', @tableName = 'Product1', @fullBackupName = 'e:\AW.bak', @undeletedTableName = '##' + @tableName + '_Undeleted', @tableWasTruncated = 0, -- copy the structure from original table to a temp table that we'll fill with restored data @sql = 'IF OBJECT_ID(''tempdb..' + @undeletedTableName + ''') IS NOT NULL DROP TABLE ' + @undeletedTableName + ' SELECT *' + ' INTO ' + @undeletedTableName + ' FROM [' + @dbName + '].[' + @schemaName + '].[' + @tableName + ']' + ' WHERE 1 = 0' EXEC (@sql) IF OBJECT_ID('tempdb..#PagesToRestore') IS NOT NULL DROP TABLE #PagesToRestore /* FIND DATA PAGES WE NEED TO RESTORE*/ CREATE TABLE #PagesToRestore ([ID] INT IDENTITY(1,1), [FileID] INT, [PageID] INT, [SQLtoExec] VARCHAR(1000)) -- DBCC PACE statement to run later RAISERROR ('Looking for deleted pages...', 10, 1) -- use T-LOG direct read to get deleted data pages INSERT INTO #PagesToRestore([FileID], [PageID], [SQLtoExec]) EXEC('USE [' + @dbName + '];SELECT FileID, PageID, ''DBCC TRACEON (3604); DBCC PAGE ([' + @dbName + '], '' + FileID + '', '' + PageID + '', 3) WITH TABLERESULTS'' as SQLToExecFROM (SELECT DISTINCT LEFT([Page ID], 4) AS FileID, CONVERT(VARCHAR(100), ' + 'CONVERT(INT, master.dbo.HexStrToVarBin(SUBSTRING([Page ID], 6, 20)))) AS PageIDFROM sys.fn_dblog(NULL, NULL)WHERE AllocUnitName LIKE ''%' + @schemaName + '.' + @tableName + '%'' ' + 'AND Context IN (''LCX_MARK_AS_GHOST'', ''LCX_HEAP'') AND Operation in (''LOP_DELETE_ROWS''))t');SELECT *FROM #PagesToRestore -- if upper EXEC returns 0 rows it means the table was truncated so find truncated pages IF (SELECT COUNT(*) FROM #PagesToRestore) = 0 BEGIN RAISERROR ('No deleted pages found. Looking for truncated pages...', 10, 1) -- use T-LOG read to get truncated data pages INSERT INTO #PagesToRestore([FileID], [PageID], [SQLtoExec]) -- dark magic happens here -- because truncation simply deallocates pages we have to find out which pages were deallocated. -- we can find this out by looking at the PFS page row's Description column. -- for every deallocated extent the Description has a CSV of 8 pages in that extent. -- then it's just a matter of parsing it. -- we also remove the pages in the extent that weren't allocated to the table itself -- marked with '0x00-->00' EXEC ('USE [' + @dbName + '];DECLARE @truncatedPages TABLE(DeallocatedPages VARCHAR(8000), IsMultipleDeallocs BIT);INSERT INTO @truncatedPagesSELECT REPLACE(REPLACE(Description, ''Deallocated '', ''Y''), ''0x00-->00 '', ''N'') + '';'' AS DeallocatedPages, CHARINDEX('';'', Description) AS IsMultipleDeallocsFROM (SELECT DISTINCT LEFT([Page ID], 4) AS FileID, CONVERT(VARCHAR(100), CONVERT(INT, master.dbo.HexStrToVarBin(SUBSTRING([Page ID], 6, 20)))) AS PageID, DescriptionFROM sys.fn_dblog(NULL, NULL)WHERE Context IN (''LCX_PFS'') AND Description LIKE ''Deallocated%'' AND AllocUnitName LIKE ''%' + @schemaName + '.' + @tableName + '%'') t;SELECT FileID, PageID , ''DBCC TRACEON (3604); DBCC PAGE ([' + @dbName + '], '' + FileID + '', '' + PageID + '', 3) WITH TABLERESULTS'' as SQLToExecFROM (SELECT LEFT(PageAndFile, 1) as WasPageAllocatedToTable , SUBSTRING(PageAndFile, 2, CHARINDEX('':'', PageAndFile) - 2 ) as FileID , CONVERT(VARCHAR(100), CONVERT(INT, master.dbo.HexStrToVarBin(SUBSTRING(PageAndFile, CHARINDEX('':'', PageAndFile) + 1, LEN(PageAndFile))))) as PageIDFROM ( SELECT SUBSTRING(DeallocatedPages, delimPosStart, delimPosEnd - delimPosStart) as PageAndFile, IsMultipleDeallocs FROM ( SELECT *, CHARINDEX('';'', DeallocatedPages)*(N-1) + 1 AS delimPosStart, CHARINDEX('';'', DeallocatedPages)*N AS delimPosEnd FROM @truncatedPages t1 CROSS APPLY (SELECT TOP (case when t1.IsMultipleDeallocs = 1 then 8 else 1 end) ROW_NUMBER() OVER(ORDER BY number) as N FROM master..spt_values) t2 )t)t)tWHERE WasPageAllocatedToTable = ''Y''') SELECT @tableWasTruncated = 1 END DECLARE @lastID INT, @pagesCount INT SELECT @lastID = 1, @pagesCount = COUNT(*) FROM #PagesToRestore SELECT @sql = 'Number of pages to restore: ' + CONVERT(VARCHAR(10), @pagesCount) IF @pagesCount = 0 RAISERROR ('No data pages to restore.', 18, 1) ELSE RAISERROR (@sql, 10, 1) -- If the table was truncated we'll read the data directly from data pages without restoring from backup IF @tableWasTruncated = 0 BEGIN -- RESTORE DATA PAGES FROM FULL BACKUP IN BATCHES OF 200 WHILE @lastID <= @pagesCount BEGIN -- create CSV string of pages to restore SELECT @sql = STUFF((SELECT ',' + CONVERT(VARCHAR(100), FileID) + ':' + CONVERT(VARCHAR(100), PageID) FROM #PagesToRestore WHERE ID BETWEEN @lastID AND @lastID + 200 ORDER BY ID FOR XML PATH('')), 1, 1, '') SELECT @sql = 'RESTORE DATABASE [' + @dbName + '] PAGE = ''' + @sql + ''' FROM DISK = ''' + @fullBackupName + '''' RAISERROR ('Starting RESTORE command:' , 10, 1) WITH NOWAIT; RAISERROR (@sql , 10, 1) WITH NOWAIT; EXEC(@sql); RAISERROR ('Restore DONE' , 10, 1) WITH NOWAIT; SELECT @lastID = @lastID + 200 END /* If you have any differential or transaction log backups you should restore them here to bring the previously restored data pages up to date */ END DECLARE @dbccSinglePage TABLE ( [ParentObject] NVARCHAR(500), [Object] NVARCHAR(500), [Field] NVARCHAR(500), [VALUE] NVARCHAR(MAX) ) DECLARE @cols NVARCHAR(MAX), @paramDefinition NVARCHAR(500), @SQLtoExec VARCHAR(1000), @FileID VARCHAR(100), @PageID VARCHAR(100), @i INT = 1 -- Get deleted table columns from information_schema view -- Need sp_executeSQL because database name can't be passed in as variable SELECT @cols = 'select @cols = STUFF((SELECT '', ['' + COLUMN_NAME + '']''FROM ' + @dbName + '.INFORMATION_SCHEMA.COLUMNSWHERE TABLE_NAME = ''' + @tableName + ''' AND TABLE_SCHEMA = ''' + @schemaName + '''ORDER BY ORDINAL_POSITIONFOR XML PATH('''')), 1, 2, '''')', @paramDefinition = N'@cols nvarchar(max) OUTPUT' EXECUTE sp_executesql @cols, @paramDefinition, @cols = @cols OUTPUT -- Loop through all the restored data pages, -- read data from them and insert them into temp table -- which you can then insert into the orignial deleted table DECLARE dbccPageCursor CURSOR GLOBAL FORWARD_ONLY FOR SELECT [FileID], [PageID], [SQLtoExec] FROM #PagesToRestore ORDER BY [FileID], [PageID] OPEN dbccPageCursor; FETCH NEXT FROM dbccPageCursor INTO @FileID, @PageID, @SQLtoExec; WHILE @@FETCH_STATUS = 0 BEGIN RAISERROR ('---------------------------------------------', 10, 1) WITH NOWAIT; SELECT @sql = 'Loop iteration: ' + CONVERT(VARCHAR(10), @i); RAISERROR (@sql, 10, 1) WITH NOWAIT; SELECT @sql = 'Running: ' + @SQLtoExec RAISERROR (@sql, 10, 1) WITH NOWAIT; -- if something goes wrong with DBCC execution or data gathering, skip it but print error BEGIN TRY INSERT INTO @dbccSinglePage EXEC (@SQLtoExec) -- make the data insert magic happen here IF (SELECT CONVERT(BIGINT, [VALUE]) FROM @dbccSinglePage WHERE [Field] LIKE '%Metadata: ObjectId%') = OBJECT_ID('['+@dbName+'].['+@schemaName +'].['+@tableName+']') BEGIN DELETE @dbccSinglePage WHERE NOT ([ParentObject] LIKE 'Slot % Offset %' AND [Object] LIKE 'Slot % Column %') SELECT @sql = 'USE tempdb; ' + 'IF (OBJECTPROPERTY(object_id(''' + @undeletedTableName + '''), ''TableHasIdentity'') = 1) ' + 'SET IDENTITY_INSERT ' + @undeletedTableName + ' ON; ' + 'INSERT INTO ' + @undeletedTableName + '(' + @cols + ') ' + STUFF((SELECT ' UNION ALL SELECT ' + STUFF((SELECT ', ' + CASE WHEN VALUE = '[NULL]' THEN 'NULL' ELSE '''' + [VALUE] + '''' END FROM ( -- the unicorn help here to correctly set ordinal numbers of columns in a data page -- it's turning STRING order into INT order (1,10,11,2,21 into 1,2,..10,11...21) SELECT [ParentObject], [Object], Field, VALUE, RIGHT('00000' + O1, 6) AS ParentObjectOrder, RIGHT('00000' + REVERSE(LEFT(O2, CHARINDEX(' ', O2)-1)), 6) AS ObjectOrder FROM ( SELECT [ParentObject], [Object], Field, VALUE, REPLACE(LEFT([ParentObject], CHARINDEX('Offset', [ParentObject])-1), 'Slot ', '') AS O1, REVERSE(LEFT([Object], CHARINDEX('Offset ', [Object])-2)) AS O2 FROM @dbccSinglePage WHERE t.ParentObject = ParentObject )t)t ORDER BY ParentObjectOrder, ObjectOrder FOR XML PATH('')), 1, 2, '') FROM @dbccSinglePage t GROUP BY ParentObject FOR XML PATH('') ), 1, 11, '') + ';' RAISERROR (@sql, 10, 1) WITH NOWAIT; EXEC (@sql) END END TRY BEGIN CATCH SELECT @sql = 'ERROR!!!' + CHAR(10) + CHAR(13) + 'ErrorNumber: ' + ERROR_NUMBER() + '; ErrorMessage' + ERROR_MESSAGE() + CHAR(10) + CHAR(13) + 'FileID: ' + @FileID + '; PageID: ' + @PageID RAISERROR (@sql, 10, 1) WITH NOWAIT; END CATCH DELETE @dbccSinglePage SELECT @sql = 'Pages left to process: ' + CONVERT(VARCHAR(10), @pagesCount - @i) + CHAR(10) + CHAR(13) + CHAR(10) + CHAR(13) + CHAR(10) + CHAR(13), @i = @i+1 RAISERROR (@sql, 10, 1) WITH NOWAIT; FETCH NEXT FROM dbccPageCursor INTO @FileID, @PageID, @SQLtoExec; END CLOSE dbccPageCursor; DEALLOCATE dbccPageCursor; EXEC ('SELECT ''' + @undeletedTableName + ''' as TableName; SELECT * FROM ' + @undeletedTableName)END TRYBEGIN CATCH SELECT ERROR_NUMBER() AS ErrorNumber, ERROR_MESSAGE() AS ErrorMessage IF CURSOR_STATUS ('global', 'dbccPageCursor') >= 0 BEGIN CLOSE dbccPageCursor; DEALLOCATE dbccPageCursor; ENDEND CATCH-- if the table was deleted we need to finish the restore page sequenceIF @tableWasTruncated = 0BEGIN -- take a log tail backup and then restore it to complete page restore process DECLARE @currentDate VARCHAR(30) SELECT @currentDate = CONVERT(VARCHAR(30), GETDATE(), 112) RAISERROR ('Starting Log Tail backup to c:\Temp ...', 10, 1) WITH NOWAIT; PRINT ('BACKUP LOG [' + @dbName + '] TO DISK = ''c:\Temp\' + @dbName + '_TailLogBackup_' + @currentDate + '.trn''') EXEC ('BACKUP LOG [' + @dbName + '] TO DISK = ''c:\Temp\' + @dbName + '_TailLogBackup_' + @currentDate + '.trn''') RAISERROR ('Log Tail backup done.', 10, 1) WITH NOWAIT; RAISERROR ('Starting Log Tail restore from c:\Temp ...', 10, 1) WITH NOWAIT; PRINT ('RESTORE LOG [' + @dbName + '] FROM DISK = ''c:\Temp\' + @dbName + '_TailLogBackup_' + @currentDate + '.trn''') EXEC ('RESTORE LOG [' + @dbName + '] FROM DISK = ''c:\Temp\' + @dbName + '_TailLogBackup_' + @currentDate + '.trn''') RAISERROR ('Log Tail restore done.', 10, 1) WITH NOWAIT;END-- The last step is manual. Insert data from our temporary table to the original deleted table The misconception here is that you can do a single table restore properly in SQL Server. You can't. But with little experimentation you can get pretty close to it. One way to possible remove a dependency on a backup to retrieve deleted pages is to quickly run a similar script to the upper one that gets data directly from data pages while the rows are still marked as ghost records. It could be done if we could beat the ghost record cleanup task.

    Read the article

  • How John Got 15x Improvement Without Really Trying

    - by rchrd
    The following article was published on a Sun Microsystems website a number of years ago by John Feo. It is still useful and worth preserving. So I'm republishing it here.  How I Got 15x Improvement Without Really Trying John Feo, Sun Microsystems Taking ten "personal" program codes used in scientific and engineering research, the author was able to get from 2 to 15 times performance improvement easily by applying some simple general optimization techniques. Introduction Scientific research based on computer simulation depends on the simulation for advancement. The research can advance only as fast as the computational codes can execute. The codes' efficiency determines both the rate and quality of results. In the same amount of time, a faster program can generate more results and can carry out a more detailed simulation of physical phenomena than a slower program. Highly optimized programs help science advance quickly and insure that monies supporting scientific research are used as effectively as possible. Scientific computer codes divide into three broad categories: ISV, community, and personal. ISV codes are large, mature production codes developed and sold commercially. The codes improve slowly over time both in methods and capabilities, and they are well tuned for most vendor platforms. Since the codes are mature and complex, there are few opportunities to improve their performance solely through code optimization. Improvements of 10% to 15% are typical. Examples of ISV codes are DYNA3D, Gaussian, and Nastran. Community codes are non-commercial production codes used by a particular research field. Generally, they are developed and distributed by a single academic or research institution with assistance from the community. Most users just run the codes, but some develop new methods and extensions that feed back into the general release. The codes are available on most vendor platforms. Since these codes are younger than ISV codes, there are more opportunities to optimize the source code. Improvements of 50% are not unusual. Examples of community codes are AMBER, CHARM, BLAST, and FASTA. Personal codes are those written by single users or small research groups for their own use. These codes are not distributed, but may be passed from professor-to-student or student-to-student over several years. They form the primordial ocean of applications from which community and ISV codes emerge. Government research grants pay for the development of most personal codes. This paper reports on the nature and performance of this class of codes. Over the last year, I have looked at over two dozen personal codes from more than a dozen research institutions. The codes cover a variety of scientific fields, including astronomy, atmospheric sciences, bioinformatics, biology, chemistry, geology, and physics. The sources range from a few hundred lines to more than ten thousand lines, and are written in Fortran, Fortran 90, C, and C++. For the most part, the codes are modular, documented, and written in a clear, straightforward manner. They do not use complex language features, advanced data structures, programming tricks, or libraries. I had little trouble understanding what the codes did or how data structures were used. Most came with a makefile. Surprisingly, only one of the applications is parallel. All developers have access to parallel machines, so availability is not an issue. Several tried to parallelize their applications, but stopped after encountering difficulties. Lack of education and a perception that parallelism is difficult prevented most from trying. I parallelized several of the codes using OpenMP, and did not judge any of the codes as difficult to parallelize. Even more surprising than the lack of parallelism is the inefficiency of the codes. I was able to get large improvements in performance in a matter of a few days applying simple optimization techniques. Table 1 lists ten representative codes [names and affiliation are omitted to preserve anonymity]. Improvements on one processor range from 2x to 15.5x with a simple average of 4.75x. I did not use sophisticated performance tools or drill deep into the program's execution character as one would do when tuning ISV or community codes. Using only a profiler and source line timers, I identified inefficient sections of code and improved their performance by inspection. The changes were at a high level. I am sure there is another factor of 2 or 3 in each code, and more if the codes are parallelized. The study’s results show that personal scientific codes are running many times slower than they should and that the problem is pervasive. Computational scientists are not sloppy programmers; however, few are trained in the art of computer programming or code optimization. I found that most have a working knowledge of some programming language and standard software engineering practices; but they do not know, or think about, how to make their programs run faster. They simply do not know the standard techniques used to make codes run faster. In fact, they do not even perceive that such techniques exist. The case studies described in this paper show that applying simple, well known techniques can significantly increase the performance of personal codes. It is important that the scientific community and the Government agencies that support scientific research find ways to better educate academic scientific programmers. The inefficiency of their codes is so bad that it is retarding both the quality and progress of scientific research. # cacheperformance redundantoperations loopstructures performanceimprovement 1 x x 15.5 2 x 2.8 3 x x 2.5 4 x 2.1 5 x x 2.0 6 x 5.0 7 x 5.8 8 x 6.3 9 2.2 10 x x 3.3 Table 1 — Area of improvement and performance gains of 10 codes The remainder of the paper is organized as follows: sections 2, 3, and 4 discuss the three most common sources of inefficiencies in the codes studied. These are cache performance, redundant operations, and loop structures. Each section includes several examples. The last section summaries the work and suggests a possible solution to the issues raised. Optimizing cache performance Commodity microprocessor systems use caches to increase memory bandwidth and reduce memory latencies. Typical latencies from processor to L1, L2, local, and remote memory are 3, 10, 50, and 200 cycles, respectively. Moreover, bandwidth falls off dramatically as memory distances increase. Programs that do not use cache effectively run many times slower than programs that do. When optimizing for cache, the biggest performance gains are achieved by accessing data in cache order and reusing data to amortize the overhead of cache misses. Secondary considerations are prefetching, associativity, and replacement; however, the understanding and analysis required to optimize for the latter are probably beyond the capabilities of the non-expert. Much can be gained simply by accessing data in the correct order and maximizing data reuse. 6 out of the 10 codes studied here benefited from such high level optimizations. Array Accesses The most important cache optimization is the most basic: accessing Fortran array elements in column order and C array elements in row order. Four of the ten codes—1, 2, 4, and 10—got it wrong. Compilers will restructure nested loops to optimize cache performance, but may not do so if the loop structure is too complex, or the loop body includes conditionals, complex addressing, or function calls. In code 1, the compiler failed to invert a key loop because of complex addressing do I = 0, 1010, delta_x IM = I - delta_x IP = I + delta_x do J = 5, 995, delta_x JM = J - delta_x JP = J + delta_x T1 = CA1(IP, J) + CA1(I, JP) T2 = CA1(IM, J) + CA1(I, JM) S1 = T1 + T2 - 4 * CA1(I, J) CA(I, J) = CA1(I, J) + D * S1 end do end do In code 2, the culprit is conditionals do I = 1, N do J = 1, N If (IFLAG(I,J) .EQ. 0) then T1 = Value(I, J-1) T2 = Value(I-1, J) T3 = Value(I, J) T4 = Value(I+1, J) T5 = Value(I, J+1) Value(I,J) = 0.25 * (T1 + T2 + T5 + T4) Delta = ABS(T3 - Value(I,J)) If (Delta .GT. MaxDelta) MaxDelta = Delta endif enddo enddo I fixed both programs by inverting the loops by hand. Code 10 has three-dimensional arrays and triply nested loops. The structure of the most computationally intensive loops is too complex to invert automatically or by hand. The only practical solution is to transpose the arrays so that the dimension accessed by the innermost loop is in cache order. The arrays can be transposed at construction or prior to entering a computationally intensive section of code. The former requires all array references to be modified, while the latter is cost effective only if the cost of the transpose is amortized over many accesses. I used the second approach to optimize code 10. Code 5 has four-dimensional arrays and loops are nested four deep. For all of the reasons cited above the compiler is not able to restructure three key loops. Assume C arrays and let the four dimensions of the arrays be i, j, k, and l. In the original code, the index structure of the three loops is L1: for i L2: for i L3: for i for l for l for j for k for j for k for j for k for l So only L3 accesses array elements in cache order. L1 is a very complex loop—much too complex to invert. I brought the loop into cache alignment by transposing the second and fourth dimensions of the arrays. Since the code uses a macro to compute all array indexes, I effected the transpose at construction and changed the macro appropriately. The dimensions of the new arrays are now: i, l, k, and j. L3 is a simple loop and easily inverted. L2 has a loop-carried scalar dependence in k. By promoting the scalar name that carries the dependence to an array, I was able to invert the third and fourth subloops aligning the loop with cache. Code 5 is by far the most difficult of the four codes to optimize for array accesses; but the knowledge required to fix the problems is no more than that required for the other codes. I would judge this code at the limits of, but not beyond, the capabilities of appropriately trained computational scientists. Array Strides When a cache miss occurs, a line (64 bytes) rather than just one word is loaded into the cache. If data is accessed stride 1, than the cost of the miss is amortized over 8 words. Any stride other than one reduces the cost savings. Two of the ten codes studied suffered from non-unit strides. The codes represent two important classes of "strided" codes. Code 1 employs a multi-grid algorithm to reduce time to convergence. The grids are every tenth, fifth, second, and unit element. Since time to convergence is inversely proportional to the distance between elements, coarse grids converge quickly providing good starting values for finer grids. The better starting values further reduce the time to convergence. The downside is that grids of every nth element, n > 1, introduce non-unit strides into the computation. In the original code, much of the savings of the multi-grid algorithm were lost due to this problem. I eliminated the problem by compressing (copying) coarse grids into continuous memory, and rewriting the computation as a function of the compressed grid. On convergence, I copied the final values of the compressed grid back to the original grid. The savings gained from unit stride access of the compressed grid more than paid for the cost of copying. Using compressed grids, the loop from code 1 included in the previous section becomes do j = 1, GZ do i = 1, GZ T1 = CA(i+0, j-1) + CA(i-1, j+0) T4 = CA1(i+1, j+0) + CA1(i+0, j+1) S1 = T1 + T4 - 4 * CA1(i+0, j+0) CA(i+0, j+0) = CA1(i+0, j+0) + DD * S1 enddo enddo where CA and CA1 are compressed arrays of size GZ. Code 7 traverses a list of objects selecting objects for later processing. The labels of the selected objects are stored in an array. The selection step has unit stride, but the processing steps have irregular stride. A fix is to save the parameters of the selected objects in temporary arrays as they are selected, and pass the temporary arrays to the processing functions. The fix is practical if the same parameters are used in selection as in processing, or if processing comprises a series of distinct steps which use overlapping subsets of the parameters. Both conditions are true for code 7, so I achieved significant improvement by copying parameters to temporary arrays during selection. Data reuse In the previous sections, we optimized for spatial locality. It is also important to optimize for temporal locality. Once read, a datum should be used as much as possible before it is forced from cache. Loop fusion and loop unrolling are two techniques that increase temporal locality. Unfortunately, both techniques increase register pressure—as loop bodies become larger, the number of registers required to hold temporary values grows. Once register spilling occurs, any gains evaporate quickly. For multiprocessors with small register sets or small caches, the sweet spot can be very small. In the ten codes presented here, I found no opportunities for loop fusion and only two opportunities for loop unrolling (codes 1 and 3). In code 1, unrolling the outer and inner loop one iteration increases the number of result values computed by the loop body from 1 to 4, do J = 1, GZ-2, 2 do I = 1, GZ-2, 2 T1 = CA1(i+0, j-1) + CA1(i-1, j+0) T2 = CA1(i+1, j-1) + CA1(i+0, j+0) T3 = CA1(i+0, j+0) + CA1(i-1, j+1) T4 = CA1(i+1, j+0) + CA1(i+0, j+1) T5 = CA1(i+2, j+0) + CA1(i+1, j+1) T6 = CA1(i+1, j+1) + CA1(i+0, j+2) T7 = CA1(i+2, j+1) + CA1(i+1, j+2) S1 = T1 + T4 - 4 * CA1(i+0, j+0) S2 = T2 + T5 - 4 * CA1(i+1, j+0) S3 = T3 + T6 - 4 * CA1(i+0, j+1) S4 = T4 + T7 - 4 * CA1(i+1, j+1) CA(i+0, j+0) = CA1(i+0, j+0) + DD * S1 CA(i+1, j+0) = CA1(i+1, j+0) + DD * S2 CA(i+0, j+1) = CA1(i+0, j+1) + DD * S3 CA(i+1, j+1) = CA1(i+1, j+1) + DD * S4 enddo enddo The loop body executes 12 reads, whereas as the rolled loop shown in the previous section executes 20 reads to compute the same four values. In code 3, two loops are unrolled 8 times and one loop is unrolled 4 times. Here is the before for (k = 0; k < NK[u]; k++) { sum = 0.0; for (y = 0; y < NY; y++) { sum += W[y][u][k] * delta[y]; } backprop[i++]=sum; } and after code for (k = 0; k < KK - 8; k+=8) { sum0 = 0.0; sum1 = 0.0; sum2 = 0.0; sum3 = 0.0; sum4 = 0.0; sum5 = 0.0; sum6 = 0.0; sum7 = 0.0; for (y = 0; y < NY; y++) { sum0 += W[y][0][k+0] * delta[y]; sum1 += W[y][0][k+1] * delta[y]; sum2 += W[y][0][k+2] * delta[y]; sum3 += W[y][0][k+3] * delta[y]; sum4 += W[y][0][k+4] * delta[y]; sum5 += W[y][0][k+5] * delta[y]; sum6 += W[y][0][k+6] * delta[y]; sum7 += W[y][0][k+7] * delta[y]; } backprop[k+0] = sum0; backprop[k+1] = sum1; backprop[k+2] = sum2; backprop[k+3] = sum3; backprop[k+4] = sum4; backprop[k+5] = sum5; backprop[k+6] = sum6; backprop[k+7] = sum7; } for one of the loops unrolled 8 times. Optimizing for temporal locality is the most difficult optimization considered in this paper. The concepts are not difficult, but the sweet spot is small. Identifying where the program can benefit from loop unrolling or loop fusion is not trivial. Moreover, it takes some effort to get it right. Still, educating scientific programmers about temporal locality and teaching them how to optimize for it will pay dividends. Reducing instruction count Execution time is a function of instruction count. Reduce the count and you usually reduce the time. The best solution is to use a more efficient algorithm; that is, an algorithm whose order of complexity is smaller, that converges quicker, or is more accurate. Optimizing source code without changing the algorithm yields smaller, but still significant, gains. This paper considers only the latter because the intent is to study how much better codes can run if written by programmers schooled in basic code optimization techniques. The ten codes studied benefited from three types of "instruction reducing" optimizations. The two most prevalent were hoisting invariant memory and data operations out of inner loops. The third was eliminating unnecessary data copying. The nature of these inefficiencies is language dependent. Memory operations The semantics of C make it difficult for the compiler to determine all the invariant memory operations in a loop. The problem is particularly acute for loops in functions since the compiler may not know the values of the function's parameters at every call site when compiling the function. Most compilers support pragmas to help resolve ambiguities; however, these pragmas are not comprehensive and there is no standard syntax. To guarantee that invariant memory operations are not executed repetitively, the user has little choice but to hoist the operations by hand. The problem is not as severe in Fortran programs because in the absence of equivalence statements, it is a violation of the language's semantics for two names to share memory. Codes 3 and 5 are C programs. In both cases, the compiler did not hoist all invariant memory operations from inner loops. Consider the following loop from code 3 for (y = 0; y < NY; y++) { i = 0; for (u = 0; u < NU; u++) { for (k = 0; k < NK[u]; k++) { dW[y][u][k] += delta[y] * I1[i++]; } } } Since dW[y][u] can point to the same memory space as delta for one or more values of y and u, assignment to dW[y][u][k] may change the value of delta[y]. In reality, dW and delta do not overlap in memory, so I rewrote the loop as for (y = 0; y < NY; y++) { i = 0; Dy = delta[y]; for (u = 0; u < NU; u++) { for (k = 0; k < NK[u]; k++) { dW[y][u][k] += Dy * I1[i++]; } } } Failure to hoist invariant memory operations may be due to complex address calculations. If the compiler can not determine that the address calculation is invariant, then it can hoist neither the calculation nor the associated memory operations. As noted above, code 5 uses a macro to address four-dimensional arrays #define MAT4D(a,q,i,j,k) (double *)((a)->data + (q)*(a)->strides[0] + (i)*(a)->strides[3] + (j)*(a)->strides[2] + (k)*(a)->strides[1]) The macro is too complex for the compiler to understand and so, it does not identify any subexpressions as loop invariant. The simplest way to eliminate the address calculation from the innermost loop (over i) is to define a0 = MAT4D(a,q,0,j,k) before the loop and then replace all instances of *MAT4D(a,q,i,j,k) in the loop with a0[i] A similar problem appears in code 6, a Fortran program. The key loop in this program is do n1 = 1, nh nx1 = (n1 - 1) / nz + 1 nz1 = n1 - nz * (nx1 - 1) do n2 = 1, nh nx2 = (n2 - 1) / nz + 1 nz2 = n2 - nz * (nx2 - 1) ndx = nx2 - nx1 ndy = nz2 - nz1 gxx = grn(1,ndx,ndy) gyy = grn(2,ndx,ndy) gxy = grn(3,ndx,ndy) balance(n1,1) = balance(n1,1) + (force(n2,1) * gxx + force(n2,2) * gxy) * h1 balance(n1,2) = balance(n1,2) + (force(n2,1) * gxy + force(n2,2) * gyy)*h1 end do end do The programmer has written this loop well—there are no loop invariant operations with respect to n1 and n2. However, the loop resides within an iterative loop over time and the index calculations are independent with respect to time. Trading space for time, I precomputed the index values prior to the entering the time loop and stored the values in two arrays. I then replaced the index calculations with reads of the arrays. Data operations Ways to reduce data operations can appear in many forms. Implementing a more efficient algorithm produces the biggest gains. The closest I came to an algorithm change was in code 4. This code computes the inner product of K-vectors A(i) and B(j), 0 = i < N, 0 = j < M, for most values of i and j. Since the program computes most of the NM possible inner products, it is more efficient to compute all the inner products in one triply-nested loop rather than one at a time when needed. The savings accrue from reading A(i) once for all B(j) vectors and from loop unrolling. for (i = 0; i < N; i+=8) { for (j = 0; j < M; j++) { sum0 = 0.0; sum1 = 0.0; sum2 = 0.0; sum3 = 0.0; sum4 = 0.0; sum5 = 0.0; sum6 = 0.0; sum7 = 0.0; for (k = 0; k < K; k++) { sum0 += A[i+0][k] * B[j][k]; sum1 += A[i+1][k] * B[j][k]; sum2 += A[i+2][k] * B[j][k]; sum3 += A[i+3][k] * B[j][k]; sum4 += A[i+4][k] * B[j][k]; sum5 += A[i+5][k] * B[j][k]; sum6 += A[i+6][k] * B[j][k]; sum7 += A[i+7][k] * B[j][k]; } C[i+0][j] = sum0; C[i+1][j] = sum1; C[i+2][j] = sum2; C[i+3][j] = sum3; C[i+4][j] = sum4; C[i+5][j] = sum5; C[i+6][j] = sum6; C[i+7][j] = sum7; }} This change requires knowledge of a typical run; i.e., that most inner products are computed. The reasons for the change, however, derive from basic optimization concepts. It is the type of change easily made at development time by a knowledgeable programmer. In code 5, we have the data version of the index optimization in code 6. Here a very expensive computation is a function of the loop indices and so cannot be hoisted out of the loop; however, the computation is invariant with respect to an outer iterative loop over time. We can compute its value for each iteration of the computation loop prior to entering the time loop and save the values in an array. The increase in memory required to store the values is small in comparison to the large savings in time. The main loop in Code 8 is doubly nested. The inner loop includes a series of guarded computations; some are a function of the inner loop index but not the outer loop index while others are a function of the outer loop index but not the inner loop index for (j = 0; j < N; j++) { for (i = 0; i < M; i++) { r = i * hrmax; R = A[j]; temp = (PRM[3] == 0.0) ? 1.0 : pow(r, PRM[3]); high = temp * kcoeff * B[j] * PRM[2] * PRM[4]; low = high * PRM[6] * PRM[6] / (1.0 + pow(PRM[4] * PRM[6], 2.0)); kap = (R > PRM[6]) ? high * R * R / (1.0 + pow(PRM[4]*r, 2.0) : low * pow(R/PRM[6], PRM[5]); < rest of loop omitted > }} Note that the value of temp is invariant to j. Thus, we can hoist the computation for temp out of the loop and save its values in an array. for (i = 0; i < M; i++) { r = i * hrmax; TEMP[i] = pow(r, PRM[3]); } [N.B. – the case for PRM[3] = 0 is omitted and will be reintroduced later.] We now hoist out of the inner loop the computations invariant to i. Since the conditional guarding the value of kap is invariant to i, it behooves us to hoist the computation out of the inner loop, thereby executing the guard once rather than M times. The final version of the code is for (j = 0; j < N; j++) { R = rig[j] / 1000.; tmp1 = kcoeff * par[2] * beta[j] * par[4]; tmp2 = 1.0 + (par[4] * par[4] * par[6] * par[6]); tmp3 = 1.0 + (par[4] * par[4] * R * R); tmp4 = par[6] * par[6] / tmp2; tmp5 = R * R / tmp3; tmp6 = pow(R / par[6], par[5]); if ((par[3] == 0.0) && (R > par[6])) { for (i = 1; i <= imax1; i++) KAP[i] = tmp1 * tmp5; } else if ((par[3] == 0.0) && (R <= par[6])) { for (i = 1; i <= imax1; i++) KAP[i] = tmp1 * tmp4 * tmp6; } else if ((par[3] != 0.0) && (R > par[6])) { for (i = 1; i <= imax1; i++) KAP[i] = tmp1 * TEMP[i] * tmp5; } else if ((par[3] != 0.0) && (R <= par[6])) { for (i = 1; i <= imax1; i++) KAP[i] = tmp1 * TEMP[i] * tmp4 * tmp6; } for (i = 0; i < M; i++) { kap = KAP[i]; r = i * hrmax; < rest of loop omitted > } } Maybe not the prettiest piece of code, but certainly much more efficient than the original loop, Copy operations Several programs unnecessarily copy data from one data structure to another. This problem occurs in both Fortran and C programs, although it manifests itself differently in the two languages. Code 1 declares two arrays—one for old values and one for new values. At the end of each iteration, the array of new values is copied to the array of old values to reset the data structures for the next iteration. This problem occurs in Fortran programs not included in this study and in both Fortran 77 and Fortran 90 code. Introducing pointers to the arrays and swapping pointer values is an obvious way to eliminate the copying; but pointers is not a feature that many Fortran programmers know well or are comfortable using. An easy solution not involving pointers is to extend the dimension of the value array by 1 and use the last dimension to differentiate between arrays at different times. For example, if the data space is N x N, declare the array (N, N, 2). Then store the problem’s initial values in (_, _, 2) and define the scalar names new = 2 and old = 1. At the start of each iteration, swap old and new to reset the arrays. The old–new copy problem did not appear in any C program. In programs that had new and old values, the code swapped pointers to reset data structures. Where unnecessary coping did occur is in structure assignment and parameter passing. Structures in C are handled much like scalars. Assignment causes the data space of the right-hand name to be copied to the data space of the left-hand name. Similarly, when a structure is passed to a function, the data space of the actual parameter is copied to the data space of the formal parameter. If the structure is large and the assignment or function call is in an inner loop, then copying costs can grow quite large. While none of the ten programs considered here manifested this problem, it did occur in programs not included in the study. A simple fix is always to refer to structures via pointers. Optimizing loop structures Since scientific programs spend almost all their time in loops, efficient loops are the key to good performance. Conditionals, function calls, little instruction level parallelism, and large numbers of temporary values make it difficult for the compiler to generate tightly packed, highly efficient code. Conditionals and function calls introduce jumps that disrupt code flow. Users should eliminate or isolate conditionls to their own loops as much as possible. Often logical expressions can be substituted for if-then-else statements. For example, code 2 includes the following snippet MaxDelta = 0.0 do J = 1, N do I = 1, M < code omitted > Delta = abs(OldValue ? NewValue) if (Delta > MaxDelta) MaxDelta = Delta enddo enddo if (MaxDelta .gt. 0.001) goto 200 Since the only use of MaxDelta is to control the jump to 200 and all that matters is whether or not it is greater than 0.001, I made MaxDelta a boolean and rewrote the snippet as MaxDelta = .false. do J = 1, N do I = 1, M < code omitted > Delta = abs(OldValue ? NewValue) MaxDelta = MaxDelta .or. (Delta .gt. 0.001) enddo enddo if (MaxDelta) goto 200 thereby, eliminating the conditional expression from the inner loop. A microprocessor can execute many instructions per instruction cycle. Typically, it can execute one or more memory, floating point, integer, and jump operations. To be executed simultaneously, the operations must be independent. Thick loops tend to have more instruction level parallelism than thin loops. Moreover, they reduce memory traffice by maximizing data reuse. Loop unrolling and loop fusion are two techniques to increase the size of loop bodies. Several of the codes studied benefitted from loop unrolling, but none benefitted from loop fusion. This observation is not too surpising since it is the general tendency of programmers to write thick loops. As loops become thicker, the number of temporary values grows, increasing register pressure. If registers spill, then memory traffic increases and code flow is disrupted. A thick loop with many temporary values may execute slower than an equivalent series of thin loops. The biggest gain will be achieved if the thick loop can be split into a series of independent loops eliminating the need to write and read temporary arrays. I found such an occasion in code 10 where I split the loop do i = 1, n do j = 1, m A24(j,i)= S24(j,i) * T24(j,i) + S25(j,i) * U25(j,i) B24(j,i)= S24(j,i) * T25(j,i) + S25(j,i) * U24(j,i) A25(j,i)= S24(j,i) * C24(j,i) + S25(j,i) * V24(j,i) B25(j,i)= S24(j,i) * U25(j,i) + S25(j,i) * V25(j,i) C24(j,i)= S26(j,i) * T26(j,i) + S27(j,i) * U26(j,i) D24(j,i)= S26(j,i) * T27(j,i) + S27(j,i) * V26(j,i) C25(j,i)= S27(j,i) * S28(j,i) + S26(j,i) * U28(j,i) D25(j,i)= S27(j,i) * T28(j,i) + S26(j,i) * V28(j,i) end do end do into two disjoint loops do i = 1, n do j = 1, m A24(j,i)= S24(j,i) * T24(j,i) + S25(j,i) * U25(j,i) B24(j,i)= S24(j,i) * T25(j,i) + S25(j,i) * U24(j,i) A25(j,i)= S24(j,i) * C24(j,i) + S25(j,i) * V24(j,i) B25(j,i)= S24(j,i) * U25(j,i) + S25(j,i) * V25(j,i) end do end do do i = 1, n do j = 1, m C24(j,i)= S26(j,i) * T26(j,i) + S27(j,i) * U26(j,i) D24(j,i)= S26(j,i) * T27(j,i) + S27(j,i) * V26(j,i) C25(j,i)= S27(j,i) * S28(j,i) + S26(j,i) * U28(j,i) D25(j,i)= S27(j,i) * T28(j,i) + S26(j,i) * V28(j,i) end do end do Conclusions Over the course of the last year, I have had the opportunity to work with over two dozen academic scientific programmers at leading research universities. Their research interests span a broad range of scientific fields. Except for two programs that relied almost exclusively on library routines (matrix multiply and fast Fourier transform), I was able to improve significantly the single processor performance of all codes. Improvements range from 2x to 15.5x with a simple average of 4.75x. Changes to the source code were at a very high level. I did not use sophisticated techniques or programming tools to discover inefficiencies or effect the changes. Only one code was parallel despite the availability of parallel systems to all developers. Clearly, we have a problem—personal scientific research codes are highly inefficient and not running parallel. The developers are unaware of simple optimization techniques to make programs run faster. They lack education in the art of code optimization and parallel programming. I do not believe we can fix the problem by publishing additional books or training manuals. To date, the developers in questions have not studied the books or manual available, and are unlikely to do so in the future. Short courses are a possible solution, but I believe they are too concentrated to be much use. The general concepts can be taught in a three or four day course, but that is not enough time for students to practice what they learn and acquire the experience to apply and extend the concepts to their codes. Practice is the key to becoming proficient at optimization. I recommend that graduate students be required to take a semester length course in optimization and parallel programming. We would never give someone access to state-of-the-art scientific equipment costing hundreds of thousands of dollars without first requiring them to demonstrate that they know how to use the equipment. Yet the criterion for time on state-of-the-art supercomputers is at most an interesting project. Requestors are never asked to demonstrate that they know how to use the system, or can use the system effectively. A semester course would teach them the required skills. Government agencies that fund academic scientific research pay for most of the computer systems supporting scientific research as well as the development of most personal scientific codes. These agencies should require graduate schools to offer a course in optimization and parallel programming as a requirement for funding. About the Author John Feo received his Ph.D. in Computer Science from The University of Texas at Austin in 1986. After graduate school, Dr. Feo worked at Lawrence Livermore National Laboratory where he was the Group Leader of the Computer Research Group and principal investigator of the Sisal Language Project. In 1997, Dr. Feo joined Tera Computer Company where he was project manager for the MTA, and oversaw the programming and evaluation of the MTA at the San Diego Supercomputer Center. In 2000, Dr. Feo joined Sun Microsystems as an HPC application specialist. He works with university research groups to optimize and parallelize scientific codes. Dr. Feo has published over two dozen research articles in the areas of parallel parallel programming, parallel programming languages, and application performance.

    Read the article

  • Optimistic non-locking copy of InnoDB .frm files

    - by jothir
    MySQL Enterprise Backup(MEB) does hot backup of innodb data and log files. Till MEB 3.6.1, the user backs up the only innodb tables in a 3 step process: STEP 1. Take backup using --only-innodb option STEP 2. Temporarily make the table read only by executing “FLUSH TABLES WITH READ LOCK” MEB 3.7.0 has an enhancement to innodb file copying. The .frm files gets copied along with the hot backup done for innodb files. I would like to make the blog a little interactive by explaining the feature as answers: 1. What are these .frm files? The files containing the metadata, such as the table definition, of a MySQL table. For backups, the full set of .frm files are always required along with the backup data, to be able to restore tables that are altered or dropped after the backup. 2. Can the .frm files not be copied by MEB itself? --only-innodb-with-frm is the new option introduced in MEB 3.7.1 to do a copy of .frm files without locking the tables during backup operation itself. This is to reduce the pain of manually copying the .frm files. The option is intended for backups where you can ensure that no ALTER TABLE, CREATE TABLE, DROP TABLE, or other DDL statements modify the .frm files for InnoDB tables during the backup operation. 3. How is data consistency ensured? MEB does validation of the .frm files after copying by comparing with the server directory to see if the timestamps of any of the .frm files is greater than the saved system time (check .frm time).  This change in timestamp of the .frm files will show if a table is altered during the process of backup. The total number of frm files in the server directory is also verified against the copied contents. If the number of .frm files is less compared to server directory, it shows that table/tables have been dropped during the process of backup. If the number of .frm files is more compared to server directory, it shows that new table/tables have been created during backup operation. 4. How does MEB handle data inconsistency? MEB copies the .frm files through several iterations,  does the validation and throws a WARNING if there is any inconsistency found in .frm files at the end of backup operation. This means the user is warned of some DDL operations that had occurred during backup operation, and has to manually copy the .frm files or do a backup again. 5. What is the option and explain its usage? The option introduced is --only-innodb-with-frm which does optimistic copy of .frm files without locking. This can be used when the user wants to backup only innodb tables along with .frm files. The option can take one of the 2 values: all | related. --only-innodb-with-frm=all does copy of all .frm files of all innodb tables. --only-innodb-with-frm=related works in conjunction with --include option.This is to allow partial backup of .frm files corresponding to the tables specified in --include. Let me show the usage with example output: ./mysqlbackup -uroot --backup-dir=/logs/backupWithFrmAll --only-innodb-with-frm=all backup MySQL Enterprise Backup version 3.7.1 [2012/06/05] Copyright (c) 2003, 2012, Oracle and/or its affiliates. All Rights Reserved. INFO: Starting with following command line ... ./mysqlbackup -uroot --backup-dir=/logs/backupWithFrmAll        --only-innodb-with-frm=all backup INFO: Got some server configuration information from running server. IMPORTANT: Please check that mysqlbackup run completes successfully.            At the end of a successful 'backup' run mysqlbackup            prints "mysqlbackup completed OK!". --------------------------------------------------------------------                       Server Repository Options: --------------------------------------------------------------------  datadir                          =  /mysql/trydb/  innodb_data_home_dir             =    innodb_data_file_path            =  ibdata1:10M:autoextend  innodb_log_group_home_dir        =  /mysql/trydb/  innodb_log_files_in_group        =  2  innodb_log_file_size             =  5242880 --------------------------------------------------------------------                       Backup Config Options: --------------------------------------------------------------------  datadir                          =  /logs/backupWithFrmAll/datadir  innodb_data_home_dir             =  /logs/backupWithFrmAll/datadir  innodb_data_file_path            =  ibdata1:10M:autoextend  innodb_log_group_home_dir        =  /logs/backupWithFrmAll/datadir  innodb_log_files_in_group        =  2  innodb_log_file_size             =  5242880 mysqlbackup: INFO: Unique generated backup id for this is 13451979804504860 mysqlbackup: INFO: Uses posix_fadvise() for performance optimization. mysqlbackup: INFO: System tablespace file format is Antelope. mysqlbackup: INFO: Found checkpoint at lsn 1656792. mysqlbackup: INFO: Starting log scan from lsn 1656320. 120817 15:36:22 mysqlbackup: INFO: Copying log... 120817 15:36:22 mysqlbackup: INFO: Log copied, lsn 1656792.          We wait 1 second before starting copying the data files... 120817 15:36:23 mysqlbackup: INFO: Copying /mysql/trydb/ibdata1 (Antelope file format). 120817 15:36:23 mysqlbackup: INFO: Copying /mysql/trydb/innodb1/table2.ibd (Antelope file format). 120817 15:36:23 mysqlbackup: INFO: Copying /mysql/trydb/innodb1/table3.ibd (Antelope file format). 120817 15:36:23 mysqlbackup: INFO: Copying /mysql/trydb/innodb1/table1.ibd (Antelope file format). mysqlbackup: INFO: Opening backup source directory '/mysql/trydb/' 120817 15:36:23 mysqlbackup: INFO: Starting to backup .frm files in the subdirectories of /mysql/trydb/ mysqlbackup: INFO: Copying innodb data and logs during final stage ... mysqlbackup: INFO: A copied database page was modified at 1656792.          (This is the highest lsn found on page)          Scanned log up to lsn 1656792.          Was able to parse the log up to lsn 1656792.          Maximum page number for a log record 0 mysqlbackup: INFO: Copying non-innodb files took 2.000 seconds 120817 15:36:25 mysqlbackup: INFO: Full backup completed! mysqlbackup: INFO: Backup created in directory '/logs/backupWithFrmAll' -------------------------------------------------------------   Parameters Summary          -------------------------------------------------------------   Start LSN                  : 1656320   End LSN                    : 1656792 ------------------------------------------------------------- mysqlbackup completed OK! bash$ ls /logs/backupWithFrmAll/datadir/innodb1/ table1.frm  table1.ibd  table2.frm  table2.ibd  table3.frm  table3.ibd Here the backup directory contains all the .frm files of all the innodb tables. ./mysqlbackup -uroot --backup-dir=/logs/backupWithFrm --include="innodb1.table3.*" --only-innodb-with-frm=related backup MySQL Enterprise Backup version 3.7.1 [2012/06/05] Copyright (c) 2003, 2012, Oracle and/or its affiliates. All Rights Reserved. INFO: Starting with following command line ... ./mysqlbackup -uroot --backup-dir=/logs/backup371frm        --include=innodb1.table3.* --only-innodb-with-frm=related backup INFO: Got some server configuration information from running server. IMPORTANT: Please check that mysqlbackup run completes successfully.            At the end of a successful 'backup' run mysqlbackup            prints "mysqlbackup completed OK!". --------------------------------------------------------------------                       Server Repository Options: --------------------------------------------------------------------  datadir                          = /mysql/trydb/  innodb_data_home_dir             =    innodb_data_file_path            =  ibdata1:10M:autoextend  innodb_log_group_home_dir        =  /mysql/trydb  innodb_log_files_in_group        =  2  innodb_log_file_size             =  5242880 --------------------------------------------------------------------                       Backup Config Options: --------------------------------------------------------------------  datadir                          =  /logs/backupWithFrm/datadir  innodb_data_home_dir             =  /logs/backupWithFrm/datadir  innodb_data_file_path            =  ibdata1:10M:autoextend  innodb_log_group_home_dir        =  /logs/backupWithFrm/datadir  innodb_log_files_in_group        =  2  innodb_log_file_size             =  5242880 mysqlbackup: INFO: Unique generated backup id for this is 13451973458118162 mysqlbackup: INFO: Uses posix_fadvise() for performance optimization. mysqlbackup: INFO: The --include option specified: innodb1.table3.* mysqlbackup: INFO: System tablespace file format is Antelope. mysqlbackup: INFO: Found checkpoint at lsn 1656792. mysqlbackup: INFO: Starting log scan from lsn 1656320. 120817 15:25:47 mysqlbackup: INFO: Copying log... 120817 15:25:47 mysqlbackup: INFO: Log copied, lsn 1656792.          We wait 1 second before starting copying the data files... 120817 15:25:48 mysqlbackup: INFO: Copying /mysql/trydbibdata1 (Antelope file format). 120817 15:25:49 mysqlbackup: INFO: Copying /mysql/trydbinnodb1/table3.ibd (Antelope file format). mysqlbackup: INFO: Opening backup source directory '/mysql/trydb' 120817 15:25:49 mysqlbackup: INFO: Starting to backup .frm files in the subdirectories of /mysql/trydb mysqlbackup: INFO: Copying innodb data and logs during final stage ... mysqlbackup: INFO: A copied database page was modified at 1656792.          (This is the highest lsn found on page)          Scanned log up to lsn 1656792.          Was able to parse the log up to lsn 1656792.          Maximum page number for a log record 0 mysqlbackup: INFO: Copying non-innodb files took 2.000 seconds 120817 15:25:51 mysqlbackup: INFO: Full backup completed! mysqlbackup: INFO: Backup created in directory '/logs/backupWithFrm' -------------------------------------------------------------   Parameters Summary          -------------------------------------------------------------   Start LSN                  : 1656320   End LSN                    : 1656792 ------------------------------------------------------------- mysqlbackup completed OK! bash$ ls /logs/backupWithFrm/datadir/innodb1/ table3.frm table3.ibd Thus the backup directory contains only the .frm file matching the innodb table name specified in --include option. In a nutshell, we present our great new option --only-innodb-with-frm which is a true hot InnoDB-only backup with .frm files, but with an additional check, if any DDL happened during the backup. If a DDL has happened, the DBA can decide if to repeat the backup, or to live with the potential inconsistency. This is the ideal solution for users that have all their "real" data in InnoDB and seldom change their schemas. You may also like: http://dev.mysql.com/doc/mysql-enterprise-backup/3.7/en/backup-partial-options.html   STEP 3. Manually copy the .frm files of innodb tables to the destination directory where backup is stored.

    Read the article

  • Boot log from remotely managed/hacked iPhone for analysis

    - by user1319903
    in reference to my other post. syslog captured immediately after a hard reset for analysis of foul play. Apr 8, 2012 10:08:36 PM - dataaccessd [53] (Notice): 137860|CoreDAV|Warn |Account "iCloud" couldn't reach the server at p03-contacts.icloud.com: Error Domain=NSURLErrorDomain Code=-1009 "The Internet connection appears to be offline." UserInfo=0xde63920 {NSErrorFailingURLStringKey=https://%[email protected]/159665024/principal/, NSErrorFailingURLKey=https://%[email protected]/ /principal/, NSLocalizedDescription=The Internet connection appears to be offline., NSUnderlyingError=0xde7dc00 "The Internet connection appears to be offline."} Apr 8, 2012 10:08:36 PM - UserEventAgent [12] (Warning): TRACE: connection interrupted Apr 8, 2012 10:08:36 PM - UserEventAgent [12] (Warning): DEBUG: disconnected Apr 8, 2012 10:08:36 PM - UserEventAgent [12] (Warning): TRACE: Canceling Apr 8, 2012 10:08:36 PM - UserEventAgent [12] (Warning): TRACE: connection invalid Apr 8, 2012 10:08:35 PM - kernel [0] (Debug): launchd[82] Builtin profile: container (sandbox) Apr 8, 2012 10:08:35 PM - kernel [0] (Debug): launchd[82] Container: /private/var/mobile/Applications/048D35CA-6427-4EC8-8B76-A194697A7CE9 [69] (sandbox) Apr 8, 2012 10:08:35 PM - wifid [29] (Error): WiFi:[355640915.904103]: Client dataaccessd set type to background application Apr 8, 2012 10:08:35 PM - dataaccessd [53] (Notice): 137860|DA|Warn |Delegate 5ADDBE3B-D5FD-43E1-87D4-C1153733EFAB finished a refresh but it is not registered with the refresh manager Apr 8, 2012 10:08:34 PM - timed [31] (Notice): (Note ) CoreTime: Not setting system time to 04/09/2012 05:08:34 from GPS because time is unchanged Apr 8, 2012 10:08:34 PM - timed [31] (Notice): (Note ) CoreTime: Not setting time zone to America/Los_Angeles from NITZ Apr 8, 2012 10:08:33 PM - kernel [0] (Debug): AppleKeyStore:cp_key_store_action(1) Apr 8, 2012 10:08:33 PM - kernel [0] (Debug): AppleKeyStore:Sending lock change Apr 8, 2012 10:08:32 PM - profiled [20] (Notice): (Note ) profiled: Device unlock notification received Apr 8, 2012 10:08:31 PM - softwareupdated [37] (Notice): 3e828d98 : Cleaning up unused prepared updates Apr 8, 2012 10:08:27 PM - mstreamd [43] (Warning): PSDLog: Can't return photoStreamsPublishStreamID because no Apple Account has Photo Streams enabled Apr 8, 2012 10:08:27 PM - mstreamd [43] (Notice): (Note ) mstreamd: Not listening to push notifications. Apr 8, 2012 10:08:27 PM - mstreamd [43] (Warning): PSDLog: Can't return photoStreamsPublishStreamID because no Apple Account has Photo Streams enabled Apr 8, 2012 10:08:27 PM - mstreamd [43] (Notice): (Note ) mstreamd: Not listening to push notifications. Apr 8, 2012 10:08:27 PM - mstreamd [43] (Notice): (Note ) mstreamd: Retrieved push tokens. Dev: 0, Prod: 0 Apr 8, 2012 10:08:27 PM - mstreamd [43] (Notice): (Note ) mstreamd: Media stream daemon starting... Apr 8, 2012 10:08:26 PM - SpringBoard [15] (Notice): SMSCTServer is available and ready to rock. Apr 8, 2012 10:08:26 PM - SpringBoard [15] (Error): mms: * isMmsConfigured = 1 Apr 8, 2012 10:08:26 PM - MobilePhone [79] (Warning): Connection lost, retrying with key exchange. Apr 8, 2012 10:08:26 PM - MobilePhone [79] (Warning): Connection lost, retrying with key exchange. Apr 8, 2012 10:08:26 PM - MobilePhone [79] (Warning): Connection lost, retrying with key exchange. Apr 8, 2012 10:08:26 PM - MobilePhone [79] (Warning): Connection lost, retrying with key exchange. Apr 8, 2012 10:08:25 PM - SpringBoard [15] (Warning): BT: failed to get connectable state with error 111 Apr 8, 2012 10:08:25 PM - SpringBoard [15] (Error): WiFi: Consulting "no-sdio-devices" property. Apr 8, 2012 10:08:25 PM - SpringBoard [15] (Error): WiFi: "no-sdio-devices" property not found. Apr 8, 2012 10:08:25 PM - SpringBoard [15] (Warning): SMS Plugin initialized. Apr 8, 2012 10:08:25 PM - SpringBoard [15] (Warning): Telephony plugin initialized Apr 8, 2012 10:08:25 PM - SpringBoard [15] (Warning): SIMToolkit plugin for SpringBoard initialized. Apr 8, 2012 10:08:25 PM - SpringBoard [15] (Error): WiFi: Consulting "no-sdio-devices" property. Apr 8, 2012 10:08:25 PM - SpringBoard [15] (Error): WiFi: "no-sdio-devices" property not found. Apr 8, 2012 10:08:25 PM - SpringBoard [15] (Warning): WiFi picker plugin initialized Apr 8, 2012 10:08:25 PM - SpringBoard [15] (Warning): EKAlarmEngine: Region monitoring not available or enabled. Trigger ignored! Apr 8, 2012 10:08:24 PM - kernel [0] (Debug): AppleH4CamIn::setPowerStateGated: 0 Apr 8, 2012 10:08:24 PM - kernel [0] (Debug): AppleH4CamIn::power_off_hardware Apr 8, 2012 10:08:24 PM - SpringBoard [15] (Notice): IOMobileFrameBufferGetMirroringCapability returning -536870201 via kIOMFBConnectMethod_GetMirroringCapability  Apr 8, 2012 10:08:24 PM - aggregated [61] (Warning): PLAggregateState Error: Leaving state unplugged_screen_off even though we are not in it, doing nothing Apr 8, 2012 10:08:24 PM - aggregated [61] (Warning): PLAggregateState Error: Entering state unplugged_screen_on even though we are already in it, doing nothing Apr 8, 2012 10:08:24 PM - wifid [29] (Error): WiFi:[355640904.616440]: Disable WoW requested by "spd" Apr 8, 2012 10:08:24 PM - SpringBoard [15] (Warning): Application windows are expected to have a root view controller at the end of application launch Apr 8, 2012 10:08:23 PM - SpringBoard [15] (Warning): BTM: attaching to BTServer Apr 8, 2012 10:08:23 PM - kernel [0] (Debug): AppleH4CamIn::ISP_LoadFirmware_gated: fw len=1232920 Apr 8, 2012 10:08:23 PM - kernel [0] (Debug): AppleH4CamIn::ISP_LoadFirmware_gated - firmware checksum: 0x05935019 Apr 8, 2012 10:08:23 PM - kernel [0] (Debug): AppleH4CamIn::power_on_hardware Apr 8, 2012 10:08:23 PM - kernel [0] (Debug): AppleH4CamIn::ISP_Init - No set-file loaded for camera channel 0 Apr 8, 2012 10:08:23 PM - kernel [0] (Debug): AppleH4CamIn::ISP_Init - No set-file loaded for camera channel 1 Apr 8, 2012 10:08:23 PM - kernel [0] (Debug): AppleH4CamIn::ISP_InitialSensorDetection - found sensor on chan 0: 0x0145 Apr 8, 2012 10:08:23 PM - kernel [0] (Debug): AppleH4CamIn::ISP_InitialSensorDetection - found sensor on chan 1: 0x7736 Apr 8, 2012 10:08:23 PM - kernel [0] (Debug): AppleH4CamIn::power_off_hardware Apr 8, 2012 10:08:23 PM - kernel [0] (Debug): AppleH4CamIn::ISP_LoadSetfile_gated (camChan=0) Apr 8, 2012 10:08:23 PM - kernel [0] (Debug): AppleH4CamIn::ISP_LoadSetfile_gated (camChan=1) Apr 8, 2012 10:08:23 PM - kernel [0] (Debug): AppleH4CamIn::setPowerStateGated: 1 Apr 8, 2012 10:08:23 PM - kernel [0] (Debug): AppleH4CamIn::power_on_hardware Apr 8, 2012 10:08:23 PM - profiled [20] (Notice): (Note ) profiled: Locking device Apr 8, 2012 10:08:22 PM - kernel [0] (Debug): HighlandParkResourceMgr::AddFirmware() {'cdma', '    '} added to resources Apr 8, 2012 10:08:22 PM - kernel [0] (Debug): AppleSynopsysOTGDevice::gated_registerFunction Register function PTP Apr 8, 2012 10:08:22 PM - kernel [0] (Debug): AppleSynopsysOTGDevice::gated_registerFunction all functions registered- we are ready to start usb stack Apr 8, 2012 10:08:22 PM - kernel [0] (Debug): AppleSynopsysOTGDevice::handleUSBCableDisconnect Apr 8, 2012 10:08:22 PM - kernel [0] (Debug): HighlandParkResourceMgr::AddFirmware() {'gsm ', 'nb  '} added to resources Apr 8, 2012 10:08:22 PM - kernel [0] (Debug): HighlandParkResourceMgr::AddFirmware() {'gsm ', 'wb  '} added to resources Apr 8, 2012 10:08:22 PM - MRMLowDiskUEA [12] (Notice): MobileDelete: LowDisk Plugin: start Apr 8, 2012 10:08:22 PM - MRMLowDiskUEA [12] (Notice): kqueue registration successful Apr 8, 2012 10:08:22 PM - mediaserverd [44] (Error): 22:08:22.522867 com.apple.AVConference: /SourceCache/GameKitServices/GameKitServices-344.21/AVConference.subproj/Sources/AVConferenceServer.m:1867: AVConferenceServerStart Apr 8, 2012 10:08:22 PM - CommCenter [18] (Notice): Carrier bundle value for recipient address: 28818773 Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): AppleSynopsysOTGDevice - Configuration: PTP Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): AppleSynopsysOTGDevice          Interface: PTP Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): AppleSynopsysOTGDevice - Configuration: iPod USB Interface Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): AppleSynopsysOTGDevice          Interface: USBAudioControl Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): AppleSynopsysOTGDevice          Interface: USBAudioStreaming Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): AppleSynopsysOTGDevice          Interface: IapOverUsbHid Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): AppleSynopsysOTGDevice - Configuration: PTP + Apple Mobile Device Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): AppleSynopsysOTGDevice          Interface: PTP Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): AppleSynopsysOTGDevice          Interface: AppleUSBMux Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): AppleSynopsysOTGDevice - Configuration: PTP + Apple Mobile Device + Apple USB Ethernet Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): AppleSynopsysOTGDevice          Interface: PTP Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): AppleSynopsysOTGDevice          Interface: AppleUSBMux Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): AppleSynopsysOTGDevice          Interface: AppleUSBEthernet Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): IOAccessoryPortUSB::start Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): AppleSynopsysOTGDevice::gated_registerFunction Register function USBAudioControl Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): virtual bool AppleUSBDeviceMux::start(IOService*) build: Feb  1 2012 23:16:46 Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): init_waste Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): AppleSynopsysOTGDevice::gated_registerFunction Register function AppleUSBMux Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): AppleSynopsysOTGDevice::gated_registerFunction Register function IapOverUsbHid Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): AppleSynopsysOTGDevice::gated_registerFunction Register function USBAudioStreaming Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): AppleSynopsysOTGDevice::gated_registerFunction Register function AppleUSBEthernet Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): AppleUSBEthernetDevice::start: Host MAC address = 02:(this Mac address does not physically exist) -edit Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): AppleUSBEthernetDevice: Ethernet address  Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): BTServer[66] Builtin profile: BlueTool (sandbox) Apr 8, 2012 10:08:21 PM - kernel [0] (Debug): BTServer[66] Builtin profile: BlueTool (sandbox) Apr 8, 2012 10:08:21 PM - hpfd [50] (Notice): firmware resource loaded { 'cdma' '    ' } Apr 8, 2012 10:08:21 PM - wifid [29] (Error): WiFi:[355640901.282776]: Could not read APPLE80211_IOC_SUPPORTED_CHANNELS err=82 Apr 8, 2012 10:08:21 PM - wifid [29] (Error): WiFi:[355640901.312786]: Client itunesstored is background application Apr 8, 2012 10:08:21 PM - timed [31] (Notice): (Note ) CoreTime: Want active time in 38.24hrs. Need active time in 121.57hrs. Apr 8, 2012 10:08:21 PM - SpringBoard [15] (Notice): MultitouchHID: detection mode: 255-0 (deferring until bootloaded) Apr 8, 2012 10:08:21 PM - CLTM [12] (Error): CLTM: resetting temps: now = 1333948101, last update = -2147483648 Apr 8, 2012 10:08:21 PM - locationd [28] (Error): WiFi:[355640901.852993]: WiFiManager now available Apr 8, 2012 10:08:21 PM - OTACrashCopier [62] (Notice): (Warn ) Failed to read attributes from '/var/mobile/Library/OTALogging/.last_successful_submission_marker' Apr 8, 2012 10:08:21 PM - hpfd [50] (Notice): firmware resource loaded { 'gsm ' 'nb  ' } Apr 8, 2012 10:08:21 PM - hpfd [50] (Notice): firmware resource loaded { 'gsm ' 'wb  ' } Apr 8, 2012 10:08:20 PM - kernel [0] (Debug): AppleBCMWLANCore::initFirmware(): successful initialization Apr 8, 2012 10:08:20 PM - kernel [0] (Debug): AppleBCMWLANCore:initFirmware(): 2496 PropTxStatus feature is not enabled for this platform  Apr 8, 2012 10:08:20 PM - kernel [0] (Debug): AppleBCMWLANCore::initDongle():: creating virtual interface with prefix = ap Apr 8, 2012 10:08:20 PM - kernel [0] (Debug): AppleBCMWLANCore::initDongle(): Core Driver Initialization Time 19.38798583 Apr 8, 2012 10:08:20 PM - kernel [0] (Debug): 000019.281423 hsic-baseband::safetyNet: port is not connected Apr 8, 2012 10:08:20 PM - lockdownd [23] (Notice): 3e828d98 _create_cesm_vault: try to create blob Apr 8, 2012 10:08:20 PM - lockdownd [23] (Notice): 3e828d98 load_activation_records: This is the default record Apr 8, 2012 10:08:20 PM - lockdownd [23] (Notice): 3e828d98 _create_cesm_vault: blob written Apr 8, 2012 10:08:20 PM - lockdownd [23] (Notice): 3e828d98 ping_configd: Not setting host name, it already has one: Pete's iPod  Apr 8, 2012 10:08:20 PM - lockdownd [23] (Notice): 3e828d98 lookup_baseband_info_new: radio not ready: kCTPostponementStatusNotReady Apr 8, 2012 10:08:20 PM - lockdownd [23] (Notice): 3e828d98 load_activation_records: This is the default record Apr 8, 2012 10:08:20 PM - SpringBoard [15] (Error): WiFi: Consulting "no-sdio-devices" property. Apr 8, 2012 10:08:20 PM - SpringBoard [15] (Error): WiFi: "no-sdio-devices" property not found. Apr 8, 2012 10:08:20 PM - lockdownd [23] (Notice): 3e828d98 determine_activation_state_new: Original act. state: Activated Apr 8, 2012 10:08:20 PM - lockdownd [23] (Notice): 3e828d98 determine_activation_state_new: radio not ready, don't change activation status, wait for notification, status: kCTPostponementStatusNotReady Apr 8, 2012 10:08:20 PM - lockdownd [23] (Notice): 3e828d98 determine_activation_state_new: Activation state now is Activated Apr 8, 2012 10:08:20 PM - SpringBoard [15] (Warning): lockdown says the device is: [Activated], state is 3 Apr 8, 2012 10:08:20 PM - SpringBoard [15] (Warning): lockdown says we've previously registered: [1], state is 1 Apr 8, 2012 10:08:20 PM - lockdownd [23] (Notice): 3e828d98 notification_worker: now listening for CT notifications Apr 8, 2012 10:08:20 PM - lockdownd [23] (Notice): 3e828d98 notification_worker: we've registered for notifications, now make sure we didn't miss one... Apr 8, 2012 10:08:20 PM - lockdownd [23] (Notice): 3e828d98 load_activation_records: This is the default record Apr 8, 2012 10:08:20 PM - lockdownd [23] (Notice): 3e828d98 determine_activation_state_new: Original act. state: Activated Apr 8, 2012 10:08:20 PM - lockdownd [23] (Notice): 3e828d98 determine_activation_state_new: radio not ready, don't change activation status, wait for notification, status: kCTPostponementStatusNotReady Apr 8, 2012 10:08:20 PM - lockdownd [23] (Notice): 3e828d98 determine_activation_state_new: Activation state now is Activated Apr 8, 2012 10:08:20 PM - SpringBoard [15] (Notice): Posting 'com.apple.iokit.hid.displayStatus' notifyState=1 Apr 8, 2012 10:08:20 PM - SpringBoard [15] (Notice): __IOHIDLoadBundles: Loaded 1 HID plugin Apr 8, 2012 10:08:19 PM - wifiFirmwareLoader [30] (Warning): [    18.778 sec] Downloaded firmware, 192512 bytes Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): AppleKeyStore:cp_key_store_action(0) Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): AppleMultitouchN1SPI: downloaded 128 bytes of prox calibration data ("built-in") Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): AppleMultitouchN1SPI: downloaded 1024 bytes of calibration data ("built-in") Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): AppleBCMWLANCore::attachBusGated(): Bus Driver Initialization Time 18.266927958 Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): AppleBCMWLANCore:attachBusGated(): Starting with MAC Address: 00:f4:b9:2f:d9:8d Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): AppleBCMWLANFirmwareManager::setNVRAMData(): received 778 bytes Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): AppleBCMWLANCore: Ethernet address 00:f4:b9:2f:d9:8d Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): Loading syscfg. Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): AppleMultitouchN1SPI: downloaded 56264 bytes of firmware data ("0x0084.bin") in 152ms. Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): AppleBCMWLANCore::apple80211_ioctl() Driver not yet initialized, cannot process ioctl Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): AppleBCMWLANCore::apple80211_ioctl() Driver not yet initialized, cannot process ioctl Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): AirPort: Enabled AppleBCMWLANCore (link 0, sys 0, user 0) Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): AppleBCMWLANCore::apple80211_ioctl() Driver not yet initialized, cannot process ioctl Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): AppleBCMWLANCore::apple80211_ioctl() Driver not yet initialized, cannot process ioctl Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): AppleBCMWLANBusInterfaceHSIC::loadFirmware(): DL Ver: chip 0x4330, chiprev 0x4 Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): BTServer[66] Builtin profile: BlueTool (sandbox) Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): BCMWLAN Firmware Version: wl0: Dec 22 2011 19:03:58 version 5.95.45 Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): AppleBCMWLANCore::initFirmware(): Firmware supports ap mode; enabling apsta feature (currently enabled) Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): AppleBCMWLANCore::initFirmware(): country code set to XX Apr 8, 2012 10:08:19 PM - configd [14] (Notice): network configuration changed. Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): AppleBCMWLANCmdManager::processResponse(): Firmware Error "BCOM Unsupported" on command "WLC_SET_VAR: bus:txglom" (263). Transaction ID 3, length 0 Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): AppleBCMWLANCore::initFirmware(): Glomming not supported on this device: BCOM Unsupported Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): AppleBCMWLANCore::initFirmware: apsta set to 1 Apr 8, 2012 10:08:19 PM - kernel [0] (Debug): AppleBCMWLANCore::handleEventPacket(): WLC_E_FIFO_CREDIT_MAP,length 6 [9 2 5 3 2] Apr 8, 2012 10:08:19 PM - iapd [49] (Error): Timed out trying to acquire capabilities data. Apr 8, 2012 10:08:19 PM - softwareupdated [37] (Notice): 3e828d98 : Cleaning up unused prepared updates Apr 8, 2012 10:08:19 PM - com.apple.misd [63] (Notice): allowing special port forwarding for test fixtures Apr 8, 2012 10:08:19 PM - hpfd [50] (Notice): resource request { 'N94 ', '    ' } Apr 8, 2012 10:08:19 PM - mstreamd [43] (Notice): (Note ) mstreamd: mstreamd starting up. Apr 8, 2012 10:08:18 PM - kernel [0] (Debug): launchd[44] Builtin profile: mediaserverd (sandbox) Apr 8, 2012 10:08:18 PM - kernel [0] (Debug): launchd[49] Builtin profile: iapd (sandbox) Apr 8, 2012 10:08:18 PM - kernel [0] (Debug): launchd[53] Builtin profile: dataaccessd (sandbox) Apr 8, 2012 10:08:18 PM - kernel [0] (Debug): launchd[60] Builtin profile: apsd (sandbox) Apr 8, 2012 10:08:18 PM - kernel [0] (Debug): launchd[66] Builtin profile: BTServer (sandbox) Apr 8, 2012 10:08:18 PM - mDNSResponder [46] (Error): mDNSResponder mDNSResponder-329.10 (Jan 15 2012 19:07:41) starting iOSVers 9 Apr 8, 2012 10:08:18 PM - mDNSResponder [46] (Error): Note: SetDomainSecrets: no keychain support Apr 8, 2012 10:08:18 PM - mDNSResponder [46] (Error): Note: Compiled without SnowLeopard Fine-Grained Power Management support Apr 8, 2012 10:08:18 PM - fseventsd [51] (Critical): event logs in /private/var/.fseventsd out of sync with volume.  destroying old logs. (10083 7 10090) Apr 8, 2012 10:08:18 PM - fseventsd [51] (Critical): log dir: /private/var/.fseventsd getting new uuid: 8778E61A-0283-4067-B7DF-F75D109983D1 Apr 8, 2012 10:08:18 PM - fseventsd [51] (Error): failed to make the directory /.fseventsd (30/Read-only file system) Apr 8, 2012 10:08:18 PM - fseventsd [51] (Critical): could not open < (No such file or directory) Apr 8, 2012 10:08:18 PM - fseventsd [51] (Critical): log dir: /tmp getting new uuid: 3919EB54-A54F-4289-864A-5158A25EF9DA Apr 8, 2012 10:08:18 PM - wifid [29] (Error): WiFi:[355640898.328610]: WiFi Preferences is up to date Apr 8, 2012 10:08:18 PM - mDNSResponder [46] (Error): D2DInitialize succeeded Apr 8, 2012 10:08:18 PM - fairplayd.N94 [52] (Notice): Vroum Apr 8, 2012 10:08:18 PM - wifid [29] (Error): WiFi:[355640898.537219]: WiFiManager starting, version: WiFiManager-260.9 Feb  4 2012 13:25:16 Apr 8, 2012 10:08:18 PM - configd [14] (Error): WiFi:[355640898.539342]: WiFiManager now available Apr 8, 2012 10:08:18 PM - keybagd [39] (Error): 3e828d98 main: System Keybag loaded Apr 8, 2012 10:08:18 PM - wifiFirmwareLoader [30] (Warning): [    18.268 sec] Found AppleBCMWLANBusInterface; downloading FW.. Apr 8, 2012 10:08:18 PM - wifiFirmwareLoader [30] (Warning): Loading "/usr/share/firmware/wifi/4330b2/bcm94330OlympicUNO3.txt", file size = 778 bytes Apr 8, 2012 10:08:18 PM - wifiFirmwareLoader [30] (Warning): [    18.276 sec] Sending NVRAM, 778 bytes Apr 8, 2012 10:08:18 PM - wifiFirmwareLoader [30] (Warning): Loading "/usr/share/firmware/wifi/4330b2/n94.trx", file size = 192512 bytes Apr 8, 2012 10:08:18 PM - wifiFirmwareLoader [30] (Warning): [    18.300 sec] Sending firmware, 192512 bytes Apr 8, 2012 10:08:18 PM - lockdownd [23] (Error): libMobileGestalt copyEthernetMacAddress: got 00:f4:b9:2f:d9:8f from syscfg Apr 8, 2012 10:08:18 PM - mediaserverd [44] (Notice): 2012-04-08 10:08:18.817015 PM [AirTunes] HAL plugin started Apr 8, 2012 10:08:18 PM - lockdownd [23] (Error): libMobileGestalt createCFStringWithCFData: Cannot convert NULL data to string Apr 8, 2012 10:08:18 PM - lockdownd [23] (Error): libMobileGestalt copyBasebandBoardSnum: Could not convert baseband board snum data to string Apr 8, 2012 10:08:18 PM - lockdownd [23] (Error): libMobileGestalt createCFStringWithCFData: Cannot convert NULL data to string Apr 8, 2012 10:08:18 PM - lockdownd [23] (Error): libMobileGestalt copyWirelessBoardSnum: Could not convert wireless board snum data to string Apr 8, 2012 10:08:18 PM - lockdownd [23] (Notice): 3e828d98 lockstart_local: Build= 9B179 Apr 8, 2012 10:08:18 PM - lockdownd [23] (Notice): 3e828d98 _load_product_type: using Raptor Certs Apr 8, 2012 10:08:17 PM - wifiFirmwareLoader [30] (Warning): [    17.590 sec] wlan AppleUSBHSICDevice found Apr 8, 2012 10:08:17 PM - wifiFirmwareLoader [30] (Warning): [    17.590 sec] WLAN Enumeration attempt 0 / 6: Apr 8, 2012 10:08:17 PM - wifiFirmwareLoader [30] (Warning): [    17.591 sec] Waiting for AppleBCMWLANBusInterface to enumerate... Apr 8, 2012 10:08:16 PM - CommCenter [18] (Notice): MMS thread running Apr 8, 2012 10:08:16 PM - CommCenter [18] (Notice): Communications Center Started. Apr 8, 2012 10:08:16 PM - CommCenter [18] (Notice): STOP LOCATION UPDATE Apr 8, 2012 10:08:16 PM - locationd [28] (Error): WiFi:[355640896.704327]: bootstrap_look_up of WiFiManager server failed Apr 8, 2012 10:08:16 PM - locationd [28] (Error): WiFi:[355640896.705542]: bootstrap_look_up of WiFiManager server failed Apr 8, 2012 10:08:16 PM - locationd [28] (Error): WiFi:[355640896.706648]: bootstrap_look_up of WiFiManager server failed Apr 8, 2012 10:08:16 PM - locationd [28] (Error): WiFi:[355640896.707418]: bootstrap_look_up of WiFiManager server failed Apr 8, 2012 10:08:15 PM - kernel [0] (Debug): bool AppleRGBOUT::power_down_hardware(), RGB_CTRL (0x00000000) clk_down_ready is not set after 60 msecs Apr 8, 2012 10:08:14 PM - lockdownd [23] (Notice): 3e828d98 main: Starting Up Apr 8, 2012 10:08:14 PM - kernel [0] (Debug): IOReturn AppleRGBOUT::set_display_device_gated(uint32_t), 1 Apr 8, 2012 10:08:14 PM - kernel [0] (Debug): virtual void AppleRGBOUT::do_power_state_change(): fSoft: 1 fHard: 1 swapBusy: 1  fController: 0 - 1 Apr 8, 2012 10:08:14 PM - kernel [0] (Debug): bool AppleRGBOUT::power_up_hardware() Apr 8, 2012 10:08:14 PM - kernel [0] (Debug): set_crc_notification_state 0 Apr 8, 2012 10:08:14 PM - kernel [0] (Debug): virtual void AppleRGBOUT::do_power_state_change(): fSoft: 0 fHard: 1 swapBusy: 0  fController: 1 - 0 Apr 8, 2012 10:08:14 PM - kernel [0] (Debug): bool AppleRGBOUT::power_down_hardware() Apr 8, 2012 10:08:14 PM - kernel [0] (Debug): IOReturn IOMobileFramebufferUserClient::set_hotplug_notify(void *, void *) 0x314b3f0d 0xe215600 Apr 8, 2012 10:08:14 PM - kernel [0] (Debug): IOReturn IOMobileFramebufferUserClient::set_hotplug_notify(void *, void *) 0x849d5000 0x876e8828 0x314b3f0d 0xe215600 Apr 8, 2012 10:08:14 PM - kernel [0] (Debug): bool AppleRGBOUT::power_down_hardware(), clock down RGBOUT Apr 8, 2012 10:08:14 PM - SpringBoard [15] (Notice): IOMobileFrameBufferGetMirroringCapability returning -536870201 via kIOMFBConnectMethod_GetMirroringCapability  Apr 8, 2012 10:08:14 PM - backupd [21] (Warning): INFO: Account changed (enabled=0, accountID=159665024) Apr 8, 2012 10:08:13 PM - kernel [0] (Debug): launchd[17] Builtin profile: ptpd (sandbox) Apr 8, 2012 10:08:13 PM - UserEventAgent [12] (Warning): Factory called Apr 8, 2012 10:08:13 PM - configd [14] (Error): WiFi:[355640893.157493]: bootstrap_look_up of WiFiManager server failed Apr 8, 2012 10:08:13 PM - configd [14] (Error): WiFi:[355640893.158197]: bootstrap_look_up of WiFiManager server failed Apr 8, 2012 10:08:13 PM - configd [14] (Error): WiFi:[355640893.158878]: bootstrap_look_up of WiFiManager server failed Apr 8, 2012 10:08:13 PM - UserEventAgent [12] (Notice): (Note ) PIH: MCUEAPlugin initialized. Apr 8, 2012 10:08:13 PM - UserEventAgent [12] (Error): Querying interface Apr 8, 2012 10:08:13 PM - configd [14] (Error): ioctl(SIOCGIFCAP) failed: Device not configured Apr 8, 2012 10:08:13 PM - configd [14] (Error): ioctl(SIOCGIFCAP) failed: Device not configured Apr 8, 2012 10:08:13 PM - configd [14] (Notice): setting hostname to "Petes-iPod" Apr 8, 2012 10:08:13 PM - configd [14] (Notice): network configuration changed. Apr 8, 2012 10:08:13 PM - UserEventAgent [12] (Warning): TRACE: sending {    command = kMBMessageAccountChanged; } Apr 8, 2012 10:08:13 PM - profiled [20] (Notice): (Note ) profiled: Service starting... Apr 8, 2012 10:08:13 PM - profiled [20] (Notice): (Note ) profiled: Performing boot time checks. Apr 8, 2012 10:08:13 PM - profiled [20] (Notice): (Note ) MC: Checking for MDM installation... Apr 8, 2012 10:08:13 PM - profiled [20] (Notice): (Note ) MC: ...finished checking for MDM installation. Apr 8, 2012 10:08:13 PM - profiled [20] (Notice): (Note ) profiled: Checking for new carrier profile... Apr 8, 2012 10:08:13 PM - profiled [20] (Notice): (Note ) profiled: Installing new carrier profile. Apr 8, 2012 10:08:13 PM - profiled [20] (Notice): (Note ) profiled: Carrier profile has already been installed. Apr 8, 2012 10:08:12 PM - com.apple.launchd [1] (Warning): (com.apple.ptpd) The exception server is already claimed! Apr 8, 2012 10:08:12 PM - CLTM [12] (Notice): CLTM: mitigation behavior enabled Apr 8, 2012 10:08:12 PM - CLTM [12] (Notice): CLTM: camera equations enabled Apr 8, 2012 10:08:12 PM - CLTM [12] (Notice): CLTM: thermal monitoring enabled Apr 8, 2012 10:08:12 PM - CLTM [12] (Notice): CLTM: registered for wake notification Apr 8, 2012 10:08:12 PM - CLTM [12] (Notice): CLTM: set decay on sensor 0 to 16384 Apr 8, 2012 10:08:12 PM - CLTM [12] (Notice): CLTM: set decay on sensor 1 to 546 Apr 8, 2012 10:08:12 PM - CLTM [12] (Notice): CLTM: set decay on sensor 2 to 5461 Apr 8, 2012 10:08:12 PM - CLTM [12] (Notice): CLTM: set decay on sensor 3 to 6553 Apr 8, 2012 10:08:12 PM - CLTM [12] (Notice): CLTM: set decay on sensor 4 to 5461 Apr 8, 2012 10:08:12 PM - CLTM [12] (Notice): CLTM: set decay on sensor 5 to 5461 Apr 8, 2012 10:08:12 PM - CLTM [12] (Notice): CLTM: set decay on sensor 6 to 16384 Apr 8, 2012 10:08:12 PM - CLTM [12] (Notice): CLTM: set decay on sensor 9 to 5461 Apr 8, 2012 10:08:12 PM - CLTM [12] (Notice): CLTM: set decay on sensor 10 to 5461 Apr 8, 2012 10:08:12 PM - CLTM [12] (Notice): CLTM: Set AppleARMPerformanceControllerDVDFactor1 dithering level to 101% Apr 8, 2012 10:08:12 PM - CLTM [12] (Notice): CLTM: Set AppleARMPerformanceControllerDVDFactor0 dithering level to 100% Apr 8, 2012 10:08:12 PM - CLTM [12] (Notice): CLTM: Set charge rate index to 0 Apr 8, 2012 10:08:12 PM - CLTM [12] (Notice): CLTM: HID not ready cannot set BL Apr 8, 2012 10:08:12 PM - CLTM [12] (Notice): CLTM: setting thermal status level to 0 (0) [-32768, -32768, -32768, -32768, -32768, -32768, -32768, -32768, -32768, -32768, -32768, -32768, -32768, -32768, -32768] Apr 8, 2012 10:08:12 PM - CLTM [12] (Notice): CLTM: set allowable transmit power limit to 24.000 dBm [-32768, -32768, -32768, -32768, -32768, -32768, -32768, -32768, -32768, -32768, -32768, -32768, -32768, -32768, -32768] Apr 8, 2012 10:08:12 PM - CLTM [12] (Error): CLTM: Could not close relay file Apr 8, 2012 10:08:12 PM - CLTM [12] (Notice): CLTM: thermtgraphrelay is not present

    Read the article

  • Problem with Entity Framework : "The underlying provider failed on Open"

    - by pokrate
    Hi, When I try to insert a record, I get this error : The underlying provider failed on Open. This error occurs only with IIS and not with VWD 2008's webserver. In the EventViewer I get this Application Error : Failed to generate a user instance of SQL Server due to a failure in starting the process for the user instance. The connection will be closed. [CLIENT: ] <add name="ASPNETDBEntities" connectionString="metadata=res://*/Models.FriendList.csdl|res://*/Models.FriendList.ssdl|res://*/Models.FriendList.msl;provider=System.Data.SqlClient;provider connection string=&quot;Data Source=.\SQLEXPRESS;AttachDbFilename=|DataDirectory|\ASPNETDB.MDF;Integrated Security=True;Connect Timeout=30;User Instance=True;MultipleActiveResultSets=True&quot;" providerName="System.Data.EntityClient" /> I am using aspnetdb.mdf file, and not any external database. I have searched enough for this, but no use. Everything works fine with VWD webserver

    Read the article

  • MVC Partial View to Call Display Pop-up window using jquery

    - by Gavin campbell
    Hi I have a index page the renders objects from my database as a treeview, each item has a link href="/MessageGroupType/Edit/1002 that makes an Ajax call to display a partial view in a DIV. Within the partial view there is a delete button which calls my controller to delete the item. However, i do a check to make sure the item can be deleted, if the item cant be deleted then i wish a pop-up to appear back on the edit form telling the user they cant delete this record. In my Edit partial view i have the following code " type="text/javascript" $(function() { $("#dialog").dialog(); }); <% if (Boolean.Parse(ViewData["DisplayWindow"].ToString())){% This Mesage group Type Cannot be deleted as is linked to other message group Types <% }% So my main questions are Can i make a reference to a javascript script within my Partial View (i dont want my master page to be called on the partial view) When i dynamically load the partial view data into my DIV - can i then after calling my controller insert another DIV into the first DIV. I am i doing this the wrong way - so any pointers is appreciated Cheers

    Read the article

  • Cannot build/create dll in Visual Studio because of admin rights??

    - by Vidar
    I have local admin rights on the PC I am using - but some things are not allowed i.e. I can't right click on My Computer and see the properties! Anyway I was trying to build my solution in Visual Studio - I just added a simple class library with barely any code in it and it won't build, it gives the error: C:\WINDOWS\Microsoft.NET\Framework\v3.5\Microsoft.Common.targets(3019,9): error MSB3216: Cannot register assembly "C:\Documents and Settings\fooUser\My Documents\Visual Studio 2008\Projects\FooSolution\Foo.Bar\bin\Debug\Foo.Bar.Utility.dll" - access denied. Please make sure you're running the application as administrator. Access to the registry key 'HKEY_CLASSES_ROOT\Record' is denied. I take it I need FULL admin rights?? - although the project did compile before I made this class library - so I dont' understand why the addition of this library has had such a drastic effect now!

    Read the article

  • Linq to sql Invalid column name

    - by Ivo
    I am using VS 2010 and Linq-to-sql I have deleted a couple of columns from my table, whatever I try I keep getting and "invalide column name" error with the old/deleted column when I try to insert a record. What tried: remove table,save, and drag it back on create new dblm with an other name created new project and create new dbml there Also the name of the deleted column does not exist in the generated code of the dbml anymore, but still I get an exception saying that column does not exists. Anyone got a suggestion how to solve this problem?

    Read the article

  • Nested SQL Select statement fails on SQL Server 2000, ok on SQL Server 2005

    - by Jay
    Here is the query: INSERT INTO @TempTable SELECT UserID, Name, Address1 = (SELECT TOP 1 [Address] FROM (SELECT TOP 1 [Address] FROM [UserAddress] ua INNER JOIN UserAddressOrder uo ON ua.UserID = uo.UserID WHERE ua.UserID = u.UserID ORDER BY uo.AddressOrder ASC) q ORDER BY AddressOrder DESC), Address2 = (SELECT TOP 1 [Address] FROM (SELECT TOP 2 [Address] FROM [UserAddress] ua INNER JOIN UserAddressOrder uo ON ua.UserID = uo.UserID WHERE ua.UserID = u.UserID ORDER BY uo.AddressOrder ASC) q ORDER BY AddressOrder DESC) FROM User u In this scenario, users have multiple address definitions, with an integer field specifying the preferred order. "Address2" (the second preferred address) attempts to take the top two preferred addresses, order them descending, then take the top one from the result. You might say, just use a subquery which does a SELECT for the record with "2" in the Order field, but the Order values are not contiguous. How can this be rewritten to conform to SQL 2000's limitations? Very much appreciated.

    Read the article

  • Java: immutability, overuse of stack -- better data structure?

    - by HH
    I overused hashSets but it was slow, then changed to Stacks, speed boost-up. Poly's reply uses Collections.emptyList() as immutable list, cutting out excess null-checkers. No Collections.emptyStack(). Combining the words stack and immutability, from the last experiences, gets "immutable stack" (probably not related to functional prog). Java Api 5 for list interface shows that Stack is an implementing class for list and arraylist, here. The java.coccurrent pkg does not have any immutable Stack data structure. The first hinted of misusing stack. The lack of immutabily in the last and poly's book recommendation leads way to list. Something very primitive, fast, no extra layers, with methods like emptyThing(). Overuse of stack and where I use it DataFile.java: public Stack<DataFile> files; FileObject.java: public Stack<String> printViews = new Stack<String>(); FileObject.java:// private static Stack<Object> getFormat(File f){return (new Format(f)).getFormat();} Format.java: private Stack<Object> getLine(File[] fs,String s){return wF;} Format.java: private Stack<Object> getFormat(){return format;} Positions.java: public static Stack<Integer[]> getPrintPoss(String s,File f,Integer maxViewPerF) Positions.java: Stack<File> possPrint = new Stack<File>(); Positions.java: Stack<Integer> positions=new Stack<Integer>(); Record.java: private String getFormatLine(Stack<Object> st) Record.java: Stack<String> lines=new Stack<String>(); SearchToUser.java: public static final Stack<File> allFiles = findf.getFs(); SearchToUser.java: public static final Stack<File> allDirs = findf.getDs(); SearchToUser.java: private Stack<Integer[]> positionsPrint=new Stack<Integer[]>(); SearchToUser.java: public Stack<String> getSearchResults(String s, Integer countPerFile, Integer resCount) SearchToUser.java: Stack<File> filesToS=Fs2Word.getFs2W(s,50); SearchToUser.java: Stack<String> rs=new Stack<String>(); View.java: public Stack<Integer[]> poss = new Stack<Integer[4]>(); View.java: public static Stack<String> getPrintViewsFileWise(String s,Object[] df,Integer maxViewsPerF) View.java: Stack<String> substrings = new Stack<String>(); View.java: private Stack<String> printViews=new Stack<String>(); View.java: MatchView(Stack<Integer> pss,File f,Integer maxViews) View.java: Stack<String> formatFile; View.java: private Stack<Search> files; View.java: private Stack<File> matchingFiles; View.java: private Stack<String> matchViews; View.java: private Stack<String> searchMatches; View.java: private Stack<String> getSearchResults(Integer numbResults) Easier with List: AllDirs and AllFs, now looping with push, but list has more pow. methods such as addAll [OLD] From Stack to some immutable data structure How to get immutable Stack data structure? Can I box it with list? Should I switch my current implementatios from stacks to Lists to get immutable? Which immutable data structure is Very fast with about similar exec time as Stack? No immutability to Stack with Final import java.io.*; import java.util.*; public class TestStack{ public static void main(String[] args) { final Stack<Integer> test = new Stack<Integer>(); Stack<Integer> test2 = new Stack<Integer>(); test.push(37707); test2.push(80437707); //WHY is there not an error to remove an elment // from FINAL stack? System.out.println(test.pop()); System.out.println(test2.pop()); } }

    Read the article

  • SQL Server 2005: Improving performance for thousands or Insert requests. logout-login time= 120ms.

    - by Rad
    Can somebody shed some lights on how SQL Server 2005 deals with may request issued by a client using ADO.NET 2.0. Below is the shortend output of SQL Trace. I can see that connection pooling is working (I believe there is only one connection being pooled). What is not clear to me is why we have so many sp_reset_connection calls i.e a series of: Audit Login, SQL:BatchStarting, RPC:Starting and Audit Logout for each loop in for loop below. I can see that there is constant switching between tempdb and master database which leads me to conclude that we lost the context when next connection is created by fetching it from the pool based on ConectionString argument. I can see that every 15ms I can get 100-200 login/logout per second (reported at the same time by Profiler). The after 15ms I have again a series fo 100-200 login/logout per second. I need clarification on how this might affect much complex insert queries in production environment. I use Enterprise Library 2006, the code is compiled with VS 2005 and it is a console application that parses a flat file with 10 of thousand of rows grouping parent-child rows, runs on an application server and runs 2 stored procedure on a remote SQL Server 2005 inserting a parent record, retrieves Identity value and using it calls the second stored procedure 1, 2 or multiple times (sometimes several thousands) inserting child records. The child table has close to 10 million records with 5-10 indexes some of them being covering non-clustered. There is a pretty complex Insert trigger that copies inserted detail record to an archive table. All in all I only have 7 inserts per second which means it can take 2-4 hours for 50 thousand records. When I run Profiler on the test server (that is almost equivalent with production server) I can see that there is about 120ms between Audit Logout and Audit Login trace entries which almost give me chance to insert about 8 records. So my question is if there is some way to improve inserting of records since the company loads 100 thousands of records and does daily planning and has SLA to fulfill client request coming as flat file orders and some big files 10 thousands have to be processed(imported quickly). 4 hours to import 60 thousands should be reduced to 30 minutes. I was thinking to use BatchSize of DataAdapter to send multiple stored procedure calls, SQL Bulk inserts to batch multiple inserts from DataReader or DataTable, SSIS fast load. But I don't know how to properly analyze re-indexing and stats population and maybe this has to take some time to finish. What is worse is that the company uses the biggest table for reporting and other online processing and indexes cannot be dropped. I manage transaction manually by setting a field to a value and do an transactional update changing that value to a new value that other applications are using to get committed rows. Please advise how to approach this problem. For now I am trying to have a staging tables with minimal logging in a separate database and no indexes and I will try to do batched (massive) parent child inserts. I believe Production DB has simple recovery model, but it could be full recovery. If DB user that is being used by my .NET console application has bulkadmin role does it mean its bulk inserts are minimally logged. I understand that when a table has clustered and many non-clustered indexes that inserts are still logged for each row. Connection pooling is working, but with many login/logouts. Why? for (int i = 1; i <= 10000; i++){ using (SqlConnection conn = new SqlConnection("server=(local);database=master;integrated security=sspi;")) {conn.Open(); using (SqlCommand cmd = conn.CreateCommand()){ cmd.CommandText = "use tempdb"; cmd.ExecuteNonQuery();}}} SQL Server Profiler trace: Audit Login master 2010-01-13 23:18:45.337 1 - Nonpooled SQL:BatchStarting use tempdb master 2010-01-13 23:18:45.337 RPC:Starting exec sp_reset_conn tempdb 2010-01-13 23:18:45.337 Audit Logout tempdb 2010-01-13 23:18:45.337 2 - Pooled Audit Login -- network protocol master 2010-01-13 23:18:45.383 2 - Pooled SQL:BatchStarting use tempdb master 2010-01-13 23:18:45.383 RPC:Starting exec sp_reset_conn tempdb 2010-01-13 23:18:45.383 Audit Logout tempdb 2010-01-13 23:18:45.383 2 - Pooled Audit Login -- network protocol master 2010-01-13 23:18:45.383 2 - Pooled SQL:BatchStarting use tempdb master 2010-01-13 23:18:45.383 RPC:Starting exec sp_reset_conn tempdb 2010-01-13 23:18:45.383 Audit Logout tempdb 2010-01-13 23:18:45.383 2 - Pooled

    Read the article

  • Trying to suppress StyleCop message SA1513:ClosingCurlyBracketMustBeFollowedByBlankLine

    - by mattruma
    I am trying to suppress the following StyleCop message for a specific property: SA1513: Statements or elements wrapped in curly brackets must be followed by a blank line. I am trying to do the following, but it doesn't seem to work: [SuppressMessage("Microsoft.StyleCop.CSharp.DocumentationRules", "SA1513:ClosingCurlyBracketMustBeFollowedByBlankLine", Justification = "There are no issues with this code")] public string CustomerId { get { return this.GetProperty(CustomerIdProperty); } set { if (this.IsNew) { this.SetProperty(CustomerIdProperty, value); } else { throw new ReadOnlyException("Id value can only be changed for a new record."); } } } Am I just doing something wrong? Or is this just not possible? It's a good rule, just not valid in my case for a property.

    Read the article

  • Entity Framework 4 ste delete foreign key relationship

    - by user169867
    I'm using EF4 and STE w/ Silverlight. I'm having trouble deleting child records from my primary entity. For some reason I can remove child entities if their foreign key to my primary entity is part of their Primary Key. But if it's not, they don't get removed. I believe these posts explains it: http://mocella.blogspot.com/2010/01/entity-framework-v4-object-graph.html http://blogs.msdn.com/dsimmons/archive/2010/01/31/deleting-foreign-key-relationships-in-ef4.aspx My question is how how do I remove a child record who's foreign key is not part of its primary key in Silverlight where I don't have access to a DeleteObject() function?

    Read the article

  • Visual studio 2010 remote debugging is very slow (across domains, over vpn)

    - by alex
    Overall debugging works, but each step through code takes dosens of seconds. I've already closed all additional windows like stack trace,watches,autos; deleted all breakpoints. server and dev machine are located in different domains, so i set up local user on both, with matching password. remote debugger is running as service. looking at security log, I found quite a lot of entries about remote debugging account logging in (record about every minute). Any suggestions on how i can speed up remote debugging? dev computer: quad core, 8 Gb mem, win 7 x64 , visual studio 2010 ultimate target server: asp.net website , 2xdual core xeon, 2Gb mem, remote debugger 2010 communication channel: vpn , 5 mbit , latency about 20ms. (seems that debbugging never uses more than 20 kb/s)

    Read the article

  • How do I insert a row with a TimeUUIDType column in Cassandra?

    - by mixmasteralan
    In Cassandra, I have the following Column Family: <ColumnFamily CompareWith="TimeUUIDType" Name="Posts"/> I'm trying to insert a record into it as follows using a C++ generated function generated by Thrift: ColumnPath new_col; new_col.__isset.column = true; /* this is required! */ new_col.column_family.assign("Posts"); new_col.super_column.assign(""); new_col.column.assign("1968ec4a-2a73-11df-9aca-00012e27a270"); client.insert("Keyspace1", "somekey", new_col, "Random Value", 1234, ONE); However, I'm getting the following error: "UUIDs must be exactly 16 bytes" I've even tried the Cassandra CLI with the following command: set Keyspace1.Posts['somekey']['1968ec4a-2a73-11df-9aca-00012e27a270'] = 'Random Value' but I still get the following error: Exception null InvalidRequestException(why:UUIDs must be exactly 16 bytes) at org.apache.cassandra.thrift.Cassandra$insert_result.read(Cassandra.java:11994) at org.apache.cassandra.thrift.Cassandra$Client.recv_insert(Cassandra.java:659) at org.apache.cassandra.thrift.Cassandra$Client.insert(Cassandra.java:632) at org.apache.cassandra.cli.CliClient.executeSet(CliClient.java:420) at org.apache.cassandra.cli.CliClient.executeCLIStmt(CliClient.java:80) at org.apache.cassandra.cli.CliMain.processCLIStmt(CliMain.java:132) at org.apache.cassandra.cli.CliMain.main(CliMain.java:173)

    Read the article

  • filter data between 2 dates in crystal report vb.net

    - by irienaoki0407
    I need some help for creating Crystal Reports in VB 2005. I want to filter data between two dates (like from date and to date) with datetimepicker. I'm using SQL Server 2000 for the connection. Update: Thanks for the link, but I'm trying using record selection formula.... Here's my code: Try Dim cryRpt As New ReportDocument With cryRpt .FileName = ("C:\Documents and Settings\Ratna Ayu\My Documents\Visual Studio 2005\Projects\Denda\Denda\CrystalReport1.rpt") .RecordSelectionFormula = "{pinjam.tglkembali}>='" & DateTimePicker1.Value.ToString("dd/MM/yyyy") & "' and {pinjam.tglkembali} =<'" & DateTimePicker2.Value.ToString("dd/MM/yyyy") & "'" End With CrystalReportViewer1.ReportSource = cryRpt CrystalReportViewer1.Refresh() Catch ex As Exception MsgBox("tdk ada data", , "") End Try

    Read the article

  • Fancy dynamic list in Android: TableLayout vs ListView

    - by Ralkie
    There is a requirement to have not-so-trivial dynamic list, each record of which consists of several columns (texts, buttons). It should look something like: Text11 Text12 Button1 Button2 Text21 Text22 Button1 Button2 ... At first obvious way to accomplish that seemed to be TableLayout. I was expecting to have layout/styling data specified in res/layout/*.xml and to populate it with some dataset from java code (as with ListView, for which its possible to specify TextView of item in *.xml and bind it to some array using ArrayAdapter). But after playing for a while, all I found to be possible is fully populating TableLayout programatically. Still, creating TableRow by TableRow and setting layout attributes directly in java code doesn't seem elegant enough. So the questions are: Am I at the right path? Is TableLayout really best View to accomplish that? Maybe it's more appropriate to extend ListView or something else to meet such requirements? Is it possible to have TableLayout/TableRow template specified in *.xml and data bound to this template in java side?

    Read the article

facebook twitter digg google delicious technorati stumbleupon myspace wordpress linkedin gmail igoogle windows live tumblr viadeo yahoo buzz yahoo mail yahoo bookmarks favorites email print

< Previous Page | 328 329 330 331 332 333 334 335 336 337 338 339  | Next Page >