chum of chance - Page 85

Red Gate Coder interviews: Alex Davies

- by Michael Williamson

Alex Davies has been a software engineer at Red Gate since graduating from university, and is currently busy working on .NET Demon. We talked about tackling parallel programming with his actors framework, a scientific approach to debugging, and how JavaScript is going to affect the programming languages we use in years to come. So, if we start at the start, how did you get started in programming? When I was seven or eight, I was given a BBC Micro for Christmas. I had asked for a Game Boy, but my dad thought it would be better to give me a proper computer. For a year or so, I only played games on it, but then I found the user guide for writing programs in it. I gradually started doing more stuff on it and found it fun. I liked creating. As I went into senior school I continued to write stuff on there, trying to write games that weren’t very good. I got a real computer when I was fourteen and found ways to write BASIC on it. Visual Basic to start with, and then something more interesting than that. How did you learn to program? Was there someone helping you out? Absolutely not! I learnt out of a book, or by experimenting. I remember the first time I found a loop, I was like “Oh my God! I don’t have to write out the same line over and over and over again any more. It’s amazing!” When did you think this might be something that you actually wanted to do as a career? For a long time, I thought it wasn’t something that you would do as a career, because it was too much fun to be a career. I thought I’d do chemistry at university and some kind of career based on chemical engineering. And then I went to a careers fair at school when I was seventeen or eighteen, and it just didn’t interest me whatsoever. I thought “I could be a programmer, and there’s loads of money there, and I’m good at it, and it’s fun”, but also that I shouldn’t spoil my hobby. Now I don’t really program in my spare time any more, which is a bit of a shame, but I program all the rest of the time, so I can live with it. Do you think you learnt much about programming at university? Yes, definitely! I went into university knowing how to make computers do anything I wanted them to do. However, I didn’t have the language to talk about algorithms, so the algorithms course in my first year was massively important. Learning other language paradigms like functional programming was really good for breadth of understanding. Functional programming influences normal programming through design rather than actually using it all the time. I draw inspiration from it to write imperative programs which I think is actually becoming really fashionable now, but I’ve been doing it for ages. I did it first! There were also some courses on really odd programming languages, a bit of Prolog, a little bit of C. Having a little bit of each of those is something that I would have never done on my own, so it was important. And then there are knowledge-based courses which are about not programming itself but things that have been programmed like TCP. Those are really important for examples for how to approach things. Did you do any internships while you were at university? Yeah, I spent both of my summers at the same company. I thought I could code well before I went there. Looking back at the crap that I produced, it was only surpassed in its crappiness by all of the other code already in that company. I’m so much better at writing nice code now than I used to be back then. Was there just not a culture of looking after your code? There was, they just didn’t hire people for their abilities in that area. They hired people for raw IQ. The first indicator of it going wrong was that they didn’t have any computer scientists, which is a bit odd in a programming company. But even beyond that they didn’t have people who learnt architecture from anyone else. Most of them had started straight out of university, so never really had experience or mentors to learn from. There wasn’t the experience to draw from to teach each other. In the second half of my second internship, I was being given tasks like looking at new technologies and teaching people stuff. Interns shouldn’t be teaching people how to do their jobs! All interns are going to have little nuggets of things that you don’t know about, but they shouldn’t consistently be the ones who know the most. It’s not a good environment to learn. I was going to ask how you found working with people who were more experienced than you… When I reached Red Gate, I found some people who were more experienced programmers than me, and that was difficult. I’ve been coding since I was tiny. At university there were people who were cleverer than me, but there weren’t very many who were more experienced programmers than me. During my internship, I didn’t find anyone who I classed as being a noticeably more experienced programmer than me. So, it was a shock to the system to have valid criticisms rather than just formatting criticisms. However, Red Gate’s not so big on the actual code review, at least it wasn’t when I started. We did an entire product release and then somebody looked over all of the UI of that product which I’d written and say what they didn’t like. By that point, it was way too late and I’d disagree with them. Do you think the lack of code reviews was a bad thing? I think if there’s going to be any oversight of new people, then it should be continuous rather than chunky. For me I don’t mind too much, I could go out and get oversight if I wanted it, and in those situations I felt comfortable without it. If I was managing the new person, then maybe I’d be keener on oversight and then the right way to do it is continuously and in very, very small chunks. Have you had any significant projects you’ve worked on outside of a job? When I was a teenager I wrote all sorts of stuff. I used to write games, I derived how to do isomorphic projections myself once. I didn’t know what the word was so I couldn’t Google for it, so I worked it out myself. It was horrifically complicated. But it sort of tailed off when I started at university, and is now basically zero. If I do side-projects now, they tend to be work-related side projects like my actors framework, NAct, which I started in a down tools week. Could you explain a little more about NAct? It is a little C# framework for writing parallel code more easily. Parallel programming is difficult when you need to write to shared data. Sometimes parallel programming is easy because you don’t need to write to shared data. When you do need to access shared data, you could just have your threads pile in and do their work, but then you would screw up the data because the threads would trample on each other’s toes. You could lock, but locks are really dangerous if you’re using more than one of them. You get interactions like deadlocks, and that’s just nasty. Actors instead allows you to say this piece of data belongs to this thread of execution, and nobody else can read it. If you want to read it, then ask that thread of execution for a piece of it by sending a message, and it will send the data back by a message. And that avoids deadlocks as long as you follow some obvious rules about not making your actors sit around waiting for other actors to do something. There are lots of ways to write actors, NAct allows you to do it as if it was method calls on other objects, which means you get all the strong type-safety that C# programmers like. Do you think that this is suitable for the majority of parallel programming, or do you think it’s only suitable for specific cases? It’s suitable for most difficult parallel programming. If you’ve just got a hundred web requests which are all independent of each other, then I wouldn’t bother because it’s easier to just spin them up in separate threads and they can proceed independently of each other. But where you’ve got difficult parallel programming, where you’ve got multiple threads accessing multiple bits of data in multiple ways at different times, then actors is at least as good as all other ways, and is, I reckon, easier to think about. When you’re using actors, you presumably still have to write your code in a different way from you would otherwise using single-threaded code. You can’t use actors with any methods that have return types, because you’re not allowed to call into another actor and wait for it. If you want to get a piece of data out of another actor, then you’ve got to use tasks so that you can use “async” and “await” to await asynchronously for it. But other than that, you can still stick things in classes so it’s not too different really. Rather than having thousands of objects with mutable state, you can use component-orientated design, where there are only a few mutable classes which each have a small number of instances. Then there can be thousands of immutable objects. If you tend to do that anyway, then actors isn’t much of a jump. If I’ve already built my system without any parallelism, how hard is it to add actors to exploit all eight cores on my desktop? Usually pretty easy. If you can identify even one boundary where things look like messages and you have components where some objects live on one side and these other objects live on the other side, then you can have a granddaddy object on one side be an actor and it will parallelise as it goes across that boundary. Not too difficult. If we do get 1000-core desktop PCs, do you think actors will scale up? It’s hard. There are always in the order of twenty to fifty actors in my whole program because I tend to write each component as actors, and I tend to have one instance of each component. So this won’t scale to a thousand cores. What you can do is write data structures out of actors. I use dictionaries all over the place, and if you need a dictionary that is going to be accessed concurrently, then you could build one of those out of actors in no time. You can use queuing to marshal requests between different slices of the dictionary which are living on different threads. So it’s like a distributed hash table but all of the chunks of it are on the same machine. That means that each of these thousand processors has cached one small piece of the dictionary. I reckon it wouldn’t be too big a leap to start doing proper parallelism. Do you think it helps if actors get baked into the language, similarly to Erlang? Erlang is excellent in that it has thread-local garbage collection. C# doesn’t, so there’s a limit to how well C# actors can possibly scale because there’s a single garbage collected heap shared between all of them. When you do a global garbage collection, you’ve got to stop all of the actors, which is seriously expensive, whereas in Erlang garbage collections happen per-actor, so they’re insanely cheap. However, Erlang deviated from all the sensible language design that people have used recently and has just come up with crazy stuff. You can definitely retrofit thread-local garbage collection to .NET, and then it’s quite well-suited to support actors, even if it’s not baked into the language. Speaking of language design, do you have a favourite programming language? I’ll choose a language which I’ve never written before. I like the idea of Scala. It sounds like C#, only with some of the niggles gone. I enjoy writing static types. It means you don’t have to writing tests so much. When you say it doesn’t have some of the niggles? C# doesn’t allow the use of a property as a method group. It doesn’t have Scala case classes, or sum types, where you can do a switch statement and the compiler checks that you’ve checked all the cases, which is really useful in functional-style programming. Pattern-matching, in other words. That’s actually the major niggle. C# is pretty good, and I’m quite happy with C#. And what about going even further with the type system to remove the need for tests to something like Haskell? Or is that a step too far? I’m quite a pragmatist, I don’t think I could deal with trying to write big systems in languages with too few other users, especially when learning how to structure things. I just don’t know anyone who can teach me, and the Internet won’t teach me. That’s the main reason I wouldn’t use it. If I turned up at a company that writes big systems in Haskell, I would have no objection to that, but I wouldn’t instigate it. What about things in C#? For instance, there’s contracts in C#, so you can try to statically verify a bit more about your code. Do you think that’s useful, or just not worthwhile? I’ve not really tried it. My hunch is that it needs to be built into the language and be quite mathematical for it to work in real life, and that doesn’t seem to have ended up true for C# contracts. I don’t think anyone who’s tried them thinks they’re any good. I might be wrong. On a slightly different note, how do you like to debug code? I think I’m quite an odd debugger. I use guesswork extremely rarely, especially if something seems quite difficult to debug. I’ve been bitten spending hours and hours on guesswork and not being scientific about debugging in the past, so now I’m scientific to a fault. What I want is to see the bug happening in the debugger, to step through the bug happening. To watch the program going from a valid state to an invalid state. When there’s a bug and I can’t work out why it’s happening, I try to find some piece of evidence which places the bug in one section of the code. From that experiment, I binary chop on the possible causes of the bug. I suppose that means binary chopping on places in the code, or binary chopping on a stage through a processing cycle. Basically, I’m very stupid about how I debug. I won’t make any guesses, I won’t use any intuition, I will only identify the experiment that’s going to binary chop most effectively and repeat rather than trying to guess anything. I suppose it’s quite top-down. Is most of the time then spent in the debugger? Absolutely, if at all possible I will never debug using print statements or logs. I don’t really hold much stock in outputting logs. If there’s any bug which can be reproduced locally, I’d rather do it in the debugger than outputting logs. And with SmartAssembly error reporting, there’s not a lot that can’t be either observed in an error report and just fixed, or reproduced locally. And in those other situations, maybe I’ll use logs. But I hate using logs. You stare at the log, trying to guess what’s going on, and that’s exactly what I don’t like doing. You have to just look at it and see does this look right or wrong. We’ve covered how you get to grip with bugs. How do you get to grips with an entire codebase? I watch it in the debugger. I find little bugs and then try to fix them, and mostly do it by watching them in the debugger and gradually getting an understanding of how the code works using my process of binary chopping. I have to do a lot of reading and watching code to choose where my slicing-in-half experiment is going to be. The last time I did it was SmartAssembly. The old code was a complete mess, but at least it did things top to bottom. There wasn’t too much of some of the big abstractions where flow of control goes all over the place, into a base class and back again. Code’s really hard to understand when that happens. So I like to choose a little bug and try to fix it, and choose a bigger bug and try to fix it. Definitely learn by doing. I want to always have an aim so that I get a little achievement after every few hours of debugging. Once I’ve learnt the codebase I might be able to fix all the bugs in an hour, but I’d rather be using them as an aim while I’m learning the codebase. If I was a maintainer of a codebase, what should I do to make it as easy as possible for you to understand? Keep distinct concepts in different places. And name your stuff so that it’s obvious which concepts live there. You shouldn’t have some variable that gets set miles up the top of somewhere, and then is read miles down to choose some later behaviour. I’m talking from a very much SmartAssembly point of view because the old SmartAssembly codebase had tons and tons of these things, where it would read some property of the code and then deal with it later. Just thousands of variables in scope. Loads of things to think about. If you can keep concepts separate, then it aids me in my process of fixing bugs one at a time, because each bug is going to more or less be understandable in the one place where it is. And what about tests? Do you think they help at all? I’ve never had the opportunity to learn a codebase which has had tests, I don’t know what it’s like! What about when you’re actually developing? How useful do you find tests in finding bugs or regressions? Finding regressions, absolutely. Running bits of code that would be quite hard to run otherwise, definitely. It doesn’t happen very often that a test finds a bug in the first place. I don’t really buy nebulous promises like tests being a good way to think about the spec of the code. My thinking goes something like “This code works at the moment, great, ship it! Ah, there’s a way that this code doesn’t work. Okay, write a test, demonstrate that it doesn’t work, fix it, use the test to demonstrate that it’s now fixed, and keep the test for future regressions.” The most valuable tests are for bugs that have actually happened at some point, because bugs that have actually happened at some point, despite the fact that you think you’ve fixed them, are way more likely to appear again than new bugs are. Does that mean that when you write your code the first time, there are no tests? Often. The chance of there being a bug in a new feature is relatively unaffected by whether I’ve written a test for that new feature because I’m not good enough at writing tests to think of bugs that I would have written into the code. So not writing regression tests for all of your code hasn’t affected you too badly? There are different kinds of features. Some of them just always work, and are just not flaky, they just continue working whatever you throw at them. Maybe because the type-checker is particularly effective around them. Writing tests for those features which just tend to always work is a waste of time. And because it’s a waste of time I’ll tend to wait until a feature has demonstrated its flakiness by having bugs in it before I start trying to test it. You can get a feel for whether it’s going to be flaky code as you’re writing it. I try to write it to make it not flaky, but there are some things that are just inherently flaky. And very occasionally, I’ll think “this is going to be flaky” as I’m writing, and then maybe do a test, but not most of the time. How do you think your programming style has changed over time? I’ve got clearer about what the right way of doing things is. I used to flip-flop a lot between different ideas. Five years ago I came up with some really good ideas and some really terrible ideas. All of them seemed great when I thought of them, but they were quite diverse ideas, whereas now I have a smaller set of reliable ideas that are actually good for structuring code. So my code is probably more similar to itself than it used to be back in the day, when I was trying stuff out. I’ve got more disciplined about encapsulation, I think. There are operational things like I use actors more now than I used to, and that forces me to use immutability more than I used to. The first code that I wrote in Red Gate was the memory profiler UI, and that was an actor, I just didn’t know the name of it at the time. I don’t really use object-orientation. By object-orientation, I mean having n objects of the same type which are mutable. I want a constant number of objects that are mutable, and they should be different types. I stick stuff in dictionaries and then have one thing that owns the dictionary and puts stuff in and out of it. That’s definitely a pattern that I’ve seen recently. I think maybe I’m doing functional programming. Possibly. It’s plausible. If you had to summarise the essence of programming in a pithy sentence, how would you do it? Programming is the form of art that, without losing any of the beauty of architecture or fine art, allows you to produce things that people love and you make money from. So you think it’s an art rather than a science? It’s a little bit of engineering, a smidgeon of maths, but it’s not science. Like architecture, programming is on that boundary between art and engineering. If you want to do it really nicely, it’s mostly art. You can get away with doing architecture and programming entirely by having a good engineering mind, but you’re not going to produce anything nice. You’re not going to have joy doing it if you’re an engineering mind. Architects who are just engineering minds are not going to enjoy their job. I suppose engineering is the foundation on which you build the art. Exactly. How do you think programming is going to change over the next ten years? There will be an unfortunate shift towards dynamically-typed languages, because of JavaScript. JavaScript has an unfair advantage. JavaScript’s unfair advantage will cause more people to be exposed to dynamically-typed languages, which means other dynamically-typed languages crop up and the best features go into dynamically-typed languages. Then people conflate the good features with the fact that it’s dynamically-typed, and more investment goes into dynamically-typed languages. They end up better, so people use them. What about the idea of compiling other languages, possibly statically-typed, to JavaScript? It’s a reasonable idea. I would like to do it, but I don’t think enough people in the world are going to do it to make it pick up. The hordes of beginners are the lifeblood of a language community. They are what makes there be good tools and what makes there be vibrant community websites. And any particular thing which is the same as JavaScript only with extra stuff added to it, although it might be technically great, is not going to have the hordes of beginners. JavaScript is always to be quickest and easiest way for a beginner to start programming in the browser. And dynamically-typed languages are great for beginners. Compilers are pretty scary and beginners don’t write big code. And having your errors come up in the same place, whether they’re statically checkable errors or not, is quite nice for a beginner. If someone asked me to teach them some programming, I’d teach them JavaScript. If dynamically-typed languages are great for beginners, when do you think the benefits of static typing start to kick in? The value of having a statically typed program is in the tools that rely on the static types to produce a smooth IDE experience rather than actually telling me my compile errors. And only once you’re experienced enough a programmer that having a really smooth IDE experience makes a blind bit of difference, does static typing make a blind bit of difference. So it’s not really about size of codebase. If I go and write up a tiny program, I’m still going to get value out of writing it in C# using ReSharper because I’m experienced with C# and ReSharper enough to be able to write code five times faster if I have that help. Any other visions of the future? Nobody’s going to use actors. Because everyone’s going to be running on single-core VMs connected over network-ready protocols like JSON over HTTP. So, parallelism within one operating system is going to die. But until then, you should use actors. More Red Gater Coder interviews

Read the article

FreeBSD performance tuning. Sysctls, loader.conf, kernel

- by SaveTheRbtz

I wanted to share knowledge of tuning FreeBSD via sysctl.conf/loader.conf/KENCONF. It was initially based on Igor Sysoev's (author of nginx) presentation about FreeBSD tuning up to 100,000-200,000 active connections. Tunings are for FreeBSD-CURRENT. Since 7.2 amd64 some of them are tuned well by default. Prior 7.0 some of them are boot only (set via /boot/loader.conf) or does not exist at all. sysctl.conf: # No zero mapping feature # May break wine # (There are also reports about broken samba3) #security.bsd.map_at_zero=0 # If you have really busy webserver with apache13 you may run out of processes #kern.maxproc=10000 # Same for servers with apache2 / Pound #kern.threads.max_threads_per_proc=4096 # Max. backlog size kern.ipc.somaxconn=4096 # Shared memory // 7.2+ can use shared memory > 2Gb kern.ipc.shmmax=2147483648 # Sockets kern.ipc.maxsockets=204800 # Can cause this on older kernels: # http://old.nabble.com/Significant-performance-regression-for-increased-maxsockbuf-on-8.0-RELEASE-tt26745981.html#a26745981 ) kern.ipc.maxsockbuf=10485760 # Mbuf 2k clusters (on amd64 7.2+ 25600 is default) # For such high value vm.kmem_size must be increased to 3G kern.ipc.nmbclusters=262144 # Jumbo pagesize(_SC_PAGESIZE) clusters # Used as general packet storage for jumbo frames # can be monitored via `netstat -m` #kern.ipc.nmbjumbop=262144 # Jumbo 9k/16k clusters # If you are using them #kern.ipc.nmbjumbo9=65536 #kern.ipc.nmbjumbo16=32768 # For lower latency you can decrease scheduler's maximum time slice # default: stathz/10 (~ 13) #kern.sched.slice=1 # Increase max command-line length showed in `ps` (e.g for Tomcat/Java) # Default is PAGE_SIZE / 16 or 256 on x86 # This avoids commands to be presented as [executable] in `ps` # For more info see: http://www.freebsd.org/cgi/query-pr.cgi?pr=120749 kern.ps_arg_cache_limit=4096 # Every socket is a file, so increase them kern.maxfiles=204800 kern.maxfilesperproc=200000 kern.maxvnodes=200000 # On some systems HPET is almost 2 times faster than default ACPI-fast # Useful on systems with lots of clock_gettime / gettimeofday calls # See http://old.nabble.com/ACPI-fast-default-timecounter,-but-HPET-83--faster-td23248172.html # After revision 222222 HPET became default: http://svnweb.freebsd.org/base?view=revision&revision=222222 kern.timecounter.hardware=HPET # Small receive space, only usable on http-server, on file server this # should be increased to 65535 or even more #net.inet.tcp.recvspace=8192 # This is useful on Fat-Long-Pipes #net.inet.tcp.recvbuf_max=10485760 #net.inet.tcp.recvbuf_inc=65535 # Small send space is useful for http servers that serve small files # Autotuned since 7.x net.inet.tcp.sendspace=16384 # This is useful on Fat-Long-Pipes #net.inet.tcp.sendbuf_max=10485760 #net.inet.tcp.sendbuf_inc=65535 # Turn off receive autotuning # You can play with it. #net.inet.tcp.recvbuf_auto=0 #net.inet.tcp.sendbuf_auto=0 # This should be enabled if you going to use big spaces (>64k) # Also timestamp field is useful when using syncookies net.inet.tcp.rfc1323=1 # Turn this off on high-speed, lossless connections (LAN 1Gbit+) # If you set it there is no need in TCP_NODELAY sockopt (see man tcp) net.inet.tcp.delayed_ack=0 # This feature is useful if you are serving data over modems, Gigabit Ethernet, # or even high speed WAN links (or any other link with a high bandwidth delay product), # especially if you are also using window scaling or have configured a large send window. # Automatically disables on small RTT ( http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/netinet/tcp_subr.c?#rev1.237 ) # This sysctl was removed in 10-CURRENT: # See: http://www.mail-archive.com/[email protected]/msg06178.html #net.inet.tcp.inflight.enable=0 # TCP slowstart algorithm tunings # We assuming we have very fast clients #net.inet.tcp.slowstart_flightsize=100 #net.inet.tcp.local_slowstart_flightsize=100 # Disable randomizing of ports to avoid false RST # Before usage check SA here www.bsdcan.org/2006/papers/ImprovingTCPIP.pdf # (it's also says that port randomization auto-disables at some conn.rates, but I didn't checked it thou) #net.inet.ip.portrange.randomized=0 # Increase portrange # For outgoing connections only. Good for seed-boxes and ftp servers. net.inet.ip.portrange.first=1024 net.inet.ip.portrange.last=65535 # # stops route cache degregation during a high-bandwidth flood # http://www.freebsd.org/doc/en/books/handbook/securing-freebsd.html #net.inet.ip.rtexpire=2 net.inet.ip.rtminexpire=2 net.inet.ip.rtmaxcache=1024 # Security net.inet.ip.redirect=0 net.inet.ip.sourceroute=0 net.inet.ip.accept_sourceroute=0 net.inet.icmp.maskrepl=0 net.inet.icmp.log_redirect=0 net.inet.icmp.drop_redirect=1 net.inet.tcp.drop_synfin=1 # # There is also good example of sysctl.conf with comments: # http://www.thern.org/projects/sysctl.conf # # icmp may NOT rst, helpful for those pesky spoofed # icmp/udp floods that end up taking up your outgoing # bandwidth/ifqueue due to all that outgoing RST traffic. # #net.inet.tcp.icmp_may_rst=0 # Security net.inet.udp.blackhole=1 net.inet.tcp.blackhole=2 # IPv6 Security # For more info see http://www.fosslc.org/drupal/content/security-implications-ipv6 # Disable Node info replies # To see this vulnerability in action run `ping6 -a sglAac ::1` or `ping6 -w ::1` on unprotected node net.inet6.icmp6.nodeinfo=0 # Turn on IPv6 privacy extensions # For more info see proposal http://unix.derkeiler.com/Mailing-Lists/FreeBSD/net/2008-06/msg00103.html net.inet6.ip6.use_tempaddr=1 net.inet6.ip6.prefer_tempaddr=1 # Disable ICMP redirect net.inet6.icmp6.rediraccept=0 # Disable acceptation of RA and auto linklocal generation if you don't use them #net.inet6.ip6.accept_rtadv=0 #net.inet6.ip6.auto_linklocal=0 # Increases default TTL, sometimes useful # Default is 64 net.inet.ip.ttl=128 # Lessen max segment life to conserve resources # ACK waiting time in miliseconds # (default: 30000. RFC from 1979 recommends 120000) net.inet.tcp.msl=5000 # Max bumber of timewait sockets net.inet.tcp.maxtcptw=200000 # Don't use tw on local connections # As of 15 Apr 2009. Igor Sysoev says that nolocaltimewait has some buggy realization. # So disable it or now till get fixed #net.inet.tcp.nolocaltimewait=1 # FIN_WAIT_2 state fast recycle net.inet.tcp.fast_finwait2_recycle=1 # Time before tcp keepalive probe is sent # default is 2 hours (7200000) #net.inet.tcp.keepidle=60000 # Should be increased until net.inet.ip.intr_queue_drops is zero net.inet.ip.intr_queue_maxlen=4096 # Interrupt handling via multiple CPU, but with context switch. # You can play with it. Default is 1; #net.isr.direct=0 # This is for routers only #net.inet.ip.forwarding=1 #net.inet.ip.fastforwarding=1 # This speed ups dummynet when channel isn't saturated net.inet.ip.dummynet.io_fast=1 # Increase dummynet(4) hash #net.inet.ip.dummynet.hash_size=2048 #net.inet.ip.dummynet.max_chain_len # Should be increased when you have A LOT of files on server # (Increase until vfs.ufs.dirhash_mem becomes lower) vfs.ufs.dirhash_maxmem=67108864 # Note from commit http://svn.freebsd.org/base/head@211031 : # For systems with RAID volumes and/or virtualization envirnments, where # read performance is very important, increasing this sysctl tunable to 32 # or even more will demonstratively yield additional performance benefits. vfs.read_max=32 # Explicit Congestion Notification (see http://en.wikipedia.org/wiki/Explicit_Congestion_Notification) net.inet.tcp.ecn.enable=1 # Flowtable - flow caching mechanism # Useful for routers #net.inet.flowtable.enable=1 #net.inet.flowtable.nmbflows=65535 # Extreme polling tuning #kern.polling.burst_max=1000 #kern.polling.each_burst=1000 #kern.polling.reg_frac=100 #kern.polling.user_frac=1 #kern.polling.idle_poll=0 # IPFW dynamic rules and timeouts tuning # Increase dyn_buckets till net.inet.ip.fw.curr_dyn_buckets is lower net.inet.ip.fw.dyn_buckets=65536 net.inet.ip.fw.dyn_max=65536 net.inet.ip.fw.dyn_ack_lifetime=120 net.inet.ip.fw.dyn_syn_lifetime=10 net.inet.ip.fw.dyn_fin_lifetime=2 net.inet.ip.fw.dyn_short_lifetime=10 # Make packets pass firewall only once when using dummynet # i.e. packets going thru pipe are passing out from firewall with accept #net.inet.ip.fw.one_pass=1 # shm_use_phys Wires all shared pages, making them unswappable # Use this to lessen Virtual Memory Manager's work when using Shared Mem. # Useful for databases #kern.ipc.shm_use_phys=1 # ZFS # Enable prefetch. Useful for sequential load type i.e fileserver. # FreeBSD sets vfs.zfs.prefetch_disable to 1 on any i386 systems and # on any amd64 systems with less than 4GB of avaiable memory # For additional info check this nabble thread http://old.nabble.com/Samba-read-speed-performance-tuning-td27964534.html #vfs.zfs.prefetch_disable=0 # On highload servers you may notice following message in dmesg: # "Approaching the limit on PV entries, consider increasing either the # vm.pmap.shpgperproc or the vm.pmap.pv_entry_max tunable" vm.pmap.shpgperproc=2048 loader.conf: # Accept filters for data, http and DNS requests # Useful when your software uses select() instead of kevent/kqueue or when you under DDoS # DNS accf available on 8.0+ accf_data_load="YES" accf_http_load="YES" accf_dns_load="YES" # Async IO system calls aio_load="YES" # Linux specific devices in /dev # As for 8.1 it only /dev/full #lindev_load="YES" # Adds NCQ support in FreeBSD # WARNING! all ad[0-9]+ devices will be renamed to ada[0-9]+ # 8.0+ only #ahci_load="YES" #siis_load="YES" # FreeBSD 8.2+ # New Congestion Control for FreeBSD # http://caia.swin.edu.au/urp/newtcp/tools/cc_chd-readme-0.1.txt # http://www.ietf.org/proceedings/78/slides/iccrg-5.pdf # Initial merge commit message http://www.mail-archive.com/[email protected]/msg31410.html #cc_chd_load="YES" # Increase kernel memory size to 3G. # # Use ONLY if you have KVA_PAGES in kernel configuration, and you have more than 3G RAM # Otherwise panic will happen on next reboot! # # It's required for high buffer sizes: kern.ipc.nmbjumbop, kern.ipc.nmbclusters, etc # Useful on highload stateful firewalls, proxies or ZFS fileservers # (FreeBSD 7.2+ amd64 users: Check that current value is lower!) #vm.kmem_size="3G" # If your server has lots of swap (>4Gb) you should increase following value # according to http://lists.freebsd.org/pipermail/freebsd-hackers/2009-October/029616.html # Otherwise you'll be getting errors # "kernel: swap zone exhausted, increase kern.maxswzone" # kern.maxswzone="256M" # Older versions of FreeBSD can't tune maxfiles on the fly #kern.maxfiles="200000" # Useful for databases # Sets maximum data size to 1G # (FreeBSD 7.2+ amd64 users: Check that current value is lower!) #kern.maxdsiz="1G" # Maximum buffer size(vfs.maxbufspace) # You can check current one via vfs.bufspace # Should be lowered/upped depending on server's load-type # Usually decreased to preserve kmem # (default is 10% of mem) #kern.maxbcache="512M" # Sendfile buffers # For i386 only #kern.ipc.nsfbufs=10240 # FreeBSD 9+ # HPET "legacy route" support. It should allow HPET to work per-CPU # See http://www.mail-archive.com/[email protected]/msg03603.html #hint.atrtc.0.clock=0 #hint.attimer.0.clock=0 #hint.hpet.0.legacy_route=1 # syncache Hash table tuning net.inet.tcp.syncache.hashsize=1024 net.inet.tcp.syncache.bucketlimit=512 net.inet.tcp.syncache.cachelimit=65536 # Increased hostcache # Later host cache can be viewed via net.inet.tcp.hostcache.list hidden sysctl # Very useful for it's RTT RTTVAR # Must be power of two net.inet.tcp.hostcache.hashsize=65536 # hashsize * bucketlimit (which is 30 by default) # It allocates 255Mb (1966080*136) of RAM net.inet.tcp.hostcache.cachelimit=1966080 # TCP control-block Hash table tuning net.inet.tcp.tcbhashsize=4096 # Disable ipfw deny all # Should be uncommented when there is a chance that # kernel and ipfw binary may be out-of sync on next reboot #net.inet.ip.fw.default_to_accept=1 # # SIFTR (Statistical Information For TCP Research) is a kernel module that # logs a range of statistics on active TCP connections to a log file. # See prerelease notes http://groups.google.com/group/mailing.freebsd.current/browse_thread/thread/b4c18be6cdce76e4 # and man 4 sitfr #siftr_load="YES" # Enable superpages, for 7.2+ only # Also read http://lists.freebsd.org/pipermail/freebsd-hackers/2009-November/030094.html vm.pmap.pg_ps_enabled=1 # Usefull if you are using Intel-Gigabit NIC #hw.em.rxd=4096 #hw.em.txd=4096 #hw.em.rx_process_limit="-1" # Also if you have ALOT interrupts on NIC - play with following parameters # NOTE: You should set them for every NIC #dev.em.0.rx_int_delay: 250 #dev.em.0.tx_int_delay: 250 #dev.em.0.rx_abs_int_delay: 250 #dev.em.0.tx_abs_int_delay: 250 # There is also multithreaded version of em/igb drivers can be found here: # http://people.yandex-team.ru/~wawa/ # # for additional em monitoring and statistics use # sysctl dev.em.0.stats=1 ; dmesg # sysctl dev.em.0.debug=1 ; dmesg # Also after r209242 (-CURRENT) there is a separate sysctl for each stat variable; # Same tunings for igb #hw.igb.rxd=4096 #hw.igb.txd=4096 #hw.igb.rx_process_limit=100 # Some useful netisr tunables. See sysctl net.isr #net.isr.maxthreads=4 #net.isr.defaultqlimit=4096 #net.isr.maxqlimit: 10240 # Bind netisr threads to CPUs #net.isr.bindthreads=1 # # FreeBSD 9.x+ # Increase interface send queue length # See commit message http://svn.freebsd.org/viewvc/base?view=revision&revision=207554 #net.link.ifqmaxlen=1024 # Nicer boot logo =) loader_logo="beastie" And finally here is KERNCONF: # Just some of them, see also # cat /sys/{i386,amd64,}/conf/NOTES # This one useful only on i386 #options KVA_PAGES=512 # You can play with HZ in environments with high interrupt rate (default is 1000) # 100 is for my notebook to prolong it's battery life #options HZ=100 # Polling is goot on network loads with high packet rates and low-end NICs # NB! Do not enable it if you want more than one netisr thread #options DEVICE_POLLING # Eliminate datacopy on socket read-write # To take advantage with zero copy sockets you should have an MTU >= 4k # This req. is only for receiving data. # Read more in man zero_copy_sockets # Also this epic thread on kernel trap: # http://kerneltrap.org/node/6506 # Here Linus says that "anybody that does it that way (FreeBSD) is totally incompetent" #options ZERO_COPY_SOCKETS # Support TCP sign. Used for IPSec options TCP_SIGNATURE # There was stackoverflow found in KAME IPSec stack: # See http://secunia.com/advisories/43995/ # For quick workaround you can use `ipfw add deny proto ipcomp` options IPSEC # This ones can be loaded as modules. They described in loader.conf section #options ACCEPT_FILTER_DATA #options ACCEPT_FILTER_HTTP # Adding ipfw, also can be loaded as modules options IPFIREWALL # On 8.1+ you can disable verbose to see blocked packets on ipfw0 interface. # Also there is no point in compiling verbose into the kernel, because # now there is net.inet.ip.fw.verbose tunable. #options IPFIREWALL_VERBOSE #options IPFIREWALL_VERBOSE_LIMIT=10 options IPFIREWALL_FORWARD # Adding kernel NAT options IPFIREWALL_NAT options LIBALIAS # Traffic shaping options DUMMYNET # Divert, i.e. for userspace NAT options IPDIVERT # This is for OpenBSD's pf firewall device pf device pflog # pf's QoS - ALTQ options ALTQ options ALTQ_CBQ # Class Bases Queuing (CBQ) options ALTQ_RED # Random Early Detection (RED) options ALTQ_RIO # RED In/Out options ALTQ_HFSC # Hierarchical Packet Scheduler (HFSC) options ALTQ_PRIQ # Priority Queuing (PRIQ) options ALTQ_NOPCC # Required for SMP build # Pretty console # Manual can be found here http://forums.freebsd.org/showthread.php?t=6134 #options VESA #options SC_PIXEL_MODE # Disable reboot on Ctrl Alt Del #options SC_DISABLE_REBOOT # Change normal|kernel messages color options SC_NORM_ATTR=(FG_GREEN|BG_BLACK) options SC_KERNEL_CONS_ATTR=(FG_YELLOW|BG_BLACK) # More scroll space options SC_HISTORY_SIZE=8192 # Adding hardware crypto device device crypto device cryptodev # Useful network interfaces device vlan device tap #Virtual Ethernet driver device gre #IP over IP tunneling device if_bridge #Bridge interface device pfsync #synchronization interface for PF device carp #Common Address Redundancy Protocol device enc #IPsec interface device lagg #Link aggregation interface device stf #IPv4-IPv6 port # Also for my notebook, but may be used with Opteron device amdtemp # Same for Intel processors device coretemp # man 4 cpuctl device cpuctl # CPU control pseudo-device # Support for ECMP. More than one route for destination # Works even with default route so one can use it as LB for two ISP # For now code is unstable and panics (panic: rtfree 2) on route deletions. #options RADIX_MPATH # Multicast routing #options MROUTING #options PIM # Debug & DTrace options KDB # Kernel debugger related code options KDB_TRACE # Print a stack trace for a panic options KDTRACE_FRAME # amd64-only(?) options KDTRACE_HOOKS # all architectures - enable general DTrace hooks #options DDB #options DDB_CTF # all architectures - kernel ELF linker loads CTF data # Adaptive spining in lockmgr (8.x+) # See http://www.mail-archive.com/[email protected]/msg10782.html options ADAPTIVE_LOCKMGRS # UTF-8 in console (8.x+) #options TEKEN_UTF8 # FreeBSD 8.1+ # Deadlock resolver thread # For additional information see http://www.mail-archive.com/[email protected]/msg18124.html # (FYI: "resolution" is panic so use with caution) #options DEADLKRES # Increase maximum size of Raw I/O and sendfile(2) readahead #options MAXPHYS=(1024*1024) #options MAXBSIZE=(1024*1024) # For scheduler debug enable following option. # Debug will be available via `kern.sched.stats` sysctl # For more information see http://svnweb.freebsd.org/base/head/sys/conf/NOTES?view=markup #options SCHED_STATS If you are tuning network for maximum performance you may wish to play with ifconfig options like: # You can list all capabilities via `ifconfig -m` ifconfig [-]rxcsum [-]txcsum [-]tso [-]lro mtu In case you've enabled DDB in kernel config, you should edit your /etc/ddb.conf and add something like this to enable automatic reboot (and textdump as bonus): script kdb.enter.panic=textdump set; capture on; show pcpu; bt; ps; alltrace; capture off; call doadump; reset script kdb.enter.default=textdump set; capture on; bt; ps; capture off; call doadump; reset And do not forget to add ddb_enable="YES" to /etc/rc.conf Since FreeBSD 9 you can select to enable/disable flowcontrol on your NIC: # See http://en.wikipedia.org/wiki/Ethernet_flow_control and # http://www.mail-archive.com/[email protected]/msg07927.html for additional info ifconfig bge0 media auto mediaopt flowcontrol PS. Also most of FreeBSD's limits can be monitored by # vmstat -z and # limits PPS. variety of network counters can be monitored via # netstat -s In FreeBSD-9 netstat's -Q option appeared, try following command to display netisr stats # netstat -Q PPPS. also see # man 7 tuning PPPPS. I wanted to thank FreeBSD community, especially author of nginx - Igor Sysoev, nginx-ru@ and FreeBSD-performance@ mailing lists for providing useful information about FreeBSD tuning. FreeBSD WIP * Whats cooking for FreeBSD 7? * Whats cooking for FreeBSD 8? * Whats cooking for FreeBSD 9? So here is the question: What tunings are you using on yours FreeBSD servers? You can also post your /etc/sysctl.conf, /boot/loader.conf, kernel options, etc with description of its' meaning (do not copy-paste from sysctl -d). Don't forget to specify server type (web, smb, gateway, etc) Let's share experience!

Read the article

Data Modeling: Logical Modeling Exercise

- by swisscheese

In trying to learn the art of data storage I have been trying to take in as much solid information as possible. PerformanceDBA posted some really helpful tutorials/examples in the following posts among others: is my data normalized? and Relational table naming convention. I already asked a subset question of this model here. So to make sure I understood the concepts he presented and I have seen elsewhere I wanted to take things a step or two further and see if I am grasping the concepts. Hence the purpose of this post, which hopefully others can also learn from. Everything I present is conceptual to me and for learning rather than applying it in some production system. It would be cool to get some input from PerformanceDBA also since I used his models to get started, but I appreciate all input given from anyone. As I am new to databases and especially modeling I will be the first to admit that I may not always ask the right questions, explain my thoughts clearly, or use the right verbage due to lack of expertise on the subject. So please keep that in mind and feel free to steer me in the right direction if I head off track. If there is enough interest in this I would like to take this from the logical to physical phases to show the evolution of the process and share it here on Stack. I will keep this thread for the Logical Diagram though and start new one for the additional steps. For my understanding I will be building a MySQL DB in the end to run some tests and see if what I came up with actually works. Here is the list of things that I want to capture in this conceptual model. Edit for V1.2 The purpose of this is to list Bands, their members, and the Events that they will be appearing at, as well as offer music and other merchandise for sale Members will be able to match up with friends Members can write reviews on the Bands, their music, and their events. There can only be one review per member on a given item, although they can edit their reviews and history will be maintained. BandMembers will have the chance to write a single Comment on Reviews about the Band they are associated with. Collectively as a Band only one Comment is allowed per Review. Members can then rate all Reviews and Comments but only once per given instance Members can select their favorite Bands, music, Merchandise, and Events Bands, Songs, and Events will be categorized into the type of Genre that they are and then further subcategorized into a SubGenre if necessary. It is ok for a Band or Event to fall into more then one Genre/SubGenre combination. Event date, time, and location will be posted for a given band and members can show that they will be attending the Event. An Event can be comprised of more than one Band, and multiple Events can take place at a single location on the same day Every party will be tied to at least one address and address history shall be maintained. Each party could also be tied to more then one address at a time (i.e. billing, shipping, physical) There will be stored profiles for Bands, BandMembers, and general members. So there it is, maybe a bit involved but could be a great learning tool for many hopefully as the process evolves and input is given by the community. Any input? EDIT v1.1 In response to PerformanceDBA U.3) That means no merchandise other than Band merchandise in the database. Correct ? That was my original thought but you got me thinking. Maybe the site would want to sell its own merchandise or even other merchandise from the bands. Not sure a mod to make for that. Would it require an entire rework of the Catalog section or just the identifying relationship that exists with the Band? Attempted a mod to sell both complete albums or song. Either way they would both be in electronic format only available for download. That is why I listed an Album as being comprised of Songs rather then 2 separate entities. U.5) I understand what you bring up about the circular relation with Favorite. I would like to get to this “It is either one Entity with some form of differentiation (FavoriteType) which identifies its treatment” but how to is not clear to me. What am I missing here? u.6) “Business Rules This is probably the only area you are weak in.” Thanks for the honest response. I will readdress these but I hope to clear up some confusion in my head first with the responses I have posted back to you. Q.1) Yes I would like to have Accepted, Rejected, and Blocked. I am not sure what you are referring to as to how this would change the logical model? Q.2) A person does not have to be a User. They can exist only as a BandMember. Is that what you are asking? Minor Issue Zero, One, or More…Oops I admit I forgot to give this attention when building the model. I am submitting this version as is and will address in a future version. I need to read up more on Constraint Checking to make sure I am understanding things. M.4) Depends if you envision OrderPurchase in the future. Can you expand as to what you mean here? EDIT V1.2 In response to PerformanceDBA input... Lessons learned. I was mixing the concept of Identifying / Non-Identifying and Cardinality (i.e. Genre / SubGenre), and doing so inconsistently to make things worse. Associative Tables are not required in Logical Diagrams as their many-to-many relationships can be depicted and then expanded in the Physical Model. I was overlooking the Cardinality in a lot of the relationships The importance of reading through relationships using effective Verb Phrases to reassure I am modeling what I want to accomplish. U.2) In the concept of this model it is only required to track a Venue as a location for an Event. No further data needs to be collected. With that being said Events will take place on a given EventDate and will be hosted at a Venue. Venues will host multiple events and possibly multiple events on a given date. In my new model my thinking was that EventDate is already tied to Event . Therefore, Venue will not need a relationship with EventDate. The 5th and 6th bullets you have listed under U.2) leave me questioning my thinking though. Am I missing something here? U.3) Is it time to move the link between Item and Band up to Item and Party instead? With the current design I don't see a possibility to sell merchandise not tied to the band as you have brought up. U.5) I left as per your input rather than making it a discrete Supertype/Subtype Relationship as I don’t see a benefit of having that type of roll up. Additional Revisions AR.1) After going through the exercise for FavoriteItem, I feel that Item to Review requires a many-to-many relationship so that is indicated. Necessary? Ok here we go for v1.3 I took a few days on this version, going back and forth with my design. Once the logical process is complete, as I want to see if I am on the right track, I will go through in depth what I had learned and the troubles I faced as a beginner going through this process. The big point for this version was it took throwing in some Keys to help see what I was missing in the past. Going through the process of doing a matrix proved to be of great help also. Regardless of anything, if it wasn't for the input given by PerformanceDBA I would still be a lost soul wondering in the dark. Who knows my current design might reaffirm that I still am, but I have learned a lot so I am know I at least have a flashlight in my hand. At this point in time I admit that I am still confused about identifying and non-identifying relationships. In my model I had to use non-identifying relationships with non nulls just to join the relationships I wanted to model. In reading a lot on the subject there seems to be a lot of disagreement and indecisiveness on the subject so I did what I thought represented the right things in my model. When to force (identifying) and when to be free (non-identifying)? Anyone have inputs? EDIT V1.4 Ok took the V1.3 inputs and cleaned things up for this V1.4 Currently working on a V1.5 to include attributes.

Read the article

Using IIS Application Request Routing (ARR) for ASP.NET MVC

- by Malcolm Frexner

I use a simple ASP.NET MVC web (the template you use when you create a new site) and the web works as expected in my live environment. I now try to use IIS Application Request Routing version 2. I have a rule that send all reuqests to a different server that match a rule. The settings are a bit like this: http://blogs.iis.net/wonyoo/archive/2008/07/09/application-request-routing-arr-as-a-reverse-proxy.aspx My rule is just a bit different it is /shop(.*). Only requests that contain shop are send to a different server. I have to use rewrite, not redirect (The same as in the Picture) This works as long as the web the original requests go to is no ASP.NET MVC web. I tried to use a plain htm file in the webfolder and it worked. If put a compiled ASP.NET application into the webfolder it worked. But as soon as I put an ASP.NET MVC web into the folder, request arr served by this application. My understanding is that the ARR should kick in before the web application gets the chance to handle the request. Did anybody use ARR sucessfully as a reverse proxy for a ASP.NET MVC web? EDIT Here is the resulting web config when the rewrite roule is entered. With this rule I get a 404 that indicates that the rule is not used. <?xml version="1.0" encoding="UTF-8"?> <configuration> <configSections> <sectionGroup name="system.web.extensions" type="System.Web.Configuration.SystemWebExtensionsSectionGroup, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35"> <sectionGroup name="scripting" type="System.Web.Configuration.ScriptingSectionGroup, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35"> <section name="scriptResourceHandler" type="System.Web.Configuration.ScriptingScriptResourceHandlerSection, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" requirePermission="false" allowDefinition="MachineToApplication" /> <sectionGroup name="webServices" type="System.Web.Configuration.ScriptingWebServicesSectionGroup, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35"> <section name="jsonSerialization" type="System.Web.Configuration.ScriptingJsonSerializationSection, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" requirePermission="false" allowDefinition="Everywhere" /> <section name="profileService" type="System.Web.Configuration.ScriptingProfileServiceSection, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" requirePermission="false" allowDefinition="MachineToApplication" /> <section name="authenticationService" type="System.Web.Configuration.ScriptingAuthenticationServiceSection, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" requirePermission="false" allowDefinition="MachineToApplication" /> <section name="roleService" type="System.Web.Configuration.ScriptingRoleServiceSection, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" requirePermission="false" allowDefinition="MachineToApplication" /> </sectionGroup> </sectionGroup> </sectionGroup> </configSections> <appSettings /> <connectionStrings> <add name="ApplicationServices" connectionString="data source=.\SQLEXPRESS;Integrated Security=SSPI;AttachDBFilename=|DataDirectory|aspnetdb.mdf;User Instance=true" providerName="System.Data.SqlClient" /> </connectionStrings> <system.web>  <compilation debug="false"> <assemblies> <add assembly="System.Core, Version=3.5.0.0, Culture=neutral, PublicKeyToken=B77A5C561934E089" /> <add assembly="System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /> <add assembly="System.Web.Abstractions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /> <add assembly="System.Web.Routing, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /> <add assembly="System.Web.Mvc, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /> <add assembly="System.Data.DataSetExtensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=B77A5C561934E089" /> <add assembly="System.Xml.Linq, Version=3.5.0.0, Culture=neutral, PublicKeyToken=B77A5C561934E089" /> <add assembly="System.Data.Linq, Version=3.5.0.0, Culture=neutral, PublicKeyToken=B77A5C561934E089" /> </assemblies> </compilation>  <authentication mode="Forms"> <forms loginUrl="~/Account/LogOn" timeout="2880" /> </authentication> <membership> <providers> <clear /> <add name="AspNetSqlMembershipProvider" type="System.Web.Security.SqlMembershipProvider, System.Web, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" connectionStringName="ApplicationServices" enablePasswordRetrieval="false" enablePasswordReset="true" requiresQuestionAndAnswer="false" requiresUniqueEmail="false" passwordFormat="Hashed" maxInvalidPasswordAttempts="5" minRequiredPasswordLength="6" minRequiredNonalphanumericCharacters="0" passwordAttemptWindow="10" passwordStrengthRegularExpression="" applicationName="/" /> </providers> </membership> <profile> <providers> <clear /> <add name="AspNetSqlProfileProvider" type="System.Web.Profile.SqlProfileProvider, System.Web, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" connectionStringName="ApplicationServices" applicationName="/" /> </providers> </profile> <roleManager enabled="false"> <providers> <clear /> <add connectionStringName="ApplicationServices" applicationName="/" name="AspNetSqlRoleProvider" type="System.Web.Security.SqlRoleProvider, System.Web, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" /> <add applicationName="/" name="AspNetWindowsTokenRoleProvider" type="System.Web.Security.WindowsTokenRoleProvider, System.Web, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" /> </providers> </roleManager>  <pages> <controls> <add tagPrefix="asp" namespace="System.Web.UI" assembly="System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /> <add tagPrefix="asp" namespace="System.Web.UI.WebControls" assembly="System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /> </controls> <namespaces> <add namespace="System.Web.Mvc" /> <add namespace="System.Web.Mvc.Ajax" /> <add namespace="System.Web.Mvc.Html" /> <add namespace="System.Web.Routing" /> <add namespace="System.Linq" /> <add namespace="System.Collections.Generic" /> </namespaces> </pages> <httpHandlers> <remove verb="*" path="*.asmx" /> <add verb="*" path="*.asmx" validate="false" type="System.Web.Script.Services.ScriptHandlerFactory, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /> <add verb="*" path="*_AppService.axd" validate="false" type="System.Web.Script.Services.ScriptHandlerFactory, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /> <add verb="GET,HEAD" path="ScriptResource.axd" type="System.Web.Handlers.ScriptResourceHandler, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" validate="false" /> <add verb="*" path="*.mvc" validate="false" type="System.Web.Mvc.MvcHttpHandler, System.Web.Mvc, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /> </httpHandlers> <httpModules> <add name="ScriptModule" type="System.Web.Handlers.ScriptModule, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /> <add name="UrlRoutingModule" type="System.Web.Routing.UrlRoutingModule, System.Web.Routing, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /> </httpModules> </system.web> <system.codedom> <compilers> <compiler language="c#;cs;csharp" extension=".cs" warningLevel="4" type="Microsoft.CSharp.CSharpCodeProvider, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"> <providerOption name="CompilerVersion" value="v3.5" /> <providerOption name="WarnAsError" value="false" /> </compiler> <compiler language="vb;vbs;visualbasic;vbscript" extension=".vb" warningLevel="4" type="Microsoft.VisualBasic.VBCodeProvider, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"> <providerOption name="CompilerVersion" value="v3.5" /> <providerOption name="OptionInfer" value="true" /> <providerOption name="WarnAsError" value="false" /> </compiler> </compilers> </system.codedom> <system.web.extensions />  <system.webServer> <rewrite> <rules> <rule name="shop" stopProcessing="true"> <match url="^shop/([_0-9a-z-.]+)" /> <action type="Rewrite" url="article.aspx?title={R:1}" logRewrittenUrl="true" /> </rule> </rules> </rewrite> <validation validateIntegratedModeConfiguration="false" /> <modules runAllManagedModulesForAllRequests="true"> <remove name="ScriptModule" /> <remove name="UrlRoutingModule" /> <add name="ScriptModule" preCondition="managedHandler" type="System.Web.Handlers.ScriptModule, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /> <add name="UrlRoutingModule" type="System.Web.Routing.UrlRoutingModule, System.Web.Routing, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /> </modules> <handlers> <remove name="WebServiceHandlerFactory-Integrated" /> <remove name="ScriptHandlerFactory" /> <remove name="ScriptHandlerFactoryAppServices" /> <remove name="ScriptResource" /> <remove name="MvcHttpHandler" /> <remove name="UrlRoutingHandler" /> <add name="ScriptHandlerFactory" verb="*" path="*.asmx" preCondition="integratedMode" type="System.Web.Script.Services.ScriptHandlerFactory, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /> <add name="ScriptHandlerFactoryAppServices" verb="*" path="*_AppService.axd" preCondition="integratedMode" type="System.Web.Script.Services.ScriptHandlerFactory, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /> <add name="ScriptResource" preCondition="integratedMode" verb="GET,HEAD" path="ScriptResource.axd" type="System.Web.Handlers.ScriptResourceHandler, System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /> <add name="MvcHttpHandler" preCondition="integratedMode" verb="*" path="*.mvc" type="System.Web.Mvc.MvcHttpHandler, System.Web.Mvc, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" /> <add name="UrlRoutingHandler" preCondition="integratedMode" verb="*" path="UrlRouting.axd" type="System.Web.HttpForbiddenHandler, System.Web, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" /> </handlers> </system.webServer> </configuration>

Read the article

Compiling a program with a legacy version of gcc

- by wyatt

This is probably a very difficult problem to troubleshoot with the information I can practically provide, but I'm hoping someone might be able to at least point me in a possible direction. I'm trying to install HTK (http://htk.eng.cam.ac.uk/), which, according to this page needs to be installed using gcc 3.4. Their method of implementing backwards compatibility: #yum install compat-gcc-34-c++ compat-gcc-34 won't work for me as I'm running Ubuntu (On that note, I take it I can't simply install YUM and the subsequent package, since it's an entirely different distro, but if I'm wrong I'd love to hear it). I instead installed two versions of gcc 3.4 - 3.4.0 and 3.4.6 using instructions from this site. I then added the lines suggested by that page to the top of the makefile (on this note, what's the difference between makefile and makefile.in? I tried adding the lines to the top of both files regardless), both for version 3.4.0 and 3.4.6, but both failed. I also tried, on the off-chance, compiling it with my current version (4.4.1), but that also failed. I got the errors: (cd HTKLib && make HTKLib.a) \ || case "" in k) fail=yes;; ) exit 1;; esac; make1: Entering directory /home/charles/bin/htk-3.4/HTKLib' gcc -ansi -D_SVID_SOURCE -DOSS_AUDIO -D'ARCH="i686"' -Wall -Wno-switch -g -O2 -I. -c -o HGraf.o HGraf.c HGraf.c:73:77: error: X11/Xlib.h: No such file or directory HGraf.c:74:23: error: X11/Xutil.h: No such file or directory HGraf.c:75:21: error: X11/Xos.h: No such file or directory HGraf.c:77:27: error: X11/keysymdef.h: No such file or directory HGraf.c:87: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token HGraf.c:88: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘rootW’ HGraf.c:91: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘theCmap’ HGraf.c:92: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘theGC’ HGraf.c:93: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘gcs’ HGraf.c:95: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token HGraf.c:96: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘report’ HGraf.c:97: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘hints’ HGraf.c:111: error: ‘GXcopy’ undeclared here (not in a function) HGraf.c:111: error: ‘GXor’ undeclared here (not in a function) HGraf.c:111: error: ‘GXxor’ undeclared here (not in a function) HGraf.c:111: error: ‘GXinvert’ undeclared here (not in a function) HGraf.c:151: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token HGraf.c: In function ‘InstallFonts’: HGraf.c:164: error: ‘FontInfo’ undeclared (first use in this function) HGraf.c:164: error: (Each undeclared identifier is reported only once HGraf.c:164: error: for each function it appears in.) HGraf.c:164: warning: implicit declaration of function ‘XLoadQueryFont’ HGraf.c:164: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:167: error: ‘DefaultFont’ undeclared (first use in this function) HGraf.c: At top level: HGraf.c:176: error: expected ‘)’ before ‘*’ token HGraf.c: In function ‘HGetEvent’: HGraf.c:219: error: ‘XEvent’ undeclared (first use in this function) HGraf.c:219: error: expected ‘;’ before ‘xev’ HGraf.c:223: warning: implicit declaration of function ‘XFlush’ HGraf.c:223: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:225: warning: implicit declaration of function ‘XEventsQueued’ HGraf.c:225: error: ‘QueuedAfterFlush’ undeclared (first use in this function) HGraf.c:226: warning: implicit declaration of function ‘XNextEvent’ HGraf.c:226: error: ‘xev’ undeclared (first use in this function) HGraf.c:228: error: ‘theWindow’ undeclared (first use in this function) HGraf.c:230: error: ‘ButtonPress’ undeclared (first use in this function) HGraf.c:235: error: ‘ButtonRelease’ undeclared (first use in this function) HGraf.c:240: error: ‘MotionNotify’ undeclared (first use in this function) HGraf.c:245: error: ‘KeyPress’ undeclared (first use in this function) HGraf.c:249: warning: implicit declaration of function ‘DecodeKeyPress’ HGraf.c:251: error: ‘KeyRelease’ undeclared (first use in this function) HGraf.c:257: error: ‘Expose’ undeclared (first use in this function) HGraf.c: In function ‘HEventsPending’: HGraf.c:281: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:281: error: ‘QueuedAfterFlush’ undeclared (first use in this function) HGraf.c: In function ‘HMousePos’: HGraf.c:288: error: ‘Window’ undeclared (first use in this function) HGraf.c:288: error: expected ‘;’ before ‘root’ HGraf.c:293: warning: implicit declaration of function ‘XQueryPointer’ HGraf.c:293: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:293: error: ‘theWindow’ undeclared (first use in this function) HGraf.c:293: error: ‘root’ undeclared (first use in this function) HGraf.c:293: error: ‘child’ undeclared (first use in this function) HGraf.c: In function ‘InstallColours’: HGraf.c:311: error: ‘XColor’ undeclared (first use in this function) HGraf.c:311: error: expected ‘;’ before ‘greyDef’ HGraf.c:317: warning: implicit declaration of function ‘XParseColor’ HGraf.c:317: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:317: error: ‘theCmap’ undeclared (first use in this function) HGraf.c:317: error: ‘colourDef’ undeclared (first use in this function) HGraf.c:320: warning: implicit declaration of function ‘XAllocColor’ HGraf.c:334: error: ‘whiteDef’ undeclared (first use in this function) HGraf.c:334: warning: implicit declaration of function ‘XQueryColor’ HGraf.c:335: error: ‘blackDef’ undeclared (first use in this function) HGraf.c:341: error: ‘greyDef’ undeclared (first use in this function) HGraf.c: In function ‘HSetColour’: HGraf.c:361: warning: implicit declaration of function ‘XSetForeground’ HGraf.c:361: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:361: error: ‘gcs’ undeclared (first use in this function) HGraf.c: In function ‘HSetGrey’: HGraf.c:370: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:370: error: ‘gcs’ undeclared (first use in this function) HGraf.c: In function ‘HDrawLines’: HGraf.c:388: warning: implicit declaration of function ‘XDrawLines’ HGraf.c:388: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:388: error: ‘theWindow’ undeclared (first use in this function) HGraf.c:388: error: ‘theGC’ undeclared (first use in this function) HGraf.c:388: error: ‘XPoint’ undeclared (first use in this function) HGraf.c:388: error: expected expression before ‘)’ token HGraf.c: In function ‘HDrawRectangle’: HGraf.c:395: warning: implicit declaration of function ‘XDrawRectangle’ HGraf.c:395: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:395: error: ‘theWindow’ undeclared (first use in this function) HGraf.c:395: error: ‘theGC’ undeclared (first use in this function) HGraf.c: In function ‘HFillRectangle’: HGraf.c:402: warning: implicit declaration of function ‘XFillRectangle’ HGraf.c:402: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:402: error: ‘theWindow’ undeclared (first use in this function) HGraf.c:402: error: ‘theGC’ undeclared (first use in this function) HGraf.c: In function ‘HDrawLine’: HGraf.c:408: warning: implicit declaration of function ‘XDrawLine’ HGraf.c:408: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:408: error: ‘theWindow’ undeclared (first use in this function) HGraf.c:408: error: ‘theGC’ undeclared (first use in this function) HGraf.c: In function ‘HFillPolygon’: HGraf.c:414: warning: implicit declaration of function ‘XFillPolygon’ HGraf.c:414: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:414: error: ‘theWindow’ undeclared (first use in this function) HGraf.c:414: error: ‘theGC’ undeclared (first use in this function) HGraf.c:414: error: ‘XPoint’ undeclared (first use in this function) HGraf.c:414: error: expected expression before ‘)’ token HGraf.c: In function ‘HDrawArc’: HGraf.c:427: warning: implicit declaration of function ‘XDrawArc’ HGraf.c:427: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:427: error: ‘theWindow’ undeclared (first use in this function) HGraf.c:427: error: ‘theGC’ undeclared (first use in this function) HGraf.c: In function ‘HFillArc’: HGraf.c:440: warning: implicit declaration of function ‘XFillArc’ HGraf.c:440: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:440: error: ‘theWindow’ undeclared (first use in this function) HGraf.c:440: error: ‘theGC’ undeclared (first use in this function) HGraf.c: In function ‘HPrintf’: HGraf.c:451: warning: implicit declaration of function ‘XDrawString’ HGraf.c:451: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:451: error: ‘theWindow’ undeclared (first use in this function) HGraf.c:451: error: ‘theGC’ undeclared (first use in this function) HGraf.c: In function ‘HCopyArea’: HGraf.c:457: warning: implicit declaration of function ‘XCopyArea’ HGraf.c:457: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:457: error: ‘theWindow’ undeclared (first use in this function) HGraf.c:457: error: ‘theGC’ undeclared (first use in this function) HGraf.c: In function ‘HPlotVector’: HGraf.c:476: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:476: error: ‘theWindow’ undeclared (first use in this function) HGraf.c:476: error: ‘theGC’ undeclared (first use in this function) HGraf.c: In function ‘HSetFontSize’: HGraf.c:490: error: ‘CurrentFont’ undeclared (first use in this function) HGraf.c:490: error: ‘DefaultFont’ undeclared (first use in this function) HGraf.c:499: error: ‘FontInfo’ undeclared (first use in this function) HGraf.c:502: warning: implicit declaration of function ‘XSetFont’ HGraf.c:502: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:502: error: ‘gcs’ undeclared (first use in this function) HGraf.c: In function ‘HSetLineWidth’: HGraf.c:511: warning: implicit declaration of function ‘XSetLineAttributes’ HGraf.c:511: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:511: error: ‘gcs’ undeclared (first use in this function) HGraf.c:511: error: ‘LineSolid’ undeclared (first use in this function) HGraf.c:511: error: ‘JoinRound’ undeclared (first use in this function) HGraf.c:511: error: ‘FillSolid’ undeclared (first use in this function) HGraf.c: In function ‘HSetXMode’: HGraf.c:517: error: ‘theGC’ undeclared (first use in this function) HGraf.c:517: error: ‘gcs’ undeclared (first use in this function) HGraf.c: In function ‘CentreX’: HGraf.c:523: warning: implicit declaration of function ‘XTextWidth’ HGraf.c:523: error: ‘CurrentFont’ undeclared (first use in this function) HGraf.c: In function ‘CentreY’: HGraf.c:529: error: ‘CurrentFont’ undeclared (first use in this function) HGraf.c: In function ‘HTextWidth’: HGraf.c:535: error: ‘CurrentFont’ undeclared (first use in this function) HGraf.c: In function ‘HTextHeight’: HGraf.c:541: error: ‘CurrentFont’ undeclared (first use in this function) HGraf.c: In function ‘HDrawImage’: HGraf.c:550: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token HGraf.c:550: error: ‘xi’ undeclared (first use in this function) HGraf.c:557: warning: implicit declaration of function ‘XDestroyImage’ HGraf.c:558: warning: implicit declaration of function ‘XGetImage’ HGraf.c:558: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:558: error: ‘theWindow’ undeclared (first use in this function) HGraf.c:558: error: ‘AllPlanes’ undeclared (first use in this function) HGraf.c:558: error: ‘XYPixmap’ undeclared (first use in this function) HGraf.c:562: warning: implicit declaration of function ‘XPutPixel’ HGraf.c:564: warning: implicit declaration of function ‘XPutImage’ HGraf.c:564: error: ‘theGC’ undeclared (first use in this function) HGraf.c: In function ‘HFlush’: HGraf.c:570: error: ‘theDisp’ undeclared (first use in this function) HGraf.c: In function ‘InitGCs’: HGraf.c:780: error: ‘XGCValues’ undeclared (first use in this function) HGraf.c:780: error: expected ‘;’ before ‘values’ HGraf.c:783: error: ‘GCLineWidth’ undeclared (first use in this function) HGraf.c:783: error: ‘GCFunction’ undeclared (first use in this function) HGraf.c:783: error: ‘GCForeground’ undeclared (first use in this function) HGraf.c:785: error: ‘values’ undeclared (first use in this function) HGraf.c:788: error: ‘gcs’ undeclared (first use in this function) HGraf.c:788: warning: implicit declaration of function ‘XCreateGC’ HGraf.c:788: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:788: error: ‘theWindow’ undeclared (first use in this function) HGraf.c:790: error: ‘GCPlaneMask’ undeclared (first use in this function) HGraf.c: In function ‘InitGlobals’: HGraf.c:800: warning: implicit declaration of function ‘DefaultScreen’ HGraf.c:800: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:801: error: ‘theCmap’ undeclared (first use in this function) HGraf.c:801: warning: implicit declaration of function ‘DefaultColormap’ HGraf.c:802: error: ‘rootW’ undeclared (first use in this function) HGraf.c:802: warning: implicit declaration of function ‘RootWindow’ HGraf.c:803: error: ‘theGC’ undeclared (first use in this function) HGraf.c:803: warning: implicit declaration of function ‘DefaultGC’ HGraf.c:804: error: ‘theVisual’ undeclared (first use in this function) HGraf.c:804: warning: implicit declaration of function ‘DefaultVisual’ HGraf.c:805: warning: implicit declaration of function ‘DisplayCells’ HGraf.c:806: warning: implicit declaration of function ‘DisplayWidth’ HGraf.c:807: warning: implicit declaration of function ‘DisplayHeight’ HGraf.c:808: warning: implicit declaration of function ‘DisplayPlanes’ HGraf.c:809: warning: implicit declaration of function ‘WhitePixel’ HGraf.c:810: warning: implicit declaration of function ‘BlackPixel’ HGraf.c: In function ‘MakeXGraf’: HGraf.c:817: error: ‘Window’ undeclared (first use in this function) HGraf.c:817: error: expected ‘;’ before ‘window’ HGraf.c:818: error: ‘XSetWindowAttributes’ undeclared (first use in this function) HGraf.c:818: error: expected ‘;’ before ‘setwinattr’ HGraf.c:823: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:823: warning: implicit declaration of function ‘XOpenDisplay’ HGraf.c:824: warning: implicit declaration of function ‘XDisplayName’ HGraf.c:828: error: ‘parent’ undeclared (first use in this function) HGraf.c:829: error: ‘window’ undeclared (first use in this function) HGraf.c:829: warning: implicit declaration of function ‘XCreateSimpleWindow’ HGraf.c:831: error: ‘CWBackingStore’ undeclared (first use in this function) HGraf.c:831: error: ‘setwinattr’ undeclared (first use in this function) HGraf.c:831: error: ‘WhenMapped’ undeclared (first use in this function) HGraf.c:832: warning: implicit declaration of function ‘XChangeWindowAttributes’ HGraf.c:834: error: ‘hints’ undeclared (first use in this function) HGraf.c:834: error: ‘PPosition’ undeclared (first use in this function) HGraf.c:834: error: ‘PSize’ undeclared (first use in this function) HGraf.c:834: error: ‘PMaxSize’ undeclared (first use in this function) HGraf.c:834: error: ‘PMinSize’ undeclared (first use in this function) HGraf.c:841: warning: implicit declaration of function ‘XSetStandardProperties’ HGraf.c:841: error: ‘None’ undeclared (first use in this function) HGraf.c:843: warning: implicit declaration of function ‘XSelectInput’ HGraf.c:843: error: ‘ExposureMask’ undeclared (first use in this function) HGraf.c:843: error: ‘KeyPressMask’ undeclared (first use in this function) HGraf.c:843: error: ‘ButtonPressMask’ undeclared (first use in this function) HGraf.c:844: error: ‘ButtonReleaseMask’ undeclared (first use in this function) HGraf.c:844: error: ‘PointerMotionHintMask’ undeclared (first use in this function) HGraf.c:844: error: ‘PointerMotionMask’ undeclared (first use in this function) HGraf.c:845: warning: implicit declaration of function ‘XMapWindow’ HGraf.c:845: error: ‘theWindow’ undeclared (first use in this function) HGraf.c:850: error: ‘report’ undeclared (first use in this function) HGraf.c:851: error: ‘Expose’ undeclared (first use in this function) HGraf.c:852: warning: implicit declaration of function ‘XSendEvent’ HGraf.c:852: error: ‘False’ undeclared (first use in this function) HGraf.c: In function ‘TermHGraf’: HGraf.c:861: error: ‘theDisp’ undeclared (first use in this function) HGraf.c:862: warning: implicit declaration of function ‘XCloseDisplay’ make[1]: *** [HGraf.o] Error 1 make[1]: Leaving directory/home/charles/bin/htk-3.4/HTKLib' make: ** [HTKLib/HTKLib.a] Error 1 Thank you for any help you can provide.

Read the article

Cache Simulator in C

- by DuffDuff

Ok this is only my second question, and it's quite a doozy. It's for a school assignment, but no one (including the TAs) seems to be able to help me. It's kind of a tall order but I'm not sure where else to turn. Essentially the assignment was to make a cache simulator. This version is direct mapping and is actually only a small portion of the whole project, but if I can't even get this down I have no chance with other associativities. I'm posting my whole code because I don't want to make any assumptions about where the problem is. This is the test case: http://www.mediafire.com/?ty5dnihydnw And you run the following command: ./sims 512 direct 32 fifo wt pinatrace.out You're supposed to get: hits: 604037 misses 138349 writes: 239269 reads: 138349 But I get: Hits: 587148 Misses: 155222 Writes: 239261 Reads: 155222 If anyone could at least point me in the right direction it would be greatly appreciated. I've been stuck on this for about 12 hours. #include <stdio.h> #include <stdlib.h> #include <string.h> #include <math.h> struct myCache { int valid; char *tag; char *block; }; /* sim [-h] <cache size> <associativity> <block size> <replace alg> <write policy> <trace file> */ //God willing I come up with a better Hex to Bin convertion that maintains the beginning 0s... void hex2bin(char input[], char output[]) { int i; int a = 0; int b = 1; int c = 2; int d = 3; int x = 4; int size; size = strlen(input); for (i = 0; i < size; i++) { if (input[i] =='0') { output[i*x +a] = '0'; output[i*x +b] = '0'; output[i*x +c] = '0'; output[i*x +d] = '0'; } else if (input[i] =='1') { output[i*x +a] = '0'; output[i*x +b] = '0'; output[i*x +c] = '0'; output[i*x +d] = '1'; } else if (input[i] =='2') { output[i*x +a] = '0'; output[i*x +b] = '0'; output[i*x +c] = '1'; output[i*x +d] = '0'; } else if (input[i] =='3') { output[i*x +a] = '0'; output[i*x +b] = '0'; output[i*x +c] = '1'; output[i*x +d] = '1'; } else if (input[i] =='x') { output[i*x +a] = '0'; output[i*x +b] = '1'; output[i*x +c] = '0'; output[i*x +d] = '0'; } else if (input[i] =='5') { output[i*x +a] = '0'; output[i*x +b] = '1'; output[i*x +c] = '0'; output[i*x +d] = '1'; } else if (input[i] =='6') { output[i*x +a] = '0'; output[i*x +b] = '1'; output[i*x +c] = '1'; output[i*x +d] = '0'; } else if (input[i] =='7') { output[i*x +a] = '0'; output[i*x +b] = '1'; output[i*x +c] = '1'; output[i*x +d] = '1'; } else if (input[i] =='8') { output[i*x +a] = '1'; output[i*x +b] = '0'; output[i*x +c] = '0'; output[i*x +d] = '0'; } else if (input[i] =='9') { output[i*x +a] = '1'; output[i*x +b] = '0'; output[i*x +c] = '0'; output[i*x +d] = '1'; } else if (input[i] =='a') { output[i*x +a] = '1'; output[i*x +b] = '0'; output[i*x +c] = '1'; output[i*x +d] = '0'; } else if (input[i] =='b') { output[i*x +a] = '1'; output[i*x +b] = '0'; output[i*x +c] = '1'; output[i*x +d] = '1'; } else if (input[i] =='c') { output[i*x +a] = '1'; output[i*x +b] = '1'; output[i*x +c] = '0'; output[i*x +d] = '0'; } else if (input[i] =='d') { output[i*x +a] = '1'; output[i*x +b] = '1'; output[i*x +c] = '0'; output[i*x +d] = '1'; } else if (input[i] =='e') { output[i*x +a] = '1'; output[i*x +b] = '1'; output[i*x +c] = '1'; output[i*x +d] = '0'; } else if (input[i] =='f') { output[i*x +a] = '1'; output[i*x +b] = '1'; output[i*x +c] = '1'; output[i*x +d] = '1'; } } output[32] = '\0'; } int main(int argc, char* argv[]) { FILE *tracefile; char readwrite; int trash; int cachesize; int blocksize; int setnumber; int blockbytes; int setbits; int blockbits; int tagsize; int m; int count = 0; int count2 = 0; int count3 = 0; int i; int j; int xindex; int jindex; int kindex; int lindex; int setadd; int totalset; int writeMiss = 0; int writeHit = 0; int cacheMiss = 0; int cacheHit = 0; int read = 0; int write = 0; int size; int extra; char bbits[100]; char sbits[100]; char tbits[100]; char output[100]; char input[100]; char origtag[100]; if (argc != 7) { if (strcmp(argv[0], "-h")) { printf("./sim2 <cache size> <associativity> <block size> <replace alg> <write policy> <trace file>\n"); return 0; } else { fprintf(stderr, "Error: wrong number of parameters.\n"); return -1; } } tracefile = fopen(argv[6], "r"); if(tracefile == NULL) { fprintf(stderr, "Error: File is NULL.\n"); return -1; } //Determining size of sbits, bbits, and tag cachesize = atoi(argv[1]); blocksize = atoi(argv[3]); setnumber = (cachesize/blocksize); printf("setnumber: %d\n", setnumber); setbits = (round((log(setnumber))/(log(2)))); printf("sbits: %d\n", setbits); blockbits = log(blocksize)/log(2); printf("bbits: %d\n", blockbits); tagsize = 32 - (blockbits + setbits); printf("t: %d\n", tagsize); struct myCache newCache[setnumber]; //Allocating Space for Tag Bits, initiating tag and valid to 0s for(i=0;i<setnumber;i++) { newCache[i].tag = (char *)malloc(sizeof(char)*(tagsize+1)); for(j=0;j<tagsize;j++) { newCache[i].tag[j] = '0'; } newCache[i].valid = 0; } while(fgetc(tracefile)!='#') { setadd = 0; totalset = 0; //read in file fseek(tracefile,-1,SEEK_CUR); fscanf(tracefile, "%x: %c %s\n", &trash, &readwrite, origtag); //shift input Hex size = strlen(origtag); extra = (10 - size); for(i=0; i<extra; i++) input[i] = '0'; for(i=extra, j=0; i<(size-(2-extra)); j++, i++) input[i]=origtag[j+2]; input[8] = '\0'; // Convert Hex to Binary hex2bin(input, output); //Resolving the Address into tbits, sbits, bbits for (xindex=0, jindex=(32-blockbits); jindex<32; jindex++, xindex++) { bbits[xindex] = output[jindex]; } bbits[xindex]='\0'; for (xindex=0, kindex=(32-(blockbits+setbits)); kindex<32-(blockbits); kindex++, xindex++){ sbits[xindex] = output[kindex]; } sbits[xindex]='\0'; for (xindex=0, lindex=0; lindex<(32-(blockbits+setbits)); lindex++, xindex++){ tbits[xindex] = output[lindex]; } tbits[xindex]='\0'; //Convert set bits from char array into ints for(xindex = 0, kindex = (setbits -1); xindex < setbits; xindex ++, kindex--) { if (sbits[xindex] == '1') setadd = 1; if (sbits[xindex] == '0') setadd = 0; setadd = setadd * pow(2, kindex); totalset += setadd; } //Calculating Hits and Misses if (newCache[totalset].valid == 0) { newCache[totalset].valid = 1; strcpy(newCache[totalset].tag, tbits); } else if (newCache[totalset].valid == 1) { if(strcmp(newCache[totalset].tag, tbits) == 0) { if (readwrite == 'W') { cacheHit++; write++; } if (readwrite == 'R') cacheHit++; } else { if (readwrite == 'R') { cacheMiss++; read++; } if (readwrite == 'W') { cacheMiss++; read++; write++; } strcpy(newCache[totalset].tag, tbits); } } } printf("Hits: %d\n", cacheHit); printf("Misses: %d\n", cacheMiss); printf("Writes: %d\n", write); printf("Reads: %d\n", read); }

Read the article

Advantage Database Server: slow stored procedure performance.

- by ie

I have a question about a performance of stored procedures in the ADS. I created a simple database with the following structure: CREATE TABLE MainTable ( Id INTEGER PRIMARY KEY, Name VARCHAR(50), Value INTEGER ); CREATE UNIQUE INDEX MainTableName_UIX ON MainTable ( Name ); CREATE TABLE SubTable ( Id INTEGER PRIMARY KEY, MainId INTEGER, Name VARCHAR(50), Value INTEGER ); CREATE INDEX SubTableMainId_UIX ON SubTable ( MainId ); CREATE UNIQUE INDEX SubTableName_UIX ON SubTable ( Name ); CREATE PROCEDURE CreateItems ( MainName VARCHAR ( 20 ), SubName VARCHAR ( 20 ), MainValue INTEGER, SubValue INTEGER, MainId INTEGER OUTPUT, SubId INTEGER OUTPUT ) BEGIN DECLARE @MainName VARCHAR ( 20 ); DECLARE @SubName VARCHAR ( 20 ); DECLARE @MainValue INTEGER; DECLARE @SubValue INTEGER; DECLARE @MainId INTEGER; DECLARE @SubId INTEGER; @MainName = (SELECT MainName FROM __input); @SubName = (SELECT SubName FROM __input); @MainValue = (SELECT MainValue FROM __input); @SubValue = (SELECT SubValue FROM __input); @MainId = (SELECT MAX(Id)+1 FROM MainTable); @SubId = (SELECT MAX(Id)+1 FROM SubTable ); INSERT INTO MainTable (Id, Name, Value) VALUES (@MainId, @MainName, @MainValue); INSERT INTO SubTable (Id, Name, MainId, Value) VALUES (@SubId, @SubName, @MainId, @SubValue); INSERT INTO __output SELECT @MainId, @SubId FROM system.iota; END; CREATE PROCEDURE UpdateItems ( MainName VARCHAR ( 20 ), MainValue INTEGER, SubValue INTEGER ) BEGIN DECLARE @MainName VARCHAR ( 20 ); DECLARE @MainValue INTEGER; DECLARE @SubValue INTEGER; DECLARE @MainId INTEGER; @MainName = (SELECT MainName FROM __input); @MainValue = (SELECT MainValue FROM __input); @SubValue = (SELECT SubValue FROM __input); @MainId = (SELECT TOP 1 Id FROM MainTable WHERE Name = @MainName); UPDATE MainTable SET Value = @MainValue WHERE Id = @MainId; UPDATE SubTable SET Value = @SubValue WHERE MainId = @MainId; END; CREATE PROCEDURE SelectItems ( MainName VARCHAR ( 20 ), CalculatedValue INTEGER OUTPUT ) BEGIN DECLARE @MainName VARCHAR ( 20 ); @MainName = (SELECT MainName FROM __input); INSERT INTO __output SELECT m.Value * s.Value FROM MainTable m INNER JOIN SubTable s ON m.Id = s.MainId WHERE m.Name = @MainName; END; CREATE PROCEDURE DeleteItems ( MainName VARCHAR ( 20 ) ) BEGIN DECLARE @MainName VARCHAR ( 20 ); DECLARE @MainId INTEGER; @MainName = (SELECT MainName FROM __input); @MainId = (SELECT TOP 1 Id FROM MainTable WHERE Name = @MainName); DELETE FROM SubTable WHERE MainId = @MainId; DELETE FROM MainTable WHERE Id = @MainId; END; Actually, the problem I had - even so light stored procedures work very-very slow (about 50-150 ms) relatively to plain queries (0-5ms). To test the performance, I created a simple test (in F# using ADS ADO.NET provider): open System; open System.Data; open System.Diagnostics; open Advantage.Data.Provider; let mainName = "main name #"; let subName = "sub name #"; // INSERT let cmdTextScriptInsert = " DECLARE @MainId INTEGER; DECLARE @SubId INTEGER; @MainId = (SELECT MAX(Id)+1 FROM MainTable); @SubId = (SELECT MAX(Id)+1 FROM SubTable ); INSERT INTO MainTable (Id, Name, Value) VALUES (@MainId, :MainName, :MainValue); INSERT INTO SubTable (Id, Name, MainId, Value) VALUES (@SubId, :SubName, @MainId, :SubValue); SELECT @MainId, @SubId FROM system.iota;"; let cmdTextProcedureInsert = "CreateItems"; // UPDATE let cmdTextScriptUpdate = " DECLARE @MainId INTEGER; @MainId = (SELECT TOP 1 Id FROM MainTable WHERE Name = :MainName); UPDATE MainTable SET Value = :MainValue WHERE Id = @MainId; UPDATE SubTable SET Value = :SubValue WHERE MainId = @MainId;"; let cmdTextProcedureUpdate = "UpdateItems"; // SELECT let cmdTextScriptSelect = " SELECT m.Value * s.Value FROM MainTable m INNER JOIN SubTable s ON m.Id = s.MainId WHERE m.Name = :MainName;"; let cmdTextProcedureSelect = "SelectItems"; // DELETE let cmdTextScriptDelete = " DECLARE @MainId INTEGER; @MainId = (SELECT TOP 1 Id FROM MainTable WHERE Name = :MainName); DELETE FROM SubTable WHERE MainId = @MainId; DELETE FROM MainTable WHERE Id = @MainId;"; let cmdTextProcedureDelete = "DeleteItems"; let cnnStr = @"data source=D:\DB\test.add; ServerType=local; user id=adssys; password=***;"; let cnn = new AdsConnection(cnnStr); try cnn.Open(); let cmd = cnn.CreateCommand(); let parametrize ix prms = cmd.Parameters.Clear(); let addParam = function | "MainName" -> cmd.Parameters.Add(":MainName" , mainName + ix.ToString()) |> ignore; | "SubName" -> cmd.Parameters.Add(":SubName" , subName + ix.ToString() ) |> ignore; | "MainValue" -> cmd.Parameters.Add(":MainValue", ix * 3 ) |> ignore; | "SubValue" -> cmd.Parameters.Add(":SubValue" , ix * 7 ) |> ignore; | _ -> () prms |> List.iter addParam; let runTest testData = let (cmdType, cmdName, cmdText, cmdParams) = testData; let toPrefix cmdType cmdName = let prefix = match cmdType with | CommandType.StoredProcedure -> "Procedure-" | CommandType.Text -> "Script -" | _ -> "Unknown -" in prefix + cmdName; let stopWatch = new Stopwatch(); let runStep ix prms = parametrize ix prms; stopWatch.Start(); cmd.ExecuteNonQuery() |> ignore; stopWatch.Stop(); cmd.CommandText <- cmdText; cmd.CommandType <- cmdType; let startId = 1500; let count = 10; for id in startId .. startId+count do runStep id cmdParams; let elapsed = stopWatch.Elapsed; Console.WriteLine("Test '{0}' - total: {1}; per call: {2}ms", toPrefix cmdType cmdName, elapsed, Convert.ToInt32(elapsed.TotalMilliseconds)/count); let lst = [ (CommandType.Text, "Insert", cmdTextScriptInsert, ["MainName"; "SubName"; "MainValue"; "SubValue"]); (CommandType.Text, "Update", cmdTextScriptUpdate, ["MainName"; "MainValue"; "SubValue"]); (CommandType.Text, "Select", cmdTextScriptSelect, ["MainName"]); (CommandType.Text, "Delete", cmdTextScriptDelete, ["MainName"]) (CommandType.StoredProcedure, "Insert", cmdTextProcedureInsert, ["MainName"; "SubName"; "MainValue"; "SubValue"]); (CommandType.StoredProcedure, "Update", cmdTextProcedureUpdate, ["MainName"; "MainValue"; "SubValue"]); (CommandType.StoredProcedure, "Select", cmdTextProcedureSelect, ["MainName"]); (CommandType.StoredProcedure, "Delete", cmdTextProcedureDelete, ["MainName"])]; lst |> List.iter runTest; finally cnn.Close(); And I'm getting the following results: Test 'Script -Insert' - total: 00:00:00.0292841; per call: 2ms Test 'Script -Update' - total: 00:00:00.0056296; per call: 0ms Test 'Script -Select' - total: 00:00:00.0051738; per call: 0ms Test 'Script -Delete' - total: 00:00:00.0059258; per call: 0ms Test 'Procedure-Insert' - total: 00:00:01.2567146; per call: 125ms Test 'Procedure-Update' - total: 00:00:00.7442440; per call: 74ms Test 'Procedure-Select' - total: 00:00:00.5120446; per call: 51ms Test 'Procedure-Delete' - total: 00:00:01.0619165; per call: 106ms The situation with the remote server is much better, but still a great gap between plaqin queries and stored procedures: Test 'Script -Insert' - total: 00:00:00.0709299; per call: 7ms Test 'Script -Update' - total: 00:00:00.0161777; per call: 1ms Test 'Script -Select' - total: 00:00:00.0258113; per call: 2ms Test 'Script -Delete' - total: 00:00:00.0166242; per call: 1ms Test 'Procedure-Insert' - total: 00:00:00.5116138; per call: 51ms Test 'Procedure-Update' - total: 00:00:00.3802251; per call: 38ms Test 'Procedure-Select' - total: 00:00:00.1241245; per call: 12ms Test 'Procedure-Delete' - total: 00:00:00.4336334; per call: 43ms Is it any chance to improve the SP performance? Please advice. ADO.NET driver version - 9.10.2.9 Server version - 9.10.0.9 (ANSI - GERMAN, OEM - GERMAN) Thanks!

Read the article

System.InvalidOperationException in Output Window

- by user318068

I constantly get the following message in my output/debug windows. The app doesn't crash but I was wondering what the deal with it is: A first chance exception of type 'System.InvalidOperationException' occurred in System.dll my code :sol.cs using System; using System.Collections.Generic; using System.Linq; using System.Text; namespace ConsoleApplication1 { class Sol { public LinkedList<int> tower1 = new LinkedList<int>(); public LinkedList<int> tower2 = new LinkedList<int>(); public LinkedList<int> tower3 = new LinkedList<int>(); public static LinkedList<string> BFS = new LinkedList<string>(); public static LinkedList<string> DFS = new LinkedList<string>(); public static LinkedList<string> IDS = new LinkedList<string>(); public int depth; public LinkedList<Sol> neighbors; public Sol() { } public Sol(LinkedList<int> tower1, LinkedList<int> tower2, LinkedList<int> tower3) { this.tower1 = tower1; this.tower2 = tower2; this.tower3 = tower3; neighbors = new LinkedList<Sol>(); } public virtual void getneighbors() { Sol temp = this.copy(); Sol neighbor1 = this.copy(); Sol neighbor2 = this.copy(); Sol neighbor3 = this.copy(); Sol neighbor4 = this.copy(); Sol neighbor5 = this.copy(); Sol neighbor6 = this.copy(); if (temp.tower1.Count != 0) { if (neighbor1.tower2.Count != 0) { if (neighbor1.tower1.First.Value < neighbor1.tower2.First.Value) { neighbor1.tower2.AddFirst(neighbor1.tower1.First); neighbor1.tower1.RemoveFirst(); neighbors.AddLast(neighbor1); } } else { neighbor1.tower2.AddFirst(neighbor1.tower1.First); neighbor1.tower1.RemoveFirst(); neighbors.AddLast(neighbor1); } if (neighbor2.tower3.Count != 0) { if (neighbor2.tower1.First.Value < neighbor2.tower3.First.Value) { neighbor2.tower3.AddFirst(neighbor2.tower1.First); neighbor2.tower1.RemoveFirst(); neighbors.AddLast(neighbor2); } } else { neighbor2.tower3.AddFirst(neighbor2.tower1.First); neighbor2.tower1.RemoveFirst(); neighbors.AddLast(neighbor2); } } //------------- if (temp.tower2.Count != 0) { if (neighbor3.tower1.Count != 0) { if (neighbor3.tower2.First.Value < neighbor3.tower1.First.Value) { neighbor3.tower1.AddFirst(neighbor3.tower2.First); neighbor3.tower2.RemoveFirst(); neighbors.AddLast(neighbor3); } } else { neighbor3.tower1.AddFirst(neighbor3.tower2.First); neighbor3.tower2.RemoveFirst(); neighbors.AddLast(neighbor3); } if (neighbor4.tower3.Count != 0) { if (neighbor4.tower2.First.Value < neighbor4.tower3.First.Value) { neighbor4.tower3.AddFirst(neighbor4.tower2.First); neighbor4.tower2.RemoveFirst(); neighbors.AddLast(neighbor4); } } else { neighbor4.tower3.AddFirst(neighbor4.tower2.First); neighbor4.tower2.RemoveFirst(); neighbors.AddLast(neighbor4); } } //------------------------ if (temp.tower3.Count() != 0) { if (neighbor5.tower1.Count() != 0) { if (neighbor5.tower3.ElementAtOrDefault(0) < neighbor5.tower1.ElementAtOrDefault(0)) { neighbor5.tower1.AddFirst(neighbor5.tower3.First); neighbor5.tower3.RemoveFirst(); neighbors.AddLast(neighbor5); } } else { neighbor5.tower1.AddFirst(neighbor5.tower3.First); neighbor5.tower3.RemoveFirst(); neighbors.AddLast(neighbor5); } if (neighbor6.tower2.Count() != 0) { if (neighbor6.tower3.ElementAtOrDefault(0) < neighbor6.tower2.ElementAtOrDefault(0)) { neighbor6.tower2.AddFirst(neighbor6.tower3.First); neighbor6.tower3.RemoveFirst(); neighbors.AddLast(neighbor6); } } else { neighbor6.tower2.AddFirst(neighbor6.tower3.First); neighbor6.tower3.RemoveFirst(); neighbors.AddLast(neighbor6); } } } public override string ToString() { string str; str = "tower1" + tower1.ToString() + " tower2" + tower2.ToString() + " tower3" + tower3.ToString(); return str; } public Sol copy() { Sol So; LinkedList<int> l1 = new LinkedList<int>(); LinkedList<int> l2 = new LinkedList<int>(); LinkedList<int> l3 = new LinkedList<int>(); for (int i = 0; i <= this.tower1.Count() - 1; i++) { l1.AddLast(tower1.ElementAt(i)); } for (int i = 0; i <= this.tower2.Count - 1; i++) { l2.AddLast(tower2.ElementAt(i)); } for (int i = 0; i <= this.tower3.Count - 1; i++) { l3.AddLast(tower3.ElementAt(i)); } So = new Sol(l1, l2, l3); return So; } public bool Equals(Sol sol) { if (this.tower1.Equals(sol.tower1) & this.tower2.Equals(sol.tower2) & this.tower3.Equals(sol.tower3)) return true; return false; } public virtual bool containedin(Stack<Sol> vec) { bool found = false; for (int i = 0; i <= vec.Count - 1; i++) { if (vec.ElementAt(i).tower1.Equals(this.tower1) && vec.ElementAt(i).tower2.Equals(this.tower2) && vec.ElementAt(i).tower3.Equals(this.tower3)) { found = true; break; } } return found; } public virtual bool breadthFirst(Sol start, Sol goal) { Stack<Sol> nextStack = new Stack<Sol>(); Stack<Sol> traversed = new Stack<Sol>(); bool found = false; start.depth = 0; nextStack.Push(start); while (nextStack.Count != 0) { Sol sol = nextStack.Pop(); BFS.AddFirst("poped State:" + sol.ToString() + "level " + sol.depth); traversed.Push(sol); if (sol.Equals(goal)) { found = true; BFS.AddFirst("Goal:" + sol.ToString()); break; } else { sol.getneighbors(); foreach (Sol neighbor in sol.neighbors) { if (!neighbor.containedin(traversed) && !neighbor.containedin(nextStack)) { neighbor.depth = (sol.depth + 1); nextStack.Push(neighbor); } } } } return found; } public virtual bool depthFirst(Sol start, Sol goal) { Stack<Sol> nextStack = new Stack<Sol>(); Stack<Sol> traversed = new Stack<Sol>(); bool found = false; start.depth = 0; nextStack.Push(start); while (nextStack.Count != 0) { //Dequeue next State for comparison //And add it 2 list of traversed States Sol sol = nextStack.Pop(); DFS.AddFirst("poped State:" + sol.ToString() + "level " + sol.depth); traversed.Push(sol); if (sol.Equals(goal)) { found = true; DFS.AddFirst("Goal:" + sol.ToString()); break; } else { sol.getneighbors(); foreach (Sol neighbor in sol.neighbors) { if (!neighbor.containedin(traversed) && !neighbor.containedin(nextStack)) { neighbor.depth = sol.depth + 1; nextStack.Push(neighbor); } } } } return found; } public virtual bool iterativedeepening(Sol start, Sol goal) { bool found = false; for (int level = 0; ; level++) { Stack<Sol> nextStack = new Stack<Sol>(); Stack<Sol> traversed = new Stack<Sol>(); start.depth = 0; nextStack.Push(start); while (nextStack.Count != 0) { Sol sol = nextStack.Pop(); IDS.AddFirst("poped State:" + sol.ToString() + "Level" + sol.depth); traversed.Push(sol); if (sol.Equals(goal)) { found = true; IDS.AddFirst("Goal:" + sol.ToString()); break; } else if (sol.depth < level) { sol.getneighbors(); foreach (Sol neighbor in sol.neighbors) { if (!neighbor.containedin(traversed) && !neighbor.containedin(nextStack)) { neighbor.depth = sol.depth + 1; nextStack.Push(neighbor); } //end if } //end for each } //end else if } // end while if (found == true) break; } // end for return found; } } } Just wondering if I may be doing something wrong somewhere or something.

Read the article

Access violation writing location, in my loop

- by numerical25

The exact error I am getting is First-chance exception at 0x0096234a in chp2.exe: 0xC0000005: Access violation writing location 0x002b0000. Windows has triggered a breakpoint in chp2.exe. And the breakpoint stops here for(DWORD i = 0; i < m; ++i) { //we are start at the top of z float z = halfDepth - i*dx; for(DWORD j = 0; j < n; ++j) { //to the left of us float x = -halfWidth + j*dx; float y = 0.0f; vertices[i*n+j].pos = D3DXVECTOR3(x, y, z); //<----- Right here vertices[i*n+j].color = D3DXVECTOR4(1.0f, 0.0f, 0.0f, 0.0f); } } I am not sure what I am doing wrong. below is the code in its entirety #include "MyGame.h" //#include "CubeVector.h" /* This code sets a projection and shows a turning cube. What has been added is the project, rotation and a rasterizer to change the rasterization of the cube. The issue that was going on was something with the effect file which was causing the vertices not to be rendered correctly.*/ typedef struct { ID3D10Effect* pEffect; ID3D10EffectTechnique* pTechnique; //vertex information ID3D10Buffer* pVertexBuffer; ID3D10Buffer* pIndicesBuffer; ID3D10InputLayout* pVertexLayout; UINT numVertices; UINT numIndices; }ModelObject; ModelObject modelObject; // World Matrix D3DXMATRIX WorldMatrix; // View Matrix D3DXMATRIX ViewMatrix; // Projection Matrix D3DXMATRIX ProjectionMatrix; ID3D10EffectMatrixVariable* pProjectionMatrixVariable = NULL; //grid information #define NUM_COLS 16 #define NUM_ROWS 16 #define CELL_WIDTH 32 #define CELL_HEIGHT 32 #define NUM_VERTSX (NUM_COLS + 1) #define NUM_VERTSY (NUM_ROWS + 1) bool MyGame::InitDirect3D() { if(!DX3dApp::InitDirect3D()) { return false; } D3D10_RASTERIZER_DESC rastDesc; rastDesc.FillMode = D3D10_FILL_WIREFRAME; rastDesc.CullMode = D3D10_CULL_FRONT; rastDesc.FrontCounterClockwise = true; rastDesc.DepthBias = false; rastDesc.DepthBiasClamp = 0; rastDesc.SlopeScaledDepthBias = 0; rastDesc.DepthClipEnable = false; rastDesc.ScissorEnable = false; rastDesc.MultisampleEnable = false; rastDesc.AntialiasedLineEnable = false; ID3D10RasterizerState *g_pRasterizerState; mpD3DDevice->CreateRasterizerState(&rastDesc, &g_pRasterizerState); mpD3DDevice->RSSetState(g_pRasterizerState); // Set up the World Matrix //The first line of code creates your identity matrix. Second line of code //second combines your camera position, target location, and which way is up respectively D3DXMatrixIdentity(&WorldMatrix); D3DXMatrixLookAtLH(&ViewMatrix, new D3DXVECTOR3(200.0f, 60.0f, -20.0f), new D3DXVECTOR3(200.0f, 50.0f, 0.0f), new D3DXVECTOR3(0.0f, 1.0f, 0.0f)); // Set up the projection matrix D3DXMatrixPerspectiveFovLH(&ProjectionMatrix, (float)D3DX_PI * 0.5f, (float)mWidth/(float)mHeight, 0.1f, 100.0f); if(!CreateObject()) { return false; } return true; } //These are actions that take place after the clearing of the buffer and before the present void MyGame::GameDraw() { static float rotationAngle = 0.0f; // create the rotation matrix using the rotation angle D3DXMatrixRotationY(&WorldMatrix, rotationAngle); rotationAngle += (float)D3DX_PI * 0.0f; // Set the input layout mpD3DDevice->IASetInputLayout(modelObject.pVertexLayout); // Set vertex buffer UINT stride = sizeof(VertexPos); UINT offset = 0; mpD3DDevice->IASetVertexBuffers(0, 1, &modelObject.pVertexBuffer, &stride, &offset); mpD3DDevice->IASetIndexBuffer(modelObject.pIndicesBuffer, DXGI_FORMAT_R32_UINT, 0); // Set primitive topology mpD3DDevice->IASetPrimitiveTopology(D3D10_PRIMITIVE_TOPOLOGY_TRIANGLELIST); // Combine and send the final matrix to the shader D3DXMATRIX finalMatrix = (WorldMatrix * ViewMatrix * ProjectionMatrix); pProjectionMatrixVariable->SetMatrix((float*)&finalMatrix); // make sure modelObject is valid // Render a model object D3D10_TECHNIQUE_DESC techniqueDescription; modelObject.pTechnique->GetDesc(&techniqueDescription); // Loop through the technique passes for(UINT p=0; p < techniqueDescription.Passes; ++p) { modelObject.pTechnique->GetPassByIndex(p)->Apply(0); // draw the cube using all 36 vertices and 12 triangles mpD3DDevice->DrawIndexed(modelObject.numIndices,0,0); } } //Render actually incapsulates Gamedraw, so you can call data before you actually clear the buffer or after you //present data void MyGame::Render() { DX3dApp::Render(); } bool MyGame::CreateObject() { //dx will represent the width and the height of the spacing of each vector float dx = 1; //Below are the number of vertices //m is the vertices of each row. n is the columns DWORD m = 30; DWORD n = 30; //This get the width of the entire land //30 - 1 = 29 rows * 1 = 29 * 0.5 = 14.5 float halfWidth = (n-1)*dx*0.5f; float halfDepth = (m-1)*dx*0.5f; float vertexsize = m * n; VertexPos vertices[80]; for(DWORD i = 0; i < m; ++i) { //we are start at the top of z float z = halfDepth - i*dx; for(DWORD j = 0; j < n; ++j) { //to the left of us float x = -halfWidth + j*dx; float y = 0.0f; vertices[i*n+j].pos = D3DXVECTOR3(x, y, z); vertices[i*n+j].color = D3DXVECTOR4(1.0f, 0.0f, 0.0f, 0.0f); } } int k = 0; DWORD indices[540]; for(DWORD i = 0; i < n-1; ++i) { for(DWORD j = 0; j < n-1; ++j) { indices[k] = (i * n) + j; indices[k + 1] = (i * n) + j + 1; indices[k + 2] = (i + 1) * n + j; indices[k + 3] = (i + 1) * n + j; indices[k + 4] = (i * n) + j + 1; indices[k + 5] = (i + 1) * n + j+ 1; k += 6; } } //Create Layout D3D10_INPUT_ELEMENT_DESC layout[] = { {"POSITION",0,DXGI_FORMAT_R32G32B32_FLOAT, 0 , 0, D3D10_INPUT_PER_VERTEX_DATA, 0}, {"COLOR",0,DXGI_FORMAT_R32G32B32A32_FLOAT, 0 , 12, D3D10_INPUT_PER_VERTEX_DATA, 0} }; UINT numElements = (sizeof(layout)/sizeof(layout[0])); modelObject.numVertices = sizeof(vertices)/sizeof(VertexPos); //Create buffer desc D3D10_BUFFER_DESC bufferDesc; bufferDesc.Usage = D3D10_USAGE_DEFAULT; bufferDesc.ByteWidth = sizeof(VertexPos) * modelObject.numVertices; bufferDesc.BindFlags = D3D10_BIND_VERTEX_BUFFER; bufferDesc.CPUAccessFlags = 0; bufferDesc.MiscFlags = 0; D3D10_SUBRESOURCE_DATA initData; initData.pSysMem = vertices; //Create the buffer HRESULT hr = mpD3DDevice->CreateBuffer(&bufferDesc, &initData, &modelObject.pVertexBuffer); if(FAILED(hr)) return false; modelObject.numIndices = sizeof(indices)/sizeof(DWORD); bufferDesc.ByteWidth = sizeof(DWORD) * modelObject.numIndices; bufferDesc.BindFlags = D3D10_BIND_INDEX_BUFFER; initData.pSysMem = indices; hr = mpD3DDevice->CreateBuffer(&bufferDesc, &initData, &modelObject.pIndicesBuffer); if(FAILED(hr)) return false; ///////////////////////////////////////////////////////////////////////////// //Set up fx files LPCWSTR effectFilename = L"effect.fx"; modelObject.pEffect = NULL; hr = D3DX10CreateEffectFromFile(effectFilename, NULL, NULL, "fx_4_0", D3D10_SHADER_ENABLE_STRICTNESS, 0, mpD3DDevice, NULL, NULL, &modelObject.pEffect, NULL, NULL); if(FAILED(hr)) return false; pProjectionMatrixVariable = modelObject.pEffect->GetVariableByName("Projection")->AsMatrix(); //Dont sweat the technique. Get it! LPCSTR effectTechniqueName = "Render"; modelObject.pTechnique = modelObject.pEffect->GetTechniqueByName(effectTechniqueName); if(modelObject.pTechnique == NULL) return false; //Create Vertex layout D3D10_PASS_DESC passDesc; modelObject.pTechnique->GetPassByIndex(0)->GetDesc(&passDesc); hr = mpD3DDevice->CreateInputLayout(layout, numElements, passDesc.pIAInputSignature, passDesc.IAInputSignatureSize, &modelObject.pVertexLayout); if(FAILED(hr)) return false; return true; }

Read the article

Zen and the Art of File and Folder Organization

- by Mark Virtue

Is your desk a paragon of neatness, or does it look like a paper-bomb has gone off? If you’ve been putting off getting organized because the task is too huge or daunting, or you don’t know where to start, we’ve got 40 tips to get you on the path to zen mastery of your filing system. For all those readers who would like to get their files and folders organized, or, if they’re already organized, better organized—we have compiled a complete guide to getting organized and staying organized, a comprehensive article that will hopefully cover every possible tip you could want. Signs that Your Computer is Poorly Organized If your computer is a mess, you’re probably already aware of it. But just in case you’re not, here are some tell-tale signs: Your Desktop has over 40 icons on it “My Documents” contains over 300 files and 60 folders, including MP3s and digital photos You use the Windows’ built-in search facility whenever you need to find a file You can’t find programs in the out-of-control list of programs in your Start Menu You save all your Word documents in one folder, all your spreadsheets in a second folder, etc Any given file that you’re looking for may be in any one of four different sets of folders But before we start, here are some quick notes: We’re going to assume you know what files and folders are, and how to create, save, rename, copy and delete them The organization principles described in this article apply equally to all computer systems. However, the screenshots here will reflect how things look on Windows (usually Windows 7). We will also mention some useful features of Windows that can help you get organized. Everyone has their own favorite methodology of organizing and filing, and it’s all too easy to get into “My Way is Better than Your Way” arguments. The reality is that there is no perfect way of getting things organized. When I wrote this article, I tried to keep a generalist and objective viewpoint. I consider myself to be unusually well organized (to the point of obsession, truth be told), and I’ve had 25 years experience in collecting and organizing files on computers. So I’ve got a lot to say on the subject. But the tips I have described here are only one way of doing it. Hopefully some of these tips will work for you too, but please don’t read this as any sort of “right” way to do it. At the end of the article we’ll be asking you, the reader, for your own organization tips. Why Bother Organizing At All? For some, the answer to this question is self-evident. And yet, in this era of powerful desktop search software (the search capabilities built into the Windows Vista and Windows 7 Start Menus, and third-party programs like Google Desktop Search), the question does need to be asked, and answered. I have a friend who puts every file he ever creates, receives or downloads into his My Documents folder and doesn’t bother filing them into subfolders at all. He relies on the search functionality built into his Windows operating system to help him find whatever he’s looking for. And he always finds it. He’s a Search Samurai. For him, filing is a waste of valuable time that could be spent enjoying life! It’s tempting to follow suit. On the face of it, why would anyone bother to take the time to organize their hard disk when such excellent search software is available? Well, if all you ever want to do with the files you own is to locate and open them individually (for listening, editing, etc), then there’s no reason to ever bother doing one scrap of organization. But consider these common tasks that are not achievable with desktop search software: Find files manually. Often it’s not convenient, speedy or even possible to utilize your desktop search software to find what you want. It doesn’t work 100% of the time, or you may not even have it installed. Sometimes its just plain faster to go straight to the file you want, if you know it’s in a particular sub-folder, rather than trawling through hundreds of search results. Find groups of similar files (e.g. all your “work” files, all the photos of your Europe holiday in 2008, all your music videos, all the MP3s from Dark Side of the Moon, all your letters you wrote to your wife, all your tax returns). Clever naming of the files will only get you so far. Sometimes it’s the date the file was created that’s important, other times it’s the file format, and other times it’s the purpose of the file. How do you name a collection of files so that they’re easy to isolate based on any of the above criteria? Short answer, you can’t. Move files to a new computer. It’s time to upgrade your computer. How do you quickly grab all the files that are important to you? Or you decide to have two computers now – one for home and one for work. How do you quickly isolate only the work-related files to move them to the work computer? Synchronize files to other computers. If you have more than one computer, and you need to mirror some of your files onto the other computer (e.g. your music collection), then you need a way to quickly determine which files are to be synced and which are not. Surely you don’t want to synchronize everything? Choose which files to back up. If your backup regime calls for multiple backups, or requires speedy backups, then you’ll need to be able to specify which files are to be backed up, and which are not. This is not possible if they’re all in the same folder. Finally, if you’re simply someone who takes pleasure in being organized, tidy and ordered (me! me!), then you don’t even need a reason. Being disorganized is simply unthinkable. Tips on Getting Organized Here we present our 40 best tips on how to get organized. Or, if you’re already organized, to get better organized. Tip #1. Choose Your Organization System Carefully The reason that most people are not organized is that it takes time. And the first thing that takes time is deciding upon a system of organization. This is always a matter of personal preference, and is not something that a geek on a website can tell you. You should always choose your own system, based on how your own brain is organized (which makes the assumption that your brain is, in fact, organized). We can’t instruct you, but we can make suggestions: You may want to start off with a system based on the users of the computer. i.e. “My Files”, “My Wife’s Files”, My Son’s Files”, etc. Inside “My Files”, you might then break it down into “Personal” and “Business”. You may then realize that there are overlaps. For example, everyone may want to share access to the music library, or the photos from the school play. So you may create another folder called “Family”, for the “common” files. You may decide that the highest-level breakdown of your files is based on the “source” of each file. In other words, who created the files. You could have “Files created by ME (business or personal)”, “Files created by people I know (family, friends, etc)”, and finally “Files created by the rest of the world (MP3 music files, downloaded or ripped movies or TV shows, software installation files, gorgeous desktop wallpaper images you’ve collected, etc).” This system happens to be the one I use myself. See below: Mark is for files created by meVC is for files created by my company (Virtual Creations)Others is for files created by my friends and familyData is the rest of the worldAlso, Settings is where I store the configuration files and other program data files for my installed software (more on this in tip #34, below). Each folder will present its own particular set of requirements for further sub-organization. For example, you may decide to organize your music collection into sub-folders based on the artist’s name, while your digital photos might get organized based on the date they were taken. It can be different for every sub-folder! Another strategy would be based on “currentness”. Files you have yet to open and look at live in one folder. Ones that have been looked at but not yet filed live in another place. Current, active projects live in yet another place. All other files (your “archive”, if you like) would live in a fourth folder. (And of course, within that last folder you’d need to create a further sub-system based on one of the previous bullet points). Put some thought into this – changing it when it proves incomplete can be a big hassle! Before you go to the trouble of implementing any system you come up with, examine a wide cross-section of the files you own and see if they will all be able to find a nice logical place to sit within your system. Tip #2. When You Decide on Your System, Stick to It! There’s nothing more pointless than going to all the trouble of creating a system and filing all your files, and then whenever you create, receive or download a new file, you simply dump it onto your Desktop. You need to be disciplined – forever! Every new file you get, spend those extra few seconds to file it where it belongs! Otherwise, in just a month or two, you’ll be worse off than before – half your files will be organized and half will be disorganized – and you won’t know which is which! Tip #3. Choose the Root Folder of Your Structure Carefully Every data file (document, photo, music file, etc) that you create, own or is important to you, no matter where it came from, should be found within one single folder, and that one single folder should be located at the root of your C: drive (as a sub-folder of C:\). In other words, do not base your folder structure in standard folders like “My Documents”. If you do, then you’re leaving it up to the operating system engineers to decide what folder structure is best for you. And every operating system has a different system! In Windows 7 your files are found in C:\Users\YourName, whilst on Windows XP it was C:\Documents and Settings\YourName\My Documents. In UNIX systems it’s often /home/YourName. These standard default folders tend to fill up with junk files and folders that are not at all important to you. “My Documents” is the worst offender. Every second piece of software you install, it seems, likes to create its own folder in the “My Documents” folder. These folders usually don’t fit within your organizational structure, so don’t use them! In fact, don’t even use the “My Documents” folder at all. Allow it to fill up with junk, and then simply ignore it. It sounds heretical, but: Don’t ever visit your “My Documents” folder! Remove your icons/links to “My Documents” and replace them with links to the folders you created and you care about! Create your own file system from scratch! Probably the best place to put it would be on your D: drive – if you have one. This way, all your files live on one drive, while all the operating system and software component files live on the C: drive – simply and elegantly separated. The benefits of that are profound. Not only are there obvious organizational benefits (see tip #10, below), but when it comes to migrate your data to a new computer, you can (sometimes) simply unplug your D: drive and plug it in as the D: drive of your new computer (this implies that the D: drive is actually a separate physical disk, and not a partition on the same disk as C:). You also get a slight speed improvement (again, only if your C: and D: drives are on separate physical disks). Warning: From tip #12, below, you will see that it’s actually a good idea to have exactly the same file system structure – including the drive it’s filed on – on all of the computers you own. So if you decide to use the D: drive as the storage system for your own files, make sure you are able to use the D: drive on all the computers you own. If you can’t ensure that, then you can still use a clever geeky trick to store your files on the D: drive, but still access them all via the C: drive (see tip #17, below). If you only have one hard disk (C:), then create a dedicated folder that will contain all your files – something like C:\Files. The name of the folder is not important, but make it a single, brief word. There are several reasons for this: When creating a backup regime, it’s easy to decide what files should be backed up – they’re all in the one folder! If you ever decide to trade in your computer for a new one, you know exactly which files to migrate You will always know where to begin a search for any file If you synchronize files with other computers, it makes your synchronization routines very simple. It also causes all your shortcuts to continue to work on the other machines (more about this in tip #24, below). Once you’ve decided where your files should go, then put all your files in there – Everything! Completely disregard the standard, default folders that are created for you by the operating system (“My Music”, “My Pictures”, etc). In fact, you can actually relocate many of those folders into your own structure (more about that below, in tip #6). The more completely you get all your data files (documents, photos, music, etc) and all your configuration settings into that one folder, then the easier it will be to perform all of the above tasks. Once this has been done, and all your files live in one folder, all the other folders in C:\ can be thought of as “operating system” folders, and therefore of little day-to-day interest for us. Here’s a screenshot of a nicely organized C: drive, where all user files are located within the \Files folder: Tip #4. Use Sub-Folders This would be our simplest and most obvious tip. It almost goes without saying. Any organizational system you decide upon (see tip #1) will require that you create sub-folders for your files. Get used to creating folders on a regular basis. Tip #5. Don’t be Shy About Depth Create as many levels of sub-folders as you need. Don’t be scared to do so. Every time you notice an opportunity to group a set of related files into a sub-folder, do so. Examples might include: All the MP3s from one music CD, all the photos from one holiday, or all the documents from one client. It’s perfectly okay to put files into a folder called C:\Files\Me\From Others\Services\WestCo Bank\Statements\2009. That’s only seven levels deep. Ten levels is not uncommon. Of course, it’s possible to take this too far. If you notice yourself creating a sub-folder to hold only one file, then you’ve probably become a little over-zealous. On the other hand, if you simply create a structure with only two levels (for example C:\Files\Work) then you really haven’t achieved any level of organization at all (unless you own only six files!). Your “Work” folder will have become a dumping ground, just like your Desktop was, with most likely hundreds of files in it. Tip #6. Move the Standard User Folders into Your Own Folder Structure Most operating systems, including Windows, create a set of standard folders for each of its users. These folders then become the default location for files such as documents, music files, digital photos and downloaded Internet files. In Windows 7, the full list is shown below: Some of these folders you may never use nor care about (for example, the Favorites folder, if you’re not using Internet Explorer as your browser). Those ones you can leave where they are. But you may be using some of the other folders to store files that are important to you. Even if you’re not using them, Windows will still often treat them as the default storage location for many types of files. When you go to save a standard file type, it can become annoying to be automatically prompted to save it in a folder that’s not part of your own file structure. But there’s a simple solution: Move the folders you care about into your own folder structure! If you do, then the next time you go to save a file of the corresponding type, Windows will prompt you to save it in the new, moved location. Moving the folders is easy. Simply drag-and-drop them to the new location. Here’s a screenshot of the default My Music folder being moved to my custom personal folder (Mark): Tip #7. Name Files and Folders Intelligently This is another one that almost goes without saying, but we’ll say it anyway: Do not allow files to be created that have meaningless names like Document1.doc, or folders called New Folder (2). Take that extra 20 seconds and come up with a meaningful name for the file/folder – one that accurately divulges its contents without repeating the entire contents in the name. Tip #8. Watch Out for Long Filenames Another way to tell if you have not yet created enough depth to your folder hierarchy is that your files often require really long names. If you need to call a file Johnson Sales Figures March 2009.xls (which might happen to live in the same folder as Abercrombie Budget Report 2008.xls), then you might want to create some sub-folders so that the first file could be simply called March.xls, and living in the Clients\Johnson\Sales Figures\2009 folder. A well-placed file needs only a brief filename! Tip #9. Use Shortcuts! Everywhere! This is probably the single most useful and important tip we can offer. A shortcut allows a file to be in two places at once. Why would you want that? Well, the file and folder structure of every popular operating system on the market today is hierarchical. This means that all objects (files and folders) always live within exactly one parent folder. It’s a bit like a tree. A tree has branches (folders) and leaves (files). Each leaf, and each branch, is supported by exactly one parent branch, all the way back to the root of the tree (which, incidentally, is exactly why C:\ is called the “root folder” of the C: drive). That hard disks are structured this way may seem obvious and even necessary, but it’s only one way of organizing data. There are others: Relational databases, for example, organize structured data entirely differently. The main limitation of hierarchical filing structures is that a file can only ever be in one branch of the tree – in only one folder – at a time. Why is this a problem? Well, there are two main reasons why this limitation is a problem for computer users: The “correct” place for a file, according to our organizational rationale, is very often a very inconvenient place for that file to be located. Just because it’s correctly filed doesn’t mean it’s easy to get to. Your file may be “correctly” buried six levels deep in your sub-folder structure, but you may need regular and speedy access to this file every day. You could always move it to a more convenient location, but that would mean that you would need to re-file back to its “correct” location it every time you’d finished working on it. Most unsatisfactory. A file may simply “belong” in two or more different locations within your file structure. For example, say you’re an accountant and you have just completed the 2009 tax return for John Smith. It might make sense to you to call this file 2009 Tax Return.doc and file it under Clients\John Smith. But it may also be important to you to have the 2009 tax returns from all your clients together in the one place. So you might also want to call the file John Smith.doc and file it under Tax Returns\2009. The problem is, in a purely hierarchical filing system, you can’t put it in both places. Grrrrr! Fortunately, Windows (and most other operating systems) offers a way for you to do exactly that: It’s called a “shortcut” (also known as an “alias” on Macs and a “symbolic link” on UNIX systems). Shortcuts allow a file to exist in one place, and an icon that represents the file to be created and put anywhere else you please. In fact, you can create a dozen such icons and scatter them all over your hard disk. Double-clicking on one of these icons/shortcuts opens up the original file, just as if you had double-clicked on the original file itself. Consider the following two icons: The one on the left is the actual Word document, while the one on the right is a shortcut that represents the Word document. Double-clicking on either icon will open the same file. There are two main visual differences between the icons: The shortcut will have a small arrow in the lower-left-hand corner (on Windows, anyway) The shortcut is allowed to have a name that does not include the file extension (the “.docx” part, in this case) You can delete the shortcut at any time without losing any actual data. The original is still intact. All you lose is the ability to get to that data from wherever the shortcut was. So why are shortcuts so great? Because they allow us to easily overcome the main limitation of hierarchical file systems, and put a file in two (or more) places at the same time. You will always have files that don’t play nice with your organizational rationale, and can’t be filed in only one place. They demand to exist in two places. Shortcuts allow this! Furthermore, they allow you to collect your most often-opened files and folders together in one spot for convenient access. The cool part is that the original files stay where they are, safe forever in their perfectly organized location. So your collection of most often-opened files can – and should – become a collection of shortcuts! If you’re still not convinced of the utility of shortcuts, consider the following well-known areas of a typical Windows computer: The Start Menu (and all the programs that live within it) The Quick Launch bar (or the Superbar in Windows 7) The “Favorite folders” area in the top-left corner of the Windows Explorer window (in Windows Vista or Windows 7) Your Internet Explorer Favorites or Firefox Bookmarks Each item in each of these areas is a shortcut! Each of those areas exist for one purpose only: For convenience – to provide you with a collection of the files and folders you access most often. It should be easy to see by now that shortcuts are designed for one single purpose: To make accessing your files more convenient. Each time you double-click on a shortcut, you are saved the hassle of locating the file (or folder, or program, or drive, or control panel icon) that it represents. Shortcuts allow us to invent a golden rule of file and folder organization: “Only ever have one copy of a file – never have two copies of the same file. Use a shortcut instead” (this rule doesn’t apply to copies created for backup purposes, of course!) There are also lesser rules, like “don’t move a file into your work area – create a shortcut there instead”, and “any time you find yourself frustrated with how long it takes to locate a file, create a shortcut to it and place that shortcut in a convenient location.” So how to we create these massively useful shortcuts? There are two main ways: “Copy” the original file or folder (click on it and type Ctrl-C, or right-click on it and select Copy): Then right-click in an empty area of the destination folder (the place where you want the shortcut to go) and select Paste shortcut: Right-drag (drag with the right mouse button) the file from the source folder to the destination folder. When you let go of the mouse button at the destination folder, a menu pops up: Select Create shortcuts here. Note that when shortcuts are created, they are often named something like Shortcut to Budget Detail.doc (windows XP) or Budget Detail – Shortcut.doc (Windows 7). If you don’t like those extra words, you can easily rename the shortcuts after they’re created, or you can configure Windows to never insert the extra words in the first place (see our article on how to do this). And of course, you can create shortcuts to folders too, not just to files! Bottom line: Whenever you have a file that you’d like to access from somewhere else (whether it’s convenience you’re after, or because the file simply belongs in two places), create a shortcut to the original file in the new location. Tip #10. Separate Application Files from Data Files Any digital organization guru will drum this rule into you. Application files are the components of the software you’ve installed (e.g. Microsoft Word, Adobe Photoshop or Internet Explorer). Data files are the files that you’ve created for yourself using that software (e.g. Word Documents, digital photos, emails or playlists). Software gets installed, uninstalled and upgraded all the time. Hopefully you always have the original installation media (or downloaded set-up file) kept somewhere safe, and can thus reinstall your software at any time. This means that the software component files are of little importance. Whereas the files you have created with that software is, by definition, important. It’s a good rule to always separate unimportant files from important files. So when your software prompts you to save a file you’ve just created, take a moment and check out where it’s suggesting that you save the file. If it’s suggesting that you save the file into the same folder as the software itself, then definitely don’t follow that suggestion. File it in your own folder! In fact, see if you can find the program’s configuration option that determines where files are saved by default (if it has one), and change it. Tip #11. Organize Files Based on Purpose, Not on File Type If you have, for example a folder called Work\Clients\Johnson, and within that folder you have two sub-folders, Word Documents and Spreadsheets (in other words, you’re separating “.doc” files from “.xls” files), then chances are that you’re not optimally organized. It makes little sense to organize your files based on the program that created them. Instead, create your sub-folders based on the purpose of the file. For example, it would make more sense to create sub-folders called Correspondence and Financials. It may well be that all the files in a given sub-folder are of the same file-type, but this should be more of a coincidence and less of a design feature of your organization system. Tip #12. Maintain the Same Folder Structure on All Your Computers In other words, whatever organizational system you create, apply it to every computer that you can. There are several benefits to this: There’s less to remember. No matter where you are, you always know where to look for your files If you copy or synchronize files from one computer to another, then setting up the synchronization job becomes very simple Shortcuts can be copied or moved from one computer to another with ease (assuming the original files are also copied/moved). There’s no need to find the target of the shortcut all over again on the second computer Ditto for linked files (e.g Word documents that link to data in a separate Excel file), playlists, and any files that reference the exact file locations of other files. This applies even to the drive that your files are stored on. If your files are stored on C: on one computer, make sure they’re stored on C: on all your computers. Otherwise all your shortcuts, playlists and linked files will stop working! Tip #13. Create an “Inbox” Folder Create yourself a folder where you store all files that you’re currently working on, or that you haven’t gotten around to filing yet. You can think of this folder as your “to-do” list. You can call it “Inbox” (making it the same metaphor as your email system), or “Work”, or “To-Do”, or “Scratch”, or whatever name makes sense to you. It doesn’t matter what you call it – just make sure you have one! Once you have finished working on a file, you then move it from the “Inbox” to its correct location within your organizational structure. You may want to use your Desktop as this “Inbox” folder. Rightly or wrongly, most people do. It’s not a bad place to put such files, but be careful: If you do decide that your Desktop represents your “to-do” list, then make sure that no other files find their way there. In other words, make sure that your “Inbox”, wherever it is, Desktop or otherwise, is kept free of junk – stray files that don’t belong there. So where should you put this folder, which, almost by definition, lives outside the structure of the rest of your filing system? Well, first and foremost, it has to be somewhere handy. This will be one of your most-visited folders, so convenience is key. Putting it on the Desktop is a great option – especially if you don’t have any other folders on your Desktop: the folder then becomes supremely easy to find in Windows Explorer: You would then create shortcuts to this folder in convenient spots all over your computer (“Favorite Links”, “Quick Launch”, etc). Tip #14. Ensure You have Only One “Inbox” Folder Once you’ve created your “Inbox” folder, don’t use any other folder location as your “to-do list”. Throw every incoming or created file into the Inbox folder as you create/receive it. This keeps the rest of your computer pristine and free of randomly created or downloaded junk. The last thing you want to be doing is checking multiple folders to see all your current tasks and projects. Gather them all together into one folder. Here are some tips to help ensure you only have one Inbox: Set the default “save” location of all your programs to this folder. Set the default “download” location for your browser to this folder. If this folder is not your desktop (recommended) then also see if you can make a point of not putting “to-do” files on your desktop. This keeps your desktop uncluttered and Zen-like: (the Inbox folder is in the bottom-right corner) Tip #15. Be Vigilant about Clearing Your “Inbox” Folder This is one of the keys to staying organized. If you let your “Inbox” overflow (i.e. allow there to be more than, say, 30 files or folders in there), then you’re probably going to start feeling like you’re overwhelmed: You’re not keeping up with your to-do list. Once your Inbox gets beyond a certain point (around 30 files, studies have shown), then you’ll simply start to avoid it. You may continue to put files in there, but you’ll be scared to look at it, fearing the “out of control” feeling that all overworked, chaotic or just plain disorganized people regularly feel. So, here’s what you can do: Visit your Inbox/to-do folder regularly (at least five times per day). Scan the folder regularly for files that you have completed working on and are ready for filing. File them immediately. Make it a source of pride to keep the number of files in this folder as small as possible. If you value peace of mind, then make the emptiness of this folder one of your highest (computer) priorities If you know that a particular file has been in the folder for more than, say, six weeks, then admit that you’re not actually going to get around to processing it, and move it to its final resting place. Tip #16. File Everything Immediately, and Use Shortcuts for Your Active Projects As soon as you create, receive or download a new file, store it away in its “correct” folder immediately. Then, whenever you need to work on it (possibly straight away), create a shortcut to it in your “Inbox” (“to-do”) folder or your desktop. That way, all your files are always in their “correct” locations, yet you still have immediate, convenient access to your current, active files. When you finish working on a file, simply delete the shortcut. Ideally, your “Inbox” folder – and your Desktop – should contain no actual files or folders. They should simply contain shortcuts. Tip #17. Use Directory Symbolic Links (or Junctions) to Maintain One Unified Folder Structure Using this tip, we can get around a potential hiccup that we can run into when creating our organizational structure – the issue of having more than one drive on our computer (C:, D:, etc). We might have files we need to store on the D: drive for space reasons, and yet want to base our organized folder structure on the C: drive (or vice-versa). Your chosen organizational structure may dictate that all your files must be accessed from the C: drive (for example, the root folder of all your files may be something like C:\Files). And yet you may still have a D: drive and wish to take advantage of the hundreds of spare Gigabytes that it offers. Did you know that it’s actually possible to store your files on the D: drive and yet access them as if they were on the C: drive? And no, we’re not talking about shortcuts here (although the concept is very similar). By using the shell command mklink, you can essentially take a folder that lives on one drive and create an alias for it on a different drive (you can do lots more than that with mklink – for a full rundown on this programs capabilities, see our dedicated article). These aliases are called directory symbolic links (and used to be known as junctions). You can think of them as “virtual” folders. They function exactly like regular folders, except they’re physically located somewhere else. For example, you may decide that your entire D: drive contains your complete organizational file structure, but that you need to reference all those files as if they were on the C: drive, under C:\Files. If that was the case you could create C:\Files as a directory symbolic link – a link to D:, as follows: mklink /d c:\files d:\ Or it may be that the only files you wish to store on the D: drive are your movie collection. You could locate all your movie files in the root of your D: drive, and then link it to C:\Files\Media\Movies, as follows: mklink /d c:\files\media\movies d:\ (Needless to say, you must run these commands from a command prompt – click the Start button, type cmd and press Enter) Tip #18. Customize Your Folder Icons This is not strictly speaking an organizational tip, but having unique icons for each folder does allow you to more quickly visually identify which folder is which, and thus saves you time when you’re finding files. An example is below (from my folder that contains all files downloaded from the Internet): To learn how to change your folder icons, please refer to our dedicated article on the subject. Tip #19. Tidy Your Start Menu The Windows Start Menu is usually one of the messiest parts of any Windows computer. Every program you install seems to adopt a completely different approach to placing icons in this menu. Some simply put a single program icon. Others create a folder based on the name of the software. And others create a folder based on the name of the software manufacturer. It’s chaos, and can make it hard to find the software you want to run. Thankfully we can avoid this chaos with useful operating system features like Quick Launch, the Superbar or pinned start menu items. Even so, it would make a lot of sense to get into the guts of the Start Menu itself and give it a good once-over. All you really need to decide is how you’re going to organize your applications. A structure based on the purpose of the application is an obvious candidate. Below is an example of one such structure: In this structure, Utilities means software whose job it is to keep the computer itself running smoothly (configuration tools, backup software, Zip programs, etc). Applications refers to any productivity software that doesn’t fit under the headings Multimedia, Graphics, Internet, etc. In case you’re not aware, every icon in your Start Menu is a shortcut and can be manipulated like any other shortcut (copied, moved, deleted, etc). With the Windows Start Menu (all version of Windows), Microsoft has decided that there be two parallel folder structures to store your Start Menu shortcuts. One for you (the logged-in user of the computer) and one for all users of the computer. Having two parallel structures can often be redundant: If you are the only user of the computer, then having two parallel structures is totally redundant. Even if you have several users that regularly log into the computer, most of your installed software will need to be made available to all users, and should thus be moved out of the “just you” version of the Start Menu and into the “all users” area. To take control of your Start Menu, so you can start organizing it, you’ll need to know how to access the actual folders and shortcut files that make up the Start Menu (both versions of it). To find these folders and files, click the Start button and then right-click on the All Programs text (Windows XP users should right-click on the Start button itself): The Open option refers to the “just you” version of the Start Menu, while the Open All Users option refers to the “all users” version. Click on the one you want to organize. A Windows Explorer window then opens with your chosen version of the Start Menu selected. From there it’s easy. Double-click on the Programs folder and you’ll see all your folders and shortcuts. Now you can delete/rename/move until it’s just the way you want it. Note: When you’re reorganizing your Start Menu, you may want to have two Explorer windows open at the same time – one showing the “just you” version and one showing the “all users” version. You can drag-and-drop between the windows. Tip #20. Keep Your Start Menu Tidy Once you have a perfectly organized Start Menu, try to be a little vigilant about keeping it that way. Every time you install a new piece of software, the icons that get created will almost certainly violate your organizational structure. So to keep your Start Menu pristine and organized, make sure you do the following whenever you install a new piece of software: Check whether the software was installed into the “just you” area of the Start Menu, or the “all users” area, and then move it to the correct area. Remove all the unnecessary icons (like the “Read me” icon, the “Help” icon (you can always open the help from within the software itself when it’s running), the “Uninstall” icon, the link(s)to the manufacturer’s website, etc) Rename the main icon(s) of the software to something brief that makes sense to you. For example, you might like to rename Microsoft Office Word 2010 to simply Word Move the icon(s) into the correct folder based on your Start Menu organizational structure And don’t forget: when you uninstall a piece of software, the software’s uninstall routine is no longer going to be able to remove the software’s icon from the Start Menu (because you moved and/or renamed it), so you’ll need to remove that icon manually. Tip #21. Tidy C:\ The root of your C: drive (C:\) is a common dumping ground for files and folders – both by the users of your computer and by the software that you install on your computer. It can become a mess. There’s almost no software these days that requires itself to be installed in C:\. 99% of the time it can and should be installed into C:\Program Files. And as for your own files, well, it’s clear that they can (and almost always should) be stored somewhere else. In an ideal world, your C:\ folder should look like this (on Windows 7): Note that there are some system files and folders in C:\ that are usually and deliberately “hidden” (such as the Windows virtual memory file pagefile.sys, the boot loader file bootmgr, and the System Volume Information folder). Hiding these files and folders is a good idea, as they need to stay where they are and are almost never needed to be opened or even seen by you, the user. Hiding them prevents you from accidentally messing with them, and enhances your sense of order and well-being when you look at your C: drive folder. Tip #22. Tidy Your Desktop The Desktop is probably the most abused part of a Windows computer (from an organization point of view). It usually serves as a dumping ground for all incoming files, as well as holding icons to oft-used applications, plus some regularly opened files and folders. It often ends up becoming an uncontrolled mess. See if you can avoid this. Here’s why… Application icons (Word, Internet Explorer, etc) are often found on the Desktop, but it’s unlikely that this is the optimum place for them. The “Quick Launch” bar (or the Superbar in Windows 7) is always visible and so represents a perfect location to put your icons. You’ll only be able to see the icons on your Desktop when all your programs are minimized. It might be time to get your application icons off your desktop… You may have decided that the Inbox/To-do folder on your computer (see tip #13, above) should be your Desktop. If so, then enough said. Simply be vigilant about clearing it and preventing it from being polluted by junk files (see tip #15, above). On the other hand, if your Desktop is not acting as your “Inbox” folder, then there’s no reason for it to have any data files or folders on it at all, except perhaps a couple of shortcuts to often-opened files and folders (either ongoing or current projects). Everything else should be moved to your “Inbox” folder. In an ideal world, it might look like this: Tip #23. Move Permanent Items on Your Desktop Away from the Top-Left Corner When files/folders are dragged onto your desktop in a Windows Explorer window, or when shortcuts are created on your Desktop from Internet Explorer, those icons are always placed in the top-left corner – or as close as they can get. If you have other files, folders or shortcuts that you keep on the Desktop permanently, then it’s a good idea to separate these permanent icons from the transient ones, so that you can quickly identify which ones the transients are. An easy way to do this is to move all your permanent icons to the right-hand side of your Desktop. That should keep them separated from incoming items. Tip #24. Synchronize If you have more than one computer, you’ll almost certainly want to share files between them. If the computers are permanently attached to the same local network, then there’s no need to store multiple copies of any one file or folder – shortcuts will suffice. However, if the computers are not always on the same network, then you will at some point need to copy files between them. For files that need to permanently live on both computers, the ideal way to do this is to synchronize the files, as opposed to simply copying them. We only have room here to write a brief summary of synchronization, not a full article. In short, there are several different types of synchronization: Where the contents of one folder are accessible anywhere, such as with Dropbox Where the contents of any number of folders are accessible anywhere, such as with Windows Live Mesh Where any files or folders from anywhere on your computer are synchronized with exactly one other computer, such as with the Windows “Briefcase”, Microsoft SyncToy, or (much more powerful, yet still free) SyncBack from 2BrightSparks. This only works when both computers are on the same local network, at least temporarily. A great advantage of synchronization solutions is that once you’ve got it configured the way you want it, then the sync process happens automatically, every time. Click a button (or schedule it to happen automatically) and all your files are automagically put where they’re supposed to be. If you maintain the same file and folder structure on both computers, then you can also sync files depend upon the correct location of other files, like shortcuts, playlists and office documents that link to other office documents, and the synchronized files still work on the other computer! Tip #25. Hide Files You Never Need to See If you have your files well organized, you will often be able to tell if a file is out of place just by glancing at the contents of a folder (for example, it should be pretty obvious if you look in a folder that contains all the MP3s from one music CD and see a Word document in there). This is a good thing – it allows you to determine if there are files out of place with a quick glance. Yet sometimes there are files in a folder that seem out of place but actually need to be there, such as the “folder art” JPEGs in music folders, and various files in the root of the C: drive. If such files never need to be opened by you, then a good idea is to simply hide them. Then, the next time you glance at the folder, you won’t have to remember whether that file was supposed to be there or not, because you won’t see it at all! To hide a file, simply right-click on it and choose Properties: Then simply tick the Hidden tick-box: Tip #26. Keep Every Setup File These days most software is downloaded from the Internet. Whenever you download a piece of software, keep it. You’ll never know when you need to reinstall the software. Further, keep with it an Internet shortcut that links back to the website where you originally downloaded it, in case you ever need to check for updates. See tip #33 below for a full description of the excellence of organizing your setup files. Tip #27. Try to Minimize the Number of Folders that Contain Both Files and Sub-folders Some of the folders in your organizational structure will contain only files. Others will contain only sub-folders. And you will also have some folders that contain both files and sub-folders. You will notice slight improvements in how long it takes you to locate a file if you try to avoid this third type of folder. It’s not always possible, of course – you’ll always have some of these folders, but see if you can avoid it. One way of doing this is to take all the leftover files that didn’t end up getting stored in a sub-folder and create a special “Miscellaneous” or “Other” folder for them. Tip #28. Starting a Filename with an Underscore Brings it to the Top of a List Further to the previous tip, if you name that “Miscellaneous” or “Other” folder in such a way that its name begins with an underscore “_”, then it will appear at the top of the list of files/folders. The screenshot below is an example of this. Each folder in the list contains a set of digital photos. The folder at the top of the list, _Misc, contains random photos that didn’t deserve their own dedicated folder: Tip #29. Clean Up those CD-ROMs and (shudder!) Floppy Disks Have you got a pile of CD-ROMs stacked on a shelf of your office? Old photos, or files you archived off onto CD-ROM (or even worse, floppy disks!) because you didn’t have enough disk space at the time? In the meantime have you upgraded your computer and now have 500 Gigabytes of space you don’t know what to do with? If so, isn’t it time you tidied up that stack of disks and filed them into your gorgeous new folder structure? So what are you waiting for? Bite the bullet, copy them all back onto your computer, file them in their appropriate folders, and then back the whole lot up onto a shiny new 1000Gig external hard drive! Useful Folders to Create This next section suggests some useful folders that you might want to create within your folder structure. I’ve personally found them to be indispensable. The first three are all about convenience – handy folders to create and then put somewhere that you can always access instantly. For each one, it’s not so important where the actual folder is located, but it’s very important where you put the shortcut(s) to the folder. You might want to locate the shortcuts: On your Desktop In your “Quick Launch” area (or pinned to your Windows 7 Superbar) In your Windows Explorer “Favorite Links” area Tip #30. Create an “Inbox” (“To-Do”) Folder This has already been mentioned in depth (see tip #13), but we wanted to reiterate its importance here. This folder contains all the recently created, received or downloaded files that you have not yet had a chance to file away properly, and it also may contain files that you have yet to process. In effect, it becomes a sort of “to-do list”. It doesn’t have to be called “Inbox” – you can call it whatever you want. Tip #31. Create a Folder where Your Current Projects are Collected Rather than going hunting for them all the time, or dumping them all on your desktop, create a special folder where you put links (or work folders) for each of the projects you’re currently working on. You can locate this folder in your “Inbox” folder, on your desktop, or anywhere at all – just so long as there’s a way of getting to it quickly, such as putting a link to it in Windows Explorer’s “Favorite Links” area: Tip #32. Create a Folder for Files and Folders that You Regularly Open You will always have a few files that you open regularly, whether it be a spreadsheet of your current accounts, or a favorite playlist. These are not necessarily “current projects”, rather they’re simply files that you always find yourself opening. Typically such files would be located on your desktop (or even better, shortcuts to those files). Why not collect all such shortcuts together and put them in their own special folder? As with the “Current Projects” folder (above), you would want to locate that folder somewhere convenient. Below is an example of a folder called “Quick links”, with about seven files (shortcuts) in it, that is accessible through the Windows Quick Launch bar: See tip #37 below for a full explanation of the power of the Quick Launch bar. Tip #33. Create a “Set-ups” Folder A typical computer has dozens of applications installed on it. For each piece of software, there are often many different pieces of information you need to keep track of, including: The original installation setup file(s). This can be anything from a simple 100Kb setup.exe file you downloaded from a website, all the way up to a 4Gig ISO file that you copied from a DVD-ROM that you purchased. The home page of the software manufacturer (in case you need to look up something on their support pages, their forum or their online help) The page containing the download link for your actual file (in case you need to re-download it, or download an upgraded version) The serial number Your proof-of-purchase documentation Any other template files, plug-ins, themes, etc that also need to get installed For each piece of software, it’s a great idea to gather all of these files together and put them in a single folder. The folder can be the name of the software (plus possibly a very brief description of what it’s for – in case you can’t remember what the software does based in its name). Then you would gather all of these folders together into one place, and call it something like “Software” or “Setups”. If you have enough of these folders (I have several hundred, being a geek, collected over 20 years), then you may want to further categorize them. My own categorization structure is based on “platform” (operating system): The last seven folders each represents one platform/operating system, while _Operating Systems contains set-up files for installing the operating systems themselves. _Hardware contains ROMs for hardware I own, such as routers. Within the Windows folder (above), you can see the beginnings of the vast library of software I’ve compiled over the years: An example of a typical application folder looks like this: Tip #34. Have a “Settings” Folder We all know that our documents are important. So are our photos and music files. We save all of these files into folders, and then locate them afterwards and double-click on them to open them. But there are many files that are important to us that can’t be saved into folders, and then searched for and double-clicked later on. These files certainly contain important information that we need, but are often created internally by an application, and saved wherever that application feels is appropriate. A good example of this is the “PST” file that Outlook creates for us and uses to store all our emails, contacts, appointments and so forth. Another example would be the collection of Bookmarks that Firefox stores on your behalf. And yet another example would be the customized settings and configuration files of our all our software. Granted, most Windows programs store their configuration in the Registry, but there are still many programs that use configuration files to store their settings. Imagine if you lost all of the above files! And yet, when people are backing up their computers, they typically only back up the files they know about – those that are stored in the “My Documents” folder, etc. If they had a hard disk failure or their computer was lost or stolen, their backup files would not include some of the most vital files they owned. Also, when migrating to a new computer, it’s vital to ensure that these files make the journey. It can be a very useful idea to create yourself a folder to store all your “settings” – files that are important to you but which you never actually search for by name and double-click on to open them. Otherwise, next time you go to set up a new computer just the way you want it, you’ll need to spend hours recreating the configuration of your previous computer! So how to we get our important files into this folder? Well, we have a few options: Some programs (such as Outlook and its PST files) allow you to place these files wherever you want. If you delve into the program’s options, you will find a setting somewhere that controls the location of the important settings files (or “personal storage” – PST – when it comes to Outlook) Some programs do not allow you to change such locations in any easy way, but if you get into the Registry, you can sometimes find a registry key that refers to the location of the file(s). Simply move the file into your Settings folder and adjust the registry key to refer to the new location. Some programs stubbornly refuse to allow their settings files to be placed anywhere other then where they stipulate. When faced with programs like these, you have three choices: (1) You can ignore those files, (2) You can copy the files into your Settings folder (let’s face it – settings don’t change very often), or (3) you can use synchronization software, such as the Windows Briefcase, to make synchronized copies of all your files in your Settings folder. All you then have to do is to remember to run your sync software periodically (perhaps just before you run your backup software!). There are some other things you may decide to locate inside this new “Settings” folder: Exports of registry keys (from the many applications that store their configurations in the Registry). This is useful for backup purposes or for migrating to a new computer Notes you’ve made about all the specific customizations you have made to a particular piece of software (so that you’ll know how to do it all again on your next computer) Shortcuts to webpages that detail how to tweak certain aspects of your operating system or applications so they are just the way you like them (such as how to remove the words “Shortcut to” from the beginning of newly created shortcuts). In other words, you’d want to create shortcuts to half the pages on the How-To Geek website! Here’s an example of a “Settings” folder: Windows Features that Help with Organization This section details some of the features of Microsoft Windows that are a boon to anyone hoping to stay optimally organized. Tip #35. Use the “Favorite Links” Area to Access Oft-Used Folders Once you’ve created your great new filing system, work out which folders you access most regularly, or which serve as great starting points for locating the rest of the files in your folder structure, and then put links to those folders in your “Favorite Links” area of the left-hand side of the Windows Explorer window (simply called “Favorites” in Windows 7): Some ideas for folders you might want to add there include: Your “Inbox” folder (or whatever you’ve called it) – most important! The base of your filing structure (e.g. C:\Files) A folder containing shortcuts to often-accessed folders on other computers around the network (shown above as Network Folders) A folder containing shortcuts to your current projects (unless that folder is in your “Inbox” folder) Getting folders into this area is very simple – just locate the folder you’re interested in and drag it there! Tip #36. Customize the Places Bar in the File/Open and File/Save Boxes Consider the screenshot below: The highlighted icons (collectively known as the “Places Bar”) can be customized to refer to any folder location you want, allowing instant access to any part of your organizational structure. Note: These File/Open and File/Save boxes have been superseded by new versions that use the Windows Vista/Windows 7 “Favorite Links”, but the older versions (shown above) are still used by a surprisingly large number of applications. The easiest way to customize these icons is to use the Group Policy Editor, but not everyone has access to this program. If you do, open it up and navigate to: User Configuration > Administrative Templates > Windows Components > Windows Explorer > Common Open File Dialog If you don’t have access to the Group Policy Editor, then you’ll need to get into the Registry. Navigate to: HKEY_CURRENT_USER \ Software \ Microsoft \ Windows \ CurrentVersion \ Policies \ comdlg32 \ Placesbar It should then be easy to make the desired changes. Log off and log on again to allow the changes to take effect. Tip #37. Use the Quick Launch Bar as a Application and File Launcher That Quick Launch bar (to the right of the Start button) is a lot more useful than people give it credit for. Most people simply have half a dozen icons in it, and use it to start just those programs. But it can actually be used to instantly access just about anything in your filing system: For complete instructions on how to set this up, visit our dedicated article on this topic. Tip #38. Put a Shortcut to Windows Explorer into Your Quick Launch Bar This is only necessary in Windows Vista and Windows XP. The Microsoft boffins finally got wise and added it to the Windows 7 Superbar by default. Windows Explorer – the program used for managing your files and folders – is one of the most useful programs in Windows. Anyone who considers themselves serious about being organized needs instant access to this program at any time. A great place to create a shortcut to this program is in the Windows XP and Windows Vista “Quick Launch” bar: To get it there, locate it in your Start Menu (usually under “Accessories”) and then right-drag it down into your Quick Launch bar (and create a copy). Tip #39. Customize the Starting Folder for Your Windows 7 Explorer Superbar Icon If you’re on Windows 7, your Superbar will include a Windows Explorer icon. Clicking on the icon will launch Windows Explorer (of course), and will start you off in your “Libraries” folder. Libraries may be fine as a starting point, but if you have created yourself an “Inbox” folder, then it would probably make more sense to start off in this folder every time you launch Windows Explorer. To change this default/starting folder location, then first right-click the Explorer icon in the Superbar, and then right-click Properties:Then, in Target field of the Windows Explorer Properties box that appears, type %windir%\explorer.exe followed by the path of the folder you wish to start in. For example: %windir%\explorer.exe C:\Files If that folder happened to be on the Desktop (and called, say, “Inbox”), then you would use the following cleverness: %windir%\explorer.exe shell:desktop\Inbox Then click OK and test it out. Tip #40. Ummmmm…. No, that’s it. I can’t think of another one. That’s all of the tips I can come up with. I only created this one because 40 is such a nice round number… Case Study – An Organized PC To finish off the article, I have included a few screenshots of my (main) computer (running Vista). The aim here is twofold: To give you a sense of what it looks like when the above, sometimes abstract, tips are applied to a real-life computer, and To offer some ideas about folders and structure that you may want to steal to use on your own PC. Let’s start with the C: drive itself. Very minimal. All my files are contained within C:\Files. I’ll confine the rest of the case study to this folder: That folder contains the following: Mark: My personal files VC: My business (Virtual Creations, Australia) Others contains files created by friends and family Data contains files from the rest of the world (can be thought of as “public” files, usually downloaded from the Net) Settings is described above in tip #34 The Data folder contains the following sub-folders: Audio: Radio plays, audio books, podcasts, etc Development: Programmer and developer resources, sample source code, etc (see below) Humour: Jokes, funnies (those emails that we all receive) Movies: Downloaded and ripped movies (all legal, of course!), their scripts, DVD covers, etc. Music: (see below) Setups: Installation files for software (explained in full in tip #33) System: (see below) TV: Downloaded TV shows Writings: Books, instruction manuals, etc (see below) The Music folder contains the following sub-folders: Album covers: JPEG scans Guitar tabs: Text files of guitar sheet music Lists: e.g. “Top 1000 songs of all time” Lyrics: Text files MIDI: Electronic music files MP3 (representing 99% of the Music folder): MP3s, either ripped from CDs or downloaded, sorted by artist/album name Music Video: Video clips Sheet Music: usually PDFs The Data\Writings folder contains the following sub-folders: (all pretty self-explanatory) The Data\Development folder contains the following sub-folders: Again, all pretty self-explanatory (if you’re a geek) The Data\System folder contains the following sub-folders: These are usually themes, plug-ins and other downloadable program-specific resources. The Mark folder contains the following sub-folders: From Others: Usually letters that other people (friends, family, etc) have written to me For Others: Letters and other things I have created for other people Green Book: None of your business Playlists: M3U files that I have compiled of my favorite songs (plus one M3U playlist file for every album I own) Writing: Fiction, philosophy and other musings of mine Mark Docs: Shortcut to C:\Users\Mark Settings: Shortcut to C:\Files\Settings\Mark The Others folder contains the following sub-folders: The VC (Virtual Creations, my business – I develop websites) folder contains the following sub-folders: And again, all of those are pretty self-explanatory. Conclusion These tips have saved my sanity and helped keep me a productive geek, but what about you? What tips and tricks do you have to keep your files organized? Please share them with us in the comments. Come on, don’t be shy… Similar Articles Productive Geek Tips Fix For When Windows Explorer in Vista Stops Showing File NamesWhy Did Windows Vista’s Music Folder Icon Turn Yellow?Print or Create a Text File List of the Contents in a Directory the Easy WayCustomize the Windows 7 or Vista Send To MenuAdd Copy To / Move To on Windows 7 or Vista Right-Click Menu TouchFreeze Alternative in AutoHotkey The Icy Undertow Desktop Windows Home Server – Backup to LAN The Clear & Clean Desktop Use This Bookmarklet to Easily Get Albums Use AutoHotkey to Assign a Hotkey to a Specific Window Latest Software Reviews Tinyhacker Random Tips Acronis Online Backup DVDFab 6 Revo Uninstaller Pro Registry Mechanic 9 for Windows Track Daily Goals With 42Goals Video Toolbox is a Superb Online Video Editor Fun with 47 charts and graphs Tomorrow is Mother’s Day Check the Average Speed of YouTube Videos You’ve Watched OutlookStatView Scans and Displays General Usage Statistics

Read the article

Profile System: User share the same id

- by Malcolm Frexner

I have a strange effect on my site when it is under heavy load. I randomly get the properties of other users settings. I have my own implementation of the profile system so I guess I can not blame the profile system itself. I just need a point to start debugging from. I guess there is a cookie-value that maps to an Profile entry somewhere. Is there any chance to see how this mapping works? Here is my profile provider: using System; using System.Text; using System.Configuration; using System.Web; using System.Web.Profile; using System.Collections; using System.Collections.Specialized; using B2CShop.Model; using log4net; using System.Collections.Generic; using System.Diagnostics; using B2CShop.DAL; using B2CShop.Model.RepositoryInterfaces; [assembly: log4net.Config.XmlConfigurator()] namespace B2CShop.Profile { public class B2CShopProfileProvider : ProfileProvider { private static readonly ILog _log = LogManager.GetLogger(typeof(B2CShopProfileProvider)); // Get an instance of the Profile DAL using the ProfileDALFactory private static readonly B2CShop.DAL.UserRepository dal = new B2CShop.DAL.UserRepository(); // Private members private const string ERR_INVALID_PARAMETER = "Invalid Profile parameter:"; private const string PROFILE_USER = "User"; private static string applicationName = B2CShop.Model.Configuration.ApplicationConfiguration.MembershipApplicationName; /// <summary> /// The name of the application using the custom profile provider. /// </summary> public override string ApplicationName { get { return applicationName; } set { applicationName = value; } } /// <summary> /// Initializes the provider. /// </summary> /// <param name="name">The friendly name of the provider.</param> /// <param name="config">A collection of the name/value pairs representing the provider-specific attributes specified in the configuration for this provider.</param> public override void Initialize(string name, NameValueCollection config) { if (config == null) throw new ArgumentNullException("config"); if (string.IsNullOrEmpty(config["description"])) { config.Remove("description"); config.Add("description", "B2C Shop Custom Provider"); } if (string.IsNullOrEmpty(name)) name = "b2c_shop"; if (config["applicationName"] != null && !string.IsNullOrEmpty(config["applicationName"].Trim())) applicationName = config["applicationName"]; base.Initialize(name, config); } /// <summary> /// Returns the collection of settings property values for the specified application instance and settings property group. /// </summary> /// <param name="context">A System.Configuration.SettingsContext describing the current application use.</param> /// <param name="collection">A System.Configuration.SettingsPropertyCollection containing the settings property group whose values are to be retrieved.</param> /// <returns>A System.Configuration.SettingsPropertyValueCollection containing the values for the specified settings property group.</returns> public override SettingsPropertyValueCollection GetPropertyValues(SettingsContext context, SettingsPropertyCollection collection) { string username = (string)context["UserName"]; bool isAuthenticated = (bool)context["IsAuthenticated"]; //if (!isAuthenticated) return null; int uniqueID = dal.GetUniqueID(username, isAuthenticated, false, ApplicationName); SettingsPropertyValueCollection svc = new SettingsPropertyValueCollection(); foreach (SettingsProperty prop in collection) { SettingsPropertyValue pv = new SettingsPropertyValue(prop); switch (pv.Property.Name) { case PROFILE_USER: if (!String.IsNullOrEmpty(username)) { pv.PropertyValue = GetUser(uniqueID); } break; default: throw new ApplicationException(ERR_INVALID_PARAMETER + " name."); } svc.Add(pv); } return svc; } /// <summary> /// Sets the values of the specified group of property settings. /// </summary> /// <param name="context">A System.Configuration.SettingsContext describing the current application usage.</param> /// <param name="collection">A System.Configuration.SettingsPropertyValueCollection representing the group of property settings to set.</param> public override void SetPropertyValues(SettingsContext context, SettingsPropertyValueCollection collection) { string username = (string)context["UserName"]; CheckUserName(username); bool isAuthenticated = (bool)context["IsAuthenticated"]; int uniqueID = dal.GetUniqueID(username, isAuthenticated, false, ApplicationName); if (uniqueID == 0) { uniqueID = dal.CreateProfileForUser(username, isAuthenticated, ApplicationName); } foreach (SettingsPropertyValue pv in collection) { if (pv.PropertyValue != null) { switch (pv.Property.Name) { case PROFILE_USER: SetUser(uniqueID, (UserInfo)pv.PropertyValue); break; default: throw new ApplicationException(ERR_INVALID_PARAMETER + " name."); } } } UpdateActivityDates(username, false); } // Profile gettters // Retrieve UserInfo private static UserInfo GetUser(int userID) { return dal.GetUser(userID); } // Update account info private static void SetUser(int uniqueID, UserInfo user) { user.UserID = uniqueID; dal.SetUser(user); } // UpdateActivityDates // Updates the LastActivityDate and LastUpdatedDate values // when profile properties are accessed by the // GetPropertyValues and SetPropertyValues methods. // Passing true as the activityOnly parameter will update // only the LastActivityDate. private static void UpdateActivityDates(string username, bool activityOnly) { dal.UpdateActivityDates(username, activityOnly, applicationName); } /// <summary> /// Deletes profile properties and information for the supplied list of profiles. /// </summary> /// <param name="profiles">A System.Web.Profile.ProfileInfoCollection of information about profiles that are to be deleted.</param> /// <returns>The number of profiles deleted from the data source.</returns> public override int DeleteProfiles(ProfileInfoCollection profiles) { int deleteCount = 0; foreach (ProfileInfo p in profiles) if (DeleteProfile(p.UserName)) deleteCount++; return deleteCount; } /// <summary> /// Deletes profile properties and information for profiles that match the supplied list of user names. /// </summary> /// <param name="usernames">A string array of user names for profiles to be deleted.</param> /// <returns>The number of profiles deleted from the data source.</returns> public override int DeleteProfiles(string[] usernames) { int deleteCount = 0; foreach (string user in usernames) if (DeleteProfile(user)) deleteCount++; return deleteCount; } // DeleteProfile // Deletes profile data from the database for the specified user name. private static bool DeleteProfile(string username) { CheckUserName(username); return dal.DeleteAnonymousProfile(username, applicationName); } // Verifies user name for sise and comma private static void CheckUserName(string userName) { if (string.IsNullOrEmpty(userName) || userName.Length > 256 || userName.IndexOf(",") > 0) throw new ApplicationException(ERR_INVALID_PARAMETER + " user name."); } /// <summary> /// Deletes all user-profile data for profiles in which the last activity date occurred before the specified date. /// </summary> /// <param name="authenticationOption">One of the System.Web.Profile.ProfileAuthenticationOption values, specifying whether anonymous, authenticated, or both types of profiles are deleted.</param> /// <param name="userInactiveSinceDate">A System.DateTime that identifies which user profiles are considered inactive. If the System.Web.Profile.ProfileInfo.LastActivityDate value of a user profile occurs on or before this date and time, the profile is considered inactive.</param> /// <returns>The number of profiles deleted from the data source.</returns> public override int DeleteInactiveProfiles(ProfileAuthenticationOption authenticationOption, DateTime userInactiveSinceDate) { string[] userArray = new string[0]; dal.GetInactiveProfiles((int)authenticationOption, userInactiveSinceDate, ApplicationName).CopyTo(userArray, 0); return DeleteProfiles(userArray); } /// <summary> /// Retrieves profile information for profiles in which the user name matches the specified user names. /// </summary> /// <param name="authenticationOption">One of the System.Web.Profile.ProfileAuthenticationOption values, specifying whether anonymous, authenticated, or both types of profiles are returned.</param> /// <param name="usernameToMatch">The user name to search for.</param> /// <param name="pageIndex">The index of the page of results to return.</param> /// <param name="pageSize">The size of the page of results to return.</param> /// <param name="totalRecords">When this method returns, contains the total number of profiles.</param> /// <returns>A System.Web.Profile.ProfileInfoCollection containing user-profile information // for profiles where the user name matches the supplied usernameToMatch parameter.</returns> public override ProfileInfoCollection FindProfilesByUserName(ProfileAuthenticationOption authenticationOption, string usernameToMatch, int pageIndex, int pageSize, out int totalRecords) { CheckParameters(pageIndex, pageSize); return GetProfileInfo(authenticationOption, usernameToMatch, null, pageIndex, pageSize, out totalRecords); } /// <summary> /// Retrieves profile information for profiles in which the last activity date occurred on or before the specified date and the user name matches the specified user name. /// </summary> /// <param name="authenticationOption">One of the System.Web.Profile.ProfileAuthenticationOption values, specifying whether anonymous, authenticated, or both types of profiles are returned.</param> /// <param name="usernameToMatch">The user name to search for.</param> /// <param name="userInactiveSinceDate">A System.DateTime that identifies which user profiles are considered inactive. If the System.Web.Profile.ProfileInfo.LastActivityDate value of a user profile occurs on or before this date and time, the profile is considered inactive.</param> /// <param name="pageIndex">The index of the page of results to return.</param> /// <param name="pageSize">The size of the page of results to return.</param> /// <param name="totalRecords">When this method returns, contains the total number of profiles.</param> /// <returns>A System.Web.Profile.ProfileInfoCollection containing user profile information for inactive profiles where the user name matches the supplied usernameToMatch parameter.</returns> public override ProfileInfoCollection FindInactiveProfilesByUserName(ProfileAuthenticationOption authenticationOption, string usernameToMatch, DateTime userInactiveSinceDate, int pageIndex, int pageSize, out int totalRecords) { CheckParameters(pageIndex, pageSize); return GetProfileInfo(authenticationOption, usernameToMatch, userInactiveSinceDate, pageIndex, pageSize, out totalRecords); } /// <summary> /// Retrieves user profile data for all profiles in the data source. /// </summary> /// <param name="authenticationOption">One of the System.Web.Profile.ProfileAuthenticationOption values, specifying whether anonymous, authenticated, or both types of profiles are returned.</param> /// <param name="pageIndex">The index of the page of results to return.</param> /// <param name="pageSize">The size of the page of results to return.</param> /// <param name="totalRecords">When this method returns, contains the total number of profiles.</param> /// <returns>A System.Web.Profile.ProfileInfoCollection containing user-profile information for all profiles in the data source.</returns> public override ProfileInfoCollection GetAllProfiles(ProfileAuthenticationOption authenticationOption, int pageIndex, int pageSize, out int totalRecords) { CheckParameters(pageIndex, pageSize); return GetProfileInfo(authenticationOption, null, null, pageIndex, pageSize, out totalRecords); } /// <summary> /// Retrieves user-profile data from the data source for profiles in which the last activity date occurred on or before the specified date. /// </summary> /// <param name="authenticationOption">One of the System.Web.Profile.ProfileAuthenticationOption values, specifying whether anonymous, authenticated, or both types of profiles are returned.</param> /// <param name="userInactiveSinceDate">A System.DateTime that identifies which user profiles are considered inactive. If the System.Web.Profile.ProfileInfo.LastActivityDate of a user profile occurs on or before this date and time, the profile is considered inactive.</param> /// <param name="pageIndex">The index of the page of results to return.</param> /// <param name="pageSize">The size of the page of results to return.</param> /// <param name="totalRecords">When this method returns, contains the total number of profiles.</param> /// <returns>A System.Web.Profile.ProfileInfoCollection containing user-profile information about the inactive profiles.</returns> public override ProfileInfoCollection GetAllInactiveProfiles(ProfileAuthenticationOption authenticationOption, DateTime userInactiveSinceDate, int pageIndex, int pageSize, out int totalRecords) { CheckParameters(pageIndex, pageSize); return GetProfileInfo(authenticationOption, null, userInactiveSinceDate, pageIndex, pageSize, out totalRecords); } /// <summary> /// Returns the number of profiles in which the last activity date occurred on or before the specified date. /// </summary> /// <param name="authenticationOption">One of the System.Web.Profile.ProfileAuthenticationOption values, specifying whether anonymous, authenticated, or both types of profiles are returned.</param> /// <param name="userInactiveSinceDate">A System.DateTime that identifies which user profiles are considered inactive. If the System.Web.Profile.ProfileInfo.LastActivityDate of a user profile occurs on or before this date and time, the profile is considered inactive.</param> /// <returns>The number of profiles in which the last activity date occurred on or before the specified date.</returns> public override int GetNumberOfInactiveProfiles(ProfileAuthenticationOption authenticationOption, DateTime userInactiveSinceDate) { int inactiveProfiles = 0; ProfileInfoCollection profiles = GetProfileInfo(authenticationOption, null, userInactiveSinceDate, 0, 0, out inactiveProfiles); return inactiveProfiles; } //Verifies input parameters for page size and page index. private static void CheckParameters(int pageIndex, int pageSize) { if (pageIndex < 1 || pageSize < 1) throw new ApplicationException(ERR_INVALID_PARAMETER + " page index."); } //GetProfileInfo //Retrieves a count of profiles and creates a //ProfileInfoCollection from the profile data in the //database. Called by GetAllProfiles, GetAllInactiveProfiles, //FindProfilesByUserName, FindInactiveProfilesByUserName, //and GetNumberOfInactiveProfiles. //Specifying a pageIndex of 0 retrieves a count of the results only. private static ProfileInfoCollection GetProfileInfo(ProfileAuthenticationOption authenticationOption, string usernameToMatch, object userInactiveSinceDate, int pageIndex, int pageSize, out int totalRecords) { ProfileInfoCollection profiles = new ProfileInfoCollection(); totalRecords = 0; // Count profiles only. if (pageSize == 0) return profiles; int counter = 0; int startIndex = pageSize * (pageIndex - 1); int endIndex = startIndex + pageSize - 1; DateTime dt = new DateTime(1900, 1, 1); if (userInactiveSinceDate != null) dt = (DateTime)userInactiveSinceDate; /* foreach(CustomProfileInfo profile in dal.GetProfileInfo((int)authenticationOption, usernameToMatch, dt, applicationName, out totalRecords)) { if(counter >= startIndex) { ProfileInfo p = new ProfileInfo(profile.UserName, profile.IsAnonymous, profile.LastActivityDate, profile.LastUpdatedDate, 0); profiles.Add(p); } if(counter >= endIndex) { break; } counter++; } */ return profiles; } } } This is how I use it in the controller: public ActionResult AddTyreToCart(CartViewModel model) { string profile = Request.IsAuthenticated ? Request.AnonymousID : User.Identity.Name; } I would like to debug: How can 2 users who provide different cookies get the same profileid? EDIT Here is the code for getuniqueid public int GetUniqueID(string userName, bool isAuthenticated, bool ignoreAuthenticationType, string appName) { SqlParameter[] parms = { new SqlParameter("@Username", SqlDbType.VarChar, 256), new SqlParameter("@ApplicationName", SqlDbType.VarChar, 256)}; parms[0].Value = userName; parms[1].Value = appName; if (!ignoreAuthenticationType) { Array.Resize(ref parms, parms.Length + 1); parms[2] = new SqlParameter("@IsAnonymous", SqlDbType.Bit) { Value = !isAuthenticated }; } int userID; object retVal = null; retVal = SqlHelper.ExecuteScalar(ConfigurationManager.ConnectionStrings["SQLOrderB2CConnString"].ConnectionString, CommandType.StoredProcedure, "getProfileUniqueID", parms); if (retVal == null) userID = CreateProfileForUser(userName, isAuthenticated, appName); else userID = Convert.ToInt32(retVal); return userID; } And this is the SP: CREATE PROCEDURE [dbo].[getProfileUniqueID] @Username VarChar( 256), @ApplicationName VarChar( 256), @IsAnonymous bit = null AS BEGIN SET NOCOUNT ON; /* [getProfileUniqueID] created 08.07.2009 mf Retrive unique id for current user */ SELECT UniqueID FROM dbo.Profiles WHERE Username = @Username AND ApplicationName = @ApplicationName AND IsAnonymous = @IsAnonymous or @IsAnonymous = null END

Read the article

c# Truncate HTML safely for article summary

- by WickedW

Hi All, Does anyone have a c# variation of this? This is so I can take some html and display it without breaking as a summary lead in to an article? http://stackoverflow.com/questions/1193500/php-truncate-html-ignoring-tags Save me from reinventing the wheel! Thank you very much ---------- edit ------------------ Sorry, new here, and your right, should have phrased the question better, heres a bit more info I wish to take a html string and truncate it to a set number of words (or even char length) so I can then show the start of it as a summary (which then leads to the main article). I wish to preserve the html so I can show the links etc in preview. The main issue I have to solve is the fact that we may well end up with unclosed html tags if we truncate in the middle of 1 or more tags! The idea I have for solution is to a) truncate the html to N words (words better but chars ok) first (be sure not to stop in the middle of a tag and truncate a require attribute) b) work through the opened html tags in this truncated string (maybe stick them on stack as I go?) c) then work through the closing tags and ensure they match the ones on stack as I pop them off? d) if any open tags left on stack after this, then write them to end of truncated string and html should be good to go!!!! -- edit 12112009 Here is what I have bumbled together so far as a unittest file in VS2008, this 'may' help someone in future My hack attempts based on Jan code are at top for char version + word version (DISCLAIMER: this is dirty rough code!! on my part) I assume working with 'well-formed' HTML in all cases (but not necessarily a full document with a root node as per XML version) Abels XML version is at bottom, but not yet got round to fully getting tests to run on this yet (plus need to understand the code) ... I will update when I get chance to refine having trouble with posting code? is there no upload facility on stack? Thanks for all comments :) using System; using System.Collections.Generic; using System.Text.RegularExpressions; using System.Xml; using System.Xml.XPath; using Microsoft.VisualStudio.TestTools.UnitTesting; namespace PINET40TestProject { [TestClass] public class UtilityUnitTest { public static string TruncateHTMLSafeishChar(string text, int charCount) { bool inTag = false; int cntr = 0; int cntrContent = 0; // loop through html, counting only viewable content foreach (Char c in text) { if (cntrContent == charCount) break; cntr++; if (c == '<') { inTag = true; continue; } if (c == '>') { inTag = false; continue; } if (!inTag) cntrContent++; } string substr = text.Substring(0, cntr); //search for nonclosed tags MatchCollection openedTags = new Regex("<[^/](.|\n)*?>").Matches(substr); MatchCollection closedTags = new Regex("<[/](.|\n)*?>").Matches(substr); // create stack Stack<string> opentagsStack = new Stack<string>(); Stack<string> closedtagsStack = new Stack<string>(); // to be honest, this seemed like a good idea then I got lost along the way // so logic is probably hanging by a thread!! foreach (Match tag in openedTags) { string openedtag = tag.Value.Substring(1, tag.Value.Length - 2); // strip any attributes, sure we can use regex for this! if (openedtag.IndexOf(" ") >= 0) { openedtag = openedtag.Substring(0, openedtag.IndexOf(" ")); } // ignore brs as self-closed if (openedtag.Trim() != "br") { opentagsStack.Push(openedtag); } } foreach (Match tag in closedTags) { string closedtag = tag.Value.Substring(2, tag.Value.Length - 3); closedtagsStack.Push(closedtag); } if (closedtagsStack.Count < opentagsStack.Count) { while (opentagsStack.Count > 0) { string tagstr = opentagsStack.Pop(); if (closedtagsStack.Count == 0 || tagstr != closedtagsStack.Peek()) { substr += "</" + tagstr + ">"; } else { closedtagsStack.Pop(); } } } return substr; } public static string TruncateHTMLSafeishWord(string text, int wordCount) { bool inTag = false; int cntr = 0; int cntrWords = 0; Char lastc = ' '; // loop through html, counting only viewable content foreach (Char c in text) { if (cntrWords == wordCount) break; cntr++; if (c == '<') { inTag = true; continue; } if (c == '>') { inTag = false; continue; } if (!inTag) { // do not count double spaces, and a space not in a tag counts as a word if (c == 32 && lastc != 32) cntrWords++; } } string substr = text.Substring(0, cntr) + " ..."; //search for nonclosed tags MatchCollection openedTags = new Regex("<[^/](.|\n)*?>").Matches(substr); MatchCollection closedTags = new Regex("<[/](.|\n)*?>").Matches(substr); // create stack Stack<string> opentagsStack = new Stack<string>(); Stack<string> closedtagsStack = new Stack<string>(); foreach (Match tag in openedTags) { string openedtag = tag.Value.Substring(1, tag.Value.Length - 2); // strip any attributes, sure we can use regex for this! if (openedtag.IndexOf(" ") >= 0) { openedtag = openedtag.Substring(0, openedtag.IndexOf(" ")); } // ignore brs as self-closed if (openedtag.Trim() != "br") { opentagsStack.Push(openedtag); } } foreach (Match tag in closedTags) { string closedtag = tag.Value.Substring(2, tag.Value.Length - 3); closedtagsStack.Push(closedtag); } if (closedtagsStack.Count < opentagsStack.Count) { while (opentagsStack.Count > 0) { string tagstr = opentagsStack.Pop(); if (closedtagsStack.Count == 0 || tagstr != closedtagsStack.Peek()) { substr += "</" + tagstr + ">"; } else { closedtagsStack.Pop(); } } } return substr; } public static string TruncateHTMLSafeishCharXML(string text, int charCount) { // your data, probably comes from somewhere, or as params to a methodint XmlDocument xml = new XmlDocument(); xml.LoadXml(text); // create a navigator, this is our primary tool XPathNavigator navigator = xml.CreateNavigator(); XPathNavigator breakPoint = null; // find the text node we need: while (navigator.MoveToFollowing(XPathNodeType.Text)) { string lastText = navigator.Value.Substring(0, Math.Min(charCount, navigator.Value.Length)); charCount -= navigator.Value.Length; if (charCount <= 0) { // truncate the last text. Here goes your "search word boundary" code: navigator.SetValue(lastText); breakPoint = navigator.Clone(); break; } } // first remove text nodes, because Microsoft unfortunately merges them without asking while (navigator.MoveToFollowing(XPathNodeType.Text)) { if (navigator.ComparePosition(breakPoint) == XmlNodeOrder.After) { navigator.DeleteSelf(); } } // moves to parent, then move the rest navigator.MoveTo(breakPoint); while (navigator.MoveToFollowing(XPathNodeType.Element)) { if (navigator.ComparePosition(breakPoint) == XmlNodeOrder.After) { navigator.DeleteSelf(); } } // moves to parent // then remove *all* empty nodes to clean up (not necessary): // TODO, add empty elements like <br />, <img /> as exclusion navigator.MoveToRoot(); while (navigator.MoveToFollowing(XPathNodeType.Element)) { while (!navigator.HasChildren && (navigator.Value ?? "").Trim() == "") { navigator.DeleteSelf(); } } // moves to parent navigator.MoveToRoot(); return navigator.InnerXml; } [TestMethod] public void TestTruncateHTMLSafeish() { // Case where we just make it to start of HREF (so effectively an empty link) // 'simple' nested none attributed tags Assert.AreEqual(@"<h1>1234</h1><b><i>56789</i>012</b>", TruncateHTMLSafeishChar( @"<h1>1234</h1><b><i>56789</i>012345</b>", 12)); // In middle of a! Assert.AreEqual(@"<h1>1234</h1><a href=""testurl""><b>567</b></a>", TruncateHTMLSafeishChar( @"<h1>1234</h1><a href=""testurl""><b>5678</b></a><i><strong>some italic nested in string</strong></i>", 7)); // more Assert.AreEqual(@"<div><b><i><strong>1</strong></i></b></div>", TruncateHTMLSafeishChar( @"<div><b><i><strong>12</strong></i></b></div>", 1)); // br Assert.AreEqual(@"<h1>1 3 5</h1><br />6", TruncateHTMLSafeishChar( @"<h1>1 3 5</h1><br />678<br />", 6)); } [TestMethod] public void TestTruncateHTMLSafeishWord() { // zero case Assert.AreEqual(@" ...", TruncateHTMLSafeishWord( @"", 5)); // 'simple' nested none attributed tags Assert.AreEqual(@"<h1>one two <br /></h1><b><i>three ...</i></b>", TruncateHTMLSafeishWord( @"<h1>one two <br /></h1><b><i>three </i>four</b>", 3), "we have added ' ...' to end of summary"); // In middle of a! Assert.AreEqual(@"<h1>one two three </h1><a href=""testurl""><b class=""mrclass"">four ...</b></a>", TruncateHTMLSafeishWord( @"<h1>one two three </h1><a href=""testurl""><b class=""mrclass"">four five </b></a><i><strong>some italic nested in string</strong></i>", 4)); // start of h1 Assert.AreEqual(@"<h1>one two three ...</h1>", TruncateHTMLSafeishWord( @"<h1>one two three </h1><a href=""testurl""><b>four five </b></a><i><strong>some italic nested in string</strong></i>", 3)); // more than words available Assert.AreEqual(@"<h1>one two three </h1><a href=""testurl""><b>four five </b></a><i><strong>some italic nested in string</strong></i> ...", TruncateHTMLSafeishWord( @"<h1>one two three </h1><a href=""testurl""><b>four five </b></a><i><strong>some italic nested in string</strong></i>", 99)); } [TestMethod] public void TestTruncateHTMLSafeishWordXML() { // zero case Assert.AreEqual(@" ...", TruncateHTMLSafeishWord( @"", 5)); // 'simple' nested none attributed tags string output = TruncateHTMLSafeishCharXML( @"<body><h1>one two </h1><b><i>three </i>four</b></body>", 13); Assert.AreEqual(@"<body>\r\n <h1>one two </h1>\r\n <b>\r\n <i>three</i>\r\n </b>\r\n</body>", output, "XML version, no ... yet and addeds '\r\n + spaces?' to format document"); // In middle of a! Assert.AreEqual(@"<h1>one two three </h1><a href=""testurl""><b class=""mrclass"">four ...</b></a>", TruncateHTMLSafeishCharXML( @"<body><h1>one two three </h1><a href=""testurl""><b class=""mrclass"">four five </b></a><i><strong>some italic nested in string</strong></i></body>", 4)); // start of h1 Assert.AreEqual(@"<h1>one two three ...</h1>", TruncateHTMLSafeishCharXML( @"<h1>one two three </h1><a href=""testurl""><b>four five </b></a><i><strong>some italic nested in string</strong></i>", 3)); // more than words available Assert.AreEqual(@"<h1>one two three </h1><a href=""testurl""><b>four five </b></a><i><strong>some italic nested in string</strong></i> ...", TruncateHTMLSafeishCharXML( @"<h1>one two three </h1><a href=""testurl""><b>four five </b></a><i><strong>some italic nested in string</strong></i>", 99)); } } }

Read the article

Delphi - TPerlRegEx / RegExBuddy Problem

- by Brad

I've got a problem with RegEx and Delphi 2k9 (Win32). I get the following Error: First chance exception at $7C812AFB. Exception class Exception with message 'TPerlRegEx.Compile() - Please specify a regular expression in RegEx first'. I've got the latest version of TPerlRegEx from the website. Using its defualt settings (Using DLL) I'm including demo source code. It's using the code generated by RegExBuddy, latest version. http://www.4shared.com/file/236428923/97478b61/googleresultstestdata.html http://www.4shared.com/file/236439483/e0acbe6d/Unit2.html Delphi FORM http://www.4shared.com/file/236439473/6734a2a2/Unit2.html Delphi PAS Thanks for any help -Brad Data is from Google External Keyword Tool RegEx could use some refinement... but works in RegExBuddy not in Delphi unit Unit2; interface uses Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms, Dialogs, StdCtrls, PerlRegEx; type TForm2 = class(TForm) Memo1: TMemo; Memo2: TMemo; Button1: TButton; procedure Button1Click(Sender: TObject); private { Private declarations } public { Public declarations } end; var Form2: TForm2; implementation {$R *.dfm} procedure TForm2.Button1Click(Sender: TObject); var Regex: TPerlRegEx; GroupIndex: Integer; begin Regex := TPerlRegEx.Create(nil); Regex.RegEx := 'criteria\.push\(new kpCriterion\('(?P<keyword>(.*?))', (?P<number1>(.*?)),'#13#10'''(?P<localsearch>(.*?))'', ''(?P<globalsearch>(.*?))'', (?P<localsearchnum>(.*?)), (?P<globalsearchnum>(.*?)), (.*+)'#13#10','#13#10''\$(?P<price>(.*?))', (?P<number2>(.*?)),'#13#10''(?P<range>(.*?))', (?P<number3>(.*+))'; Regex.Options := [preMultiLine]; Regex.Subject := memo1.text; if Regex.Match then begin memo2.Lines.Add('Matches Found'); repeat for GroupIndex := 0 to Regex.SubExpressionCount do begin memo2.lines.add( Regex.SubExpressions[GroupIndex]); //Add Results to memo // backreference text: Regex.SubExpressions[GroupIndex]; // backreference start: Regex.SubExpressionOffsets[GroupIndex]; // backreference length: Regex.SubExpressionLengths[GroupIndex]; end; until not Regex.MatchAgain; end else memo2.Lines.Add('No-Matches Found'); end; end. DFM object Form2: TForm2 Left = 0 Top = 0 Caption = 'Form2' ClientHeight = 247 ClientWidth = 480 Color = clBtnFace Font.Charset = DEFAULT_CHARSET Font.Color = clWindowText Font.Height = -11 Font.Name = 'Tahoma' Font.Style = [] OldCreateOrder = False PixelsPerInch = 96 TextHeight = 13 object Memo1: TMemo Left = 8 Top = 8 Width = 185 Height = 89 Lines.Strings = ( 'var showImpressions = false; var ' 'criteriaSuggestor = ' ''sensei_keyword'; var ' 'historicalTimePeriod = 'Mar ' '2009 - Feb 2010'; var ' 'historicalStartMonth = 2; var ' 'impressionTimePeriod = ' ''February'; var ' 'criteriaGroupsArray = new Array(); ' 'var captchaError = false; var ' 'quotaExceeded = false;' 'var criteria = new Array();' 'var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.52' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.67' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.82' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.73' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.5' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.45' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.45' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.43' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.4' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.47' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.45' '));' 'criteria.push(new ' 'kpCriterion('thunderstorm&#3' '9;, 1.9117305278778076,' #39'201,000'#39', '#39'550,000'#39', 201000, ' '550000, 0.8666667' ',' ''$0.49', 493102,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '5' ',' '''' ',' 'kpView.MATCH_BROAD' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.57' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.7' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.57' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.45' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.42' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.47' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.46' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.43' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.36' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.45' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.43' '));' 'criteria.push(new ' 'kpCriterion('[thunderstorm]&' '#39;, 1.9117305278778076,' #39'33,100'#39', '#39'90,500'#39', 33100, 90500, ' '0.8666667' ',' ''$0.49', 493102,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '3' ',' '''' ',' 'kpView.MATCH_EXACT' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.52' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.67' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.82' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.73' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.5' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.45' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.45' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.43' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.4' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.47' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.45' '));' 'criteria.push(new ' 'kpCriterion('\42thunderstorm\' '042', 1.9117305278778076,' #39'201,000'#39', '#39'450,000'#39', 201000, ' '450000, 0.8666667' ',' ''$0.49', 493102,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '5' ',' '''' ',' 'kpView.MATCH_PHRASE' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.75' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.81' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.87' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.64' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.56' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.52' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.6' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.53' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.47' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.58' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.61' '));' 'criteria.push(new ' 'kpCriterion('thunderstorms&#' '39;, 1.8268921375274658,' #39'110,000'#39', '#39'201,000'#39', 110000, ' '201000, 0.8' ',' ''$0.56', 559074,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '4' ',' '''' ',' 'kpView.MATCH_BROAD' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.83' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.82' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.67' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.42' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.41' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.47' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.56' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.47' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.39' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.5' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.51' '));' 'criteria.push(new ' 'kpCriterion('[thunderstorms]&' '#39;, 1.8268921375274658,' #39'22,200'#39', '#39'40,500'#39', 22200, 40500, ' '0.8' ',' ''$0.56', 559074,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '4' ',' '''' ',' 'kpView.MATCH_EXACT' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.75' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.81' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.87' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.64' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.56' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.52' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.6' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.53' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.47' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.58' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.61' '));' 'criteria.push(new ' 'kpCriterion('\42thunderstorms' '\042', 1.8268921375274658,' #39'110,000'#39', '#39'165,000'#39', 110000, ' '165000, 0.8' ',' ''$0.56', 559074,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '4' ',' '''' ',' 'kpView.MATCH_PHRASE' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.71' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.73' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.82' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.87' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.92' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.82' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.7' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.75' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.68' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.77' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.79' '));' 'criteria.push(new ' 'kpCriterion('lightning ' 'storm', 1.774579644203186,' #39'49,500'#39', '#39'90,500'#39', 49500, 90500, ' '0.73333335' ',' ''$0.54', 535666,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '5' ',' '''' ',' 'kpView.MATCH_BROAD' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.76' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.87' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.97' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.87' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.98' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.87' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.84' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.68' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.86' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.81' '));' 'criteria.push(new ' 'kpCriterion('[lightning ' 'storm]', 1.774579644203186,' #39'12,100'#39', '#39'22,200'#39', 12100, 22200, ' '0.73333335' ',' ''$0.54', 535666,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '5' ',' '''' ',' 'kpView.MATCH_EXACT' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.68' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.72' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.81' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.85' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.92' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.81' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.67' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.71' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.65' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.76' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.73' '));' 'criteria.push(new ' 'kpCriterion('\42lightning ' 'storm\042', ' '1.774579644203186,' #39'33,100'#39', '#39'60,500'#39', 33100, 60500, ' '0.73333335' ',' ''$0.54', 535666,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '5' ',' '''' ',' 'kpView.MATCH_PHRASE' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.69' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.69' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.71' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.66' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.68' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.7' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.75' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.79' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.74' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.72' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.7' '));' 'criteria.push(new ' 'kpCriterion('rain storm', ' '1.7464053630828857,' #39'27,100'#39', '#39'49,500'#39', 27100, 49500, ' '0.6666667' ',' ''$0.53', 526334,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '0' ',' '''' ',' 'kpView.MATCH_BROAD' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.87' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.79' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.57' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.55' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.57' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.74' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.76' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.69' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.61' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.89' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.73' '));' 'criteria.push(new ' 'kpCriterion('[rain ' 'storm]', ' '1.7464053630828857,' #39'5,400'#39', '#39'8,100'#39', 5400, 8100, ' '0.6666667' ',' ''$0.53', 526334,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '2' ',' '''' ',' 'kpView.MATCH_EXACT' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.73' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.7' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.68' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.61' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.68' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.69' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.73' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.72' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.62' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.59' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.66' '));' 'criteria.push(new ' 'kpCriterion('\42rain ' 'storm\042', ' '1.7464053630828857,' #39'14,800'#39', '#39'27,100'#39', 14800, 27100, ' '0.6666667' ',' ''$0.53', 526334,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '0' ',' '''' ',' 'kpView.MATCH_PHRASE' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.82' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.87' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.78' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.82' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.84' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.79' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.77' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.61' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.92' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.82' '));' 'criteria.push(new ' 'kpCriterion('lightning ' 'storms', ' '1.6842896938323975,' #39'14,800'#39', '#39'27,100'#39', 14800, 27100, ' '0.73333335' ',' ''$0.42', 417108,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '4' ',' '''' ',' 'kpView.MATCH_BROAD' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.9' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.9' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.84' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.7' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.81' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.88' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.77' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.76' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.57' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.75' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.63' '));' 'criteria.push(new ' 'kpCriterion('[lightning ' 'storms]', ' '1.6842896938323975,' #39'3,600'#39', '#39'8,100'#39', 3600, 8100, ' '0.73333335' ',' ''$0.42', 417108,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '4' ',' '''' ',' 'kpView.MATCH_EXACT' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.8' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.86' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.99' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.77' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.83' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.85' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.78' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.77' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.6' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.91' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.81' '));' 'criteria.push(new ' 'kpCriterion('\42lightning ' 'storms\042', ' '1.6842896938323975,' #39'12,100'#39', '#39'22,200'#39', 12100, 22200, ' '0.73333335' ',' ''$0.42', 417108,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '4' ',' '''' ',' 'kpView.MATCH_PHRASE' ',' '0' ')); var monthlyVariation =

Read the article

Delphi - TPerlRegEx / RegExBuddy Problem

- by Brad

I've got a problem with RegEx and Delphi 2k9 (Win32). I get the following Error: First chance exception at $7C812AFB. Exception class Exception with message 'TPerlRegEx.Compile() - Please specify a regular expression in RegEx first'. I've got the latest version of TPerlRegEx from the website. Using its defualt settings (Using DLL) I'm including demo source code. It's using the code generated by RegExBuddy, latest version. http://www.4shared.com/file/236428923/97478b61/googleresultstestdata.html http://www.4shared.com/file/236439483/e0acbe6d/Unit2.html Delphi FORM http://www.4shared.com/file/236439473/6734a2a2/Unit2.html Delphi PAS Thanks for any help -Brad Data is from Google External Keyword Tool RegEx could use some refinement... but works in RegExBuddy not in Delphi unit Unit2; interface uses Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms, Dialogs, StdCtrls, PerlRegEx; type TForm2 = class(TForm) Memo1: TMemo; Memo2: TMemo; Button1: TButton; procedure Button1Click(Sender: TObject); private { Private declarations } public { Public declarations } end; var Form2: TForm2; implementation {$R *.dfm} procedure TForm2.Button1Click(Sender: TObject); var Regex: TPerlRegEx; GroupIndex: Integer; begin Regex := TPerlRegEx.Create(nil); Regex.RegEx := 'criteria.push(new kpCriterion('(?P(.?))', (?P(.?)),'#13#10'''(?P(.?))'', ''(?P(.?))'', (?P(.?)), (?P(.?)), (.+)'#13#10','#13#10''\$(?P(.?))', (?P(.?)),'#13#10''(?P(.?))', (?P(.*+))'; Regex.Options := [preMultiLine]; Regex.Subject := memo1.text; if Regex.Match then begin memo2.Lines.Add('Matches Found'); repeat for GroupIndex := 0 to Regex.SubExpressionCount do begin memo2.lines.add( Regex.SubExpressions[GroupIndex]); //Add Results to memo // backreference text: Regex.SubExpressions[GroupIndex]; // backreference start: Regex.SubExpressionOffsets[GroupIndex]; // backreference length: Regex.SubExpressionLengths[GroupIndex]; end; until not Regex.MatchAgain; end else memo2.Lines.Add('No-Matches Found'); end; end. DFM object Form2: TForm2 Left = 0 Top = 0 Caption = 'Form2' ClientHeight = 247 ClientWidth = 480 Color = clBtnFace Font.Charset = DEFAULT_CHARSET Font.Color = clWindowText Font.Height = -11 Font.Name = 'Tahoma' Font.Style = [] OldCreateOrder = False PixelsPerInch = 96 TextHeight = 13 object Memo1: TMemo Left = 8 Top = 8 Width = 185 Height = 89 Lines.Strings = ( 'var showImpressions = false; var ' 'criteriaSuggestor = ' ''sensei_keyword'; var ' 'historicalTimePeriod = 'Mar ' '2009 - Feb 2010'; var ' 'historicalStartMonth = 2; var ' 'impressionTimePeriod = ' ''February'; var ' 'criteriaGroupsArray = new Array(); ' 'var captchaError = false; var ' 'quotaExceeded = false;' 'var criteria = new Array();' 'var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.52' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.67' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.82' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.73' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.5' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.45' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.45' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.43' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.4' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.47' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.45' '));' 'criteria.push(new ' 'kpCriterion('thunderstorm' '9;, 1.9117305278778076,' #39'201,000'#39', '#39'550,000'#39', 201000, ' '550000, 0.8666667' ',' ''$0.49', 493102,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '5' ',' '''' ',' 'kpView.MATCH_BROAD' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.57' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.7' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.57' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.45' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.42' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.47' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.46' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.43' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.36' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.45' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.43' '));' 'criteria.push(new ' 'kpCriterion('[thunderstorm]&' '#39;, 1.9117305278778076,' #39'33,100'#39', '#39'90,500'#39', 33100, 90500, ' '0.8666667' ',' ''$0.49', 493102,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '3' ',' '''' ',' 'kpView.MATCH_EXACT' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.52' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.67' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.82' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.73' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.5' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.45' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.45' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.43' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.4' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.47' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.45' '));' 'criteria.push(new ' 'kpCriterion('\42thunderstorm\' '042', 1.9117305278778076,' #39'201,000'#39', '#39'450,000'#39', 201000, ' '450000, 0.8666667' ',' ''$0.49', 493102,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '5' ',' '''' ',' 'kpView.MATCH_PHRASE' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.75' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.81' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.87' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.64' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.56' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.52' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.6' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.53' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.47' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.58' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.61' '));' 'criteria.push(new ' 'kpCriterion('thunderstorms&#' '39;, 1.8268921375274658,' #39'110,000'#39', '#39'201,000'#39', 110000, ' '201000, 0.8' ',' ''$0.56', 559074,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '4' ',' '''' ',' 'kpView.MATCH_BROAD' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.83' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.82' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.67' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.42' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.41' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.47' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.56' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.47' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.39' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.5' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.51' '));' 'criteria.push(new ' 'kpCriterion('[thunderstorms]&' '#39;, 1.8268921375274658,' #39'22,200'#39', '#39'40,500'#39', 22200, 40500, ' '0.8' ',' ''$0.56', 559074,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '4' ',' '''' ',' 'kpView.MATCH_EXACT' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.75' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.81' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.87' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.64' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.56' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.52' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.6' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.53' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.47' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.58' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.61' '));' 'criteria.push(new ' 'kpCriterion('\42thunderstorms' '\042', 1.8268921375274658,' #39'110,000'#39', '#39'165,000'#39', 110000, ' '165000, 0.8' ',' ''$0.56', 559074,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '4' ',' '''' ',' 'kpView.MATCH_PHRASE' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.71' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.73' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.82' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.87' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.92' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.82' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.7' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.75' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.68' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.77' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.79' '));' 'criteria.push(new ' 'kpCriterion('lightning ' 'storm', 1.774579644203186,' #39'49,500'#39', '#39'90,500'#39', 49500, 90500, ' '0.73333335' ',' ''$0.54', 535666,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '5' ',' '''' ',' 'kpView.MATCH_BROAD' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.76' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.87' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.97' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.87' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.98' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.87' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.84' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.68' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.86' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.81' '));' 'criteria.push(new ' 'kpCriterion('[lightning ' 'storm]', 1.774579644203186,' #39'12,100'#39', '#39'22,200'#39', 12100, 22200, ' '0.73333335' ',' ''$0.54', 535666,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '5' ',' '''' ',' 'kpView.MATCH_EXACT' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.68' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.72' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.81' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.85' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.92' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.81' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.67' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.71' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.65' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.76' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.73' '));' 'criteria.push(new ' 'kpCriterion('\42lightning ' 'storm\042', ' '1.774579644203186,' #39'33,100'#39', '#39'60,500'#39', 33100, 60500, ' '0.73333335' ',' ''$0.54', 535666,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '5' ',' '''' ',' 'kpView.MATCH_PHRASE' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.69' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.69' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.71' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.66' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.68' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.7' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.75' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.79' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.74' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.72' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.7' '));' 'criteria.push(new ' 'kpCriterion('rain storm', ' '1.7464053630828857,' #39'27,100'#39', '#39'49,500'#39', 27100, 49500, ' '0.6666667' ',' ''$0.53', 526334,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '0' ',' '''' ',' 'kpView.MATCH_BROAD' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.87' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.79' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.57' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.55' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.57' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.74' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.76' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.69' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.61' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.89' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.73' '));' 'criteria.push(new ' 'kpCriterion('[rain ' 'storm]', ' '1.7464053630828857,' #39'5,400'#39', '#39'8,100'#39', 5400, 8100, ' '0.6666667' ',' ''$0.53', 526334,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '2' ',' '''' ',' 'kpView.MATCH_EXACT' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.73' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.7' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.68' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.61' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.68' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.69' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.73' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.72' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.62' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.59' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.66' '));' 'criteria.push(new ' 'kpCriterion('\42rain ' 'storm\042', ' '1.7464053630828857,' #39'14,800'#39', '#39'27,100'#39', 14800, 27100, ' '0.6666667' ',' ''$0.53', 526334,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '0' ',' '''' ',' 'kpView.MATCH_PHRASE' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.82' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.87' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.78' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.82' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.84' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.79' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.77' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.61' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.92' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.82' '));' 'criteria.push(new ' 'kpCriterion('lightning ' 'storms', ' '1.6842896938323975,' #39'14,800'#39', '#39'27,100'#39', 14800, 27100, ' '0.73333335' ',' ''$0.42', 417108,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '4' ',' '''' ',' 'kpView.MATCH_BROAD' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.9' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.9' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.84' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.7' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.81' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.88' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.77' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.76' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.57' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.75' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.63' '));' 'criteria.push(new ' 'kpCriterion('[lightning ' 'storms]', ' '1.6842896938323975,' #39'3,600'#39', '#39'8,100'#39', 3600, 8100, ' '0.73333335' ',' ''$0.42', 417108,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '4' ',' '''' ',' 'kpView.MATCH_EXACT' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.8' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.86' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.99' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.77' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.83' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.85' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.78' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.77' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.6' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.91' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.81' '));' 'criteria.push(new ' 'kpCriterion('\42lightning ' 'storms\042', ' '1.6842896938323975,' #39'12,100'#39', '#39'22,200'#39', 12100, 22200, ' '0.73333335' ',' ''$0.42', 417108,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '4' ',' '''' ',' 'kpView.MATCH_PHRASE' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.68' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.66' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.7' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.54' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.52' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.5' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.7' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.7' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.6' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.5' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.66' '));' 'criteria.push(new ' 'kpCriterion('rain ' 'storms', ' '1.421982765197754,' #39'6,600'#39', '#39'9,900'#39', 6600, 9900, 0.6' ',' ''$0.32', 324834,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '0' ',' '''' ',' 'kpView.MATCH_BROAD' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '1.0' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.97' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.91' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.57' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.52' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.51' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.69' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.64' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.6' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.51' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.77' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.6' '));' 'criteria.push(new ' 'kpCriterion('[rain ' 'storms]', ' '1.421982765197754,' #39'1,300'#39', '#39'1,900'#39', 1300, 1900, 0.6' ',' ''$0.32', 324834,' ''1 - 3', 2' ',' '0' ',' '0' ',' 'monthlyVariation,' '2' ',' '''' ',' 'kpView.MATCH_EXACT' ',' '0' ')); var monthlyVariation = new ' 'Array();' 'monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.68' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.68' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.7' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.53' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.53' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.49' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.71' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.67' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.57' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity(' '0.48' ')); monthlyVariation.push(new ' 'kpMonthlyPopularity('

Read the article

What are good design practices when working with Entity Framework

- by AD

This will apply mostly for an asp.net application where the data is not accessed via soa. Meaning that you get access to the objects loaded from the framework, not Transfer Objects, although some recommendation still apply. This is a community post, so please add to it as you see fit. Applies to: Entity Framework 1.0 shipped with Visual Studio 2008 sp1. Why pick EF in the first place? Considering it is a young technology with plenty of problems (see below), it may be a hard sell to get on the EF bandwagon for your project. However, it is the technology Microsoft is pushing (at the expense of Linq2Sql, which is a subset of EF). In addition, you may not be satisfied with NHibernate or other solutions out there. Whatever the reasons, there are people out there (including me) working with EF and life is not bad.make you think. EF and inheritance The first big subject is inheritance. EF does support mapping for inherited classes that are persisted in 2 ways: table per class and table the hierarchy. The modeling is easy and there are no programming issues with that part. (The following applies to table per class model as I don't have experience with table per hierarchy, which is, anyway, limited.) The real problem comes when you are trying to run queries that include one or many objects that are part of an inheritance tree: the generated sql is incredibly awful, takes a long time to get parsed by the EF and takes a long time to execute as well. This is a real show stopper. Enough that EF should probably not be used with inheritance or as little as possible. Here is an example of how bad it was. My EF model had ~30 classes, ~10 of which were part of an inheritance tree. On running a query to get one item from the Base class, something as simple as Base.Get(id), the generated SQL was over 50,000 characters. Then when you are trying to return some Associations, it degenerates even more, going as far as throwing SQL exceptions about not being able to query more than 256 tables at once. Ok, this is bad, EF concept is to allow you to create your object structure without (or with as little as possible) consideration on the actual database implementation of your table. It completely fails at this. So, recommendations? Avoid inheritance if you can, the performance will be so much better. Use it sparingly where you have to. In my opinion, this makes EF a glorified sql-generation tool for querying, but there are still advantages to using it. And ways to implement mechanism that are similar to inheritance. Bypassing inheritance with Interfaces First thing to know with trying to get some kind of inheritance going with EF is that you cannot assign a non-EF-modeled class a base class. Don't even try it, it will get overwritten by the modeler. So what to do? You can use interfaces to enforce that classes implement some functionality. For example here is a IEntity interface that allow you to define Associations between EF entities where you don't know at design time what the type of the entity would be. public enum EntityTypes{ Unknown = -1, Dog = 0, Cat } public interface IEntity { int EntityID { get; } string Name { get; } Type EntityType { get; } } public partial class Dog : IEntity { // implement EntityID and Name which could actually be fields // from your EF model Type EntityType{ get{ return EntityTypes.Dog; } } } Using this IEntity, you can then work with undefined associations in other classes // lets take a class that you defined in your model. // that class has a mapping to the columns: PetID, PetType public partial class Person { public IEntity GetPet() { return IEntityController.Get(PetID,PetType); } } which makes use of some extension functions: public class IEntityController { static public IEntity Get(int id, EntityTypes type) { switch (type) { case EntityTypes.Dog: return Dog.Get(id); case EntityTypes.Cat: return Cat.Get(id); default: throw new Exception("Invalid EntityType"); } } } Not as neat as having plain inheritance, particularly considering you have to store the PetType in an extra database field, but considering the performance gains, I would not look back. It also cannot model one-to-many, many-to-many relationship, but with creative uses of 'Union' it could be made to work. Finally, it creates the side effet of loading data in a property/function of the object, which you need to be careful about. Using a clear naming convention like GetXYZ() helps in that regards. Compiled Queries Entity Framework performance is not as good as direct database access with ADO (obviously) or Linq2SQL. There are ways to improve it however, one of which is compiling your queries. The performance of a compiled query is similar to Linq2Sql. What is a compiled query? It is simply a query for which you tell the framework to keep the parsed tree in memory so it doesn't need to be regenerated the next time you run it. So the next run, you will save the time it takes to parse the tree. Do not discount that as it is a very costly operation that gets even worse with more complex queries. There are 2 ways to compile a query: creating an ObjectQuery with EntitySQL and using CompiledQuery.Compile() function. (Note that by using an EntityDataSource in your page, you will in fact be using ObjectQuery with EntitySQL, so that gets compiled and cached). An aside here in case you don't know what EntitySQL is. It is a string-based way of writing queries against the EF. Here is an example: "select value dog from Entities.DogSet as dog where dog.ID = @ID". The syntax is pretty similar to SQL syntax. You can also do pretty complex object manipulation, which is well explained [here][1]. Ok, so here is how to do it using ObjectQuery< string query = "select value dog " + "from Entities.DogSet as dog " + "where dog.ID = @ID"; ObjectQuery<Dog> oQuery = new ObjectQuery<Dog>(query, EntityContext.Instance)); oQuery.Parameters.Add(new ObjectParameter("ID", id)); oQuery.EnablePlanCaching = true; return oQuery.FirstOrDefault(); The first time you run this query, the framework will generate the expression tree and keep it in memory. So the next time it gets executed, you will save on that costly step. In that example EnablePlanCaching = true, which is unnecessary since that is the default option. The other way to compile a query for later use is the CompiledQuery.Compile method. This uses a delegate: static readonly Func<Entities, int, Dog> query_GetDog = CompiledQuery.Compile<Entities, int, Dog>((ctx, id) => ctx.DogSet.FirstOrDefault(it => it.ID == id)); or using linq static readonly Func<Entities, int, Dog> query_GetDog = CompiledQuery.Compile<Entities, int, Dog>((ctx, id) => (from dog in ctx.DogSet where dog.ID == id select dog).FirstOrDefault()); to call the query: query_GetDog.Invoke( YourContext, id ); The advantage of CompiledQuery is that the syntax of your query is checked at compile time, where as EntitySQL is not. However, there are other consideration... Includes Lets say you want to have the data for the dog owner to be returned by the query to avoid making 2 calls to the database. Easy to do, right? EntitySQL string query = "select value dog " + "from Entities.DogSet as dog " + "where dog.ID = @ID"; ObjectQuery<Dog> oQuery = new ObjectQuery<Dog>(query, EntityContext.Instance)).Include("Owner"); oQuery.Parameters.Add(new ObjectParameter("ID", id)); oQuery.EnablePlanCaching = true; return oQuery.FirstOrDefault(); CompiledQuery static readonly Func<Entities, int, Dog> query_GetDog = CompiledQuery.Compile<Entities, int, Dog>((ctx, id) => (from dog in ctx.DogSet.Include("Owner") where dog.ID == id select dog).FirstOrDefault()); Now, what if you want to have the Include parametrized? What I mean is that you want to have a single Get() function that is called from different pages that care about different relationships for the dog. One cares about the Owner, another about his FavoriteFood, another about his FavotireToy and so on. Basicly, you want to tell the query which associations to load. It is easy to do with EntitySQL public Dog Get(int id, string include) { string query = "select value dog " + "from Entities.DogSet as dog " + "where dog.ID = @ID"; ObjectQuery<Dog> oQuery = new ObjectQuery<Dog>(query, EntityContext.Instance)) .IncludeMany(include); oQuery.Parameters.Add(new ObjectParameter("ID", id)); oQuery.EnablePlanCaching = true; return oQuery.FirstOrDefault(); } The include simply uses the passed string. Easy enough. Note that it is possible to improve on the Include(string) function (that accepts only a single path) with an IncludeMany(string) that will let you pass a string of comma-separated associations to load. Look further in the extension section for this function. If we try to do it with CompiledQuery however, we run into numerous problems: The obvious static readonly Func<Entities, int, string, Dog> query_GetDog = CompiledQuery.Compile<Entities, int, string, Dog>((ctx, id, include) => (from dog in ctx.DogSet.Include(include) where dog.ID == id select dog).FirstOrDefault()); will choke when called with: query_GetDog.Invoke( YourContext, id, "Owner,FavoriteFood" ); Because, as mentionned above, Include() only wants to see a single path in the string and here we are giving it 2: "Owner" and "FavoriteFood" (which is not to be confused with "Owner.FavoriteFood"!). Then, let's use IncludeMany(), which is an extension function static readonly Func<Entities, int, string, Dog> query_GetDog = CompiledQuery.Compile<Entities, int, string, Dog>((ctx, id, include) => (from dog in ctx.DogSet.IncludeMany(include) where dog.ID == id select dog).FirstOrDefault()); Wrong again, this time it is because the EF cannot parse IncludeMany because it is not part of the functions that is recognizes: it is an extension. Ok, so you want to pass an arbitrary number of paths to your function and Includes() only takes a single one. What to do? You could decide that you will never ever need more than, say 20 Includes, and pass each separated strings in a struct to CompiledQuery. But now the query looks like this: from dog in ctx.DogSet.Include(include1).Include(include2).Include(include3) .Include(include4).Include(include5).Include(include6) .[...].Include(include19).Include(include20) where dog.ID == id select dog which is awful as well. Ok, then, but wait a minute. Can't we return an ObjectQuery< with CompiledQuery? Then set the includes on that? Well, that what I would have thought so as well: static readonly Func<Entities, int, ObjectQuery<Dog>> query_GetDog = CompiledQuery.Compile<Entities, int, string, ObjectQuery<Dog>>((ctx, id) => (ObjectQuery<Dog>)(from dog in ctx.DogSet where dog.ID == id select dog)); public Dog GetDog( int id, string include ) { ObjectQuery<Dog> oQuery = query_GetDog(id); oQuery = oQuery.IncludeMany(include); return oQuery.FirstOrDefault; } That should have worked, except that when you call IncludeMany (or Include, Where, OrderBy...) you invalidate the cached compiled query because it is an entirely new one now! So, the expression tree needs to be reparsed and you get that performance hit again. So what is the solution? You simply cannot use CompiledQueries with parametrized Includes. Use EntitySQL instead. This doesn't mean that there aren't uses for CompiledQueries. It is great for localized queries that will always be called in the same context. Ideally CompiledQuery should always be used because the syntax is checked at compile time, but due to limitation, that's not possible. An example of use would be: you may want to have a page that queries which two dogs have the same favorite food, which is a bit narrow for a BusinessLayer function, so you put it in your page and know exactly what type of includes are required. Passing more than 3 parameters to a CompiledQuery Func is limited to 5 parameters, of which the last one is the return type and the first one is your Entities object from the model. So that leaves you with 3 parameters. A pitance, but it can be improved on very easily. public struct MyParams { public string param1; public int param2; public DateTime param3; } static readonly Func<Entities, MyParams, IEnumerable<Dog>> query_GetDog = CompiledQuery.Compile<Entities, MyParams, IEnumerable<Dog>>((ctx, myParams) => from dog in ctx.DogSet where dog.Age == myParams.param2 && dog.Name == myParams.param1 and dog.BirthDate > myParams.param3 select dog); public List<Dog> GetSomeDogs( int age, string Name, DateTime birthDate ) { MyParams myParams = new MyParams(); myParams.param1 = name; myParams.param2 = age; myParams.param3 = birthDate; return query_GetDog(YourContext,myParams).ToList(); } Return Types (this does not apply to EntitySQL queries as they aren't compiled at the same time during execution as the CompiledQuery method) Working with Linq, you usually don't force the execution of the query until the very last moment, in case some other functions downstream wants to change the query in some way: static readonly Func<Entities, int, string, IEnumerable<Dog>> query_GetDog = CompiledQuery.Compile<Entities, int, string, IEnumerable<Dog>>((ctx, age, name) => from dog in ctx.DogSet where dog.Age == age && dog.Name == name select dog); public IEnumerable<Dog> GetSomeDogs( int age, string name ) { return query_GetDog(YourContext,age,name); } public void DataBindStuff() { IEnumerable<Dog> dogs = GetSomeDogs(4,"Bud"); // but I want the dogs ordered by BirthDate gridView.DataSource = dogs.OrderBy( it => it.BirthDate ); } What is going to happen here? By still playing with the original ObjectQuery (that is the actual return type of the Linq statement, which implements IEnumerable), it will invalidate the compiled query and be force to re-parse. So, the rule of thumb is to return a List< of objects instead. static readonly Func<Entities, int, string, IEnumerable<Dog>> query_GetDog = CompiledQuery.Compile<Entities, int, string, IEnumerable<Dog>>((ctx, age, name) => from dog in ctx.DogSet where dog.Age == age && dog.Name == name select dog); public List<Dog> GetSomeDogs( int age, string name ) { return query_GetDog(YourContext,age,name).ToList(); //<== change here } public void DataBindStuff() { List<Dog> dogs = GetSomeDogs(4,"Bud"); // but I want the dogs ordered by BirthDate gridView.DataSource = dogs.OrderBy( it => it.BirthDate ); } When you call ToList(), the query gets executed as per the compiled query and then, later, the OrderBy is executed against the objects in memory. It may be a little bit slower, but I'm not even sure. One sure thing is that you have no worries about mis-handling the ObjectQuery and invalidating the compiled query plan. Once again, that is not a blanket statement. ToList() is a defensive programming trick, but if you have a valid reason not to use ToList(), go ahead. There are many cases in which you would want to refine the query before executing it. Performance What is the performance impact of compiling a query? It can actually be fairly large. A rule of thumb is that compiling and caching the query for reuse takes at least double the time of simply executing it without caching. For complex queries (read inherirante), I have seen upwards to 10 seconds. So, the first time a pre-compiled query gets called, you get a performance hit. After that first hit, performance is noticeably better than the same non-pre-compiled query. Practically the same as Linq2Sql When you load a page with pre-compiled queries the first time you will get a hit. It will load in maybe 5-15 seconds (obviously more than one pre-compiled queries will end up being called), while subsequent loads will take less than 300ms. Dramatic difference, and it is up to you to decide if it is ok for your first user to take a hit or you want a script to call your pages to force a compilation of the queries. Can this query be cached? { Dog dog = from dog in YourContext.DogSet where dog.ID == id select dog; } No, ad-hoc Linq queries are not cached and you will incur the cost of generating the tree every single time you call it. Parametrized Queries Most search capabilities involve heavily parametrized queries. There are even libraries available that will let you build a parametrized query out of lamba expressions. The problem is that you cannot use pre-compiled queries with those. One way around that is to map out all the possible criteria in the query and flag which one you want to use: public struct MyParams { public string name; public bool checkName; public int age; public bool checkAge; } static readonly Func<Entities, MyParams, IEnumerable<Dog>> query_GetDog = CompiledQuery.Compile<Entities, MyParams, IEnumerable<Dog>>((ctx, myParams) => from dog in ctx.DogSet where (myParams.checkAge == true && dog.Age == myParams.age) && (myParams.checkName == true && dog.Name == myParams.name ) select dog); protected List<Dog> GetSomeDogs() { MyParams myParams = new MyParams(); myParams.name = "Bud"; myParams.checkName = true; myParams.age = 0; myParams.checkAge = false; return query_GetDog(YourContext,myParams).ToList(); } The advantage here is that you get all the benifits of a pre-compiled quert. The disadvantages are that you most likely will end up with a where clause that is pretty difficult to maintain, that you will incur a bigger penalty for pre-compiling the query and that each query you run is not as efficient as it could be (particularly with joins thrown in). Another way is to build an EntitySQL query piece by piece, like we all did with SQL. protected List<Dod> GetSomeDogs( string name, int age) { string query = "select value dog from Entities.DogSet where 1 = 1 "; if( !String.IsNullOrEmpty(name) ) query = query + " and dog.Name == @Name "; if( age > 0 ) query = query + " and dog.Age == @Age "; ObjectQuery<Dog> oQuery = new ObjectQuery<Dog>( query, YourContext ); if( !String.IsNullOrEmpty(name) ) oQuery.Parameters.Add( new ObjectParameter( "Name", name ) ); if( age > 0 ) oQuery.Parameters.Add( new ObjectParameter( "Age", age ) ); return oQuery.ToList(); } Here the problems are: - there is no syntax checking during compilation - each different combination of parameters generate a different query which will need to be pre-compiled when it is first run. In this case, there are only 4 different possible queries (no params, age-only, name-only and both params), but you can see that there can be way more with a normal world search. - Noone likes to concatenate strings! Another option is to query a large subset of the data and then narrow it down in memory. This is particularly useful if you are working with a definite subset of the data, like all the dogs in a city. You know there are a lot but you also know there aren't that many... so your CityDog search page can load all the dogs for the city in memory, which is a single pre-compiled query and then refine the results protected List<Dod> GetSomeDogs( string name, int age, string city) { string query = "select value dog from Entities.DogSet where dog.Owner.Address.City == @City "; ObjectQuery<Dog> oQuery = new ObjectQuery<Dog>( query, YourContext ); oQuery.Parameters.Add( new ObjectParameter( "City", city ) ); List<Dog> dogs = oQuery.ToList(); if( !String.IsNullOrEmpty(name) ) dogs = dogs.Where( it => it.Name == name ); if( age > 0 ) dogs = dogs.Where( it => it.Age == age ); return dogs; } It is particularly useful when you start displaying all the data then allow for filtering. Problems: - Could lead to serious data transfer if you are not careful about your subset. - You can only filter on the data that you returned. It means that if you don't return the Dog.Owner association, you will not be able to filter on the Dog.Owner.Name So what is the best solution? There isn't any. You need to pick the solution that works best for you and your problem: - Use lambda-based query building when you don't care about pre-compiling your queries. - Use fully-defined pre-compiled Linq query when your object structure is not too complex. - Use EntitySQL/string concatenation when the structure could be complex and when the possible number of different resulting queries are small (which means fewer pre-compilation hits). - Use in-memory filtering when you are working with a smallish subset of the data or when you had to fetch all of the data on the data at first anyway (if the performance is fine with all the data, then filtering in memory will not cause any time to be spent in the db). Singleton access The best way to deal with your context and entities accross all your pages is to use the singleton pattern: public sealed class YourContext { private const string instanceKey = "On3GoModelKey"; YourContext(){} public static YourEntities Instance { get { HttpContext context = HttpContext.Current; if( context == null ) return Nested.instance; if (context.Items[instanceKey] == null) { On3GoEntities entity = new On3GoEntities(); context.Items[instanceKey] = entity; } return (YourEntities)context.Items[instanceKey]; } } class Nested { // Explicit static constructor to tell C# compiler // not to mark type as beforefieldinit static Nested() { } internal static readonly YourEntities instance = new YourEntities(); } } NoTracking, is it worth it? When executing a query, you can tell the framework to track the objects it will return or not. What does it mean? With tracking enabled (the default option), the framework will track what is going on with the object (has it been modified? Created? Deleted?) and will also link objects together, when further queries are made from the database, which is what is of interest here. For example, lets assume that Dog with ID == 2 has an owner which ID == 10. Dog dog = (from dog in YourContext.DogSet where dog.ID == 2 select dog).FirstOrDefault(); //dog.OwnerReference.IsLoaded == false; Person owner = (from o in YourContext.PersonSet where o.ID == 10 select dog).FirstOrDefault(); //dog.OwnerReference.IsLoaded == true; If we were to do the same with no tracking, the result would be different. ObjectQuery<Dog> oDogQuery = (ObjectQuery<Dog>) (from dog in YourContext.DogSet where dog.ID == 2 select dog); oDogQuery.MergeOption = MergeOption.NoTracking; Dog dog = oDogQuery.FirstOrDefault(); //dog.OwnerReference.IsLoaded == false; ObjectQuery<Person> oPersonQuery = (ObjectQuery<Person>) (from o in YourContext.PersonSet where o.ID == 10 select o); oPersonQuery.MergeOption = MergeOption.NoTracking; Owner owner = oPersonQuery.FirstOrDefault(); //dog.OwnerReference.IsLoaded == false; Tracking is very useful and in a perfect world without performance issue, it would always be on. But in this world, there is a price for it, in terms of performance. So, should you use NoTracking to speed things up? It depends on what you are planning to use the data for. Is there any chance that the data your query with NoTracking can be used to make update/insert/delete in the database? If so, don't use NoTracking because associations are not tracked and will causes exceptions to be thrown. In a page where there are absolutly no updates to the database, you can use NoTracking. Mixing tracking and NoTracking is possible, but it requires you to be extra careful with updates/inserts/deletes. The problem is that if you mix then you risk having the framework trying to Attach() a NoTracking object to the context where another copy of the same object exist with tracking on. Basicly, what I am saying is that Dog dog1 = (from dog in YourContext.DogSet where dog.ID == 2).FirstOrDefault(); ObjectQuery<Dog> oDogQuery = (ObjectQuery<Dog>) (from dog in YourContext.DogSet where dog.ID == 2 select dog); oDogQuery.MergeOption = MergeOption.NoTracking; Dog dog2 = oDogQuery.FirstOrDefault(); dog1 and dog2 are 2 different objects, one tracked and one not. Using the detached object in an update/insert will force an Attach() that will say "Wait a minute, I do already have an object here with the same database key. Fail". And when you Attach() one object, all of its hierarchy gets attached as well, causing problems everywhere. Be extra careful. How much faster is it with NoTracking It depends on the queries. Some are much more succeptible to tracking than other. I don't have a fast an easy rule for it, but it helps. So I should use NoTracking everywhere then? Not exactly. There are some advantages to tracking object. The first one is that the object is cached, so subsequent call for that object will not hit the database. That cache is only valid for the lifetime of the YourEntities object, which, if you use the singleton code above, is the same as the page lifetime. One page request == one YourEntity object. So for multiple calls for the same object, it will load only once per page request. (Other caching mechanism could extend that). What happens when you are using NoTracking and try to load the same object multiple times? The database will be queried each time, so there is an impact there. How often do/should you call for the same object during a single page request? As little as possible of course, but it does happens. Also remember the piece above about having the associations connected automatically for your? You don't have that with NoTracking, so if you load your data in multiple batches, you will not have a link to between them: ObjectQuery<Dog> oDogQuery = (ObjectQuery<Dog>)(from dog in YourContext.DogSet select dog); oDogQuery.MergeOption = MergeOption.NoTracking; List<Dog> dogs = oDogQuery.ToList(); ObjectQuery<Person> oPersonQuery = (ObjectQuery<Person>)(from o in YourContext.PersonSet select o); oPersonQuery.MergeOption = MergeOption.NoTracking; List<Person> owners = oPersonQuery.ToList(); In this case, no dog will have its .Owner property set. Some things to keep in mind when you are trying to optimize the performance. No lazy loading, what am I to do? This can be seen as a blessing in disguise. Of course it is annoying to load everything manually. However, it decreases the number of calls to the db and forces you to think about when you should load data. The more you can load in one database call the better. That was always true, but it is enforced now with this 'feature' of EF. Of course, you can call if( !ObjectReference.IsLoaded ) ObjectReference.Load(); if you want to, but a better practice is to force the framework to load the objects you know you will need in one shot. This is where the discussion about parametrized Includes begins to make sense. Lets say you have you Dog object public class Dog { public Dog Get(int id) { return YourContext.DogSet.FirstOrDefault(it => it.ID == id ); } } This is the type of function you work with all the time. It gets called from all over the place and once you have that Dog object, you will do very different things to it in different functions. First, it should be pre-compiled, because you will call that very often. Second, each different pages will want to have access to a different subset of the Dog data. Some will want the Owner, some the FavoriteToy, etc. Of course, you could call Load() for each reference you need anytime you need one. But that will generate a call to the database each time. Bad idea. So instead, each page will ask for the data it wants to see when it first request for the Dog object: static public Dog Get(int id) { return GetDog(entity,"");} static public Dog Get(int id, string includePath) { string query = "select value o " + " from YourEntities.DogSet as o " +

Search Results

Search found 2115 results on 85 pages for 'chum of chance'.

Page 85/85 | < Previous Page | 81 82 83 84 85

- by Michael Williamson

- by SaveTheRbtz

- by swisscheese

- by Malcolm Frexner

- by wyatt

- by DuffDuff

- by ie

- by user318068

- by numerical25

- by Mark Virtue

- by Malcolm Frexner

- by WickedW

- by Brad

- by Brad

- by AD

< Previous Page | 81 82 83 84 85