Search Results

Search found 12588 results on 504 pages for 'memory allocation'.

Page 244/504 | < Previous Page | 240 241 242 243 244 245 246 247 248 249 250 251  | Next Page >

• Ubuntu, No wireless networks found after correctly installed madwifi

  - by Peter

  Hi, I just installed madwifi on my MSI laptop with an Atheros AR5001 wifi card & Lucid. As far as I can see and according to System - Administration - Hardware drivers the install was successful and the card + driver is up and running. However, I don't see any wireless network (my windows PC can see about 5 wireless networks). I tried it with the network manager applet as well as with wicd. If I try to connect to "Hidden Wireless Network" via nm-applet, it will start to connect for a while but is unable too (although I supply it with the correct WEP settings & key) So, I'm unable to use my wireless network. What am i doing wrong? Some information about my system: iwconfig lo no wireless extensions. eth0 no wireless extensions. wifi0 no wireless extensions. ath0 IEEE 802.11g ESSID:"" Mode:Managed Frequency:2.437 GHz Access Point: Not-Associated Bit Rate:0 kb/s Tx-Power:17 dBm Sensitivity=1/1 Retry:off RTS thr:off Fragment thr:off Power Management:off Link Quality=0/70 Signal level=-96 dBm Noise level=-96 dBm Rx invalid nwid:0 Rx invalid crypt:0 Rx invalid frag:0 Tx excessive retries:0 Invalid misc:0 Missed beacon:0 pan0 no wireless extensions. ifconfig ath0 Link encap:Ethernet HWaddr 00:15:af:cf:e2:ca inet6 addr: fe80::215:afff:fecf:e2ca/64 Scope:Link UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) eth0 Link encap:Ethernet HWaddr 00:21:85:4d:82:78 inet addr:192.168.2.101 Bcast:192.168.2.255 Mask:255.255.255.0 inet6 addr: fe80::221:85ff:fe4d:8278/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:3800 errors:0 dropped:0 overruns:0 frame:0 TX packets:2944 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3940261 (3.9 MB) TX bytes:525218 (525.2 KB) Interrupt:27 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:12 errors:0 dropped:0 overruns:0 frame:0 TX packets:12 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:720 (720.0 B) TX bytes:720 (720.0 B) wifi0 Link encap:UNSPEC HWaddr 00-15-AF-CF-E2-CA-00-00-00-00-00-00-00-00-00-00 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:3497 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:280 RX bytes:0 (0.0 B) TX bytes:179947 (179.9 KB) Interrupt:16 lshw -C network *-network description: Wireless interface product: AR5001 Wireless Network Adapter vendor: Atheros Communications Inc. physical id: 0 bus info: pci@0000:02:00.0 logical name: wifi0 version: 01 serial: 00:15:af:cf:e2:ca width: 64 bits clock: 33MHz capabilities: pm msi pciexpress msix bus_master cap_list logical ethernet physical wireless configuration: broadcast=yes driver=ath_pci latency=0 multicast=yes wireless=IEEE 802.11g resources: irq:16 memory:fd7f0000-fd7fffff *-network description: Ethernet interface product: RTL8111/8168B PCI Express Gigabit Ethernet controller vendor: Realtek Semiconductor Co., Ltd. physical id: 0 bus info: pci@0000:05:00.0 logical name: eth0 version: 01 serial: 00:21:85:4d:82:78 size: 100MB/s capacity: 1GB/s width: 64 bits clock: 33MHz capabilities: pm vpd msi pciexpress bus_master cap_list rom ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=r8169 driverversion=2.3LK-NAPI duplex=full ip=192.168.2.101 latency=0 link=yes multicast=yes port=MII speed=100MB/s resources: irq:27 ioport:c800(size=256) memory:fe2ff000-fe2fffff memory:fe2c0000-fe2dffff(prefetchable) lspci 00:00.0 Host bridge: ATI Technologies Inc RS690 Host Bridge 00:01.0 PCI bridge: ATI Technologies Inc RS690 PCI to PCI Bridge (Internal gfx) 00:04.0 PCI bridge: ATI Technologies Inc Device 7914 00:06.0 PCI bridge: ATI Technologies Inc RS690 PCI to PCI Bridge (PCI Express Port 2) 00:07.0 PCI bridge: ATI Technologies Inc RS690 PCI to PCI Bridge (PCI Express Port 3) 00:12.0 SATA controller: ATI Technologies Inc SB600 Non-Raid-5 SATA 00:13.0 USB Controller: ATI Technologies Inc SB600 USB (OHCI0) 00:13.1 USB Controller: ATI Technologies Inc SB600 USB (OHCI1) 00:13.2 USB Controller: ATI Technologies Inc SB600 USB (OHCI2) 00:13.3 USB Controller: ATI Technologies Inc SB600 USB (OHCI3) 00:13.4 USB Controller: ATI Technologies Inc SB600 USB (OHCI4) 00:13.5 USB Controller: ATI Technologies Inc SB600 USB Controller (EHCI) 00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 14) 00:14.1 IDE interface: ATI Technologies Inc SB600 IDE 00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) 00:14.3 ISA bridge: ATI Technologies Inc SB600 PCI to LPC Bridge 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge 00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration 00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map 00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller 00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control 01:05.0 VGA compatible controller: ATI Technologies Inc RS690M [Radeon X1200 Series] 01:05.2 Audio device: ATI Technologies Inc Radeon X1200 Series Audio Controller 02:00.0 Ethernet controller: Atheros Communications Inc. AR5001 Wireless Network Adapter (rev 01) 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01) 06:04.0 CardBus bridge: O2 Micro, Inc. OZ711MP1/MS1 MemoryCardBus Controller (rev 21) 06:04.2 SD Host controller: O2 Micro, Inc. Integrated MMC/SD Controller (rev 01) 06:04.3 Bridge: O2 Micro, Inc. Integrated MS/xD Controller (rev 01) 06:04.4 FireWire (IEEE 1394): O2 Micro, Inc. Firewire (IEEE 1394) (rev 02) less /proc/modules | grep ath ath_rate_sample 11476 1 - Live 0xf812b000 ath_pci 193197 0 - Live 0xf85c3000 wlan 222892 5 wlan_wep,wlan_scan_sta,ath_rate_sample,ath_pci, Live 0xf8537000 ath_hal 398604 3 ath_rate_sample,ath_pci, Live 0xf8480000 I've been at this for hours now, also tried ndiswrapper and ath5k drivers with no luck, and really could use some help. Cheers.

  Read the article

• Can someone explain the "use-cases" for the default munin graphs?

  - by exhuma

  When installing munin, it activates a default set of plugins (at least on ubuntu). Alternatively, you can simply run munin-node-configure to figure out which plugins are supported on your system. Most of these plugins plot straight-forward data. My question is not to explain the nature of the data (well... maybe for some) but what is it that you look for in these graphs? It is easy to install munin and see fancy graphs. But having the graphs and not being able to "read" them renders them totally useless. I am going to list standard plugins which are enabled by default on my system. So it's going to be a long list. For completeness, I am also going to list plugins which I think to understand and give a short explanation as to what I think it's used for. Pleas correct if I am wrong with any of them. So let me split this questions in three parts: Plugins where I don't even understand the data Plugins where I understand the data but don't know what I should look out for Plugins which I think to understand Plugins where I don't even understand the data These may contain questions that are not necessarily aimed at munin alone. Not understanding the data usually mean a gap in fundamental knowledge on operating systems/hardware.... ;) Feel free to respond with a "giyf" answer. These are plugins where I can only guess what's going on... I hardly want to look at these "guessing"... Disk IOs per device (IOs/second)What's an IO. I know it stands for input/output. But that's as far as it goes. Disk latency per device (Average IO wait)Not a clue what an "IO wait" is... IO Service TimeThis one is a huge mess, and it's near impossible to see something in the graph at all. Plugins where I understand the data but don't know what I should look out for IOStat (blocks/second read/written)I assume, the thing to look out for in here are spikes? Which would mean that the device is in heavy use? Available entropy (bytes)I assume that this is important for random number generation? Why would I graph this? So far the value has always been near constant. VMStat (running/I/O sleep processes)What's the difference between this one and the "processes" graph? Both show running/sleeping processes, whereas the "Processes" graph seems to have more details. Disk throughput per device (bytes/second read/written) What's thedifference between this one and the "IOStat" graph? inode table usageWhat should I look for in this graph? Plugins which I think to understand I'll be guessing some things here... correct me if I am wrong. Disk usage in percent (percent)How much disk space is used/remaining. As this is approaching 100%, you should consider cleaning up or extend the partition. This is extremely important for the root partition. Firewall Throughput (packets/second)The number of packets passing through the firewall. If this is spiking for a longer period of time, it could be a sign of a DOS attack (or we are simply recieving a large file). It can also give you an idea about your firewall performance. If it's levelling out and you need more "power" you should consider load balancing. If it's levelling out and see a correlation with your CPU load, it could also mean that your hardware is not fast enough. Correlations with disk usage could point to excessive LOG targets in you FW config. eth0 errors (packets in/out)Network errors. If this value is increasing, it could be a sign of faulty hardware. eth0 traffic (bits/second in/out)Raw network traffic. This should correlate with Firewall throughput. number of threadsAn ever-increasing value might point to a process not properly closing threads. Investigate! processesBreakdown of active processes (including sleeping). A quick spike in here might point to a fork-bomb. A slowly, but ever-increasing value might point to an application spawning sub-processes but not properly closing them. Investigate using ps faux. process priorityThis shows the distribution of process priorities. Having only high-priority processes is not of much use. Consider de-prioritizing some. cpu usageFairly straight-forward. If this is spiking, you may have an attack going on, or a process is hogging the CPU. Idf it's slowly increasing and approaching max in normal operations, you should consider upgrading your hardware (or load-balancing). file table usageNumber of actively open files. If this is reaching max, you may have a process opening, but not properly releasing files. load averageShows an summarized value for the system load. Should correlate with CPU usage. Increasing values can come from a number of sources. Look for correlations with other graphs. memory usageA graphical representation of you memory. As long as you have a lot of unused+cache+buffers you are fine. swap in/outShows the activity on your swap partition. This should always be 0. If you see activity on this, you should add more memory to your machine!

  Read the article

• Mount TMPFS instead of ro /dev

  - by schiggn

  I am working on a ARM-Based embedded system with a custom Debian Linux based on kernel 2.6.31. In the final system, the Root file system is stored as squashfs on flash. Now, the folder /dev is created by udev, but since there is no hot plugging functionality needed and booting time is critical, I wanted to delete udev and "hard code" the /dev folder (read here, page 5). because i still need to change parameters of the devices (with ioctl /sysfs) this does not work for me in this case. so i thought of mounting a tmpfs on /dev and change the parameters there. is this possible? and how to do best? my approach would be: delete /dev from RFS create tar containing basic devices mount tmpfs /dev untar tar-file into /dev change parameters Could this work? Do you see any problems? I found out, that you can mount on top of already mounted mount point, is it somehow possible just to take data with while mounting the new file system? if so that would be very convenient! Thanks Update: I just tried that out, but I'm stuck at a certain point. I packed all my devices into devices.tar, packed it into /usr of my squashfs and added the following lines to mountkernfs.sh, which is executed right after INIT. #mount /dev on tmpfs echo -n "Mounting /dev on tmpfs..." mount -o size=5M,mode=0755 -t tmpfs tmpfs /dev mknod -m 600 /dev/console c 5 1 mknod -m 600 /dev/null c 1 3 echo "done." echo -n "Populating /dev..." tar -xf /usr/devices.tar -C /dev echo "done." This works fine on the version over NFS, if I place printf's in the code, I can see it executing, if I comment out the extracting part, its complaining about missing devices. Booting OK mmc0: new high speed SDHC card at address 0007 mmcblk0: mmc0:0007 SD04G 3.67 GiB mmcblk0: p1 IP-Config: Unable to set interface netmask (-22). Looking up port of RPC 100003/2 on 192.168.1.234 Looking up port of RPC 100005/1 on 192.168.1.234 VFS: Mounted root (nfs filesystem) on device 0:14. Freeing init memory: 136K INIT: version 2.86 booting Mounting /dev on tmpfs...done. Populating /dev...done. Initializing /var...done. Setting the system clock. System Clock set to: Thu Sep 13 11:26:23 UTC 2012. INIT: Entering runlevel: 2 UBI: attaching mtd8 to ubi0 Commenting out the extraction of the tar mmc0: new high speed SDHC card at address 0007 mmcblk0: mmc0:0007 SD04G 3.67 GiB mmcblk0: p1 IP-Config: Unable to set interface netmask (-22). Looking up port of RPC 100003/2 on 192.168.1.234 Looking up port of RPC 100005/1 on 192.168.1.234 VFS: Mounted root (nfs filesystem) on device 0:14. Freeing init memory: 136K INIT: version 2.86 booting Mounting /dev on tmpfs...done. Populating /dev...done. Initializing /var...done. Setting the system clock. Cannot access the Hardware Clock via any known method. Use the --debug option to see the details of our search for an access method. Unable to set System Clock to: Thu Sep 13 12:24:00 UTC 2012 ... (warning). INIT: Entering runlevel: 2 libubi: error!: cannot open "/dev/ubi_ctrl" So far so good. But if I pack the whole story into a squashfs and boot from there, it is acting strange. It's telling me while booting that it is unable to open an initial console and its throwing errors on mounting the UBIFS devices, but finally provides a login anyway. Over that my echo's are not executed. If I then log in, /dev is mounted as TMPFS as desired and all the devices reside inside. When I redo the "mount" command to mount the UBIFS partitions it is executed whitout problem and useable. From squashfs VFS: Mounted root (squashfs filesystem) readonly on device 31:15. Freeing init memory: 136K Warning: unable to open an initial console. mmc0: new high speed SDHC card at address 0007 mmcblk0: mmc0:0007 SD04G 3.67 GiB mmcblk0: p1 UBIFS error (pid 484): ubifs_get_sb: cannot open "ubi1_0", error -19 Additionally, a part of the rest of the bootscripts is still exexuted, but not all of them. Does anyone has a clue why? Other question, is 5MB enough/too much for /dev?

  Read the article

• Android: OutOfMemoryError while uploading video...

  - by AP257

  Hi all, I have the same problem as described here, but I will supply a few more details. While trying to upload a video in Android, I'm reading it into memory, and if the video is large I get an OutOfMemoryError. Here's my code: // get bytestream to upload videoByteArray = getBytesFromFile(cR, fileUriString); public static byte[] getBytesFromFile(ContentResolver cR, String fileUriString) throws IOException { Uri tempuri = Uri.parse(fileUriString); InputStream is = cR.openInputStream(tempuri); byte[] b3 = readBytes(is); is.close(); return b3; } public static byte[] readBytes(InputStream inputStream) throws IOException { ByteArrayOutputStream byteBuffer = new ByteArrayOutputStream(); // this is storage overwritten on each iteration with bytes int bufferSize = 1024; byte[] buffer = new byte[bufferSize]; int len = 0; while ((len = inputStream.read(buffer)) != -1) { byteBuffer.write(buffer, 0, len); } return byteBuffer.toByteArray(); } And here's the traceback (the error is thrown on the byteBuffer.write(buffer, 0, len) line): 04-08 11:56:20.456: ERROR/dalvikvm-heap(6088): Out of memory on a 16775184-byte allocation. 04-08 11:56:20.456: INFO/dalvikvm(6088): "IntentService[UploadService]" prio=5 tid=17 RUNNABLE 04-08 11:56:20.456: INFO/dalvikvm(6088): | group="main" sCount=0 dsCount=0 s=N obj=0x449a3cf0 self=0x38d410 04-08 11:56:20.456: INFO/dalvikvm(6088): | sysTid=6119 nice=0 sched=0/0 cgrp=default handle=4010416 04-08 11:56:20.456: INFO/dalvikvm(6088): at java.io.ByteArrayOutputStream.expand(ByteArrayOutputStream.java:~93) 04-08 11:56:20.456: INFO/dalvikvm(6088): at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:218) 04-08 11:56:20.456: INFO/dalvikvm(6088): at com.android.election2010.UploadService.readBytes(UploadService.java:199) 04-08 11:56:20.456: INFO/dalvikvm(6088): at com.android.election2010.UploadService.getBytesFromFile(UploadService.java:182) 04-08 11:56:20.456: INFO/dalvikvm(6088): at com.android.election2010.UploadService.doUploadinBackground(UploadService.java:118) 04-08 11:56:20.456: INFO/dalvikvm(6088): at com.android.election2010.UploadService.onHandleIntent(UploadService.java:85) 04-08 11:56:20.456: INFO/dalvikvm(6088): at android.app.IntentService$ServiceHandler.handleMessage(IntentService.java:30) 04-08 11:56:20.456: INFO/dalvikvm(6088): at android.os.Handler.dispatchMessage(Handler.java:99) 04-08 11:56:20.456: INFO/dalvikvm(6088): at android.os.Looper.loop(Looper.java:123) 04-08 11:56:20.456: INFO/dalvikvm(6088): at android.os.HandlerThread.run(HandlerThread.java:60) 04-08 11:56:20.467: WARN/dalvikvm(6088): threadid=17: thread exiting with uncaught exception (group=0x4001b180) 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): Uncaught handler: thread IntentService[UploadService] exiting due to uncaught exception 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): java.lang.OutOfMemoryError 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): at java.io.ByteArrayOutputStream.expand(ByteArrayOutputStream.java:93) 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:218) 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): at com.android.election2010.UploadService.readBytes(UploadService.java:199) 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): at com.android.election2010.UploadService.getBytesFromFile(UploadService.java:182) 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): at com.android.election2010.UploadService.doUploadinBackground(UploadService.java:118) 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): at com.android.election2010.UploadService.onHandleIntent(UploadService.java:85) 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): at android.app.IntentService$ServiceHandler.handleMessage(IntentService.java:30) 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): at android.os.Handler.dispatchMessage(Handler.java:99) 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): at android.os.Looper.loop(Looper.java:123) 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): at android.os.HandlerThread.run(HandlerThread.java:60) 04-08 11:56:20.496: INFO/Process(4657): Sending signal. PID: 6088 SIG: 3 I guess that as @DroidIn suggests, I need to upload it in chunks. But (newbie question alert) does that mean that I should make multiple PostMethod requests, and glue the file together at the server end? Or can I load the bytestream into memory in chunks, and glue it together in the Android code? If anyone could give me a clue as to the best approach, I would be very grateful.

  Read the article

• SEVERE error in Eclipse Web application Startup

  - by lakshmanan

  I use tomcat. I am developing a web application using struts2, and I use eclipse as my IDE for development. I get this error Mar 7, 2010 7:38:42 PM org.apache.catalina.core.StandardContext filterStart SEVERE: Exception starting filter struts2 Unable to load bean: type:com.opensymphony.xwork2.ObjectFactory class:org.apache.struts2.spring.StrutsSpringObjectFactory - bean - jar:file:/C:/Documents%20and%20Settings/Lakshmanan/My%20Documents/My%20Dropbox/Final%20year%20project/Workspace/.metadata/.plugins/org.eclipse.wst.server.core/tmp0/wtpwebapps/projit1/WEB-INF/lib/struts2-spring-plugin-2.1.8.1.jar!/struts-plugin.xml:29:132 at com.opensymphony.xwork2.config.providers.XmlConfigurationProvider.register(XmlConfigurationProvider.java:208) at org.apache.struts2.config.StrutsXmlConfigurationProvider.register(StrutsXmlConfigurationProvider.java:101) at com.opensymphony.xwork2.config.impl.DefaultConfiguration.reload(DefaultConfiguration.java:131) at com.opensymphony.xwork2.config.ConfigurationManager.getConfiguration(ConfigurationManager.java:52) at org.apache.struts2.dispatcher.Dispatcher.init_PreloadConfiguration(Dispatcher.java:395) at org.apache.struts2.dispatcher.Dispatcher.init(Dispatcher.java:452) at org.apache.struts2.dispatcher.FilterDispatcher.init(FilterDispatcher.java:201) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:295) at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:422) at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:115) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3838) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4488) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at org.apache.catalina.core.StandardHost.start(StandardHost.java:722) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443) at org.apache.catalina.core.StandardService.start(StandardService.java:516) at org.apache.catalina.core.StandardServer.start(StandardServer.java:710) at org.apache.catalina.startup.Catalina.start(Catalina.java:593) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: java.lang.NoClassDefFoundError: com/opensymphony/xwork2/util/classloader/ReloadingClassLoader at java.lang.Class.getDeclaredConstructors0(Native Method) at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389) at java.lang.Class.getDeclaredConstructors(Class.java:1836) at com.opensymphony.xwork2.config.providers.XmlConfigurationProvider.register(XmlConfigurationProvider.java:198) ... 24 more Caused by: java.lang.ClassNotFoundException: com.opensymphony.xwork2.util.classloader.ReloadingClassLoader at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1484) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329) ... 28 more Mar 7, 2010 7:38:42 PM org.apache.catalina.core.StandardContext start SEVERE: Error filterStart Mar 7, 2010 7:38:42 PM org.apache.catalina.core.StandardContext start SEVERE: Context [/projit1] startup failed due to previous errors Mar 7, 2010 7:38:42 PM org.apache.catalina.core.ApplicationContext log INFO: Closing Spring root WebApplicationContext Mar 7, 2010 7:38:42 PM org.springframework.context.support.AbstractApplicationContext doClose INFO: Closing Root WebApplicationContext: startup date [Sun Mar 07 19:38:41 GMT+05:30 2010]; root of context hierarchy Mar 7, 2010 7:38:42 PM org.springframework.beans.factory.support.DefaultSingletonBeanRegistry destroySingletons INFO: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@afae4a: defining beans [personCrud,companyCrud,projectCrud,discussionCrud]; root of factory hierarchy Mar 7, 2010 7:38:42 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc SEVERE: A web application registered the JBDC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered. Mar 7, 2010 7:38:42 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap SEVERE: A web application created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Prototype beans currently in creation]) and a value of type [null] (value [null]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed. Mar 7, 2010 7:38:42 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap SEVERE: A web application created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [XML bean definition resources currently being loaded]) and a value of type [null] (value [null]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed. I am confused . Please help me in rectifying this error.

  Read the article

• Running multiple image manipulations in parallel causing OutOfMemory exception

  - by Tom

  I am working on a site where I need to be able to split and image around 4000x6000 into 4 parts (amongst many other tasks) and I need this to be as quick as possible for multiple users. My current code for doing this is var bitmaps = new RenderTargetBitmap[elements.Length]; using (var stream = blobService.Stream(key)) { BitmapImage bi = new BitmapImage(); bi.BeginInit(); bi.StreamSource = stream; bi.EndInit(); for (var i = 0; i < elements.Length; i++) { var element = elements[i]; TransformGroup transformGroup = new TransformGroup(); TranslateTransform translateTransform = new TranslateTransform(); translateTransform.X = -element.Left; translateTransform.Y = -element.Top; transformGroup.Children.Add(translateTransform); DrawingVisual vis = new DrawingVisual(); DrawingContext cont = vis.RenderOpen(); cont.PushTransform(transformGroup); cont.DrawImage(bi, new Rect(new Size(bi.PixelWidth, bi.PixelHeight))); cont.Close(); RenderTargetBitmap rtb = new RenderTargetBitmap(element.Width, element.Height, 96d, 96d, PixelFormats.Default); rtb.Render(vis); bitmaps[i] = rtb; } } for (var i = 0; i < bitmaps.Length; i++) { using (MemoryStream ms = new MemoryStream()) { PngBitmapEncoder encoder = new PngBitmapEncoder(); encoder.Frames.Add(BitmapFrame.Create(bitmaps[i])); encoder.Save(ms); var regionKey = WebPath.Variant(key, elements[i].Id); saveBlobService.Save("image/png", regionKey, ms); } } I am running multiple threads which take jobs off a queue. I am finding that if this part of code is hit by 4 threads at once I get an OutOfMemory exception. I can stop this happening by wrapping all the code above in a lock(obj) but this isn't ideal. I have tried wrapping just the first using block (where the file is read from disk and split) but I still get the out of memory exceptions (this part of the code executes quite quickly). I this normal considering the amount of memory this should be taking up? Are there any optimisations I could make? Can I increase the memory available? UPDATE: My new code as per Moozhe's help public static void GenerateRegions(this IBlobService blobService, string key, Element[] elements) { using (var stream = blobService.Stream(key)) { foreach (var element in elements) { stream.Position = 0; BitmapImage bi = new BitmapImage(); bi.BeginInit(); bi.SourceRect = new Int32Rect(element.Left, element.Top, element.Width, element.Height); bi.StreamSource = stream; bi.EndInit(); DrawingVisual vis = new DrawingVisual(); DrawingContext cont = vis.RenderOpen(); cont.DrawImage(bi, new Rect(new Size(element.Width, element.Height))); cont.Close(); RenderTargetBitmap rtb = new RenderTargetBitmap(element.Width, element.Height, 96d, 96d, PixelFormats.Default); rtb.Render(vis); using (MemoryStream ms = new MemoryStream()) { PngBitmapEncoder encoder = new PngBitmapEncoder(); encoder.Frames.Add(BitmapFrame.Create(rtb)); encoder.Save(ms); var regionKey = WebPath.Variant(key, element.Id); blobService.Save("image/png", regionKey, ms); } } } }

  Read the article

• Access violation when accessing a COM object from .Net

  - by Groo

  Dear Sirs, I am sorry if the post is too long, but I would be happy if someone would at least point read the bolded titles, and point me in the right direction. I am having this problem for couple of days, but was unable to found the answer on the net. These are the things I have found out so far. 1. "Access violation" exception crushes my managed application My C# WinForms app sometimes closes with an "Access violation" exception ("Attempted to read or write protected memory"), right in the moment when selecting a TabPage in a windows form TabControl. From the stack trace (try/catch around Application.Run) I can see that the exception happens at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg), called inside UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32 dwComponentID, Int32 reason, Int32 pvLoopData). -- Message: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. -- Stack trace: at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg) at System.Windows.Forms.Application.ComponentManager .System.Windows.Forms.UnsafeNativeMethods .IMsoComponentManager.FPushMessageLoop (Int32 dwComponentID, Int32 reason, Int32 pvLoopData) at System.Windows.Forms.Application.ThreadContext .RunMessageLoopInner(Int32 reason, ApplicationContext context) at System.Windows.Forms.Application.ThreadContext .RunMessageLoop(Int32 reason, ApplicationContext context) at System.Windows.Forms.Application.Run(ApplicationContext context) at MyApp.Program.Main() 2. The faulting module seems to be a COM object (ChartFX Client Server 6.2) Using WinDbg (with SoS loaded), I caught it on the unmanaged side, inside ChartFX.ClientServer.Core.dll (that's a COM charting component we are using): (ca84.c98c): Access violation - code c0000005 (first chance) First chance exceptions are reported before any exception handling. This exception may be expected and handled. eax=00000000 ebx=06e67c38 ecx=06e67c38 edx=000018c6 esi=06e7df30 edi=317a9e80 eip=31666110 esp=0015e040 ebp=0015e08c iopl=0 nv up ei pl zr na pe nc cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010246 ChartFX_ClientServer_Core!Ordinal5507+0x97b7: 31666110 8a404d mov al,byte ptr [eax+4Dh] ds:0023:0000004d=?? [edit:] I also wasn't able to get the unmamanged stack details from WinDbg (it said "Stack unwind info not available"): 0:000 kP ChildEBP RetAddr WARNING: Stack unwind information not available. Following frames may be wrong. 0015e08c 3166288b ChartFX_ClientServer_Core!Ordinal5507+0x97b7 0015e394 3165a921 ChartFX_ClientServer_Core!Ordinal5507+0x5f32 0015e480 31678685 ChartFX_ClientServer_Core!Ordinal5496+0x26a 0015e568 3167bef4 ChartFX_ClientServer_Core!Ordinal5492+0x975 0015e668 316a356b ChartFX_ClientServer_Core!Ordinal5492+0x41e4 0015e77c 31709496 ChartFX_ClientServer_Core!Ordinal443+0x5745 0015e7d0 31707f70 ChartFX_ClientServer_Core!Ordinal2584+0x3cdc 0015e7f8 3170817d ChartFX_ClientServer_Core!Ordinal2584+0x27b6 0015e81c 3162fd76 ChartFX_ClientServer_Core!Ordinal2584+0x29c3 0015e86c 7719f8d2 ChartFX_ClientServer_Core!Ordinal899+0x6b6 0015e898 7719f794 USER32!GetMessageW+0x93 0015e910 771a06f6 USER32!GetWindowLongW+0x115 0015e940 771a069c USER32!CallWindowProcW+0x75 0015e960 747fcef4 USER32!CallWindowProcW+0x1b 0015e97c 747fd073 comctl32!Ordinal377+0x5c 0015e9e0 747fd027 comctl32!DefSubclassProc+0x92 0015ea04 747fd4e6 comctl32!DefSubclassProc+0x46 0015ea20 747fd073 comctl32!DefSubclassProc+0x505 0015ea84 747fd118 comctl32!DefSubclassProc+0x92 0015eae4 7719f8d2 comctl32!DefSubclassProc+0x137 3. Bug is not easy to reproduce (although it can be provoked usually in less than 5 min.) I have several Chart instances in several TabPages, and this usually happens while I am switching the tabs. I still don't know how to reproduce it, besides switching those tabs for several minutes before it happens, so I cannot use our source control to reliably find the build which didn't have this problem. I am accessing the charts through the managed AxChart wrapper class (derived from AxHost), which was created by VS designer automatically. 4. What should be my next step? If someone could point me to the next step I should do to find the actual cause, I would be very grateful. Experimenting (removing and returning code) does not do much good, because I don't know how to reproduce it, so it would take large amounts of time on each iteration just to convince myself that the bug is still there. I have found that people often suggest something like "switching compiler optimizations", but since the exception is not thrown deterministically, I don't want to simply rearrange some bytes and hope that it never returns. Thanks a lot in advance! Best regards, Groo

  Read the article

• Is the Cloud ready for an Enterprise Java web application? Seeking a JEE hosting advice.

  - by Jakub Holý

  Greetings to all the smart people around here! I'd like to ask whether it is feasible or a good idea at all to deploy a Java enterprise web application to a Cloud such as Amazon EC2. More exactly, I'm looking for infrastructure options for an application that shall handle few hundred users with long but neither CPU nor memory intensive sessions. I'm considering dedicated servers, virtual private servers (VPSs) and EC2. I've noticed that there is a project called JBoss Cloud so people are working on enabling such a deployment, on the other hand it doesn't seem to be mature yet and I'm not sure that the cloud is ready for this kind of applications, which differs from the typical cloud-based applications like Twitter. Would you recommend to deploy it to the cloud? What are the pros and cons? The application is a Java EE 5 web application whose main function is to enable users to compose their own customized Product by combining the available Parts. It uses stateless and stateful session beans and JPA for persistence of entities to a RDBMS and fetches information about Parts from the company's inventory system via a web service. Aside of external users it's used also by few internal ones, who are authenticated against the company's LDAP. The application should handle around 300-400 concurrent users building their product and should be reasonably scalable and available though these qualities are only of a medium importance at this stage. I've proposed an architecture consisting of a firewall (FW) and load balancer supporting sticky sessions and https (in the Cloud this would be replaced with EC2's Elastic Load Balancing service and FW on the app. servers, in a physical architecture the load-balancer would be a HW), then two physical clustered application servers combined with web servers (so that if one fails, a user doesn't loose his/her long built product) and finally a database server. The DB server would need a slave backup instance that can replace the master instance if it fails. This should provide reasonable availability and fault tolerance and provide good scalability as long as a single RDBMS can keep with the load, which should be OK for quite a while because most of the operations are done in the memory using a stateful bean and only occasionally stored or retrieved from the DB and the amount of data is low too. A problematic part could be the dependency on the remote inventory system webservice but with good caching of its outputs in the application it should be OK too. Unfortunately I've only vague idea of the system resources (memory size, number and speed of CPUs/cores) that such an "average Java EE application" for few hundred users needs. My rough and mostly unfounded estimate based on actual Amazon offerings is that 1.7GB and a single, 2-core "modern CPU" with speed around 2.5GHz (the High-CPU Medium Instance) should be sufficient for any of the two application servers (since we can handle higher load by provisioning more of them). Alternatively I would consider using the Large instance (64b, 7.5GB RAM, 2 cores at 1GHz) So my question is whether such a deployment to the cloud is technically and financially feasible or whether dedicated/VPS servers would be a better option and whether there are some real-world experiences with something similar. Thank you very much! /Jakub Holy PS: I've found the JBoss EAP in a Cloud Case Study that shows that it is possible to deploy a real-world Java EE application to the EC2 cloud but unfortunately there're no details regarding topology, instance types, or anything :-(

  Read the article

• Android: OutOfMemoryError while uploading video - how best to chunk?

  - by AP257

  Hi all, I have the same problem as described here, but I will supply a few more details. While trying to upload a video in Android, I'm reading it into memory, and if the video is large I get an OutOfMemoryError. Here's my code: // get bytestream to upload videoByteArray = getBytesFromFile(cR, fileUriString); public static byte[] getBytesFromFile(ContentResolver cR, String fileUriString) throws IOException { Uri tempuri = Uri.parse(fileUriString); InputStream is = cR.openInputStream(tempuri); byte[] b3 = readBytes(is); is.close(); return b3; } public static byte[] readBytes(InputStream inputStream) throws IOException { ByteArrayOutputStream byteBuffer = new ByteArrayOutputStream(); // this is storage overwritten on each iteration with bytes int bufferSize = 1024; byte[] buffer = new byte[bufferSize]; int len = 0; while ((len = inputStream.read(buffer)) != -1) { byteBuffer.write(buffer, 0, len); } return byteBuffer.toByteArray(); } And here's the traceback (the error is thrown on the byteBuffer.write(buffer, 0, len) line): 04-08 11:56:20.456: ERROR/dalvikvm-heap(6088): Out of memory on a 16775184-byte allocation. 04-08 11:56:20.456: INFO/dalvikvm(6088): "IntentService[UploadService]" prio=5 tid=17 RUNNABLE 04-08 11:56:20.456: INFO/dalvikvm(6088): | group="main" sCount=0 dsCount=0 s=N obj=0x449a3cf0 self=0x38d410 04-08 11:56:20.456: INFO/dalvikvm(6088): | sysTid=6119 nice=0 sched=0/0 cgrp=default handle=4010416 04-08 11:56:20.456: INFO/dalvikvm(6088): at java.io.ByteArrayOutputStream.expand(ByteArrayOutputStream.java:~93) 04-08 11:56:20.456: INFO/dalvikvm(6088): at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:218) 04-08 11:56:20.456: INFO/dalvikvm(6088): at com.android.election2010.UploadService.readBytes(UploadService.java:199) 04-08 11:56:20.456: INFO/dalvikvm(6088): at com.android.election2010.UploadService.getBytesFromFile(UploadService.java:182) 04-08 11:56:20.456: INFO/dalvikvm(6088): at com.android.election2010.UploadService.doUploadinBackground(UploadService.java:118) 04-08 11:56:20.456: INFO/dalvikvm(6088): at com.android.election2010.UploadService.onHandleIntent(UploadService.java:85) 04-08 11:56:20.456: INFO/dalvikvm(6088): at android.app.IntentService$ServiceHandler.handleMessage(IntentService.java:30) 04-08 11:56:20.456: INFO/dalvikvm(6088): at android.os.Handler.dispatchMessage(Handler.java:99) 04-08 11:56:20.456: INFO/dalvikvm(6088): at android.os.Looper.loop(Looper.java:123) 04-08 11:56:20.456: INFO/dalvikvm(6088): at android.os.HandlerThread.run(HandlerThread.java:60) 04-08 11:56:20.467: WARN/dalvikvm(6088): threadid=17: thread exiting with uncaught exception (group=0x4001b180) 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): Uncaught handler: thread IntentService[UploadService] exiting due to uncaught exception 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): java.lang.OutOfMemoryError 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): at java.io.ByteArrayOutputStream.expand(ByteArrayOutputStream.java:93) 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:218) 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): at com.android.election2010.UploadService.readBytes(UploadService.java:199) 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): at com.android.election2010.UploadService.getBytesFromFile(UploadService.java:182) 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): at com.android.election2010.UploadService.doUploadinBackground(UploadService.java:118) 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): at com.android.election2010.UploadService.onHandleIntent(UploadService.java:85) 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): at android.app.IntentService$ServiceHandler.handleMessage(IntentService.java:30) 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): at android.os.Handler.dispatchMessage(Handler.java:99) 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): at android.os.Looper.loop(Looper.java:123) 04-08 11:56:20.467: ERROR/AndroidRuntime(6088): at android.os.HandlerThread.run(HandlerThread.java:60) 04-08 11:56:20.496: INFO/Process(4657): Sending signal. PID: 6088 SIG: 3 I guess that as @DroidIn suggests, I need to upload it in chunks. But (newbie question alert) does that mean that I should make multiple PostMethod requests, and glue the file together at the server end? Or can I load the bytestream into memory in chunks, and glue it together in the Android code? If anyone could give me a clue as to the best approach, I would be very grateful.

  Read the article

• How to debug EXC_BAD_ACCESS that occurs only on release target for an iPhone app?

  - by Bobby Moretti

  I'm developing an iPhone application. I have an EXC_BAD_ACCESS that occurs only in the release target; when I build the debug target the exception does not occur. However, when I set the NSZombieEnabled environment variable to YES, I still get the EXC_BAD_ACCESS with no further information. Is it even possible for NSZombieEnabled to work when executing the release target? I don't see why not, since gdb is running in both cases... Update: here is a printout of the top of the stack: #0 0x33369ebc in objc_msgSend () #1 0x3144f968 in -[EAInputStream _streamEventTrigger] () #2 0x3144fe78 in __streamEventTrigger () #3 0x338ae3a6 in CFRunLoopRunSpecific () #4 0x338adc1e in CFRunLoopRunInMode () #5 0x32ed6966 in -[NSRunLoop(NSRunLoop) runMode:beforeDate:] () #6 0x00005b06 in -[IOStreamDelegate removeMsg:] (self=0x142cc0, _cmd=<value temporarily unavailable, due to optimizations>, message=0x2fffe544) at /Users/robertmoretti/Documents/XXXXXXX/IOStreamDelegate.m:191 Here is a gdb session from inside the objc_msgSend call at the top: (gdb) p/x $r0 $6 = 0x3100000 (gdb) x/s $r1 0x32d7cff8: "release" (gdb) disassemble $pc Dump of assembler code for function objc_msgSend: 0x33369ea8 <objc_msgSend+0>: teq r0, #0 ; 0x0 0x33369eac <objc_msgSend+4>: moveq r1, #0 ; 0x0 0x33369eb0 <objc_msgSend+8>: bxeq lr 0x33369eb4 <objc_msgSend+12>: push {r3, r4, r5, r6} 0x33369eb8 <objc_msgSend+16>: ldr r4, [r0] 0x33369ebc <objc_msgSend+20>: ldr r5, [r4, #8] 0x33369ec0 <objc_msgSend+24>: ldr r6, [r5] 0x33369ec4 <objc_msgSend+28>: add r3, r5, #8 ; 0x8 0x33369ec8 <objc_msgSend+32>: and r5, r6, r1, lsr #2 0x33369ecc <objc_msgSend+36>: ldr r4, [r3, r5, lsl #2] 0x33369ed0 <objc_msgSend+40>: teq r4, #0 ; 0x0 0x33369ed4 <objc_msgSend+44>: add r5, r5, #1 ; 0x1 0x33369ed8 <objc_msgSend+48>: beq 0x33369efc <objc_msgSend+84> 0x33369edc <objc_msgSend+52>: ldr r12, [r4] 0x33369ee0 <objc_msgSend+56>: teq r1, r12 0x33369ee4 <objc_msgSend+60>: and r5, r5, r6 0x33369ee8 <objc_msgSend+64>: bne 0x33369ecc <objc_msgSend+36> 0x33369eec <objc_msgSend+68>: ldr r12, [r4, #8] 0x33369ef0 <objc_msgSend+72>: teq r4, r4 0x33369ef4 <objc_msgSend+76>: pop {r3, r4, r5, r6} 0x33369ef8 <objc_msgSend+80>: bx r12 0x33369efc <objc_msgSend+84>: pop {r3, r4, r5, r6} 0x33369f00 <objc_msgSend+88>: b 0x33369f04 <objc_msgSend_uncached> End of assembler dump. (gdb) p/x *$r0 $9 = 0x0 The exception occurs on the line 0x33369ebc <objc_msgSend+20>: ldr r5, [r4, #8]. r4 has just been given the value pointed to by r0, which happens to be 0. I am wondering what is supposed to be in the memory region of 0x3100000. Here's a memory dump of that area: (gdb) x/256w 0x3100000 0x3100000: 0x00000000 0x0000a293 0xaa650505 0x00000000 0x3100010: 0x0000a294 0xaa670505 0x00000000 0x0000a295 0x3100020: 0xaa690505 0x00000000 0x0000a296 0xaa6b0505 0x3100030: 0x00000000 0x0000a297 0xaa6d0505 0x00000000 0x3100040: 0x0000a298 0xaa6f0505 0x00000000 0x0000a299 0x3100050: 0xaa710505 0x00000000 0x0000a29a 0xaa730505 0x3100060: 0x00000000 0x0000a29b 0xaa750505 0x00000000 0x3100070: 0x0000a29c 0xaa770505 0x00000000 0x0000a29d 0x3100080: 0xaa790505 0x00000000 0x0000a29e 0xaa7b0505 0x3100090: 0x00000000 0x0000a29f 0xaa7d0505 0x00000000 0x31000a0: 0x0000a2a0 0xaa7f0505 0x00000000 0x0000a2a1 0x31000b0: 0xaa810505 0x00000000 0x0000a2a2 0xaa830505 0x31000c0: 0x00000000 0x0000a2a3 0xaa850505 0x00000000 0x31000d0: 0x0000a2a4 0xaa870505 0x00000000 0x0000a2a5 0x31000e0: 0xaa890505 0x00000000 0x0000a2a6 0xaa8b0505 0x31000f0: 0x00000000 0x0000a2a7 0xaa8d0505 0x00000000 0x3100100: 0x0000a2a8 0xaa8f0505 0x00000000 0x0000a2a9 0x3100110: 0xaa910505 0x00000000 0x0000a2aa 0xaa930505 0x3100120: 0x00000000 0x0000a2ab 0xaa950505 0x00000000 0x3100130: 0x0000a2ac 0xaa970505 0x00000000 0x0000a2ad 0x3100140: 0xaa990505 0x00000000 0x0000a2ae 0xaa9b0505 0x3100150: 0x00000000 0x0000a2af 0xaa9d0505 0x00000000 0x3100160: 0x0000a2b0 0xaa9f0505 0x00000000 0x0000a2b1 0x3100170: 0xaaa10505 0x00000000 0x0000a2b2 0xaaa30505 0x3100180: 0x00000000 0x0000a2b3 0xaaa50505 0x00000000 0x3100190: 0x0000a2b4 0xaaa70505 0x00000000 0x0000a2b5 0x31001a0: 0xaaa90505 0x00000000 0x0000a2b6 0xaaab0505 0x31001b0: 0x00000000 0x0000a2b7 0xaaad0505 0x00000000 0x31001c0: 0x0000a2b8 0xaaaf0505 0x00000000 0x0000a2b9 0x31001d0: 0xaab10505 0x00000000 0x0000a2ba 0xaab30505 0x31001e0: 0x00000000 0x0000a2bb 0xaab50505 0x00000000 0x31001f0: 0x0000a2bc 0xaab70505 0x00000000 0x0000a2bd 0x3100200: 0xaab90505 0x00000000 0x0000a2be 0xaabb0505 0x3100210: 0x00000000 0x0000a2bf 0xaabd0505 0x00000000 0x3100220: 0x0000a2c0 0xaabf0505 0x00000000 0x0000a2c1 0x3100230: 0xaac10505 0x00000000 0x0000a2c2 0xaac30505 0x3100240: 0x00000000 0x0000a2c3 0xaac50505 0x00000000 0x3100250: 0x0000a2c4 0xaac70505 0x00000000 0x0000a2c5 0x3100260: 0xaac90505 0x00000000 0x0000a2c6 0xaacb0505 0x3100270: 0x00000000 0x0000a2c7 0xaacd0505 0x00000000 0x3100280: 0x0000a2c8 0xaacf0505 0x00000000 0x0000a2c9 0x3100290: 0xaad10505 0x00000000 0x0000a2ca 0xaad30505 0x31002a0: 0x00000000 0x0000a2cb 0xaad50505 0x00000000 0x31002b0: 0x0000a2cc 0xaad70505 0x00000000 0x0000a2cd 0x31002c0: 0xaad90505 0x00000000 0x0000a2ce 0xaadb0505 0x31002d0: 0x00000000 0x0000a2cf 0xaadd0505 0x00000000 0x31002e0: 0x0000a2d0 0xaadf0505 0x00000000 0x0000a2d1 0x31002f0: 0xaae10505 0x00000000 0x0000a2d2 0xaae30505 0x3100300: 0x00000000 0x0000a2d3 0xaae50505 0x00000000 0x3100310: 0x0000a2d4 0xaae70505 0x00000000 0x0000a2d5 0x3100320: 0xaae90505 0x00000000 0x0000a2d6 0xaaeb0505 0x3100330: 0x00000000 0x0000a2d7 0xaaed0505 0x00000000 0x3100340: 0x0000a2d8 0xaaef0505 0x00000000 0x0000a2d9 0x3100350: 0xaaf10505 0x00000000 0x0000a2da 0xaaf30505 0x3100360: 0x00000000 0x0000a2db 0xaaf50505 0x00000000 0x3100370: 0x0000a2dc 0xaaf70505 0x00000000 0x0000a2dd 0x3100380: 0xaaf90505 0x00000000 0x0000a2de 0xaafb0505 0x3100390: 0x00000000 0x0000a2df 0xaafd0505 0x00000000 0x31003a0: 0x0000a2e0 0xab050505 0x00000000 0x0000a2e1 0x31003b0: 0xab070505 0x00000000 0x0000a2e2 0xab090505 0x31003c0: 0x00000000 0x0000a2e3 0xab0b0505 0x00000000 0x31003d0: 0x0000a2e4 0xab0d0505 0x00000000 0x0000a2e5 0x31003e0: 0xab0f0505 0x00000000 0x0000a2e6 0xab110505 0x31003f0: 0x00000000 0x0000a2e7 0xab130505 0x00000000 I don't really know what else to try; hopefully someone with more iphone experience will be able to recognize this memory as something meaningful.

  Read the article

• Writing more efficient xquery code (avoiding redundant iteration)

  - by Coquelicot

  Here's a simplified version of a problem I'm working on: I have a bunch of xml data that encodes information about people. Each person is uniquely identified by an 'id' attribute, but they may go by many names. For example, in one document, I might find <person id=1>Paul Mcartney</person> <person id=2>Ringo Starr</person> And in another I might find: <person id=1>Sir Paul McCartney</person> <person id=2>Richard Starkey</person> I want to use xquery to produce a new document that lists every name associated with a given id. i.e.: <person id=1> <name>Paul McCartney</name> <name>Sir Paul McCartney</name> <name>James Paul McCartney</name> </person> <person id=2> ... </person> The way I'm doing this now in xquery is something like this (pseudocode-esque): let $ids := distinct-terms( [all the id attributes on people] ) for $id in $ids return <person id={$id}> { for $unique-name in distinct-values ( for $name in ( [all names] ) where $name/@id=$id return $name ) return <name>{$unique-name}</name> } </person> The problem is that this is really slow. I imagine the bottleneck is the innermost loop, which executes once for every id (of which there are about 1200). I'm dealing with a fair bit of data (300 MB, spread over about 800 xml files), so even a single execution of the query in the inner loop takes about 12 seconds, which means that repeating it 1200 times will take about 4 hours (which might be optimistic - the process has been running for 3 hours so far). Not only is it slow, it's using a whole lot of virtual memory. I'm using Saxon, and I had to set java's maximum heap size to 10 GB (!) to avoid getting out of memory errors, and it's currently using 6 GB of physical memory. So here's how I'd really like to do this (in Pythonic pseudocode): persons = {} for id in ids: person[id] = set() for person in all_the_people_in_my_xml_document: persons[person.id].add(person.name) There, I just did it in linear time, with only one sweep of the xml document. Now, is there some way to do something similar in xquery? Surely if I can imagine it, a reasonable programming language should be able to do it (he said quixotically). The problem, I suppose, is that unlike Python, xquery doesn't (as far as I know) have anything like an associative array. Is there some clever way around this? Failing that, is there something better than xquery that I might use to accomplish my goal? Because really, the computational resources I'm throwing at this relatively simple problem are kind of ridiculous.

  Read the article

• Compress file to bytes for uploading to SQL Server

  - by Chris

  I am trying to zip files to an SQL Server database table. I can't ensure that the user of the tool has write priveledges on the source file folder so I want to load the file into memory, compress it to an array of bytes and insert it into my database. This below does not work. class ZipFileToSql { public event MessageHandler Message; protected virtual void OnMessage(string msg) { if (Message != null) { MessageHandlerEventArgs args = new MessageHandlerEventArgs(); args.Message = msg; Message(this, args); } } private int sourceFileId; private SqlConnection Conn; private string PathToFile; private bool isExecuting; public bool IsExecuting { get { return isExecuting; } } public int SourceFileId { get { return sourceFileId; } } public ZipFileToSql(string pathToFile, SqlConnection conn) { isExecuting = false; PathToFile = pathToFile; Conn = conn; } public void Execute() { isExecuting = true; byte[] data; byte[] cmpData; //create temp zip file OnMessage("Reading file to memory"); FileStream fs = File.OpenRead(PathToFile); data = new byte[fs.Length]; ReadWholeArray(fs, data); OnMessage("Zipping file to memory"); MemoryStream ms = new MemoryStream(); GZipStream zip = new GZipStream(ms, CompressionMode.Compress, true); zip.Write(data, 0, data.Length); cmpData = new byte[ms.Length]; ReadWholeArray(ms, cmpData); OnMessage("Saving file to database"); using (SqlCommand cmd = Conn.CreateCommand()) { cmd.CommandText = @"MergeFileUploads"; cmd.CommandType = CommandType.StoredProcedure; //cmd.Parameters.Add("@File", SqlDbType.VarBinary).Value = data; cmd.Parameters.Add("@File", SqlDbType.VarBinary).Value = cmpData; SqlParameter p = new SqlParameter(); p.ParameterName = "@SourceFileId"; p.Direction = ParameterDirection.Output; p.SqlDbType = SqlDbType.Int; cmd.Parameters.Add(p); cmd.ExecuteNonQuery(); sourceFileId = (int)p.Value; } OnMessage("File Saved"); isExecuting = false; } private void ReadWholeArray(Stream stream, byte[] data) { int offset = 0; int remaining = data.Length; float Step = data.Length / 100; float NextStep = data.Length - Step; while (remaining > 0) { int read = stream.Read(data, offset, remaining); if (read <= 0) throw new EndOfStreamException (String.Format("End of stream reached with {0} bytes left to read", remaining)); remaining -= read; offset += read; if (remaining < NextStep) { NextStep -= Step; } } } }

  Read the article

• Large number of soft page faults when assigning a TJpegImage to a TBitmap

  - by Robert Oschler

  I have a Delphi 6 Pro application that processes incoming jpeg frames from a streaming video server. The code works but I recently noticed that it generates a huge number of soft page faults over time. After doing some investigation, the page faults appear to be coming from one particular graphics operation. Note, the uncompressed bitmaps in question are 320 x 240 or about 300 KB in size so it's not due to the handling of large images. The number of page faults being generated isn't tolerable. Over an hour it can easily top 1000000 page faults. I created a stripped down test case that executes the code I have included below on a timer, 10 times a second. The page faults appear to happen when I try to assign the TJpegImage to a TBitmap in the GetBitmap() method. I know this because I commented out that line and the page faults do not occur. The assign() triggers a decompression operation on the part of TJpegImage as it pushes the decompressed bits into a newly created bitmap that GetBitmap() returns. When I run Microsoft's pfmon utility (page fault monitor), I get a huge number of soft page fault error lines concerning RtlFillMemoryUlong, so it appears to happen during a memory buffer fill operation. One puzzling note. The summary part of pfmon's report where it shows which DLL caused what page fault does not show any DLL names in the far left column. I tried this on another system and it happens there too. Can anyone suggest a fix or a workaround? Here's the code. Note, IReceiveBufferForClientSocket is a simple class object that holds bytes in an accumulating buffer. function GetBitmap(theJpegImage: TJpegImage): Graphics.TBitmap; begin Result := TBitmap.Create; Result.Assign(theJpegImage); end; // --------------------------------------------------------------- procedure processJpegFrame(intfReceiveBuffer: IReceiveBufferForClientSocket); var theBitmap: TBitmap; theJpegStream, theBitmapStream: TMemoryStream; theJpegImage: TJpegImage; begin theBitmap := nil; theJpegImage := TJPEGImage.Create; theJpegStream:= TMemoryStream.Create; theBitmapStream := TMemoryStream.Create; try // 2 // ************************ BEGIN JPEG FRAME PROCESSING // Load the JPEG image from the receive buffer. theJpegStream.Size := intfReceiveBuffer.numBytesInBuffer; Move(intfReceiveBuffer.bufPtr^, theJpegStream.Memory^, intfReceiveBuffer.numBytesInBuffer); theJpegImage.LoadFromStream(theJpegStream); // Convert to bitmap. theBitmap := GetBitmap(theJpegImage); finally // Free memory objects. if Assigned(theBitmap) then theBitmap.Free; if Assigned(theJpegImage) then theJpegImage.Free; if Assigned(theBitmapStream) then theBitmapStream.Free; if Assigned(theJpegStream) then theJpegStream.Free; end; // try() end; // --------------------------------------------------------------- procedure TForm1.Timer1Timer(Sender: TObject); begin processJpegFrame(FIntfReceiveBufferForClientSocket); end; // --------------------------------------------------------------- procedure TForm1.FormCreate(Sender: TObject); var S: string; begin FIntfReceiveBufferForClientSocket := TReceiveBufferForClientSocket.Create(1000000); S := loadStringFromFile('c:\test.jpg'); FIntfReceiveBufferForClientSocket.assign(S); end; // --------------------------------------------------------------- Thanks, Robert

  Read the article

• Java: how to do fast copy of a BufferedImage's pixels? (include unit test)

  - by WizardOfOdds

  I want to do a copy (of a rectangle area) of the ARGB values from a source BufferedImage into a destination BufferedImage. No compositing should be done: if I copy a pixel with an ARGB value of 0x8000BE50 (alpha value at 128), then the destination pixel must be exactly 0x8000BE50, totally overriding the destination pixel. I've got a very precise question and I made a unit test to show what I need. The unit test is fully functional and self-contained and is passing fine and is doing precisely what I want. However, I want a faster and more memory efficient method to replace copySrcIntoDstAt(...). That's the whole point of my question: I'm not after how to "fill" the image in a faster way (what I did is just an example to have a unit test). All I want is to know what would be a fast and memory efficient way to do it (ie fast and not creating needless objects). The proof-of-concept implementation I've made is obviously very memory efficient, but it is slow (doing one getRGB and one setRGB for every pixel). Schematically, I've got this: (where A indicates corresponding pixels from the destination image before the copy) AAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAA And I want to have this: AAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAABBBBAAA AAAAAAAAAAAAABBBBAAA AAAAAAAAAAAAAAAAAAAA where 'B' represents the pixels from the src image. I'm looking for an exact replacement of the method, not for an API link/quote. import org.junit.Test; import java.awt.image.BufferedImage; import static org.junit.Assert.*; public class TestCopy { private static final int COL1 = 0x8000BE50; // alpha at 128 private static final int COL2 = 0x1732FE87; // alpha at 23 @Test public void testPixelsCopy() { final BufferedImage src = new BufferedImage( 5, 5, BufferedImage.TYPE_INT_ARGB ); final BufferedImage dst = new BufferedImage( 20, 20, BufferedImage.TYPE_INT_ARGB ); convenienceFill( src, COL1 ); convenienceFill( dst, COL2 ); copySrcIntoDstAt( src, dst, 3, 4 ); for (int x = 0; x < dst.getWidth(); x++) { for (int y = 0; y < dst.getHeight(); y++) { if ( x >= 3 && x <= 7 && y >= 4 && y <= 8 ) { assertEquals( COL1, dst.getRGB(x,y) ); } else { assertEquals( COL2, dst.getRGB(x,y) ); } } } } // clipping is unnecessary private static void copySrcIntoDstAt( final BufferedImage src, final BufferedImage dst, final int dx, final int dy ) { // TODO: replace this by a much more efficient method for (int x = 0; x < src.getWidth(); x++) { for (int y = 0; y < src.getHeight(); y++) { dst.setRGB( dx + x, dy + y, src.getRGB(x,y) ); } } } // This method is just a convenience method, there's // no point in optimizing this method, this is not what // this question is about private static void convenienceFill( final BufferedImage bi, final int color ) { for (int x = 0; x < bi.getWidth(); x++) { for (int y = 0; y < bi.getHeight(); y++) { bi.setRGB( x, y, color ); } } } }

  Read the article

• Debugging EXC_BAD_ACCESS that occurs only on release target for an iPhone app?

  - by Bobby Moretti

  I'm developing an iPhone application. I have an EXC_BAD_ACCESS that occurs only in the release target; when I build the debug target the exception does not occur. However, when I set the NSZombieEnabled environment variable to YES, I still get the EXC_BAD_ACCESS with no further information. Is it even possible for NSZombieEnabled to work when executing the release target? I don't see why not, since gdb is running in both cases... Update: here is a printout of the top of the stack: #0 0x33369ebc in objc_msgSend () #1 0x3144f968 in -[EAInputStream _streamEventTrigger] () #2 0x3144fe78 in __streamEventTrigger () #3 0x338ae3a6 in CFRunLoopRunSpecific () #4 0x338adc1e in CFRunLoopRunInMode () #5 0x32ed6966 in -[NSRunLoop(NSRunLoop) runMode:beforeDate:] () #6 0x00005b06 in -[IOStreamDelegate removeMsg:] (self=0x142cc0, _cmd=<value temporarily unavailable, due to optimizations>, message=0x2fffe544) at /Users/robertmoretti/Documents/XXXXXXX/IOStreamDelegate.m:191 Here is a gdb session from inside the objc_msgSend call at the top: (gdb) p/x $r0 $6 = 0x3100000 (gdb) x/s $r1 0x32d7cff8: "release" (gdb) disassemble $pc Dump of assembler code for function objc_msgSend: 0x33369ea8 <objc_msgSend+0>: teq r0, #0 ; 0x0 0x33369eac <objc_msgSend+4>: moveq r1, #0 ; 0x0 0x33369eb0 <objc_msgSend+8>: bxeq lr 0x33369eb4 <objc_msgSend+12>: push {r3, r4, r5, r6} 0x33369eb8 <objc_msgSend+16>: ldr r4, [r0] 0x33369ebc <objc_msgSend+20>: ldr r5, [r4, #8] 0x33369ec0 <objc_msgSend+24>: ldr r6, [r5] 0x33369ec4 <objc_msgSend+28>: add r3, r5, #8 ; 0x8 0x33369ec8 <objc_msgSend+32>: and r5, r6, r1, lsr #2 0x33369ecc <objc_msgSend+36>: ldr r4, [r3, r5, lsl #2] 0x33369ed0 <objc_msgSend+40>: teq r4, #0 ; 0x0 0x33369ed4 <objc_msgSend+44>: add r5, r5, #1 ; 0x1 0x33369ed8 <objc_msgSend+48>: beq 0x33369efc <objc_msgSend+84> 0x33369edc <objc_msgSend+52>: ldr r12, [r4] 0x33369ee0 <objc_msgSend+56>: teq r1, r12 0x33369ee4 <objc_msgSend+60>: and r5, r5, r6 0x33369ee8 <objc_msgSend+64>: bne 0x33369ecc <objc_msgSend+36> 0x33369eec <objc_msgSend+68>: ldr r12, [r4, #8] 0x33369ef0 <objc_msgSend+72>: teq r4, r4 0x33369ef4 <objc_msgSend+76>: pop {r3, r4, r5, r6} 0x33369ef8 <objc_msgSend+80>: bx r12 0x33369efc <objc_msgSend+84>: pop {r3, r4, r5, r6} 0x33369f00 <objc_msgSend+88>: b 0x33369f04 <objc_msgSend_uncached> End of assembler dump. (gdb) p/x *$r0 $9 = 0x0 The exception occurs on the line 0x33369ebc <objc_msgSend+20>: ldr r5, [r4, #8]. r4 has just been given the value pointed to by r0, which happens to be 0. I am wondering what is supposed to be in the memory region of 0x3100000. Here's a memory dump of that area: (gdb) x/256w 0x3100000 0x3100000: 0x00000000 0x0000a293 0xaa650505 0x00000000 0x3100010: 0x0000a294 0xaa670505 0x00000000 0x0000a295 0x3100020: 0xaa690505 0x00000000 0x0000a296 0xaa6b0505 0x3100030: 0x00000000 0x0000a297 0xaa6d0505 0x00000000 0x3100040: 0x0000a298 0xaa6f0505 0x00000000 0x0000a299 0x3100050: 0xaa710505 0x00000000 0x0000a29a 0xaa730505 0x3100060: 0x00000000 0x0000a29b 0xaa750505 0x00000000 0x3100070: 0x0000a29c 0xaa770505 0x00000000 0x0000a29d 0x3100080: 0xaa790505 0x00000000 0x0000a29e 0xaa7b0505 0x3100090: 0x00000000 0x0000a29f 0xaa7d0505 0x00000000 0x31000a0: 0x0000a2a0 0xaa7f0505 0x00000000 0x0000a2a1 0x31000b0: 0xaa810505 0x00000000 0x0000a2a2 0xaa830505 0x31000c0: 0x00000000 0x0000a2a3 0xaa850505 0x00000000 0x31000d0: 0x0000a2a4 0xaa870505 0x00000000 0x0000a2a5 0x31000e0: 0xaa890505 0x00000000 0x0000a2a6 0xaa8b0505 0x31000f0: 0x00000000 0x0000a2a7 0xaa8d0505 0x00000000 0x3100100: 0x0000a2a8 0xaa8f0505 0x00000000 0x0000a2a9 0x3100110: 0xaa910505 0x00000000 0x0000a2aa 0xaa930505 0x3100120: 0x00000000 0x0000a2ab 0xaa950505 0x00000000 0x3100130: 0x0000a2ac 0xaa970505 0x00000000 0x0000a2ad 0x3100140: 0xaa990505 0x00000000 0x0000a2ae 0xaa9b0505 0x3100150: 0x00000000 0x0000a2af 0xaa9d0505 0x00000000 0x3100160: 0x0000a2b0 0xaa9f0505 0x00000000 0x0000a2b1 0x3100170: 0xaaa10505 0x00000000 0x0000a2b2 0xaaa30505 0x3100180: 0x00000000 0x0000a2b3 0xaaa50505 0x00000000 0x3100190: 0x0000a2b4 0xaaa70505 0x00000000 0x0000a2b5 0x31001a0: 0xaaa90505 0x00000000 0x0000a2b6 0xaaab0505 0x31001b0: 0x00000000 0x0000a2b7 0xaaad0505 0x00000000 0x31001c0: 0x0000a2b8 0xaaaf0505 0x00000000 0x0000a2b9 0x31001d0: 0xaab10505 0x00000000 0x0000a2ba 0xaab30505 0x31001e0: 0x00000000 0x0000a2bb 0xaab50505 0x00000000 0x31001f0: 0x0000a2bc 0xaab70505 0x00000000 0x0000a2bd 0x3100200: 0xaab90505 0x00000000 0x0000a2be 0xaabb0505 0x3100210: 0x00000000 0x0000a2bf 0xaabd0505 0x00000000 0x3100220: 0x0000a2c0 0xaabf0505 0x00000000 0x0000a2c1 0x3100230: 0xaac10505 0x00000000 0x0000a2c2 0xaac30505 0x3100240: 0x00000000 0x0000a2c3 0xaac50505 0x00000000 0x3100250: 0x0000a2c4 0xaac70505 0x00000000 0x0000a2c5 0x3100260: 0xaac90505 0x00000000 0x0000a2c6 0xaacb0505 0x3100270: 0x00000000 0x0000a2c7 0xaacd0505 0x00000000 0x3100280: 0x0000a2c8 0xaacf0505 0x00000000 0x0000a2c9 0x3100290: 0xaad10505 0x00000000 0x0000a2ca 0xaad30505 0x31002a0: 0x00000000 0x0000a2cb 0xaad50505 0x00000000 0x31002b0: 0x0000a2cc 0xaad70505 0x00000000 0x0000a2cd 0x31002c0: 0xaad90505 0x00000000 0x0000a2ce 0xaadb0505 0x31002d0: 0x00000000 0x0000a2cf 0xaadd0505 0x00000000 0x31002e0: 0x0000a2d0 0xaadf0505 0x00000000 0x0000a2d1 0x31002f0: 0xaae10505 0x00000000 0x0000a2d2 0xaae30505 0x3100300: 0x00000000 0x0000a2d3 0xaae50505 0x00000000 0x3100310: 0x0000a2d4 0xaae70505 0x00000000 0x0000a2d5 0x3100320: 0xaae90505 0x00000000 0x0000a2d6 0xaaeb0505 0x3100330: 0x00000000 0x0000a2d7 0xaaed0505 0x00000000 0x3100340: 0x0000a2d8 0xaaef0505 0x00000000 0x0000a2d9 0x3100350: 0xaaf10505 0x00000000 0x0000a2da 0xaaf30505 0x3100360: 0x00000000 0x0000a2db 0xaaf50505 0x00000000 0x3100370: 0x0000a2dc 0xaaf70505 0x00000000 0x0000a2dd 0x3100380: 0xaaf90505 0x00000000 0x0000a2de 0xaafb0505 0x3100390: 0x00000000 0x0000a2df 0xaafd0505 0x00000000 0x31003a0: 0x0000a2e0 0xab050505 0x00000000 0x0000a2e1 0x31003b0: 0xab070505 0x00000000 0x0000a2e2 0xab090505 0x31003c0: 0x00000000 0x0000a2e3 0xab0b0505 0x00000000 0x31003d0: 0x0000a2e4 0xab0d0505 0x00000000 0x0000a2e5 0x31003e0: 0xab0f0505 0x00000000 0x0000a2e6 0xab110505 0x31003f0: 0x00000000 0x0000a2e7 0xab130505 0x00000000 I don't really know what else to try; hopefully someone with more iphone experience will be able to recognize this memory as something meaningful.

  Read the article

• Faster method for Matrix vector product for large matrix in C or C++ for use in GMRES

  - by user35959

  I have a large, dense matrix A, and I aim to find the solution to the linear system Ax=b using an iterative method (in MATLAB was the plan using its built in GMRES). For more than 10,000 rows, this is too much for my computer to store in memory, but I know that the entries in A are constructed by two known vectors x and y of length N and the entries satisfy: A(i,j) = .5*(x[i]-x[j])^2+([y[i]-y[j])^2 * log(x[i]-x[j])^2+([y[i]-y[j]^2). MATLAB's GMRES command accepts as input a function call that can compute the matrix vector product A*x, which allows me to handle larger matrices than I can store in memory. To write the matrix-vecotr product function, I first tried this in matlab by going row by row and using some vectorization, but I avoid spawning the entire array A (since it would be too large). This was fairly slow unfortnately in my application for GMRES. My plan was to write a mex file for MATLAB to, which is in C, and ideally should be significantly faster than the matlab code. I'm rather new to C, so this went rather poorly and my naive attempt at writing the code in C was slower than my partially vectorized attempt in Matlab. #include <math.h> #include "mex.h" void Aproduct(double *x, double *ctrs_x, double *ctrs_y, double *b, mwSize n) { mwSize i; mwSize j; double val; for (i=0; i<n; i++) { for (j=0; j<i; j++) { val = pow(ctrs_x[i]-ctrs_x[j],2)+pow(ctrs_y[i]-ctrs_y[j],2); b[i] = b[i] + .5* val * log(val) * x[j]; } for (j=i+1; j<n; j++) { val = pow(ctrs_x[i]-ctrs_x[j],2)+pow(ctrs_y[i]-ctrs_y[j],2); b[i] = b[i] + .5* val * log(val) * x[j]; } } } The above is the computational portion of the code for the matlab mex file (which is slightly modified C, if I understand correctly). Please note that I skip the case i=j, since in that case the variable val will be a 0*log(0), which should be interpreted as 0 for me, so I just skip it. Is there a more efficient or faster way to write this? When I call this C function via the mex file in matlab, it is quite slow, slower even than the matlab method I used. This surprises me since I suspected that C code should be much faster than matlab. The alternative matlab method which is partially vectorized that I am comparing it with is function Ax = Aprod(x,ctrs) n = length(x); Ax = zeros(n,1); for j=1:(n-3) v = .5*((ctrs(j,1)-ctrs(:,1)).^2+(ctrs(j,2)-ctrs(:,2)).^2).*log((ctrs(j,1)-ctrs(:,1)).^2+(ctrs(j,2)-ctrs(:,2)).^2); v(j)=0; Ax(j) = dot(v,x(1:n-3); end (the n-3 is because there is actually 3 extra components, but they are dealt with separately,so I excluded that code). This is partly vectorized and only needs one for loop, so it makes some sense that it is faster. However, I was hoping I could go even faster with C+mex file. Any suggestions or help would be greatly appreciated! Thanks! EDIT: I should be more clear. I am open to any faster method that can help me use GMRES to invert this matrix that I am interested in, which requires a faster way of doing the matrix vector product without explicitly loading the array into memory. Thanks!

  Read the article

• parallel computation for an Iterator of elements in Java

  - by Brian Harris

  I've had the same need a few times now and wanted to get other thoughts on the right way to structure a solution. The need is to perform some operation on many elements on many threads without needing to have all elements in memory at once, just the ones under computation. As in, Iterables.partition is insufficient because it brings all elements into memory up front. Expressing it in code, I want to write a BulkCalc2 that does the same thing as BulkCalc1, just in parallel. Below is sample code that illustrates my best attempt. I'm not satisfied because it's big and ugly, but it does seem to accomplish my goals of keeping threads highly utilized until the work is done, propagating any exceptions during computation, and not having more than numThreads instances of BigThing necessarily in memory at once. I'll accept the answer which meets the stated goals in the most concise way, whether it's a way to improve my BulkCalc2 or a completely different solution. interface BigThing { int getId(); String getString(); } class Calc { // somewhat expensive computation double calc(BigThing bigThing) { Random r = new Random(bigThing.getString().hashCode()); double d = 0; for (int i = 0; i < 100000; i++) { d += r.nextDouble(); } return d; } } class BulkCalc1 { final Calc calc; public BulkCalc1(Calc calc) { this.calc = calc; } public TreeMap<Integer, Double> calc(Iterator<BigThing> in) { TreeMap<Integer, Double> results = Maps.newTreeMap(); while (in.hasNext()) { BigThing o = in.next(); results.put(o.getId(), calc.calc(o)); } return results; } } class SafeIterator<T> { final Iterator<T> in; SafeIterator(Iterator<T> in) { this.in = in; } synchronized T nextOrNull() { if (in.hasNext()) { return in.next(); } return null; } } class BulkCalc2 { final Calc calc; final int numThreads; public BulkCalc2(Calc calc, int numThreads) { this.calc = calc; this.numThreads = numThreads; } public TreeMap<Integer, Double> calc(Iterator<BigThing> in) { ExecutorService e = Executors.newFixedThreadPool(numThreads); List<Future<?>> futures = Lists.newLinkedList(); final Map<Integer, Double> results = new MapMaker().concurrencyLevel(numThreads).makeMap(); final SafeIterator<BigThing> it = new SafeIterator<BigThing>(in); for (int i = 0; i < numThreads; i++) { futures.add(e.submit(new Runnable() { @Override public void run() { while (true) { BigThing o = it.nextOrNull(); if (o == null) { return; } results.put(o.getId(), calc.calc(o)); } } })); } e.shutdown(); for (Future<?> future : futures) { try { future.get(); } catch (InterruptedException ex) { // swallowing is OK } catch (ExecutionException ex) { throw Throwables.propagate(ex.getCause()); } } return new TreeMap<Integer, Double>(results); } }

  Read the article

• Lock free multiple readers single writer

  - by dummzeuch

  I have got an in memory data structure that is read by multiple threads and written by only one thread. Currently I am using a critical section to make this access threadsafe. Unfortunately this has the effect of blocking readers even though only another reader is accessing it. There are two options to remedy this: use TMultiReadExclusiveWriteSynchronizer do away with any blocking by using a lock free approach For 2. I have got the following so far (any code that doesn't matter has been left out): type TDataManager = class private FAccessCount: integer; FData: TDataClass; public procedure Read(out _Some: integer; out _Data: double); procedure Write(_Some: integer; _Data: double); end; procedure TDataManager.Read(out _Some: integer; out _Data: double); var Data: TDAtaClass; begin InterlockedIncrement(FAccessCount); try // make sure we get both values from the same TDataClass instance Data := FData; // read the actual data _Some := Data.Some; _Data := Data.Data; finally InterlockedDecrement(FAccessCount); end; end; procedure TDataManager.Write(_Some: integer; _Data: double); var NewData: TDataClass; OldData: TDataClass; ReaderCount: integer; begin NewData := TDataClass.Create(_Some, _Data); InterlockedIncrement(FAccessCount); OldData := TDataClass(InterlockedExchange(integer(FData), integer(NewData)); // now FData points to the new instance but there might still be // readers that got the old one before we exchanged it. ReaderCount := InterlockedDecrement(FAccessCount); if ReaderCount = 0 then // no active readers, so we can safely free the old instance FreeAndNil(OldData) else begin /// here is the problem end; end; Unfortunately there is the small problem of getting rid of the OldData instance after it has been replaced. If no other thread is currently within the Read method (ReaderCount=0), it can safely be disposed and that's it. But what can I do if that's not the case? I could just store it until the next call and dispose it there, but Windows scheduling could in theory let a reader thread sleep while it is within the Read method and still has got a reference to OldData. If you see any other problem with the above code, please tell me about it. This is to be run on computers with multiple cores and the above methods are to be called very frequently. In case this matters: I am using Delphi 2007 with the builtin memory manager. I am aware that the memory manager probably enforces some lock anyway when creating a new class but I want to ignore that for the moment. Edit: It may not have been clear from the above: For the full lifetime of the TDataManager object there is only one thread that writes to the data, not several that might compete for write access. So this is a special case of MREW.

  Read the article

• Severe Tomcat crash has me stumped

  - by Eric Banderhide

  I'm pretty used to crashing my tomcat server. But I don't think I made any changes at all and seem to have broken it completely. Was working one min then not the next. I really hope someone can help here is the Catalina.out: Dec 20, 2012 1:35:56 AM org.apache.catalina.startup.Embedded initDirs SEVERE: Cannot find specified temporary folder at Dec 20, 2012 1:35:57 AM org.apache.catalina.core.AprLifecycleListener init INFO: Loaded APR based Apache Tomcat Native library 1.1.23. Dec 20, 2012 1:35:57 AM org.apache.catalina.core.AprLifecycleListener init INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true]. Dec 20, 2012 1:35:57 AM org.apache.coyote.http11.Http11AprProtocol init INFO: Initializing Coyote HTTP/1.1 on http-8040 Dec 20, 2012 1:35:57 AM org.apache.coyote.ajp.AjpAprProtocol init INFO: Initializing Coyote AJP/1.3 on ajp-8042 Dec 20, 2012 1:35:57 AM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 587 ms Dec 20, 2012 1:35:57 AM org.apache.catalina.core.StandardService start INFO: Starting service Catalina Dec 20, 2012 1:35:57 AM org.apache.catalina.core.StandardEngine start INFO: Starting Servlet Engine: Apache Tomcat/6.0.33 Dec 20, 2012 1:35:57 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory docs Dec 20, 2012 1:35:57 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory ROOT Dec 20, 2012 1:35:57 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory myApp Dec 20, 2012 1:35:57 AM org.apache.catalina.loader.WebappClassLoader validateJarFile INFO: validateJarFile(/lhome/me/tomcat6/webapps/myApp/WEB-INF/lib/javax.servlet.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class Dec 20, 2012 1:35:58 AM org.apache.coyote.http11.Http11AprProtocol start INFO: Starting Coyote HTTP/1.1 on http-8040 Dec 20, 2012 1:35:58 AM org.apache.coyote.ajp.AjpAprProtocol start INFO: Starting Coyote AJP/1.3 on ajp-8042 Dec 20, 2012 1:35:58 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 880 ms Error Reading TLE line 1: java.lang.Exception: TLE line 1 not valid first line Error Creating SGP4 Satellite Dec 20, 2012 1:36:26 AM org.apache.coyote.http11.Http11AprProtocol pause INFO: Pausing Coyote HTTP/1.1 on http-8040 Dec 20, 2012 1:36:26 AM org.apache.coyote.ajp.AjpAprProtocol pause INFO: Pausing Coyote AJP/1.3 on ajp-8042 Dec 20, 2012 1:36:27 AM org.apache.catalina.core.StandardService stop INFO: Stopping service Catalina Dec 20, 2012 1:36:30 AM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc SEVERE: The web application [/myApp] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered. Dec 20, 2012 1:36:30 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/myApp] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. Dec 20, 2012 1:36:30 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/myApp] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak. Dec 20, 2012 1:36:30 AM org.apache.coyote.http11.Http11AprProtocol destroy INFO: Stopping Coyote HTTP/1.1 on http-8040 Dec 20, 2012 1:36:30 AM org.apache.coyote.ajp.AjpAprProtocol destroy INFO: Stopping Coyote AJP/1.3 on ajp-8042 Dec 20, 2012 1:37:12 AM org.apache.catalina.startup.Catalina stopServer SEVERE: Catalina.stop: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at java.net.Socket.<init>(Socket.java:425) at java.net.Socket.<init>(Socket.java:208) at org.apache.catalina.startup.Catalina.stopServer(Catalina.java:422) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.catalina.startup.Bootstrap.stopServer(Bootstrap.java:338) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:416) And I have a new log that has been created, I've never seen it before but it seems to think its something severe. A long one but here it is: Massive Error Log I've never seen before

  Read the article

• Problem using delete[] (Heap corruption) when implementing operator+= (C++)

  - by Darel

  I've been trying to figure this out for hours now, and I'm at my wit's end. I would surely appreciate it if someone could tell me when I'm doing wrong. I have written a simple class to emulate basic functionality of strings. The class's members include a character pointer data (which points to a dynamically created char array) and an integer strSize (which holds the length of the string, sans terminator.) Since I'm using new and delete, I've implemented the copy constructor and destructor. My problem occurs when I try to implement the operator+=. The LHS object builds the new string correctly - I can even print it using cout - but the problem comes when I try to deallocate the data pointer in the destructor: I get a "Heap Corruption Detected after normal block" at the memory address pointed to by the data array the destructor is trying to deallocate. Here's my complete class and test program: #include <iostream> using namespace std; // Class to emulate string class Str { public: // Default constructor Str(): data(0), strSize(0) { } // Constructor from string literal Str(const char* cp) { data = new char[strlen(cp) + 1]; char *p = data; const char* q = cp; while (*q) *p++ = *q++; *p = '\0'; strSize = strlen(cp); } Str& operator+=(const Str& rhs) { // create new dynamic memory to hold concatenated string char* str = new char[strSize + rhs.strSize + 1]; char* p = str; // new data char* i = data; // old data const char* q = rhs.data; // data to append // append old string to new string in new dynamic memory while (*p++ = *i++) ; p--; while (*p++ = *q++) ; *p = '\0'; // assign new values to data and strSize delete[] data; data = str; strSize += rhs.strSize; return *this; } // Copy constructor Str(const Str& s) { data = new char[s.strSize + 1]; char *p = data; char *q = s.data; while (*q) *p++ = *q++; *p = '\0'; strSize = s.strSize; } // destructor ~Str() { delete[] data; } const char& operator[](int i) const { return data[i]; } int size() const { return strSize; } private: char *data; int strSize; }; ostream& operator<<(ostream& os, const Str& s) { for (int i = 0; i != s.size(); ++i) os << s[i]; return os; } // Test constructor, copy constructor, and += operator int main() { Str s = "hello"; // destructor for s works ok Str x = s; // destructor for x works ok s += "world!"; // destructor for s gives error cout << s << endl; cout << x << endl; return 0; }

  Read the article

• How to implement this .NET client and PHP server synchronization scenario?

  - by pbean

  I have a PHP webserver and a .NET mobile application. The .NET application needs data from a database, which is provided (for now) by the php webserver. I'm fairly new to this kind of scenario so I'm not sure what the best practices are. I ran into a couple of problems and I am not certain how to overcome them. For now, I have the following setup. I run a PHP SOAP server which has a couple of operations which simply retrieve data from the database. For this web service I have created a WSDL file. In Visual Studio I added a web reference to my project using the WSDL file and it generated some classes for it. I simply call something like "MyWebService.GetItems();" and I get an array of items in my .NET application, which come straight from the database. Next I also serialize all these retrieved objects to local (permanent) storage. I face a couple of challenges which I don't know how to resolve properly. The idea is for the mobile client to synchronize the data once (at the start of the day), before working, and then use the local storage throughout the day, and synchronize it back at the end of the day. At the moment all data is downloaded through SOAP, and not a subset (only what is needed). How would I know which new information should be sent to the client? Only the server knows what is new, but only the client knows for sure which data it already has. I seem to be doing double work now. The data which is transferred with SOAP basically already are serialized objects. But at the moment I first retrieve all objects through SOAP and the .NET framework automatically deserializes it. Then I serialize all data again myself. It would be more efficient to simply download it to storage, and then deserialize it. The mobile device does not have a lot of memory. In my test data this is not really a problem, but with potentially tenths of thousands of records I can imagine this will become a problem. Currently when I call the SOAP method, it loads all data into memory (as an array) but as described above perhaps it would be better to have it stored into storage directly, and only retrieve from storage the objects that are needed. But how would I store this? An array would serialize to one big XML (or binary) file from which I cannot choose which objects to load. Would I make (possible tenths of thousands) separate files? Also at the end of the day when I want to send the changes back to the server, how would I know which objects to send... since they're not in memory? I hope my troubles are clear and I hope you are able to help me figure out how to implement this the best way. :) Some things are already fixed (like using .NET on the mobile device, using PHP and having a MySQL database server) but other things can most certainly be changed (like using SOAP).

  Read the article

• Sending typedef struct containing void* by creating MPI drived datatype.

  - by hankol

  what I understand studying MPI specification is that an MPI send primitive refer to a memory location (or a send buffer) pointed by the data to be sent and take the data in that location which then passed as a message to the another Process. Though it is true that virtual address of a give process will be meaningless in another process memory address; It is ok to send data pointed by pointer such as void pointer as MPI will any way pass the data itself as a message For example the following works correctly: // Sender Side. int x = 100; void* snd; MPI_Send(snd,4,MPI_BYTE,1,0,MPI_COMM_WORLD); // Receiver Side. void* rcv; MPI_Recv(rcv, 4,MPI_BYTE,0,0,MPI_COMM_WORLD); but when I add void* snd in a struct and try to send the struct this will no succeed. I don't understand why the previous example work correctly but not the following. Here, I have defined a typedef struct and then create an MPI_DataType from it. With the same explanation of the above the following should also have succeed, unfortunately it is not working. here is the code: #include "mpi.h" #include<stdio.h> int main(int args, char *argv[]) { int rank, source =0, tag=1, dest=1; int bloackCount[2]; MPI_Init(&args, &argv); typedef struct { void* data; int tag; } data; data myData; MPI_Datatype structType, oldType[2]; MPI_Status stat; /* MPI_Aint type used to idetify byte displacement of each block (array)*/ MPI_Aint offsets[2], extent; MPI_Comm_rank(MPI_COMM_WORLD, &rank); offsets[0] = 0; oldType[0] = MPI_BYTE; bloackCount[0] = 1; MPI_Type_extent(MPI_INT, &extent); offsets[1] = 4 * extent; /*let say the MPI_BYTE will contain ineteger : size of int * extent */ oldType[1] = MPI_INT; bloackCount[1] = 1; MPI_Type_create_struct(2, bloackCount,offsets,oldType, &structType); MPI_Type_commit(&structType); if(rank == 0){ int x = 100; myData.data = &x; myData.tag = 99; MPI_Send(&myData,1,structType, dest, tag, MPI_COMM_WORLD); } if(rank == 1 ){ MPI_Recv(&myData, 1, structType, source, tag, MPI_COMM_WORLD, &stat); // with out this the following printf() will properly print the value 99 for // myData.tag int x = *(int *) myData.data; printf(" \n Process %d, Received : %d , %d \n\n", rank , myData.tag, x); } MPI_Type_free(&structType); MPI_Finalize(); } Error message running the code: [Looks like I am trying to access an invalid memory address space in the second process] [ubuntu:04123] *** Process received signal *** [ubuntu:04123] Signal: Segmentation fault (11) [ubuntu:04123] Signal code: Address not mapped (1) [ubuntu:04123] Failing at address: 0xbfe008bc [ubuntu:04123] [ 0] [0xb778240c] [ubuntu:04123] [ 1] GenericstructType(main+0x161) [0x8048935] [ubuntu:04123] [ 2] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0xb750f4d3] [ubuntu:04123] [ 3] GenericstructType() [0x8048741] [ubuntu:04123] *** End of error message *** Can some please explain to me why it is not working. any advice will also be appreciated thanks,

  Read the article

• Fitting it together, database, reporting, applications in C#

  - by alvonellos

  Introduction Preamble I was hesitant to post this, since it's an application whose intricate details are defined elsewhere, and answers may not be helpful to others. Within the past few weeks (I was actually going to write a blog post about this after I finished) I've discovered that the barrier I'm encountering is one that's actually quite common for newer developers. This question is not so much about a specific thing as it is about piecing those things together. I've searched the internet far and wide, and found many tutorials on how to create applications that are kind of similar to what I'm looking for. I've also looked at hiring another, more experienced, developer to help me along, but all I've gotten are unqualified candidates that don't have the experience necessary and won't take care of the client or project like I will. I'd rather have the project never transpire than to release a solution that is half-baked. I've asked professors at my school, but they've not turned up answers to my question. I'm an experienced developer, and I've written many applications that are -- very abstractly -- close to what I'm doing, but my experiences from those applications aren't giving me enough leverage to solve this particular problem. I just hope that posting this article isn't a mistake for me to write. Project Description I have a project I'm working on for a client that is a rewrite of an application, originally written in Foxpro 2.6 by someone before me, that performs some analysis (which, sadly, I'm not allowed to disclose as per of my employment contract) on financial data. One day, after a long talk between the client and I -- where he intimately described his frustrations with all the bugs I've been hacking out of this code for 6 months now -- he told me to just rewrite it and gave me a month to write a good 1/8 of this 65k LOC Foxpro monstrosity. this 65k line of code foxpro monstrosity. It'll take me a good 3 - 6 months to rewrite this software (I know things the original programmer did not, like inheritance) going as I am right now, but I'm quickly discovering that I'm going to need to use databases. Prior to this contract I didn't even know about foxpro, and so I've had to learn foxpro on the fly, write procedures and make modifications to the database. I've actually come to like it, and this project would be rewritten in Foxpro if it were still a supported language, because over the past few months, I've come to like the features of Foxpro that make it so easy to develop data-driven applications. I once perfomed an experiment, comparing C# to Foxpro. What took me 45 minutes in C# took me two in Foxpro, and I knew C# prior to Foxpro. I was hoping to leverage the power of C#, but it intimidates me that in foxpro, you can have one line of code and be using a database. Prior to this, I have never written any serious database development from scratch. All the applications that I've written are in a different league. They are either completely data-naive or data-naive enough that I can get away with not using a database through serialization or by designing algorithms that work with the data in a manner that is stateless, so there is no need to worry about databases. I've come to realize, very quickly, that serialization and my efficacy with data structures has been my crutch all these years that's prevented me from adventuring into databases, and has consequently hindered my success in real-world programming. Sure, I've written some database stuff in Perl and Python, and I've done forms and worked with relational databases and tables, I'm a wizard in Access and Excel (seriously) and can do just about anything, but it just feels unnatural writing SQL code in another language... I don't mind writing SQL, and I don't It's that bridge between the database and the program code that drives me absolutely bonkers. I hope I'm not the only one to think this, but it bothers me that I have to create statements like the following string sSql = "SELECT * from tablename" When there's really no reason for that kind of unchecked language binding between two languages and two API's. Don't get my wrong, SQL is great, but I don't like the idea that, when executing commands on a SQL database, that one must intermix database and application software, and there's no database independence, which means that different versions of different databases can break code. This isn't very nice. The nicest thing about Foxpro is the cohesiveness between programming language and database. It's so easy, and Foxpro makes it easy, because the tool just fits the task. I can see why so many developers have created a career with this language, because it lowered the barrier of entry to data-driven applications that so many businesses need. It was wonderful. For my purposes today, with the demands and need for community support, extensibility, and language features, Foxpro isn't a solution that I feel would be the right tool for the job. I'm also worried about working too heavy with the database, because I've seen data-driven .NET applications have issues with database caches, running out of memory, and objects in the database not being collected. (Memory leaks) And OH the queries. Which one, how, and why? There are a plethora of different ways that a database can be setup, I think I counted 5 or 6 different kinds of database applications alone that I can chose from. That is a great mountain for me to climb when I don't even know where to begin when it comes to writing data-driven applications. The problem isn't that I don't know SQL or that I don't know C#. I know both and have worked with both extensively. It's making them work together that's the problem, and it's something I've never done in C# before. Reports The client likes paper. The data needs to be printed out in a format that is extensible, layered, and easy to use. I have never done reporting before, and so this is a bit of a problem. From the data source comes crystal reports, and so there's a dependency on the database, from what I understand. Code reuse A large part of the design decision that I've gone through so far is to break the task of writing a piece of this software into routines and modular DLL's and so forth such that much of the code can be reused. For example, when I setup this database, I want to be able to reuse the same database code over and over again. I also want to make sure that when the day comes that another developer is here, that he/she will be able to pick up just where I left off. The quicker I develop these applications, the better off I am. Tasks & Goals In my project, I need to write routines that apply algorithms and look for predefined patterns in financial data. Additionally, I need to simulate trading based on predefined algorithms and data. Then I need to prepare reports on that data. Additionally, I need to have a way to change the code base for this application quickly and effectively, without hacking together some band-aid solution for a problem that really needs a trauma ward. Special Considerations The solution must be fast, run quickly on existing hardware, and not be too much of a pain to maintain and write. I understand that anything I write I'm married to -- I'm responsible for the things that I write because my reputation and livelihood is dependent on it. Do I really need a database? What about performance? Performance was such a big issue that I hand wrote a data structure that is capable of performing 2 billion operations, using a total of 4 gigs of memory in under 1/4 of a second using the standard core two duo processor. I could not find a similar, pre-written data structure in C# to perform this task. What setup do I use in terms of database? What about reporting? I'd prefer to have PDF's generated, but I'd like to be able to visually sketch those reports and then just have a ReportFactory of some sort, that when I pass some variables in, it just does that data. About Me I'm a lone developer for a small business in this area. This is the first time I've done this and I've never had the breadth and depth of my knowledge tested. I'm incredibly frustrated with this project because I feel incredibly overwhelmed with the task at hand. I'm looking for that entry level point where I can draw a line and say "this is what I need to do" Conclusion I may have not been clear enough on my post. I'm still new to this whole thing, and I've been doing my best to contribute back to the community that I've leached so much knowledge from. I'd be glad to edit my post and add more information if possible. I'm looking for a big-picture solution or design process that helps me get off the ground in this world of data-driven applications, because I have a feeling that it's going to be concentric to my entire career as a programmer for some time. Specifically, if you didn't get it from the rest of the post (I may not have been clear enough) I really need some guidance as to where to go in terms of the design decisions for this project. Some things that'll be useful will be a pro/con list for the different kinds of database projects available in VS2010. I've tried, but generating that list has been as hard as solving the problem itself... If you could walk a developer writing a data-driven application for the first time in C#, how would you do that? Where would you point them to?

  Read the article

• C# Neural Networks with Encog

  - by JoshReuben

  Neural Networks ·       I recently read a book Introduction to Neural Networks for C# , by Jeff Heaton. http://www.amazon.com/Introduction-Neural-Networks-C-2nd/dp/1604390093/ref=sr_1_2?ie=UTF8&s=books&qid=1296821004&sr=8-2-spell. Not the 1st ANN book I've perused, but a nice revision.   ·       Artificial Neural Networks (ANNs) are a mechanism of machine learning – see http://en.wikipedia.org/wiki/Artificial_neural_network , http://en.wikipedia.org/wiki/Category:Machine_learning ·       Problems Not Suited to a Neural Network Solution- Programs that are easily written out as flowcharts consisting of well-defined steps, program logic that is unlikely to change, problems in which you must know exactly how the solution was derived. ·       Problems Suited to a Neural Network – pattern recognition, classification, series prediction, and data mining. Pattern recognition - network attempts to determine if the input data matches a pattern that it has been trained to recognize. Classification - take input samples and classify them into fuzzy groups. ·       As far as machine learning approaches go, I thing SVMs are superior (see http://en.wikipedia.org/wiki/Support_vector_machine ) - a neural network has certain disadvantages in comparison: an ANN can be overtrained, different training sets can produce non-deterministic weights and it is not possible to discern the underlying decision function of an ANN from its weight matrix – they are black box. ·       In this post, I'm not going to go into internals (believe me I know them). An autoassociative network (e.g. a Hopfield network) will echo back a pattern if it is recognized. ·       Under the hood, there is very little maths. In a nutshell - Some simple matrix operations occur during training: the input array is processed (normalized into bipolar values of 1, -1) - transposed from input column vector into a row vector, these are subject to matrix multiplication and then subtraction of the identity matrix to get a contribution matrix. The dot product is taken against the weight matrix to yield a boolean match result. For backpropogation training, a derivative function is required. In learning, hill climbing mechanisms such as Genetic Algorithms and Simulated Annealing are used to escape local minima. For unsupervised training, such as found in Self Organizing Maps used for OCR, Hebbs rule is applied. ·       The purpose of this post is not to mire you in technical and conceptual details, but to show you how to leverage neural networks via an abstraction API - Encog   Encog ·       Encog is a neural network API ·       Links to Encog: http://www.encog.org , http://www.heatonresearch.com/encog, http://www.heatonresearch.com/forum ·       Encog requires .Net 3.5 or higher – there is also a Silverlight version. Third-Party Libraries – log4net and nunit. ·       Encog supports feedforward, recurrent, self-organizing maps, radial basis function and Hopfield neural networks. ·       Encog neural networks, and related data, can be stored in .EG XML files. ·       Encog Workbench allows you to edit, train and visualize neural networks. The Encog Workbench can generate code. Synapses and layers ·       the primary building blocks - Almost every neural network will have, at a minimum, an input and output layer. In some cases, the same layer will function as both input and output layer. ·       To adapt a problem to a neural network, you must determine how to feed the problem into the input layer of a neural network, and receive the solution through the output layer of a neural network. ·       The Input Layer - For each input neuron, one double value is stored. An array is passed as input to a layer. Encog uses the interface INeuralData to hold these arrays. The class BasicNeuralData implements the INeuralData interface. Once the neural network processes the input, an INeuralData based class will be returned from the neural network's output layer. ·       convert a double array into an INeuralData object : INeuralData data = new BasicNeuralData(= new double[10]); ·       the Output Layer- The neural network outputs an array of doubles, wraped in a class based on the INeuralData interface. ·        The real power of a neural network comes from its pattern recognition capabilities. The neural network should be able to produce the desired output even if the input has been slightly distorted. ·       Hidden Layers– optional. between the input and output layers. very much a “black box”. If the structure of the hidden layer is too simple it may not learn the problem. If the structure is too complex, it will learn the problem but will be very slow to train and execute. Some neural networks have no hidden layers. The input layer may be directly connected to the output layer. Further, some neural networks have only a single layer. A single layer neural network has the single layer self-connected. ·       connections, called synapses, contain individual weight matrixes. These values are changed as the neural network learns. Constructing a Neural Network ·       the XOR operator is a frequent “first example” -the “Hello World” application for neural networks. ·       The XOR Operator- only returns true when both inputs differ. 0 XOR 0 = 0 1 XOR 0 = 1 0 XOR 1 = 1 1 XOR 1 = 0 ·       Structuring a Neural Network for XOR  - two inputs to the XOR operator and one output. ·       input: 0.0,0.0 1.0,0.0 0.0,1.0 1.0,1.0 ·       Expected output: 0.0 1.0 1.0 0.0 ·       A Perceptron - a simple feedforward neural network to learn the XOR operator. ·       Because the XOR operator has two inputs and one output, the neural network will follow suit. Additionally, the neural network will have a single hidden layer, with two neurons to help process the data. The choice for 2 neurons in the hidden layer is arbitrary, and often comes down to trial and error. ·       Neuron Diagram for the XOR Network ·       ·       The Encog workbench displays neural networks on a layer-by-layer basis. ·       Encog Layer Diagram for the XOR Network:   ·       Create a BasicNetwork - Three layers are added to this network. the FinalizeStructure method must be called to inform the network that no more layers are to be added. The call to Reset randomizes the weights in the connections between these layers. var network = new BasicNetwork(); network.AddLayer(new BasicLayer(2)); network.AddLayer(new BasicLayer(2)); network.AddLayer(new BasicLayer(1)); network.Structure.FinalizeStructure(); network.Reset(); ·       Neural networks frequently start with a random weight matrix. This provides a starting point for the training methods. These random values will be tested and refined into an acceptable solution. However, sometimes the initial random values are too far off. Sometimes it may be necessary to reset the weights again, if training is ineffective. These weights make up the long-term memory of the neural network. Additionally, some layers have threshold values that also contribute to the long-term memory of the neural network. Some neural networks also contain context layers, which give the neural network a short-term memory as well. The neural network learns by modifying these weight and threshold values. ·       Now that the neural network has been created, it must be trained. Training a Neural Network ·       construct a INeuralDataSet object - contains the input array and the expected output array (of corresponding range). Even though there is only one output value, we must still use a two-dimensional array to represent the output. public static double[][] XOR_INPUT ={ new double[2] { 0.0, 0.0 }, new double[2] { 1.0, 0.0 }, new double[2] { 0.0, 1.0 }, new double[2] { 1.0, 1.0 } };   public static double[][] XOR_IDEAL = { new double[1] { 0.0 }, new double[1] { 1.0 }, new double[1] { 1.0 }, new double[1] { 0.0 } };   INeuralDataSet trainingSet = new BasicNeuralDataSet(XOR_INPUT, XOR_IDEAL); ·       Training is the process where the neural network's weights are adjusted to better produce the expected output. Training will continue for many iterations, until the error rate of the network is below an acceptable level. Encog supports many different types of training. Resilient Propagation (RPROP) - general-purpose training algorithm. All training classes implement the ITrain interface. The RPROP algorithm is implemented by the ResilientPropagation class. Training the neural network involves calling the Iteration method on the ITrain class until the error is below a specific value. The code loops through as many iterations, or epochs, as it takes to get the error rate for the neural network to be below 1%. Once the neural network has been trained, it is ready for use. ITrain train = new ResilientPropagation(network, trainingSet);   for (int epoch=0; epoch < 10000; epoch++) { train.Iteration(); Debug.Print("Epoch #" + epoch + " Error:" + train.Error); if (train.Error > 0.01) break; } Executing a Neural Network ·       Call the Compute method on the BasicNetwork class. Console.WriteLine("Neural Network Results:"); foreach (INeuralDataPair pair in trainingSet) { INeuralData output = network.Compute(pair.Input); Console.WriteLine(pair.Input[0] + "," + pair.Input[1] + ", actual=" + output[0] + ",ideal=" + pair.Ideal[0]); } ·       The Compute method accepts an INeuralData class and also returns a INeuralData object. Neural Network Results: 0.0,0.0, actual=0.002782538818034049,ideal=0.0 1.0,0.0, actual=0.9903741937121177,ideal=1.0 0.0,1.0, actual=0.9836807956566187,ideal=1.0 1.0,1.0, actual=0.0011646072586172778,ideal=0.0 ·       the network has not been trained to give the exact results. This is normal. Because the network was trained to 1% error, each of the results will also be within generally 1% of the expected value.

  Read the article

• CodePlex Daily Summary for Saturday, June 05, 2010

  CodePlex Daily Summary for Saturday, June 05, 2010New Projects555 Calculator: A simple calculator to help choosing resistor and capacitor values to get the frequency you're looking for on a 555 timer.BleQua .NET: PL: Program sieciowy BleQua .NET jest multi-komunikatorem. EN: Network program BleQua .NET is multicomunicator.ChatDiplomaWork: ChatSample is the sample project.CSUFVGDC Summer Jam: Repository for VGDC summer game jam.Database Export Wizard: ExportWizard is a Step Wizard for Database Export using ASP.net and SQL Server. It allows easy export in any of the standard formats: CSV, TXT, HTM...Dozer Enterprise Library for .NET: a light .net framework for enterprise applications developmentEmployee Management System: This is an Employee Management System. the goal here is to offer a software that caters to small to mid sized businesses for free. This program a...Fanray: My project on Codeplex.Infragistics Analytics Framework: This project includes wrappers for the Infragistics controls that integrate with the recently launched Microsoft Silverlight Analytics Framework. ...KIME Simio Extensions: This is the official project of KIME Solutions. Here you find any current developments of KIME that are publicly available. KIME develops extension...MindTouch Community Extensions: MindTouch Community Extentions is a Native C# extention library for MindTouch Core that will have Dekiscript functions for full coverage of the API...NginxTray: NginxTray allows you manage easily Nginx Web Server by a Tray icon.NStore: NStore is a virtual store example done with ASP.NET MVC 2.0 tecnologyProjet Campus Numerique + Appli Mobile: Projet de création d'un campus numérique pour l'ISEN et d'un application mobile d'accès.SharePoint 2010 Feature Upgrade Kit: A set of tools for managing upgradable Features in SharePoint 2010. (Upgrading Features is a means of deploying code/artifact updates to existing S...SiteMap Utility for DNN Blog Module: This is a mini-project which allows you to easily add or generate an XML site map to your DotNetNuke® website for the search engines to use to inde...Space Explorer: A small app to help users examine folders to see which files and subfolders are taking up space. Still in development, no releases available curren...SQL Compact Toolbox: SQL Compact Toolbox is a Visual Studio 2010 add-in, that adds scripting, import, export, migrate, rename, run script and other upcoming SQL Server ...Twilverlight: Twliverlight is TweetDeck mixed with Silverlight. Much as I like using TweetDeck, it hogs my memory out, so this is an attempt to write a memory-ef...Visual Studio 2010 FxCop Extension: Visual Studio 2010 FxCop Extension allows to integrate stand-alone FxCop into Visual Studio 2010. You'll be able to analysis your source code with ...VisualStudio 2010 JavaScript Outlining: Visual Studio 2010 editor extension for JavaScript code blocks and custom regions outlining Wiki Shelf: Wiki Shelf is a Wikipedia browser app. The goal is to bring the library experience of browsing books, studying, and researching to the Wikipedia u...X-Arena - Magic: Projeto de PDS2 no Curso de Tecnologia em Desenvolvimento de Software no CEFETRN. O desenvolvimento do jogo Quiz Arena possui como pr...New Releases555 Calculator: 555Calc release v1.0: The initial 1 point uh-oh release of 555Calc.BleQua .NET: BleQua .NET 1.0.0.0: First releaseChatterbot JBot: JBot 1.0.1.155: Change presentation technology from Window Forms to Widndown Presentation Fundation.Community Forums NNTP bridge: Community Forums NNTP Bridge V26: Release of the Community Forums NNTP Bridge to access the social and anwsers MS forums with a single, open source NNTP bridge. This release has ad...Community Forums NNTP bridge: Community Forums NNTP Bridge V27: Release of the Community Forums NNTP Bridge to access the social and anwsers MS forums with a single, open source NNTP bridge. This release has ad...Community Forums NNTP bridge: Community Forums NNTP Bridge V28: Release of the Community Forums NNTP Bridge to access the social and anwsers MS forums with a single, open source NNTP bridge. This release has ad...CSS 360 Planetary Calendar: Final Release: =============================================================================== Final Release Version: 2.0 Tools Used: - Collaboration, Releas...Database Export Wizard: Version 3: As described in CodeProject article: Database Export Wizard for ASP.net and SQL Server. http://www.codeproject.com/KB/aspnet/DatabaseExportWizard.aspxEdu Math: Edu Math 2.0.10.122: Change version .NET Framework.Employee Management System: V1 (beta): This version is still in beta testing. Any issues, comments or suggestions are greatly appreciated. The export to excel function in this release o...ESB.NET: ESBDeploy_7.0.27.0 (x64 and x86) [ESB.NET 7.0 RC1]: Release Details Changes Since Last Release (since 6.3.47.1) - Targets .NET Framework 4, Visual Studio.NET 2010, Workflow 4 - Flowchart workflow ada...Free Silverlight & WPF Chart Control - Visifire: Visifire SL and WPF Charts v3.1.1 beta 2 Released: Hi, This release contains the following enhancements: *ShowIndicator() and HideIndicator() function has been implemented in Chart. So now user wi...Free Silverlight & WPF Chart Control - Visifire: Visifire SL and WPF Charts v3.5.4 beta 2 Released: Hi, This release contains the following enhancements: *ShowIndicator() and HideIndicator() function has been implemented in Chart. So now user wi...FsCheck: A random testing framework: FsCheck 0.7: What to download? If you use F# April 2010 CTP with Visual Studio 2008 or Visual Studio 2010, either Source or Binaries will do. To open source in...Git Source Control Provider: V 0.5: For VS 2010 users, it is recommanded to install it within Visual Studio by selecting Tools | Extension Manager. Run Visual Studio. Go to Tools ...GPdotNET - Genetic Programming Tool: GPdotNETv0.95: 1. Localization support 2. Export functionality for GP Model with training and testing data. 3. Export GPModel with Testing and Training data. 4....JoshDOS: JoshDOS 1.1: 1.1 adds a toutorial of how to create new commands and of course, you need the COMSOS user kit or dev kit. Ver 1.1 also includes a demo called Gue...KIME Simio Extensions: KIME.SimioDebugStep: This simple Simio step allows you to debug any number of expressions from within the simulation run. The debug information is displayed using a mes...NginxTray: NginxTray 0.3 Beta 2: NginxTray 0.3 Beta 2NginxTray: NginxTray 0.5 Beta 3: NginxTray 0.5 Beta 3NginxTray: NginxTray 0.6 RC1: NginxTray 0.6 RC1Open Source PLM Activities: Prodeos_OC beta 1.0: The “Innovator – MS Office Connector” is a product developed by Prodeos (www.prodeos.com). It is a light connector made to facilitate the use of Mi...Paint.NET PSD Plugin: 1.5.1: Changes in this release: Bitmap-mode images can now be loaded. Thanks to dhnc for filing the bug. Plugin no longer crashes on files with user m...SharePoint 2010 Feature Upgrade Kit: 1.0.0.0: This release contains:- - Custom application page to manage the upgrade of Site and Web-scoped Features. To come in the next release: - Companio...SiteMap Utility for DNN Blog Module: Blog SiteMap Utility v01.00.01: This is the first public release of the SiteMap Utility for the core DotNetNuke® Blog Module. Please see the documentation on this site on how to...SqlDiffFramework-A Visual Differencing Engine for Dissimilar Data Sources: SqlDiffFramework 1.0.2.0: Maintenance Release Defect Fixes: Issue # 3: 3 Issue # 4: 4 Enhancements: About Box now displays regional and language settings in effect. SDF...SuperSocket: First release of SuperSocket: !First release of SuperSocketThe Fastcopy Helper: FastcopyHelper: Fastcopy Helper 2.0 This is a final one. You can use it on the way. In order to use it , you should have the .NET3.5 ! 此软件必须下载 .NET3.5平台，方可使用！TV Show Renamer: TV Show Renamer Beta 3: I found the bug the prevented it from closing correctly so I fixed it and had to release it right away. If anyone else finds any problems. contact me.UrzaGatherer: UrzaGatherer v2.0: UrzaGatherer is the first stable version. This release include UrzaBackgroundPictures.VisualStudio 2010 JavaScript Outlining: VisualStudion 2010 Javascript Outlining 1.0: Features Outlines JavaScript codeblock regions for the code placed between { }. Both places on a new line. Outlines custom regions defined by: //...Wouter's SharePoint Demo Land: Navigation Service with Proxy: A SharePoint 2010 Service Application that uses service proxies to relay commands to the actual service. The demo proxy makes use of in-memory comm...盘古分词-开源中文分词组件: V2.0.0.0: 进一步优化性能，分词速度达到将近 500K ，1.2.0.1 版本只有 320K 修改 PanGu.Lucene.Analyzer, 支持 Lucene.net 2.9 版本。 增加对字典中以数字开头的专业非中文词汇的识别 增加英文分词开关，权重由英文小写权重和英文词根权重两个参数来决定...Most Popular ProjectsCommunity Forums NNTP bridgeOutSyncASP.NET MVC Time PlannerNeatUploadMoonyDesk (windows desktop widgets)AgUnit - Silverlight unit testing with ReSharperViperWorks IgnitionASP.NET MVC ExtensionsAviva Solutions C# Coding GuidelinesMute4Most Active ProjectsCommunity Forums NNTP bridgeRawrpatterns & practices – Enterprise LibraryIonics Isapi Rewrite FilterGMap.NET - Great Maps for Windows Forms & PresentationN2 CMSStyleCopFarseer Physics Enginesmark C# LibraryMirror Testing System

  Read the article

< Previous Page | 240 241 242 243 244 245 246 247 248 249 250 251  | Next Page >