Search Results

Search found 21702 results on 869 pages for 'large objects'.

Page 45/869 | < Previous Page | 41 42 43 44 45 46 47 48 49 50 51 52  | Next Page >

  • Cannot upload files bigger than 8GB to Amazon S3 by multi-part upload due to broken pipe

    - by spencerho
    I implemented S3 multi-part upload, both high level and low level version, based on the sample code from http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?HLuploadFileJava.html and http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?llJavaUploadFile.html When I uploaded files of size less than 4 GB, the upload processes completed without any problem. When I uploaded a file of size 13 GB, the code started to show IO exception, broken pipes. After retries, it still failed. Here is the way to repeat the scenario. Take 1.1.7.1 release, create a new bucket in US standard region create a large EC2 instance as the client to upload file create a file of 13GB in size on the EC2 instance. run the sample code on either one of the high-level or low-level API S3 documentation pages from the EC2 instance test either one of the three part size: default part size (5 MB) or set the part size to 100,000,000 or 200,000,000 bytes. So far the problem shows up consistently. I attached here a tcpdump file for you to compare. In there, the host on the S3 side kept resetting the socket.

    Read the article

  • android Emulator always stop at "waiting for Home..."

    - by wuwupp
    hi,there, I freshed install Eclipse, jdk, android sdk 1.5 in winxp. but when I run the "hello world" app, the emulator always stop at "andorid" loading message. In eclipse console, it shows "waiting for HOME..." and in DDMS LogCat, it shows following msg: there are some error and warning. So, what's wrong with my case? I have googled lots of results, but no one can help me. Please help me. Many thx 06-13 00:07:54.323: INFO/DEBUG(551): debuggerd: Jun 30 2009 17:00:51 06-13 00:07:54.383: INFO/vold(550): Android Volume Daemon version 2.0 06-13 00:07:54.724: ERROR/flash_image(556): can't find recovery partition 06-13 00:07:55.223: DEBUG/qemud(558): entering main loop 06-13 00:07:55.323: DEBUG/qemud(558): multiplexer_handle_control: unknown control message (18 bytes): 'ko:unknown command' 06-13 00:07:55.493: INFO/vold(550): New MMC card 'SU02G' (serial 1012966) added @ /devices/platform/goldfish_mmc.0/mmc_host/mmc0/mmc0:e118 06-13 00:07:55.773: INFO/vold(550): Disk (blkdev 179:0), 262144 secs (128 MB) 0 partitions 06-13 00:07:55.773: INFO/vold(550): New blkdev 179.0 on media SU02G, media path /devices/platform/goldfish_mmc.0/mmc_host/mmc0/mmc0:e118, Dpp 0 06-13 00:07:55.814: INFO/vold(550): Evaluating dev '/devices/platform/goldfish_mmc.0/mmc_host/mmc0/mmc0:e118/block/mmcblk0' for mountable filesystems for '/sdcard' 06-13 00:07:56.014: ERROR/vold(550): Error opening switch name path '/sys/class/switch/test2' (No such file or directory) 06-13 00:07:56.014: ERROR/vold(550): Error bootstrapping switch '/sys/class/switch/test2' (m) 06-13 00:07:56.073: ERROR/vold(550): Error opening switch name path '/sys/class/switch/test' (No such file or directory) 06-13 00:07:56.073: ERROR/vold(550): Error bootstrapping switch '/sys/class/switch/test' (m) 06-13 00:07:56.073: DEBUG/vold(550): Bootstrapping complete 06-13 00:07:56.743: INFO//system/bin/dosfsck(550): dosfsck 3.0.1 (23 Nov 2008) 06-13 00:07:56.753: INFO//system/bin/dosfsck(550): dosfsck 3.0.1, 23 Nov 2008, FAT32, LFN 06-13 00:07:56.783: INFO//system/bin/dosfsck(550): Checking we can access the last sector of the filesystem 06-13 00:07:56.893: INFO//system/bin/dosfsck(550): Boot sector contents: 06-13 00:07:56.924: INFO//system/bin/dosfsck(550): System ID "MSWIN4.1" 06-13 00:07:56.934: INFO//system/bin/dosfsck(550): Media byte 0xf8 (hard disk) 06-13 00:07:56.953: INFO//system/bin/dosfsck(550): 512 bytes per logical sector 06-13 00:07:56.974: INFO//system/bin/dosfsck(550): 512 bytes per cluster 06-13 00:07:57.005: INFO//system/bin/dosfsck(550): 32 reserved sectors 06-13 00:07:57.013: INFO//system/bin/dosfsck(550): First FAT starts at byte 16384 (sector 32) 06-13 00:07:57.013: INFO//system/bin/dosfsck(550): 2 FATs, 32 bit entries 06-13 00:07:57.023: INFO//system/bin/dosfsck(550): 1040384 bytes per FAT (= 2032 sectors) 06-13 00:07:57.043: INFO//system/bin/dosfsck(550): Root directory start at cluster 2 (arbitrary size) 06-13 00:07:57.043: INFO//system/bin/dosfsck(550): Data area starts at byte 2097152 (sector 4096) 06-13 00:07:57.043: INFO//system/bin/dosfsck(550): 258048 data clusters (132120576 bytes) 06-13 00:07:57.103: INFO//system/bin/dosfsck(550): 9 sectors/track, 2 heads 06-13 00:07:57.103: INFO//system/bin/dosfsck(550): 0 hidden sectors 06-13 00:07:57.123: INFO//system/bin/dosfsck(550): 262144 sectors total 06-13 00:07:57.313: DEBUG/qemud(558): fdhandler_accept_event: accepting on fd 10 06-13 00:07:57.313: DEBUG/qemud(558): created client 0xe078 listening on fd 8 06-13 00:07:57.313: DEBUG/qemud(558): fdhandler_event: disconnect on fd 8 06-13 00:07:57.623: DEBUG/qemud(558): fdhandler_accept_event: accepting on fd 10 06-13 00:07:57.623: DEBUG/qemud(558): created client 0xf028 listening on fd 8 06-13 00:07:57.643: DEBUG/qemud(558): client_fd_receive: attempting registration for service 'gsm' 06-13 00:07:57.763: DEBUG/qemud(558): client_fd_receive: - received channel id 1 06-13 00:08:12.553: INFO//system/bin/dosfsck(550): Checking for unused clusters. 06-13 00:08:13.483: INFO//system/bin/dosfsck(550): Checking free cluster summary. 06-13 00:08:13.643: DEBUG/AndroidRuntime(553): AndroidRuntime START <<<<<<<<<<<<<< 06-13 00:08:13.705: DEBUG/AndroidRuntime(553): CheckJNI is ON 06-13 00:08:13.793: INFO//system/bin/dosfsck(550): /dev/block//vold/179:0: 0 files, 1/258048 clusters 06-13 00:08:14.063: INFO/logwrapper(550): /system/bin/dosfsck terminated by exit(0) 06-13 00:08:14.143: DEBUG/vold(550): Filesystem check completed OK 06-13 00:08:14.683: INFO/vold(550): Sucessfully mounted vfat filesystem 179:0 on /sdcard (safe-mode on) 06-13 00:08:17.023: INFO/(554): ServiceManager: 0xac38 06-13 00:08:17.883: INFO/AudioFlinger(554): AudioFlinger's thread ready to run for output 0 06-13 00:08:18.163: INFO/CameraService(554): CameraService started: pid=554 06-13 00:08:21.824: DEBUG/AndroidRuntime(553): --- registering native functions --- 06-13 00:08:27.813: INFO/Zygote(553): Preloading classes... 06-13 00:08:27.994: DEBUG/dalvikvm(553): GC freed 764 objects / 42216 bytes in 88ms 06-13 00:08:30.234: DEBUG/dalvikvm(553): GC freed 278 objects / 17160 bytes in 48ms 06-13 00:08:33.094: DEBUG/dalvikvm(553): GC freed 208 objects / 12696 bytes in 44ms 06-13 00:08:34.343: DEBUG/dalvikvm(553): Trying to load lib /system/lib/libmedia_jni.so 0x0 06-13 00:08:35.803: DEBUG/dalvikvm(553): Added shared lib /system/lib/libmedia_jni.so 0x0 06-13 00:08:35.903: DEBUG/dalvikvm(553): Trying to load lib /system/lib/libmedia_jni.so 0x0 06-13 00:08:35.903: DEBUG/dalvikvm(553): Shared lib '/system/lib/libmedia_jni.so' already loaded in same CL 0x0 06-13 00:08:36.003: DEBUG/dalvikvm(553): Trying to load lib /system/lib/libmedia_jni.so 0x0 06-13 00:08:36.003: DEBUG/dalvikvm(553): Shared lib '/system/lib/libmedia_jni.so' already loaded in same CL 0x0 06-13 00:08:36.215: DEBUG/dalvikvm(553): Trying to load lib /system/lib/libmedia_jni.so 0x0 06-13 00:08:36.244: DEBUG/dalvikvm(553): Shared lib '/system/lib/libmedia_jni.so' already loaded in same CL 0x0 06-13 00:08:36.455: DEBUG/dalvikvm(553): GC freed 462 objects / 29144 bytes in 70ms 06-13 00:08:44.123: DEBUG/dalvikvm(553): GC freed 3584 objects / 171648 bytes in 125ms 06-13 00:09:10.473: DEBUG/dalvikvm(553): GC freed 11329 objects / 400856 bytes in 196ms 06-13 00:09:17.373: DEBUG/dalvikvm(553): GC freed 10472 objects / 438272 bytes in 199ms 06-13 00:09:24.563: DEBUG/dalvikvm(553): GC freed 10975 objects / 459800 bytes in 202ms 06-13 00:09:46.403: DEBUG/dalvikvm(553): GC freed 14372 objects / 506896 bytes in 252ms 06-13 00:09:53.793: DEBUG/dalvikvm(553): GC freed 11314 objects / 481360 bytes in 215ms 06-13 00:09:57.743: DEBUG/dalvikvm(553): GC freed 5928 objects / 248640 bytes in 195ms 06-13 00:10:01.324: DEBUG/dalvikvm(553): GC freed 349 objects / 37032 bytes in 190ms 06-13 00:10:05.253: DEBUG/dalvikvm(553): GC freed 778 objects / 48376 bytes in 217ms 06-13 00:10:06.564: DEBUG/dalvikvm(553): GC freed 321 objects / 37288 bytes in 219ms 06-13 00:10:08.194: DEBUG/dalvikvm(553): GC freed 477 objects / 29584 bytes in 212ms 06-13 00:10:08.663: DEBUG/dalvikvm(553): Trying to load lib /system/lib/libwebcore.so 0x0 06-13 00:10:09.743: DEBUG/dalvikvm(553): Added shared lib /system/lib/libwebcore.so 0x0 06-13 00:10:11.634: DEBUG/dalvikvm(553): GC freed 441 objects / 26224 bytes in 236ms 06-13 00:10:12.893: DEBUG/dalvikvm(553): GC freed 506 objects / 41464 bytes in 235ms 06-13 00:10:14.153: DEBUG/dalvikvm(553): GC freed 537 objects / 38832 bytes in 239ms 06-13 00:10:15.883: DEBUG/dalvikvm(553): GC freed 342 objects / 22552 bytes in 248ms 06-13 00:10:17.124: DEBUG/dalvikvm(553): GC freed 338 objects / 18736 bytes in 264ms 06-13 00:10:18.523: DEBUG/dalvikvm(553): GC freed 629 objects / 32136 bytes in 260ms 06-13 00:10:38.933: DEBUG/dalvikvm(553): GC freed 14257 objects / 497280 bytes in 368ms 06-13 00:10:46.453: DEBUG/dalvikvm(553): GC freed 11164 objects / 469576 bytes in 360ms 06-13 00:10:52.973: DEBUG/dalvikvm(553): GC freed 7134 objects / 311432 bytes in 339ms 06-13 00:10:55.595: DEBUG/dalvikvm(553): GC freed 752 objects / 43224 bytes in 520ms 06-13 00:10:56.863: DEBUG/dalvikvm(553): GC freed 598 objects / 31496 bytes in 307ms 06-13 00:10:58.543: DEBUG/dalvikvm(553): GC freed 413 objects / 26336 bytes in 355ms 06-13 00:10:59.263: INFO/Zygote(553): ...preloaded 1166 classes in 151403ms. 06-13 00:10:59.683: DEBUG/dalvikvm(553): GC freed 313 objects / 19952 bytes in 343ms 06-13 00:10:59.793: INFO/Zygote(553): Preloading resources... 06-13 00:11:00.683: DEBUG/dalvikvm(553): GC freed 54 objects / 11248 bytes in 340ms 06-13 00:11:05.723: DEBUG/dalvikvm(553): GC freed 337 objects / 15008 bytes in 317ms 06-13 00:11:08.703: DEBUG/dalvikvm(553): GC freed 280 objects / 11768 bytes in 312ms 06-13 00:11:09.303: INFO/Zygote(553): ...preloaded 48 resources in 9513ms. 06-13 00:11:09.795: INFO/Zygote(553): ...preloaded 15 resources in 454ms. 06-13 00:11:10.303: DEBUG/dalvikvm(553): GC freed 118 objects / 8616 bytes in 420ms 06-13 00:11:10.913: DEBUG/dalvikvm(553): GC freed 205 objects / 8104 bytes in 308ms 06-13 00:11:11.344: DEBUG/dalvikvm(553): GC freed 36 objects / 1400 bytes in 320ms 06-13 00:11:11.543: INFO/dalvikvm(553): Splitting out new zygote heap 06-13 00:11:12.973: INFO/dalvikvm(553): System server process 585 has been created 06-13 00:11:13.336: INFO/Zygote(553): Accepting command socket connections 06-13 00:11:14.963: INFO/jdwp(585): received file descriptor 10 from ADB 06-13 00:11:16.843: WARN/System.err(585): Can't dispatch DDM chunk 46454154: no handler defined 06-13 00:11:16.953: WARN/System.err(585): Can't dispatch DDM chunk 4d505251: no handler defined 06-13 00:11:17.763: DEBUG/dalvikvm(585): Trying to load lib /system/lib/libandroid_servers.so 0x0 06-13 00:11:19.714: DEBUG/dalvikvm(585): Added shared lib /system/lib/libandroid_servers.so 0x0 06-13 00:11:20.123: INFO/sysproc(585): Entered system_init() 06-13 00:11:20.223: INFO/sysproc(585): ServiceManager: 0x1017b8 06-13 00:11:20.359: INFO/SurfaceFlinger(585): SurfaceFlinger is starting 06-13 00:11:20.493: INFO/SurfaceFlinger(585): SurfaceFlinger's main thread ready to run. Initializing graphics H/W... 06-13 00:11:20.634: ERROR/MemoryHeapBase(585): error opening /dev/pmem: No such file or directory 06-13 00:11:20.704: ERROR/SurfaceFlinger(585): Couldn't open /sys/power/wait_for_fb_sleep or /sys/power/wait_for_fb_wake 06-13 00:11:22.013: ERROR/GLLogger(585): couldn't load library (Cannot find library) 06-13 00:11:22.103: INFO/SurfaceFlinger(585): EGL informations: 06-13 00:11:22.113: INFO/SurfaceFlinger(585): # of configs : 6 06-13 00:11:22.123: INFO/SurfaceFlinger(585): vendor : Android 06-13 00:11:22.123: INFO/SurfaceFlinger(585): version : 1.31 Android META-EGL 06-13 00:11:22.134: INFO/SurfaceFlinger(585): extensions: 06-13 00:11:22.134: INFO/SurfaceFlinger(585): Client API: OpenGL ES 06-13 00:11:22.193: INFO/EGLDisplaySurface(585): using (fd=22) 06-13 00:11:22.193: INFO/EGLDisplaySurface(585): id = 06-13 00:11:22.193: INFO/EGLDisplaySurface(585): xres = 320 px 06-13 00:11:22.193: INFO/EGLDisplaySurface(585): yres = 480 px 06-13 00:11:22.193: INFO/EGLDisplaySurface(585): xres_virtual = 320 px 06-13 00:11:22.193: INFO/EGLDisplaySurface(585): yres_virtual = 960 px 06-13 00:11:22.193: INFO/EGLDisplaySurface(585): bpp = 16 06-13 00:11:22.193: INFO/EGLDisplaySurface(585): r = 11:5 06-13 00:11:22.193: INFO/EGLDisplaySurface(585): g = 5:6 06-13 00:11:22.193: INFO/EGLDisplaySurface(585): b = 0:5 06-13 00:11:22.193: INFO/EGLDisplaySurface(585): width = 49 mm (165.877548 dpi) 06-13 00:11:22.193: INFO/EGLDisplaySurface(585): height = 74 mm (164.756760 dpi) 06-13 00:11:22.193: INFO/EGLDisplaySurface(585): refresh rate = 60.00 Hz 06-13 00:11:22.533: WARN/HAL(585): load: module=/system/lib/hw/copybit.goldfish.so error=Cannot find library 06-13 00:11:22.543: WARN/HAL(585): load: module=/system/lib/hw/copybit.default.so error=Cannot find library 06-13 00:11:22.553: WARN/SurfaceFlinger(585): ro.sf.lcd_density not defined, using 160 dpi by default. 06-13 00:11:22.644: INFO/SurfaceFlinger(585): OpenGL informations: 06-13 00:11:22.654: INFO/SurfaceFlinger(585): vendor : Android 06-13 00:11:22.654: INFO/SurfaceFlinger(585): renderer : Android PixelFlinger 1.0 06-13 00:11:22.654: INFO/SurfaceFlinger(585): version : OpenGL ES-CM 1.0 06-13 00:11:22.654: INFO/SurfaceFlinger(585): extensions: GL_OES_byte_coordinates GL_OES_fixed_point GL_OES_single_precision GL_OES_read_format GL_OES_compressed_paletted_texture GL_OES_draw_texture GL_OES_matrix_get GL_OES_query_matrix GL_ARB_texture_compression GL_ARB_texture_non_power_of_two GL_ANDROID_direct_texture GL_ANDROID_user_clip_plane GL_ANDROID_vertex_buffer_object GL_ANDROID_generate_mipmap 06-13 00:11:22.673: WARN/HAL(585): load: module=/system/lib/hw/copybit.goldfish.so error=Cannot find library 06-13 00:11:22.683: WARN/HAL(585): load: module=/system/lib/hw/copybit.default.so error=Cannot find library 06-13 00:11:22.703: WARN/HAL(585): load: module=/system/lib/hw/overlay.goldfish.so error=Cannot find library 06-13 00:11:22.713: WARN/HAL(585): load: module=/system/lib/hw/overlay.default.so error=Cannot find library 06-13 00:11:23.663: INFO/sysproc(585): System server: starting Android runtime. 06-13 00:11:23.733: INFO/sysproc(585): System server: starting Android services. 06-13 00:11:23.953: INFO/SystemServer(585): Entered the Android system server! 06-13 00:11:24.303: INFO/sysproc(585): System server: entering thread pool. 06-13 00:11:24.763: ERROR/GLLogger(585): couldn't load library (Cannot find library) 06-13 00:11:25.893: INFO/ARMAssembler(585): generated scanline__00000077:03545404_00000A01_00000000 [ 30 ipp] (51 ins) at [0x18f708:0x18f7d4] in 72796961 ns 06-13 00:11:26.193: INFO/SystemServer(585): Starting Power Manager. 06-13 00:11:26.953: INFO/SystemServer(585): Starting Activity Manager. 06-13 00:11:31.733: INFO/SystemServer(585): Starting telephony registry 06-13 00:11:32.054: INFO/SystemServer(585): Starting Package Manager. 06-13 00:11:32.553: INFO/Installer(585): connecting... 06-13 00:11:32.914: INFO/installd(555): new connection 06-13 00:11:35.193: INFO/PackageManager(585): Got library android.awt in /system/framework/android.awt.jar 06-13 00:11:35.313: INFO/PackageManager(585): Got library android.test.runner in /system/framework/android.test.runner.jar 06-13 00:11:35.324: INFO/PackageManager(585): Got library com.android.im.plugin in /system/framework/com.android.im.plugin.jar 06-13 00:11:44.643: DEBUG/PackageManager(585): Scanning app dir /system/framework 06-13 00:11:49.513: DEBUG/PackageManager(585): Scanning app dir /system/app 06-13 00:11:51.493: DEBUG/dalvikvm(585): GC freed 6088 objects / 251280 bytes in 1237ms 06-13 00:12:27.497: DEBUG/dalvikvm(585): GC freed 3435 objects / 216088 bytes in 792ms 06-13 00:12:29.213: DEBUG/PackageManager(585): Scanning app dir /data/app 06-13 00:12:30.223: DEBUG/PackageManager(585): Scanning app dir /data/app-private 06-13 00:12:30.425: INFO/PackageManager(585): Time to scan packages: 47.319 seconds 06-13 00:12:30.703: WARN/PackageManager(585): Unknown permission com.google.android.googleapps.permission.GOOGLE_AUTH in package com.android.providers.contacts 06-13 00:12:30.803: WARN/PackageManager(585): Unknown permission com.google.android.googleapps.permission.GOOGLE_AUTH.cp in package com.android.providers.contacts 06-13 00:12:30.853: WARN/PackageManager(585): Unknown permission com.google.android.googleapps.permission.GOOGLE_AUTH in package com.android.development 06-13 00:12:30.913: WARN/PackageManager(585): Unknown permission com.google.android.googleapps.permission.GOOGLE_AUTH.ALL_SERVICES in package com.android.development 06-13 00:12:31.133: WARN/PackageManager(585): Unknown permission com.google.android.googleapps.permission.GOOGLE_AUTH.YouTubeUser in package com.android.development 06-13 00:12:31.143: WARN/PackageManager(585): Unknown permission com.google.android.googleapps.permission.ACCESS_GOOGLE_PASSWORD in package com.android.development 06-13 00:12:31.234: WARN/PackageManager(585): Unknown permission com.google.android.providers.gmail.permission.WRITE_GMAIL in package com.android.settings 06-13 00:12:31.254: WARN/PackageManager(585): Unknown permission com.google.android.providers.gmail.permission.READ_GMAIL in package com.android.settings 06-13 00:12:31.303: WARN/PackageManager(585): Unknown permission com.google.android.googleapps.permission.GOOGLE_AUTH in package com.android.settings 06-13 00:12:31.683: WARN/PackageManager(585): Unknown permission com.google.android.googleapps.permission.GOOGLE_AUTH in package com.android.browser 06-13 00:12:31.803: WARN/PackageManager(585): Unknown permission com.google.android.googleapps.permission.GOOGLE_AUTH.mail in package com.android.contacts 06-13 00:12:34.603: DEBUG/dalvikvm(585): GC freed 2851 objects / 161304 bytes in 845ms 06-13 00:12:35.403: INFO/SystemServer(585): Starting Content Manager. 06-13 00:12:39.954: WARN/ActivityManager(585): Unable to start service Intent { action=android.accounts.IAccountsService comp={com.google.android.googleapps/com.google.android.googleapps.GoogleLoginService} }: not found 06-13 00:12:40.063: WARN/AccountMonitor(585): Couldn't connect to Intent { action=android.accounts.IAccountsService comp={com.google.android.googleapps/com.google.android.googleapps.GoogleLoginService} } (Missing service?) 06-13 00:12:40.253: INFO/SystemServer(585): Starting System Content Providers. 06-13 00:12:40.553: INFO/ActivityThread(585): Publishing provider settings: com.android.providers.settings.SettingsProvider 06-13 00:12:41.433: INFO/ActivityThread(585): Publishing provider sync: android.content.SyncProvider 06-13 00:12:41.683: INFO/SystemServer(585): Starting Battery Service. 06-13 00:12:42.293: ERROR/BatteryService(585): Could not open '/sys/class/power_supply/usb/online' 06-13 00:12:42.433: ERROR/BatteryService(585): Could not open '/sys/class/power_supply/battery/batt_vol' 06-13 00:12:42.543: ERROR/BatteryService(585): Could not open '/sys/class/power_supply/battery/batt_temp' 06-13 00:12:42.933: INFO/SystemServer(585): Starting Hardware Service. 06-13 00:12:43.398: DEBUG/qemud(558): fdhandler_accept_event: accepting on fd 10 06-13 00:12:43.623: DEBUG/qemud(558): created client 0x10fd8 listening on fd 11 06-13 00:12:43.743: DEBUG/qemud(558): client_fd_receive: attempting registration for service 'hw-control' 06-13 00:12:43.873: DEBUG/qemud(558): client_fd_receive: - received channel id 2 06-13 00:15:20.695: WARN/SurfaceFlinger(585): executeScheduledBroadcasts() skipped, contention on the client. We'll try again later...

    Read the article

  • Practical size limitations for RDBMS

    - by grenade
    I am working on a project that must store very large datasets and associated reference data. I have never come across a project that required tables quite this large. I have proved that at least one development environment cannot cope at the database tier with the processing required by the complex queries against views that the application layer generates (views with multiple inner and outer joins, grouping, summing and averaging against tables with 90 million rows). The RDBMS that I have tested against is DB2 on AIX. The dev environment that failed was loaded with 1/20th of the volume that will be processed in production. I am assured that the production hardware is superior to the dev and staging hardware but I just don't believe that it will cope with the sheer volume of data and complexity of queries. Before the dev environment failed, it was taking in excess of 5 minutes to return a small dataset (several hundred rows) that was produced by a complex query (many joins, lots of grouping, summing and averaging) against the large tables. My gut feeling is that the db architecture must change so that the aggregations currently provided by the views are performed as part of an off-peak batch process. Now for my question. I am assured by people who claim to have experience of this sort of thing (which I do not) that my fears are unfounded. Are they? Can a modern RDBMS (SQL Server 2008, Oracle, DB2) cope with the volume and complexity I have described (given an appropriate amount of hardware) or are we in the realm of technologies like Google's BigTable? I'm hoping for answers from folks who have actually had to work with this sort of volume at a non-theoretical level.

    Read the article

  • How to Avoid PHP Object Nesting/Creation Limit?

    - by Will Shaver
    I've got a handmade ORM in PHP that seems to be bumping up against an object limit and causing php to crash. Here's a simple script that will cause crashes: <? class Bob { protected $parent; public function Bob($parent) { $this->parent = $parent; } public function __toString() { if($this->parent) return (string) "x " . $this->parent; return "top"; } } $bobs = array(); for($i = 1; $i < 40000; $i++) { $bobs[] = new Bob($bobs[$1 -1]); } ?> Even running this from the command line will cause issues. Some boxes take more than 40,000 objects. I've tried it on Linux/Appache (fail) but my app runs on IIS/FastCGI. On FastCGI this causes the famous "The FastCGI process exited unexpectedly" error. Obviously 20k objects is a bit high, but it crashes with far fewer objects if they have data and nested complexity. Fast CGI isn't the issue - I've tried running it from the command line. I've tried setting the memory to something really high - 6,000MB and to something really low - 24MB. If I set it low enough I'll get the "allocated memory size xxx bytes exhausted" error. I'm thinking that it has to do with the number of functions that are called - some kind of nesting prevention. I didn't think that my ORM's nesting was that complicated but perhaps it is. I've got some pretty clear cases where if I load just ONE more object it dies, but loads in under 3 seconds if it works.

    Read the article

  • How to parse text as JavaScript?

    - by Danjah
    This question of mine (currently unanswered), drove me toward finding a better solution to what I'm attempting. My requirements: chunks of code which can be arbitrarily added into a document, without an identifier: [div class="thing"] [elements... /] [/div] the objects are scanned for and found by an external script: var things = yd.getElementsBy(function(el){ return yd.hasClass('thing'); },null,document ); the objects must be individually configurable, what I have currently is identifier-based: [div class="thing" id="thing0"] [elements... /] [script type="text/javascript"] new Thing().init({ id:'thing0'; }); [/script] [/div] So I need to ditch the identifier (id="thing0") so there are no duplicates when more than one chunk of the same code is added to a page I still need to be able to config these objects individually, without an identifier SO! All of that said, I wondered about creating a dynamic global variable within the script block of each added chunk of code, within its script tag. As each 'thing' is found, I figure it would be legit to grab the innerHTML of the script tag and somehow convert that text into a useable JS object. Discuss. Ok, don't discuss if you like, but if you get the drift then feel free to correct my wayward thinking or provide a better solution - please! d

    Read the article

  • Should a Trim method generally in the Data Access Layer or with in the Domain Layer?

    - by jpierson
    I'm dealing with a database that contains data with inconsistencies such as white leading and trailing white space. In general I see a lot of developers practice defensive coding by trimming almost all strings that come from the database that may have been entered by a user at some point. In my oppinoin it is better to do such formating before data is persisted so that it is done only once and then the data can be in a consistent and reliable state. Unfortunatley this is not the case however which leads me to the next best solution, using a Trim method. If I trim all data as part of my data access layer then I don't have to concern myself with defensive trimming within the business objects of my domain layer. If I instead put the trimming responsibility in my business objects, such as with set accessors of my C# properties, I should get the same net results however the trim will be operating on all values assigned to my business objects properties not just the ones that come from the inconsistent database. I guess as a somewhat philisophical question that may determine the answer I could ask "Should the domain later be responsible for defensive/coercive formatting of data?" Would it make sense to have a set accessor for a PhoneNumber property on a business object accept a unformatted or formatted string and then attempt to format it as required or should this responsibility be pushed to the presentation and data access layers leaving the domain layer more strict in the type of data that it will accept? I think this may be the more fundamental question. Update: Below are a few links that I thought I should share about the topic of data cleansing. Information service patterns, Part 3: Data cleansing pattern LINQ to SQL - Format a string before saving? How to trim values using Linq to Sql?

    Read the article

  • gzip: stdout: File too large when running customized backup script

    - by Roland
    I've create a plain and siple backup script that only backs up certain files and folders. tar -zcf $DIRECTORY/var.www.tar.gz /var/www tar -zcf $DIRECTORY/development.tar.gz /development tar -zcf $DIRECTORY/home.tar.gz /home Now this script runs for about 30mins then gives me the following error gzip: stdout: File too large Any other solutions that I can use to backup my files using shell scripting or a way to solve this error? I'm grateful for any help.

    Read the article

  • Saving State of Objects in GXT

    - by 8EM
    Is there a way to store the state of objects in GXT? That is, having a dynamically configurable GUI built in GXT, you can add your own widgets 'on-the-fly' in any order you like - with your own custom everything. Is there a way to save the state of all the objects, so one can load the profile back at a later date?

    Read the article

  • How can I insert or remove bytes from the middle of a large file in .NET

    - by Eric
    Is it possible to efficiently insert or remove bytes from the middle of a large file, and if so how? Or am I stuck rewriting the entire file after the point where the data was inserted or removed? [A lot of Bytes][Unwanted Bytes][A lot of Bytes] - > [A lot of Bytes][A lot of Bytes] or [A lot of Bytes][A lot of Bytes] - > [A loto f Bytes][New Inserted Bytes][A lot of Bytes]

    Read the article

  • Uploading Large Files

    - by mohamed hadad
    Hello, I'm using a windows service on my server to receive large files (1 GB) from desktop clients when I use the TCPClient class it creates astream to send the file which bloks my memory, What is the well designe for this problem

    Read the article

  • Django custom managers - how do I return only objects created by the logged-in user?

    - by Tom Tom
    I want to overwrite the custom objects model manager to only return objects a specific user created. Admin users should still return all objects using the objects model manager. Now I have found an approach that could work. They propose to create your own middleware looking like this: #### myproject/middleware/threadlocals.py try: from threading import local except ImportError: # Python 2.3 compatibility from django.utils._threading_local import local _thread_locals = local() def get_current_user(): return getattr(_thread_locals, 'user', None) class ThreadLocals(object): """Middleware that gets various objects from the request object and saves them in thread local storage.""" def process_request(self, request): _thread_locals.user = getattr(request, 'user', None) #### end And in the Custom manager you could call the get_current_user() method to return only objects a specific user created. class UserContactManager(models.Manager): def get_query_set(self): return super(UserContactManager, self).get_query_set().filter(creator=get_current_user()) Is this a good approach to this use-case? Will this work? Or is this like "using a sledgehammer to crack a nut" ? ;-) Just using: Contact.objects.filter(created_by= user) in each view doesn`t look very neat to me. EDIT Do not use this middleware approach !!! use the approach stated by Jack M. below After a while of testing this approach behaved pretty strange and with this approach you mix up a global-state with a current request. Use the approach presented below. It is really easy and no need to hack around with the middleware. create a custom manager in your model with a function that expects the current user or any other user as an input. #in your models.py class HourRecordManager(models.Manager): def for_user(self, user): return self.get_query_set().filter(created_by=user) class HourRecord(models.Model): #Managers objects = HourRecordManager() #in vour view you can call the manager like this and get returned only the objects from the currently logged-in user. hr_set = HourRecord.objects.for_user(request.user)

    Read the article

  • Django - testing using large tables of static data

    - by Michael B
    I am using "manage.py test" along with a JSON fixture I created using using 'dumpdata' My problem is that several of the tables in the fixture are very large (for example one containing the names of all cities in the US) which makes running a test incredibly slow. Seeing as several of these tables are never modified by the program (eg - the city names will never need to be modified), it doesn't make much sense to create and tear down these tables for every test run. Is there a better way to be testing this code using this kind of data?

    Read the article

  • How to use NSPredicate to catch child objects?

    - by Konstantin
    I'm new to core data and try to get all children objects of various types with one query. Say there's an "Animal" type as parent and "Cat", "Dog" and "Bird" as children. I'd like to get both cats and dogs, but not Birds in single query returned as Animal objects. Is it possible?

    Read the article

  • Mercurial hook to disallow committing large binary files

    - by hekevintran
    I want to have a Mercurial hook that will run before committing a transaction that will abort the transaction if a binary file being committed is greater than 1 megabyte. I found the following code which works fine except for one problem. If my changeset involves removing a file, this hook will throw an exception. The hook (I'm using pretxncommit = python:checksize.newbinsize): from mercurial import context, util from mercurial.i18n import _ import mercurial.node as dpynode '''hooks to forbid adding binary file over a given size Ensure the PYTHONPATH is pointing where hg_checksize.py is and setup your repo .hg/hgrc like this: [hooks] pretxncommit = python:checksize.newbinsize pretxnchangegroup = python:checksize.newbinsize preoutgoing = python:checksize.nopull [limits] maxnewbinsize = 10240 ''' def newbinsize(ui, repo, node=None, **kwargs): '''forbid to add binary files over a given size''' forbid = False # default limit is 10 MB limit = int(ui.config('limits', 'maxnewbinsize', 10000000)) tip = context.changectx(repo, 'tip').rev() ctx = context.changectx(repo, node) for rev in range(ctx.rev(), tip+1): ctx = context.changectx(repo, rev) print ctx.files() for f in ctx.files(): fctx = ctx.filectx(f) filecontent = fctx.data() # check only for new files if not fctx.parents(): if len(filecontent) > limit and util.binary(filecontent): msg = 'new binary file %s of %s is too large: %ld > %ld\n' hname = dpynode.short(ctx.node()) ui.write(_(msg) % (f, hname, len(filecontent), limit)) forbid = True return forbid The exception: $ hg commit -m 'commit message' error: pretxncommit hook raised an exception: apps/helpers/templatetags/include_extends.py@bced6272d8f4: not found in manifest transaction abort! rollback completed abort: apps/helpers/templatetags/include_extends.py@bced6272d8f4: not found in manifest! I'm not familiar with writing Mercurial hooks, so I'm pretty confused about what's going on. Why does the hook care that a file was removed if hg already knows about it? Is there a way to fix this hook so that it works all the time? Update (solved): I modified the hook to filter out files that were removed in the changeset. def newbinsize(ui, repo, node=None, **kwargs): '''forbid to add binary files over a given size''' forbid = False # default limit is 10 MB limit = int(ui.config('limits', 'maxnewbinsize', 10000000)) ctx = repo[node] for rev in xrange(ctx.rev(), len(repo)): ctx = context.changectx(repo, rev) # do not check the size of files that have been removed # files that have been removed do not have filecontexts # to test for whether a file was removed, test for the existence of a filecontext filecontexts = list(ctx) def file_was_removed(f): """Returns True if the file was removed""" if f not in filecontexts: return True else: return False for f in itertools.ifilterfalse(file_was_removed, ctx.files()): fctx = ctx.filectx(f) filecontent = fctx.data() # check only for new files if not fctx.parents(): if len(filecontent) > limit and util.binary(filecontent): msg = 'new binary file %s of %s is too large: %ld > %ld\n' hname = dpynode.short(ctx.node()) ui.write(_(msg) % (f, hname, len(filecontent), limit)) forbid = True return forbid

    Read the article

  • Long overdue (for me) question about disposing managed objects in .Net, VB.Net, C#

    - by Jules
    I can't believe I'm still confused about this but, any way, lets finally nail it: I have a class that overrides OnPaint to do some drawing. To speed things up, I create the pens, brushes etc before hand, in the construtor, so that OnPaint does not need to keep creating and disposing them. Now, I make sure that I always dispose of such objects, but I have the feeling I don't need to because, despite the fact they implement IDisposable, they're managed objects. Is this correct?

    Read the article

  • remote desktop client with panning of large desktops?

    - by mikewse
    When using the standard Windows remote desktop client to connect to a large remote desktop, the result is usually that the remote desktop is resized to a smaller size matching the local display. Are there any RDP clients that instead leave the remote desktop at its original size and pan this larger area when the mouse reaches the border edges of the local display? Thanks Mike

    Read the article

  • WCF service to send large binaries to server

    - by Ali Shafai
    I need to upload large (100 meg max) binairies to server using WCF. I followed instructions from this: http://www.c-sharpcorner.com/UploadFile/dhananjaycoder/fileuploadsilverlightwcf07142009104020AM/fileuploadsilverlightwcf.aspx it workds for anything less than 50K. going above that I get 415 errors. any idea?

    Read the article

  • NoHostAvailableException With Cassandra & DataStax Java Driver If Large ResultSet

    - by hughj
    The setup: 2-node Cassandra 1.2.6 cluster replicas=2 very large CQL3 table with no secondary index Rowkey is a UUID.randomUUID().toString() read consistency set to ONE Using DataStax java driver 1.0 The request: Attempting to do a table scan by "SELECT some-col from schema.table LIMIT nnn;" The fail: Once I go beyond a certain nnn LIMIT, I start to get NoHostAvailableExceptions from the driver. It reads like this: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.181.13.239 ([/10.181.13.239] Unexpected exception triggered)) at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64) at com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:214) at com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:169) at com.jpmc.es.rtm.storage.impl.EventExtract.main(EventExtract.java:36) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120) Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.181.13.239 ([/10.181.13.239] Unexpected exception triggered)) at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:98) at com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:165) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) Given: This is probably not the most enlightened thing to do to a large table with millions of rows, but this is how I learn what not to do, so I would really appreciate someone who could volunteer how this kind of error can be debugged. For example, when this happens, there are no indications that the nodes in the cluster ever had an issue with the request (there is nothing in the logs on either node that indicate any timeout or failure). Also, I enabled the trace on the driver, which gives you some nice autotrace (ala Oracle) info as long as the query succeeds. But in this case, the driver blows a NoHostAvailableException and no ExecutionInfo is available, so tracing has not provided any benefit in this case. I also find it interesting that this does not seem to be recorded as a timeout (my JMX consoles tell me no timeouts have occurred). So, I am left not understanding WHERE the failure is actually occurring. I am left with the idea that it is the driver that is having a problem, but I don't know how to debug it (and I would really like to). I have read several posts from folks that state that query'g for resultSets 10000 rows is probably not a good idea, and I am willing to accept this, but I would like to understand what is causing the exception and where the exception is happening. FWIW, I also tried bumping the timeout properties in the cassandra.yaml, but this made no difference whatsoever. I welcome any suggestions, anecdotes, insults, or monetary contributions for my registration in the house of moron-developers. Regards!!

    Read the article

  • How to quickly search an array of objects in Objective-C

    - by randombits
    Is there a way in Objective-C to search an array of objects by the contained object's properties if the properties are of type string? For instance, I have an NSArray of Person objects. Person has two properties, NSString *firstName and NSString *lastName. What's the best way to search through the array to find everyone who matches 'Ken' anywhere in the firstName OR lastName properties?

    Read the article

  • Testing for undefined and null child objects in ActionsScript/Flex

    - by dbasch
    Hi Everyone, I use this pattern to test for undefined and null values in ActionScript/Flex : if(obj) { execute() } Unfortunately, a ReferenceError is always thrown when I use the pattern to test for child objects : if(obj.child) { execute() } ReferenceError: Error #1069: Property child not found on obj and there is no default value. Why does testing for child objects with if statements throw a ReferenceError? Thanks!

    Read the article

  • Algorithm for dividing very large numbers

    - by pocoa
    I need to write an algorithm(I cannot use any 3rd party library, because this is an assignment) to divide(integer division, floating parts are not important) very large numbers like 100 - 1000 digits. I found http://en.wikipedia.org/wiki/Fourier_division algorithm but I don't know if it's the right way to go. Do you have any suggestions?

    Read the article

< Previous Page | 41 42 43 44 45 46 47 48 49 50 51 52  | Next Page >