Search Results

Search found 1137 results on 46 pages for 'barrier synchronization'.

Page 46/46 | < Previous Page | 42 43 44 45 46 

  • Fedora error log file

    - by user111196
    I am running a java application using this wrapper service yajsw. The problem it just stopped without any error in its logs file. So I was wondering will there be any system log file which will indicate the cause of it going down? Partial of the log file. Apr 6 00:12:20 localhost kernel: imklog 3.22.1, log source = /proc/kmsg started. Apr 6 00:12:20 localhost rsyslogd: [origin software="rsyslogd" swVersion="3.22.1" x-pid="2234" x-info="http://www.rsyslog.com"] (re)start Apr 6 00:12:20 localhost kernel: Initializing cgroup subsys cpuset Apr 6 00:12:20 localhost kernel: Initializing cgroup subsys cpu Apr 6 00:12:20 localhost kernel: Linux version 2.6.27.41-170.2.117.fc10.x86_64 ([email protected]) (gcc version 4.3.2 20081105 (Red Hat 4.3.2-7) (GCC) ) #1 SMP Thu Dec 10 10:36:29 EST 2009 Apr 6 00:12:20 localhost kernel: Command line: ro root=UUID=722ebf87-437f-4634-9c68-a82d157fa948 rhgb quiet Apr 6 00:12:20 localhost kernel: KERNEL supported cpus: Apr 6 00:12:20 localhost kernel: Intel GenuineIntel Apr 6 00:12:20 localhost kernel: AMD AuthenticAMD Apr 6 00:12:20 localhost kernel: Centaur CentaurHauls Apr 6 00:12:20 localhost kernel: BIOS-provided physical RAM map: Apr 6 00:12:20 localhost kernel: BIOS-e820: 0000000000000000 - 00000000000a0000 (usable) Apr 6 00:12:20 localhost kernel: BIOS-e820: 0000000000100000 - 00000000cfb50000 (usable) Apr 6 00:12:20 localhost kernel: BIOS-e820: 00000000cfb50000 - 00000000cfb66000 (reserved) Apr 6 00:12:20 localhost kernel: BIOS-e820: 00000000cfb66000 - 00000000cfb85c00 (ACPI data) Apr 6 00:12:20 localhost kernel: BIOS-e820: 00000000cfb85c00 - 00000000d0000000 (reserved) Apr 6 00:12:20 localhost kernel: BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved) Apr 6 00:12:20 localhost kernel: BIOS-e820: 00000000fe000000 - 0000000100000000 (reserved) Apr 6 00:12:20 localhost kernel: BIOS-e820: 0000000100000000 - 0000000330000000 (usable) Apr 6 00:12:20 localhost kernel: DMI 2.5 present. Apr 6 00:12:20 localhost kernel: last_pfn = 0x330000 max_arch_pfn = 0x3ffffffff Apr 6 00:12:20 localhost kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106 Apr 6 00:12:20 localhost kernel: last_pfn = 0xcfb50 max_arch_pfn = 0x3ffffffff Apr 6 00:12:20 localhost kernel: init_memory_mapping Apr 6 00:12:20 localhost kernel: last_map_addr: cfb50000 end: cfb50000 Apr 6 00:12:20 localhost kernel: init_memory_mapping Apr 6 00:12:20 localhost kernel: last_map_addr: 330000000 end: 330000000 Apr 6 00:12:20 localhost kernel: RAMDISK: 37bfc000 - 37fef6c8 Apr 6 00:12:20 localhost kernel: ACPI: RSDP 000F21B0, 0024 (r2 DELL ) Apr 6 00:12:20 localhost kernel: ACPI: XSDT 000F224C, 0084 (r1 DELL PE_SC3 1 DELL 1) Apr 6 00:12:20 localhost kernel: ACPI: FACP CFB83524, 00F4 (r3 DELL PE_SC3 1 DELL 1) Apr 6 00:12:20 localhost kernel: ACPI: DSDT CFB66000, 4974 (r1 DELL PE_SC3 1 INTL 20050624) Apr 6 00:12:20 localhost kernel: ACPI: FACS CFB85C00, 0040 Apr 6 00:12:20 localhost kernel: ACPI: APIC CFB83078, 00B6 (r1 DELL PE_SC3 1 DELL 1) Apr 6 00:12:20 localhost kernel: ACPI: SPCR CFB83130, 0050 (r1 DELL PE_SC3 1 DELL 1) Apr 6 00:12:20 localhost kernel: ACPI: HPET CFB83184, 0038 (r1 DELL PE_SC3 1 DELL 1) Apr 6 00:12:20 localhost kernel: ACPI: MCFG CFB831C0, 003C (r1 DELL PE_SC3 1 DELL 1) Apr 6 00:12:20 localhost kernel: ACPI: WD__ CFB83200, 0134 (r1 DELL PE_SC3 1 DELL 1) Apr 6 00:12:20 localhost kernel: ACPI: SLIC CFB83338, 0176 (r1 DELL PE_SC3 1 DELL 1) Apr 6 00:12:20 localhost kernel: ACPI: ERST CFB6AAF4, 0210 (r1 DELL PE_SC3 1 DELL 1) Apr 6 00:12:20 localhost kernel: ACPI: HEST CFB6AD04, 027C (r1 DELL PE_SC3 1 DELL 1) Apr 6 00:12:20 localhost kernel: ACPI: BERT CFB6A974, 0030 (r1 DELL PE_SC3 1 DELL 1) Apr 6 00:12:20 localhost kernel: ACPI: EINJ CFB6A9A4, 0150 (r1 DELL PE_SC3 1 DELL 1) Apr 6 00:12:20 localhost kernel: ACPI: TCPA CFB834BC, 0064 (r1 DELL PE_SC3 1 DELL 1) Apr 6 00:12:20 localhost kernel: No NUMA configuration found Apr 6 00:12:20 localhost kernel: Faking a node at 0000000000000000-0000000330000000 Apr 6 00:12:20 localhost kernel: Bootmem setup node 0 0000000000000000-0000000330000000 Apr 6 00:12:20 localhost kernel: NODE_DATA [0000000000015000 - 0000000000029fff] Apr 6 00:12:20 localhost kernel: bootmap [000000000002a000 - 000000000008ffff] pages 66 Apr 6 00:12:20 localhost kernel: (7 early reservations) ==> bootmem [0000000000 - 0330000000] Apr 6 00:12:20 localhost kernel: #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] Apr 6 00:12:20 localhost kernel: #1 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000] Apr 6 00:12:20 localhost kernel: #2 [0000200000 - 0000a310cc] TEXT DATA BSS ==> [0000200000 - 0000a310cc] Apr 6 00:12:20 localhost kernel: #3 [0037bfc000 - 0037fef6c8] RAMDISK ==> [0037bfc000 - 0037fef6c8] Apr 6 00:12:20 localhost kernel: #4 [000009f000 - 0000100000] BIOS reserved ==> [000009f000 - 0000100000] Apr 6 00:12:20 localhost kernel: #5 [0000008000 - 000000c000] PGTABLE ==> [0000008000 - 000000c000] Apr 6 00:12:20 localhost kernel: #6 [000000c000 - 0000015000] PGTABLE ==> [000000c000 - 0000015000] Apr 6 00:12:20 localhost kernel: found SMP MP-table at [ffff8800000fe710] 000fe710 Apr 6 00:12:20 localhost kernel: Zone PFN ranges: Apr 6 00:12:20 localhost kernel: DMA 0x00000000 -> 0x00001000 Apr 6 00:12:20 localhost kernel: DMA32 0x00001000 -> 0x00100000 Apr 6 00:12:20 localhost kernel: Normal 0x00100000 -> 0x00330000 Apr 6 00:12:20 localhost kernel: Movable zone start PFN for each node Apr 6 00:12:20 localhost kernel: early_node_map[3] active PFN ranges Apr 6 00:12:20 localhost kernel: 0: 0x00000000 -> 0x000000a0 Apr 6 00:12:20 localhost kernel: 0: 0x00000100 -> 0x000cfb50 Apr 6 00:12:20 localhost kernel: 0: 0x00100000 -> 0x00330000 Apr 6 00:12:20 localhost kernel: ACPI: PM-Timer IO Port: 0x808 Apr 6 00:12:20 localhost kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) Apr 6 00:12:20 localhost kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled) Apr 6 00:12:20 localhost kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled) Apr 6 00:12:20 localhost kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled) Apr 6 00:12:20 localhost kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled) Apr 6 00:12:20 localhost kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x05] enabled) Apr 6 00:12:20 localhost kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x03] enabled) Apr 6 00:12:20 localhost kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled) Apr 6 00:12:20 localhost kernel: ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1]) Apr 6 00:12:20 localhost kernel: ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0]) Apr 6 00:12:20 localhost kernel: IOAPIC[0]: apic_id 8, version 0, address 0xfec00000, GSI 0-23 Apr 6 00:12:20 localhost kernel: ACPI: IOAPIC (id[0x09] address[0xfec81000] gsi_base[64]) Apr 6 00:12:20 localhost kernel: IOAPIC[1]: apic_id 9, version 0, address 0xfec81000, GSI 64-87 Apr 6 00:12:20 localhost kernel: ACPI: IOAPIC (id[0x0a] address[0xfec84000] gsi_base[160]) Apr 6 00:12:20 localhost kernel: IOAPIC[2]: apic_id 10, version 0, address 0xfec84000, GSI 160-183 Apr 6 00:12:20 localhost kernel: ACPI: IOAPIC (id[0x0b] address[0xfec84800] gsi_base[224]) Apr 6 00:12:20 localhost kernel: IOAPIC[3]: apic_id 11, version 0, address 0xfec84800, GSI 224-247 Apr 6 00:12:20 localhost kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) Apr 6 00:12:20 localhost kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) Apr 6 00:12:20 localhost kernel: Setting APIC routing to flat Apr 6 00:12:20 localhost kernel: ACPI: HPET id: 0x8086a201 base: 0xfed00000 Apr 6 00:12:20 localhost kernel: Using ACPI (MADT) for SMP configuration information Apr 6 00:12:20 localhost kernel: SMP: Allowing 8 CPUs, 0 hotplug CPUs Apr 6 00:12:20 localhost kernel: PM: Registered nosave memory: 00000000000a0000 - 0000000000100000 Apr 6 00:12:20 localhost kernel: PM: Registered nosave memory: 00000000cfb50000 - 00000000cfb66000 Apr 6 00:12:20 localhost kernel: PM: Registered nosave memory: 00000000cfb66000 - 00000000cfb85000 Apr 6 00:12:20 localhost kernel: PM: Registered nosave memory: 00000000cfb85000 - 00000000cfb86000 Apr 6 00:12:20 localhost kernel: PM: Registered nosave memory: 00000000cfb86000 - 00000000d0000000 Apr 6 00:12:20 localhost kernel: PM: Registered nosave memory: 00000000d0000000 - 00000000e0000000 Apr 6 00:12:20 localhost kernel: PM: Registered nosave memory: 00000000e0000000 - 00000000f0000000 Apr 6 00:12:20 localhost kernel: PM: Registered nosave memory: 00000000f0000000 - 00000000fe000000 Apr 6 00:12:20 localhost kernel: PM: Registered nosave memory: 00000000fe000000 - 0000000100000000 Apr 6 00:12:20 localhost kernel: Allocating PCI resources starting at d1000000 (gap: d0000000:10000000) Apr 6 00:12:20 localhost kernel: PERCPU: Allocating 65184 bytes of per cpu data Apr 6 00:12:20 localhost kernel: Built 1 zonelists in Zone order, mobility grouping on. Total pages: 3096524 Apr 6 00:12:20 localhost kernel: Policy zone: Normal Apr 6 00:12:20 localhost kernel: Kernel command line: ro root=UUID=722ebf87-437f-4634-9c68-a82d157fa948 rhgb quiet Apr 6 00:12:20 localhost kernel: Initializing CPU#0 Apr 6 00:12:20 localhost kernel: PID hash table entries: 4096 (order: 12, 32768 bytes) Apr 6 00:12:20 localhost kernel: Extended CMOS year: 2000 Apr 6 00:12:20 localhost kernel: TSC: PIT calibration confirmed by PMTIMER. Apr 6 00:12:20 localhost kernel: TSC: using PMTIMER calibration value Apr 6 00:12:20 localhost kernel: Detected 1994.992 MHz processor. Apr 6 00:12:20 localhost kernel: Console: colour VGA+ 80x25 Apr 6 00:12:20 localhost kernel: console [tty0] enabled Apr 6 00:12:20 localhost kernel: Checking aperture... Apr 6 00:12:20 localhost kernel: No AGP bridge found Apr 6 00:12:20 localhost kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB) Apr 6 00:12:20 localhost kernel: Placing software IO TLB between 0x20000000 - 0x24000000 Apr 6 00:12:20 localhost kernel: Memory: 12324244k/13369344k available (3311k kernel code, 253484k reserved, 1844k data, 1296k init) Apr 6 00:12:20 localhost kernel: SLUB: Genslabs=13, HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1 Apr 6 00:12:20 localhost kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 3989.98 BogoMIPS (lpj=1994992) Apr 6 00:12:20 localhost kernel: Security Framework initialized Apr 6 00:12:20 localhost kernel: SELinux: Initializing. Apr 6 00:12:20 localhost kernel: Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes) Apr 6 00:12:20 localhost kernel: Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes) Apr 6 00:12:20 localhost kernel: Mount-cache hash table entries: 256 Apr 6 00:12:20 localhost kernel: Initializing cgroup subsys ns Apr 6 00:12:20 localhost kernel: Initializing cgroup subsys cpuacct Apr 6 00:12:20 localhost kernel: Initializing cgroup subsys devices Apr 6 00:12:20 localhost kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Apr 6 00:12:20 localhost kernel: CPU: L2 cache: 4096K Apr 6 00:12:20 localhost kernel: CPU 0/0 -> Node 0 Apr 6 00:12:20 localhost kernel: CPU: Physical Processor ID: 0 Apr 6 00:12:20 localhost kernel: CPU: Processor Core ID: 0 Apr 6 00:12:20 localhost kernel: CPU0: Thermal monitoring enabled (TM1) Apr 6 00:12:20 localhost kernel: using mwait in idle threads. Apr 6 00:12:20 localhost kernel: ACPI: Core revision 20080609 Apr 6 00:12:20 localhost kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 Apr 6 00:12:20 localhost kernel: CPU0: Intel(R) Xeon(R) CPU E5335 @ 2.00GHz stepping 07 Apr 6 00:12:20 localhost kernel: Using local APIC timer interrupts. Apr 6 00:12:20 localhost kernel: Detected 20.781 MHz APIC timer. Apr 6 00:12:20 localhost kernel: Booting processor 1/4 ip 6000 Apr 6 00:12:20 localhost kernel: Initializing CPU#1 Apr 6 00:12:20 localhost kernel: Calibrating delay using timer specific routine.. 3990.05 BogoMIPS (lpj=1995026) Apr 6 00:12:20 localhost kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Apr 6 00:12:20 localhost kernel: CPU: L2 cache: 4096K Apr 6 00:12:20 localhost kernel: CPU 1/4 -> Node 0 Apr 6 00:12:20 localhost kernel: CPU: Physical Processor ID: 1 Apr 6 00:12:20 localhost kernel: CPU: Processor Core ID: 0 Apr 6 00:12:20 localhost kernel: CPU1: Thermal monitoring enabled (TM2) Apr 6 00:12:20 localhost kernel: x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106 Apr 6 00:12:20 localhost kernel: CPU1: Intel(R) Xeon(R) CPU E5335 @ 2.00GHz stepping 07 Apr 6 00:12:20 localhost kernel: checking TSC synchronization [CPU#0 -> CPU#1]: passed. Apr 6 00:12:20 localhost kernel: Booting processor 2/2 ip 6000 Apr 6 00:12:20 localhost kernel: Initializing CPU#2 Apr 6 00:12:20 localhost kernel: Calibrating delay using timer specific routine.. 3990.05 BogoMIPS (lpj=1995029)

    Read the article

  • How to tune down the Hyperic built-in postgresql database for a small setup

    - by Svish
    We are testing out Hyperic 4.5.1 in a quite small environment for now. Currently there are just 1-5 agents and there probably won't be any more than 10-15. When I run ps ax there are 20(!) postgres processes running. For a small setup like this, that can't be necessary, can it? I'm a software developer and don't have much experience with setting up servers and such though, so don't really know. Either way, what settings are appropriate for a small Hyperic setup like this? Current, default and untouched configuration file, hqdb/data/postgresql.conf: # ----------------------------- # PostgreSQL configuration file # ----------------------------- # # This file consists of lines of the form: # # name = value # # (The '=' is optional.) White space may be used. Comments are introduced # with '#' anywhere on a line. The complete list of option names and # allowed values can be found in the PostgreSQL documentation. The # commented-out settings shown in this file represent the default values. # # Please note that re-commenting a setting is NOT sufficient to revert it # to the default value, unless you restart the server. # # Any option can also be given as a command line switch to the server, # e.g., 'postgres -c log_connections=on'. Some options can be changed at # run-time with the 'SET' SQL command. # # This file is read on server startup and when the server receives a # SIGHUP. If you edit the file on a running system, you have to SIGHUP the # server for the changes to take effect, or use "pg_ctl reload". Some # settings, which are marked below, require a server shutdown and restart # to take effect. # # Memory units: kB = kilobytes MB = megabytes GB = gigabytes # Time units: ms = milliseconds s = seconds min = minutes h = hours d = days #--------------------------------------------------------------------------- # FILE LOCATIONS #--------------------------------------------------------------------------- # The default values of these variables are driven from the -D command line # switch or PGDATA environment variable, represented here as ConfigDir. #data_directory = 'ConfigDir' # use data in another directory # (change requires restart) #hba_file = 'ConfigDir/pg_hba.conf' # host-based authentication file # (change requires restart) #ident_file = 'ConfigDir/pg_ident.conf' # ident configuration file # (change requires restart) # If external_pid_file is not explicitly set, no extra PID file is written. #external_pid_file = '(none)' # write an extra PID file # (change requires restart) #--------------------------------------------------------------------------- # CONNECTIONS AND AUTHENTICATION #--------------------------------------------------------------------------- # - Connection Settings - #listen_addresses = 'localhost' # what IP address(es) to listen on; # comma-separated list of addresses; # defaults to 'localhost', '*' = all # (change requires restart) port = 9432 # (change requires restart) max_connections = 100 # (change requires restart) # Note: increasing max_connections costs ~400 bytes of shared memory per # connection slot, plus lock space (see max_locks_per_transaction). You # might also need to raise shared_buffers to support more connections. #superuser_reserved_connections = 3 # (change requires restart) #unix_socket_directory = '' # (change requires restart) #unix_socket_group = '' # (change requires restart) #unix_socket_permissions = 0777 # octal # (change requires restart) #bonjour_name = '' # defaults to the computer name # (change requires restart) # - Security & Authentication - #authentication_timeout = 1min # 1s-600s #ssl = off # (change requires restart) #password_encryption = on #db_user_namespace = off # Kerberos #krb_server_keyfile = '' # (change requires restart) #krb_srvname = 'postgres' # (change requires restart) #krb_server_hostname = '' # empty string matches any keytab entry # (change requires restart) #krb_caseins_users = off # (change requires restart) # - TCP Keepalives - # see 'man 7 tcp' for details #tcp_keepalives_idle = 0 # TCP_KEEPIDLE, in seconds; # 0 selects the system default #tcp_keepalives_interval = 0 # TCP_KEEPINTVL, in seconds; # 0 selects the system default #tcp_keepalives_count = 0 # TCP_KEEPCNT; # 0 selects the system default #--------------------------------------------------------------------------- # RESOURCE USAGE (except WAL) #--------------------------------------------------------------------------- # - Memory - shared_buffers = 64MB # min 128kB or max_connections*16kB # (change requires restart) #temp_buffers = 8MB # min 800kB #max_prepared_transactions = 5 # can be 0 or more # (change requires restart) # Note: increasing max_prepared_transactions costs ~600 bytes of shared memory # per transaction slot, plus lock space (see max_locks_per_transaction). work_mem = 2MB # min 64kB maintenance_work_mem = 32MB # min 1MB #max_stack_depth = 2MB # min 100kB # - Free Space Map - max_fsm_pages = 204800 # min max_fsm_relations*16, 6 bytes each # (change requires restart) #max_fsm_relations = 1000 # min 100, ~70 bytes each # (change requires restart) # - Kernel Resource Usage - #max_files_per_process = 1000 # min 25 # (change requires restart) #shared_preload_libraries = '' # (change requires restart) # - Cost-Based Vacuum Delay - #vacuum_cost_delay = 0 # 0-1000 milliseconds #vacuum_cost_page_hit = 1 # 0-10000 credits #vacuum_cost_page_miss = 10 # 0-10000 credits #vacuum_cost_page_dirty = 20 # 0-10000 credits #vacuum_cost_limit = 200 # 0-10000 credits # - Background writer - #bgwriter_delay = 200ms # 10-10000ms between rounds #bgwriter_lru_percent = 1.0 # 0-100% of LRU buffers scanned/round #bgwriter_lru_maxpages = 5 # 0-1000 buffers max written/round #bgwriter_all_percent = 0.333 # 0-100% of all buffers scanned/round #bgwriter_all_maxpages = 5 # 0-1000 buffers max written/round #--------------------------------------------------------------------------- # WRITE AHEAD LOG #--------------------------------------------------------------------------- # - Settings - fsync = on # turns forced synchronization on or off #wal_sync_method = fsync # the default is the first option # supported by the operating system: # open_datasync # fdatasync # fsync # fsync_writethrough # open_sync #full_page_writes = on # recover from partial page writes #wal_buffers = 64kB # min 32kB # (change requires restart) commit_delay = 100000 # range 0-100000, in microseconds #commit_siblings = 5 # range 1-1000 # - Checkpoints - checkpoint_segments = 10 # in logfile segments, min 1, 16MB each #checkpoint_timeout = 5min # range 30s-1h #checkpoint_warning = 30s # 0 is off # - Archiving - #archive_command = '' # command to use to archive a logfile segment #archive_timeout = 0 # force a logfile segment switch after this # many seconds; 0 is off #--------------------------------------------------------------------------- # QUERY TUNING #--------------------------------------------------------------------------- # - Planner Method Configuration - #enable_bitmapscan = on #enable_hashagg = on #enable_hashjoin = on #enable_indexscan = on #enable_mergejoin = on #enable_nestloop = on #enable_seqscan = on #enable_sort = on #enable_tidscan = on # - Planner Cost Constants - #seq_page_cost = 1.0 # measured on an arbitrary scale #random_page_cost = 4.0 # same scale as above #cpu_tuple_cost = 0.01 # same scale as above #cpu_index_tuple_cost = 0.005 # same scale as above #cpu_operator_cost = 0.0025 # same scale as above #effective_cache_size = 128MB # - Genetic Query Optimizer - #geqo = on #geqo_threshold = 12 #geqo_effort = 5 # range 1-10 #geqo_pool_size = 0 # selects default based on effort #geqo_generations = 0 # selects default based on effort #geqo_selection_bias = 2.0 # range 1.5-2.0 # - Other Planner Options - #default_statistics_target = 10 # range 1-1000 #constraint_exclusion = off #from_collapse_limit = 8 #join_collapse_limit = 8 # 1 disables collapsing of explicit # JOINs #--------------------------------------------------------------------------- # ERROR REPORTING AND LOGGING #--------------------------------------------------------------------------- # - Where to Log - log_destination = 'stderr' # Valid values are combinations of # stderr, syslog and eventlog, # depending on platform. # This is used when logging to stderr: redirect_stderr = on # Enable capturing of stderr into log # files # (change requires restart) # These are only used if redirect_stderr is on: log_directory = '../../logs' # Directory where log files are written # Can be absolute or relative to PGDATA log_filename = 'hqdb-%Y-%m-%d.log' # Log file name pattern. # Can include strftime() escapes #log_truncate_on_rotation = off # If on, any existing log file of the same # name as the new log file will be # truncated rather than appended to. But # such truncation only occurs on # time-driven rotation, not on restarts # or size-driven rotation. Default is # off, meaning append to existing files # in all cases. log_rotation_age = 1d # Automatic rotation of logfiles will # happen after that time. 0 to # disable. #log_rotation_size = 10MB # Automatic rotation of logfiles will # happen after that much log # output. 0 to disable. # These are relevant when logging to syslog: #syslog_facility = 'LOCAL0' #syslog_ident = 'postgres' # - When to Log - #client_min_messages = notice # Values, in order of decreasing detail: # debug5 # debug4 # debug3 # debug2 # debug1 # log # notice # warning # error #log_min_messages = notice # Values, in order of decreasing detail: # debug5 # debug4 # debug3 # debug2 # debug1 # info # notice # warning # error # log # fatal # panic #log_error_verbosity = default # terse, default, or verbose messages #log_min_error_statement = error # Values in order of increasing severity: # debug5 # debug4 # debug3 # debug2 # debug1 # info # notice # warning # error # fatal # panic (effectively off) log_min_duration_statement = 10000 # -1 is disabled, 0 logs all statements # and their durations. #silent_mode = off # DO NOT USE without syslog or # redirect_stderr # (change requires restart) # - What to Log - #debug_print_parse = off #debug_print_rewritten = off #debug_print_plan = off #debug_pretty_print = off #log_connections = off #log_disconnections = off #log_duration = off #log_line_prefix = '' # Special values: # %u = user name # %d = database name # %r = remote host and port # %h = remote host # %p = PID # %t = timestamp (no milliseconds) # %m = timestamp with milliseconds # %i = command tag # %c = session id # %l = session line number # %s = session start timestamp # %x = transaction id # %q = stop here in non-session # processes # %% = '%' # e.g. '<%u%%%d> ' #log_statement = 'none' # none, ddl, mod, all #log_hostname = off #--------------------------------------------------------------------------- # RUNTIME STATISTICS #--------------------------------------------------------------------------- # - Query/Index Statistics Collector - #stats_command_string = on #update_process_title = on stats_start_collector = on # needed for block or row stats # (change requires restart) stats_block_level = on stats_row_level = on stats_reset_on_server_start = off # (change requires restart) # - Statistics Monitoring - #log_parser_stats = off #log_planner_stats = off #log_executor_stats = off #log_statement_stats = off #--------------------------------------------------------------------------- # AUTOVACUUM PARAMETERS #--------------------------------------------------------------------------- #autovacuum = off # enable autovacuum subprocess? # 'on' requires stats_start_collector # and stats_row_level to also be on #autovacuum_naptime = 1min # time between autovacuum runs #autovacuum_vacuum_threshold = 500 # min # of tuple updates before # vacuum #autovacuum_analyze_threshold = 250 # min # of tuple updates before # analyze #autovacuum_vacuum_scale_factor = 0.2 # fraction of rel size before # vacuum #autovacuum_analyze_scale_factor = 0.1 # fraction of rel size before # analyze #autovacuum_freeze_max_age = 200000000 # maximum XID age before forced vacuum # (change requires restart) #autovacuum_vacuum_cost_delay = -1 # default vacuum cost delay for # autovacuum, -1 means use # vacuum_cost_delay #autovacuum_vacuum_cost_limit = -1 # default vacuum cost limit for # autovacuum, -1 means use # vacuum_cost_limit #--------------------------------------------------------------------------- # CLIENT CONNECTION DEFAULTS #--------------------------------------------------------------------------- # - Statement Behavior - #search_path = '"$user",public' # schema names #default_tablespace = '' # a tablespace name, '' uses # the default #check_function_bodies = on #default_transaction_isolation = 'read committed' #default_transaction_read_only = off #statement_timeout = 0 # 0 is disabled #vacuum_freeze_min_age = 100000000 # - Locale and Formatting - datestyle = 'iso, mdy' #timezone = unknown # actually, defaults to TZ # environment setting #timezone_abbreviations = 'Default' # select the set of available timezone # abbreviations. Currently, there are # Default # Australia # India # However you can also create your own # file in share/timezonesets/. #extra_float_digits = 0 # min -15, max 2 #client_encoding = sql_ascii # actually, defaults to database # encoding # These settings are initialized by initdb -- they might be changed lc_messages = 'C' # locale for system error message # strings lc_monetary = 'C' # locale for monetary formatting lc_numeric = 'C' # locale for number formatting lc_time = 'C' # locale for time formatting # - Other Defaults - #explain_pretty_print = on #dynamic_library_path = '$libdir' #local_preload_libraries = '' #--------------------------------------------------------------------------- # LOCK MANAGEMENT #--------------------------------------------------------------------------- #deadlock_timeout = 1s #max_locks_per_transaction = 64 # min 10 # (change requires restart) # Note: each lock table slot uses ~270 bytes of shared memory, and there are # max_locks_per_transaction * (max_connections + max_prepared_transactions) # lock table slots. #--------------------------------------------------------------------------- # VERSION/PLATFORM COMPATIBILITY #--------------------------------------------------------------------------- # - Previous Postgres Versions - #add_missing_from = off #array_nulls = on #backslash_quote = safe_encoding # on, off, or safe_encoding #default_with_oids = off #escape_string_warning = on #standard_conforming_strings = off #regex_flavor = advanced # advanced, extended, or basic #sql_inheritance = on # - Other Platforms & Clients - #transform_null_equals = off #--------------------------------------------------------------------------- # CUSTOMIZED OPTIONS #--------------------------------------------------------------------------- #custom_variable_classes = '' # list of custom variable class names SELECT * FROM pg_stat_activity; datid | datname | procpid | usesysid | usename | current_query | waiting | query_start | backend_start | client_addr | client_port -------+---------+---------+----------+---------+---------------------------------+---------+-------------------------------+-------------------------------+-------------+------------- 16384 | hqdb | 3267 | 10 | hqadmin | <IDLE> | f | 2011-02-08 15:51:20.036781+01 | 2011-02-08 15:51:20.02413+01 | 127.0.0.1 | 47892 16384 | hqdb | 3268 | 10 | hqadmin | <IDLE> | f | 2011-02-08 15:51:20.050994+01 | 2011-02-08 15:51:20.047393+01 | 127.0.0.1 | 47893 16384 | hqdb | 3269 | 10 | hqadmin | <IDLE> | f | 2011-02-08 15:51:20.056661+01 | 2011-02-08 15:51:20.053201+01 | 127.0.0.1 | 47894 16384 | hqdb | 3271 | 10 | hqadmin | <IDLE> | f | 2011-02-08 15:51:20.062351+01 | 2011-02-08 15:51:20.058822+01 | 127.0.0.1 | 47895 16384 | hqdb | 3272 | 10 | hqadmin | <IDLE> | f | 2011-02-08 15:51:20.068328+01 | 2011-02-08 15:51:20.064517+01 | 127.0.0.1 | 47896 16384 | hqdb | 3273 | 10 | hqadmin | <IDLE> | f | 2011-02-08 15:51:20.07444+01 | 2011-02-08 15:51:20.070755+01 | 127.0.0.1 | 47897 16384 | hqdb | 3274 | 10 | hqadmin | <IDLE> | f | 2011-02-08 15:51:20.080941+01 | 2011-02-08 15:51:20.076983+01 | 127.0.0.1 | 47898 16384 | hqdb | 3275 | 10 | hqadmin | <IDLE> | f | 2011-02-08 15:51:20.08741+01 | 2011-02-08 15:51:20.083697+01 | 127.0.0.1 | 47899 16384 | hqdb | 3276 | 10 | hqadmin | <IDLE> | f | 2011-02-08 15:51:20.093597+01 | 2011-02-08 15:51:20.089977+01 | 127.0.0.1 | 47900 16384 | hqdb | 3277 | 10 | hqadmin | <IDLE> in transaction | f | 2011-02-08 15:51:20.133974+01 | 2011-02-08 15:51:20.096149+01 | 127.0.0.1 | 47901 16384 | hqdb | 3308 | 10 | hqadmin | <IDLE> | f | 2011-02-09 10:49:27.402197+01 | 2011-02-08 15:51:29.826321+01 | 127.0.0.1 | 47902 16384 | hqdb | 3309 | 10 | hqadmin | <IDLE> | f | 2011-02-08 15:51:55.572395+01 | 2011-02-08 15:51:29.865243+01 | 127.0.0.1 | 47903 16384 | hqdb | 3310 | 10 | hqadmin | <IDLE> | f | 2011-02-08 15:51:55.586273+01 | 2011-02-08 15:51:29.874346+01 | 127.0.0.1 | 47904 16384 | hqdb | 3311 | 10 | hqadmin | <IDLE> | f | 2011-02-09 10:10:03.024088+01 | 2011-02-08 15:51:29.883598+01 | 127.0.0.1 | 47905 16384 | hqdb | 3312 | 10 | hqadmin | <IDLE> in transaction | f | 2011-02-08 15:51:35.804457+01 | 2011-02-08 15:51:29.892925+01 | 127.0.0.1 | 47906 16384 | hqdb | 3418 | 10 | hqadmin | <IDLE> | f | 2011-02-08 15:51:55.580207+01 | 2011-02-08 15:51:55.56911+01 | 127.0.0.1 | 47910 16384 | hqdb | 3419 | 10 | hqadmin | <IDLE> | f | 2011-02-08 15:51:55.59781+01 | 2011-02-08 15:51:55.588609+01 | 127.0.0.1 | 47911 16384 | hqdb | 3422 | 10 | hqadmin | <IDLE> | f | 2011-02-09 10:10:02.668836+01 | 2011-02-08 15:51:55.603076+01 | 127.0.0.1 | 47914 16384 | hqdb | 3421 | 10 | hqadmin | <IDLE> | f | 2011-02-08 15:51:55.770427+01 | 2011-02-08 15:51:55.603086+01 | 127.0.0.1 | 47913 16384 | hqdb | 3420 | 10 | hqadmin | <IDLE> | f | 2011-02-08 15:51:55.680785+01 | 2011-02-08 15:51:55.637058+01 | 127.0.0.1 | 47912 16384 | hqdb | 18233 | 10 | hqadmin | SELECT * FROM pg_stat_activity; | f | 2011-02-09 10:49:29.688949+01 | 2011-02-09 10:48:13.031475+01 | | -1 (21 rows)

    Read the article

  • FreeBSD performance tuning. Sysctls, loader.conf, kernel.

    - by SaveTheRbtz
    I wanted to share knowledge of tuning FreeBSD via sysctls, so i'm posting them with comments. Based on Igor Sysoev (author of nginx) presentation about FreeBSD tuning up to 100,000-200,000 active connections. Sysctls are for 7.x FreeBSD. Since 7.2 amd64 some of them are tuned well by default. Prior 7.0 some of them are boot only (set via /boot/loader.conf) or does not exist at all. Highload web server sysctls: # Max. backlog size kern.ipc.somaxconn=4096 # Shared memory // 7.2+ can use shared memory > 2Gb kern.ipc.shmmax=2147483648 # Sockets kern.ipc.maxsockets=204800 # Do not use lager sockbufs on 8.0 # ( http://old.nabble.com/Significant-performance-regression-for-increased-maxsockbuf-on-8.0-RELEASE-tt26745981.html#a26745981 ) kern.ipc.maxsockbuf=262144 # Recive clusters (on amd64 7.2+ 65k is default) # For such high value vm.kmem_size must be increased to 3G #kern.ipc.nmbclusters=229376 # Jumbo pagesize(4k/8k) clusters # Used as general packet storage for jumbo frames # can be monitored via `netstat -m` #kern.ipc.nmbjumbop=192000 # Jumbo 9k/16k clusters # If you are using them #kern.ipc.nmbjumbo9=24000 #kern.ipc.nmbjumbo16=10240 # Every socket is a file, so increase them kern.maxfiles=204800 kern.maxfilesperproc=200000 kern.maxvnodes=200000 # Turn off receive autotuning #net.inet.tcp.recvbuf_auto=0 # Small receive space, only usable on http-server, on file server this # should be increased to 65535 or even more #net.inet.tcp.recvspace=8192 # Small send space is useful for http servers that serve small files # Autotuned since 7.x net.inet.tcp.sendspace=16384 # This should be enabled if you going to use big spaces (>64k) #net.inet.tcp.rfc1323=1 # Turn this off on highspeed, lossless connections (LAN 1Gbit+) #net.inet.tcp.delayed_ack=0 # This feature is useful if you are serving data over modems, Gigabit Ethernet, # or even high speed WAN links (or any other link with a high bandwidth delay product), # especially if you are also using window scaling or have configured a large send window. # You can try setting it to 0 on fileserver with 1GBit+ interfaces # Automatically disables on small RTT ( http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/netinet/tcp_subr.c?#rev1.237 ) #net.inet.tcp.inflight.enable=0 # Disable randomizing of ports to avoid false RST # Before usage check SA here www.bsdcan.org/2006/papers/ImprovingTCPIP.pdf # (it's also says that port randomization auto-disables at some conn.rates, but I didn't tested it thou) #net.inet.ip.portrange.randomized=0 # Increase portrange # For outgoing connections only. Good for seed-boxes and ftp servers. net.inet.ip.portrange.first=1024 net.inet.ip.portrange.last=65535 # Security net.inet.ip.redirect=0 net.inet.ip.sourceroute=0 net.inet.ip.accept_sourceroute=0 net.inet.icmp.maskrepl=0 net.inet.icmp.log_redirect=0 net.inet.icmp.drop_redirect=1 net.inet.tcp.drop_synfin=1 # Security net.inet.udp.blackhole=1 net.inet.tcp.blackhole=2 # Increases default TTL, sometimes useful # Default is 64 net.inet.ip.ttl=128 # Lessen max segment life to conserve resources # ACK waiting time in miliseconds (default: 30000 from RFC) net.inet.tcp.msl=5000 # Max bumber of timewait sockets net.inet.tcp.maxtcptw=40960 # Don't use tw on local connections # As of 15 Apr 2009. Igor Sysoev says that nolocaltimewait has some buggy realization. # So disable it or now till get fixed #net.inet.tcp.nolocaltimewait=1 # FIN_WAIT_2 state fast recycle net.inet.tcp.fast_finwait2_recycle=1 # Time before tcp keepalive probe is sent # default is 2 hours (7200000) #net.inet.tcp.keepidle=60000 # Should be increased until net.inet.ip.intr_queue_drops is zero net.inet.ip.intr_queue_maxlen=4096 # Interrupt handling via multiple CPU, but with context switch. # You can play with it. Default is 1; #net.isr.direct=0 # This is for routers only #net.inet.ip.forwarding=1 #net.inet.ip.fastforwarding=1 # This speed ups dummynet when channel isn't saturated net.inet.ip.dummynet.io_fast=1 # Increase dummynet(4) hash #net.inet.ip.dummynet.hash_size=2048 #net.inet.ip.dummynet.max_chain_len # Should be increased when you have A LOT of files on server # (Increase until vfs.ufs.dirhash_mem becames lower) vfs.ufs.dirhash_maxmem=67108864 # Explicit Congestion Notification (see http://en.wikipedia.org/wiki/Explicit_Congestion_Notification) net.inet.tcp.ecn.enable=1 # Flowtable - flow caching mechanism # Useful for routers #net.inet.flowtable.enable=1 #net.inet.flowtable.nmbflows=65535 # Extreme polling tuning #kern.polling.burst_max=1000 #kern.polling.each_burst=1000 #kern.polling.reg_frac=100 #kern.polling.user_frac=1 #kern.polling.idle_poll=0 # IPFW dynamic rules and timeouts tuning # Increase dyn_buckets till net.inet.ip.fw.curr_dyn_buckets is lower net.inet.ip.fw.dyn_buckets=65536 net.inet.ip.fw.dyn_max=65536 net.inet.ip.fw.dyn_ack_lifetime=120 net.inet.ip.fw.dyn_syn_lifetime=10 net.inet.ip.fw.dyn_fin_lifetime=2 net.inet.ip.fw.dyn_short_lifetime=10 # Make packets pass firewall only once when using dummynet # i.e. packets going thru pipe are passing out from firewall with accept #net.inet.ip.fw.one_pass=1 # shm_use_phys Wires all shared pages, making them unswappable # Use this to lessen Virtual Memory Manager's work when using Shared Mem. # Useful for databases #kern.ipc.shm_use_phys=1 /boot/loader.conf: # Accept filters for data, http and DNS requests # Usefull when your software uses select() instead of kevent/kqueue or when you under DDoS # DNS accf available on 8.0+ accf_data_load="YES" accf_http_load="YES" accf_dns_load="YES" # Async IO system calls aio_load="YES" # Adds NCQ support in FreeBSD # WARNING! all ad[0-9]+ devices will be renamed to ada[0-9]+ # 8.0+ only #ahci_load= #siis_load= # Increase kernel memory size to 3G. # # Use ONLY if you have KVA_PAGES in kernel configuration, and you have more than 3G RAM # Otherwise panic will happen on next reboot! # # It's required for high buffer sizes: kern.ipc.nmbjumbop, kern.ipc.nmbclusters, etc # Useful on highload stateful firewalls, proxies or ZFS fileservers # (FreeBSD 7.2+ amd64 users: Check that current value is lower!) #vm.kmem_size="3G" # Older versions of FreeBSD can't tune maxfiles on the fly #kern.maxfiles="200000" # Useful for databases # Sets maximum data size to 1G # (FreeBSD 7.2+ amd64 users: Check that current value is lower!) #kern.maxdsiz="1G" # Maximum buffer size(vfs.maxbufspace) # You can check current one via vfs.bufspace # Should be lowered/upped depending on server's load-type # Usually decreased to preserve kmem # (default is 200M) #kern.maxbcache="512M" # Sendfile buffers # For i386 only #kern.ipc.nsfbufs=10240 # syncache Hash table tuning net.inet.tcp.syncache.hashsize=1024 net.inet.tcp.syncache.bucketlimit=100 # Incresed hostcache net.inet.tcp.hostcache.hashsize="16384" net.inet.tcp.hostcache.bucketlimit="100" # TCP control-block Hash table tuning net.inet.tcp.tcbhashsize=4096 # Enable superpages, for 7.2+ only # Also read http://lists.freebsd.org/pipermail/freebsd-hackers/2009-November/030094.html vm.pmap.pg_ps_enabled=1 # Usefull if you are using Intel-Gigabit NIC #hw.em.rxd=4096 #hw.em.txd=4096 #hw.em.rx_process_limit="-1" # Also if you have ALOT interrupts on NIC - play with following parameters # NOTE: You should set them for every NIC #dev.em.0.rx_int_delay: 250 #dev.em.0.tx_int_delay: 250 #dev.em.0.rx_abs_int_delay: 250 #dev.em.0.tx_abs_int_delay: 250 # There is also multithreaded version of em drivers can be found here: # http://people.yandex-team.ru/~wawa/ # # for additional em monitoring and statistics use # `sysctl dev.em.0.stats=1 ; dmesg` # #Same tunings for igb #hw.igb.rxd=4096 #hw.igb.txd=4096 #hw.igb.rx_process_limit=100 # Some useful netisr tunables. See sysctl net.isr #net.isr.defaultqlimit=4096 #net.isr.maxqlimit: 10240 # Bind netisr threads to CPUs #net.isr.bindthreads=1 # Nicer boot logo =) loader_logo="beastie" And finally here is my additions to GENERIC kernel # Just some of them, see also # cat /sys/{i386,amd64,}/conf/NOTES # This one useful only on i386 #options KVA_PAGES=512 # You can play with HZ in environments with high interrupt rate (default is 1000) # 100 is for my notebook to prolong it's battery life #options HZ=100 # Polling is goot on network loads with high packet rates and low-end NICs # NB! Do not enable it if you want more than one netisr thread #options DEVICE_POLLING # Eliminate datacopy on socket read-write # To take advantage with zero copy sockets you should have an MTU of 8K(amd64) # (4k for i386). This req. is only for receiving data. # Read more in man zero_copy_sockets #options ZERO_COPY_SOCKETS # Support TCP sign. Used for IPSec options TCP_SIGNATURE options IPSEC # This ones can be loaded as modules. They described in loader.conf section #options ACCEPT_FILTER_DATA #options ACCEPT_FILTER_HTTP # Adding ipfw, also can be loaded as modules options IPFIREWALL options IPFIREWALL_VERBOSE options IPFIREWALL_VERBOSE_LIMIT=10 options IPFIREWALL_DEFAULT_TO_ACCEPT options IPFIREWALL_FORWARD # Adding kernel NAT options IPFIREWALL_NAT options LIBALIAS # Traffic shaping options DUMMYNET # Divert, i.e. for userspace NAT options IPDIVERT # This is for OpenBSD's pf firewall device pf device pflog # pf's QoS - ALTQ options ALTQ options ALTQ_CBQ # Class Bases Queuing (CBQ) options ALTQ_RED # Random Early Detection (RED) options ALTQ_RIO # RED In/Out options ALTQ_HFSC # Hierarchical Packet Scheduler (HFSC) options ALTQ_PRIQ # Priority Queuing (PRIQ) options ALTQ_NOPCC # Required for SMP build # Pretty console # Manual can be found here http://forums.freebsd.org/showthread.php?t=6134 #options VESA #options SC_PIXEL_MODE # Disable reboot on Ctrl Alt Del #options SC_DISABLE_REBOOT # Change normal|kernel messages color options SC_NORM_ATTR=(FG_GREEN|BG_BLACK) options SC_KERNEL_CONS_ATTR=(FG_YELLOW|BG_BLACK) # More scroll space options SC_HISTORY_SIZE=8192 # Adding hardware crypto device device crypto device cryptodev # Useful network interfaces device vlan device tap #Virtual Ethernet driver device gre #IP over IP tunneling device if_bridge #Bridge interface device pfsync #synchronization interface for PF device carp #Common Address Redundancy Protocol device enc #IPsec interface device lagg #Link aggregation interface device stf #IPv4-IPv6 port # Also for my notebook, but may be used with Opteron #device amdtemp # Support for ECMP. More than one route for destination # Works even with default route so one can use it as LB for two ISP # For now code is unstable and panics (panic: rtfree 2) on route deletions. #options RADIX_MPATH # Multicast routing #options MROUTING #options PIM # DTrace options KDTRACE_HOOKS # all architectures - enable general DTrace hooks options DDB_CTF # all architectures - kernel ELF linker loads CTF data #options KDTRACE_FRAME # amd64-only # Adaptive spining in lockmgr (8.x+) # See http://www.mail-archive.com/[email protected]/msg10782.html options ADAPTIVE_LOCKMGRS # UTF-8 in console (9.x+) #options TEKEN_UTF8 #options TEKEN_XTERM # NCQ support # WARNING! all ad[0-9]+ devices will be renamed to ada[0-9]+ #options ATA_CAM # FreeBSD 9+ # Deadlock resolver thread # For additional information see http://www.mail-archive.com/[email protected]/msg18124.html #options DEADLKRES PS. Also most of FreeBSD's limits can be monitored by # vmstat -z and # limits PPS. variety of network counters can be monitored via # netstat -s In FreeBSD-9 netstat's -Q option appeared, try following command to display netisr stats # netstat -Q PPPS. also see # man 7 tuning PPPPS. I wanted to thank FreeBSD community, especially author of nginx - Igor Sysoev, nginx-ru@ and FreeBSD-performance@ mailing lists for providing useful information about FreeBSD tuning. So here is the question: What tunings are you using on yours FreeBSD servers? You can also post your /etc/sysctl.conf, /boot/loader.conf, kernel options, etc with description of its' meaning (do not copy-paste from sysctl -d). Don't forget to specify server type (web, smb, gateway, etc) Let's share experience!

    Read the article

  • Yum Update Failing mod_ssl and glibc_devel

    - by Kerry
    Any ideas on how to get this to not fail? # yum update Freeing read locks for locker 0x82: 4189/140342084876032 Freeing read locks for locker 0x84: 4189/140342084876032 Freeing read locks for locker 0x85: 4189/140342084876032 Freeing read locks for locker 0x86: 4189/140342084876032 Freeing read locks for locker 0x87: 4189/140342084876032 Freeing read locks for locker 0x9a: 4189/140342084876032 Freeing read locks for locker 0x9c: 4189/140342084876032 Freeing read locks for locker 0x9d: 4189/140342084876032 Freeing read locks for locker 0x9e: 4189/140342084876032 Freeing read locks for locker 0x9f: 4189/140342084876032 Freeing read locks for locker 0xa0: 4189/140342084876032 Freeing read locks for locker 0xa1: 4189/140342084876032 Freeing read locks for locker 0xa2: 4189/140342084876032 Freeing read locks for locker 0xa3: 4189/140342084876032 Freeing read locks for locker 0xa4: 4189/140342084876032 Freeing read locks for locker 0xa5: 4189/140342084876032 Freeing read locks for locker 0xa6: 4189/140342084876032 Freeing read locks for locker 0xa7: 4189/140342084876032 Freeing read locks for locker 0xa8: 4189/140342084876032 Freeing read locks for locker 0xa9: 4189/140342084876032 Freeing read locks for locker 0xaa: 4189/140342084876032 Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.hmc.edu * epel: mirrors.kernel.org * extras: centos.mirror.freedomvoice.com * updates: mirrors.sonic.net Setting up Update Process Resolving Dependencies There are unfinished transactions remaining. You might consider running yum-complete-transaction first to finish them. The program yum-complete-transaction is found in the yum-utils package. --> Running transaction check ---> Package device-mapper-persistent-data.x86_64 0:0.2.8-2.el6 will be updated ---> Package device-mapper-persistent-data.x86_64 0:0.2.8-4.el6_5 will be an update ---> Package glibc-headers.x86_64 0:2.12-1.132.el6 will be updated --> Processing Dependency: glibc-headers = 2.12-1.132.el6 for package: glibc-devel-2.12-1.132.el6.x86_64 ---> Package glibc-headers.x86_64 0:2.12-1.132.el6_5.2 will be an update ---> Package httpd.x86_64 0:2.2.15-29.el6.centos will be updated --> Processing Dependency: httpd = 2.2.15-29.el6.centos for package: 1:mod_ssl-2.2.15-29.el6.centos.x86_64 ---> Package httpd.x86_64 0:2.2.15-30.el6.centos will be an update ---> Package kernel.x86_64 0:2.6.32-431.17.1.el6 will be installed ---> Package kernel-devel.x86_64 0:2.6.32-431.17.1.el6 will be installed ---> Package selinux-policy-targeted.noarch 0:3.7.19-231.el6_5.1 will be updated ---> Package selinux-policy-targeted.noarch 0:3.7.19-231.el6_5.3 will be an update --> Finished Dependency Resolution Error: Package: 1:mod_ssl-2.2.15-29.el6.centos.x86_64 (@base) Requires: httpd = 2.2.15-29.el6.centos Removing: httpd-2.2.15-29.el6.centos.x86_64 (@base) httpd = 2.2.15-29.el6.centos Updated By: httpd-2.2.15-30.el6.centos.x86_64 (updates) httpd = 2.2.15-30.el6.centos Error: Package: glibc-devel-2.12-1.132.el6.x86_64 (@base) Requires: glibc-headers = 2.12-1.132.el6 Removing: glibc-headers-2.12-1.132.el6.x86_64 (@base) glibc-headers = 2.12-1.132.el6 Updated By: glibc-headers-2.12-1.132.el6_5.2.x86_64 (updates) glibc-headers = 2.12-1.132.el6_5.2 Available: glibc-headers-2.12-1.132.el6_5.1.x86_64 (updates) glibc-headers = 2.12-1.132.el6_5.1 You could try using --skip-broken to work around the problem ** Found 34 pre-existing rpmdb problem(s), 'yum check' output follows: audit-2.2-4.el6_5.x86_64 is a duplicate with audit-2.2-2.el6.x86_64 audit-libs-2.2-4.el6_5.x86_64 is a duplicate with audit-libs-2.2-2.el6.x86_64 curl-7.19.7-37.el6_5.3.x86_64 is a duplicate with curl-7.19.7-37.el6_4.x86_64 device-mapper-multipath-0.4.9-72.el6_5.2.x86_64 is a duplicate with device-mapper-multipath-0.4.9-72.el6_5.1.x86_64 device-mapper-multipath-libs-0.4.9-72.el6_5.2.x86_64 is a duplicate with device-mapper-multipath-libs-0.4.9-72.el6_5.1.x86_64 2:ethtool-3.5-1.4.el6_5.x86_64 is a duplicate with 2:ethtool-3.5-1.2.el6_5.x86_64 glibc-2.12-1.132.el6_5.2.x86_64 is a duplicate with glibc-2.12-1.132.el6.x86_64 glibc-common-2.12-1.132.el6_5.2.x86_64 is a duplicate with glibc-common-2.12-1.132.el6.x86_64 glibc-devel-2.12-1.132.el6_5.2.x86_64 is a duplicate with glibc-devel-2.12-1.132.el6.x86_64 glibc-devel-2.12-1.132.el6_5.2.x86_64 has missing requires of glibc-headers = ('0', '2.12', '1.132.el6_5.2') gnutls-2.8.5-14.el6_5.x86_64 is a duplicate with gnutls-2.8.5-13.el6_5.x86_64 httpd-2.2.15-29.el6.centos.x86_64 has missing requires of httpd-tools = ('0', '2.2.15', '29.el6.centos') httpd-manual-2.2.15-30.el6.centos.noarch has missing requires of httpd = ('0', '2.2.15', '30.el6.centos') iproute-2.6.32-32.el6_5.x86_64 is a duplicate with iproute-2.6.32-31.el6.x86_64 kernel-firmware-2.6.32-431.17.1.el6.noarch is a duplicate with kernel-firmware-2.6.32-431.11.2.el6.noarch kernel-headers-2.6.32-431.17.1.el6.x86_64 is a duplicate with kernel-headers-2.6.32-431.11.2.el6.x86_64 kpartx-0.4.9-72.el6_5.2.x86_64 is a duplicate with kpartx-0.4.9-72.el6_5.1.x86_64 krb5-libs-1.10.3-15.el6_5.1.x86_64 is a duplicate with krb5-libs-1.10.3-10.el6_4.6.x86_64 libblkid-2.17.2-12.14.el6_5.x86_64 is a duplicate with libblkid-2.17.2-12.14.el6.x86_64 libcurl-7.19.7-37.el6_5.3.x86_64 is a duplicate with libcurl-7.19.7-37.el6_4.x86_64 libcurl-devel-7.19.7-37.el6_5.3.x86_64 is a duplicate with libcurl-devel-7.19.7-37.el6_4.x86_64 libtasn1-2.3-6.el6_5.x86_64 is a duplicate with libtasn1-2.3-3.el6_2.1.x86_64 libuuid-2.17.2-12.14.el6_5.x86_64 is a duplicate with libuuid-2.17.2-12.14.el6.x86_64 libxml2-2.7.6-14.el6_5.1.x86_64 is a duplicate with libxml2-2.7.6-14.el6.x86_64 mdadm-3.2.6-7.el6_5.2.x86_64 is a duplicate with mdadm-3.2.6-7.el6.x86_64 1:mod_ssl-2.2.15-30.el6.centos.x86_64 is a duplicate with 1:mod_ssl-2.2.15-29.el6.centos.x86_64 1:mod_ssl-2.2.15-30.el6.centos.x86_64 has missing requires of httpd = ('0', '2.2.15', '30.el6.centos') nss-softokn-3.14.3-10.el6_5.x86_64 is a duplicate with nss-softokn-3.14.3-9.el6.x86_64 openssl-1.0.1e-16.el6_5.7.x86_64 is a duplicate with openssl-1.0.1e-16.el6_5.4.x86_64 openssl-1.0.1e-16.el6_5.14.x86_64 is a duplicate with openssl-1.0.1e-16.el6_5.7.x86_64 openssl-devel-1.0.1e-16.el6_5.14.x86_64 is a duplicate with openssl-devel-1.0.1e-16.el6_5.7.x86_64 selinux-policy-3.7.19-231.el6_5.3.noarch is a duplicate with selinux-policy-3.7.19-231.el6_5.1.noarch tzdata-2014d-1.el6.noarch is a duplicate with tzdata-2014b-1.el6.noarch util-linux-ng-2.17.2-12.14.el6_5.x86_64 is a duplicate with util-linux-ng-2.17.2-12.14.el6.x86_64 UPDATE I installed and ran yum-complete-transaction as requested, it finished some things and suggested I run package-cleanup --problems, which yielded this: package-cleanup --problems Loaded plugins: fastestmirror Package httpd-manual-2.2.15-30.el6.centos.noarch requires httpd = ('0', '2.2.15', '30.el6.centos') Package httpd-2.2.15-29.el6.centos.x86_64 requires httpd-tools = ('0', '2.2.15', '29.el6.centos') Package mod_ssl-2.2.15-30.el6.centos.x86_64 requires httpd = ('0', '2.2.15', '30.el6.centos') Package glibc-devel-2.12-1.132.el6_5.2.x86_64 requires glibc-headers = ('0', '2.12', '1.132.el6_5.2') I'm definitely not a sys-admin, what would be the next step? UPDATE 2 I ran yum distro-sync: # yum distro-sync Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.hmc.edu * epel: mirrors.kernel.org * extras: centos.mirror.freedomvoice.com * updates: mirrors.sonic.net Setting up Distribution Synchronization Process Resolving Dependencies --> Running transaction check ---> Package glibc-headers.x86_64 0:2.12-1.132.el6 will be updated --> Processing Dependency: glibc-headers = 2.12-1.132.el6 for package: glibc-devel-2.12-1.132.el6.x86_64 ---> Package glibc-headers.x86_64 0:2.12-1.132.el6_5.2 will be an update ---> Package httpd.x86_64 0:2.2.15-29.el6.centos will be updated --> Processing Dependency: httpd = 2.2.15-29.el6.centos for package: 1:mod_ssl-2.2.15-29.el6.centos.x86_64 ---> Package httpd.x86_64 0:2.2.15-30.el6.centos will be an update --> Finished Dependency Resolution Error: Package: 1:mod_ssl-2.2.15-29.el6.centos.x86_64 (@base) Requires: httpd = 2.2.15-29.el6.centos Removing: httpd-2.2.15-29.el6.centos.x86_64 (@base) httpd = 2.2.15-29.el6.centos Updated By: httpd-2.2.15-30.el6.centos.x86_64 (updates) httpd = 2.2.15-30.el6.centos Error: Package: glibc-devel-2.12-1.132.el6.x86_64 (@base) Requires: glibc-headers = 2.12-1.132.el6 Removing: glibc-headers-2.12-1.132.el6.x86_64 (@base) glibc-headers = 2.12-1.132.el6 Updated By: glibc-headers-2.12-1.132.el6_5.2.x86_64 (updates) glibc-headers = 2.12-1.132.el6_5.2 Available: glibc-headers-2.12-1.132.el6_5.1.x86_64 (updates) glibc-headers = 2.12-1.132.el6_5.1 You could try using --skip-broken to work around the problem ** Found 34 pre-existing rpmdb problem(s), 'yum check' output follows: audit-2.2-4.el6_5.x86_64 is a duplicate with audit-2.2-2.el6.x86_64 audit-libs-2.2-4.el6_5.x86_64 is a duplicate with audit-libs-2.2-2.el6.x86_64 curl-7.19.7-37.el6_5.3.x86_64 is a duplicate with curl-7.19.7-37.el6_4.x86_64 device-mapper-multipath-0.4.9-72.el6_5.2.x86_64 is a duplicate with device-mapper-multipath-0.4.9-72.el6_5.1.x86_64 device-mapper-multipath-libs-0.4.9-72.el6_5.2.x86_64 is a duplicate with device-mapper-multipath-libs-0.4.9-72.el6_5.1.x86_64 2:ethtool-3.5-1.4.el6_5.x86_64 is a duplicate with 2:ethtool-3.5-1.2.el6_5.x86_64 glibc-2.12-1.132.el6_5.2.x86_64 is a duplicate with glibc-2.12-1.132.el6.x86_64 glibc-common-2.12-1.132.el6_5.2.x86_64 is a duplicate with glibc-common-2.12-1.132.el6.x86_64 glibc-devel-2.12-1.132.el6_5.2.x86_64 is a duplicate with glibc-devel-2.12-1.132.el6.x86_64 glibc-devel-2.12-1.132.el6_5.2.x86_64 has missing requires of glibc-headers = ('0', '2.12', '1.132.el6_5.2') gnutls-2.8.5-14.el6_5.x86_64 is a duplicate with gnutls-2.8.5-13.el6_5.x86_64 httpd-2.2.15-29.el6.centos.x86_64 has missing requires of httpd-tools = ('0', '2.2.15', '29.el6.centos') httpd-manual-2.2.15-30.el6.centos.noarch has missing requires of httpd = ('0', '2.2.15', '30.el6.centos') iproute-2.6.32-32.el6_5.x86_64 is a duplicate with iproute-2.6.32-31.el6.x86_64 kernel-firmware-2.6.32-431.17.1.el6.noarch is a duplicate with kernel-firmware-2.6.32-431.11.2.el6.noarch kernel-headers-2.6.32-431.17.1.el6.x86_64 is a duplicate with kernel-headers-2.6.32-431.11.2.el6.x86_64 kpartx-0.4.9-72.el6_5.2.x86_64 is a duplicate with kpartx-0.4.9-72.el6_5.1.x86_64 krb5-libs-1.10.3-15.el6_5.1.x86_64 is a duplicate with krb5-libs-1.10.3-10.el6_4.6.x86_64 libblkid-2.17.2-12.14.el6_5.x86_64 is a duplicate with libblkid-2.17.2-12.14.el6.x86_64 libcurl-7.19.7-37.el6_5.3.x86_64 is a duplicate with libcurl-7.19.7-37.el6_4.x86_64 libcurl-devel-7.19.7-37.el6_5.3.x86_64 is a duplicate with libcurl-devel-7.19.7-37.el6_4.x86_64 libtasn1-2.3-6.el6_5.x86_64 is a duplicate with libtasn1-2.3-3.el6_2.1.x86_64 libuuid-2.17.2-12.14.el6_5.x86_64 is a duplicate with libuuid-2.17.2-12.14.el6.x86_64 libxml2-2.7.6-14.el6_5.1.x86_64 is a duplicate with libxml2-2.7.6-14.el6.x86_64 mdadm-3.2.6-7.el6_5.2.x86_64 is a duplicate with mdadm-3.2.6-7.el6.x86_64 1:mod_ssl-2.2.15-30.el6.centos.x86_64 is a duplicate with 1:mod_ssl-2.2.15-29.el6.centos.x86_64 1:mod_ssl-2.2.15-30.el6.centos.x86_64 has missing requires of httpd = ('0', '2.2.15', '30.el6.centos') nss-softokn-3.14.3-10.el6_5.x86_64 is a duplicate with nss-softokn-3.14.3-9.el6.x86_64 openssl-1.0.1e-16.el6_5.7.x86_64 is a duplicate with openssl-1.0.1e-16.el6_5.4.x86_64 openssl-1.0.1e-16.el6_5.14.x86_64 is a duplicate with openssl-1.0.1e-16.el6_5.7.x86_64 openssl-devel-1.0.1e-16.el6_5.14.x86_64 is a duplicate with openssl-devel-1.0.1e-16.el6_5.7.x86_64 selinux-policy-3.7.19-231.el6_5.3.noarch is a duplicate with selinux-policy-3.7.19-231.el6_5.1.noarch tzdata-2014d-1.el6.noarch is a duplicate with tzdata-2014b-1.el6.noarch util-linux-ng-2.17.2-12.14.el6_5.x86_64 is a duplicate with util-linux-ng-2.17.2-12.14.el6.x86_64

    Read the article

  • value types in the vm

    - by john.rose
    value types in the vm p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Times} p.p2 {margin: 0.0px 0.0px 14.0px 0.0px; font: 14.0px Times} p.p3 {margin: 0.0px 0.0px 12.0px 0.0px; font: 14.0px Times} p.p4 {margin: 0.0px 0.0px 15.0px 0.0px; font: 14.0px Times} p.p5 {margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Courier} p.p6 {margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Courier; min-height: 17.0px} p.p7 {margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Times; min-height: 18.0px} p.p8 {margin: 0.0px 0.0px 0.0px 36.0px; text-indent: -36.0px; font: 14.0px Times; min-height: 18.0px} p.p9 {margin: 0.0px 0.0px 12.0px 0.0px; font: 14.0px Times; min-height: 18.0px} p.p10 {margin: 0.0px 0.0px 12.0px 0.0px; font: 14.0px Times; color: #000000} li.li1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Times} li.li7 {margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Times; min-height: 18.0px} span.s1 {font: 14.0px Courier} span.s2 {color: #000000} span.s3 {font: 14.0px Courier; color: #000000} ol.ol1 {list-style-type: decimal} Or, enduring values for a changing world. Introduction A value type is a data type which, generally speaking, is designed for being passed by value in and out of methods, and stored by value in data structures. The only value types which the Java language directly supports are the eight primitive types. Java indirectly and approximately supports value types, if they are implemented in terms of classes. For example, both Integer and String may be viewed as value types, especially if their usage is restricted to avoid operations appropriate to Object. In this note, we propose a definition of value types in terms of a design pattern for Java classes, accompanied by a set of usage restrictions. We also sketch the relation of such value types to tuple types (which are a JVM-level notion), and point out JVM optimizations that can apply to value types. This note is a thought experiment to extend the JVM’s performance model in support of value types. The demonstration has two phases.  Initially the extension can simply use design patterns, within the current bytecode architecture, and in today’s Java language. But if the performance model is to be realized in practice, it will probably require new JVM bytecode features, changes to the Java language, or both.  We will look at a few possibilities for these new features. An Axiom of Value In the context of the JVM, a value type is a data type equipped with construction, assignment, and equality operations, and a set of typed components, such that, whenever two variables of the value type produce equal corresponding values for their components, the values of the two variables cannot be distinguished by any JVM operation. Here are some corollaries: A value type is immutable, since otherwise a copy could be constructed and the original could be modified in one of its components, allowing the copies to be distinguished. Changing the component of a value type requires construction of a new value. The equals and hashCode operations are strictly component-wise. If a value type is represented by a JVM reference, that reference cannot be successfully synchronized on, and cannot be usefully compared for reference equality. A value type can be viewed in terms of what it doesn’t do. We can say that a value type omits all value-unsafe operations, which could violate the constraints on value types.  These operations, which are ordinarily allowed for Java object types, are pointer equality comparison (the acmp instruction), synchronization (the monitor instructions), all the wait and notify methods of class Object, and non-trivial finalize methods. The clone method is also value-unsafe, although for value types it could be treated as the identity function. Finally, and most importantly, any side effect on an object (however visible) also counts as an value-unsafe operation. A value type may have methods, but such methods must not change the components of the value. It is reasonable and useful to define methods like toString, equals, and hashCode on value types, and also methods which are specifically valuable to users of the value type. Representations of Value Value types have two natural representations in the JVM, unboxed and boxed. An unboxed value consists of the components, as simple variables. For example, the complex number x=(1+2i), in rectangular coordinate form, may be represented in unboxed form by the following pair of variables: /*Complex x = Complex.valueOf(1.0, 2.0):*/ double x_re = 1.0, x_im = 2.0; These variables might be locals, parameters, or fields. Their association as components of a single value is not defined to the JVM. Here is a sample computation which computes the norm of the difference between two complex numbers: double distance(/*Complex x:*/ double x_re, double x_im,         /*Complex y:*/ double y_re, double y_im) {     /*Complex z = x.minus(y):*/     double z_re = x_re - y_re, z_im = x_im - y_im;     /*return z.abs():*/     return Math.sqrt(z_re*z_re + z_im*z_im); } A boxed representation groups component values under a single object reference. The reference is to a ‘wrapper class’ that carries the component values in its fields. (A primitive type can naturally be equated with a trivial value type with just one component of that type. In that view, the wrapper class Integer can serve as a boxed representation of value type int.) The unboxed representation of complex numbers is practical for many uses, but it fails to cover several major use cases: return values, array elements, and generic APIs. The two components of a complex number cannot be directly returned from a Java function, since Java does not support multiple return values. The same story applies to array elements: Java has no ’array of structs’ feature. (Double-length arrays are a possible workaround for complex numbers, but not for value types with heterogeneous components.) By generic APIs I mean both those which use generic types, like Arrays.asList and those which have special case support for primitive types, like String.valueOf and PrintStream.println. Those APIs do not support unboxed values, and offer some problems to boxed values. Any ’real’ JVM type should have a story for returns, arrays, and API interoperability. The basic problem here is that value types fall between primitive types and object types. Value types are clearly more complex than primitive types, and object types are slightly too complicated. Objects are a little bit dangerous to use as value carriers, since object references can be compared for pointer equality, and can be synchronized on. Also, as many Java programmers have observed, there is often a performance cost to using wrapper objects, even on modern JVMs. Even so, wrapper classes are a good starting point for talking about value types. If there were a set of structural rules and restrictions which would prevent value-unsafe operations on value types, wrapper classes would provide a good notation for defining value types. This note attempts to define such rules and restrictions. Let’s Start Coding Now it is time to look at some real code. Here is a definition, written in Java, of a complex number value type. @ValueSafe public final class Complex implements java.io.Serializable {     // immutable component structure:     public final double re, im;     private Complex(double re, double im) {         this.re = re; this.im = im;     }     // interoperability methods:     public String toString() { return "Complex("+re+","+im+")"; }     public List<Double> asList() { return Arrays.asList(re, im); }     public boolean equals(Complex c) {         return re == c.re && im == c.im;     }     public boolean equals(@ValueSafe Object x) {         return x instanceof Complex && equals((Complex) x);     }     public int hashCode() {         return 31*Double.valueOf(re).hashCode()                 + Double.valueOf(im).hashCode();     }     // factory methods:     public static Complex valueOf(double re, double im) {         return new Complex(re, im);     }     public Complex changeRe(double re2) { return valueOf(re2, im); }     public Complex changeIm(double im2) { return valueOf(re, im2); }     public static Complex cast(@ValueSafe Object x) {         return x == null ? ZERO : (Complex) x;     }     // utility methods and constants:     public Complex plus(Complex c)  { return new Complex(re+c.re, im+c.im); }     public Complex minus(Complex c) { return new Complex(re-c.re, im-c.im); }     public double abs() { return Math.sqrt(re*re + im*im); }     public static final Complex PI = valueOf(Math.PI, 0.0);     public static final Complex ZERO = valueOf(0.0, 0.0); } This is not a minimal definition, because it includes some utility methods and other optional parts.  The essential elements are as follows: The class is marked as a value type with an annotation. The class is final, because it does not make sense to create subclasses of value types. The fields of the class are all non-private and final.  (I.e., the type is immutable and structurally transparent.) From the supertype Object, all public non-final methods are overridden. The constructor is private. Beyond these bare essentials, we can observe the following features in this example, which are likely to be typical of all value types: One or more factory methods are responsible for value creation, including a component-wise valueOf method. There are utility methods for complex arithmetic and instance creation, such as plus and changeIm. There are static utility constants, such as PI. The type is serializable, using the default mechanisms. There are methods for converting to and from dynamically typed references, such as asList and cast. The Rules In order to use value types properly, the programmer must avoid value-unsafe operations.  A helpful Java compiler should issue errors (or at least warnings) for code which provably applies value-unsafe operations, and should issue warnings for code which might be correct but does not provably avoid value-unsafe operations.  No such compilers exist today, but to simplify our account here, we will pretend that they do exist. A value-safe type is any class, interface, or type parameter marked with the @ValueSafe annotation, or any subtype of a value-safe type.  If a value-safe class is marked final, it is in fact a value type.  All other value-safe classes must be abstract.  The non-static fields of a value class must be non-public and final, and all its constructors must be private. Under the above rules, a standard interface could be helpful to define value types like Complex.  Here is an example: @ValueSafe public interface ValueType extends java.io.Serializable {     // All methods listed here must get redefined.     // Definitions must be value-safe, which means     // they may depend on component values only.     List<? extends Object> asList();     int hashCode();     boolean equals(@ValueSafe Object c);     String toString(); } //@ValueSafe inherited from supertype: public final class Complex implements ValueType { … The main advantage of such a conventional interface is that (unlike an annotation) it is reified in the runtime type system.  It could appear as an element type or parameter bound, for facilities which are designed to work on value types only.  More broadly, it might assist the JVM to perform dynamic enforcement of the rules for value types. Besides types, the annotation @ValueSafe can mark fields, parameters, local variables, and methods.  (This is redundant when the type is also value-safe, but may be useful when the type is Object or another supertype of a value type.)  Working forward from these annotations, an expression E is defined as value-safe if it satisfies one or more of the following: The type of E is a value-safe type. E names a field, parameter, or local variable whose declaration is marked @ValueSafe. E is a call to a method whose declaration is marked @ValueSafe. E is an assignment to a value-safe variable, field reference, or array reference. E is a cast to a value-safe type from a value-safe expression. E is a conditional expression E0 ? E1 : E2, and both E1 and E2 are value-safe. Assignments to value-safe expressions and initializations of value-safe names must take their values from value-safe expressions. A value-safe expression may not be the subject of a value-unsafe operation.  In particular, it cannot be synchronized on, nor can it be compared with the “==” operator, not even with a null or with another value-safe type. In a program where all of these rules are followed, no value-type value will be subject to a value-unsafe operation.  Thus, the prime axiom of value types will be satisfied, that no two value type will be distinguishable as long as their component values are equal. More Code To illustrate these rules, here are some usage examples for Complex: Complex pi = Complex.valueOf(Math.PI, 0); Complex zero = pi.changeRe(0);  //zero = pi; zero.re = 0; ValueType vtype = pi; @SuppressWarnings("value-unsafe")   Object obj = pi; @ValueSafe Object obj2 = pi; obj2 = new Object();  // ok List<Complex> clist = new ArrayList<Complex>(); clist.add(pi);  // (ok assuming List.add param is @ValueSafe) List<ValueType> vlist = new ArrayList<ValueType>(); vlist.add(pi);  // (ok) List<Object> olist = new ArrayList<Object>(); olist.add(pi);  // warning: "value-unsafe" boolean z = pi.equals(zero); boolean z1 = (pi == zero);  // error: reference comparison on value type boolean z2 = (pi == null);  // error: reference comparison on value type boolean z3 = (pi == obj2);  // error: reference comparison on value type synchronized (pi) { }  // error: synch of value, unpredictable result synchronized (obj2) { }  // unpredictable result Complex qq = pi; qq = null;  // possible NPE; warning: “null-unsafe" qq = (Complex) obj;  // warning: “null-unsafe" qq = Complex.cast(obj);  // OK @SuppressWarnings("null-unsafe")   Complex empty = null;  // possible NPE qq = empty;  // possible NPE (null pollution) The Payoffs It follows from this that either the JVM or the java compiler can replace boxed value-type values with unboxed ones, without affecting normal computations.  Fields and variables of value types can be split into their unboxed components.  Non-static methods on value types can be transformed into static methods which take the components as value parameters. Some common questions arise around this point in any discussion of value types. Why burden the programmer with all these extra rules?  Why not detect programs automagically and perform unboxing transparently?  The answer is that it is easy to break the rules accidently unless they are agreed to by the programmer and enforced.  Automatic unboxing optimizations are tantalizing but (so far) unreachable ideal.  In the current state of the art, it is possible exhibit benchmarks in which automatic unboxing provides the desired effects, but it is not possible to provide a JVM with a performance model that assures the programmer when unboxing will occur.  This is why I’m writing this note, to enlist help from, and provide assurances to, the programmer.  Basically, I’m shooting for a good set of user-supplied “pragmas” to frame the desired optimization. Again, the important thing is that the unboxing must be done reliably, or else programmers will have no reason to work with the extra complexity of the value-safety rules.  There must be a reasonably stable performance model, wherein using a value type has approximately the same performance characteristics as writing the unboxed components as separate Java variables. There are some rough corners to the present scheme.  Since Java fields and array elements are initialized to null, value-type computations which incorporate uninitialized variables can produce null pointer exceptions.  One workaround for this is to require such variables to be null-tested, and the result replaced with a suitable all-zero value of the value type.  That is what the “cast” method does above. Generically typed APIs like List<T> will continue to manipulate boxed values always, at least until we figure out how to do reification of generic type instances.  Use of such APIs will elicit warnings until their type parameters (and/or relevant members) are annotated or typed as value-safe.  Retrofitting List<T> is likely to expose flaws in the present scheme, which we will need to engineer around.  Here are a couple of first approaches: public interface java.util.List<@ValueSafe T> extends Collection<T> { … public interface java.util.List<T extends Object|ValueType> extends Collection<T> { … (The second approach would require disjunctive types, in which value-safety is “contagious” from the constituent types.) With more transformations, the return value types of methods can also be unboxed.  This may require significant bytecode-level transformations, and would work best in the presence of a bytecode representation for multiple value groups, which I have proposed elsewhere under the title “Tuples in the VM”. But for starters, the JVM can apply this transformation under the covers, to internally compiled methods.  This would give a way to express multiple return values and structured return values, which is a significant pain-point for Java programmers, especially those who work with low-level structure types favored by modern vector and graphics processors.  The lack of multiple return values has a strong distorting effect on many Java APIs. Even if the JVM fails to unbox a value, there is still potential benefit to the value type.  Clustered computing systems something have copy operations (serialization or something similar) which apply implicitly to command operands.  When copying JVM objects, it is extremely helpful to know when an object’s identity is important or not.  If an object reference is a copied operand, the system may have to create a proxy handle which points back to the original object, so that side effects are visible.  Proxies must be managed carefully, and this can be expensive.  On the other hand, value types are exactly those types which a JVM can “copy and forget” with no downside. Array types are crucial to bulk data interfaces.  (As data sizes and rates increase, bulk data becomes more important than scalar data, so arrays are definitely accompanying us into the future of computing.)  Value types are very helpful for adding structure to bulk data, so a successful value type mechanism will make it easier for us to express richer forms of bulk data. Unboxing arrays (i.e., arrays containing unboxed values) will provide better cache and memory density, and more direct data movement within clustered or heterogeneous computing systems.  They require the deepest transformations, relative to today’s JVM.  There is an impedance mismatch between value-type arrays and Java’s covariant array typing, so compromises will need to be struck with existing Java semantics.  It is probably worth the effort, since arrays of unboxed value types are inherently more memory-efficient than standard Java arrays, which rely on dependent pointer chains. It may be sufficient to extend the “value-safe” concept to array declarations, and allow low-level transformations to change value-safe array declarations from the standard boxed form into an unboxed tuple-based form.  Such value-safe arrays would not be convertible to Object[] arrays.  Certain connection points, such as Arrays.copyOf and System.arraycopy might need additional input/output combinations, to allow smooth conversion between arrays with boxed and unboxed elements. Alternatively, the correct solution may have to wait until we have enough reification of generic types, and enough operator overloading, to enable an overhaul of Java arrays. Implicit Method Definitions The example of class Complex above may be unattractively complex.  I believe most or all of the elements of the example class are required by the logic of value types. If this is true, a programmer who writes a value type will have to write lots of error-prone boilerplate code.  On the other hand, I think nearly all of the code (except for the domain-specific parts like plus and minus) can be implicitly generated. Java has a rule for implicitly defining a class’s constructor, if no it defines no constructors explicitly.  Likewise, there are rules for providing default access modifiers for interface members.  Because of the highly regular structure of value types, it might be reasonable to perform similar implicit transformations on value types.  Here’s an example of a “highly implicit” definition of a complex number type: public class Complex implements ValueType {  // implicitly final     public double re, im;  // implicitly public final     //implicit methods are defined elementwise from te fields:     //  toString, asList, equals(2), hashCode, valueOf, cast     //optionally, explicit methods (plus, abs, etc.) would go here } In other words, with the right defaults, a simple value type definition can be a one-liner.  The observant reader will have noticed the similarities (and suitable differences) between the explicit methods above and the corresponding methods for List<T>. Another way to abbreviate such a class would be to make an annotation the primary trigger of the functionality, and to add the interface(s) implicitly: public @ValueType class Complex { … // implicitly final, implements ValueType (But to me it seems better to communicate the “magic” via an interface, even if it is rooted in an annotation.) Implicitly Defined Value Types So far we have been working with nominal value types, which is to say that the sequence of typed components is associated with a name and additional methods that convey the intention of the programmer.  A simple ordered pair of floating point numbers can be variously interpreted as (to name a few possibilities) a rectangular or polar complex number or Cartesian point.  The name and the methods convey the intended meaning. But what if we need a truly simple ordered pair of floating point numbers, without any further conceptual baggage?  Perhaps we are writing a method (like “divideAndRemainder”) which naturally returns a pair of numbers instead of a single number.  Wrapping the pair of numbers in a nominal type (like “QuotientAndRemainder”) makes as little sense as wrapping a single return value in a nominal type (like “Quotient”).  What we need here are structural value types commonly known as tuples. For the present discussion, let us assign a conventional, JVM-friendly name to tuples, roughly as follows: public class java.lang.tuple.$DD extends java.lang.tuple.Tuple {      double $1, $2; } Here the component names are fixed and all the required methods are defined implicitly.  The supertype is an abstract class which has suitable shared declarations.  The name itself mentions a JVM-style method parameter descriptor, which may be “cracked” to determine the number and types of the component fields. The odd thing about such a tuple type (and structural types in general) is it must be instantiated lazily, in response to linkage requests from one or more classes that need it.  The JVM and/or its class loaders must be prepared to spin a tuple type on demand, given a simple name reference, $xyz, where the xyz is cracked into a series of component types.  (Specifics of naming and name mangling need some tasteful engineering.) Tuples also seem to demand, even more than nominal types, some support from the language.  (This is probably because notations for non-nominal types work best as combinations of punctuation and type names, rather than named constructors like Function3 or Tuple2.)  At a minimum, languages with tuples usually (I think) have some sort of simple bracket notation for creating tuples, and a corresponding pattern-matching syntax (or “destructuring bind”) for taking tuples apart, at least when they are parameter lists.  Designing such a syntax is no simple thing, because it ought to play well with nominal value types, and also with pre-existing Java features, such as method parameter lists, implicit conversions, generic types, and reflection.  That is a task for another day. Other Use Cases Besides complex numbers and simple tuples there are many use cases for value types.  Many tuple-like types have natural value-type representations. These include rational numbers, point locations and pixel colors, and various kinds of dates and addresses. Other types have a variable-length ‘tail’ of internal values. The most common example of this is String, which is (mathematically) a sequence of UTF-16 character values. Similarly, bit vectors, multiple-precision numbers, and polynomials are composed of sequences of values. Such types include, in their representation, a reference to a variable-sized data structure (often an array) which (somehow) represents the sequence of values. The value type may also include ’header’ information. Variable-sized values often have a length distribution which favors short lengths. In that case, the design of the value type can make the first few values in the sequence be direct ’header’ fields of the value type. In the common case where the header is enough to represent the whole value, the tail can be a shared null value, or even just a null reference. Note that the tail need not be an immutable object, as long as the header type encapsulates it well enough. This is the case with String, where the tail is a mutable (but never mutated) character array. Field types and their order must be a globally visible part of the API.  The structure of the value type must be transparent enough to have a globally consistent unboxed representation, so that all callers and callees agree about the type and order of components  that appear as parameters, return types, and array elements.  This is a trade-off between efficiency and encapsulation, which is forced on us when we remove an indirection enjoyed by boxed representations.  A JVM-only transformation would not care about such visibility, but a bytecode transformation would need to take care that (say) the components of complex numbers would not get swapped after a redefinition of Complex and a partial recompile.  Perhaps constant pool references to value types need to declare the field order as assumed by each API user. This brings up the delicate status of private fields in a value type.  It must always be possible to load, store, and copy value types as coordinated groups, and the JVM performs those movements by moving individual scalar values between locals and stack.  If a component field is not public, what is to prevent hostile code from plucking it out of the tuple using a rogue aload or astore instruction?  Nothing but the verifier, so we may need to give it more smarts, so that it treats value types as inseparable groups of stack slots or locals (something like long or double). My initial thought was to make the fields always public, which would make the security problem moot.  But public is not always the right answer; consider the case of String, where the underlying mutable character array must be encapsulated to prevent security holes.  I believe we can win back both sides of the tradeoff, by training the verifier never to split up the components in an unboxed value.  Just as the verifier encapsulates the two halves of a 64-bit primitive, it can encapsulate the the header and body of an unboxed String, so that no code other than that of class String itself can take apart the values. Similar to String, we could build an efficient multi-precision decimal type along these lines: public final class DecimalValue extends ValueType {     protected final long header;     protected private final BigInteger digits;     public DecimalValue valueOf(int value, int scale) {         assert(scale >= 0);         return new DecimalValue(((long)value << 32) + scale, null);     }     public DecimalValue valueOf(long value, int scale) {         if (value == (int) value)             return valueOf((int)value, scale);         return new DecimalValue(-scale, new BigInteger(value));     } } Values of this type would be passed between methods as two machine words. Small values (those with a significand which fits into 32 bits) would be represented without any heap data at all, unless the DecimalValue itself were boxed. (Note the tension between encapsulation and unboxing in this case.  It would be better if the header and digits fields were private, but depending on where the unboxing information must “leak”, it is probably safer to make a public revelation of the internal structure.) Note that, although an array of Complex can be faked with a double-length array of double, there is no easy way to fake an array of unboxed DecimalValues.  (Either an array of boxed values or a transposed pair of homogeneous arrays would be reasonable fallbacks, in a current JVM.)  Getting the full benefit of unboxing and arrays will require some new JVM magic. Although the JVM emphasizes portability, system dependent code will benefit from using machine-level types larger than 64 bits.  For example, the back end of a linear algebra package might benefit from value types like Float4 which map to stock vector types.  This is probably only worthwhile if the unboxing arrays can be packed with such values. More Daydreams A more finely-divided design for dynamic enforcement of value safety could feature separate marker interfaces for each invariant.  An empty marker interface Unsynchronizable could cause suitable exceptions for monitor instructions on objects in marked classes.  More radically, a Interchangeable marker interface could cause JVM primitives that are sensitive to object identity to raise exceptions; the strangest result would be that the acmp instruction would have to be specified as raising an exception. @ValueSafe public interface ValueType extends java.io.Serializable,         Unsynchronizable, Interchangeable { … public class Complex implements ValueType {     // inherits Serializable, Unsynchronizable, Interchangeable, @ValueSafe     … It seems possible that Integer and the other wrapper types could be retro-fitted as value-safe types.  This is a major change, since wrapper objects would be unsynchronizable and their references interchangeable.  It is likely that code which violates value-safety for wrapper types exists but is uncommon.  It is less plausible to retro-fit String, since the prominent operation String.intern is often used with value-unsafe code. We should also reconsider the distinction between boxed and unboxed values in code.  The design presented above obscures that distinction.  As another thought experiment, we could imagine making a first class distinction in the type system between boxed and unboxed representations.  Since only primitive types are named with a lower-case initial letter, we could define that the capitalized version of a value type name always refers to the boxed representation, while the initial lower-case variant always refers to boxed.  For example: complex pi = complex.valueOf(Math.PI, 0); Complex boxPi = pi;  // convert to boxed myList.add(boxPi); complex z = myList.get(0);  // unbox Such a convention could perhaps absorb the current difference between int and Integer, double and Double. It might also allow the programmer to express a helpful distinction among array types. As said above, array types are crucial to bulk data interfaces, but are limited in the JVM.  Extending arrays beyond the present limitations is worth thinking about; for example, the Maxine JVM implementation has a hybrid object/array type.  Something like this which can also accommodate value type components seems worthwhile.  On the other hand, does it make sense for value types to contain short arrays?  And why should random-access arrays be the end of our design process, when bulk data is often sequentially accessed, and it might make sense to have heterogeneous streams of data as the natural “jumbo” data structure.  These considerations must wait for another day and another note. More Work It seems to me that a good sequence for introducing such value types would be as follows: Add the value-safety restrictions to an experimental version of javac. Code some sample applications with value types, including Complex and DecimalValue. Create an experimental JVM which internally unboxes value types but does not require new bytecodes to do so.  Ensure the feasibility of the performance model for the sample applications. Add tuple-like bytecodes (with or without generic type reification) to a major revision of the JVM, and teach the Java compiler to switch in the new bytecodes without code changes. A staggered roll-out like this would decouple language changes from bytecode changes, which is always a convenient thing. A similar investigation should be applied (concurrently) to array types.  In this case, it seems to me that the starting point is in the JVM: Add an experimental unboxing array data structure to a production JVM, perhaps along the lines of Maxine hybrids.  No bytecode or language support is required at first; everything can be done with encapsulated unsafe operations and/or method handles. Create an experimental JVM which internally unboxes value types but does not require new bytecodes to do so.  Ensure the feasibility of the performance model for the sample applications. Add tuple-like bytecodes (with or without generic type reification) to a major revision of the JVM, and teach the Java compiler to switch in the new bytecodes without code changes. That’s enough musing me for now.  Back to work!

    Read the article

  • Implementing an async "read all currently available data from stream" operation

    - by Jon
    I recently provided an answer to this question: C# - Realtime console output redirection. As often happens, explaining stuff (here "stuff" was how I tackled a similar problem) leads you to greater understanding and/or, as is the case here, "oops" moments. I realized that my solution, as implemented, has a bug. The bug has little practical importance, but it has an extremely large importance to me as a developer: I can't rest easy knowing that my code has the potential to blow up. Squashing the bug is the purpose of this question. I apologize for the long intro, so let's get dirty. I wanted to build a class that allows me to receive input from a console's standard output Stream. Console output streams are of type FileStream; the implementation can cast to that, if needed. There is also an associated StreamReader already present to leverage. There is only one thing I need to implement in this class to achieve my desired functionality: an async "read all the data available this moment" operation. Reading to the end of the stream is not viable because the stream will not end unless the process closes the console output handle, and it will not do that because it is interactive and expecting input before continuing. I will be using that hypothetical async operation to implement event-based notification, which will be more convenient for my callers. The public interface of the class is this: public class ConsoleAutomator { public event EventHandler<ConsoleOutputReadEventArgs> StandardOutputRead; public void StartSendingEvents(); public void StopSendingEvents(); } StartSendingEvents and StopSendingEvents do what they advertise; for the purposes of this discussion, we can assume that events are always being sent without loss of generality. The class uses these two fields internally: protected readonly StringBuilder inputAccumulator = new StringBuilder(); protected readonly byte[] buffer = new byte[256]; The functionality of the class is implemented in the methods below. To get the ball rolling: public void StartSendingEvents(); { this.stopAutomation = false; this.BeginReadAsync(); } To read data out of the Stream without blocking, and also without requiring a carriage return char, BeginRead is called: protected void BeginReadAsync() { if (!this.stopAutomation) { this.StandardOutput.BaseStream.BeginRead( this.buffer, 0, this.buffer.Length, this.ReadHappened, null); } } The challenging part: BeginRead requires using a buffer. This means that when reading from the stream, it is possible that the bytes available to read ("incoming chunk") are larger than the buffer. Remember that the goal here is to read all of the chunk and call event subscribers exactly once for each chunk. To this end, if the buffer is full after EndRead, we don't send its contents to subscribers immediately but instead append them to a StringBuilder. The contents of the StringBuilder are only sent back whenever there is no more to read from the stream. private void ReadHappened(IAsyncResult asyncResult) { var bytesRead = this.StandardOutput.BaseStream.EndRead(asyncResult); if (bytesRead == 0) { this.OnAutomationStopped(); return; } var input = this.StandardOutput.CurrentEncoding.GetString( this.buffer, 0, bytesRead); this.inputAccumulator.Append(input); if (bytesRead < this.buffer.Length) { this.OnInputRead(); // only send back if we 're sure we got it all } this.BeginReadAsync(); // continue "looping" with BeginRead } After any read which is not enough to fill the buffer (in which case we know that there was no more data to be read during the last read operation), all accumulated data is sent to the subscribers: private void OnInputRead() { var handler = this.StandardOutputRead; if (handler == null) { return; } handler(this, new ConsoleOutputReadEventArgs(this.inputAccumulator.ToString())); this.inputAccumulator.Clear(); } (I know that as long as there are no subscribers the data gets accumulated forever. This is a deliberate decision). The good This scheme works almost perfectly: Async functionality without spawning any threads Very convenient to the calling code (just subscribe to an event) Never more than one event for each time data is available to be read Is almost agnostic to the buffer size The bad That last almost is a very big one. Consider what happens when there is an incoming chunk with length exactly equal to the size of the buffer. The chunk will be read and buffered, but the event will not be triggered. This will be followed up by a BeginRead that expects to find more data belonging to the current chunk in order to send it back all in one piece, but... there will be no more data in the stream. In fact, as long as data is put into the stream in chunks with length exactly equal to the buffer size, the data will be buffered and the event will never be triggered. This scenario may be highly unlikely to occur in practice, especially since we can pick any number for the buffer size, but the problem is there. Solution? Unfortunately, after checking the available methods on FileStream and StreamReader, I can't find anything which lets me peek into the stream while also allowing async methods to be used on it. One "solution" would be to have a thread wait on a ManualResetEvent after the "buffer filled" condition is detected. If the event is not signaled (by the async callback) in a small amount of time, then more data from the stream will not be forthcoming and the data accumulated so far should be sent to subscribers. However, this introduces the need for another thread, requires thread synchronization, and is plain inelegant. Specifying a timeout for BeginRead would also suffice (call back into my code every now and then so I can check if there's data to be sent back; most of the time there will not be anything to do, so I expect the performance hit to be negligible). But it looks like timeouts are not supported in FileStream. Since I imagine that async calls with timeouts are an option in bare Win32, another approach might be to PInvoke the hell out of the problem. But this is also undesirable as it will introduce complexity and simply be a pain to code. Is there an elegant way to get around the problem? Thanks for being patient enough to read all of this. Update: I definitely did not communicate the scenario well in my initial writeup. I have since revised the writeup quite a bit, but to be extra sure: The question is about how to implement an async "read all the data available this moment" operation. My apologies to the people who took the time to read and answer without me making my intent clear enough.

    Read the article

  • Implementing a robust async stream reader

    - by Jon
    I recently provided an answer to this question: C# - Realtime console output redirection. As often happens, explaining stuff (here "stuff" was how I tackled a similar problem) leads you to greater understanding and/or, as is the case here, "oops" moments. I realized that my solution, as implemented, has a bug. The bug has little practical importance, but it has an extremely large importance to me as a developer: I can't rest easy knowing that my code has the potential to blow up. Squashing the bug is the purpose of this question. I apologize for the long intro, so let's get dirty. I wanted to build a class that allows me to receive input from a Stream in an event-based manner. The stream, in my scenario, is guaranteed to be a FileStream and there is also an associated StreamReader already present to leverage. The public interface of the class is this: public class MyStreamManager { public event EventHandler<ConsoleOutputReadEventArgs> StandardOutputRead; public void StartSendingEvents(); public void StopSendingEvents(); } Obviously this specific scenario has to do with a console's standard output, but that is a detail and does not play an important role. StartSendingEvents and StopSendingEvents do what they advertise; for the purposes of this discussion, we can assume that events are always being sent without loss of generality. The class uses these two fields internally: protected readonly StringBuilder inputAccumulator = new StringBuilder(); protected readonly byte[] buffer = new byte[256]; The functionality of the class is implemented in the methods below. To get the ball rolling: public void StartSendingEvents(); { this.stopAutomation = false; this.BeginReadAsync(); } To read data out of the Stream without blocking, and also without requiring a carriage return char, BeginRead is called: protected void BeginReadAsync() { if (!this.stopAutomation) { this.StandardOutput.BaseStream.BeginRead( this.buffer, 0, this.buffer.Length, this.ReadHappened, null); } } The challenging part: BeginRead requires using a buffer. This means that when reading from the stream, it is possible that the bytes available to read ("incoming chunk") are larger than the buffer. Since we are only handing off data from the stream to a consumer, and that consumer may well have inside knowledge about the size and/or format of these chunks, I want to call event subscribers exactly once for each chunk. Otherwise the abstraction breaks down and the subscribers have to buffer the incoming data and reconstruct the chunks themselves using said knowledge. This is much less convenient to the calling code, and detracts from the usefulness of my class. To this end, if the buffer is full after EndRead, we don't send its contents to subscribers immediately but instead append them to a StringBuilder. The contents of the StringBuilder are only sent back whenever there is no more to read from the stream (thus preserving the chunks). private void ReadHappened(IAsyncResult asyncResult) { var bytesRead = this.StandardOutput.BaseStream.EndRead(asyncResult); if (bytesRead == 0) { this.OnAutomationStopped(); return; } var input = this.StandardOutput.CurrentEncoding.GetString( this.buffer, 0, bytesRead); this.inputAccumulator.Append(input); if (bytesRead < this.buffer.Length) { this.OnInputRead(); // only send back if we 're sure we got it all } this.BeginReadAsync(); // continue "looping" with BeginRead } After any read which is not enough to fill the buffer, all accumulated data is sent to the subscribers: private void OnInputRead() { var handler = this.StandardOutputRead; if (handler == null) { return; } handler(this, new ConsoleOutputReadEventArgs(this.inputAccumulator.ToString())); this.inputAccumulator.Clear(); } (I know that as long as there are no subscribers the data gets accumulated forever. This is a deliberate decision). The good This scheme works almost perfectly: Async functionality without spawning any threads Very convenient to the calling code (just subscribe to an event) Maintains the "chunkiness" of the data; this allows the calling code to use inside knowledge of the data without doing any extra work Is almost agnostic to the buffer size (it will work correctly with any size buffer irrespective of the data being read) The bad That last almost is a very big one. Consider what happens when there is an incoming chunk with length exactly equal to the size of the buffer. The chunk will be read and buffered, but the event will not be triggered. This will be followed up by a BeginRead that expects to find more data belonging to the current chunk in order to send it back all in one piece, but... there will be no more data in the stream. In fact, as long as data is put into the stream in chunks with length exactly equal to the buffer size, the data will be buffered and the event will never be triggered. This scenario may be highly unlikely to occur in practice, especially since we can pick any number for the buffer size, but the problem is there. Solution? Unfortunately, after checking the available methods on FileStream and StreamReader, I can't find anything which lets me peek into the stream while also allowing async methods to be used on it. One "solution" would be to have a thread wait on a ManualResetEvent after the "buffer filled" condition is detected. If the event is not signaled (by the async callback) in a small amount of time, then more data from the stream will not be forthcoming and the data accumulated so far should be sent to subscribers. However, this introduces the need for another thread, requires thread synchronization, and is plain inelegant. Specifying a timeout for BeginRead would also suffice (call back into my code every now and then so I can check if there's data to be sent back; most of the time there will not be anything to do, so I expect the performance hit to be negligible). But it looks like timeouts are not supported in FileStream. Since I imagine that async calls with timeouts are an option in bare Win32, another approach might be to PInvoke the hell out of the problem. But this is also undesirable as it will introduce complexity and simply be a pain to code. Is there an elegant way to get around the problem? Thanks for being patient enough to read all of this.

    Read the article

  • Implementing a robust async stream reader for a console

    - by Jon
    I recently provided an answer to this question: C# - Realtime console output redirection. As often happens, explaining stuff (here "stuff" was how I tackled a similar problem) leads you to greater understanding and/or, as is the case here, "oops" moments. I realized that my solution, as implemented, has a bug. The bug has little practical importance, but it has an extremely large importance to me as a developer: I can't rest easy knowing that my code has the potential to blow up. Squashing the bug is the purpose of this question. I apologize for the long intro, so let's get dirty. I wanted to build a class that allows me to receive input from a Stream in an event-based manner. The stream, in my scenario, is guaranteed to be a FileStream and there is also an associated StreamReader already present to leverage. The public interface of the class is this: public class MyStreamManager { public event EventHandler<ConsoleOutputReadEventArgs> StandardOutputRead; public void StartSendingEvents(); public void StopSendingEvents(); } Obviously this specific scenario has to do with a console's standard output. StartSendingEvents and StopSendingEvents do what they advertise; for the purposes of this discussion, we can assume that events are always being sent without loss of generality. The class uses these two fields internally: protected readonly StringBuilder inputAccumulator = new StringBuilder(); protected readonly byte[] buffer = new byte[256]; The functionality of the class is implemented in the methods below. To get the ball rolling: public void StartSendingEvents(); { this.stopAutomation = false; this.BeginReadAsync(); } To read data out of the Stream without blocking, and also without requiring a carriage return char, BeginRead is called: protected void BeginReadAsync() { if (!this.stopAutomation) { this.StandardOutput.BaseStream.BeginRead( this.buffer, 0, this.buffer.Length, this.ReadHappened, null); } } The challenging part: BeginRead requires using a buffer. This means that when reading from the stream, it is possible that the bytes available to read ("incoming chunk") are larger than the buffer. Since we are only handing off data from the stream to a consumer, and that consumer may well have inside knowledge about the size and/or format of these chunks, I want to call event subscribers exactly once for each chunk. Otherwise the abstraction breaks down and the subscribers have to buffer the incoming data and reconstruct the chunks themselves using said knowledge. This is much less convenient to the calling code, and detracts from the usefulness of my class. Edit: There are comments below correctly stating that since the data is coming from a stream, there is absolutely nothing that the receiver can infer about the structure of the data unless it is fully prepared to parse it. What I am trying to do here is leverage the "flush the output" "structure" that the owner of the console imparts while writing on it. I am prepared to assume (better: allow my caller to have the option to assume) that the OS will pass me the data written between two flushes of the stream in exactly one piece. To this end, if the buffer is full after EndRead, we don't send its contents to subscribers immediately but instead append them to a StringBuilder. The contents of the StringBuilder are only sent back whenever there is no more to read from the stream (thus preserving the chunks). private void ReadHappened(IAsyncResult asyncResult) { var bytesRead = this.StandardOutput.BaseStream.EndRead(asyncResult); if (bytesRead == 0) { this.OnAutomationStopped(); return; } var input = this.StandardOutput.CurrentEncoding.GetString( this.buffer, 0, bytesRead); this.inputAccumulator.Append(input); if (bytesRead < this.buffer.Length) { this.OnInputRead(); // only send back if we 're sure we got it all } this.BeginReadAsync(); // continue "looping" with BeginRead } After any read which is not enough to fill the buffer, all accumulated data is sent to the subscribers: private void OnInputRead() { var handler = this.StandardOutputRead; if (handler == null) { return; } handler(this, new ConsoleOutputReadEventArgs(this.inputAccumulator.ToString())); this.inputAccumulator.Clear(); } (I know that as long as there are no subscribers the data gets accumulated forever. This is a deliberate decision). The good This scheme works almost perfectly: Async functionality without spawning any threads Very convenient to the calling code (just subscribe to an event) Maintains the "chunkiness" of the data; this allows the calling code to use inside knowledge of the data without doing any extra work Is almost agnostic to the buffer size (it will work correctly with any size buffer irrespective of the data being read) The bad That last almost is a very big one. Consider what happens when there is an incoming chunk with length exactly equal to the size of the buffer. The chunk will be read and buffered, but the event will not be triggered. This will be followed up by a BeginRead that expects to find more data belonging to the current chunk in order to send it back all in one piece, but... there will be no more data in the stream. In fact, as long as data is put into the stream in chunks with length exactly equal to the buffer size, the data will be buffered and the event will never be triggered. This scenario may be highly unlikely to occur in practice, especially since we can pick any number for the buffer size, but the problem is there. Solution? Unfortunately, after checking the available methods on FileStream and StreamReader, I can't find anything which lets me peek into the stream while also allowing async methods to be used on it. One "solution" would be to have a thread wait on a ManualResetEvent after the "buffer filled" condition is detected. If the event is not signaled (by the async callback) in a small amount of time, then more data from the stream will not be forthcoming and the data accumulated so far should be sent to subscribers. However, this introduces the need for another thread, requires thread synchronization, and is plain inelegant. Specifying a timeout for BeginRead would also suffice (call back into my code every now and then so I can check if there's data to be sent back; most of the time there will not be anything to do, so I expect the performance hit to be negligible). But it looks like timeouts are not supported in FileStream. Since I imagine that async calls with timeouts are an option in bare Win32, another approach might be to PInvoke the hell out of the problem. But this is also undesirable as it will introduce complexity and simply be a pain to code. Is there an elegant way to get around the problem? Thanks for being patient enough to read all of this.

    Read the article

  • Producer and Consumer Threads Hang

    - by user972425
    So this is my first foray into threads and thus far it is driving me insane. My problem seems to be some kind of synchronization error that causes my consumer thread to hang. I've looked at other code and just about everything I could find and I can't find what my error is. There also seems to be a discrepancy between the code being executed in Eclipse and via javac in the command line. Intention - Using a bounded buffer (with 1000 slots) create and consume 1,000,000 doubles. Use only notify and wait. Problem - In Eclipse the consumer thread will occasionally hang around 940,000 iterations, but other times completes. In the command line the consumer thread always hangs. Output - Eclipse - Successful Producer has produced 100000 doubles. Consumer has consumed 100000 doubles. Producer has produced 200000 doubles. Consumer has consumed 200000 doubles. Producer has produced 300000 doubles. Consumer has consumed 300000 doubles. Producer has produced 400000 doubles. Consumer has consumed 400000 doubles. Producer has produced 500000 doubles. Consumer has consumed 500000 doubles. Producer has produced 600000 doubles. Consumer has consumed 600000 doubles. Producer has produced 700000 doubles. Consumer has consumed 700000 doubles. Producer has produced 800000 doubles. Consumer has consumed 800000 doubles. Producer has produced 900000 doubles. Consumer has consumed 900000 doubles. Producer has produced 1000000 doubles. Producer has produced all items. Consumer has consumed 1000000 doubles. Consumer has consumed all items. Exitting Output - Command Line/Eclipse - Unsuccessful Producer has produced 100000 doubles. Consumer has consumed 100000 doubles. Producer has produced 200000 doubles. Consumer has consumed 200000 doubles. Producer has produced 300000 doubles. Consumer has consumed 300000 doubles. Producer has produced 400000 doubles. Consumer has consumed 400000 doubles. Producer has produced 500000 doubles. Consumer has consumed 500000 doubles. Producer has produced 600000 doubles. Consumer has consumed 600000 doubles. Producer has produced 700000 doubles. Consumer has consumed 700000 doubles. Producer has produced 800000 doubles. Consumer has consumed 800000 doubles. Producer has produced 900000 doubles. Consumer has consumed 900000 doubles. Producer has produced 1000000 doubles. Producer has produced all items. At this point it just sits and hangs. Any help you can provide about where I might have misstepped is greatly appreciated. Code - Producer thread import java.text.DecimalFormat;+ " doubles. Cumulative value of generated items= " + temp) import java.util.*; import java.io.*; public class producer implements Runnable{ private buffer produceBuff; public producer (buffer buff){ produceBuff = buff; } public void run(){ Random random = new Random(); double temp = 0, randomElem; DecimalFormat df = new DecimalFormat("#.###"); for(int i = 1; i<=1000000; i++) { randomElem = (Double.parseDouble( df.format(random.nextDouble() * 100.0))); try { produceBuff.add(randomElem); } catch (InterruptedException e) { // TODO Auto-generated catch block e.printStackTrace(); } temp+= randomElem; if(i%100000 == 0) {produceBuff.print("Producer has produced "+ i ); } } produceBuff.print("Producer has produced all items."); } } Consumer thread import java.util.*; import java.io.*; public class consumer implements Runnable{ private buffer consumBuff; public consumer (buffer buff){ consumBuff = buff; } public void run(){ double temp = 0; for(int i = 1; i<=1000000; i++) { try { temp += consumBuff.get(); } catch (InterruptedException e) { // TODO Auto-generated catch block e.printStackTrace(); } if(i%100000 == 0) {consumBuff.print("Consumer has consumed "+ i ); //if(i>999000) //{System.out.println("Consuming item " + i);} } consumBuff.print("Consumer has consumed all items."); } } Buffer/Main import java.util.*; import java.io.*; public class buffer { private double buff[]; private int addPlace; private int getPlace; public buffer(){ buff = new double[1000]; addPlace = 0; getPlace = 0; } public synchronized void add(double add) throws InterruptedException{ if((addPlace+1 == getPlace) ) { try { wait(); } catch (InterruptedException e) {throw e;} } buff[addPlace] = add; addPlace = (addPlace+1)%1000; notify(); } public synchronized double get()throws InterruptedException{ if(getPlace == addPlace) { try { wait(); } catch (InterruptedException e) {throw e;} } double temp = buff[getPlace]; getPlace = (getPlace+1)%1000; notify(); return temp; } public synchronized void print(String view) { System.out.println(view); } public static void main(String args[]){ buffer buf = new buffer(); Thread produce = new Thread(new producer(buf)); Thread consume = new Thread(new consumer(buf)); produce.start(); consume.start(); try { produce.join(); consume.join(); } catch (InterruptedException e) {return;} System.out.println("Exitting"); } }

    Read the article

  • How to synchronize cuda threads when they are in the same loop and we need to synchronize them to ex

    - by Vickey
    Hi all, I have written a code and Now I want to implement this on cuda GPU but I'm new to synchronization so please help me with this, It's little urgent to me. Below I'm presenting the code and I want to that LOOP1 to be executed by all threads (heance I want to this portion to take advantage of cuda and the remaining portion (the portion other from the LOOP1) is to be executed by only a single thread. do{ point_set = master_Q[(*num_mas) - 1].q; List* temp = point_set; List* pa = point_set; if(master_Q[num_mas[0] - 1].max) max_level = (int) (ceilf(il2 * log(master_Q[num_mas[0] - 1].max))); *num_mas = (*num_mas) - 1; while(point_set){ List* insert_ele = temp; while(temp){ insert_ele = temp; if((insert_ele->dist[insert_ele->dist_index-1] <= pow(2, max_level-1)) || (top_level == max_level)){ if(point_set == temp){ point_set = temp->next; pa = temp->next; } else{ pa->next = temp->next; } temp = NULL; List* new_point_set = point_set; float maximum_dist = 0; if(parent->p_index != insert_ele->point_index){ List* tmp = new_point_set; float *b = &(data[(insert_ele->point_index)*point_len]); **LOOP 1:** while(tmp){ float *c = &(data[(tmp->point_index)*point_len]); float sum = 0.; for(int j = 0; j < point_len; j+=2){ float d1 = b[j] - c[j]; float d2 = b[j+1] - c[j+1]; d1 *= d1; d2 *= d2; sum = sum + d1 + d2; } tmp->dist[tmp->dist_index] = sqrt(sum); if(maximum_dist < tmp->dist[tmp->dist_index]) maximum_dist = tmp->dist[tmp->dist_index]; tmp->dist_index = tmp->dist_index+1; tmp = tmp->next; } max_distance = maximum_dist; } while(new_point_set || insert_ele){ List* far, *par, *tmp, *tmp_new; far = NULL; tmp = new_point_set; tmp_new = NULL; float level_dist = pow(2, max_level-1); float maxdist = 0, maxp = 0; while(tmp){ if(tmp->dist[(tmp->dist_index)-1] > level_dist){ if(maxdist < tmp->dist[tmp->dist_index-1]) maxdist = tmp->dist[tmp->dist_index-1]; if(tmp == new_point_set){ new_point_set = tmp->next; par = tmp->next; } else{ par->next = tmp->next; } if(far == NULL){ far = tmp; tmp_new = far; } else{ tmp_new->next = tmp; tmp_new = tmp; } if(parent->p_index != insert_ele->point_index) tmp->dist_index = tmp->dist_index - 1; tmp = tmp->next; tmp_new->next = NULL; } else{ par = tmp; if(maxp < tmp->dist[(tmp->dist_index)-1]) maxp = tmp->dist[(tmp->dist_index)-1]; tmp = tmp->next; } } if(0 == maxp){ tmp = new_point_set; aloc_mem[*tree_index].p_index = insert_ele->point_index; aloc_mem[*tree_index].no_child = 0; aloc_mem[*tree_index].level = max_level--; parent->children_index[parent->no_child++] = *tree_index; parent = &(aloc_mem[*tree_index]); tree_index[0] = tree_index[0]+1; while(tmp){ aloc_mem[*tree_index].p_index = tmp->point_index; aloc_mem[(*tree_index)].no_child = 0; aloc_mem[(*tree_index)].level = master_Q[(*cur_count_Q)-1].level; parent->children_index[parent->no_child] = *tree_index; parent->no_child = parent->no_child + 1; (*tree_index)++; tmp = tmp->next; } cur_count_Q[0] = cur_count_Q[0]-1; new_point_set = NULL; } master_Q[*num_mas].q = far; master_Q[*num_mas].parent = parent; master_Q[*num_mas].valid = true; master_Q[*num_mas].max = maxdist; master_Q[*num_mas].level = max_level; num_mas[0] = num_mas[0]+1; if(0 != maxp){ aloc_mem[*tree_index].p_index = insert_ele->point_index; aloc_mem[*tree_index].no_child = 0; aloc_mem[*tree_index].level = max_level; parent->children_index[parent->no_child++] = *tree_index; parent = &(aloc_mem[*tree_index]); tree_index[0] = tree_index[0]+1; if(maxp){ int new_level = ((int) (ceilf(il2 * log(maxp)))) +1; if (new_level < (max_level-1)) max_level = new_level; else max_level--; } else max_level--; } if( 0 == maxp ) insert_ele = NULL; } } else{ if(NULL == temp->next){ master_Q[*num_mas].q = point_set; master_Q[*num_mas].parent = parent; master_Q[*num_mas].valid = true; master_Q[*num_mas].level = max_level; num_mas[0] = num_mas[0]+1; } pa = temp; temp = temp->next; } } if((*num_mas) > 1){ List *temp2 = master_Q[(*num_mas)-1].q; while(temp2){ List* temp3 = master_Q[(*num_mas)-2].q; master_Q[(*num_mas)-2].q = temp2; if((master_Q[(*num_mas)-1].parent)->p_index != (master_Q[(*num_mas)-2].parent)->p_index){ temp2->dist_index = temp2->dist_index - 1; } temp2 = temp2->next; master_Q[(*num_mas)-2].q->next = temp3; } num_mas[0] = num_mas[0]-1; } point_set = master_Q[(*num_mas)-1].q; temp = point_set; pa = point_set; parent = master_Q[(*num_mas)-1].parent; max_level = master_Q[(*num_mas)-1].level; if(master_Q[(*num_mas)-1].max) if( max_level > ((int) (ceilf(il2 * log(master_Q[(*num_mas)-1].max)))) +1) max_level = ((int) (ceilf(il2 * log(master_Q[(*num_mas)-1].max)))) +1; num_mas[0] = num_mas[0]-1; } }while(*num_mas > 0);

    Read the article

  • FreeBSD performance tuning. Sysctls, loader.conf, kernel

    - by SaveTheRbtz
    I wanted to share knowledge of tuning FreeBSD via sysctl.conf/loader.conf/KENCONF. It was initially based on Igor Sysoev's (author of nginx) presentation about FreeBSD tuning up to 100,000-200,000 active connections. Tunings are for FreeBSD-CURRENT. Since 7.2 amd64 some of them are tuned well by default. Prior 7.0 some of them are boot only (set via /boot/loader.conf) or does not exist at all. sysctl.conf: # No zero mapping feature # May break wine # (There are also reports about broken samba3) #security.bsd.map_at_zero=0 # If you have really busy webserver with apache13 you may run out of processes #kern.maxproc=10000 # Same for servers with apache2 / Pound #kern.threads.max_threads_per_proc=4096 # Max. backlog size kern.ipc.somaxconn=4096 # Shared memory // 7.2+ can use shared memory > 2Gb kern.ipc.shmmax=2147483648 # Sockets kern.ipc.maxsockets=204800 # Can cause this on older kernels: # http://old.nabble.com/Significant-performance-regression-for-increased-maxsockbuf-on-8.0-RELEASE-tt26745981.html#a26745981 ) kern.ipc.maxsockbuf=10485760 # Mbuf 2k clusters (on amd64 7.2+ 25600 is default) # For such high value vm.kmem_size must be increased to 3G kern.ipc.nmbclusters=262144 # Jumbo pagesize(_SC_PAGESIZE) clusters # Used as general packet storage for jumbo frames # can be monitored via `netstat -m` #kern.ipc.nmbjumbop=262144 # Jumbo 9k/16k clusters # If you are using them #kern.ipc.nmbjumbo9=65536 #kern.ipc.nmbjumbo16=32768 # For lower latency you can decrease scheduler's maximum time slice # default: stathz/10 (~ 13) #kern.sched.slice=1 # Increase max command-line length showed in `ps` (e.g for Tomcat/Java) # Default is PAGE_SIZE / 16 or 256 on x86 # This avoids commands to be presented as [executable] in `ps` # For more info see: http://www.freebsd.org/cgi/query-pr.cgi?pr=120749 kern.ps_arg_cache_limit=4096 # Every socket is a file, so increase them kern.maxfiles=204800 kern.maxfilesperproc=200000 kern.maxvnodes=200000 # On some systems HPET is almost 2 times faster than default ACPI-fast # Useful on systems with lots of clock_gettime / gettimeofday calls # See http://old.nabble.com/ACPI-fast-default-timecounter,-but-HPET-83--faster-td23248172.html # After revision 222222 HPET became default: http://svnweb.freebsd.org/base?view=revision&revision=222222 kern.timecounter.hardware=HPET # Small receive space, only usable on http-server, on file server this # should be increased to 65535 or even more #net.inet.tcp.recvspace=8192 # This is useful on Fat-Long-Pipes #net.inet.tcp.recvbuf_max=10485760 #net.inet.tcp.recvbuf_inc=65535 # Small send space is useful for http servers that serve small files # Autotuned since 7.x net.inet.tcp.sendspace=16384 # This is useful on Fat-Long-Pipes #net.inet.tcp.sendbuf_max=10485760 #net.inet.tcp.sendbuf_inc=65535 # Turn off receive autotuning # You can play with it. #net.inet.tcp.recvbuf_auto=0 #net.inet.tcp.sendbuf_auto=0 # This should be enabled if you going to use big spaces (>64k) # Also timestamp field is useful when using syncookies net.inet.tcp.rfc1323=1 # Turn this off on high-speed, lossless connections (LAN 1Gbit+) # If you set it there is no need in TCP_NODELAY sockopt (see man tcp) net.inet.tcp.delayed_ack=0 # This feature is useful if you are serving data over modems, Gigabit Ethernet, # or even high speed WAN links (or any other link with a high bandwidth delay product), # especially if you are also using window scaling or have configured a large send window. # Automatically disables on small RTT ( http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/netinet/tcp_subr.c?#rev1.237 ) # This sysctl was removed in 10-CURRENT: # See: http://www.mail-archive.com/[email protected]/msg06178.html #net.inet.tcp.inflight.enable=0 # TCP slowstart algorithm tunings # We assuming we have very fast clients #net.inet.tcp.slowstart_flightsize=100 #net.inet.tcp.local_slowstart_flightsize=100 # Disable randomizing of ports to avoid false RST # Before usage check SA here www.bsdcan.org/2006/papers/ImprovingTCPIP.pdf # (it's also says that port randomization auto-disables at some conn.rates, but I didn't checked it thou) #net.inet.ip.portrange.randomized=0 # Increase portrange # For outgoing connections only. Good for seed-boxes and ftp servers. net.inet.ip.portrange.first=1024 net.inet.ip.portrange.last=65535 # # stops route cache degregation during a high-bandwidth flood # http://www.freebsd.org/doc/en/books/handbook/securing-freebsd.html #net.inet.ip.rtexpire=2 net.inet.ip.rtminexpire=2 net.inet.ip.rtmaxcache=1024 # Security net.inet.ip.redirect=0 net.inet.ip.sourceroute=0 net.inet.ip.accept_sourceroute=0 net.inet.icmp.maskrepl=0 net.inet.icmp.log_redirect=0 net.inet.icmp.drop_redirect=1 net.inet.tcp.drop_synfin=1 # # There is also good example of sysctl.conf with comments: # http://www.thern.org/projects/sysctl.conf # # icmp may NOT rst, helpful for those pesky spoofed # icmp/udp floods that end up taking up your outgoing # bandwidth/ifqueue due to all that outgoing RST traffic. # #net.inet.tcp.icmp_may_rst=0 # Security net.inet.udp.blackhole=1 net.inet.tcp.blackhole=2 # IPv6 Security # For more info see http://www.fosslc.org/drupal/content/security-implications-ipv6 # Disable Node info replies # To see this vulnerability in action run `ping6 -a sglAac ::1` or `ping6 -w ::1` on unprotected node net.inet6.icmp6.nodeinfo=0 # Turn on IPv6 privacy extensions # For more info see proposal http://unix.derkeiler.com/Mailing-Lists/FreeBSD/net/2008-06/msg00103.html net.inet6.ip6.use_tempaddr=1 net.inet6.ip6.prefer_tempaddr=1 # Disable ICMP redirect net.inet6.icmp6.rediraccept=0 # Disable acceptation of RA and auto linklocal generation if you don't use them #net.inet6.ip6.accept_rtadv=0 #net.inet6.ip6.auto_linklocal=0 # Increases default TTL, sometimes useful # Default is 64 net.inet.ip.ttl=128 # Lessen max segment life to conserve resources # ACK waiting time in miliseconds # (default: 30000. RFC from 1979 recommends 120000) net.inet.tcp.msl=5000 # Max bumber of timewait sockets net.inet.tcp.maxtcptw=200000 # Don't use tw on local connections # As of 15 Apr 2009. Igor Sysoev says that nolocaltimewait has some buggy realization. # So disable it or now till get fixed #net.inet.tcp.nolocaltimewait=1 # FIN_WAIT_2 state fast recycle net.inet.tcp.fast_finwait2_recycle=1 # Time before tcp keepalive probe is sent # default is 2 hours (7200000) #net.inet.tcp.keepidle=60000 # Should be increased until net.inet.ip.intr_queue_drops is zero net.inet.ip.intr_queue_maxlen=4096 # Interrupt handling via multiple CPU, but with context switch. # You can play with it. Default is 1; #net.isr.direct=0 # This is for routers only #net.inet.ip.forwarding=1 #net.inet.ip.fastforwarding=1 # This speed ups dummynet when channel isn't saturated net.inet.ip.dummynet.io_fast=1 # Increase dummynet(4) hash #net.inet.ip.dummynet.hash_size=2048 #net.inet.ip.dummynet.max_chain_len # Should be increased when you have A LOT of files on server # (Increase until vfs.ufs.dirhash_mem becomes lower) vfs.ufs.dirhash_maxmem=67108864 # Note from commit http://svn.freebsd.org/base/head@211031 : # For systems with RAID volumes and/or virtualization envirnments, where # read performance is very important, increasing this sysctl tunable to 32 # or even more will demonstratively yield additional performance benefits. vfs.read_max=32 # Explicit Congestion Notification (see http://en.wikipedia.org/wiki/Explicit_Congestion_Notification) net.inet.tcp.ecn.enable=1 # Flowtable - flow caching mechanism # Useful for routers #net.inet.flowtable.enable=1 #net.inet.flowtable.nmbflows=65535 # Extreme polling tuning #kern.polling.burst_max=1000 #kern.polling.each_burst=1000 #kern.polling.reg_frac=100 #kern.polling.user_frac=1 #kern.polling.idle_poll=0 # IPFW dynamic rules and timeouts tuning # Increase dyn_buckets till net.inet.ip.fw.curr_dyn_buckets is lower net.inet.ip.fw.dyn_buckets=65536 net.inet.ip.fw.dyn_max=65536 net.inet.ip.fw.dyn_ack_lifetime=120 net.inet.ip.fw.dyn_syn_lifetime=10 net.inet.ip.fw.dyn_fin_lifetime=2 net.inet.ip.fw.dyn_short_lifetime=10 # Make packets pass firewall only once when using dummynet # i.e. packets going thru pipe are passing out from firewall with accept #net.inet.ip.fw.one_pass=1 # shm_use_phys Wires all shared pages, making them unswappable # Use this to lessen Virtual Memory Manager's work when using Shared Mem. # Useful for databases #kern.ipc.shm_use_phys=1 # ZFS # Enable prefetch. Useful for sequential load type i.e fileserver. # FreeBSD sets vfs.zfs.prefetch_disable to 1 on any i386 systems and # on any amd64 systems with less than 4GB of avaiable memory # For additional info check this nabble thread http://old.nabble.com/Samba-read-speed-performance-tuning-td27964534.html #vfs.zfs.prefetch_disable=0 # On highload servers you may notice following message in dmesg: # "Approaching the limit on PV entries, consider increasing either the # vm.pmap.shpgperproc or the vm.pmap.pv_entry_max tunable" vm.pmap.shpgperproc=2048 loader.conf: # Accept filters for data, http and DNS requests # Useful when your software uses select() instead of kevent/kqueue or when you under DDoS # DNS accf available on 8.0+ accf_data_load="YES" accf_http_load="YES" accf_dns_load="YES" # Async IO system calls aio_load="YES" # Linux specific devices in /dev # As for 8.1 it only /dev/full #lindev_load="YES" # Adds NCQ support in FreeBSD # WARNING! all ad[0-9]+ devices will be renamed to ada[0-9]+ # 8.0+ only #ahci_load="YES" #siis_load="YES" # FreeBSD 8.2+ # New Congestion Control for FreeBSD # http://caia.swin.edu.au/urp/newtcp/tools/cc_chd-readme-0.1.txt # http://www.ietf.org/proceedings/78/slides/iccrg-5.pdf # Initial merge commit message http://www.mail-archive.com/[email protected]/msg31410.html #cc_chd_load="YES" # Increase kernel memory size to 3G. # # Use ONLY if you have KVA_PAGES in kernel configuration, and you have more than 3G RAM # Otherwise panic will happen on next reboot! # # It's required for high buffer sizes: kern.ipc.nmbjumbop, kern.ipc.nmbclusters, etc # Useful on highload stateful firewalls, proxies or ZFS fileservers # (FreeBSD 7.2+ amd64 users: Check that current value is lower!) #vm.kmem_size="3G" # If your server has lots of swap (>4Gb) you should increase following value # according to http://lists.freebsd.org/pipermail/freebsd-hackers/2009-October/029616.html # Otherwise you'll be getting errors # "kernel: swap zone exhausted, increase kern.maxswzone" # kern.maxswzone="256M" # Older versions of FreeBSD can't tune maxfiles on the fly #kern.maxfiles="200000" # Useful for databases # Sets maximum data size to 1G # (FreeBSD 7.2+ amd64 users: Check that current value is lower!) #kern.maxdsiz="1G" # Maximum buffer size(vfs.maxbufspace) # You can check current one via vfs.bufspace # Should be lowered/upped depending on server's load-type # Usually decreased to preserve kmem # (default is 10% of mem) #kern.maxbcache="512M" # Sendfile buffers # For i386 only #kern.ipc.nsfbufs=10240 # FreeBSD 9+ # HPET "legacy route" support. It should allow HPET to work per-CPU # See http://www.mail-archive.com/[email protected]/msg03603.html #hint.atrtc.0.clock=0 #hint.attimer.0.clock=0 #hint.hpet.0.legacy_route=1 # syncache Hash table tuning net.inet.tcp.syncache.hashsize=1024 net.inet.tcp.syncache.bucketlimit=512 net.inet.tcp.syncache.cachelimit=65536 # Increased hostcache # Later host cache can be viewed via net.inet.tcp.hostcache.list hidden sysctl # Very useful for it's RTT RTTVAR # Must be power of two net.inet.tcp.hostcache.hashsize=65536 # hashsize * bucketlimit (which is 30 by default) # It allocates 255Mb (1966080*136) of RAM net.inet.tcp.hostcache.cachelimit=1966080 # TCP control-block Hash table tuning net.inet.tcp.tcbhashsize=4096 # Disable ipfw deny all # Should be uncommented when there is a chance that # kernel and ipfw binary may be out-of sync on next reboot #net.inet.ip.fw.default_to_accept=1 # # SIFTR (Statistical Information For TCP Research) is a kernel module that # logs a range of statistics on active TCP connections to a log file. # See prerelease notes http://groups.google.com/group/mailing.freebsd.current/browse_thread/thread/b4c18be6cdce76e4 # and man 4 sitfr #siftr_load="YES" # Enable superpages, for 7.2+ only # Also read http://lists.freebsd.org/pipermail/freebsd-hackers/2009-November/030094.html vm.pmap.pg_ps_enabled=1 # Usefull if you are using Intel-Gigabit NIC #hw.em.rxd=4096 #hw.em.txd=4096 #hw.em.rx_process_limit="-1" # Also if you have ALOT interrupts on NIC - play with following parameters # NOTE: You should set them for every NIC #dev.em.0.rx_int_delay: 250 #dev.em.0.tx_int_delay: 250 #dev.em.0.rx_abs_int_delay: 250 #dev.em.0.tx_abs_int_delay: 250 # There is also multithreaded version of em/igb drivers can be found here: # http://people.yandex-team.ru/~wawa/ # # for additional em monitoring and statistics use # sysctl dev.em.0.stats=1 ; dmesg # sysctl dev.em.0.debug=1 ; dmesg # Also after r209242 (-CURRENT) there is a separate sysctl for each stat variable; # Same tunings for igb #hw.igb.rxd=4096 #hw.igb.txd=4096 #hw.igb.rx_process_limit=100 # Some useful netisr tunables. See sysctl net.isr #net.isr.maxthreads=4 #net.isr.defaultqlimit=4096 #net.isr.maxqlimit: 10240 # Bind netisr threads to CPUs #net.isr.bindthreads=1 # # FreeBSD 9.x+ # Increase interface send queue length # See commit message http://svn.freebsd.org/viewvc/base?view=revision&revision=207554 #net.link.ifqmaxlen=1024 # Nicer boot logo =) loader_logo="beastie" And finally here is KERNCONF: # Just some of them, see also # cat /sys/{i386,amd64,}/conf/NOTES # This one useful only on i386 #options KVA_PAGES=512 # You can play with HZ in environments with high interrupt rate (default is 1000) # 100 is for my notebook to prolong it's battery life #options HZ=100 # Polling is goot on network loads with high packet rates and low-end NICs # NB! Do not enable it if you want more than one netisr thread #options DEVICE_POLLING # Eliminate datacopy on socket read-write # To take advantage with zero copy sockets you should have an MTU >= 4k # This req. is only for receiving data. # Read more in man zero_copy_sockets # Also this epic thread on kernel trap: # http://kerneltrap.org/node/6506 # Here Linus says that "anybody that does it that way (FreeBSD) is totally incompetent" #options ZERO_COPY_SOCKETS # Support TCP sign. Used for IPSec options TCP_SIGNATURE # There was stackoverflow found in KAME IPSec stack: # See http://secunia.com/advisories/43995/ # For quick workaround you can use `ipfw add deny proto ipcomp` options IPSEC # This ones can be loaded as modules. They described in loader.conf section #options ACCEPT_FILTER_DATA #options ACCEPT_FILTER_HTTP # Adding ipfw, also can be loaded as modules options IPFIREWALL # On 8.1+ you can disable verbose to see blocked packets on ipfw0 interface. # Also there is no point in compiling verbose into the kernel, because # now there is net.inet.ip.fw.verbose tunable. #options IPFIREWALL_VERBOSE #options IPFIREWALL_VERBOSE_LIMIT=10 options IPFIREWALL_FORWARD # Adding kernel NAT options IPFIREWALL_NAT options LIBALIAS # Traffic shaping options DUMMYNET # Divert, i.e. for userspace NAT options IPDIVERT # This is for OpenBSD's pf firewall device pf device pflog # pf's QoS - ALTQ options ALTQ options ALTQ_CBQ # Class Bases Queuing (CBQ) options ALTQ_RED # Random Early Detection (RED) options ALTQ_RIO # RED In/Out options ALTQ_HFSC # Hierarchical Packet Scheduler (HFSC) options ALTQ_PRIQ # Priority Queuing (PRIQ) options ALTQ_NOPCC # Required for SMP build # Pretty console # Manual can be found here http://forums.freebsd.org/showthread.php?t=6134 #options VESA #options SC_PIXEL_MODE # Disable reboot on Ctrl Alt Del #options SC_DISABLE_REBOOT # Change normal|kernel messages color options SC_NORM_ATTR=(FG_GREEN|BG_BLACK) options SC_KERNEL_CONS_ATTR=(FG_YELLOW|BG_BLACK) # More scroll space options SC_HISTORY_SIZE=8192 # Adding hardware crypto device device crypto device cryptodev # Useful network interfaces device vlan device tap #Virtual Ethernet driver device gre #IP over IP tunneling device if_bridge #Bridge interface device pfsync #synchronization interface for PF device carp #Common Address Redundancy Protocol device enc #IPsec interface device lagg #Link aggregation interface device stf #IPv4-IPv6 port # Also for my notebook, but may be used with Opteron device amdtemp # Same for Intel processors device coretemp # man 4 cpuctl device cpuctl # CPU control pseudo-device # Support for ECMP. More than one route for destination # Works even with default route so one can use it as LB for two ISP # For now code is unstable and panics (panic: rtfree 2) on route deletions. #options RADIX_MPATH # Multicast routing #options MROUTING #options PIM # Debug & DTrace options KDB # Kernel debugger related code options KDB_TRACE # Print a stack trace for a panic options KDTRACE_FRAME # amd64-only(?) options KDTRACE_HOOKS # all architectures - enable general DTrace hooks #options DDB #options DDB_CTF # all architectures - kernel ELF linker loads CTF data # Adaptive spining in lockmgr (8.x+) # See http://www.mail-archive.com/[email protected]/msg10782.html options ADAPTIVE_LOCKMGRS # UTF-8 in console (8.x+) #options TEKEN_UTF8 # FreeBSD 8.1+ # Deadlock resolver thread # For additional information see http://www.mail-archive.com/[email protected]/msg18124.html # (FYI: "resolution" is panic so use with caution) #options DEADLKRES # Increase maximum size of Raw I/O and sendfile(2) readahead #options MAXPHYS=(1024*1024) #options MAXBSIZE=(1024*1024) # For scheduler debug enable following option. # Debug will be available via `kern.sched.stats` sysctl # For more information see http://svnweb.freebsd.org/base/head/sys/conf/NOTES?view=markup #options SCHED_STATS If you are tuning network for maximum performance you may wish to play with ifconfig options like: # You can list all capabilities via `ifconfig -m` ifconfig [-]rxcsum [-]txcsum [-]tso [-]lro mtu In case you've enabled DDB in kernel config, you should edit your /etc/ddb.conf and add something like this to enable automatic reboot (and textdump as bonus): script kdb.enter.panic=textdump set; capture on; show pcpu; bt; ps; alltrace; capture off; call doadump; reset script kdb.enter.default=textdump set; capture on; bt; ps; capture off; call doadump; reset And do not forget to add ddb_enable="YES" to /etc/rc.conf Since FreeBSD 9 you can select to enable/disable flowcontrol on your NIC: # See http://en.wikipedia.org/wiki/Ethernet_flow_control and # http://www.mail-archive.com/[email protected]/msg07927.html for additional info ifconfig bge0 media auto mediaopt flowcontrol PS. Also most of FreeBSD's limits can be monitored by # vmstat -z and # limits PPS. variety of network counters can be monitored via # netstat -s In FreeBSD-9 netstat's -Q option appeared, try following command to display netisr stats # netstat -Q PPPS. also see # man 7 tuning PPPPS. I wanted to thank FreeBSD community, especially author of nginx - Igor Sysoev, nginx-ru@ and FreeBSD-performance@ mailing lists for providing useful information about FreeBSD tuning. FreeBSD WIP * Whats cooking for FreeBSD 7? * Whats cooking for FreeBSD 8? * Whats cooking for FreeBSD 9? So here is the question: What tunings are you using on yours FreeBSD servers? You can also post your /etc/sysctl.conf, /boot/loader.conf, kernel options, etc with description of its' meaning (do not copy-paste from sysctl -d). Don't forget to specify server type (web, smb, gateway, etc) Let's share experience!

    Read the article

  • Process is killed without a (obvious) reason and program stops working

    - by Krzysiek Gurniak
    Here's what my program is supposed to do: create 4 child processes: process 0 is reading 1 byte at a time from STDIN, then writing it into FIFO process 1 is reading this 1 byte from fifo and write its value as HEX into shared memory process 2 is reading HEX value from shared memory and writing it into pipe finally process 3 is reading from pipe and writing into STDOUT (in my case: terminal) I can't change communication channels. FIFO, then shared memory, then pipes are the only option. My problem: Program stops at random moments when some file is directed into stdin (for example:./program < /dev/urandom). Sometimes after writing 5 HEX values, sometimes after 100. Weird thing is that when it is working and in another terminal I write "pstree -c" there is 1 main process with 4 children processes (which is what I want), but when I write "pstree -c" after it stopped writing (but still runs) there are only 3 child processes. For some reason 1 is gone even though they all have while(1) in them.. I think I might have problem with synchronization here, but I am unable to spot it (I've tried for many hours). Here's the code: #include <unistd.h> #include <fcntl.h> #include <stdio.h> #include <string.h> #include <stdlib.h> #include <sys/shm.h> #include <sys/sem.h> #include <sys/types.h> #include <sys/wait.h> #include <sys/stat.h> #include <string.h> #include <signal.h> #define BUFSIZE 1 #define R 0 #define W 1 // processes ID pid_t p0, p1, p2, p3; // FIFO variables int fifo_fd; unsigned char bufor[BUFSIZE] = {}; unsigned char bufor1[BUFSIZE] = {}; // Shared memory variables key_t key; int shmid; char * tab; // zmienne do pipes int file_des[2]; char bufor_pipe[BUFSIZE*30] = {}; void proces0() { ssize_t n; while(1) { fifo_fd = open("/tmp/fifo",O_WRONLY); if(fifo_fd == -1) { perror("blad przy otwieraniu kolejki FIFO w p0\n"); exit(1); } n = read(STDIN_FILENO, bufor, BUFSIZE); if(n<0) { perror("read error w p0\n"); exit(1); } if(n > 0) { if(write(fifo_fd, bufor, n) != n) { perror("blad zapisu do kolejki fifo w p0\n"); exit(1); } memset(bufor, 0, n); // czyszczenie bufora } close(fifo_fd); } } void proces1() { ssize_t m, x; char wartosc_hex[30] = {}; while(1) { if(tab[0] == 0) { fifo_fd = open("/tmp/fifo", O_RDONLY); // otwiera plik typu fifo do odczytu if(fifo_fd == -1) { perror("blad przy otwieraniu kolejki FIFO w p1\n"); exit(1); } m = read(fifo_fd, bufor1, BUFSIZE); x = m; if(x < 0) { perror("read error p1\n"); exit(1); } if(x > 0) { // Konwersja na HEX if(bufor1[0] < 16) { if(bufor1[0] == 10) // gdy enter { sprintf(wartosc_hex, "0x0%X\n", bufor1[0]); } else { sprintf(wartosc_hex, "0x0%X ", bufor1[0]); } } else { sprintf(wartosc_hex, "0x%X ", bufor1[0]); } // poczekaj az pamiec bedzie pusta (gotowa do zapisu) strcpy(&tab[0], wartosc_hex); memset(bufor1, 0, sizeof(bufor1)); // czyszczenie bufora memset(wartosc_hex, 0, sizeof(wartosc_hex)); // przygotowanie tablicy na zapis wartosci hex x = 0; } close(fifo_fd); } } } void proces2() { close(file_des[0]); // zablokuj kanal do odczytu while(1) { if(tab[0] != 0) { if(write(file_des[1], tab, strlen(tab)) != strlen(tab)) { perror("blad write w p2"); exit(1); } // wyczysc pamiec dzielona by przyjac kolejny bajt memset(tab, 0, sizeof(tab)); } } } void proces3() { ssize_t n; close(file_des[1]); // zablokuj kanal do zapisu while(1) { if(tab[0] == 0) { if((n = read(file_des[0], bufor_pipe, sizeof(bufor_pipe))) > 0) { if(write(STDOUT_FILENO, bufor_pipe, n) != n) { perror("write error w proces3()"); exit(1); } memset(bufor_pipe, 0, sizeof(bufor_pipe)); } } } } int main(void) { key = 5678; int status; // Tworzenie plikow przechowujacych ID procesow int des_pid[2] = {}; char bufor_proces[50] = {}; mknod("pid0", S_IFREG | 0777, 0); mknod("pid1", S_IFREG | 0777, 0); mknod("pid2", S_IFREG | 0777, 0); mknod("pid3", S_IFREG | 0777, 0); // Tworzenie semaforow key_t klucz; klucz = ftok(".", 'a'); // na podstawie pliku i pojedynczego znaku id wyznacza klucz semafora if(klucz == -1) { perror("blad wyznaczania klucza semafora"); exit(1); } semafor = semget(klucz, 1, IPC_CREAT | 0777); // tworzy na podstawie klucza semafor. 1 - ilosc semaforow if(semafor == -1) { perror("blad przy tworzeniu semafora"); exit(1); } if(semctl(semafor, 0, SETVAL, 0) == -1) // ustawia poczatkowa wartosc semafora (klucz, numer w zbiorze od 0, polecenie, argument 0/1/2) { perror("blad przy ustawianiu wartosci poczatkowej semafora"); exit(1); } // Tworzenie lacza nazwanego FIFO if(access("/tmp/fifo", F_OK) == -1) // sprawdza czy plik istnieje, jesli nie - tworzy go { if(mkfifo("/tmp/fifo", 0777) != 0) { perror("blad tworzenia FIFO w main"); exit(1); } } // Tworzenie pamieci dzielonej // Lista pamieci wspoldzielonych, komenda "ipcs" // usuwanie pamieci wspoldzielonej, komenta "ipcrm -m ID_PAMIECI" shmid = shmget(key, (BUFSIZE*30), 0666 | IPC_CREAT); if(shmid == -1) { perror("shmget"); exit(1); } tab = (char *) shmat(shmid, NULL, 0); if(tab == (char *)(-1)) { perror("shmat"); exit(1); } memset(tab, 0, (BUFSIZE*30)); // Tworzenie lacza nienazwanego pipe if(pipe(file_des) == -1) { perror("pipe"); exit(1); } // Tworzenie procesow potomnych if(!(p0 = fork())) { des_pid[W] = open("pid0", O_WRONLY | O_TRUNC | O_CREAT); // 1 - zapis, 0 - odczyt sprintf(bufor_proces, "Proces0 ma ID: %d\n", getpid()); if(write(des_pid[W], bufor_proces, sizeof(bufor_proces)) != sizeof(bufor_proces)) { perror("blad przy zapisie pid do pliku w p0"); exit(1); } close(des_pid[W]); proces0(); } else if(p0 == -1) { perror("blad przy p0 fork w main"); exit(1); } else { if(!(p1 = fork())) { des_pid[W] = open("pid1", O_WRONLY | O_TRUNC | O_CREAT); // 1 - zapis, 0 - odczyt sprintf(bufor_proces, "Proces1 ma ID: %d\n", getpid()); if(write(des_pid[W], bufor_proces, sizeof(bufor_proces)) != sizeof(bufor_proces)) { perror("blad przy zapisie pid do pliku w p1"); exit(1); } close(des_pid[W]); proces1(); } else if(p1 == -1) { perror("blad przy p1 fork w main"); exit(1); } else { if(!(p2 = fork())) { des_pid[W] = open("pid2", O_WRONLY | O_TRUNC | O_CREAT); // 1 - zapis, 0 - odczyt sprintf(bufor_proces, "Proces2 ma ID: %d\n", getpid()); if(write(des_pid[W], bufor_proces, sizeof(bufor_proces)) != sizeof(bufor_proces)) { perror("blad przy zapisie pid do pliku w p2"); exit(1); } close(des_pid[W]); proces2(); } else if(p2 == -1) { perror("blad przy p2 fork w main"); exit(1); } else { if(!(p3 = fork())) { des_pid[W] = open("pid3", O_WRONLY | O_TRUNC | O_CREAT); // 1 - zapis, 0 - odczyt sprintf(bufor_proces, "Proces3 ma ID: %d\n", getpid()); if(write(des_pid[W], bufor_proces, sizeof(bufor_proces)) != sizeof(bufor_proces)) { perror("blad przy zapisie pid do pliku w p3"); exit(1); } close(des_pid[W]); proces3(); } else if(p3 == -1) { perror("blad przy p3 fork w main"); exit(1); } else { // proces macierzysty waitpid(p0, &status, 0); waitpid(p1, &status, 0); waitpid(p2, &status, 0); waitpid(p3, &status, 0); //wait(NULL); unlink("/tmp/fifo"); shmdt(tab); // odlaczenie pamieci dzielonej shmctl(shmid, IPC_RMID, NULL); // usuwanie pamieci wspoldzielonej printf("\nKONIEC PROGRAMU\n"); } } } } exit(0); }

    Read the article

< Previous Page | 42 43 44 45 46