Search Results

Search found 9279 results on 372 pages for 'job queue'.

Page 73/372 | < Previous Page | 69 70 71 72 73 74 75 76 77 78 79 80  | Next Page >

• Working with Queues in SQL Server

  This article shows sow to create a queue from a SQL table to serve each operator exactly one item from the queue that has not been picked up by any other.

  Read the article

• Properly Configured Rsyslog on CentOS

  - by Gaia

  I'm trying to configure Rsyslog 5.8.10 on CentOS 6.4 to send Apache's error and access logs to a remote server. It's working, but I have a couple questions. A) I would like to use as few queues (and resources) as possible. I send error logs to server A, send access logs to server A, send both logs in one stream to server B. Should I specify one queue per external service (2 queues) or one queue per stream (3 queues, as I have now)? This is what I have: $ActionResumeInterval 10 $ActionQueueSize 100000 $ActionQueueDiscardMark 97500 $ActionQueueHighWaterMark 80000 $ActionQueueType LinkedList $ActionQueueFileName logglyaccessqueue $ActionQueueCheckpointInterval 100 $ActionQueueMaxDiskSpace 1g $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on $ActionQueueTimeoutEnqueue 10 $ActionQueueDiscardSeverity 0 if $syslogtag startswith 'www-access' then @@logs-01.loggly.com:514;logglyaccess $ActionResumeInterval 10 $ActionQueueSize 100000 $ActionQueueDiscardMark 97500 $ActionQueueHighWaterMark 80000 $ActionQueueType LinkedList $ActionQueueFileName logglyerrorsqueue $ActionQueueCheckpointInterval 100 $ActionQueueMaxDiskSpace 1g $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on $ActionQueueTimeoutEnqueue 10 $ActionQueueDiscardSeverity 0 if $syslogtag startswith 'www-errors' then @@logs-01.loggly.com:514;logglyerrors $DefaultNetstreamDriverCAFile /etc/syslog.papertrail.crt # trust these CAs $ActionSendStreamDriver gtls # use gtls netstream driver $ActionSendStreamDriverMode 1 # require TLS $ActionSendStreamDriverAuthMode x509/name # authenticate by hostname $ActionResumeInterval 10 $ActionQueueSize 100000 $ActionQueueDiscardMark 97500 $ActionQueueHighWaterMark 80000 $ActionQueueType LinkedList $ActionQueueFileName papertrailqueue $ActionQueueCheckpointInterval 100 $ActionQueueMaxDiskSpace 1g $ActionResumeRetryCount -1 $ActionQueueSaveOnShutdown on $ActionQueueTimeoutEnqueue 10 $ActionQueueDiscardSeverity 0 *.* @@logs.papertrailapp.com:XXXXX;papertrailstandard & ~ B) Does a queue block get used over and over by every send action that follows it or only by the first one or only until it encounters a send followed by a discard action (~)? C) How do I reset a queue block so that an upcoming send action does not use a queue at all? D) Does a TLS block get used over and over by every send action that follows it or only by the first one or only until it encounters a send followed by a discard action (~)? E) How do I reset a TLS block so that an upcoming send action does not use TLS at all? F) If I run rsyslog -N1 I get: rsyslogd -N1 rsyslogd: version 5.8.10, config validation run (level 1), master config /etc/rsyslog.conf rsyslogd: WARNING: rsyslogd is running in compatibility mode. Automatically generated config directives may interfer with your rsyslog.conf settings. We suggest upgrading your config and adding -c5 as the first rsyslogd option. rsyslogd: Warning: backward compatibility layer added to following directive to rsyslog.conf: ModLoad immark rsyslogd: Warning: backward compatibility layer added to following directive to rsyslog.conf: MarkMessagePeriod 1200 rsyslogd: Warning: backward compatibility layer added to following directive to rsyslog.conf: ModLoad imuxsock rsyslogd: End of config validation run. Bye. Where do I place the -c5 so that it doesnt run in compatibility mode anymore?

  Read the article

• Proxy / Squid 2.7 / Debian Wheezy 6.7 / lots of TCP Timed-out

  - by Maroon Ibrahim

  i'm facing a lot of TCP timed-out on a busy cache server and here below my sysctl.conf configuration as well as an output of "netstat -st" Kernel 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64 GNU/Linux Any advice or help would be highly appreciated #################### Sysctl.conf cat /etc/sysctl.conf net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_tw_recycle = 1 fs.file-max = 65536 net.ipv4.tcp_low_latency = 1 net.core.wmem_max = 8388608 net.core.rmem_max = 8388608 net.ipv4.ip_local_port_range = 1024 65000 fs.aio-max-nr = 131072 net.ipv4.tcp_fin_timeout = 10 net.ipv4.tcp_keepalive_time = 60 net.ipv4.tcp_keepalive_intvl = 10 net.ipv4.tcp_keepalive_probes = 3 kernel.threads-max = 131072 kernel.msgmax = 32768 kernel.msgmni = 64 kernel.msgmnb = 65536 kernel.shmmax = 68719476736 kernel.shmall = 4294967296 net.ipv4.ip_forward = 1 net.ipv4.tcp_timestamps = 0 net.ipv4.conf.all.accept_redirects = 0 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_sack = 0 net.ipv4.tcp_syncookies = 1 net.ipv4.ip_dynaddr = 1 vm.swappiness = 0 vm.drop_caches = 3 net.ipv4.tcp_moderate_rcvbuf = 1 net.ipv4.tcp_no_metrics_save = 1 net.ipv4.tcp_ecn = 0 net.ipv4.tcp_max_orphans = 131072 net.ipv4.tcp_orphan_retries = 1 net.ipv4.conf.default.rp_filter = 0 net.ipv4.conf.default.accept_source_route = 0 net.ipv4.tcp_max_syn_backlog = 32768 net.core.netdev_max_backlog = 131072 net.ipv4.tcp_mem = 6085248 16227328 67108864 net.ipv4.tcp_wmem = 4096 131072 33554432 net.ipv4.tcp_rmem = 4096 174760 33554432 net.core.rmem_default = 33554432 net.core.rmem_max = 33554432 net.core.wmem_default = 33554432 net.core.wmem_max = 33554432 net.core.somaxconn = 10000 # ################ Netstat results /# netstat -st IcmpMsg: InType0: 2 InType3: 233754 InType8: 56251 InType11: 23192 OutType0: 56251 OutType3: 437 OutType8: 4 Tcp: 20680741 active connections openings 63642431 passive connection openings 1126690 failed connection attempts 2093143 connection resets received 13059 connections established 2649651696 segments received 2195445642 segments send out 183401499 segments retransmited 38299 bad segments received. 14648899 resets sent UdpLite: TcpExt: 507 SYN cookies sent 178 SYN cookies received 1376771 invalid SYN cookies received 1014577 resets received for embryonic SYN_RECV sockets 4530970 packets pruned from receive queue because of socket buffer overrun 7233 packets pruned from receive queue 688 packets dropped from out-of-order queue because of socket buffer overrun 12445 ICMP packets dropped because they were out-of-window 446 ICMP packets dropped because socket was locked 33812202 TCP sockets finished time wait in fast timer 622 TCP sockets finished time wait in slow timer 573656 packets rejects in established connections because of timestamp 133357718 delayed acks sent 23593 delayed acks further delayed because of locked socket Quick ack mode was activated 21288857 times 839 times the listen queue of a socket overflowed 839 SYNs to LISTEN sockets dropped 41 packets directly queued to recvmsg prequeue. 79166 bytes directly in process context from backlog 24 bytes directly received in process context from prequeue 2713742130 packet headers predicted 84 packets header predicted and directly queued to user 1925423249 acknowledgments not containing data payload received 877898013 predicted acknowledgments 16449673 times recovered from packet loss due to fast retransmit 17687820 times recovered from packet loss by selective acknowledgements 5047 bad SACK blocks received Detected reordering 11 times using FACK Detected reordering 1778091 times using SACK Detected reordering 97955 times using reno fast retransmit Detected reordering 280414 times using time stamp 839369 congestion windows fully recovered without slow start 4173098 congestion windows partially recovered using Hoe heuristic 305254 congestion windows recovered without slow start by DSACK 933682 congestion windows recovered without slow start after partial ack 77828 TCP data loss events TCPLostRetransmit: 5066 2618430 timeouts after reno fast retransmit 2927294 timeouts after SACK recovery 3059394 timeouts in loss state 75953830 fast retransmits 11929429 forward retransmits 51963833 retransmits in slow start 19418337 other TCP timeouts 2330398 classic Reno fast retransmits failed 2177787 SACK retransmits failed 742371590 packets collapsed in receive queue due to low socket buffer 13595689 DSACKs sent for old packets 50523 DSACKs sent for out of order packets 4658236 DSACKs received 175441 DSACKs for out of order packets received 880664 connections reset due to unexpected data 346356 connections reset due to early user close 2364841 connections aborted due to timeout TCPSACKDiscard: 1590 TCPDSACKIgnoredOld: 241849 TCPDSACKIgnoredNoUndo: 1636687 TCPSpuriousRTOs: 766073 TCPSackShifted: 74562088 TCPSackMerged: 169015212 TCPSackShiftFallback: 78391303 TCPBacklogDrop: 29 TCPReqQFullDoCookies: 507 TCPChallengeACK: 424921 TCPSYNChallenge: 170388 IpExt: InBcastPkts: 351510 InOctets: -609466797 OutOctets: -1057794685 InBcastOctets: 75631402 #

  Read the article

• Sendmail Tuning For Batch Mail Jobs

  - by Kyle Brandt

  I have a webservers that send out emails to a sendmail relay server as a batch job. The emails need to be accepted by the relay sendmail server as fast as possible, however, they do not need to go out (be relayed) very quickly. I am seeing a couple timeouts once and a while from the webserver trying to connect to the relay server. The load currently is about 30 emails a second for a couple minutes. There are quite a few tuning options for sendmail in the sendmail tuning guide. What I am focusing on now is the Delivery Mode: Delivery Mode There are a number of delivery modes that sendmail can operate in, set by the DeliveryMode ( d) configuration option. These modes specify how quickly mail will be delivered. Legal modes are: i deliver interactively (synchronously) b deliver in background (asynchronously) q queue only (don't deliver) d defer delivery attempts (don't deliver) There are tradeoffs. Mode i gives the sender the quickest feedback, but may slow down some mailers and is hardly ever necessary. Mode b delivers promptly but can cause large numbers of processes if you have a mailer that takes a long time to deliver a message. Mode q minimizes the load on your machine, but means that delivery may be delayed for up to the queue interval. Mode d is identical to mode q except that it also prevents lookups in maps including the -D flag from working during the initial queue phase; it is intended for ``dial on demand'' sites where DNS lookups might cost real money. Some simple error messages (e.g., host unknown during the SMTP protocol) will be delayed using this mode. Mode b is the usual default. If you run in mode q (queue only), d (defer), or b (deliver in background) sendmail will not expand aliases and follow .forward files upon initial receipt of the mail. This speeds up the response to RCPT commands. Mode i should not be used by the SMTP server. I currently have the CentOS default modes: Sendmail.cf: DeliveryMode=background Submit.cf: DeliveryMode=i Is sendmail.cf/mc for outgoing email from relay (to the intertubes) and sumbit.cf/mc for incoming eamil (from my webservers). Would it make sense to change the outgoing delivery mode to queue? If I did, what would the outbound email flow behave like? If this is the right thing to do, can anyone show me example mc configurations for this change? If it isn't, what recommendations are there for these constraints?

  Read the article

• How to Modify Data Security in Fusion Applications

  - by Elie Wazen

  The reference implementation in Fusion Applications is designed with built-in data security on business objects that implement the most common business practices.  For example, the “Sales Representative” job has the following two data security rules implemented on an “Opportunity” to restrict the list of Opportunities that are visible to an Sales Representative: Can view all the Opportunities where they are a member of the Opportunity Team Can view all the Opportunities where they are a resource of a territory in the Opportunity territory team While the above conditions may represent the most common access requirements of an Opportunity, some customers may have additional access constraints. This blog post explains: How to discover the data security implemented in Fusion Applications. How to customize data security Illustrative example. a.) How to discover seeded data security definitions The Security Reference Manuals explain the Function and Data Security implemented on each job role.  Security Reference Manuals are available on Oracle Enterprise Repository for Oracle Fusion Applications. The following is a snap shot of the security documented for the “Sales Representative” Job. The two data security policies define the list of Opportunities a Sales Representative can view. Here is a sample of data security policies on an Opportunity. Business Object Policy Description Policy Store Implementation Opportunity A Sales Representative can view opportunity where they are a territory resource in the opportunity territory team Role: Opportunity Territory Resource Duty Privilege: View Opportunity (Data) Resource: Opportunity A Sales Representative can view opportunity where they are an opportunity sales team member with view, edit, or full access Role: Opportunity Sales Representative Duty Privilege: View Opportunity (Data) Resource: Opportunity Description of Columns Column Name Description Policy Description Explains the data filters that are implemented as a SQL Where Clause in a Data Security Grant Policy Store Implementation Provides the implementation details of the Data Security Grant for this policy. In this example the Opportunities listed for a “Sales Representative” job role are derived from a combination of two grants defined on two separate duty roles at are inherited by the Sales Representative job role. b.) How to customize data security Requirement 1: Opportunities should be viewed only by members of the opportunity team and not by all the members of all the territories on the opportunity. Solution: Remove the role “Opportunity Territory Resource Duty” from the hierarchy of the “Sales Representative” job role. Best Practice: Do not modify the seeded role hierarchy. Create a custom “Sales Representative” job role and build the role hierarchy with the seeded duty roles. Requirement 2: Opportunities must be more restrictive based on a custom attribute that identifies if a Opportunity is confidential or not. Confidential Opportunities must be visible only the owner of the Opportunity. Solution: Modify the (2) data security policy in the above example as follows: A Sales Representative can view opportunity where they are a territory resource in the opportunity territory team and the opportunity is not confidential. Implementation of this policy is more invasive. The seeded SQL where clause of the data security grant on “Opportunity Territory Resource Duty” has to be modified and the condition that checks for the confidential flag must be added. Best Practice: Do not modify the seeded grant. Create a new grant with the modified condition. End Date the seeded grant. c.) Illustrative Example (Implementing Requirement 2) A data security policy contains the following components: Role Object Instance Set Action Of the above four components, the Role and Instance Set are the only components that are customizable. Object and Actions for that object are seed data and cannot be modified. To customize a seeded policy, “A Sales Representative can view opportunity where they are a territory resource in the opportunity territory team”, Find the seeded policy Identify the Role, Object, Instance Set and Action components of the policy Create a new custom instance set based on the seeded instance set. End Date the seeded policies Create a new data security policy with custom instance set c-1: Find the seeded policy Step 1: 1. Find the Role 2. Open 3. Find Policies Step 2: Click on the Data Security Tab Sort by “Resource Name” Find all the policies with the “Condition” as “where they are a territory resource in the opportunity territory team” In this example, we can see there are 5 policies for “Opportunity Territory Resource Duty” on Opportunity object. Step 3: Now that we know the policy details, we need to create new instance set with the custom condition. All instance sets are linked to the object. Find the object using global search option. Open it and click on “condition” tab Sort by Display name Find the Instance set Edit the instance set and copy the “SQL Predicate” to a notepad. Create a new instance set with the modified SQL Predicate from above by clicking on the icon as shown below. Step 4: End date the seeded data security policies on the duty role and create new policies with your custom instance set. Repeat the navigation in step Edit each of the 5 policies and end date them 3. Create new custom policies with the same information as the seeded policies in the “General Information”, “Roles” and “Action” tabs. 4. In the “Rules” tab, please pick the new instance set that was created in Step 3.

  Read the article

• where to get free 1440 cron jobs a day??

  - by Nok Imchen

  Well, I'm making a program for my own use. In this program, i need to set up cron job. The cron job should run every minute (24 hr * 60 mins = 1440 times). Thus, I'll need to set up a cron job with a frequency of 1 minute. I think Google app engine gives free cron job. But i'm very new to it. I downloaded the java SDK and read the document but understood nothing :( So, i cant use Google app engine. Is here any other free service like Google app engine which but with easier inferface??? all i want is a cron job with 1 minute frequency please help/suggest me.... thank you

  Read the article

• Are there clean ways to do these Quartz Triggers?

  - by Ryan Elkins

  I'm using Quartz to schedule some jobs but I have a few scenarios that I'm not sure how to resolve. 1) Lets say I have a job that is scheduled to run every 5 minutes. Generally that works well but periodically the job takes more than 5 minutes and I don't really want multiple instances of the job running simultaneously. 2) I have a job that can take between 1 and 60 minutes to complete. I want it to run continuously but pause for 10 minutes between runs, regardless of how long it took previously. I like using Quartz for this rather than some sort of loop because if a job crashes Quartz will still spin up a new one based on the schedule. I am using Quartz in Java right now.

  Read the article

• Where to get 1440 free cron jobs a day?

  - by Nok Imchen

  I'm making a program for my own use. In this program, I need to set up cron job. The cron job should run every minute (24 hr * 60 mins = 1440 times). Thus, I'll need to set up a cron job with a frequency of 1 minute. I think Google App Engine gives free cron job. But I'm very new to it. I downloaded the java SDK and read the document but understood nothing :( So, I can't use Google App Engine. Is here any other free service like Google app engine which but with easier inferface??? All I want is a cron job with 1 minute frequency Please help/suggest me... Thank you

  Read the article

• Best way to store application images taken via camera

  - by Dave

  Hi all, I'm just looking for some insight into what would be the best way for me to store images as part of my app. I have an activity that represents a 'Job' which has a couple of edittext's and underneath was planning on using the Gallery component to show images relevant to this job. The job data is stored in a database (on the sdcard) so was also thinking of creating a table to store 'JobImages' and having each image stored as a byte array. But I'm not sure if it would be better to store the images directly on sdcard under a folder structure specific to my application and the job. E.g. using the job ID number as a folder name. Depending on which method I use will greatly determine the code that goes into an 'adapter' that allows me to bind to the gallery component so before I begin I was wondering if anyone has had the same design problem and what option they chose. Thanks, Dave

  Read the article

• fast way for finding GUIDs.

  - by Behrooz

  hi. I have lots(+2000) of GUIDs(in some network class) and my program must find one of them when it receives a message and do the job associated with it. the positive point is i have a hard-code generator, but the fastest way is my goal(and i don't know how to implement it). my code should do something like this: switch(received guid) { case guid1: do job 1; break; case guid2: do job 2; break; case guid3: do job 3; break; case guid4: do job 4; break; .... }

  Read the article

• IPsec tunnel to Android device not created even though there is an IKE SA

  - by Quentin Swain

  I'm trying to configure a VPN tunnel between an Android device running 4.1 and a Fedora 17 Linux box running strongSwan 5.0. The device reports that it is connected and strongSwan statusall returns that there is an IKE SA, but doesn't display a tunnel. I used the instructions for iOS in the wiki to generate certificates and configure strongSwan. Since Android uses a modified version of racoon this should work and since the connection is partly established I think I am on the right track. I don't see any errors about not being able to create the tunnel. This is the configuration for the strongSwan connection conn android2 keyexchange=ikev1 authby=xauthrsasig xauth=server left=96.244.142.28 leftsubnet=0.0.0.0/0 leftfirewall=yes leftcert=serverCert.pem right=%any rightsubnet=10.0.0.0/24 rightsourceip=10.0.0.2 rightcert=clientCert.pem ike=aes256-sha1-modp1024 auto=add This is the output of strongswan statusall Status of IKE charon daemon (strongSwan 5.0.0, Linux 3.3.4-5.fc17.x86_64, x86_64): uptime: 20 minutes, since Oct 31 10:27:31 2012 malloc: sbrk 270336, mmap 0, used 198144, free 72192 worker threads: 8 of 16 idle, 7/1/0/0 working, job queue: 0/0/0/0, scheduled: 7 loaded plugins: charon aes des sha1 sha2 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs8 pgp dnskey pem openssl fips-prf gmp xcbc cmac hmac attr kernel-netlink resolve socket-default stroke updown xauth-generic Virtual IP pools (size/online/offline): android-hybrid: 1/0/0 android2: 1/1/0 Listening IP addresses: 96.244.142.28 Connections: android-hybrid: %any...%any IKEv1 android-hybrid: local: [C=CH, O=strongSwan, CN=vpn.strongswan.org] uses public key authentication android-hybrid: cert: "C=CH, O=strongSwan, CN=vpn.strongswan.org" android-hybrid: remote: [%any] uses XAuth authentication: any android-hybrid: child: dynamic === dynamic TUNNEL android2: 96.244.142.28...%any IKEv1 android2: local: [C=CH, O=strongSwan, CN=vpn.strongswan.org] uses public key authentication android2: cert: "C=CH, O=strongSwan, CN=vpn.strongswan.org" android2: remote: [C=CH, O=strongSwan, CN=client] uses public key authentication android2: cert: "C=CH, O=strongSwan, CN=client" android2: remote: [%any] uses XAuth authentication: any android2: child: 0.0.0.0/0 === 10.0.0.0/24 TUNNEL Security Associations (1 up, 0 connecting): android2[3]: ESTABLISHED 10 seconds ago, 96.244.142.28[C=CH, O=strongSwan, CN=vpn.strongswan.org]...208.54.35.241[C=CH, O=strongSwan, CN=client] android2[3]: Remote XAuth identity: android android2[3]: IKEv1 SPIs: 4151e371ad46b20d_i 59a56390d74792d2_r*, public key reauthentication in 56 minutes android2[3]: IKE proposal: AES_CBC_256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024 The output of ip -s xfrm policy src ::/0 dst ::/0 uid 0 socket in action allow index 3851 priority 0 ptype main share any flag (0x00000000) lifetime config: limit: soft 0(bytes), hard 0(bytes) limit: soft 0(packets), hard 0(packets) expire add: soft 0(sec), hard 0(sec) expire use: soft 0(sec), hard 0(sec) lifetime current: 0(bytes), 0(packets) add 2012-10-31 13:29:08 use - src ::/0 dst ::/0 uid 0 socket out action allow index 3844 priority 0 ptype main share any flag (0x00000000) lifetime config: limit: soft 0(bytes), hard 0(bytes) limit: soft 0(packets), hard 0(packets) expire add: soft 0(sec), hard 0(sec) expire use: soft 0(sec), hard 0(sec) lifetime current: 0(bytes), 0(packets) add 2012-10-31 13:29:08 use - src ::/0 dst ::/0 uid 0 socket in action allow index 3835 priority 0 ptype main share any flag (0x00000000) lifetime config: limit: soft 0(bytes), hard 0(bytes) limit: soft 0(packets), hard 0(packets) expire add: soft 0(sec), hard 0(sec) expire use: soft 0(sec), hard 0(sec) lifetime current: 0(bytes), 0(packets) add 2012-10-31 13:29:08 use - src ::/0 dst ::/0 uid 0 socket out action allow index 3828 priority 0 ptype main share any flag (0x00000000) lifetime config: limit: soft 0(bytes), hard 0(bytes) limit: soft 0(packets), hard 0(packets) expire add: soft 0(sec), hard 0(sec) expire use: soft 0(sec), hard 0(sec) lifetime current: 0(bytes), 0(packets) add 2012-10-31 13:29:08 use - src 0.0.0.0/0 dst 0.0.0.0/0 uid 0 socket in action allow index 3819 priority 0 ptype main share any flag (0x00000000) lifetime config: limit: soft 0(bytes), hard 0(bytes) limit: soft 0(packets), hard 0(packets) expire add: soft 0(sec), hard 0(sec) expire use: soft 0(sec), hard 0(sec) lifetime current: 0(bytes), 0(packets) add 2012-10-31 13:29:08 use 2012-10-31 13:29:39 src 0.0.0.0/0 dst 0.0.0.0/0 uid 0 socket out action allow index 3812 priority 0 ptype main share any flag (0x00000000) lifetime config: limit: soft 0(bytes), hard 0(bytes) limit: soft 0(packets), hard 0(packets) expire add: soft 0(sec), hard 0(sec) expire use: soft 0(sec), hard 0(sec) lifetime current: 0(bytes), 0(packets) add 2012-10-31 13:29:08 use 2012-10-31 13:29:22 src 0.0.0.0/0 dst 0.0.0.0/0 uid 0 socket in action allow index 3803 priority 0 ptype main share any flag (0x00000000) lifetime config: limit: soft 0(bytes), hard 0(bytes) limit: soft 0(packets), hard 0(packets) expire add: soft 0(sec), hard 0(sec) expire use: soft 0(sec), hard 0(sec) lifetime current: 0(bytes), 0(packets) add 2012-10-31 13:29:08 use 2012-10-31 13:29:20 src 0.0.0.0/0 dst 0.0.0.0/0 uid 0 socket out action allow index 3796 priority 0 ptype main share any flag (0x00000000) lifetime config: limit: soft 0(bytes), hard 0(bytes) limit: soft 0(packets), hard 0(packets) expire add: soft 0(sec), hard 0(sec) expire use: soft 0(sec), hard 0(sec) lifetime current: 0(bytes), 0(packets) add 2012-10-31 13:29:08 use 2012-10-31 13:29:20 So a xfrm policy isn't being created for the connection, even though there is an SA between device and strongswan. Executing ip -s xfrm policy on the android device results in the following output: src 0.0.0.0/0 dst 10.0.0.2/32 uid 0 dir in action allow index 40 priority 2147483648 share any flag (0x00000000) lifetime config: limit: soft (INF)(bytes), hard (INF)(bytes) limit: soft (INF)(packets), hard (INF)(packets) expire add: soft 0(sec), hard 0(sec) expire use: soft 0(sec), hard 0(sec) lifetime current: 0(bytes), 0(packets) add 2012-10-31 13:42:08 use - tmpl src 96.244.142.28 dst 25.239.33.30 proto esp spi 0x00000000(0) reqid 0(0x00000000) mode tunnel level required share any enc-mask 00000000 auth-mask 00000000 comp-mask 00000000 src 10.0.0.2/32 dst 0.0.0.0/0 uid 0 dir out action allow index 33 priority 2147483648 share any flag (0x00000000) lifetime config: limit: soft (INF)(bytes), hard (INF)(bytes) limit: soft (INF)(packets), hard (INF)(packets) expire add: soft 0(sec), hard 0(sec) expire use: soft 0(sec), hard 0(sec) lifetime current: 0(bytes), 0(packets) add 2012-10-31 13:42:08 use - tmpl src 25.239.33.30 dst 96.244.142.28 proto esp spi 0x00000000(0) reqid 0(0x00000000) mode tunnel level required share any enc-mask 00000000 auth-mask 00000000 comp-mask 00000000 src 0.0.0.0/0 dst 0.0.0.0/0 uid 0 dir 4 action allow index 28 priority 0 share any flag (0x00000000) lifetime config: limit: soft (INF)(bytes), hard (INF)(bytes) limit: soft (INF)(packets), hard (INF)(packets) expire add: soft 0(sec), hard 0(sec) expire use: soft 0(sec), hard 0(sec) lifetime current: 0(bytes), 0(packets) add 2012-10-31 13:42:04 use 2012-10-31 13:42:08 src 0.0.0.0/0 dst 0.0.0.0/0 uid 0 dir 3 action allow index 19 priority 0 share any flag (0x00000000) lifetime config: limit: soft (INF)(bytes), hard (INF)(bytes) limit: soft (INF)(packets), hard (INF)(packets) expire add: soft 0(sec), hard 0(sec) expire use: soft 0(sec), hard 0(sec) lifetime current: 0(bytes), 0(packets) add 2012-10-31 13:42:04 use 2012-10-31 13:42:08 src 0.0.0.0/0 dst 0.0.0.0/0 uid 0 dir 4 action allow index 12 priority 0 share any flag (0x00000000) lifetime config: limit: soft (INF)(bytes), hard (INF)(bytes) limit: soft (INF)(packets), hard (INF)(packets) expire add: soft 0(sec), hard 0(sec) expire use: soft 0(sec), hard 0(sec) lifetime current: 0(bytes), 0(packets) add 2012-10-31 13:42:04 use 2012-10-31 13:42:06 src 0.0.0.0/0 dst 0.0.0.0/0 uid 0 dir 3 action allow index 3 priority 0 share any flag (0x00000000) lifetime config: limit: soft (INF)(bytes), hard (INF)(bytes) limit: soft (INF)(packets), hard (INF)(packets) expire add: soft 0(sec), hard 0(sec) expire use: soft 0(sec), hard 0(sec) lifetime current: 0(bytes), 0(packets) add 2012-10-31 13:42:04 use 2012-10-31 13:42:07 Logs from charon: 00[DMN] Starting IKE charon daemon (strongSwan 5.0.0, Linux 3.3.4-5.fc17.x86_64, x86_64) 00[KNL] listening on interfaces: 00[KNL] em1 00[KNL] 96.244.142.28 00[KNL] fe80::224:e8ff:fed2:18b2 00[CFG] loading ca certificates from '/etc/strongswan/ipsec.d/cacerts' 00[CFG] loaded ca certificate "C=CH, O=strongSwan, CN=strongSwan CA" from '/etc/strongswan/ipsec.d/cacerts/caCert.pem' 00[CFG] loading aa certificates from '/etc/strongswan/ipsec.d/aacerts' 00[CFG] loading ocsp signer certificates from '/etc/strongswan/ipsec.d/ocspcerts' 00[CFG] loading attribute certificates from '/etc/strongswan/ipsec.d/acerts' 00[CFG] loading crls from '/etc/strongswan/ipsec.d/crls' 00[CFG] loading secrets from '/etc/strongswan/ipsec.secrets' 00[CFG] loaded RSA private key from '/etc/strongswan/ipsec.d/private/clientKey.pem' 00[CFG] loaded IKE secret for %any 00[CFG] loaded EAP secret for android 00[CFG] loaded EAP secret for android 00[DMN] loaded plugins: charon aes des sha1 sha2 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs8 pgp dnskey pem openssl fips-prf gmp xcbc cmac hmac attr kernel-netlink resolve socket-default stroke updown xauth-generic 08[NET] waiting for data on sockets 16[LIB] created thread 16 [15338] 16[JOB] started worker thread 16 11[CFG] received stroke: add connection 'android-hybrid' 11[CFG] conn android-hybrid 11[CFG] left=%any 11[CFG] leftsubnet=(null) 11[CFG] leftsourceip=(null) 11[CFG] leftauth=pubkey 11[CFG] leftauth2=(null) 11[CFG] leftid=(null) 11[CFG] leftid2=(null) 11[CFG] leftrsakey=(null) 11[CFG] leftcert=serverCert.pem 11[CFG] leftcert2=(null) 11[CFG] leftca=(null) 11[CFG] leftca2=(null) 11[CFG] leftgroups=(null) 11[CFG] leftupdown=ipsec _updown iptables 11[CFG] right=%any 11[CFG] rightsubnet=(null) 11[CFG] rightsourceip=96.244.142.3 11[CFG] rightauth=xauth 11[CFG] rightauth2=(null) 11[CFG] rightid=%any 11[CFG] rightid2=(null) 11[CFG] rightrsakey=(null) 11[CFG] rightcert=(null) 11[CFG] rightcert2=(null) 11[CFG] rightca=(null) 11[CFG] rightca2=(null) 11[CFG] rightgroups=(null) 11[CFG] rightupdown=(null) 11[CFG] eap_identity=(null) 11[CFG] aaa_identity=(null) 11[CFG] xauth_identity=(null) 11[CFG] ike=aes256-sha1-modp1024 11[CFG] esp=aes128-sha1-modp2048,3des-sha1-modp1536 11[CFG] dpddelay=30 11[CFG] dpdtimeout=150 11[CFG] dpdaction=0 11[CFG] closeaction=0 11[CFG] mediation=no 11[CFG] mediated_by=(null) 11[CFG] me_peerid=(null) 11[CFG] keyexchange=ikev1 11[KNL] getting interface name for %any 11[KNL] %any is not a local address 11[KNL] getting interface name for %any 11[KNL] %any is not a local address 11[CFG] left nor right host is our side, assuming left=local 11[CFG] loaded certificate "C=CH, O=strongSwan, CN=vpn.strongswan.org" from 'serverCert.pem' 11[CFG] id '%any' not confirmed by certificate, defaulting to 'C=CH, O=strongSwan, CN=vpn.strongswan.org' 11[CFG] added configuration 'android-hybrid' 11[CFG] adding virtual IP address pool 'android-hybrid': 96.244.142.3/32 13[CFG] received stroke: add connection 'android2' 13[CFG] conn android2 13[CFG] left=96.244.142.28 13[CFG] leftsubnet=0.0.0.0/0 13[CFG] leftsourceip=(null) 13[CFG] leftauth=pubkey 13[CFG] leftauth2=(null) 13[CFG] leftid=(null) 13[CFG] leftid2=(null) 13[CFG] leftrsakey=(null) 13[CFG] leftcert=serverCert.pem 13[CFG] leftcert2=(null) 13[CFG] leftca=(null) 13[CFG] leftca2=(null) 13[CFG] leftgroups=(null) 13[CFG] leftupdown=ipsec _updown iptables 13[CFG] right=%any 13[CFG] rightsubnet=10.0.0.0/24 13[CFG] rightsourceip=10.0.0.2 13[CFG] rightauth=pubkey 13[CFG] rightauth2=xauth 13[CFG] rightid=(null) 13[CFG] rightid2=(null) 13[CFG] rightrsakey=(null) 13[CFG] rightcert=clientCert.pem 13[CFG] rightcert2=(null) 13[CFG] rightca=(null) 13[CFG] rightca2=(null) 13[CFG] rightgroups=(null) 13[CFG] rightupdown=(null) 13[CFG] eap_identity=(null) 13[CFG] aaa_identity=(null) 13[CFG] xauth_identity=(null) 13[CFG] ike=aes256-sha1-modp1024 13[CFG] esp=aes128-sha1-modp2048,3des-sha1-modp1536 13[CFG] dpddelay=30 13[CFG] dpdtimeout=150 13[CFG] dpdaction=0 13[CFG] closeaction=0 13[CFG] mediation=no 13[CFG] mediated_by=(null) 13[CFG] me_peerid=(null) 13[CFG] keyexchange=ikev0 13[KNL] getting interface name for %any 13[KNL] %any is not a local address 13[KNL] getting interface name for 96.244.142.28 13[KNL] 96.244.142.28 is on interface em1 13[CFG] loaded certificate "C=CH, O=strongSwan, CN=vpn.strongswan.org" from 'serverCert.pem' 13[CFG] id '96.244.142.28' not confirmed by certificate, defaulting to 'C=CH, O=strongSwan, CN=vpn.strongswan.org' 13[CFG] loaded certificate "C=CH, O=strongSwan, CN=client" from 'clientCert.pem' 13[CFG] id '%any' not confirmed by certificate, defaulting to 'C=CH, O=strongSwan, CN=client' 13[CFG] added configuration 'android2' 13[CFG] adding virtual IP address pool 'android2': 10.0.0.2/32 08[NET] received packet: from 208.54.35.241[32235] to 96.244.142.28[500] 15[CFG] looking for an ike config for 96.244.142.28...208.54.35.241 15[CFG] candidate: %any...%any, prio 2 15[CFG] candidate: 96.244.142.28...%any, prio 5 15[CFG] found matching ike config: 96.244.142.28...%any with prio 5 01[JOB] next event in 29s 999ms, waiting 15[IKE] received NAT-T (RFC 3947) vendor ID 15[IKE] received draft-ietf-ipsec-nat-t-ike-02 vendor ID 15[IKE] received draft-ietf-ipsec-nat-t-ike-02\n vendor ID 15[IKE] received draft-ietf-ipsec-nat-t-ike-00 vendor ID 15[IKE] received XAuth vendor ID 15[IKE] received Cisco Unity vendor ID 15[IKE] received DPD vendor ID 15[IKE] 208.54.35.241 is initiating a Main Mode IKE_SA 15[IKE] IKE_SA (unnamed)[1] state change: CREATED => CONNECTING 15[CFG] selecting proposal: 15[CFG] proposal matches 15[CFG] received proposals: IKE:AES_CBC_256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024, IKE:AES_CBC_256/HMAC_MD5_96/PRF_HMAC_MD5/MODP_1024, IKE:AES_CBC_128/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024, IKE:AES_CBC_128/HMAC_MD5_96/PRF_HMAC_MD5/MODP_1024, IKE:3DES_CBC/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024, IKE:3DES_CBC/HMAC_MD5_96/PRF_HMAC_MD5/MODP_1024, IKE:DES_CBC/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024, IKE:DES_CBC/HMAC_MD5_96/PRF_HMAC_MD5/MODP_1024 15[CFG] configured proposals: IKE:AES_CBC_256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024, IKE:AES_CBC_128/AES_CBC_192/AES_CBC_256/3DES_CBC/CAMELLIA_CBC_128/CAMELLIA_CBC_192/CAMELLIA_CBC_256/HMAC_MD5_96/HMAC_SHA1_96/HMAC_SHA2_256_128/HMAC_SHA2_384_192/HMAC_SHA2_512_256/AES_XCBC_96/AES_CMAC_96/PRF_HMAC_MD5/PRF_HMAC_SHA1/PRF_HMAC_SHA2_256/PRF_HMAC_SHA2_384/PRF_HMAC_SHA2_512/PRF_AES128_XCBC/PRF_AES128_CMAC/MODP_2048/MODP_2048_224/MODP_2048_256/MODP_1536/MODP_4096/MODP_8192/MODP_1024/MODP_1024_160 15[CFG] selected proposal: IKE:AES_CBC_256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024 15[NET] sending packet: from 96.244.142.28[500] to 208.54.35.241[32235] 04[NET] sending packet: from 96.244.142.28[500] to 208.54.35.241[32235] 15[MGR] checkin IKE_SA (unnamed)[1] 15[MGR] check-in of IKE_SA successful. 08[NET] received packet: from 208.54.35.241[32235] to 96.244.142.28[500] 08[NET] waiting for data on sockets 07[MGR] checkout IKE_SA by message 07[MGR] IKE_SA (unnamed)[1] successfully checked out 07[NET] received packet: from 208.54.35.241[32235] to 96.244.142.28[500] 07[LIB] size of DH secret exponent: 1023 bits 07[IKE] remote host is behind NAT 07[IKE] sending cert request for "C=CH, O=strongSwan, CN=strongSwan CA" 07[ENC] generating NAT_D_V1 payload finished 07[NET] sending packet: from 96.244.142.28[500] to 208.54.35.241[32235] 07[MGR] checkin IKE_SA (unnamed)[1] 07[MGR] check-in of IKE_SA successful. 04[NET] sending packet: from 96.244.142.28[500] to 208.54.35.241[32235] 08[NET] received packet: from 208.54.35.241[35595] to 96.244.142.28[4500] 10[IKE] ignoring certificate request without data 10[IKE] received end entity cert "C=CH, O=strongSwan, CN=client" 10[CFG] looking for XAuthInitRSA peer configs matching 96.244.142.28...208.54.35.241[C=CH, O=strongSwan, CN=client] 10[CFG] candidate "android-hybrid", match: 1/1/2/2 (me/other/ike/version) 10[CFG] candidate "android2", match: 1/20/5/1 (me/other/ike/version) 10[CFG] selected peer config "android2" 10[CFG] certificate "C=CH, O=strongSwan, CN=client" key: 2048 bit RSA 10[CFG] using trusted ca certificate "C=CH, O=strongSwan, CN=strongSwan CA" 10[CFG] checking certificate status of "C=CH, O=strongSwan, CN=client" 10[CFG] ocsp check skipped, no ocsp found 10[CFG] certificate status is not available 10[CFG] certificate "C=CH, O=strongSwan, CN=strongSwan CA" key: 2048 bit RSA 10[CFG] reached self-signed root ca with a path length of 0 10[CFG] using trusted certificate "C=CH, O=strongSwan, CN=client" 10[IKE] authentication of 'C=CH, O=strongSwan, CN=client' with RSA successful 10[ENC] added payload of type ID_V1 to message 10[ENC] added payload of type SIGNATURE_V1 to message 10[IKE] authentication of 'C=CH, O=strongSwan, CN=vpn.strongswan.org' (myself) successful 10[IKE] queueing XAUTH task 10[IKE] sending end entity cert "C=CH, O=strongSwan, CN=vpn.strongswan.org" 10[NET] sending packet: from 96.244.142.28[4500] to 208.54.35.241[35595] 04[NET] sending packet: from 96.244.142.28[4500] to 208.54.35.241[35595] 10[IKE] activating new tasks 10[IKE] activating XAUTH task 10[NET] sending packet: from 96.244.142.28[4500] to 208.54.35.241[35595] 04[NET] sending packet: from 96.244.142.28[4500] to 208.54.35.241[35595] 01[JOB] next event in 3s 999ms, waiting 10[MGR] checkin IKE_SA android2[1] 10[MGR] check-in of IKE_SA successful. 08[NET] received packet: from 208.54.35.241[35595] to 96.244.142.28[4500] 08[NET] waiting for data on sockets 12[MGR] checkout IKE_SA by message 12[MGR] IKE_SA android2[1] successfully checked out 12[NET] received packet: from 208.54.35.241[35595] to 96.244.142.28[4500] 12[MGR] checkin IKE_SA android2[1] 12[MGR] check-in of IKE_SA successful. 08[NET] received packet: from 208.54.35.241[35595] to 96.244.142.28[4500] 16[MGR] checkout IKE_SA by message 16[MGR] IKE_SA android2[1] successfully checked out 16[NET] received packet: from 208.54.35.241[35595] to 96.244.142.28[4500] 08[NET] waiting for data on sockets 16[IKE] XAuth authentication of 'android' successful 16[IKE] reinitiating already active tasks 16[IKE] XAUTH task 16[NET] sending packet: from 96.244.142.28[4500] to 208.54.35.241[35595] 04[NET] sending packet: from 96.244.142.28[4500] to 208.54.35.241[35595] 16[MGR] checkin IKE_SA android2[1] 01[JOB] next event in 3s 907ms, waiting 16[MGR] check-in of IKE_SA successful. 08[NET] received packet: from 208.54.35.241[35595] to 96.244.142.28[4500] 09[MGR] checkout IKE_SA by message 09[MGR] IKE_SA android2[1] successfully checked out 09[NET] received packet: from 208.54.35.241[35595] to 96.244.142.28[4500] .8rS 09[IKE] IKE_SA android2[1] established between 96.244.142.28[C=CH, O=strongSwan, CN=vpn.strongswan.org]...208.54.35.241[C=CH, O=strongSwan, CN=client] 09[IKE] IKE_SA android2[1] state change: CONNECTING => ESTABLISHED 09[IKE] scheduling reauthentication in 3409s 09[IKE] maximum IKE_SA lifetime 3589s 09[IKE] activating new tasks 09[IKE] nothing to initiate 09[MGR] checkin IKE_SA android2[1] 09[MGR] check-in of IKE_SA successful. 09[MGR] checkout IKE_SA 09[MGR] IKE_SA android2[1] successfully checked out 09[MGR] checkin IKE_SA android2[1] 09[MGR] check-in of IKE_SA successful. 01[JOB] next event in 3s 854ms, waiting 08[NET] waiting for data on sockets 08[NET] received packet: from 208.54.35.241[35595] to 96.244.142.28[4500] 14[MGR] checkout IKE_SA by message 14[MGR] IKE_SA android2[1] successfully checked out 14[NET] received packet: from 208.54.35.241[35595] to 96.244.142.28[4500] 14[IKE] processing INTERNAL_IP4_ADDRESS attribute 14[IKE] processing INTERNAL_IP4_NETMASK attribute 14[IKE] processing INTERNAL_IP4_DNS attribute 14[IKE] processing INTERNAL_IP4_NBNS attribute 14[IKE] processing UNITY_BANNER attribute 14[IKE] processing UNITY_DEF_DOMAIN attribute 14[IKE] processing UNITY_SPLITDNS_NAME attribute 14[IKE] processing UNITY_SPLIT_INCLUDE attribute 14[IKE] processing UNITY_LOCAL_LAN attribute 14[IKE] processing APPLICATION_VERSION attribute 14[IKE] peer requested virtual IP %any 14[CFG] assigning new lease to 'android' 14[IKE] assigning virtual IP 10.0.0.2 to peer 'android' 14[NET] sending packet: from 96.244.142.28[4500] to 208.54.35.241[35595] 14[MGR] checkin IKE_SA android2[1] 14[MGR] check-in of IKE_SA successful. 04[NET] sending packet: from 96.244.142.28[4500] to 208.54.35.241[35595] 08[NET] waiting for data on sockets 01[JOB] got event, queuing job for execution 01[JOB] next event in 91ms, waiting 13[MGR] checkout IKE_SA 13[MGR] IKE_SA android2[1] successfully checked out 13[MGR] checkin IKE_SA android2[1] 13[MGR] check-in of IKE_SA successful. 01[JOB] got event, queuing job for execution 01[JOB] next event in 24s 136ms, waiting 15[MGR] checkout IKE_SA 15[MGR] IKE_SA android2[1] successfully checked out 15[MGR] checkin IKE_SA android2[1] 15[MGR] check-in of IKE_SA successful.

  Read the article

• Framework 4 Features: Support for Timed Jobs

  - by Anthony Shorten

  One of the new features of the Oracle Utilities Application Framework V4 is the ability for the batch framework to support Timed Batch. Traditionally batch is associated with set processing in the background in a fixed time frame. For example, billing customers. Over the last few versions their has been functionality required by the products required a more monitoring style batch process. The monitor is a batch process that looks for specific business events based upon record status or other pieces of data. For example, the framework contains a fact monitor (F1-FCTRN) that can be configured to look for specific status's or other conditions. The batch process then uses the instructions on the object to determine what to do. To support monitor style processing, you need to run the process regularly a number of times a day (for example, every ten minutes). Traditional batch could support this but it was not as optimal as expected (if you are a site using the old Workflow subsystem, you understand what I mean). The Batch framework was extended to add additional facilities to support times (and continuous batch which is another new feature for another blog entry). The new facilities include: The batch control now defines the job as Timed or Not Timed. Non-Timed batch are traditional batch jobs. The timer interval (the interval between executions) can be specified The timer can be made active or inactive. Only active timers are executed. Setting the Timer Active to inactive will stop the job at the next time interval. Setting the Timer Active to Active will start the execution of the timed job. You can specify the credentials, language to view the messages and an email address to send the a summary of the execution to. The email address is optional and requires an email server to be specified in the relevant feature configuration. You can specify the thread limits and commit intervals to be sued for the multiple executions. Once a timer job is defined it will be executed automatically by the Business Application Server process if the DEFAULT threadpool is active. This threadpool can be started using the online batch daemon (for non-production) or externally using the threadpoolworker utility. At that time any batch process with the Timer Active set to Active and Batch Control Type of Timed will begin executing. As Timed jobs are executed automatically then they do not appear in any external schedule or are managed by an external scheduler (except via the DEFAULT threadpool itself of course). Now, if the job has no work to do as the timer interval is being reached then that instance of the job is stopped and the next instance started at the timer interval. If there is still work to complete when the interval interval is reached, the instance will continue processing till the work is complete, then the instance will be stopped and the next instance scheduled for the next timer interval. One of the key ways of optimizing this processing is to set the timer interval correctly for the expected workload. This is an interesting new feature of the batch framework and we anticipate it will come in handy for specific business situations with the monitor processes.

  Read the article

• What Counts For a DBA: Imagination

  - by drsql

  "Imagination…One little spark, of inspiration… is at the heart, of all creation." – From the song "One Little Spark", by the Sherman Brothers I have a confession to make. Despite my great enthusiasm for databases and programming, it occurs to me that every database system I've ever worked on has been, in terms of its inputs and outputs, downright dull. Most have been glorified e-spreadsheets, many replacing manual systems built on actual spreadsheets. I've created a lot of database-driven software whose main job was to "count stuff"; phone calls, web visitors, payments, donations, pieces of equipment and so on. Sometimes, instead of counting stuff, the database recorded values from other stuff, such as data from sensors or networking devices. Yee hah! So how do we, as DBAs, maintain high standards and high spirits when we realize that so much of our work would fail to raise the pulse of even the most easily excitable soul? The answer lies in our imagination. To understand what I mean by this, consider a role that, in terms of its output, offers an extreme counterpoint to that of the DBA: the Disney Imagineer. Their job is to design Disney's Theme Parks, of which I'm a huge fan. To me this has always seemed like a fascinating and exciting job. What must an Imagineer do, every day, to inspire the feats of creativity that are so clearly evident in those spectacular rides and shows? Here, if ever there was one, is a role where "dull moments" must be rare indeed, surely? I wanted to find out, and so parted with a considerable sum of money for my wife and I to have lunch with one; I reasoned that if I found one small way to apply their secrets to my own career, it would be money well spent. Early in the conversation with our Imagineer (Cindy Cote), the job did indeed sound magical. However, as talk turned to management meetings, budget-wrangling and insane deadlines, I came to the strange realization that, in fact, her job was a lot more like mine than I would ever have guessed. Much like databases, all those spectacular Disney rides bring with them a vast array of complex plumbing, lighting, safety features, and all manner of other "boring bits", kept well out of sight of the end user, but vital for creating the desired experience; and, of course, it is these "boring bits" that take up much of the Imagineer's time. Naturally, there is still a vital part of their job that is spent testing out new ideas, putting themselves in the place of a park visitor, from a 9-year-old boy to a 90-year-old grandmother, and trying to imagine what experiences they'd like to have. It is these small, but vital, sparks of imagination and creativity that have the biggest impact. The real feat of a successful Imagineer is clearly to never to lose sight of this fact, in among all the rote tasks. It is the same for a DBA. Not matter how seemingly dull is the task at hand, try to put yourself in the shoes of the end user, and imagine how your input will affect the experience he or she will have with the database you're building, and how that may affect the world beyond the bits stored in your database. Then, despite the inevitable rush to be "done", find time to go the extra mile and hone the design so that it delivers something as close to that imagined experience as you can get. OK, our output still can't and won't reach the same spectacular heights as the "Journey into The Imagination" ride at EPCOT Theme Park in Orlando, where I first heard "One Little Spark". However, our imaginative sparks and efforts can, and will, make a difference to the user who now feels slightly more at home with a database application, or to the manager holding a report presented with enough clarity to drive an interesting decision or two. They are small victories, but worth having, and appreciated, or at least that's how I imagine it.

  Read the article

• JBossMQ - Clustered Queues/NameNotFoundException: QueueConnectionFactory error

  - by mfarver

  I am trying to get an application working on a JBoss Cluster. It uses Queues internally, and the developer claims that it should work correctly in a clustered environment. I have jbossmq setup as a ha-singleton on the cluster. The application works correctly on whichever node currently is running the queue, but fails on the other nodes with a: "javax.naming.NameNotFoundException: QueueConnectionFactory not bound" error. I can look at JNDIview from the jmx-console and see that indeed the QueueConnectionFactory class only appears on the primary node in the Global context. Is there a way to see the Cluster's JNDI listing instead of each server? The steps I took from a default Jboss 4.2.3.GA installation were to use the "all" configuration. Then removed /server/all/deploy/hsqldb-ds.xml and /deploy-hasingleton/jms/hsqldb-jdbc2-service.xml, copying the example/jms/mysq-jdbc2-service.xml file into its place (editing that file to use DefaultDS instead of MySqlDS). Finally I created a mysql-ds.xml file in the deploy directory pointing "DefaultDS" at an empty database. I created a -services.xml file in the deploy directory with the queue definition. like the one below: <server> <mbean code="org.jboss.mq.server.jmx.Queue" name="jboss.mq.destination:service=Queue,name=myfirstqueue"> <depends optional-attribute-name="DestinationManager"> jboss.mq:service=DestinationManager </depends> </mbean> </server> All of the other cluster features of working, the servers list each other in the view, and sessions are replicating back and forth. The JBoss documentation is somewhat light in this area, is there another setting I might have missed? Or is this likely to be a code issue (is there different code to do a JNDI lookup in a clusted environment?) Thanks

  Read the article

• using pf for packet filtering and ipfw's dummynet for bandwidth limiting at the same time

  - by krdx

  I would like to ask if it's fine to use pf for all packet filtering (including using altq for traffic shaping) and ipfw's dummynet for bandwidth limiting certain IPs or subnets at the same time. I am using FreeBSD 10 and I couldn't find a definitive answer to this. Googling returns such results as: It works It doesn't work Might work but it's not stable and not recommended It can work as long as you load the kernel modules in the right order It used to work but with recent FreeBSD versions it doesn't You can make it work provided you use a patch from pfsense Then there's a mention that this patch might had been merged back to FreeBSD, but I can't find it. One certain thing is that pfsense uses both firewalls simultaneously so the question is, is it possible with stock FreeBSD 10 (and where to obtain the patch if it's still necessary). For reference here's a sample of what I have for now and how I load things /etc/rc.conf ifconfig_vtnet0="inet 80.224.45.100 netmask 255.255.255.0 -rxcsum -txcsum" ifconfig_vtnet1="inet 10.20.20.1 netmask 255.255.255.0 -rxcsum -txcsum" defaultrouter="80.224.45.1" gateway_enable="YES" firewall_enable="YES" firewall_script="/etc/ipfw.rules" pf_enable="YES" pf_rules="/etc/pf.conf" /etc/pf.conf WAN1="vtnet0" LAN1="vtnet1" set skip on lo0 set block-policy return scrub on $WAN1 all fragment reassemble scrub on $LAN1 all fragment reassemble altq on $WAN1 hfsc bandwidth 30Mb queue { q_ssh, q_default } queue q_ssh bandwidth 10% priority 2 hfsc (upperlimit 99%) queue q_default bandwidth 90% priority 1 hfsc (default upperlimit 99%) nat on $WAN1 from $LAN1:network to any -> ($WAN1) block in all block out all antispoof quick for $WAN1 antispoof quick for $LAN1 pass in on $WAN1 inet proto icmp from any to $WAN1 keep state pass in on $WAN1 proto tcp from any to $WAN1 port www pass in on $WAN1 proto tcp from any to $WAN1 port ssh pass out quick on $WAN1 proto tcp from $WAN1 to any port ssh queue q_ssh keep state pass out on $WAN1 keep state pass in on $LAN1 from $LAN1:network to any keep state /etc/ipfw.rules ipfw -q -f flush ipfw -q add 65534 allow all from any to any ipfw -q pipe 1 config bw 2048KBit/s ipfw -q pipe 2 config bw 2048KBit/s ipfw -q add pipe 1 ip from any to 10.20.20.4 via vtnet1 out ipfw -q add pipe 2 ip from 10.20.20.4 to any via vtnet1 in

  Read the article

• JBossMQ - Clustered Queues/NameNotFoundException: QueueConnectionFactory error

  - by mfarver

  I am trying to get an application working on a JBoss Cluster. It uses Queues internally, and the developer claims that it should work correctly in a clustered environment. I have jbossmq setup as a ha-singleton on the cluster. The application works correctly on whichever node currently is running the queue, but fails on the other nodes with a: "javax.naming.NameNotFoundException: QueueConnectionFactory not bound" error. I can look at JNDIview from the jmx-console and see that indeed the QueueConnectionFactory class only appears on the primary node in the Global context. Is there a way to see the Cluster's JNDI listing instead of each server? The steps I took from a default Jboss 4.2.3.GA installation were to use the "all" configuration. Then removed /server/all/deploy/hsqldb-ds.xml and /deploy-hasingleton/jms/hsqldb-jdbc2-service.xml, copying the example/jms/mysq-jdbc2-service.xml file into its place (editing that file to use DefaultDS instead of MySqlDS). Finally I created a mysql-ds.xml file in the deploy directory pointing "DefaultDS" at an empty database. I created a -services.xml file in the deploy directory with the queue definition. like the one below: <server> <mbean code="org.jboss.mq.server.jmx.Queue" name="jboss.mq.destination:service=Queue,name=myfirstqueue"> <depends optional-attribute-name="DestinationManager"> jboss.mq:service=DestinationManager </depends> </mbean> </server> All of the other cluster features of working, the servers list each other in the view, and sessions are replicating back and forth. The JBoss documentation is somewhat light in this area, is there another setting I might have missed? Or is this likely to be a code issue (is there different code to do a JNDI lookup in a clusted environment?) Thanks

  Read the article

• Linux - real-world hardware RAID controller tuning (scsi and cciss)

  - by ewwhite

  Most of the Linux systems I manage feature hardware RAID controllers (mostly HP Smart Array). They're all running RHEL or CentOS. I'm looking for real-world tunables to help optimize performance for setups that incorporate hardware RAID controllers with SAS disks (Smart Array, Perc, LSI, etc.) and battery-backed or flash-backed cache. Assume RAID 1+0 and multiple spindles (4+ disks). I spend a considerable amount of time tuning Linux network settings for low-latency and financial trading applications. But many of those options are well-documented (changing send/receive buffers, modifying TCP window settings, etc.). What are engineers doing on the storage side? Historically, I've made changes to the I/O scheduling elevator, recently opting for the deadline and noop schedulers to improve performance within my applications. As RHEL versions have progressed, I've also noticed that the compiled-in defaults for SCSI and CCISS block devices have changed as well. This has had an impact on the recommended storage subsystem settings over time. However, it's been awhile since I've seen any clear recommendations. And I know that the OS defaults aren't optimal. For example, it seems that the default read-ahead buffer of 128kb is extremely small for a deployment on server-class hardware. The following articles explore the performance impact of changing read-ahead cache and nr_requests values on the block queues. http://zackreed.me/articles/54-hp-smart-array-p410-controller-tuning http://www.overclock.net/t/515068/tuning-a-hp-smart-array-p400-with-linux-why-tuning-really-matters http://yoshinorimatsunobu.blogspot.com/2009/04/linux-io-scheduler-queue-size-and.html For example, these are suggested changes for an HP Smart Array RAID controller: echo "noop" > /sys/block/cciss\!c0d0/queue/scheduler blockdev --setra 65536 /dev/cciss/c0d0 echo 512 > /sys/block/cciss\!c0d0/queue/nr_requests echo 2048 > /sys/block/cciss\!c0d0/queue/read_ahead_kb What else can be reliably tuned to improve storage performance? I'm specifically looking for sysctl and sysfs options in production scenarios.

  Read the article

• Possible disk IO issue

  - by Tim Meers

  I've been trying to really figure out what my IOPS are on my DB server array and see if it's just too much. The array is four 72.6gb 15k rpm drives in RAID 5. To calculate IOPS for RAID 5 the following formula is used: (reads + (4 * Writes)) / Number of disks = total IOPS. The formula is from MSDN. I also want to calculate the Avg Queue Length but I'm not sure where they are getting the formula from, but i think it reads on that page as avg que length/number of disks = actual queue. To populate that formula I used the perfmon to gather the needed information. I came up with this, under normal production load: (873.982 + (4 * 28.999)) / 4 = 247.495. Also the disk queue lengh of 14.454/4 = 3.614. So to the question, am I wrong in thinking this array has a very high disk IO? Edit I got the chance to review it again this morning under normal/high load. This time with even bigger numbers and IOPS in excess of 600 for about 5 minutes then it died down again. But I also took a look at the Avg sec/Transfer, %Disk Time, and %Idle Time. These number were taken when the reads/writes per sec were only 332.997/17.999 respectively. %Disk Time: 219.436 %Idle Time: 0.300 Avg Disk Queue Length: 2.194 Avg Disk sec/Transfer: 0.006 Pages/sec: 2927.802 % Processor Time: 21.877 Edit (again) Looks like I have that issue solved. Thanks for the help. Also for a pretty slick parser I found this: http://pal.codeplex.com/ It works pretty well for breaking down the data into something usable.

  Read the article

• 256 Windows Azure Worker Roles, Windows Kinect and a 90's Text-Based Ray-Tracer

  - by Alan Smith

  For a couple of years I have been demoing a simple render farm hosted in Windows Azure using worker roles and the Azure Storage service. At the start of the presentation I deploy an Azure application that uses 16 worker roles to render a 1,500 frame 3D ray-traced animation. At the end of the presentation, when the animation was complete, I would play the animation delete the Azure deployment. The standing joke with the audience was that it was that it was a “$2 demo”, as the compute charges for running the 16 instances for an hour was $1.92, factor in the bandwidth charges and it’s a couple of dollars. The point of the demo is that it highlights one of the great benefits of cloud computing, you pay for what you use, and if you need massive compute power for a short period of time using Windows Azure can work out very cost effective. The “$2 demo” was great for presenting at user groups and conferences in that it could be deployed to Azure, used to render an animation, and then removed in a one hour session. I have always had the idea of doing something a bit more impressive with the demo, and scaling it from a “$2 demo” to a “$30 demo”. The challenge was to create a visually appealing animation in high definition format and keep the demo time down to one hour.  This article will take a run through how I achieved this. Ray Tracing Ray tracing, a technique for generating high quality photorealistic images, gained popularity in the 90’s with companies like Pixar creating feature length computer animations, and also the emergence of shareware text-based ray tracers that could run on a home PC. In order to render a ray traced image, the ray of light that would pass from the view point must be tracked until it intersects with an object. At the intersection, the color, reflectiveness, transparency, and refractive index of the object are used to calculate if the ray will be reflected or refracted. Each pixel may require thousands of calculations to determine what color it will be in the rendered image. Pin-Board Toys Having very little artistic talent and a basic understanding of maths I decided to focus on an animation that could be modeled fairly easily and would look visually impressive. I’ve always liked the pin-board desktop toys that become popular in the 80’s and when I was working as a 3D animator back in the 90’s I always had the idea of creating a 3D ray-traced animation of a pin-board, but never found the energy to do it. Even if I had a go at it, the render time to produce an animation that would look respectable on a 486 would have been measured in months. PolyRay Back in 1995 I landed my first real job, after spending three years being a beach-ski-climbing-paragliding-bum, and was employed to create 3D ray-traced animations for a CD-ROM that school kids would use to learn physics. I had got into the strange and wonderful world of text-based ray tracing, and was using a shareware ray-tracer called PolyRay. PolyRay takes a text file describing a scene as input and, after a few hours processing on a 486, produced a high quality ray-traced image. The following is an example of a basic PolyRay scene file. background Midnight_Blue   static define matte surface { ambient 0.1 diffuse 0.7 } define matte_white texture { matte { color white } } define matte_black texture { matte { color dark_slate_gray } } define position_cylindrical 3 define lookup_sawtooth 1 define light_wood <0.6, 0.24, 0.1> define median_wood <0.3, 0.12, 0.03> define dark_wood <0.05, 0.01, 0.005>     define wooden texture { noise surface { ambient 0.2  diffuse 0.7  specular white, 0.5 microfacet Reitz 10 position_fn position_cylindrical position_scale 1  lookup_fn lookup_sawtooth octaves 1 turbulence 1 color_map( [0.0, 0.2, light_wood, light_wood] [0.2, 0.3, light_wood, median_wood] [0.3, 0.4, median_wood, light_wood] [0.4, 0.7, light_wood, light_wood] [0.7, 0.8, light_wood, median_wood] [0.8, 0.9, median_wood, light_wood] [0.9, 1.0, light_wood, dark_wood]) } } define glass texture { surface { ambient 0 diffuse 0 specular 0.2 reflection white, 0.1 transmission white, 1, 1.5 }} define shiny surface { ambient 0.1 diffuse 0.6 specular white, 0.6 microfacet Phong 7  } define steely_blue texture { shiny { color black } } define chrome texture { surface { color white ambient 0.0 diffuse 0.2 specular 0.4 microfacet Phong 10 reflection 0.8 } }   viewpoint {     from <4.000, -1.000, 1.000> at <0.000, 0.000, 0.000> up <0, 1, 0> angle 60     resolution 640, 480 aspect 1.6 image_format 0 }       light <-10, 30, 20> light <-10, 30, -20>   object { disc <0, -2, 0>, <0, 1, 0>, 30 wooden }   object { sphere <0.000, 0.000, 0.000>, 1.00 chrome } object { cylinder <0.000, 0.000, 0.000>, <0.000, 0.000, -4.000>, 0.50 chrome }   After setting up the background and defining colors and textures, the viewpoint is specified. The “camera” is located at a point in 3D space, and it looks towards another point. The angle, image resolution, and aspect ratio are specified. Two lights are present in the image at defined coordinates. The three objects in the image are a wooden disc to represent a table top, and a sphere and cylinder that intersect to form a pin that will be used for the pin board toy in the final animation. When the image is rendered, the following image is produced. The pins are modeled with a chrome surface, so they reflect the environment around them. Note that the scale of the pin shaft is not correct, this will be fixed later. Modeling the Pin Board The frame of the pin-board is made up of three boxes, and six cylinders, the front box is modeled using a clear, slightly reflective solid, with the same refractive index of glass. The other shapes are modeled as metal. object { box <-5.5, -1.5, 1>, <5.5, 5.5, 1.2> glass } object { box <-5.5, -1.5, -0.04>, <5.5, 5.5, -0.09> steely_blue } object { box <-5.5, -1.5, -0.52>, <5.5, 5.5, -0.59> steely_blue } object { cylinder <-5.2, -1.2, 1.4>, <-5.2, -1.2, -0.74>, 0.2 steely_blue } object { cylinder <5.2, -1.2, 1.4>, <5.2, -1.2, -0.74>, 0.2 steely_blue } object { cylinder <-5.2, 5.2, 1.4>, <-5.2, 5.2, -0.74>, 0.2 steely_blue } object { cylinder <5.2, 5.2, 1.4>, <5.2, 5.2, -0.74>, 0.2 steely_blue } object { cylinder <0, -1.2, 1.4>, <0, -1.2, -0.74>, 0.2 steely_blue } object { cylinder <0, 5.2, 1.4>, <0, 5.2, -0.74>, 0.2 steely_blue }   In order to create the matrix of pins that make up the pin board I used a basic console application with a few nested loops to create two intersecting matrixes of pins, which models the layout used in the pin boards. The resulting image is shown below. The pin board contains 11,481 pins, with the scene file containing 23,709 lines of code. For the complete animation 2,000 scene files will be created, which is over 47 million lines of code. Each pin in the pin-board will slide out a specific distance when an object is pressed into the back of the board. This is easily modeled by setting the Z coordinate of the pin to a specific value. In order to set all of the pins in the pin-board to the correct position, a bitmap image can be used. The position of the pin can be set based on the color of the pixel at the appropriate position in the image. When the Windows Azure logo is used to set the Z coordinate of the pins, the following image is generated. The challenge now was to make a cool animation. The Azure Logo is fine, but it is static. Using a normal video to animate the pins would not work; the colors in the video would not be the same as the depth of the objects from the camera. In order to simulate the pin board accurately a series of frames from a depth camera could be used. Windows Kinect The Kenect controllers for the X-Box 360 and Windows feature a depth camera. The Kinect SDK for Windows provides a programming interface for Kenect, providing easy access for .NET developers to the Kinect sensors. The Kinect Explorer provided with the Kinect SDK is a great starting point for exploring Kinect from a developers perspective. Both the X-Box 360 Kinect and the Windows Kinect will work with the Kinect SDK, the Windows Kinect is required for commercial applications, but the X-Box Kinect can be used for hobby projects. The Windows Kinect has the advantage of providing a mode to allow depth capture with objects closer to the camera, which makes for a more accurate depth image for setting the pin positions. Creating a Depth Field Animation The depth field animation used to set the positions of the pin in the pin board was created using a modified version of the Kinect Explorer sample application. In order to simulate the pin board accurately, a small section of the depth range from the depth sensor will be used. Any part of the object in front of the depth range will result in a white pixel; anything behind the depth range will be black. Within the depth range the pixels in the image will be set to RGB values from 0,0,0 to 255,255,255. A screen shot of the modified Kinect Explorer application is shown below. The Kinect Explorer sample application was modified to include slider controls that are used to set the depth range that forms the image from the depth stream. This allows the fine tuning of the depth image that is required for simulating the position of the pins in the pin board. The Kinect Explorer was also modified to record a series of images from the depth camera and save them as a sequence JPEG files that will be used to animate the pins in the animation the Start and Stop buttons are used to start and stop the image recording. En example of one of the depth images is shown below. Once a series of 2,000 depth images has been captured, the task of creating the animation can begin. Rendering a Test Frame In order to test the creation of frames and get an approximation of the time required to render each frame a test frame was rendered on-premise using PolyRay. The output of the rendering process is shown below. The test frame contained 23,629 primitive shapes, most of which are the spheres and cylinders that are used for the 11,800 or so pins in the pin board. The 1280x720 image contains 921,600 pixels, but as anti-aliasing was used the number of rays that were calculated was 4,235,777, with 3,478,754,073 object boundaries checked. The test frame of the pin board with the depth field image applied is shown below. The tracing time for the test frame was 4 minutes 27 seconds, which means rendering the2,000 frames in the animation would take over 148 hours, or a little over 6 days. Although this is much faster that an old 486, waiting almost a week to see the results of an animation would make it challenging for animators to create, view, and refine their animations. It would be much better if the animation could be rendered in less than one hour. Windows Azure Worker Roles The cost of creating an on-premise render farm to render animations increases in proportion to the number of servers. The table below shows the cost of servers for creating a render farm, assuming a cost of $500 per server. Number of Servers Cost 1 $500 16 $8,000 256 $128,000   As well as the cost of the servers, there would be additional costs for networking, racks etc. Hosting an environment of 256 servers on-premise would require a server room with cooling, and some pretty hefty power cabling. The Windows Azure compute services provide worker roles, which are ideal for performing processor intensive compute tasks. With the scalability available in Windows Azure a job that takes 256 hours to complete could be perfumed using different numbers of worker roles. The time and cost of using 1, 16 or 256 worker roles is shown below. Number of Worker Roles Render Time Cost 1 256 hours $30.72 16 16 hours $30.72 256 1 hour $30.72   Using worker roles in Windows Azure provides the same cost for the 256 hour job, irrespective of the number of worker roles used. Provided the compute task can be broken down into many small units, and the worker role compute power can be used effectively, it makes sense to scale the application so that the task is completed quickly, making the results available in a timely fashion. The task of rendering 2,000 frames in an animation is one that can easily be broken down into 2,000 individual pieces, which can be performed by a number of worker roles. Creating a Render Farm in Windows Azure The architecture of the render farm is shown in the following diagram. The render farm is a hybrid application with the following components: ·         On-Premise o   Windows Kinect – Used combined with the Kinect Explorer to create a stream of depth images. o   Animation Creator – This application uses the depth images from the Kinect sensor to create scene description files for PolyRay. These files are then uploaded to the jobs blob container, and job messages added to the jobs queue. o   Process Monitor – This application queries the role instance lifecycle table and displays statistics about the render farm environment and render process. o   Image Downloader – This application polls the image queue and downloads the rendered animation files once they are complete. ·         Windows Azure o   Azure Storage – Queues and blobs are used for the scene description files and completed frames. A table is used to store the statistics about the rendering environment.   The architecture of each worker role is shown below.   The worker role is configured to use local storage, which provides file storage on the worker role instance that can be use by the applications to render the image and transform the format of the image. The service definition for the worker role with the local storage configuration highlighted is shown below. <?xml version="1.0" encoding="utf-8"?> <ServiceDefinition name="CloudRay" >   <WorkerRole name="CloudRayWorkerRole" vmsize="Small">     <Imports>     </Imports>     <ConfigurationSettings>       <Setting name="DataConnectionString" />     </ConfigurationSettings>     <LocalResources>       <LocalStorage name="RayFolder" cleanOnRoleRecycle="true" />     </LocalResources>   </WorkerRole> </ServiceDefinition>     The two executable programs, PolyRay.exe and DTA.exe are included in the Azure project, with Copy Always set as the property. PolyRay will take the scene description file and render it to a Truevision TGA file. As the TGA format has not seen much use since the mid 90’s it is converted to a JPG image using Dave's Targa Animator, another shareware application from the 90’s. Each worker roll will use the following process to render the animation frames. 1.       The worker process polls the job queue, if a job is available the scene description file is downloaded from blob storage to local storage. 2.       PolyRay.exe is started in a process with the appropriate command line arguments to render the image as a TGA file. 3.       DTA.exe is started in a process with the appropriate command line arguments convert the TGA file to a JPG file. 4.       The JPG file is uploaded from local storage to the images blob container. 5.       A message is placed on the images queue to indicate a new image is available for download. 6.       The job message is deleted from the job queue. 7.       The role instance lifecycle table is updated with statistics on the number of frames rendered by the worker role instance, and the CPU time used. The code for this is shown below. public override void Run() {     // Set environment variables     string polyRayPath = Path.Combine(Environment.GetEnvironmentVariable("RoleRoot"), PolyRayLocation);     string dtaPath = Path.Combine(Environment.GetEnvironmentVariable("RoleRoot"), DTALocation);       LocalResource rayStorage = RoleEnvironment.GetLocalResource("RayFolder");     string localStorageRootPath = rayStorage.RootPath;       JobQueue jobQueue = new JobQueue("renderjobs");     JobQueue downloadQueue = new JobQueue("renderimagedownloadjobs");     CloudRayBlob sceneBlob = new CloudRayBlob("scenes");     CloudRayBlob imageBlob = new CloudRayBlob("images");     RoleLifecycleDataSource roleLifecycleDataSource = new RoleLifecycleDataSource();       Frames = 0;       while (true)     {         // Get the render job from the queue         CloudQueueMessage jobMsg = jobQueue.Get();           if (jobMsg != null)         {             // Get the file details             string sceneFile = jobMsg.AsString;             string tgaFile = sceneFile.Replace(".pi", ".tga");             string jpgFile = sceneFile.Replace(".pi", ".jpg");               string sceneFilePath = Path.Combine(localStorageRootPath, sceneFile);             string tgaFilePath = Path.Combine(localStorageRootPath, tgaFile);             string jpgFilePath = Path.Combine(localStorageRootPath, jpgFile);               // Copy the scene file to local storage             sceneBlob.DownloadFile(sceneFilePath);               // Run the ray tracer.             string polyrayArguments =                 string.Format("\"{0}\" -o \"{1}\" -a 2", sceneFilePath, tgaFilePath);             Process polyRayProcess = new Process();             polyRayProcess.StartInfo.FileName =                 Path.Combine(Environment.GetEnvironmentVariable("RoleRoot"), polyRayPath);             polyRayProcess.StartInfo.Arguments = polyrayArguments;             polyRayProcess.Start();             polyRayProcess.WaitForExit();               // Convert the image             string dtaArguments =                 string.Format(" {0} /FJ /P{1}", tgaFilePath, Path.GetDirectoryName (jpgFilePath));             Process dtaProcess = new Process();             dtaProcess.StartInfo.FileName =                 Path.Combine(Environment.GetEnvironmentVariable("RoleRoot"), dtaPath);             dtaProcess.StartInfo.Arguments = dtaArguments;             dtaProcess.Start();             dtaProcess.WaitForExit();               // Upload the image to blob storage             imageBlob.UploadFile(jpgFilePath);               // Add a download job.             downloadQueue.Add(jpgFile);               // Delete the render job message             jobQueue.Delete(jobMsg);               Frames++;         }         else         {             Thread.Sleep(1000);         }           // Log the worker role activity.         roleLifecycleDataSource.Alive             ("CloudRayWorker", RoleLifecycleDataSource.RoleLifecycleId, Frames);     } }     Monitoring Worker Role Instance Lifecycle In order to get more accurate statistics about the lifecycle of the worker role instances used to render the animation data was tracked in an Azure storage table. The following class was used to track the worker role lifecycles in Azure storage.   public class RoleLifecycle : TableServiceEntity {     public string ServerName { get; set; }     public string Status { get; set; }     public DateTime StartTime { get; set; }     public DateTime EndTime { get; set; }     public long SecondsRunning { get; set; }     public DateTime LastActiveTime { get; set; }     public int Frames { get; set; }     public string Comment { get; set; }       public RoleLifecycle()     {     }       public RoleLifecycle(string roleName)     {         PartitionKey = roleName;         RowKey = Utils.GetAscendingRowKey();         Status = "Started";         StartTime = DateTime.UtcNow;         LastActiveTime = StartTime;         EndTime = StartTime;         SecondsRunning = 0;         Frames = 0;     } }     A new instance of this class is created and added to the storage table when the role starts. It is then updated each time the worker renders a frame to record the total number of frames rendered and the total processing time. These statistics are used be the monitoring application to determine the effectiveness of use of resources in the render farm. Rendering the Animation The Azure solution was deployed to Windows Azure with the service configuration set to 16 worker role instances. This allows for the application to be tested in the cloud environment, and the performance of the application determined. When I demo the application at conferences and user groups I often start with 16 instances, and then scale up the application to the full 256 instances. The configuration to run 16 instances is shown below. <?xml version="1.0" encoding="utf-8"?> <ServiceConfiguration serviceName="CloudRay" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" osFamily="1" osVersion="*">   <Role name="CloudRayWorkerRole">     <Instances count="16" />     <ConfigurationSettings>       <Setting name="DataConnectionString"         value="DefaultEndpointsProtocol=https;AccountName=cloudraydata;AccountKey=..." />     </ConfigurationSettings>   </Role> </ServiceConfiguration>     About six minutes after deploying the application the first worker roles become active and start to render the first frames of the animation. The CloudRay Monitor application displays an icon for each worker role instance, with a number indicating the number of frames that the worker role has rendered. The statistics on the left show the number of active worker roles and statistics about the render process. The render time is the time since the first worker role became active; the CPU time is the total amount of processing time used by all worker role instances to render the frames.   Five minutes after the first worker role became active the last of the 16 worker roles activated. By this time the first seven worker roles had each rendered one frame of the animation.   With 16 worker roles u and running it can be seen that one hour and 45 minutes CPU time has been used to render 32 frames with a render time of just under 10 minutes.     At this rate it would take over 10 hours to render the 2,000 frames of the full animation. In order to complete the animation in under an hour more processing power will be required. Scaling the render farm from 16 instances to 256 instances is easy using the new management portal. The slider is set to 256 instances, and the configuration saved. We do not need to re-deploy the application, and the 16 instances that are up and running will not be affected. Alternatively, the configuration file for the Azure service could be modified to specify 256 instances.   <?xml version="1.0" encoding="utf-8"?> <ServiceConfiguration serviceName="CloudRay" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" osFamily="1" osVersion="*">   <Role name="CloudRayWorkerRole">     <Instances count="256" />     <ConfigurationSettings>       <Setting name="DataConnectionString"         value="DefaultEndpointsProtocol=https;AccountName=cloudraydata;AccountKey=..." />     </ConfigurationSettings>   </Role> </ServiceConfiguration>     Six minutes after the new configuration has been applied 75 new worker roles have activated and are processing their first frames.   Five minutes later the full configuration of 256 worker roles is up and running. We can see that the average rate of frame rendering has increased from 3 to 12 frames per minute, and that over 17 hours of CPU time has been utilized in 23 minutes. In this test the time to provision 140 worker roles was about 11 minutes, which works out at about one every five seconds.   We are now half way through the rendering, with 1,000 frames complete. This has utilized just under three days of CPU time in a little over 35 minutes.   The animation is now complete, with 2,000 frames rendered in a little over 52 minutes. The CPU time used by the 256 worker roles is 6 days, 7 hours and 22 minutes with an average frame rate of 38 frames per minute. The rendering of the last 1,000 frames took 16 minutes 27 seconds, which works out at a rendering rate of 60 frames per minute. The frame counts in the server instances indicate that the use of a queue to distribute the workload has been very effective in distributing the load across the 256 worker role instances. The first 16 instances that were deployed first have rendered between 11 and 13 frames each, whilst the 240 instances that were added when the application was scaled have rendered between 6 and 9 frames each.   Completed Animation I’ve uploaded the completed animation to YouTube, a low resolution preview is shown below. Pin Board Animation Created using Windows Kinect and 256 Windows Azure Worker Roles   The animation can be viewed in 1280x720 resolution at the following link: http://www.youtube.com/watch?v=n5jy6bvSxWc Effective Use of Resources According to the CloudRay monitor statistics the animation took 6 days, 7 hours and 22 minutes CPU to render, this works out at 152 hours of compute time, rounded up to the nearest hour. As the usage for the worker role instances are billed for the full hour, it may have been possible to render the animation using fewer than 256 worker roles. When deciding the optimal usage of resources, the time required to provision and start the worker roles must also be considered. In the demo I started with 16 worker roles, and then scaled the application to 256 worker roles. It would have been more optimal to start the application with maybe 200 worker roles, and utilized the full hour that I was being billed for. This would, however, have prevented showing the ease of scalability of the application. The new management portal displays the CPU usage across the worker roles in the deployment. The average CPU usage across all instances is 93.27%, with over 99% used when all the instances are up and running. This shows that the worker role resources are being used very effectively. Grid Computing Scenarios Although I am using this scenario for a hobby project, there are many scenarios where a large amount of compute power is required for a short period of time. Windows Azure provides a great platform for developing these types of grid computing applications, and can work out very cost effective. ·         Windows Azure can provide massive compute power, on demand, in a matter of minutes. ·         The use of queues to manage the load balancing of jobs between role instances is a simple and effective solution. ·         Using a cloud-computing platform like Windows Azure allows proof-of-concept scenarios to be tested and evaluated on a very low budget. ·         No charges for inbound data transfer makes the uploading of large data sets to Windows Azure Storage services cost effective. (Transaction charges still apply.) Tips for using Windows Azure for Grid Computing Scenarios I found the implementation of a render farm using Windows Azure a fairly simple scenario to implement. I was impressed by ease of scalability that Azure provides, and by the short time that the application took to scale from 16 to 256 worker role instances. In this case it was around 13 minutes, in other tests it took between 10 and 20 minutes. The following tips may be useful when implementing a grid computing project in Windows Azure. ·         Using an Azure Storage queue to load-balance the units of work across multiple worker roles is simple and very effective. The design I have used in this scenario could easily scale to many thousands of worker role instances. ·         Windows Azure accounts are typically limited to 20 cores. If you need to use more than this, a call to support and a credit card check will be required. ·         Be aware of how the billing model works. You will be charged for worker role instances for the full clock our in which the instance is deployed. Schedule the workload to start just after the clock hour has started. ·         Monitor the utilization of the resources you are provisioning, ensure that you are not paying for worker roles that are idle. ·         If you are deploying third party applications to worker roles, you may well run into licensing issues. Purchasing software licenses on a per-processor basis when using hundreds of processors for a short time period would not be cost effective. ·         Third party software may also require installation onto the worker roles, which can be accomplished using start-up tasks. Bear in mind that adding a startup task and possible re-boot will add to the time required for the worker role instance to start and activate. An alternative may be to use a prepared VM and use VM roles. ·         Consider using the Windows Azure Autoscaling Application Block (WASABi) to autoscale the worker roles in your application. When using a large number of worker roles, the utilization must be carefully monitored, if the scaling algorithms are not optimal it could get very expensive!

  Read the article

• Problem with custom Equality in Entity Framework

  - by Shimmy

  Hello! I am using Entity Framework in my application. I implemented with the partial class of an entity the IEquatable<T> interface: Partial Class Address : Implements IEquatable(Of Address) 'Other part generated Public Overloads Function Equals(ByVal other As Address) As Boolean _ Implements System.IEquatable(Of Address).Equals If ReferenceEquals(Me, other) Then Return True Return AddressId = other.AddressId End Function Public Overrides Function Equals(ByVal obj As Object) As Boolean If obj Is Nothing Then Return MyBase.Equals(obj) If TypeOf obj Is Address Then Return Equals(DirectCast(obj, Address)) Else Return False End Function Public Overrides Function GetHashCode() As Integer Return AddressId.GetHashCode End Function End Class Now in my code I use it this way: Sub Main() Using e As New CompleteKitchenEntities Dim job = e.Job.FirstOrDefault Dim address As New Address() job.Addresses.Add(address) Dim contains1 = job.Addresses.Contains(address) 'True e.SaveChanges() Dim contains2 = job.Addresses.Contains(address) 'False 'The problem is that I can't remove it: Dim removed = job.Addresses.Remoeve(address) 'False End Using End Sub Note (I checked in the debugger visualizer) that the EntityCollection class stores its entities in HashSet so it has to do with the GetHashCode function, I want it to depend on the ID so entities are compared by their IDs. Please help me find what's wrong in the GetHashCode function (by ID) and what can I change to make it work. Thanks a lot.

  Read the article

• Linq PredicateBuilder with conditional AND, OR and NOT filters.

  - by richeym

  We have a project using LINQ to SQL, for which I need to rewrite a couple of search pages to allow the client to select whether they wish to perform an and or an or search. I though about redoing the LINQ queries using PredicateBuilder and have got this working pretty well I think. I effectively have a class containing my predicates, e.g.: internal static Expression<Func<Job, bool>> Description(string term) { return p => p.Description.Contains(term); } To perform the search i'm doing this (some code omitted for brevity): public Expression<Func<Job, bool>> ToLinqExpression() { var predicates = new List<Expression<Func<Job, bool>>>(); // build up predicates here if (SearchType == SearchType.And) { query = PredicateBuilder.True<Job>(); } else { query = PredicateBuilder.False<Job>(); } foreach (var predicate in predicates) { if (SearchType == SearchType.And) { query = query.And(predicate); } else { query = query.Or(predicate); } } return query; } While i'm reasonably happy with this, I have two concerns: The if/else blocks that evaluate a SearchType property feel like they could be a potential code smell. The client is now insisting on being able to perform 'and not' / 'or not' searches. To address point 2, I think I could do this by simply rewriting my expressions, e.g.: internal static Expression<Func<Job, bool>> Description(string term, bool invert) { if (invert) { return p => !p.Description.Contains(term); } else { return p => p.Description.Contains(term); } } However this feels like a bit of a kludge, which usually means there's a better solution out there. Can anyone recommend how this could be improved? I'm aware of dynamic LINQ, but I don't really want to lose LINQ's strong typing.

  Read the article

• Making that move from junior > mid level

  - by dotnetdev

  Hi, Before I start, I know there is another thread about this very issue (http://stackoverflow.com/questions/2352874/moving-from-junior-developer-to-mid-level). I am in this very same situation, but of course every person and the company/employment-history is not the same. In my current company, I have not done one piece of coding from start to finish with the oversight of my manager and a Project Manager to manage the work/deadlines etc. I am basically an odd-jobs type of guy. The coding I do is on the side to whatever boring spreadsheet/word document I have to write. Very illogical that you're a coder and you're doing it in secret. In another job I had for 3 months (Was made redundant), it required 1 years experience, perhaps because of the fact I was the sole developer. It wasn't too hard, but then I was solely responsible and I learnt a lot from that. I had 2 other 3 months jobs (contracts), so I have been working for 1 year 9 months. I know found a job which I'm in the last stage for, which needs 3 years .NET experience and 2 years Sharepoint. How can I know if I am ready for this job? My current job has been going on for 1 year, but it doesn't mean squat apart from explaining how I have spent my time. It does not tell me what level I am at (apart from the huge skills gap I have opened up against my peers because I practise at home). So 1 year of doing nothing at work, but 1 year of doing loads at home. In fact, I take 1 week off and do more at home then in the company since I started. How can I know if I am ready for such a job? I am generally very confident given all I've achieved in coding, but I have no idea what a job with this sort of experience entails (what day-to-day-problems I would be facing). Is there any advice on how to handle this transition? Thanks

  Read the article

• Problem with custom Equality and GetHashCode in a mutable object

  - by Shimmy

  Hello! I am using Entity Framework in my application. I implemented with the partial class of an entity the IEquatable<T> interface: Partial Class Address : Implements IEquatable(Of Address) 'Other part generated Public Overloads Function Equals(ByVal other As Address) As Boolean _ Implements System.IEquatable(Of Address).Equals If ReferenceEquals(Me, other) Then Return True Return AddressId = other.AddressId End Function Public Overrides Function Equals(ByVal obj As Object) As Boolean If obj Is Nothing Then Return MyBase.Equals(obj) If TypeOf obj Is Address Then Return Equals(DirectCast(obj, Address)) Else Return False End Function Public Overrides Function GetHashCode() As Integer Return AddressId.GetHashCode End Function End Class Now in my code I use it this way: Sub Main() Using e As New CompleteKitchenEntities Dim job = e.Job.FirstOrDefault Dim address As New Address() job.Addresses.Add(address) Dim contains1 = job.Addresses.Contains(address) 'True e.SaveChanges() Dim contains2 = job.Addresses.Contains(address) 'False 'The problem is that I can't remove it: Dim removed = job.Addresses.Remoeve(address) 'False End Using End Sub Note (I checked in the debugger visualizer) that the EntityCollection class stores its entities in HashSet so it has to do with the GetHashCode function, I want it to depend on the ID so entities are compared by their IDs. The problem is that when I hit save, the ID changes from 0 to its db value. So the question is how can I have an equatable object, being properly hashed. Please help me find what's wrong in the GetHashCode function (by ID) and what can I change to make it work. Thanks a lot.

  Read the article

• NMock2.0 - how to stub a non interface call?

  - by dferraro

  Hello, I have a class API which has full code coverage and uses DI to mock out all the logic in the main class function (Job.Run) which does all the work. I found a bug in production where we werent doing some validation on one of the data input fields. So, I added a stub function called ValidateFoo()... Wrote a unit test against this function to Expect a JobFailedException, ran the test - it failed obviously because that function was empty. I added the validation logic, and now the test passes. Great, now we know the validation works. Problem is - how do I write the test to make sure that ValidateFoo() is actually called inside Job.Run()? ValidateFoo() is a private method of the Job class - so it's not an interface... Is there anyway to do this with NMock2.0? I know TypeMock supports fakes of non interface types. But changing mock libs right now is not an option. At this point if NMock can't support it, I will simply just add the ValidateFoo() call to the Run() method and test things manually - which obviously I'd prefer not to do considering my Job.Run() method has 100% coverage right now. Any Advice? Thanks very much it is appreciated. EDIT: the other option I have in mind is to just create an integration test for my Job.Run functionality (injecting to it true implementations of the composite objects instead of mocks). I will give it a bad input value for that field and then validate that the job failed. This works and covers my test - but it's not really a unit test but instead an integration test that tests one unit of functionality.... hmm.. EDIT2: IS there any way to do tihs? Anyone have ideas? Maybe TypeMock - or a better design?

  Read the article

• How to implement IEquatable<T> when mutable fields are part of the equality - Problem with GetHashCo

  - by Shimmy

  Hello! I am using Entity Framework in my application. I implemented with the partial class of an entity the IEquatable<T> interface: Partial Class Address : Implements IEquatable(Of Address) 'Other part generated Public Overloads Function Equals(ByVal other As Address) As Boolean _ Implements System.IEquatable(Of Address).Equals If ReferenceEquals(Me, other) Then Return True Return AddressId = other.AddressId End Function Public Overrides Function Equals(ByVal obj As Object) As Boolean If obj Is Nothing Then Return MyBase.Equals(obj) If TypeOf obj Is Address Then Return Equals(DirectCast(obj, Address)) Else Return False End Function Public Overrides Function GetHashCode() As Integer Return AddressId.GetHashCode End Function End Class Now in my code I use it this way: Sub Main() Using e As New CompleteKitchenEntities Dim job = e.Job.FirstOrDefault Dim address As New Address() job.Addresses.Add(address) Dim contains1 = job.Addresses.Contains(address) 'True e.SaveChanges() Dim contains2 = job.Addresses.Contains(address) 'False 'The problem is that I can't remove it: Dim removed = job.Addresses.Remoeve(address) 'False End Using End Sub Note (I checked in the debugger visualizer) that the EntityCollection class stores its entities in HashSet so it has to do with the GetHashCode function, I want it to depend on the ID so entities are compared by their IDs. The problem is that when I hit save, the ID changes from 0 to its db value. So the question is how can I have an equatable object, being properly hashed. Please help me find what's wrong in the GetHashCode function (by ID) and what can I change to make it work. Thanks a lot.

  Read the article

< Previous Page | 69 70 71 72 73 74 75 76 77 78 79 80  | Next Page >