Search Results

Search found 303 results on 13 pages for 'rac'.

Page 2/13 | < Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >

• ???????/?????????!RAC??????????

  - by user788995

  ????? ??:2012/01/23 ??:??????/?? RAC ??????????????????????????RAC ???????????????????????????????????????????????????????????????????????? RAC ????????????????????? ????RAC ??????????????????????·???????? ????????? ????????????????? http://otndnld.oracle.co.jp/ondemand/otn-seminar/movie/111121_B-10_RACTuning.wmv http://otndnld.oracle.co.jp/ondemand/otn-seminar/movie/mp4/111121_B-10_RACTuning.mp4 http://www.oracle.com/technetwork/jp/ondemand/b-10-ractuning-1448392-ja.pdf

  Read the article

• Is RAC One Node Certified for E-Business Suite?

  - by Steven Chan

  Oracle Real Application Clusters (RAC) is a cluster database with a shared cache architecture that supports the transparent deployment of a single database across a pool of servers.  RAC is certified with both Oracle E-Business Suite Release 11i and 12.  We publish best-practices documentation for specific combinations of EBS + RAC versions.  For example, if you were planning on implementing RAC for EBS 12, you would use this documentation:Using Oracle 11g Release 2 Real Application Clusters with Oracle E-Business Suite Release 12 (Note 823587.1)Many of the largest E-Business Suite users in the world run RAC today, including Oracle; see this Oracle R12 case study for details.A number of customers have recently asked whether RAC One Node can be used with the E-Business Suite.  From the RAC website:Oracle RAC One Node is a new option available with Oracle Database 11g Release 2. Oracle RAC One Node is a single instance of an Oracle RAC-enabled database running on one node in a cluster.

  Read the article

• Oracle RAC interconnect in a Dell M1000e Blade Enclosure

  - by Antitribu

  We are looking at a Dell M1000e enclosure and appropriate Blades with 4 NICs each. We are planning on running Linux/Oracle 11g RAC on two blades, storage will be handled on an iSCSI SAN for which two NICs (via passthrough) will be connected leaving us with two NICs (via blade centre switches). We would like to have an interconnect (obviously) , an external IP and an internal IP. Would best practice be to: bond the remaining two interfaces and VLAN as appropriate to provide three virtual interfaces? run the interconnect on one interface and VLAN the external/internal interfaces? purchase a blade with more NICs as the above is a terrible idea? Another option? Please feel free to point out the blindingly obvious or to relevant documentation on support.oracle. I am specifically interested in supported configurations and best practices. Thanks!

  Read the article

• ???????/???Windows?RAC??!??????????

  - by Yusuke.Yamamoto

  ????? ??:2010/09/01 ??:??????/?? Windows ? RAC ????????????MSCS ???HA????????Oracle Database 11g R2 ?? RAC ???·??????????? Windows Server?Oracle Database????/ Windows ???????? Oracle DatabaseMSCS/MSFC + OFS???HA??????/ OFS ???MSCS/MSFC vs. RACRAC on Windows ?????/ Oracle Database 11g R2 ?? RAC ????11g R2 RAC ???·??TIPS?.NET ?????? RAC ?? ????????? ????????????????? http://otndnld.oracle.co.jp/ondemand/otn-seminar/movie/RAConWin_09011500.wmv http://www.oracle.com/technetwork/jp/content/20100901-raconwindows-251670-ja.pdf

  Read the article

• McAfee Virus Scan and Oracle RAC

  - by Lee Gathercole

  Hi, We're experiencing a strange problem with Oracle RAC and McAfee anti-virus. As part of the installation of the Oracle RAC we disable anti virus as directed. We have had our RAC running fine, but when we came to re-enable the AV and reboot we got the BSOD. Abnormal Program Termination (BugCheck, STOP: 0x00000035 (0x8E984678, 0x00000000, 0x00000000, 0x00000000 NO_MORE_IRP_STACK_LOCATIONS Following the standard process of raising this problem with Microsoft they identify the problem and also a fix. Microsoft talk about too many file filter drivers being present and pushing the DFS upper limit beyond the default size. Upping this value, as per msdn, has no impact. We're able to recover from this BSOD by disabling AV. We don't have the problem if we run the AV service manually whilst the system is up. However, if we make the service automatic we fail to boot. Tech Details 2 Node Oracle 10g Cluster 2 * Windows 2003 SP2, 16GB RAM, Quad Core 3ghz Processor SAN attached storage McAfee VirusScan Enterprise 8.5.0i, Scan Engine (5300.2777), DAT Version (5536.0000) Thanks Lee

  Read the article

• So what is Active GridLink for RAC?

  - by Ruma Sanyal

  Normal 0 false false false EN-US X-NONE X-NONE MicrosoftInternetExplorer4 I had referred to Active GridLink for RAC in my blog yesterday and since then got several questions on this topic. So I decided to re-visit Active GridLink. With the release of version 11g, Oracle WebLogic Server started to provide strong support for the Real Application Clusters (RAC) features in Oracle Database 11g, minimizing database access time while allowing transparent access to rich pooling management functions that maximizes both connection performance and availability. WebLogic is the only application server in the marketplace which has been fully integrated and certified with Oracle Database RAC 11g without losing any rich functionality. Active GridLink provides Fast Connection Failover (FCF), Runtime Connection Load-Balancing (RCLB), and RAC instance graceful shutdown. With the key foundation for providing deeper integration with Oracle RAC, this single data source implementation in Oracle WebLogic Server supports the full and unrestricted use of database services as the connection target for a data source. For more details and to understand how our customer NEC leverages this capability, read the whitepapers on this topic. Get in depth ‘how-to’ details from this youtube video from our resident expert, Frances Zhao.

  Read the article

• aufs user permissions

  - by user56395

  Anyone know why this doesn't work? Is this user error, AUFS feature or bug maybe: rac@tecraS1:~/tmp$ mkdir orig tmp au rac@tecraS1:~/tmp$ sudo mount -t tmpfs none tmp rac@tecraS1:~/tmp$ sudo chown -R rac tmp rac@tecraS1:~/tmp$ echo hello > orig/hello rac@tecraS1:~/tmp$ sudo mount -t aufs -o br=tmp:orig none au rac@tecraS1:~/tmp$ ls -al au total 8 drwxrwxrwt 4 rac root 100 2011-01-06 13:53 . drwxr-xr-x 5 rac rac 4096 2011-01-06 13:52 .. -rw-r--r-- 1 rac rac 6 2011-01-06 13:53 hello rac@tecraS1:~/tmp$ rm au/hello rm: cannot remove `au/hello': Operation not permitted rac@tecraS1:~/tmp$ Seems the aufs files were created as root and user has no access to them: rac@tecraS1:~/tmp$ sudo rm au/hello rac@tecraS1:~/tmp$ ls -al tmp total 4 drwxrwxrwt 4 rac root 120 2011-01-06 13:53 . drwxr-xr-x 5 rac rac 4096 2011-01-06 13:52 .. -r--r--r-- 2 root root 0 2011-01-06 13:53 .wh.hello -r--r--r-- 2 root root 0 2011-01-06 13:53 .wh..wh.aufs drwx------ 2 root root 40 2011-01-06 13:53 .wh..wh.orph drwx------ 2 root root 40 2011-01-06 13:53 .wh..wh.plnk rac@tecraS1:~/tmp$ OS is the latest Lucid with 2.6.35-23 stock kernel. No idea about aufs version. Using sudo chown -R rac tmp/.wh* fixes the problem. Thanks for looking.

  Read the article

• ???????/???RAC????10??!US Oracle?????????????? Oracle Real Application Clusters??????????

  - by user788995

  ????? ??:2012/02/20 ??:??????/?? Oracle 9i ? Real Application Clusters ?????????????10??????? RAC ???????????????? 11g R2 ???????????????Oracle US??? RAC ?????????????????? Oracle RAC ???Oracle RAC ? More than a DatabaseOracle RAC ?????????Oracle RAC ????????????????? Oracle RAC ????????? ????????????????? http://otndnld.oracle.co.jp/ondemand/otn-seminar/movie/120130_B-8_RAC.wmv http://otndnld.oracle.co.jp/ondemand/otn-seminar/movie/mp4/120130_B-8_RAC.mp4 http://www.oracle.com/technetwork/jp/ondemand/db-new/b-8-rac-1524112-ja.pdf

  Read the article

• 11g R2 RAC Expert ????????? ???????

  - by user763243

  ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????(??????????????????????????????????????????????)200?????????????????????????????????????????????????????? ?????????????????????????????????????10????????60??????? ???????????????????????????????iPhone???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????(?????????????????????) ??????????????????????????????????????????????????????????????????????????????

  Read the article

• New Oracles VM RAC template with support for oracle vm 3 built-in

  - by wcoekaer

  The RAC team did it again (thanks Saar!) - another awesome set of Oracle VM templates published and uploaded to My Oracle Support. You can find the main page here. What's special about the latest version of DeployCluster is that it integrates tightly with Oracle VM 3 manager. It basically is an Oracle VM frontend that helps start VMs, pass arguments down automatically and there is absolutely no need to log into the Oracle VM servers or the guests. Once it completes, you have an entire Oracle RAC database setup ready to go. Here's a short summary of the steps : Set up an Oracle VM 3 server pool Download the Oracle VM RAC template from oracle.com Import the template into Oracle VM using Oracle VM Manager repository - import Create a public and private network in Oracle VM Manager in the network tab Configure the template with the right public and private virtual networks Create a set of shared disks (physical or virtual) to assign to the VMs you want to create (for ASM/at least 5) Clone a set of VMs from the template (as many RAC nodes as you plan to configure) With Oracle VM 3.1 you can clone with a number so one clone command for, say 8 VMs is easy. Assign the shared devices/disks to the cloned VMs Create a netconfig.ini file on your manager node or a client where you plan to run DeployCluster This little text file just contains the IP addresses, hostnames etc for your cluster. It is a very simple small textfile. Run deploycluster.py with the VM names as argument Done. At this point, the tool will connect to Oracle VM Manager, start the VMs and configure each one, Configure the OS (Oracle Linux) Configure the disks with ASM Configure the clusterware (CRS) Configure ASM Create database instances on each node. Now you are ready to log in, and use your x node database cluster. x No need to download various products from various websites, click on trial licenses for the OS, go to a Virtual Machine store with sample and test versions only - this is production ready and supported. Software. Complete. example netconfig.ini : # Node specific information NODE1=racnode1 NODE1VIP=racnode1-vip NODE1PRIV=racnode1-priv NODE1IP=192.168.1.2 NODE1VIPIP=192.168.1.22 NODE1PRIVIP=10.0.0.22 NODE2=racnode2 NODE2VIP=racnode2-vip NODE2PRIV=racnode2-priv NODE2IP=192.168.1.3 NODE2VIPIP=192.168.1.23 NODE2PRIVIP=10.0.0.23 # Common data PUBADAP=eth0 PUBMASK=255.255.255.0 PUBGW=192.168.1.1 PRIVADAP=eth1 PRIVMASK=255.255.255.0 RACCLUSTERNAME=raccluster DOMAINNAME=mydomain.com DNSIP= # Device used to transfer network information to second node # in interview mode NETCONFIG_DEV=/dev/xvdc # 11gR2 specific data SCANNAME=racnode12-scan SCANIP=192.168.1.50

  Read the article

• ?11gR2 RAC???ASM DISK Path????

  - by Liu Maclean(???)

  ????T.askmaclean.com???????11gR2?ASM DISK?????,??????: aix 6.1,grid 11.2.0.3+asm11.2.0.3+rac ???????????aix????????mpio,??diskgroup ?????veritas dmp???,?????asm?disk_strings=/dev/vx/rdmp/*,crs/asm??????????????/dev/vx/rdmp/?????,?????????diskgroup??? crs???????:2012-07-13 15:07:29.748: [ GPNP][1286]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2108 clsgpnp_profileCallUrlInt] get-profile call to url “ipc://GPNPD_ggtest1? disco “” [f=0 claimed- host: cname: seq: auth:]2012-07-13 15:07:29.762: [ GPNP][1286]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2236 clsgpnp_profileCallUrlInt] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote “ipc://GPNPD_ggtest1? disco “”2012-07-13 15:07:29.762: [ CSSD][1286]clssnmReadDiscoveryProfile: voting file discovery string(/dev/vx/rdmp/*)2012-07-13 15:07:29.762: [ CSSD][1286]clssnmvDDiscThread: using discovery string /dev/vx/rdmp/* for initial discovery2012-07-13 15:07:29.762: [ SKGFD][1286]Discovery with str:/dev/vx/rdmp/*: 2012-07-13 15:07:29.762: [ SKGFD][1286]UFS discovery with :/dev/vx/rdmp/*: 2012-07-13 15:07:29.769: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_919: 2012-07-13 15:07:29.770: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_212: 2012-07-13 15:07:29.770: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_211: 2012-07-13 15:07:29.770: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_210: 2012-07-13 15:07:29.770: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_209: 2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_181: 2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_180: 2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/disk_3: 2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/disk_2: 2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/disk_1: 2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/disk_0: 2012-07-13 15:07:29.771: [ SKGFD][1286]OSS discovery with :/dev/vx/rdmp/*: 2012-07-13 15:07:29.771: [ SKGFD][1286]Handle 1115e7510 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_916: 2012-07-13 15:07:29.772: [ SKGFD][1286]Handle 1118758b0 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_912: 2012-07-13 15:07:29.773: [ SKGFD][1286]Handle 1118d9cf0 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_908: 2012-07-13 15:07:29.773: [ SKGFD][1286]Handle 1118da450 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_904: 2012-07-13 15:07:29.773: [ SKGFD][1286]Handle 1118dad70 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_903: 2012-07-13 15:07:29.802: [ CLSF][1286]checksum failed for disk:/dev/vx/rdmp/v_df8000_916:2012-07-13 15:07:29.803: [ SKGFD][1286]Lib :UFS:: closing handle 1115e7510 for disk :/dev/vx/rdmp/v_df8000_916: 2012-07-13 15:07:29.803: [ SKGFD][1286]Lib :UFS:: closing handle 1118758b0 for disk :/dev/vx/rdmp/v_df8000_912: 2012-07-13 15:07:29.804: [ SKGFD][1286]Handle 1115e6710 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_202: 2012-07-13 15:07:29.808: [ SKGFD][1286]Handle 1115e7030 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_201: 2012-07-13 15:07:29.809: [ SKGFD][1286]Handle 1115e7ad0 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_200: 2012-07-13 15:07:29.809: [ SKGFD][1286]Handle 1118733f0 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_199: 2012-07-13 15:07:29.816: [ CLSF][1286]checksum failed for disk:/dev/vx/rdmp/v_df8000_186:2012-07-13 15:07:29.816: [ SKGFD][1286]Lib :UFS:: closing handle 1118de5d0 for disk :/dev/vx/rdmp/v_df8000_186: 2012-07-13 15:07:29.816: [ CSSD][1286]clssnmvDiskVerify: Successful discovery of 0 disks2012-07-13 15:07:29.816: [ CSSD][1286]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery2012-07-13 15:07:29.816: [ CSSD][1286]clssnmvFindInitialConfigs: No voting files found2012-07-13 15:07:29.816: [ CSSD][1286](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds2012-07-13 15:07:30.169: [ CSSD][1029]clssgmExecuteClientRequest(): type(37) size(80) only connect and exit messages are allowed before lease acquisition proc(1115e4870) client(0) ??????ASM DISK PATH???????,????11gR2 RAC+ASM????,??CRS??????,????crsctl start crs -excl -nocrs???????CSS???ASM??, ???????(clssnmCompleteInitVFDiscovery: Voting file not found),????Voteing file????????????????? ?????????,???????11gR2 RAC+ASM??ASM DISK??: 1.?????????ASM DISK?????,??????UDEV????????,???UDEV????ASM DISK?/dev/asm-disk* ??? /dev/rasm-disk*???, ??????udev rule??????: [grid@maclean1 ~]$ export ORACLE_HOME=/g01/grid/app/11.2.0/grid [grid@maclean1 ~]$ /g01/grid/app/11.2.0/grid/bin/sqlplus / as sysasm SQL*Plus: Release 11.2.0.3.0 Production on Sun Jul 15 04:09:28 2012 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options SQL> show parameter diskstri NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ asm_diskstring string /dev/asm* ??????ASM?????asm_diskstring ?/dev/asm*, ???root????UDEV RULE?? : [root@maclean1 rules.d]# cp 99-oracle-asmdevices.rules 99-oracle-asmdevices.rules.bak [root@maclean1 rules.d]# vi 99-oracle-asmdevices.rules [root@maclean1 rules.d]# cat 99-oracle-asmdevices.rules KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB09cadb31-cfbea255_", NAME="rasm-diskb", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB5f097069-59efb82f_", NAME="rasm-diskc", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB4e1a81c0-20478bc4_", NAME="rasm-diskd", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VBdcce9285-b13c5a27_", NAME="rasm-diske", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB82effe1a-dbca7dff_", NAME="rasm-diskf", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB950d279f-c581cb51_", NAME="rasm-diskg", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB14400d81-651672d7_", NAME="rasm-diskh", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB31b1237b-78aa22bb_", NAME="rasm-diski", OWNER="grid", GROUP="asmadmin", MODE="0660" ???????99-oracle-asmdevices.rules?UDEV RULE????,??????????/dev/rasm-disk*???,??????ASM DISK???, ????????????????RAC CRS??????? ??????votedisk?ocr ????: [root@maclean1 rules.d]# /g01/grid/app/11.2.0/grid/bin/crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 6896bfc3d1464f9fbf0ea9df87e023ad (/dev/asm-diskb) [SYSTEMDG] 2. ONLINE 58eb81b656084ff2bfd315d9badd08b7 (/dev/asm-diskc) [SYSTEMDG] 3. ONLINE 6bf7324625c54f3abf2c942b1e7f70d9 (/dev/asm-diskd) [SYSTEMDG] 4. ONLINE 43ad8ae20c354f5ebf7083bc30bf94cc (/dev/asm-diske) [SYSTEMDG] 5. ONLINE 4c225359d51b4f93bfba01080664b3d7 (/dev/asm-diskf) [SYSTEMDG] Located 5 voting disk(s). [root@maclean1 rules.d]# /g01/grid/app/11.2.0/grid/bin/ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 2844 Available space (kbytes) : 259276 ID : 879001605 Device/File Name : +SYSTEMDG Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded ??votedisk file?????????ASM DISK,?????????crsctl replace votedisk, ??????LINUX OS: [root@maclean1 rules.d]# init 6 rebooting ............ [root@maclean1 dev]# ls -l *asm* brw-rw---- 1 grid asmadmin 8, 16 Jul 15 04:15 rasm-diskb brw-rw---- 1 grid asmadmin 8, 32 Jul 15 04:15 rasm-diskc brw-rw---- 1 grid asmadmin 8, 48 Jul 15 04:15 rasm-diskd brw-rw---- 1 grid asmadmin 8, 64 Jul 15 04:15 rasm-diske brw-rw---- 1 grid asmadmin 8, 80 Jul 15 04:15 rasm-diskf brw-rw---- 1 grid asmadmin 8, 96 Jul 15 04:15 rasm-diskg brw-rw---- 1 grid asmadmin 8, 112 Jul 15 04:15 rasm-diskh brw-rw---- 1 grid asmadmin 8, 128 Jul 15 04:15 rasm-diski ??????????/dev/rasm-disk*?ASM DISK,??ASM??????css?????/dev/asm*?????ASM DISK,??????????????ASM DISK: more /g01/grid/app/11.2.0/grid/log/maclean1/cssd/ocssd.log 2012-07-15 04:17:45.208: [ SKGFD][1099548992]Discovery with str:/dev/asm*: 2012-07-15 04:17:45.208: [ SKGFD][1099548992]UFS discovery with :/dev/asm*: 2012-07-15 04:17:45.208: [ SKGFD][1099548992]OSS discovery with :/dev/asm*: 2012-07-15 04:17:45.208: [ CSSD][1099548992]clssnmvDiskVerify: Successful discovery of 0 disks 2012-07-15 04:17:45.208: [ CSSD][1099548992]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery 2012-07-15 04:17:45.208: [ CSSD][1099548992]clssnmvFindInitialConfigs: No voting files found 2012-07-15 04:17:45.208: [ CSSD][1099548992](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds 2012-07-15 04:17:45.251: [ CSSD][1096661312]clssgmExecuteClientRequest(): type(37) size(80) only connect and exit messages are allowed before lease acquisition proc(0x26a8ba0) client((nil)) 2012-07-15 04:17:45.251: [ CSSD][1096661312]clssgmDeadProc: proc 0x26a8ba0 2012-07-15 04:17:45.251: [ CSSD][1096661312]clssgmDestroyProc: cleaning up proc(0x26a8ba0) con(0xfe6) skgpid ospid 3751 with 0 clients, refcount 0 2012-07-15 04:17:45.252: [ CSSD][1096661312]clssgmDiscEndpcl: gipcDestroy 0xfe6 2012-07-15 04:17:45.829: [ CSSD][1096661312]clssscSelect: cookie accept request 0x2318ea0 2012-07-15 04:17:45.829: [ CSSD][1096661312]clssgmAllocProc: (0x2659480) allocated 2012-07-15 04:17:45.830: [ CSSD][1096661312]clssgmClientConnectMsg: properties of cmProc 0x2659480 - 1,2,3,4,5 2012-07-15 04:17:45.830: [ CSSD][1096661312]clssgmClientConnectMsg: Connect from con(0x114e) proc(0x2659480) pid(3751) version 11:2:1:4, properties: 1,2,3,4,5 2012-07-15 04:17:45.830: [ CSSD][1096661312]clssgmClientConnectMsg: msg flags 0x0000 2012-07-15 04:17:45.939: [ CSSD][1096661312]clssscSelect: cookie accept request 0x253ddd0 2012-07-15 04:17:45.939: [ CSSD][1096661312]clssscevtypSHRCON: getting client with cmproc 0x253ddd0 2012-07-15 04:17:45.939: [ CSSD][1096661312]clssgmRegisterClient: proc(3/0x253ddd0), client(61/0x26877b0) 2012-07-15 04:17:45.939: [ CSSD][1096661312]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are  allowed before lease acquisition proc(0x253ddd0) client(0x26877b0) 2012-07-15 04:17:45.939: [ CSSD][1096661312]clssgmDiscEndpcl: gipcDestroy 0x1174 2012-07-15 04:17:46.070: [ CSSD][1096661312]clssscSelect: cookie accept request 0x26368a0 2012-07-15 04:17:46.070: [ CSSD][1096661312]clssscevtypSHRCON: getting client with cmproc 0x26368a0 2012-07-15 04:17:46.070: [ CSSD][1096661312]clssgmRegisterClient: proc(5/0x26368a0), client(50/0x26877b0) ??11gR2?CRS?????ASM,??ocr???ASM?,??ASM???????,???CRS?????????: [root@maclean1 ~]# crsctl check has CRS-4638: Oracle High Availability Services is online [root@maclean1 ~]# crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4535: Cannot communicate with Cluster Ready Services CRS-4530: Communications failure contacting Cluster Synchronization Services daemon CRS-4534: Cannot communicate with Event Manager 2. ?????ASM DISK PATH???????,?????????????CRS: ??????OHASD??: [root@maclean1 ~]# crsctl stop has -f CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'maclean1' CRS-2673: Attempting to stop 'ora.mdnsd' on 'maclean1' CRS-2673: Attempting to stop 'ora.crf' on 'maclean1' CRS-2677: Stop of 'ora.mdnsd' on 'maclean1' succeeded CRS-2677: Stop of 'ora.crf' on 'maclean1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'maclean1' CRS-2677: Stop of 'ora.gipcd' on 'maclean1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'maclean1' CRS-2677: Stop of 'ora.gpnpd' on 'maclean1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'maclean1' has completed CRS-4133: Oracle High Availability Services has been stopped. 3. ?-excl -nocrs????CRS,?????ASM ???????CRS??: [root@maclean1 ~]# crsctl start crs -excl -nocrs  CRS-4123: Oracle High Availability Services has been started. CRS-2672: Attempting to start 'ora.mdnsd' on 'maclean1' CRS-2676: Start of 'ora.mdnsd' on 'maclean1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'maclean1' CRS-2676: Start of 'ora.gpnpd' on 'maclean1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'maclean1' CRS-2672: Attempting to start 'ora.gipcd' on 'maclean1' CRS-2676: Start of 'ora.cssdmonitor' on 'maclean1' succeeded CRS-2676: Start of 'ora.gipcd' on 'maclean1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'maclean1' CRS-2672: Attempting to start 'ora.diskmon' on 'maclean1' CRS-2676: Start of 'ora.diskmon' on 'maclean1' succeeded CRS-2676: Start of 'ora.cssd' on 'maclean1' succeeded CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'maclean1' CRS-2672: Attempting to start 'ora.ctssd' on 'maclean1' CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'maclean1' succeeded CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'maclean1' CRS-2676: Start of 'ora.ctssd' on 'maclean1' succeeded CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'maclean1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'maclean1' CRS-2676: Start of 'ora.asm' on 'maclean1' succeeded #??????CRS_HOME???ORACLE_BASE?777??,??????? [root@maclean1 ~]# chmod 777 /g01 4.??ASM???disk_strings????ASM DISK PATH??: [root@maclean1 ~]# su - grid [grid@maclean1 ~]$ sqlplus / as sysasm SQL*Plus: Release 11.2.0.3.0 Production on Sun Jul 15 04:40:40 2012 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options SQL> alter system set asm_diskstring='/dev/rasm*'; System altered. SQL> alter diskgroup systemdg mount; Diskgroup altered. SQL> create spfile from memory; File created. SQL> startup force mount; ORA-32004: obsolete or deprecated parameter(s) specified for ASM instance ASM instance started Total System Global Area 283930624 bytes Fixed Size 2227664 bytes Variable Size 256537136 bytes ASM Cache 25165824 bytes ASM diskgroups mounted SQL> show parameter spfile NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ spfile string /g01/grid/app/11.2.0/grid/dbs/ spfile+ASM1.ora SQL> show parameter disk NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ asm_diskgroups string SYSTEMDG asm_diskstring string /dev/rasm* SQL> create pfile from spfile; File created. SQL> create spfile='+SYSTEMDG' from pfile; File created. SQL> startup force; ORA-32004: obsolete or deprecated parameter(s) specified for ASM instance ASM instance started Total System Global Area 283930624 bytes Fixed Size 2227664 bytes Variable Size 256537136 bytes ASM Cache 25165824 bytes ASM diskgroups mounted SQL> show parameter spfile NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ spfile string +SYSTEMDG/maclean-cluster/asmp arameterfile/registry.253.7886 82933 ???????asm_diskstring ,????ASM DISKGROUP??SPFILE , ??ASM?????SPFILE?????????????????? 5. crsctl replace votedisk ???votedisk????: [root@maclean1 ~]# crsctl replace votedisk +systemdg Successful addition of voting disk 864a00efcfbe4f42bfd0f4f6b60472a0. Successful addition of voting disk ab14d6e727614f29bf53b9870052a5c8. Successful addition of voting disk 754c03c168854f46bf2daee7287bf260. Successful addition of voting disk 9ed58f37f3e84f28bfcd9b101f2af9f3. Successful addition of voting disk 4ce7b7c682364f12bf4df5ce1fb7814e. Successfully replaced voting disk group with +systemdg. CRS-4266: Voting file(s) successfully replaced [root@maclean1 ~]# crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 864a00efcfbe4f42bfd0f4f6b60472a0 (/dev/rasm-diskb) [SYSTEMDG] 2. ONLINE ab14d6e727614f29bf53b9870052a5c8 (/dev/rasm-diskc) [SYSTEMDG] 3. ONLINE 754c03c168854f46bf2daee7287bf260 (/dev/rasm-diskd) [SYSTEMDG] 4. ONLINE 9ed58f37f3e84f28bfcd9b101f2af9f3 (/dev/rasm-diske) [SYSTEMDG] 5. ONLINE 4ce7b7c682364f12bf4df5ce1fb7814e (/dev/rasm-diskf) [SYSTEMDG] Located 5 voting disk(s). [root@maclean1 ~]# ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 2844 Available space (kbytes) : 259276 ID : 879001605 Device/File Name : +SYSTEMDG Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded ??replace?votedisk??? ASM DISK?,???votedisk?OCR??????? 6.??CRS??: [root@maclean1 ~]# crsctl stop crs CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'maclean1' CRS-2673: Attempting to stop 'ora.mdnsd' on 'maclean1' CRS-2673: Attempting to stop 'ora.ctssd' on 'maclean1' CRS-2673: Attempting to stop 'ora.asm' on 'maclean1' CRS-2677: Stop of 'ora.mdnsd' on 'maclean1' succeeded CRS-2677: Stop of 'ora.asm' on 'maclean1' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'maclean1' CRS-2677: Stop of 'ora.ctssd' on 'maclean1' succeeded CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'maclean1' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'maclean1' CRS-2677: Stop of 'ora.cssd' on 'maclean1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'maclean1' CRS-2677: Stop of 'ora.gipcd' on 'maclean1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'maclean1' CRS-2677: Stop of 'ora.gpnpd' on 'maclean1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'maclean1' has completed CRS-4133: Oracle High Availability Services has been stopped. [root@maclean1 ~]# crsctl stat res -t -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.BACKUPDG.dg ONLINE ONLINE maclean1 ora.DATA.dg ONLINE ONLINE maclean1 ora.LISTENER.lsnr ONLINE ONLINE maclean1 ora.SYSTEMDG.dg ONLINE ONLINE maclean1 ora.asm ONLINE ONLINE maclean1 Started ora.gsd OFFLINE OFFLINE maclean1 ora.net1.network ONLINE ONLINE maclean1 ora.ons ONLINE ONLINE maclean1 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE maclean1 ora.cvu 1 ONLINE ONLINE maclean1 ora.maclean1.vip 1 ONLINE ONLINE maclean1 ora.maclean2.vip 1 ONLINE INTERMEDIATE maclean1 FAILED OVER ora.oc4j 1 ONLINE OFFLINE STARTING ora.prod.db 1 ONLINE OFFLINE Instance Shutdown,S TARTING 2 ONLINE OFFLINE ora.scan1.vip 1 ONLINE ONLINE maclean1 ???????ASM?????SPFILE,???????????????,?????CRS??????? ??11gR2 RAC+ASM?????????,????????????????ASM DISK PATH??????????

  Read the article

• ?11gR2 RAC???ASM DISK Path????

  - by Liu Maclean(???)

  ????T.askmaclean.com???????11gR2?ASM DISK?????,??????: aix 6.1,grid 11.2.0.3+asm11.2.0.3+rac ???????????aix????????mpio,??diskgroup ?????veritas dmp???,?????asm?disk_strings=/dev/vx/rdmp/*,crs/asm??????????????/dev/vx/rdmp/?????,?????????diskgroup??? crs???????:2012-07-13 15:07:29.748: [ GPNP][1286]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2108 clsgpnp_profileCallUrlInt] get-profile call to url “ipc://GPNPD_ggtest1? disco “” [f=0 claimed- host: cname: seq: auth:]2012-07-13 15:07:29.762: [ GPNP][1286]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2236 clsgpnp_profileCallUrlInt] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote “ipc://GPNPD_ggtest1? disco “”2012-07-13 15:07:29.762: [ CSSD][1286]clssnmReadDiscoveryProfile: voting file discovery string(/dev/vx/rdmp/*)2012-07-13 15:07:29.762: [ CSSD][1286]clssnmvDDiscThread: using discovery string /dev/vx/rdmp/* for initial discovery2012-07-13 15:07:29.762: [ SKGFD][1286]Discovery with str:/dev/vx/rdmp/*: 2012-07-13 15:07:29.762: [ SKGFD][1286]UFS discovery with :/dev/vx/rdmp/*: 2012-07-13 15:07:29.769: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_919: 2012-07-13 15:07:29.770: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_212: 2012-07-13 15:07:29.770: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_211: 2012-07-13 15:07:29.770: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_210: 2012-07-13 15:07:29.770: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_209: 2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_181: 2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_180: 2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/disk_3: 2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/disk_2: 2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/disk_1: 2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/disk_0: 2012-07-13 15:07:29.771: [ SKGFD][1286]OSS discovery with :/dev/vx/rdmp/*: 2012-07-13 15:07:29.771: [ SKGFD][1286]Handle 1115e7510 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_916: 2012-07-13 15:07:29.772: [ SKGFD][1286]Handle 1118758b0 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_912: 2012-07-13 15:07:29.773: [ SKGFD][1286]Handle 1118d9cf0 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_908: 2012-07-13 15:07:29.773: [ SKGFD][1286]Handle 1118da450 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_904: 2012-07-13 15:07:29.773: [ SKGFD][1286]Handle 1118dad70 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_903: 2012-07-13 15:07:29.802: [ CLSF][1286]checksum failed for disk:/dev/vx/rdmp/v_df8000_916:2012-07-13 15:07:29.803: [ SKGFD][1286]Lib :UFS:: closing handle 1115e7510 for disk :/dev/vx/rdmp/v_df8000_916: 2012-07-13 15:07:29.803: [ SKGFD][1286]Lib :UFS:: closing handle 1118758b0 for disk :/dev/vx/rdmp/v_df8000_912: 2012-07-13 15:07:29.804: [ SKGFD][1286]Handle 1115e6710 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_202: 2012-07-13 15:07:29.808: [ SKGFD][1286]Handle 1115e7030 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_201: 2012-07-13 15:07:29.809: [ SKGFD][1286]Handle 1115e7ad0 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_200: 2012-07-13 15:07:29.809: [ SKGFD][1286]Handle 1118733f0 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_199: 2012-07-13 15:07:29.816: [ CLSF][1286]checksum failed for disk:/dev/vx/rdmp/v_df8000_186:2012-07-13 15:07:29.816: [ SKGFD][1286]Lib :UFS:: closing handle 1118de5d0 for disk :/dev/vx/rdmp/v_df8000_186: 2012-07-13 15:07:29.816: [ CSSD][1286]clssnmvDiskVerify: Successful discovery of 0 disks2012-07-13 15:07:29.816: [ CSSD][1286]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery2012-07-13 15:07:29.816: [ CSSD][1286]clssnmvFindInitialConfigs: No voting files found2012-07-13 15:07:29.816: [ CSSD][1286](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds2012-07-13 15:07:30.169: [ CSSD][1029]clssgmExecuteClientRequest(): type(37) size(80) only connect and exit messages are allowed before lease acquisition proc(1115e4870) client(0) ??????ASM DISK PATH???????,????11gR2 RAC+ASM????,??CRS??????,????crsctl start crs -excl -nocrs???????CSS???ASM??, ???????(clssnmCompleteInitVFDiscovery: Voting file not found),????Voteing file????????????????? ?????????,???????11gR2 RAC+ASM??ASM DISK??: 1.?????????ASM DISK?????,??????UDEV????????,???UDEV????ASM DISK?/dev/asm-disk* ??? /dev/rasm-disk*???, ??????udev rule??????: [grid@maclean1 ~]$ export ORACLE_HOME=/g01/grid/app/11.2.0/grid [grid@maclean1 ~]$ /g01/grid/app/11.2.0/grid/bin/sqlplus / as sysasm SQL*Plus: Release 11.2.0.3.0 Production on Sun Jul 15 04:09:28 2012 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options SQL> show parameter diskstri NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ asm_diskstring string /dev/asm* ??????ASM?????asm_diskstring ?/dev/asm*, ???root????UDEV RULE?? : [root@maclean1 rules.d]# cp 99-oracle-asmdevices.rules 99-oracle-asmdevices.rules.bak [root@maclean1 rules.d]# vi 99-oracle-asmdevices.rules [root@maclean1 rules.d]# cat 99-oracle-asmdevices.rules KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB09cadb31-cfbea255_", NAME="rasm-diskb", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB5f097069-59efb82f_", NAME="rasm-diskc", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB4e1a81c0-20478bc4_", NAME="rasm-diskd", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VBdcce9285-b13c5a27_", NAME="rasm-diske", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB82effe1a-dbca7dff_", NAME="rasm-diskf", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB950d279f-c581cb51_", NAME="rasm-diskg", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB14400d81-651672d7_", NAME="rasm-diskh", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB31b1237b-78aa22bb_", NAME="rasm-diski", OWNER="grid", GROUP="asmadmin", MODE="0660" ???????99-oracle-asmdevices.rules?UDEV RULE????,??????????/dev/rasm-disk*???,??????ASM DISK???, ????????????????RAC CRS??????? ??????votedisk?ocr ????: [root@maclean1 rules.d]# /g01/grid/app/11.2.0/grid/bin/crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 6896bfc3d1464f9fbf0ea9df87e023ad (/dev/asm-diskb) [SYSTEMDG] 2. ONLINE 58eb81b656084ff2bfd315d9badd08b7 (/dev/asm-diskc) [SYSTEMDG] 3. ONLINE 6bf7324625c54f3abf2c942b1e7f70d9 (/dev/asm-diskd) [SYSTEMDG] 4. ONLINE 43ad8ae20c354f5ebf7083bc30bf94cc (/dev/asm-diske) [SYSTEMDG] 5. ONLINE 4c225359d51b4f93bfba01080664b3d7 (/dev/asm-diskf) [SYSTEMDG] Located 5 voting disk(s). [root@maclean1 rules.d]# /g01/grid/app/11.2.0/grid/bin/ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 2844 Available space (kbytes) : 259276 ID : 879001605 Device/File Name : +SYSTEMDG Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded ??votedisk file?????????ASM DISK,?????????crsctl replace votedisk, ??????LINUX OS: [root@maclean1 rules.d]# init 6 rebooting ............ [root@maclean1 dev]# ls -l *asm* brw-rw---- 1 grid asmadmin 8, 16 Jul 15 04:15 rasm-diskb brw-rw---- 1 grid asmadmin 8, 32 Jul 15 04:15 rasm-diskc brw-rw---- 1 grid asmadmin 8, 48 Jul 15 04:15 rasm-diskd brw-rw---- 1 grid asmadmin 8, 64 Jul 15 04:15 rasm-diske brw-rw---- 1 grid asmadmin 8, 80 Jul 15 04:15 rasm-diskf brw-rw---- 1 grid asmadmin 8, 96 Jul 15 04:15 rasm-diskg brw-rw---- 1 grid asmadmin 8, 112 Jul 15 04:15 rasm-diskh brw-rw---- 1 grid asmadmin 8, 128 Jul 15 04:15 rasm-diski ??????????/dev/rasm-disk*?ASM DISK,??ASM??????css?????/dev/asm*?????ASM DISK,??????????????ASM DISK: more /g01/grid/app/11.2.0/grid/log/maclean1/cssd/ocssd.log 2012-07-15 04:17:45.208: [ SKGFD][1099548992]Discovery with str:/dev/asm*: 2012-07-15 04:17:45.208: [ SKGFD][1099548992]UFS discovery with :/dev/asm*: 2012-07-15 04:17:45.208: [ SKGFD][1099548992]OSS discovery with :/dev/asm*: 2012-07-15 04:17:45.208: [ CSSD][1099548992]clssnmvDiskVerify: Successful discovery of 0 disks 2012-07-15 04:17:45.208: [ CSSD][1099548992]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery 2012-07-15 04:17:45.208: [ CSSD][1099548992]clssnmvFindInitialConfigs: No voting files found 2012-07-15 04:17:45.208: [ CSSD][1099548992](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds 2012-07-15 04:17:45.251: [ CSSD][1096661312]clssgmExecuteClientRequest(): type(37) size(80) only connect and exit messages are allowed before lease acquisition proc(0x26a8ba0) client((nil)) 2012-07-15 04:17:45.251: [ CSSD][1096661312]clssgmDeadProc: proc 0x26a8ba0 2012-07-15 04:17:45.251: [ CSSD][1096661312]clssgmDestroyProc: cleaning up proc(0x26a8ba0) con(0xfe6) skgpid ospid 3751 with 0 clients, refcount 0 2012-07-15 04:17:45.252: [ CSSD][1096661312]clssgmDiscEndpcl: gipcDestroy 0xfe6 2012-07-15 04:17:45.829: [ CSSD][1096661312]clssscSelect: cookie accept request 0x2318ea0 2012-07-15 04:17:45.829: [ CSSD][1096661312]clssgmAllocProc: (0x2659480) allocated 2012-07-15 04:17:45.830: [ CSSD][1096661312]clssgmClientConnectMsg: properties of cmProc 0x2659480 - 1,2,3,4,5 2012-07-15 04:17:45.830: [ CSSD][1096661312]clssgmClientConnectMsg: Connect from con(0x114e) proc(0x2659480) pid(3751) version 11:2:1:4, properties: 1,2,3,4,5 2012-07-15 04:17:45.830: [ CSSD][1096661312]clssgmClientConnectMsg: msg flags 0x0000 2012-07-15 04:17:45.939: [ CSSD][1096661312]clssscSelect: cookie accept request 0x253ddd0 2012-07-15 04:17:45.939: [ CSSD][1096661312]clssscevtypSHRCON: getting client with cmproc 0x253ddd0 2012-07-15 04:17:45.939: [ CSSD][1096661312]clssgmRegisterClient: proc(3/0x253ddd0), client(61/0x26877b0) 2012-07-15 04:17:45.939: [ CSSD][1096661312]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are  allowed before lease acquisition proc(0x253ddd0) client(0x26877b0) 2012-07-15 04:17:45.939: [ CSSD][1096661312]clssgmDiscEndpcl: gipcDestroy 0x1174 2012-07-15 04:17:46.070: [ CSSD][1096661312]clssscSelect: cookie accept request 0x26368a0 2012-07-15 04:17:46.070: [ CSSD][1096661312]clssscevtypSHRCON: getting client with cmproc 0x26368a0 2012-07-15 04:17:46.070: [ CSSD][1096661312]clssgmRegisterClient: proc(5/0x26368a0), client(50/0x26877b0) ??11gR2?CRS?????ASM,??ocr???ASM?,??ASM???????,???CRS?????????: [root@maclean1 ~]# crsctl check has CRS-4638: Oracle High Availability Services is online [root@maclean1 ~]# crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4535: Cannot communicate with Cluster Ready Services CRS-4530: Communications failure contacting Cluster Synchronization Services daemon CRS-4534: Cannot communicate with Event Manager 2. ?????ASM DISK PATH???????,?????????????CRS: ??????OHASD??: [root@maclean1 ~]# crsctl stop has -f CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'maclean1' CRS-2673: Attempting to stop 'ora.mdnsd' on 'maclean1' CRS-2673: Attempting to stop 'ora.crf' on 'maclean1' CRS-2677: Stop of 'ora.mdnsd' on 'maclean1' succeeded CRS-2677: Stop of 'ora.crf' on 'maclean1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'maclean1' CRS-2677: Stop of 'ora.gipcd' on 'maclean1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'maclean1' CRS-2677: Stop of 'ora.gpnpd' on 'maclean1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'maclean1' has completed CRS-4133: Oracle High Availability Services has been stopped. 3. ?-excl -nocrs????CRS,?????ASM ???????CRS??: [root@maclean1 ~]# crsctl start crs -excl -nocrs  CRS-4123: Oracle High Availability Services has been started. CRS-2672: Attempting to start 'ora.mdnsd' on 'maclean1' CRS-2676: Start of 'ora.mdnsd' on 'maclean1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'maclean1' CRS-2676: Start of 'ora.gpnpd' on 'maclean1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'maclean1' CRS-2672: Attempting to start 'ora.gipcd' on 'maclean1' CRS-2676: Start of 'ora.cssdmonitor' on 'maclean1' succeeded CRS-2676: Start of 'ora.gipcd' on 'maclean1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'maclean1' CRS-2672: Attempting to start 'ora.diskmon' on 'maclean1' CRS-2676: Start of 'ora.diskmon' on 'maclean1' succeeded CRS-2676: Start of 'ora.cssd' on 'maclean1' succeeded CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'maclean1' CRS-2672: Attempting to start 'ora.ctssd' on 'maclean1' CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'maclean1' succeeded CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'maclean1' CRS-2676: Start of 'ora.ctssd' on 'maclean1' succeeded CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'maclean1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'maclean1' CRS-2676: Start of 'ora.asm' on 'maclean1' succeeded #??????CRS_HOME???ORACLE_BASE?777??,??????? [root@maclean1 ~]# chmod 777 /g01 4.??ASM???disk_strings????ASM DISK PATH??: [root@maclean1 ~]# su - grid [grid@maclean1 ~]$ sqlplus / as sysasm SQL*Plus: Release 11.2.0.3.0 Production on Sun Jul 15 04:40:40 2012 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options SQL> alter system set asm_diskstring='/dev/rasm*'; System altered. SQL> alter diskgroup systemdg mount; Diskgroup altered. SQL> create spfile from memory; File created. SQL> startup force mount; ORA-32004: obsolete or deprecated parameter(s) specified for ASM instance ASM instance started Total System Global Area 283930624 bytes Fixed Size 2227664 bytes Variable Size 256537136 bytes ASM Cache 25165824 bytes ASM diskgroups mounted SQL> show parameter spfile NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ spfile string /g01/grid/app/11.2.0/grid/dbs/ spfile+ASM1.ora SQL> show parameter disk NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ asm_diskgroups string SYSTEMDG asm_diskstring string /dev/rasm* SQL> create pfile from spfile; File created. SQL> create spfile='+SYSTEMDG' from pfile; File created. SQL> startup force; ORA-32004: obsolete or deprecated parameter(s) specified for ASM instance ASM instance started Total System Global Area 283930624 bytes Fixed Size 2227664 bytes Variable Size 256537136 bytes ASM Cache 25165824 bytes ASM diskgroups mounted SQL> show parameter spfile NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ spfile string +SYSTEMDG/maclean-cluster/asmp arameterfile/registry.253.7886 82933 ???????asm_diskstring ,????ASM DISKGROUP??SPFILE , ??ASM?????SPFILE?????????????????? 5. crsctl replace votedisk ???votedisk????: [root@maclean1 ~]# crsctl replace votedisk +systemdg Successful addition of voting disk 864a00efcfbe4f42bfd0f4f6b60472a0. Successful addition of voting disk ab14d6e727614f29bf53b9870052a5c8. Successful addition of voting disk 754c03c168854f46bf2daee7287bf260. Successful addition of voting disk 9ed58f37f3e84f28bfcd9b101f2af9f3. Successful addition of voting disk 4ce7b7c682364f12bf4df5ce1fb7814e. Successfully replaced voting disk group with +systemdg. CRS-4266: Voting file(s) successfully replaced [root@maclean1 ~]# crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 864a00efcfbe4f42bfd0f4f6b60472a0 (/dev/rasm-diskb) [SYSTEMDG] 2. ONLINE ab14d6e727614f29bf53b9870052a5c8 (/dev/rasm-diskc) [SYSTEMDG] 3. ONLINE 754c03c168854f46bf2daee7287bf260 (/dev/rasm-diskd) [SYSTEMDG] 4. ONLINE 9ed58f37f3e84f28bfcd9b101f2af9f3 (/dev/rasm-diske) [SYSTEMDG] 5. ONLINE 4ce7b7c682364f12bf4df5ce1fb7814e (/dev/rasm-diskf) [SYSTEMDG] Located 5 voting disk(s). [root@maclean1 ~]# ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 2844 Available space (kbytes) : 259276 ID : 879001605 Device/File Name : +SYSTEMDG Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded ??replace?votedisk??? ASM DISK?,???votedisk?OCR??????? 6.??CRS??: [root@maclean1 ~]# crsctl stop crs CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'maclean1' CRS-2673: Attempting to stop 'ora.mdnsd' on 'maclean1' CRS-2673: Attempting to stop 'ora.ctssd' on 'maclean1' CRS-2673: Attempting to stop 'ora.asm' on 'maclean1' CRS-2677: Stop of 'ora.mdnsd' on 'maclean1' succeeded CRS-2677: Stop of 'ora.asm' on 'maclean1' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'maclean1' CRS-2677: Stop of 'ora.ctssd' on 'maclean1' succeeded CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'maclean1' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'maclean1' CRS-2677: Stop of 'ora.cssd' on 'maclean1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'maclean1' CRS-2677: Stop of 'ora.gipcd' on 'maclean1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'maclean1' CRS-2677: Stop of 'ora.gpnpd' on 'maclean1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'maclean1' has completed CRS-4133: Oracle High Availability Services has been stopped. [root@maclean1 ~]# crsctl stat res -t -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.BACKUPDG.dg ONLINE ONLINE maclean1 ora.DATA.dg ONLINE ONLINE maclean1 ora.LISTENER.lsnr ONLINE ONLINE maclean1 ora.SYSTEMDG.dg ONLINE ONLINE maclean1 ora.asm ONLINE ONLINE maclean1 Started ora.gsd OFFLINE OFFLINE maclean1 ora.net1.network ONLINE ONLINE maclean1 ora.ons ONLINE ONLINE maclean1 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE maclean1 ora.cvu 1 ONLINE ONLINE maclean1 ora.maclean1.vip 1 ONLINE ONLINE maclean1 ora.maclean2.vip 1 ONLINE INTERMEDIATE maclean1 FAILED OVER ora.oc4j 1 ONLINE OFFLINE STARTING ora.prod.db 1 ONLINE OFFLINE Instance Shutdown,S TARTING 2 ONLINE OFFLINE ora.scan1.vip 1 ONLINE ONLINE maclean1 ???????ASM?????SPFILE,???????????????,?????CRS??????? ??11gR2 RAC+ASM?????????,????????????????ASM DISK PATH?????????, ???????????????,????!

  Read the article

• 11gR2 RAC ASM????

  - by Liu Maclean(???)

  11gR2 RAC?ocr?votedisk???????ASM??, ????10g??????2?RAC????????????,  ?? 11gR2 ?ASM?spfile??????ASM diskgroup???????ASM??????? ????????????,????? ASM?????mount diskgroup??????diskgroup????, ??ASM??????ASM spfile????????,?2???????? ????T.askmaclean.com?????ASM?????: hello maclean, ??spfile??ASMCMD> spget+CRSDG/rac/asmparameterfile/registry.253.787925627?????,ASM ?????ORACLE instance,?????????????diskgroup,????????????????????????????thanks.! ?????????: ?11.2??Oracle Cluterware??voting disk files?????????11.1?10.2????,11.2??voting disk file??????OCR?, ?????11.2??ocr?votedisk?????ASM? , ???11.2?voting disk file??GPNP profile??CSS voting file discovery string???? CSS voting disk file?discovery string???ASM,??????ASM discovery string???  ????????udev???????ASM???LUN, ??udev????????/dev/rasm-disk* , ????gpnptool get????gpnp profile: [grid@maclean1 trace]$ gpnptool get Warning: some command line parameters were defaulted. Resulting command line: /g01/grid/app/11.2.0/grid/bin/gpnptool.bin get -o- <?xml version="1.0" encoding="UTF-8"?><gpnp:GPnP-Profile Version="1.0" xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile" xmlns:gpnp="http://www.grid-pnp.org/2005/11/gpnp-profile" xmlns:orcl="http://www.oracle.com/gpnp/2005/11/gpnp-profile" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.grid-pnp.org/2005/11/gpnp-profile gpnp-profile.xsd" ProfileSequence="9" ClusterUId="452185be9cd14ff4ffdc7688ec5439bf" ClusterName="maclean-cluster" PALocation=""><gpnp:Network-Profile><gpnp:HostNetwork id="gen" HostName="*"><gpnp:Network id="net1" IP="192.168.1.0" Adapter="eth0" Use="public"/><gpnp:Network id="net2" IP="172.168.1.0" Adapter="eth1" Use="cluster_interconnect"/></gpnp:HostNetwork></gpnp:Network-Profile>< orcl:CSS-Profile id="css" DiscoveryString="+asm" LeaseDuration="400"/><orcl:ASM-Profile id="asm" DiscoveryString="/dev/rasm*" SPFile="+SYSTEMDG/maclean-cluster/asmparameterfile/registry.253.788682933"/>< ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#"><ds:SignedInfo><ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/><ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/><ds:Reference URI=""><ds:Transforms><ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/><ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"> <InclusiveNamespaces xmlns="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="gpnp orcl xsi"/></ds:Transform></ds:Transforms>< ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/><ds:DigestValue>L1SLg10AqGEauCQ4ne9quucITZA=</ds:DigestValue>< /ds:Reference></ds:SignedInfo><ds:SignatureValue>rTyZm9vfcQCMuian6isnAThUmsV4xPoK2fteMc1l0GIvRvHncMwLQzPM/QrXCGGTCEvgvXzUPEKzmdX2oy5vLcztN60UHr6AJtA2JYYodmrsFwEyVBQ1D6wH+HQiOe2SG9UzdQnNtWSbjD4jfZkeQWyMPfWdKm071Ek0Rfb4nxE=</ds:SignatureValue></ds:Signature></gpnp:GPnP-Profile> Success. ?????2???: <orcl:CSS-Profile id=”css” DiscoveryString=”+asm” LeaseDuration=”400?/>==»css voting disk??+ASM<orcl:ASM-Profile id=”asm” DiscoveryString=”/dev/rasm*” SPFile=”+SYSTEMDG/maclean-cluster/asmparameterfile/registry.253.788682933?/>==»??????ASM?DiscoveryString=”/dev/rasm*”,?ASM??????????????,SPFILE???ASM Parameter FILE?ALIAS ???????GPNP???ASM Parameter FILE?ALIAS,?????ASM???????SPFILE,???Diskgroup?Mount???????ASM ALIAS?????? ??????+SYSTEMDG/maclean-cluster/asmparameterfile/registry.253.788682933??SPFILE?ASM??????: [grid@maclean1 wallets]$ sqlplus / as sysasm SQL*Plus: Release 11.2.0.3.0 Production on Tue Jul 17 05:45:35 2012 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options SQL> set linesize 140 pagesize 1400 col "FILE NAME" format a40 set head on select NAME "FILE NAME", AU_KFFXP "AU NUMBER", NUMBER_KFFXP "FILE NUMBER", DISK_KFFXP "DISK NUMBER" from x$kffxp, v$asm_alias where GROUP_KFFXP = GROUP_NUMBER and NUMBER_KFFXP = FILE_NUMBER and name in ('REGISTRY.253.788682933') order by DISK_KFFXP,AU_KFFXP; FILE NAME AU NUMBER FILE NUMBER DISK NUMBER ---------------------------------------- ---------- ----------- ----------- REGISTRY.253.788682933 39 253 1 REGISTRY.253.788682933 35 253 3 REGISTRY.253.788682933 35 253 4 SQL> col path for a50 SQL> select disk_number,path from v$asm_disk where disk_number in (1,3,4) and GROUP_NUMBER=3; DISK_NUMBER PATH ----------- -------------------------------------------------- 3 /dev/rasm-diske 4 /dev/rasm-diskf 1 /dev/rasm-diskc ?????ASM SPFILE??????(redundancy=high),????? /dev/rasm-diskc?AU=39?/dev/rasm-diske AU=35?/dev/rasm-diskf AU=35? ????kfed?????????ASM DISK?header: [grid@maclean1 wallets]$ kfed read /dev/rasm-diske|grep spfile kfdhdb.spfile: 35 ; 0x0f4: 0x00000023 [grid@maclean1 wallets]$ kfed read /dev/rasm-diskc|grep spfile kfdhdb.spfile: 39 ; 0x0f4: 0x00000027 [grid@maclean1 wallets]$ kfed read /dev/rasm-diskf|grep spfile kfdhdb.spfile: 35 ; 0x0f4: 0x00000023 ????ASM disk header?kfdhdb.spfile??ASM SPFILE???DISK??AU NUMBER????, ASM???????????GPNP PROFILE?? DiscoveryString?????????,????ASM disk header?????kfdhdb.spfile??????,?????MOUNT DISKGROUP??????ASM SPFILE,?????ASM, ?????????????????

  Read the article

• New EPS: Oracle RAC 11g & Grid Infrastructure Administration

  - by Brandye Barrington

  I'm happy to announce the availability of a brand new Exam Prep Seminar titled Certification Exam Prep Seminar: Oracle RAC 11g and Grid Infrastructure Administration. This new Exam Prep Seminar is available standalone, or through a Certification Value Package, which includes (1) the Seminar, and (2) a certification exam voucher with a free retake. For those of you preparing for the Oracle Certified Expert (OCE) - Oracle Real Application Clusters 11g and Grid Infrastructure certification, this seminar is a great value and and an excellent way to gain valuable insight from one of Oracle University's top Database instructors. This Exam Prep Seminar will accelerate your preparation, make your prep time more efficient and give you insight to the breadth and depth of the certification exam. This type of exam preparation has traditionally only been available at the Oracle OpenWorld conference, but is now available to anyone through this new format. Of course with online video, you can now start, stop, rewind, and review as needed! Also note that because this seminar is in the Oracle Training On Demand format, you can also watch it on your your iPad through Oracle University's new free iPad app. QUICK LINKS Oracle Certified Expert (OCE) - Oracle Real Application Clusters 11g and Grid Infrastructure Oracle RAC 11g & Grid Infrastructure Admin. Certified Expert Certification Value Package

  Read the article

• ?Linux 6???UDEV??RAC ASM???????

  - by Liu Maclean(???)

  Maclean?????UDEV??ASMLIB?RAC???????????,???????????????????:Why ASMLIB and why not???UDEV????RAC ASM?????  ?«??UDEV????RAC ASM????? »???????????????,????????udev rule????: for i in b c d e f g h i j k ; do echo "KERNEL==\"sd*\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id -g -u -s %p\", RESULT==\"`scsi_id -g -u -s /block/sd$i`\", NAME=\"asm-disk$i\", OWNER=\"grid\", GROUP=\"asmadmin\", MODE=\"0660\"" done ?????Linux 5?????, ????????redhat/Oracle Linux 6???????????? ????: ?OEL6??RHEL6?,????????? ??????:1. scsi_id??????????,scsi_id -g -u -s??????????2. udevtest???????,????udevadm??How to use udev for Oracle ASM in Oracle Linux 6   ???????????,???redhat/Oracle Linux 6??????udev rule ????: 1. #????? Linux 6.0???? [root@vrh6 dev]# cat /etc/issue Oracle Linux Server release 6.2 Kernel \r on an \m 2. #?????/etc/scsi_id.config echo "options=--whitelisted --replace-whitespace" >> /etc/scsi_id.config 3. #?????????udev?? [root@vrh6 dev]# ls -l sd* brw-rw----. 1 root disk 8, 0 Jun 30 09:29 sda brw-rw----. 1 root disk 8, 1 Jun 30 09:29 sda1 brw-rw----. 1 root disk 8, 2 Jun 30 09:29 sda2 brw-rw----. 1 root disk 8, 16 Jun 30 09:29 sdb brw-rw----. 1 root disk 8, 32 Jun 30 09:29 sdc brw-rw----. 1 root disk 8, 48 Jun 30 09:29 sdd brw-rw----. 1 root disk 8, 64 Jun 30 09:29 sde brw-rw----. 1 root disk 8, 80 Jun 30 09:29 sdf ??????? sdb-> sdf???????? 4. ? b->f?????for ???,??: # AUTO UDEV RULE BY Maclean Liu 2012/06/30 for i in b c d e f ; do echo "KERNEL==\"sd*\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/\$name\", RESULT==\"`/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/sd$i`\", NAME=\"asm-disk$i\", OWNER=\"grid\", GROUP=\"asmadmin\", MODE=\"0660\"" done ????sdb->sdf ?????RULE,????RULE???/etc/udev/rules.d/99-oracle-asmdevices.rules? ??????????? ,??RULE?99-oracle-asmdevices.rules # AUTO UDEV RULE BY Maclean Liu 2012/06/30 for i in b c d e f ; do echo "KERNEL==\"sd*\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/\$name\", RESULT==\"`/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/sd$i`\", NAME=\"asm-disk$i\", OWNER=\"grid\", GROUP=\"asmadmin\", MODE=\"0660\"" >> /etc/udev/rules.d/99-oracle-asmdevices.rules done 5. ?????root??/sbin/start_udev ?? ??????: [root@vrh6 dev]# echo "options=--whitelisted --replace-whitespace" >> /etc/scsi_id.config [root@vrh6 dev]# for i in b c d e f ; > do > echo "KERNEL==\"sd*\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/\$name\", RESULT==\"`/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/sd$i`\", NAME=\"asm-disk$i\", OWNER=\"grid\", GROUP=\"asmadmin\", MODE=\"0660\"" >> /etc/udev/rules.d/99-oracle-asmdevices.rules > done [root@vrh6 dev]# [root@vrh6 dev]# cat /etc/udev/rules.d/99-oracle-asmdevices.rules KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB09cadb31-cfbea255", NAME="asm-diskb", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB5f097069-59efb82f", NAME="asm-diskc", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB4e1a81c0-20478bc4", NAME="asm-diskd", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VBdcce9285-b13c5a27", NAME="asm-diske", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="1ATA_VBOX_HARDDISK_VB82effe1a-dbca7dff", NAME="asm-diskf", OWNER="grid", GROUP="asmadmin", MODE="0660" [root@vrh6 dev]# [root@vrh6 dev]# /sbin/start_udev Starting udev: [ OK ] [root@vrh6 dev]# ls -l asm* brw-rw----. 1 grid asmadmin 8, 16 Jun 30 09:34 asm-diskb brw-rw----. 1 grid asmadmin 8, 32 Jun 30 09:34 asm-diskc brw-rw----. 1 grid asmadmin 8, 48 Jun 30 09:34 asm-diskd brw-rw----. 1 grid asmadmin 8, 64 Jun 30 09:34 asm-diske brw-rw----. 1 grid asmadmin 8, 80 Jun 30 09:34 asm-diskf

  Read the article

• ??11.2 RAC??OCR?Votedisk??ASM Diskgroup?????

  - by Liu Maclean(???)

  ????????Oracle Allstarts??????????ocr?votedisk?ASM diskgroup??11gR2 RAC cluster?????????,????«?11gR2 RAC???ASM DISK Path????»??????,??????CRS??????11.2??ASM???????, ????????????”crsctl start crs -excl -nocrs “; ?????????,??ASM????ocr?????votedisk?????,??11.2????ocr?votedisk???ASM?,?ASM???????ocr?votedisk,?????ocr?votedisk????????cluter??????;???????????CRS????,?????diskgroup??????????,?????????????????? ??:?????????????????ASM LUN DISK,???OCR?????,????????4??????????,???????$GI_HOME,?????????;????votedisk?? ????: ??dd????ocr?votedisk??diskgroup header,??diskgroup corruption: 1. ??votedisk? ocr?? [root@vrh1 ~]# crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE a853d6204bbc4feabfd8c73d4c3b3001 (/dev/asm-diskh) [SYSTEMDG] 2. ONLINE a5b37704c3574f0fbf21d1d9f58c4a6b (/dev/asm-diskg) [SYSTEMDG] 3. ONLINE 36e5c51ff0294fc3bf2a042266650331 (/dev/asm-diski) [SYSTEMDG] 4. ONLINE af337d1512824fe4bf6ad45283517aaa (/dev/asm-diskj) [SYSTEMDG] 5. ONLINE 3c4a349e2e304ff6bf64b2b1c9d9cf5d (/dev/asm-diskk) [SYSTEMDG] Located 5 voting disk(s). su - grid [grid@vrh1 ~]$ ocrconfig -showbackup PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy vrh1 2012/08/09 01:59:56 /g01/11.2.0/maclean/grid/cdata/vrh-cluster/backup00.ocr vrh1 2012/08/08 21:59:56 /g01/11.2.0/maclean/grid/cdata/vrh-cluster/backup01.ocr vrh1 2012/08/08 17:59:55 /g01/11.2.0/maclean/grid/cdata/vrh-cluster/backup02.ocr vrh1 2012/08/08 05:59:54 /g01/11.2.0/grid/cdata/vrh-cluster/day.ocr vrh1 2012/08/08 05:59:54 /g01/11.2.0/grid/cdata/vrh-cluster/week.ocr PROT-25: Manual backups for the Oracle Cluster Registry are not available 2. ??????????clusterware ,OHASD crsctl stop has -f 3. GetAsmDH.sh ==> GetAsmDH.sh?ASM disk header????? ????????,????????asm header [grid@vrh1 ~]$ ./GetAsmDH.sh ############################################ 1) Collecting Information About the Disks: ############################################ SQL*Plus: Release 11.2.0.3.0 Production on Thu Aug 9 03:28:13 2012 Copyright (c) 1982, 2011, Oracle. All rights reserved. SQL> Connected. SQL> SQL> SQL> SQL> SQL> SQL> SQL> 1 0 /dev/asm-diske 1 1 /dev/asm-diskd 2 0 /dev/asm-diskb 2 1 /dev/asm-diskc 2 2 /dev/asm-diskf 3 0 /dev/asm-diskh 3 1 /dev/asm-diskg 3 2 /dev/asm-diski 3 3 /dev/asm-diskj 3 4 /dev/asm-diskk SQL> SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options -rw-r--r-- 1 grid oinstall 1048 Aug 9 03:28 /tmp/HC/asmdisks.lst ############################################ 2) Generating asm_diskh.sh script. ############################################ -rwx------ 1 grid oinstall 666 Aug 9 03:28 /tmp/HC/asm_diskh.sh ############################################ 3) Executing asm_diskh.sh script to generate dd dumps. ############################################ -rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_1_0.dd -rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_1_1.dd -rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_2_0.dd -rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_2_1.dd -rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_2_2.dd -rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_3_0.dd -rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_3_1.dd -rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_3_2.dd -rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_3_3.dd -rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_3_4.dd ############################################ 4) Compressing dd dumps in the next format: (asm_dd_header_all_.tar) ############################################ /tmp/HC/dsk_1_0.dd /tmp/HC/dsk_1_1.dd /tmp/HC/dsk_2_0.dd /tmp/HC/dsk_2_1.dd /tmp/HC/dsk_2_2.dd /tmp/HC/dsk_3_0.dd /tmp/HC/dsk_3_1.dd /tmp/HC/dsk_3_2.dd /tmp/HC/dsk_3_3.dd /tmp/HC/dsk_3_4.dd ./GetAsmDH.sh: line 81: compress: command not found ls: /tmp/HC/*.Z: No such file or directory [grid@vrh1 ~]$ 4. ??dd ?? ??ocr?votedisk??diskgroup [root@vrh1 ~]# dd if=/dev/zero of=/dev/asm-diskh bs=1024k count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00423853 seconds, 247 MB/s [root@vrh1 ~]# dd if=/dev/zero of=/dev/asm-diskg bs=1024k count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0045179 seconds, 232 MB/s [root@vrh1 ~]# dd if=/dev/zero of=/dev/asm-diski bs=1024k count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00469976 seconds, 223 MB/s [root@vrh1 ~]# dd if=/dev/zero of=/dev/asm-diskj bs=1024k count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00344262 seconds, 305 MB/s [root@vrh1 ~]# dd if=/dev/zero of=/dev/asm-diskk bs=1024k count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0053518 seconds, 196 MB/s 5. ????????????HAS [root@vrh1 ~]# crsctl start has CRS-4123: Oracle High Availability Services has been started. ????ocr?votedisk??diskgroup??,??CSS???????,???????: alertvrh1.log [cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log 2012-08-09 03:35:41.207 [cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log 2012-08-09 03:35:56.240 [cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log 2012-08-09 03:36:11.284 [cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log 2012-08-09 03:36:26.305 [cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log 2012-08-09 03:36:41.328 ocssd.log 2012-08-09 03:40:26.662: [ CSSD][1078700352]clssnmReadDiscoveryProfile: voting file discovery string(/dev/asm*) 2012-08-09 03:40:26.662: [ CSSD][1078700352]clssnmvDDiscThread: using discovery string /dev/asm* for initial discovery 2012-08-09 03:40:26.662: [ SKGFD][1078700352]Discovery with str:/dev/asm*: 2012-08-09 03:40:26.662: [ SKGFD][1078700352]UFS discovery with :/dev/asm*: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskf: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskb: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskj: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskh: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskc: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskd: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diske: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskg: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diski: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskk: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]OSS discovery with :/dev/asm*: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Handle 0xdf22a0 from lib :UFS:: for disk :/dev/asm-diskf: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Handle 0xf412a0 from lib :UFS:: for disk :/dev/asm-diskb: 2012-08-09 03:40:26.666: [ SKGFD][1078700352]Handle 0xf3a680 from lib :UFS:: for disk :/dev/asm-diskj: 2012-08-09 03:40:26.666: [ SKGFD][1078700352]Handle 0xf93da0 from lib :UFS:: for disk :/dev/asm-diskh: 2012-08-09 03:40:26.667: [ CSSD][1078700352]clssnmvDiskVerify: Successful discovery of 0 disks 2012-08-09 03:40:26.667: [ CSSD][1078700352]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery 2012-08-09 03:40:26.667: [ CSSD][1078700352]clssnmvFindInitialConfigs: No voting files found 2012-08-09 03:40:26.667: [ CSSD][1078700352](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds ?????ocr?votedisk??diskgroup?????: 1. ?-excl -nocrs ????cluster,??????ASM?? ????CRS [root@vrh1 vrh1]# crsctl start crs -excl -nocrs CRS-4123: Oracle High Availability Services has been started. CRS-2672: Attempting to start 'ora.mdnsd' on 'vrh1' CRS-2676: Start of 'ora.mdnsd' on 'vrh1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'vrh1' CRS-2676: Start of 'ora.gpnpd' on 'vrh1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'vrh1' CRS-2672: Attempting to start 'ora.gipcd' on 'vrh1' CRS-2676: Start of 'ora.cssdmonitor' on 'vrh1' succeeded CRS-2676: Start of 'ora.gipcd' on 'vrh1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'vrh1' CRS-2672: Attempting to start 'ora.diskmon' on 'vrh1' CRS-2676: Start of 'ora.diskmon' on 'vrh1' succeeded CRS-2676: Start of 'ora.cssd' on 'vrh1' succeeded CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'vrh1' CRS-2672: Attempting to start 'ora.ctssd' on 'vrh1' CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'vrh1' succeeded CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'vrh1' CRS-2676: Start of 'ora.ctssd' on 'vrh1' succeeded CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'vrh1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'vrh1' CRS-2676: Start of 'ora.asm' on 'vrh1' succeeded 2.???ocr?votedisk??diskgroup,??compatible.asm???11.2: [root@vrh1 vrh1]# su - grid [grid@vrh1 ~]$ sqlplus / as sysasm SQL*Plus: Release 11.2.0.3.0 Production on Thu Aug 9 04:16:58 2012 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options SQL> create diskgroup systemdg high redundancy disk '/dev/asm-diskh','/dev/asm-diskg','/dev/asm-diski','/dev/asm-diskj','/dev/asm-diskk' ATTRIBUTE 'compatible.rdbms' = '11.2', 'compatible.asm' = '11.2'; 3.?ocr backup???ocr??ocrcheck??: [root@vrh1 ~]# ocrconfig -restore /g01/11.2.0/grid/cdata/vrh-cluster/backup00.ocr [root@vrh1 ~]# ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 3180 Available space (kbytes) : 258940 ID : 1238458014 Device/File Name : +systemdg Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded 4. ????votedisk ,??????????: [grid@vrh1 ~]$ crsctl replace votedisk +SYSTEMDG CRS-4602: Failed 27 to add voting file 2e4e0fe285924f86bf5473d00dcc0388. CRS-4602: Failed 27 to add voting file 4fa54bb0cc5c4fafbf1a9be5479bf389. CRS-4602: Failed 27 to add voting file a109ead9ea4e4f28bfe233188623616a. CRS-4602: Failed 27 to add voting file 042c9fbd71b54f5abfcd3ab3408f3cf3. CRS-4602: Failed 27 to add voting file 7b5a8cd24f954fafbf835ad78615763f. Failed to replace voting disk group with +SYSTEMDG. CRS-4000: Command Replace failed, or completed with errors. ????????ASM???,???ASM: SQL> alter system set asm_diskstring='/dev/asm*'; System altered. SQL> create spfile from memory; File created. SQL> startup force mount; ORA-32004: obsolete or deprecated parameter(s) specified for ASM instance ASM instance started Total System Global Area 283930624 bytes Fixed Size 2227664 bytes Variable Size 256537136 bytes ASM Cache 25165824 bytes ASM diskgroups mounted SQL> show parameter spfile NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ spfile string /g01/11.2.0/grid/dbs/spfile+AS M1.ora [grid@vrh1 trace]$ crsctl replace votedisk +SYSTEMDG CRS-4256: Updating the profile Successful addition of voting disk 85edc0e82d274f78bfc58cdc73b8c68a. Successful addition of voting disk 201ffffc8ba44faabfe2efec2aa75840. Successful addition of voting disk 6f2a25c589964faabf6980f7c5f621ce. Successful addition of voting disk 93eb315648454f25bf3717df1a2c73d5. Successful addition of voting disk 3737240678964f88bfbfbd31d8b3829f. Successfully replaced voting disk group with +SYSTEMDG. CRS-4256: Updating the profile CRS-4266: Voting file(s) successfully replaced 5. ??has??,??cluster????: [root@vrh1 ~]# crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online [root@vrh1 ~]# crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 85edc0e82d274f78bfc58cdc73b8c68a (/dev/asm-diskh) [SYSTEMDG] 2. ONLINE 201ffffc8ba44faabfe2efec2aa75840 (/dev/asm-diskg) [SYSTEMDG] 3. ONLINE 6f2a25c589964faabf6980f7c5f621ce (/dev/asm-diski) [SYSTEMDG] 4. ONLINE 93eb315648454f25bf3717df1a2c73d5 (/dev/asm-diskj) [SYSTEMDG] 5. ONLINE 3737240678964f88bfbfbd31d8b3829f (/dev/asm-diskk) [SYSTEMDG] Located 5 voting disk(s). [root@vrh1 ~]# crsctl stat res -t -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.BACKUPDG.dg ONLINE ONLINE vrh1 ora.DATA.dg ONLINE ONLINE vrh1 ora.LISTENER.lsnr ONLINE ONLINE vrh1 ora.LSN_MACLEAN.lsnr ONLINE ONLINE vrh1 ora.SYSTEMDG.dg ONLINE ONLINE vrh1 ora.asm ONLINE ONLINE vrh1 Started ora.gsd OFFLINE OFFLINE vrh1 ora.net1.network ONLINE ONLINE vrh1 ora.ons ONLINE ONLINE vrh1 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr http://www.askmaclean.com 1 ONLINE ONLINE vrh1 ora.cvu 1 OFFLINE OFFLINE ora.oc4j 1 OFFLINE OFFLINE ora.scan1.vip 1 ONLINE ONLINE vrh1 ora.vprod.db 1 ONLINE OFFLINE 2 ONLINE OFFLINE ora.vrh1.vip 1 ONLINE ONLINE vrh1 ora.vrh2.vip 1 ONLINE INTERMEDIATE vrh1 FAILED OVER

  Read the article

• ??11.2 RAC??OCR?Votedisk??ASM Diskgroup?????

  - by Liu Maclean(???)

  ????????Oracle Allstarts??????????ocr?votedisk?ASM diskgroup??11gR2 RAC cluster?????????,????«?11gR2 RAC???ASM DISK Path????»??????,??????CRS??????11.2??ASM???????, ????????????”crsctl start crs -excl -nocrs “; ?????????,??ASM????ocr?????votedisk?????,??11.2????ocr?votedisk???ASM?,?ASM???????ocr?votedisk,?????ocr?votedisk????????cluter??????;???????????CRS????,?????diskgroup??????????,?????????????????? ??:?????????????????ASM LUN DISK,???OCR?????,????????4??????????,???????$GI_HOME,?????????;????votedisk?? ????: ??dd????ocr?votedisk??diskgroup header,??diskgroup corruption: 1. ??votedisk? ocr?? [root@vrh1 ~]# crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE a853d6204bbc4feabfd8c73d4c3b3001 (/dev/asm-diskh) [SYSTEMDG] 2. ONLINE a5b37704c3574f0fbf21d1d9f58c4a6b (/dev/asm-diskg) [SYSTEMDG] 3. ONLINE 36e5c51ff0294fc3bf2a042266650331 (/dev/asm-diski) [SYSTEMDG] 4. ONLINE af337d1512824fe4bf6ad45283517aaa (/dev/asm-diskj) [SYSTEMDG] 5. ONLINE 3c4a349e2e304ff6bf64b2b1c9d9cf5d (/dev/asm-diskk) [SYSTEMDG] Located 5 voting disk(s). su - grid [grid@vrh1 ~]$ ocrconfig -showbackup PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy vrh1 2012/08/09 01:59:56 /g01/11.2.0/maclean/grid/cdata/vrh-cluster/backup00.ocr vrh1 2012/08/08 21:59:56 /g01/11.2.0/maclean/grid/cdata/vrh-cluster/backup01.ocr vrh1 2012/08/08 17:59:55 /g01/11.2.0/maclean/grid/cdata/vrh-cluster/backup02.ocr vrh1 2012/08/08 05:59:54 /g01/11.2.0/grid/cdata/vrh-cluster/day.ocr vrh1 2012/08/08 05:59:54 /g01/11.2.0/grid/cdata/vrh-cluster/week.ocr PROT-25: Manual backups for the Oracle Cluster Registry are not available 2. ??????????clusterware ,OHASD crsctl stop has -f 3. GetAsmDH.sh ==> GetAsmDH.sh?ASM disk header????? ????????,????????asm header [grid@vrh1 ~]$ ./GetAsmDH.sh ############################################ 1) Collecting Information About the Disks: ############################################ SQL*Plus: Release 11.2.0.3.0 Production on Thu Aug 9 03:28:13 2012 Copyright (c) 1982, 2011, Oracle. All rights reserved. SQL> Connected. SQL> SQL> SQL> SQL> SQL> SQL> SQL> 1 0 /dev/asm-diske 1 1 /dev/asm-diskd 2 0 /dev/asm-diskb 2 1 /dev/asm-diskc 2 2 /dev/asm-diskf 3 0 /dev/asm-diskh 3 1 /dev/asm-diskg 3 2 /dev/asm-diski 3 3 /dev/asm-diskj 3 4 /dev/asm-diskk SQL> SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options -rw-r--r-- 1 grid oinstall 1048 Aug 9 03:28 /tmp/HC/asmdisks.lst ############################################ 2) Generating asm_diskh.sh script. ############################################ -rwx------ 1 grid oinstall 666 Aug 9 03:28 /tmp/HC/asm_diskh.sh ############################################ 3) Executing asm_diskh.sh script to generate dd dumps. ############################################ -rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_1_0.dd -rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_1_1.dd -rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_2_0.dd -rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_2_1.dd -rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_2_2.dd -rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_3_0.dd -rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_3_1.dd -rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_3_2.dd -rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_3_3.dd -rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_3_4.dd ############################################ 4) Compressing dd dumps in the next format: (asm_dd_header_all_.tar) ############################################ /tmp/HC/dsk_1_0.dd /tmp/HC/dsk_1_1.dd /tmp/HC/dsk_2_0.dd /tmp/HC/dsk_2_1.dd /tmp/HC/dsk_2_2.dd /tmp/HC/dsk_3_0.dd /tmp/HC/dsk_3_1.dd /tmp/HC/dsk_3_2.dd /tmp/HC/dsk_3_3.dd /tmp/HC/dsk_3_4.dd ./GetAsmDH.sh: line 81: compress: command not found ls: /tmp/HC/*.Z: No such file or directory [grid@vrh1 ~]$ 4. ??dd ?? ??ocr?votedisk??diskgroup [root@vrh1 ~]# dd if=/dev/zero of=/dev/asm-diskh bs=1024k count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00423853 seconds, 247 MB/s [root@vrh1 ~]# dd if=/dev/zero of=/dev/asm-diskg bs=1024k count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0045179 seconds, 232 MB/s [root@vrh1 ~]# dd if=/dev/zero of=/dev/asm-diski bs=1024k count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00469976 seconds, 223 MB/s [root@vrh1 ~]# dd if=/dev/zero of=/dev/asm-diskj bs=1024k count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00344262 seconds, 305 MB/s [root@vrh1 ~]# dd if=/dev/zero of=/dev/asm-diskk bs=1024k count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0053518 seconds, 196 MB/s 5. ????????????HAS [root@vrh1 ~]# crsctl start has CRS-4123: Oracle High Availability Services has been started. ????ocr?votedisk??diskgroup??,??CSS???????,???????: alertvrh1.log [cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log 2012-08-09 03:35:41.207 [cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log 2012-08-09 03:35:56.240 [cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log 2012-08-09 03:36:11.284 [cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log 2012-08-09 03:36:26.305 [cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log 2012-08-09 03:36:41.328 ocssd.log 2012-08-09 03:40:26.662: [ CSSD][1078700352]clssnmReadDiscoveryProfile: voting file discovery string(/dev/asm*) 2012-08-09 03:40:26.662: [ CSSD][1078700352]clssnmvDDiscThread: using discovery string /dev/asm* for initial discovery 2012-08-09 03:40:26.662: [ SKGFD][1078700352]Discovery with str:/dev/asm*: 2012-08-09 03:40:26.662: [ SKGFD][1078700352]UFS discovery with :/dev/asm*: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskf: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskb: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskj: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskh: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskc: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskd: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diske: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskg: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diski: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskk: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]OSS discovery with :/dev/asm*: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Handle 0xdf22a0 from lib :UFS:: for disk :/dev/asm-diskf: 2012-08-09 03:40:26.665: [ SKGFD][1078700352]Handle 0xf412a0 from lib :UFS:: for disk :/dev/asm-diskb: 2012-08-09 03:40:26.666: [ SKGFD][1078700352]Handle 0xf3a680 from lib :UFS:: for disk :/dev/asm-diskj: 2012-08-09 03:40:26.666: [ SKGFD][1078700352]Handle 0xf93da0 from lib :UFS:: for disk :/dev/asm-diskh: 2012-08-09 03:40:26.667: [ CSSD][1078700352]clssnmvDiskVerify: Successful discovery of 0 disks 2012-08-09 03:40:26.667: [ CSSD][1078700352]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery 2012-08-09 03:40:26.667: [ CSSD][1078700352]clssnmvFindInitialConfigs: No voting files found 2012-08-09 03:40:26.667: [ CSSD][1078700352](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds ?????ocr?votedisk??diskgroup?????: 1. ?-excl -nocrs ????cluster,??????ASM?? ????CRS [root@vrh1 vrh1]# crsctl start crs -excl -nocrs CRS-4123: Oracle High Availability Services has been started. CRS-2672: Attempting to start 'ora.mdnsd' on 'vrh1' CRS-2676: Start of 'ora.mdnsd' on 'vrh1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'vrh1' CRS-2676: Start of 'ora.gpnpd' on 'vrh1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'vrh1' CRS-2672: Attempting to start 'ora.gipcd' on 'vrh1' CRS-2676: Start of 'ora.cssdmonitor' on 'vrh1' succeeded CRS-2676: Start of 'ora.gipcd' on 'vrh1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'vrh1' CRS-2672: Attempting to start 'ora.diskmon' on 'vrh1' CRS-2676: Start of 'ora.diskmon' on 'vrh1' succeeded CRS-2676: Start of 'ora.cssd' on 'vrh1' succeeded CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'vrh1' CRS-2672: Attempting to start 'ora.ctssd' on 'vrh1' CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'vrh1' succeeded CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'vrh1' CRS-2676: Start of 'ora.ctssd' on 'vrh1' succeeded CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'vrh1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'vrh1' CRS-2676: Start of 'ora.asm' on 'vrh1' succeeded 2.???ocr?votedisk??diskgroup,??compatible.asm???11.2: [root@vrh1 vrh1]# su - grid [grid@vrh1 ~]$ sqlplus / as sysasm SQL*Plus: Release 11.2.0.3.0 Production on Thu Aug 9 04:16:58 2012 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options SQL> create diskgroup systemdg high redundancy disk '/dev/asm-diskh','/dev/asm-diskg','/dev/asm-diski','/dev/asm-diskj','/dev/asm-diskk' ATTRIBUTE 'compatible.rdbms' = '11.2', 'compatible.asm' = '11.2'; 3.?ocr backup???ocr??ocrcheck??: [root@vrh1 ~]# ocrconfig -restore /g01/11.2.0/grid/cdata/vrh-cluster/backup00.ocr [root@vrh1 ~]# ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 3180 Available space (kbytes) : 258940 ID : 1238458014 Device/File Name : +systemdg Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded 4. ????votedisk ,??????????: [grid@vrh1 ~]$ crsctl replace votedisk +SYSTEMDG CRS-4602: Failed 27 to add voting file 2e4e0fe285924f86bf5473d00dcc0388. CRS-4602: Failed 27 to add voting file 4fa54bb0cc5c4fafbf1a9be5479bf389. CRS-4602: Failed 27 to add voting file a109ead9ea4e4f28bfe233188623616a. CRS-4602: Failed 27 to add voting file 042c9fbd71b54f5abfcd3ab3408f3cf3. CRS-4602: Failed 27 to add voting file 7b5a8cd24f954fafbf835ad78615763f. Failed to replace voting disk group with +SYSTEMDG. CRS-4000: Command Replace failed, or completed with errors. ????????ASM???,???ASM: SQL> alter system set asm_diskstring='/dev/asm*'; System altered. SQL> create spfile from memory; File created. SQL> startup force mount; ORA-32004: obsolete or deprecated parameter(s) specified for ASM instance ASM instance started Total System Global Area 283930624 bytes Fixed Size 2227664 bytes Variable Size 256537136 bytes ASM Cache 25165824 bytes ASM diskgroups mounted SQL> show parameter spfile NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ spfile string /g01/11.2.0/grid/dbs/spfile+AS M1.ora [grid@vrh1 trace]$ crsctl replace votedisk +SYSTEMDG CRS-4256: Updating the profile Successful addition of voting disk 85edc0e82d274f78bfc58cdc73b8c68a. Successful addition of voting disk 201ffffc8ba44faabfe2efec2aa75840. Successful addition of voting disk 6f2a25c589964faabf6980f7c5f621ce. Successful addition of voting disk 93eb315648454f25bf3717df1a2c73d5. Successful addition of voting disk 3737240678964f88bfbfbd31d8b3829f. Successfully replaced voting disk group with +SYSTEMDG. CRS-4256: Updating the profile CRS-4266: Voting file(s) successfully replaced 5. ??has??,??cluster????: [root@vrh1 ~]# crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online [root@vrh1 ~]# crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 85edc0e82d274f78bfc58cdc73b8c68a (/dev/asm-diskh) [SYSTEMDG] 2. ONLINE 201ffffc8ba44faabfe2efec2aa75840 (/dev/asm-diskg) [SYSTEMDG] 3. ONLINE 6f2a25c589964faabf6980f7c5f621ce (/dev/asm-diski) [SYSTEMDG] 4. ONLINE 93eb315648454f25bf3717df1a2c73d5 (/dev/asm-diskj) [SYSTEMDG] 5. ONLINE 3737240678964f88bfbfbd31d8b3829f (/dev/asm-diskk) [SYSTEMDG] Located 5 voting disk(s). [root@vrh1 ~]# crsctl stat res -t -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.BACKUPDG.dg ONLINE ONLINE vrh1 ora.DATA.dg ONLINE ONLINE vrh1 ora.LISTENER.lsnr ONLINE ONLINE vrh1 ora.LSN_MACLEAN.lsnr ONLINE ONLINE vrh1 ora.SYSTEMDG.dg ONLINE ONLINE vrh1 ora.asm ONLINE ONLINE vrh1 Started ora.gsd OFFLINE OFFLINE vrh1 ora.net1.network ONLINE ONLINE vrh1 ora.ons ONLINE ONLINE vrh1 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr http://www.askmaclean.com 1 ONLINE ONLINE vrh1 ora.cvu 1 OFFLINE OFFLINE ora.oc4j 1 OFFLINE OFFLINE ora.scan1.vip 1 ONLINE ONLINE vrh1 ora.vprod.db 1 ONLINE OFFLINE 2 ONLINE OFFLINE ora.vrh1.vip 1 ONLINE ONLINE vrh1 ora.vrh2.vip 1 ONLINE INTERMEDIATE vrh1 FAILED OVER

  Read the article

• SPARC T5-4 LDoms for RAC and WebLogic Clusters

  - by Jeff Taylor-Oracle

  I wanted to use two Oracle SPARC T5-4 servers to simultaneously host both Oracle RAC and a WebLogic Server Cluster. I chose to use Oracle VM Server for SPARC to create a cluster like this: There are plenty of trade offs and decisions that need to be made, for example: Rather than configuring the system by hand, you might want to use an Oracle SuperCluster T5-8 My configuration is similar to jsavit's: Availability Best Practices - Example configuring a T5-8 but I chose to ignore some of the advice. Maybe I should have included an  alternate service domain, but I decided that I already had enough redundancy Both Oracle SPARC T5-4 servers were to be configured like this: Cntl 0.25  4  64GB                     App LDom                    2.75 CPU's                                        44 cores                                          704 GB              DB LDom      One CPU         16 cores         256 GB   The systems started with everything in the primary domain: # ldm list NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME primary          active     -n-c--  UART    512   1023G    0.0%  0.0%  11m # ldm list-spconfig factory-default [current] primary # ldm list -o core,memory,physio NAME              primary           CORE     CID    CPUSET     0      (0, 1, 2, 3, 4, 5, 6, 7)     1      (8, 9, 10, 11, 12, 13, 14, 15)     2      (16, 17, 18, 19, 20, 21, 22, 23) -- SNIP     62     (496, 497, 498, 499, 500, 501, 502, 503)     63     (504, 505, 506, 507, 508, 509, 510, 511) MEMORY     RA               PA               SIZE                 0x30000000       0x30000000       255G     0x80000000000    0x80000000000    256G     0x100000000000   0x100000000000   256G     0x180000000000   0x180000000000   256G # Give this memory block to the DB LDom IO     DEVICE                           PSEUDONYM        OPTIONS     pci@300                          pci_0                pci@340                          pci_1                pci@380                          pci_2                pci@3c0                          pci_3                pci@400                          pci_4                pci@440                          pci_5                pci@480                          pci_6                pci@4c0                          pci_7                pci@300/pci@1/pci@0/pci@6        /SYS/RCSA/PCIE1     pci@300/pci@1/pci@0/pci@c        /SYS/RCSA/PCIE2     pci@300/pci@1/pci@0/pci@4/pci@0/pci@c /SYS/MB/SASHBA0     pci@300/pci@1/pci@0/pci@4/pci@0/pci@8 /SYS/RIO/NET0        pci@340/pci@1/pci@0/pci@6        /SYS/RCSA/PCIE3     pci@340/pci@1/pci@0/pci@c        /SYS/RCSA/PCIE4     pci@380/pci@1/pci@0/pci@a        /SYS/RCSA/PCIE9     pci@380/pci@1/pci@0/pci@4        /SYS/RCSA/PCIE10     pci@3c0/pci@1/pci@0/pci@e        /SYS/RCSA/PCIE11     pci@3c0/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE12     pci@400/pci@1/pci@0/pci@e        /SYS/RCSA/PCIE5     pci@400/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE6     pci@440/pci@1/pci@0/pci@e        /SYS/RCSA/PCIE7     pci@440/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE8     pci@480/pci@1/pci@0/pci@a        /SYS/RCSA/PCIE13     pci@480/pci@1/pci@0/pci@4        /SYS/RCSA/PCIE14     pci@4c0/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE15     pci@4c0/pci@1/pci@0/pci@4        /SYS/RCSA/PCIE16     pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c /SYS/MB/SASHBA1     pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4 /SYS/RIO/NET2    Added an additional service processor configuration: # ldm add-spconfig split # ldm list-spconfig factory-default primary split [current] And removed many of the resources from the primary domain: # ldm start-reconf primary # ldm set-core 4 primary # ldm set-memory 32G primary # ldm rm-io pci@340 primary # ldm rm-io pci@380 primary # ldm rm-io pci@3c0 primary # ldm rm-io pci@400 primary # ldm rm-io pci@440 primary # ldm rm-io pci@480 primary # ldm rm-io pci@4c0 primary # init 6 Needed to add resources to the guest domains: # ldm add-domain db # ldm set-core cid=`seq -s"," 48 63` db # ldm add-memory mblock=0x180000000000:256G db # ldm add-io pci@480 db # ldm add-io pci@4c0 db # ldm add-domain app # ldm set-core 44 app # ldm set-memory 704G  app # ldm add-io pci@340 app # ldm add-io pci@380 app # ldm add-io pci@3c0 app # ldm add-io pci@400 app # ldm add-io pci@440 app Needed to set up services: # ldm add-vds primary-vds0 primary # ldm add-vcc port-range=5000-5100 primary-vcc0 primary Needed to add a virtual network port for the WebLogic application domain: # ipadm NAME              CLASS/TYPE STATE        UNDER      ADDR lo0               loopback   ok           --         --    lo0/v4         static     ok           --         ...    lo0/v6         static     ok           --         ... net0              ip         ok           --         ...    net0/v4        static     ok           --         xxx.xxx.xxx.xxx/24    net0/v6        addrconf   ok           --         ....    net0/v6        addrconf   ok           --         ... net8              ip         ok           --         --    net8/v4        static     ok           --         ... # dladm show-phys LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE net1              Ethernet             unknown    0      unknown   ixgbe1 net0              Ethernet             up         1000   full      ixgbe0 net8              Ethernet             up         10     full      usbecm2 # ldm add-vsw net-dev=net0 primary-vsw0 primary # ldm add-vnet vnet1 primary-vsw0 app Needed to add a virtual disk to the WebLogic application domain: # format Searching for disks...done AVAILABLE DISK SELECTIONS:        0. c0t5000CCA02505F874d0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB>           /scsi_vhci/disk@g5000cca02505f874           /dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD0/disk        1. c0t5000CCA02506C468d0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB>           /scsi_vhci/disk@g5000cca02506c468           /dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD1/disk        2. c0t5000CCA025067E5Cd0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB>           /scsi_vhci/disk@g5000cca025067e5c           /dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD2/disk        3. c0t5000CCA02506C258d0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB>           /scsi_vhci/disk@g5000cca02506c258           /dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD3/disk Specify disk (enter its number): ^C # ldm add-vdsdev /dev/dsk/c0t5000CCA02506C468d0s2 HDD1@primary-vds0 # ldm add-vdisk HDD1 HDD1@primary-vds0 app Add some additional spice to the pot: # ldm set-variable auto-boot\\?=false db # ldm set-variable auto-boot\\?=false app # ldm set-var boot-device=HDD1 app Bind the logical domains: # ldm bind db # ldm bind app At the end of the process, the system is set up like this: # ldm list -o core,memory,physio NAME             primary          CORE     CID    CPUSET     0      (0, 1, 2, 3, 4, 5, 6, 7)     1      (8, 9, 10, 11, 12, 13, 14, 15)     2      (16, 17, 18, 19, 20, 21, 22, 23)     3      (24, 25, 26, 27, 28, 29, 30, 31) MEMORY     RA               PA               SIZE                0x30000000       0x30000000       32G IO     DEVICE                           PSEUDONYM        OPTIONS     pci@300                          pci_0               pci@300/pci@1/pci@0/pci@6        /SYS/RCSA/PCIE1     pci@300/pci@1/pci@0/pci@c        /SYS/RCSA/PCIE2     pci@300/pci@1/pci@0/pci@4/pci@0/pci@c /SYS/MB/SASHBA0     pci@300/pci@1/pci@0/pci@4/pci@0/pci@8 /SYS/RIO/NET0   ------------------------------------------------------------------------------ NAME             app              CORE     CID    CPUSET     4      (32, 33, 34, 35, 36, 37, 38, 39)     5      (40, 41, 42, 43, 44, 45, 46, 47)     6      (48, 49, 50, 51, 52, 53, 54, 55)     7      (56, 57, 58, 59, 60, 61, 62, 63)     8      (64, 65, 66, 67, 68, 69, 70, 71)     9      (72, 73, 74, 75, 76, 77, 78, 79)     10     (80, 81, 82, 83, 84, 85, 86, 87)     11     (88, 89, 90, 91, 92, 93, 94, 95)     12     (96, 97, 98, 99, 100, 101, 102, 103)     13     (104, 105, 106, 107, 108, 109, 110, 111)     14     (112, 113, 114, 115, 116, 117, 118, 119)     15     (120, 121, 122, 123, 124, 125, 126, 127)     16     (128, 129, 130, 131, 132, 133, 134, 135)     17     (136, 137, 138, 139, 140, 141, 142, 143)     18     (144, 145, 146, 147, 148, 149, 150, 151)     19     (152, 153, 154, 155, 156, 157, 158, 159)     20     (160, 161, 162, 163, 164, 165, 166, 167)     21     (168, 169, 170, 171, 172, 173, 174, 175)     22     (176, 177, 178, 179, 180, 181, 182, 183)     23     (184, 185, 186, 187, 188, 189, 190, 191)     24     (192, 193, 194, 195, 196, 197, 198, 199)     25     (200, 201, 202, 203, 204, 205, 206, 207)     26     (208, 209, 210, 211, 212, 213, 214, 215)     27     (216, 217, 218, 219, 220, 221, 222, 223)     28     (224, 225, 226, 227, 228, 229, 230, 231)     29     (232, 233, 234, 235, 236, 237, 238, 239)     30     (240, 241, 242, 243, 244, 245, 246, 247)     31     (248, 249, 250, 251, 252, 253, 254, 255)     32     (256, 257, 258, 259, 260, 261, 262, 263)     33     (264, 265, 266, 267, 268, 269, 270, 271)     34     (272, 273, 274, 275, 276, 277, 278, 279)     35     (280, 281, 282, 283, 284, 285, 286, 287)     36     (288, 289, 290, 291, 292, 293, 294, 295)     37     (296, 297, 298, 299, 300, 301, 302, 303)     38     (304, 305, 306, 307, 308, 309, 310, 311)     39     (312, 313, 314, 315, 316, 317, 318, 319)     40     (320, 321, 322, 323, 324, 325, 326, 327)     41     (328, 329, 330, 331, 332, 333, 334, 335)     42     (336, 337, 338, 339, 340, 341, 342, 343)     43     (344, 345, 346, 347, 348, 349, 350, 351)     44     (352, 353, 354, 355, 356, 357, 358, 359)     45     (360, 361, 362, 363, 364, 365, 366, 367)     46     (368, 369, 370, 371, 372, 373, 374, 375)     47     (376, 377, 378, 379, 380, 381, 382, 383) MEMORY     RA               PA               SIZE                0x30000000       0x830000000      192G     0x4000000000     0x80000000000    256G     0x8080000000     0x100000000000   256G IO     DEVICE                           PSEUDONYM        OPTIONS     pci@340                          pci_1               pci@380                          pci_2               pci@3c0                          pci_3               pci@400                          pci_4               pci@440                          pci_5               pci@340/pci@1/pci@0/pci@6        /SYS/RCSA/PCIE3     pci@340/pci@1/pci@0/pci@c        /SYS/RCSA/PCIE4     pci@380/pci@1/pci@0/pci@a        /SYS/RCSA/PCIE9     pci@380/pci@1/pci@0/pci@4        /SYS/RCSA/PCIE10     pci@3c0/pci@1/pci@0/pci@e        /SYS/RCSA/PCIE11     pci@3c0/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE12     pci@400/pci@1/pci@0/pci@e        /SYS/RCSA/PCIE5     pci@400/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE6     pci@440/pci@1/pci@0/pci@e        /SYS/RCSA/PCIE7     pci@440/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE8 ------------------------------------------------------------------------------ NAME             db               CORE     CID    CPUSET     48     (384, 385, 386, 387, 388, 389, 390, 391)     49     (392, 393, 394, 395, 396, 397, 398, 399)     50     (400, 401, 402, 403, 404, 405, 406, 407)     51     (408, 409, 410, 411, 412, 413, 414, 415)     52     (416, 417, 418, 419, 420, 421, 422, 423)     53     (424, 425, 426, 427, 428, 429, 430, 431)     54     (432, 433, 434, 435, 436, 437, 438, 439)     55     (440, 441, 442, 443, 444, 445, 446, 447)     56     (448, 449, 450, 451, 452, 453, 454, 455)     57     (456, 457, 458, 459, 460, 461, 462, 463)     58     (464, 465, 466, 467, 468, 469, 470, 471)     59     (472, 473, 474, 475, 476, 477, 478, 479)     60     (480, 481, 482, 483, 484, 485, 486, 487)     61     (488, 489, 490, 491, 492, 493, 494, 495)     62     (496, 497, 498, 499, 500, 501, 502, 503)     63     (504, 505, 506, 507, 508, 509, 510, 511) MEMORY     RA               PA               SIZE                0x80000000       0x180000000000   256G IO     DEVICE                           PSEUDONYM        OPTIONS     pci@480                          pci_6               pci@4c0                          pci_7               pci@480/pci@1/pci@0/pci@a        /SYS/RCSA/PCIE13     pci@480/pci@1/pci@0/pci@4        /SYS/RCSA/PCIE14     pci@4c0/pci@1/pci@0/pci@8        /SYS/RCSA/PCIE15     pci@4c0/pci@1/pci@0/pci@4        /SYS/RCSA/PCIE16     pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c /SYS/MB/SASHBA1     pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4 /SYS/RIO/NET2   Start the domains: # ldm start app LDom app started # ldm start db LDom db started Make sure to start the vntsd service that was created, above. # svcs -a | grep ldo disabled        8:38:38 svc:/ldoms/vntsd:default online          8:38:58 svc:/ldoms/agents:default online          8:39:25 svc:/ldoms/ldmd:default # svcadm enable vntsd Now use the MAC address to configure the Solaris 11 Automated Installation. Database Logical Domain # telnet localhost 5000 {0} ok devalias screen                   /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@7/display@0 disk7                    /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0/disk@p3 disk6                    /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0/disk@p2 disk5                    /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0/disk@p1 disk4                    /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0/disk@p0 scsi1                    /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0 net3                     /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4/network@0,1 net2                     /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4/network@0 virtual-console          /virtual-devices/console@1 name                     aliases {0} ok boot net2 Boot device: /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4/network@0  File and args: 1000 Mbps full duplex Link up Requesting Internet Address for xx:xx:xx:xx:xx:xx Requesting Internet Address for xx:xx:xx:xx:xx:xx WLS Logical Domain # telnet localhost 5001 {0} ok devalias hdd1                     /virtual-devices@100/channel-devices@200/disk@0 vnet1                    /virtual-devices@100/channel-devices@200/network@0 net                      /virtual-devices@100/channel-devices@200/network@0 disk                     /virtual-devices@100/channel-devices@200/disk@0 virtual-console          /virtual-devices/console@1 name                     aliases {0} ok boot net Boot device: /virtual-devices@100/channel-devices@200/network@0  File and args: Requesting Internet Address for xx:xx:xx:xx:xx:xx Requesting Internet Address for xx:xx:xx:xx:xx:xx Repeat the process for the second SPARC T5-4, install Solaris, RAC and WebLogic Cluster, and you are ready to go. Maybe buying a SuperCluster would have been easier.

  Read the article

• ?????????????????????TCO???????????????????WebLogic???????|WebLogic Channel|??????

  - by ???02

  ????????????/????????????????????????????·???????????????????????????????????????????????????????????WebLogic Server??????(????????)???????????????????????????Oracle Database??????????????????????TCO??????????????????·??????·??????????WebLogic Server???????????????????????(???)???????????????????????????????????????????WebLogic Server?????????????????????????????Oracle Database?????????????????????????????WebLogic Server????????1???Oracle RAC???????????·?????????WebLogic Server??????Oracle Database?????????·??????·?????????????????????????????Oracle Real Application Clusters(RAC)?????????????????????? WebLogic Server???Oracle RAC???????????Active GridLink for RAC???????????????????????Oracle RAC???WebLogic Server?????????????????????????????????Oracle RAC????????????????????????????1?????????????????????????????????????????????????????????????Oracle RAC???"??"?????????????????????????????????·???????????????????????????????????????????????????????·??????????????????????????????????????WebLogic Server?Oracle RAC?????????????????????????Oracle RAC????????????????????????????????WebLogic Server??Oracle RAC????????????????·?????????????????????????????·?????Oracle WebLogic Server?Real Application Clusters? ···?????·???????Oracle WebLogic Server 10.3?????????Oracle RAC?????????????????????????????????????????????????????????????????????????????????????????????????????????????????Oracle WebLogic Server JDBC???·???????Oracle RAC?????????????????????????????????????·????????????????????DB?Oracle Coherence???????????/???????????????????/???????????????????????????????NOSQL?(SQL???????????????/??????????)???????????????????????????·????????????Oracle Coherence???????????????????/??????????????????????? ??Coherence??????????WebLogic Server????1???Coherence?????????·??????????????????"????????"????????????????WebLogic Server?????????????????????O/R?????·?????????Oracle TopLink?????????TopLink?Java???????????????Java Persistence API(JPA)????????JPA?????????????????????TopLink?????Oracle Database????????Coherence?????????????????????? ???WebLogic Server????????????????????Coherence?????????????????????·???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ???????WebLogic Server??Coherence?HTTP????????????????????WebLogic Server????????????????????Coherence????????????????????????????????????????????????????????????WebLogic Server???????????????Coherence?????????????????????????????????????????Coherence???????????????EC??????????·??????????????????????Web?????????????????????????????????????????????????????????????????????·????????????????WebLogic Server?????????1?????Oracle Enterprise Manager????????????????????????????????????Oracle Enterprise Manager(EM)??????????????Oracle EM???????URL???????????????????????????????Oracle Database?????????????????????????????????????"?????"??????TCO??????????????/??????????????????IT?TCO(??????)???????WebLogic Server???????????? ???????????????????????????????????????????????????????????WebLogic Server??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ??????????Oracle EM??????Oracle Database??????????????????????????????????????????????????????????????????TCO????????????? ????Java EE??????????????????????????????????????????TCO????????????????WebLogic Server?????????????????????·????????????????WebLogic Server????????????????????????????IT???????????????????????????WebLogic Server????????????????????????? ???????????????????????????????????????????????JRockit????????????????Sun JVM???????????????WebLogic Server????????Sun JVM????????????????????????????????? ????????????????????????????????????????????????????????????????????? ?????????????????Oracle EM????????????????Oracle EM???????????????????????????????????????????????????????????????????????????????????????????????????????????????? ??????Java EE??????????????????????????????????????????????????WebLogic Server?????????????????????????????????????????????????????????????WebLogic Server????????????????????????????WebLogic Server???????????????????????????????????????????????????????????????? ??????TCO???????????????????Java EE???????????????????????????????????????????????"?????"?????????·???????? ??????WebLogic Server??????????????????????·??????·??????????????????????????????????WebLogic Server??????????????????????????????????????????????

  Read the article

• Exadata V2+Oracle EBS R12 RAC Best Practices

  - by longchun.zhu

  ?????,???,??????,???????,??2????,????,???????????????,????????????????,?????????, ??..???????????,?????????????... ??: - Exadata V2 - Oracle ES Linux 5.3 - Oracle EBS DB 11.1.07 - 11.2.0.1 Upgrade - Oracle EBS R12.1.1 + Chinese Package - Muti Application +PCP (on going)

  Read the article

• HealthSouth Upgrades to Oracle Database 11g Release 2 and Oracle RAC

  - by jenny.gelhausen

  HealthSouth Corporation, the nation's largest provider of inpatient rehabilitation services, has upgraded to Oracle Database 11g Release 2 underneath PeopleSoft Enterprise Human Capital Management. Additionally, HealthSouth improved the availability and performance of its Oracle PeopleSoft Enterprise applications and Enterprise Data Warehouse using Oracle Database 11g and Oracle Real Application Clusters. Oracle Database options -- Oracle Advanced Compression and Oracle Partitioning are key to HealthSouth's data lifecycle management practices and to utilizing storage systems more efficiently. Using compression on both partitioned as well as non-partitioned tables in its data warehouse, HealthSouth has seen a 4X storage reduction without any cost to performance. "Oracle Database 11g, along with Oracle Real Application Clusters, Advanced Compression and Partitioning, all lend themselves to delivering highly available, performant data warehousing," said Henry Lovoy, Data Manager, HealthSouth Corporation. Press Release var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); try { var pageTracker = _gat._getTracker("UA-13185312-1"); pageTracker._trackPageview(); } catch(err) {}

  Read the article

• ORACLE MASTER Expert ??????!11g R2 RAC ????????

  - by M.Morozumi

  Oracle Certified Expert, Oracle Real Application Clusters 11g and Grid Infrastructure Administrator ??????????(2012/4/12~) -------------------------------------------------------------------- 2012?4?12???OPN ?????????(OPN Certified Specialist)??????Oracle Database 11g???????????? ?????????Oracle Certified Expert, Oracle Real Application Clusters 11g and Grid Infrastructure Administrator ???????????????????? ?Oracle Certified Expert, Oracle Real Application Clusters 11g and Grid Infrastructure Administrator?????????????5? ????Oracle University? Web???????????????????????? -------------------------------------------------------------------- ???????????! Oracle Certified Expert, Oracle Real Application Clusters 11g and Grid Infrastructure Administrator ????????????????????! ?????????????????

  Read the article

• ???????!11g R2 RAC ????

  - by M.Morozumi

  ??(2012?4?12?)OPN ?????????(OPN Certified Specialist)??????Oracle Database 11g???????????????? ???:Oracle Certified Expert, Oracle Real Application Clusters 11g and Grid Infrastructure Administrator ???:Oracle Real Application Clusters 11g Release 2 and Grid Infrastructure Administration (1Z0-058) ?????????Oracle Certified Expert, Oracle Real Application Clusters 11g and Grid Infrastructure Administrator ???????????????????? ?Oracle Certified Expert, Oracle Real Application Clusters 11g and Grid Infrastructure Administrator?????????????5? ????Oracle University? Web???????????????????????? ????????????????????! ?????????????????

  Read the article

• ????: PostgreSQL??Oracle RAC????

  - by Kumiko Fujita

  ?????????????????????????????????????????????????????????????????????????? ????????????????????????? * * * ?????????????????????????????????????DBMS??????????????????????????????DBMS????????????????????????????????????????????? 1. ???? ?????????????????????????????????????????????????????????????????????1?????? ???????????????? ?????????????????????????????DB???????OSS?PostgreSQL?????AP?????DB??????????????????? ???????? ?????10?????????????GB????????????????????????????DB?????????????????????????? ?????????????3,500?????????24????????????????????????????????????? ??AP?????????????????????????????????????????DB??PostgreSQL??????????????????PostgreSQL ????????????????????Vacuum????????????????????????????????????????????????????? ??????????????????PostgreSQL?OSS??????????????????????????????????????????????????DB MS??????Oracle Database 11gR2???????????????????????500GB???????????????????????????Partitioning ???????? Oracle Database Enterprise Edition?????????????????????????????????????????????? ????SAN?????Active/Standby???HA????????????????? 2. ????? 2.1. ???? PostgreSQL??????Oracle??????????????????????????????????????????????????????TEXT????? ????????????????????Oracle??????????????????????????PostgreSQL??csv???????Oracle Database?SQL*Loa der????????????? ??????????????????????????????DB??????????????Windows?Liunx??????????????????????? ????????????????????????????????????????????????? ?????????????PostgreSQL?NULL?????''????????????Oracle Database???????????????????????? ?????????? table { border-collapse: collapse; } th { border: solid 1px #666666; color: #000000; background-color: #ff9999; } td { border: solid 1px #666666; color: #000000; background-color: #ffffff; } ???? PostgreSQL Oracle Database ??? CHAR(n) CHAR(n),CLOB VARCHAR(n) VARCHAR2(n),CLOB TEXT CLOB ??? NUMERIC NUMBER INTEGER NUMBER SMALLINT NUMBER BIGINT NUMBER REAL NUMBER DOUBLE PRECISION NUMBER ??? DATE DATE TIMESTAMP TIMESTAMP ????? Bytea BLOB LOB BFILE/SecureFiles ??? OID ROWID 2.2. ????? ?????????????PostgreSQL?Oracle Database??????????SQL???????????????????????????????????Postg reSQL?LIMIT?OFFSET??Oracle Database?????????????????????? LIMIT,OFFSET???SELECT?????? /* PostgreSQL LIMIT,OFFSET */ SELECT ??? FROM ????? ORDER BY ???? LIMIT 2 OFFSET 5; /* Oracle Database????? */ SELECT ??? FROM (SELECT ???, ROWNUM line_no FROM (SELECT??? FROM ????? OREDR BY ???? ) ) WHERE line_no BETWEEN 6 AND 7; ??????????????????????????????????????????????????????????????????????????? ?????????????????? ????????????????????????????????????????????????Oracle Database??????????????????????Oracle Database????WHERE??????????????????????????????????????????????????????WHERE?????????????????????? 3. ???? ???????????????????????30%~40%????????????????????80%????????????????????? ?ITpro???:???????4????? ??????????????????????????????????? ·?????·??????????????????????????? ·????????????????????????????? ????????????????????????????????????????? 3.1. ??????? ????????????????????????????????????????·??????????????????????????????????? ???????????????????????????????????????????????????????·?????????????????? ???????????????????????????? (1)???????????????????? (2)???????????????????????????????????????????? (3)??????????????? (4)???????????????????????????????? ???????????·???????????????????????????????????????????????????????????????? ????????????????????? ????????·?? table { border-collapse: collapse; } th { border: solid 1px #666666; color: #000000; background-color: #ff9999; } td { border: solid 1px #666666; color: #000000; background-color: #ffffff; } ?? ?? ?? (1) ?????????? ????????????·???????????????????????? (2) ???????????????????? ?????????????????????????????? (3) ?????4????????????????? ???????????????????????DB????????? (4) ??????????(3)???????? ???????????????????????? ?????????????????????GB???????????????????????????????????????????(3)?????????? ??????? ??????????????????????????????????????????????csv??????????SQL*Loader?Oracle Database?????????????????????Oracle Database???????????????????????????INSERT????????????? ???????????????????????????????????????????????????????????????????????????? ?????????????????????? 3.2. ????? ???????????????????????????????????????????????????????????????????????????? ????????????????????????????????????????????????????????????????????????????????????????????? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 3.3. ????? ??????????????????????????????????????????????????????????????????????????? ??????????????????????? DBMS????????????????????????SQL??????????????????????????????????????????????????PostgreSQL?Oracle Database???????????MVCC?????????????????????????Read Committed??????????????????????????????????????????????????????????????????????????????????? ????????????????DBMS?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 4. ??? PostgreSQL??Oracle Database?????????????????????????????? ????????????·????????????????????????????????????? ??????4???????????????????????·??????????????????? ???????????????????????????????????????????????? ?????????????????????????????????????????????DBMS???????????????????DBMS???????? ?????SQL?????????????????????????????DB???????????????????????????? ???????????????????????????DBMS?????????????????????????????????????????????????????? ??????????????????????????????

  Read the article

• ????????2013?11??: ??RACCheck?????RAC??

  - by Allen Gao

  ???1?????????,????RACCheck????????????,????RACCheck ???????????,??RACCheck ????“11.2.0.3 Upgrade Readiness Feature”? ??,???????????,???????RACCheck ???????????????? :+ ?????????+ RACCheck ??,??,??????????+ RACcheck ??? 11.2.0.3 Upgrade Readiness+ RACcheck ??WebEx??????(???)?????:2013?11?13?15:00(????) ????: 592 310 106????????  1. ??????:https://oracleaw.webex.com/oracleaw/onstage/g.php?d=592310106&t=a2. ????????????????,???????????,?????????InterCall????????Webex???????,???????????,??????:   - ????ID: 25261419   - ????????: 108001201870   - ?????????: 108007121869    - ????: 8009 661 55   - ????: 00801044259   - ????????????????MOS?? 1148600.1 ???????:????????????,??????????(25261419)??????(First Name and Last Name) ????????????????? MOS ?? 1456176.1??????????????????????,?????????????????????????????MOS ?? 1456176.1“Archived 2013" (???)

  Read the article

< Previous Page | 1 2 3 4 5 6 7 8 9 10 11 12  | Next Page >