I wanted to use two Oracle SPARC T5-4 servers to simultaneously host both Oracle RAC and a WebLogic Server Cluster. I chose to use Oracle VM Server for SPARC to create a cluster like this:
There are plenty of trade offs and decisions that need to be made, for example:
Rather than configuring the system by hand, you might want to use an Oracle SuperCluster T5-8
My configuration is similar to jsavit's: Availability Best Practices - Example configuring a T5-8
but I chose to ignore some of the advice. Maybe I should have included an alternate service domain, but I decided that I already had enough
redundancy
Both Oracle SPARC T5-4 servers were to be configured like this:
Cntl
0.25 4 64GB
App LDom 2.75 CPU's 44 cores 704 GB
DB LDom One CPU 16 cores 256 GB
The systems started with everything in the primary domain: # ldm list
NAME STATE FLAGS CONS VCPU MEMORY UTIL NORM UPTIME
primary active -n-c-- UART 512 1023G 0.0% 0.0% 11m
# ldm list-spconfig
factory-default [current]
primary
# ldm list -o core,memory,physio
NAME
primary
CORE
CID CPUSET
0 (0, 1, 2, 3, 4, 5, 6, 7)
1 (8, 9, 10, 11, 12, 13, 14, 15)
2 (16, 17, 18, 19, 20, 21, 22, 23)
-- SNIP
62 (496, 497, 498, 499, 500, 501, 502, 503)
63 (504, 505, 506, 507, 508, 509, 510, 511)
MEMORY
RA PA SIZE
0x30000000 0x30000000 255G
0x80000000000 0x80000000000 256G
0x100000000000 0x100000000000 256G
0x180000000000 0x180000000000 256G # Give this memory block to the DB LDom
IO
DEVICE PSEUDONYM OPTIONS
pci@300 pci_0
pci@340 pci_1
pci@380 pci_2
pci@3c0 pci_3
pci@400 pci_4
pci@440 pci_5
pci@480 pci_6
pci@4c0 pci_7
pci@300/pci@1/pci@0/pci@6 /SYS/RCSA/PCIE1
pci@300/pci@1/pci@0/pci@c /SYS/RCSA/PCIE2
pci@300/pci@1/pci@0/pci@4/pci@0/pci@c /SYS/MB/SASHBA0
pci@300/pci@1/pci@0/pci@4/pci@0/pci@8 /SYS/RIO/NET0
pci@340/pci@1/pci@0/pci@6 /SYS/RCSA/PCIE3
pci@340/pci@1/pci@0/pci@c /SYS/RCSA/PCIE4
pci@380/pci@1/pci@0/pci@a /SYS/RCSA/PCIE9
pci@380/pci@1/pci@0/pci@4 /SYS/RCSA/PCIE10
pci@3c0/pci@1/pci@0/pci@e /SYS/RCSA/PCIE11
pci@3c0/pci@1/pci@0/pci@8 /SYS/RCSA/PCIE12
pci@400/pci@1/pci@0/pci@e /SYS/RCSA/PCIE5
pci@400/pci@1/pci@0/pci@8 /SYS/RCSA/PCIE6
pci@440/pci@1/pci@0/pci@e /SYS/RCSA/PCIE7
pci@440/pci@1/pci@0/pci@8 /SYS/RCSA/PCIE8
pci@480/pci@1/pci@0/pci@a /SYS/RCSA/PCIE13
pci@480/pci@1/pci@0/pci@4 /SYS/RCSA/PCIE14
pci@4c0/pci@1/pci@0/pci@8 /SYS/RCSA/PCIE15
pci@4c0/pci@1/pci@0/pci@4 /SYS/RCSA/PCIE16
pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c /SYS/MB/SASHBA1
pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4 /SYS/RIO/NET2
Added an additional service processor configuration: # ldm add-spconfig split # ldm list-spconfig factory-default primary split [current]
And removed many of the resources from the primary domain: # ldm start-reconf primary
# ldm set-core 4 primary
# ldm set-memory 32G primary
# ldm rm-io pci@340 primary
# ldm rm-io pci@380 primary
# ldm rm-io pci@3c0 primary
# ldm rm-io pci@400 primary
# ldm rm-io pci@440 primary
# ldm rm-io pci@480 primary
# ldm rm-io pci@4c0 primary
# init 6
Needed to add resources to the guest domains: # ldm add-domain db
# ldm set-core cid=`seq -s"," 48 63` db # ldm add-memory mblock=0x180000000000:256G db
# ldm add-io pci@480 db
# ldm add-io pci@4c0 db # ldm add-domain app
# ldm set-core 44 app
# ldm set-memory 704G app
# ldm add-io pci@340 app
# ldm add-io pci@380 app
# ldm add-io pci@3c0 app
# ldm add-io pci@400 app
# ldm add-io pci@440 app
Needed to set up services: # ldm add-vds primary-vds0 primary # ldm add-vcc port-range=5000-5100 primary-vcc0 primary
Needed to add a virtual network port for the WebLogic application domain: # ipadm NAME CLASS/TYPE STATE UNDER ADDR lo0 loopback ok -- -- lo0/v4 static ok -- ... lo0/v6 static ok -- ... net0 ip ok -- ... net0/v4 static ok -- xxx.xxx.xxx.xxx/24 net0/v6 addrconf ok -- .... net0/v6 addrconf ok -- ... net8 ip ok -- -- net8/v4 static ok -- ... # dladm show-phys LINK MEDIA STATE SPEED DUPLEX DEVICE net1 Ethernet unknown 0 unknown ixgbe1 net0 Ethernet up 1000 full ixgbe0 net8 Ethernet up 10 full usbecm2 # ldm add-vsw net-dev=net0 primary-vsw0 primary # ldm add-vnet vnet1 primary-vsw0 app
Needed to add a virtual disk to the WebLogic application domain: # format Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c0t5000CCA02505F874d0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB> /scsi_vhci/disk@g5000cca02505f874 /dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD0/disk 1. c0t5000CCA02506C468d0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB> /scsi_vhci/disk@g5000cca02506c468 /dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD1/disk 2. c0t5000CCA025067E5Cd0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB> /scsi_vhci/disk@g5000cca025067e5c /dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD2/disk 3. c0t5000CCA02506C258d0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB> /scsi_vhci/disk@g5000cca02506c258 /dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD3/disk Specify disk (enter its number): ^C # ldm add-vdsdev /dev/dsk/c0t5000CCA02506C468d0s2 HDD1@primary-vds0 # ldm add-vdisk HDD1 HDD1@primary-vds0 app
Add some additional spice to the pot: # ldm set-variable auto-boot\\?=false db # ldm set-variable auto-boot\\?=false app # ldm set-var boot-device=HDD1 app
Bind the logical domains: # ldm bind db # ldm bind app
At the end of the process, the system is set up like this: # ldm list -o core,memory,physio NAME primary CORE CID CPUSET 0 (0, 1, 2, 3, 4, 5, 6, 7) 1 (8, 9, 10, 11, 12, 13, 14, 15) 2 (16, 17, 18, 19, 20, 21, 22, 23) 3 (24, 25, 26, 27, 28, 29, 30, 31) MEMORY RA PA SIZE 0x30000000 0x30000000 32G IO DEVICE PSEUDONYM OPTIONS pci@300 pci_0 pci@300/pci@1/pci@0/pci@6 /SYS/RCSA/PCIE1 pci@300/pci@1/pci@0/pci@c /SYS/RCSA/PCIE2 pci@300/pci@1/pci@0/pci@4/pci@0/pci@c /SYS/MB/SASHBA0 pci@300/pci@1/pci@0/pci@4/pci@0/pci@8 /SYS/RIO/NET0 ------------------------------------------------------------------------------ NAME app CORE CID CPUSET 4 (32, 33, 34, 35, 36, 37, 38, 39) 5 (40, 41, 42, 43, 44, 45, 46, 47) 6 (48, 49, 50, 51, 52, 53, 54, 55) 7 (56, 57, 58, 59, 60, 61, 62, 63) 8 (64, 65, 66, 67, 68, 69, 70, 71) 9 (72, 73, 74, 75, 76, 77, 78, 79) 10 (80, 81, 82, 83, 84, 85, 86, 87) 11 (88, 89, 90, 91, 92, 93, 94, 95) 12 (96, 97, 98, 99, 100, 101, 102, 103) 13 (104, 105, 106, 107, 108, 109, 110, 111) 14 (112, 113, 114, 115, 116, 117, 118, 119) 15 (120, 121, 122, 123, 124, 125, 126, 127) 16 (128, 129, 130, 131, 132, 133, 134, 135) 17 (136, 137, 138, 139, 140, 141, 142, 143) 18 (144, 145, 146, 147, 148, 149, 150, 151) 19 (152, 153, 154, 155, 156, 157, 158, 159) 20 (160, 161, 162, 163, 164, 165, 166, 167) 21 (168, 169, 170, 171, 172, 173, 174, 175) 22 (176, 177, 178, 179, 180, 181, 182, 183) 23 (184, 185, 186, 187, 188, 189, 190, 191) 24 (192, 193, 194, 195, 196, 197, 198, 199) 25 (200, 201, 202, 203, 204, 205, 206, 207) 26 (208, 209, 210, 211, 212, 213, 214, 215) 27 (216, 217, 218, 219, 220, 221, 222, 223) 28 (224, 225, 226, 227, 228, 229, 230, 231) 29 (232, 233, 234, 235, 236, 237, 238, 239) 30 (240, 241, 242, 243, 244, 245, 246, 247) 31 (248, 249, 250, 251, 252, 253, 254, 255) 32 (256, 257, 258, 259, 260, 261, 262, 263) 33 (264, 265, 266, 267, 268, 269, 270, 271) 34 (272, 273, 274, 275, 276, 277, 278, 279) 35 (280, 281, 282, 283, 284, 285, 286, 287) 36 (288, 289, 290, 291, 292, 293, 294, 295) 37 (296, 297, 298, 299, 300, 301, 302, 303) 38 (304, 305, 306, 307, 308, 309, 310, 311) 39 (312, 313, 314, 315, 316, 317, 318, 319) 40 (320, 321, 322, 323, 324, 325, 326, 327) 41 (328, 329, 330, 331, 332, 333, 334, 335) 42 (336, 337, 338, 339, 340, 341, 342, 343) 43 (344, 345, 346, 347, 348, 349, 350, 351) 44 (352, 353, 354, 355, 356, 357, 358, 359) 45 (360, 361, 362, 363, 364, 365, 366, 367) 46 (368, 369, 370, 371, 372, 373, 374, 375) 47 (376, 377, 378, 379, 380, 381, 382, 383) MEMORY RA PA SIZE 0x30000000 0x830000000 192G 0x4000000000 0x80000000000 256G 0x8080000000 0x100000000000 256G IO DEVICE PSEUDONYM OPTIONS pci@340 pci_1 pci@380 pci_2 pci@3c0 pci_3 pci@400 pci_4 pci@440 pci_5 pci@340/pci@1/pci@0/pci@6 /SYS/RCSA/PCIE3 pci@340/pci@1/pci@0/pci@c /SYS/RCSA/PCIE4 pci@380/pci@1/pci@0/pci@a /SYS/RCSA/PCIE9 pci@380/pci@1/pci@0/pci@4 /SYS/RCSA/PCIE10 pci@3c0/pci@1/pci@0/pci@e /SYS/RCSA/PCIE11 pci@3c0/pci@1/pci@0/pci@8 /SYS/RCSA/PCIE12 pci@400/pci@1/pci@0/pci@e /SYS/RCSA/PCIE5 pci@400/pci@1/pci@0/pci@8 /SYS/RCSA/PCIE6 pci@440/pci@1/pci@0/pci@e /SYS/RCSA/PCIE7 pci@440/pci@1/pci@0/pci@8 /SYS/RCSA/PCIE8 ------------------------------------------------------------------------------ NAME db CORE CID CPUSET 48 (384, 385, 386, 387, 388, 389, 390, 391) 49 (392, 393, 394, 395, 396, 397, 398, 399) 50 (400, 401, 402, 403, 404, 405, 406, 407) 51 (408, 409, 410, 411, 412, 413, 414, 415) 52 (416, 417, 418, 419, 420, 421, 422, 423) 53 (424, 425, 426, 427, 428, 429, 430, 431) 54 (432, 433, 434, 435, 436, 437, 438, 439) 55 (440, 441, 442, 443, 444, 445, 446, 447) 56 (448, 449, 450, 451, 452, 453, 454, 455) 57 (456, 457, 458, 459, 460, 461, 462, 463) 58 (464, 465, 466, 467, 468, 469, 470, 471) 59 (472, 473, 474, 475, 476, 477, 478, 479) 60 (480, 481, 482, 483, 484, 485, 486, 487) 61 (488, 489, 490, 491, 492, 493, 494, 495) 62 (496, 497, 498, 499, 500, 501, 502, 503) 63 (504, 505, 506, 507, 508, 509, 510, 511) MEMORY RA PA SIZE 0x80000000 0x180000000000 256G IO DEVICE PSEUDONYM OPTIONS pci@480 pci_6 pci@4c0 pci_7 pci@480/pci@1/pci@0/pci@a /SYS/RCSA/PCIE13 pci@480/pci@1/pci@0/pci@4 /SYS/RCSA/PCIE14 pci@4c0/pci@1/pci@0/pci@8 /SYS/RCSA/PCIE15 pci@4c0/pci@1/pci@0/pci@4 /SYS/RCSA/PCIE16 pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c /SYS/MB/SASHBA1 pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4 /SYS/RIO/NET2
Start the domains: # ldm start app LDom app started # ldm start db LDom db started
Make sure to start the vntsd service that was created, above. # svcs -a | grep ldo disabled 8:38:38 svc:/ldoms/vntsd:default online 8:38:58 svc:/ldoms/agents:default online 8:39:25 svc:/ldoms/ldmd:default # svcadm enable vntsd
Now use the MAC address to configure the Solaris 11 Automated Installation.
Database Logical Domain
# telnet localhost
5000
{0} ok devalias
screen
/pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@7/display@0
disk7
/pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0/disk@p3
disk6
/pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0/disk@p2
disk5
/pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0/disk@p1
disk4
/pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0/disk@p0
scsi1
/pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0
net3
/pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4/network@0,1
net2
/pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4/network@0
virtual-console /virtual-devices/console@1
name aliases
{0} ok boot net2
Boot device:
/pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4/network@0 File
and args:
1000 Mbps full duplex Link up Requesting Internet Address for xx:xx:xx:xx:xx:xx Requesting Internet Address for xx:xx:xx:xx:xx:xx
WLS Logical Domain
# telnet localhost
5001
{0} ok devalias hdd1
/virtual-devices@100/channel-devices@200/disk@0 vnet1
/virtual-devices@100/channel-devices@200/network@0 net
/virtual-devices@100/channel-devices@200/network@0 disk
/virtual-devices@100/channel-devices@200/disk@0 virtual-console
/virtual-devices/console@1 name aliases
{0} ok boot net Boot device:
/virtual-devices@100/channel-devices@200/network@0 File
and args: Requesting Internet Address for xx:xx:xx:xx:xx:xx Requesting Internet Address for xx:xx:xx:xx:xx:xx
Repeat the process for the second SPARC T5-4, install Solaris, RAC
and WebLogic Cluster, and you are ready to go. Maybe buying a
SuperCluster would have been easier.
