SPARC T5-4 LDoms for RAC and WebLogic Clusters
Posted
by Jeff Taylor-Oracle
on Oracle Blogs
See other posts from Oracle Blogs
or by Jeff Taylor-Oracle
Published on Mon, 18 Aug 2014 17:33:50 +0000
Indexed on
2014/08/18
22:27 UTC
Read the original article
Hit count: 404
/Sun
I wanted to use two Oracle SPARC T5-4 servers to simultaneously host both Oracle RAC and a WebLogic Server Cluster. I chose to use Oracle VM Server for SPARC to create a cluster like this:
There are plenty of trade offs and decisions that need to be made, for example:
- Rather than configuring the system by hand, you might want to use an Oracle SuperCluster T5-8
- My configuration is similar to jsavit's: Availability Best Practices - Example configuring a T5-8 but I chose to ignore some of the advice. Maybe I should have included an alternate service domain, but I decided that I already had enough redundancy
Both Oracle SPARC T5-4 servers were to be configured like this:
Cntl 0.25 4 64GB |
App LDom 2.75 CPU's 44 cores 704 GB |
DB LDom One CPU 16 cores 256 GB |
The systems started with everything in the primary domain:
# ldm list
NAME STATE FLAGS CONS VCPU MEMORY UTIL NORM UPTIME
primary active -n-c-- UART 512 1023G 0.0% 0.0% 11m
# ldm list-spconfig
factory-default [current]
primary
# ldm list -o core,memory,physio
NAME
primary
CORE
CID CPUSET
0 (0, 1, 2, 3, 4, 5, 6, 7)
1 (8, 9, 10, 11, 12, 13, 14, 15)
2 (16, 17, 18, 19, 20, 21, 22, 23)
-- SNIP
62 (496, 497, 498, 499, 500, 501, 502, 503)
63 (504, 505, 506, 507, 508, 509, 510, 511)
MEMORY
RA PA SIZE
0x30000000 0x30000000 255G
0x80000000000 0x80000000000 256G
0x100000000000 0x100000000000 256G
0x180000000000 0x180000000000 256G # Give this memory block to the DB LDom
IO
DEVICE PSEUDONYM OPTIONS
pci@300 pci_0
pci@340 pci_1
pci@380 pci_2
pci@3c0 pci_3
pci@400 pci_4
pci@440 pci_5
pci@480 pci_6
pci@4c0 pci_7
pci@300/pci@1/pci@0/pci@6 /SYS/RCSA/PCIE1
pci@300/pci@1/pci@0/pci@c /SYS/RCSA/PCIE2
pci@300/pci@1/pci@0/pci@4/pci@0/pci@c /SYS/MB/SASHBA0
pci@300/pci@1/pci@0/pci@4/pci@0/pci@8 /SYS/RIO/NET0
pci@340/pci@1/pci@0/pci@6 /SYS/RCSA/PCIE3
pci@340/pci@1/pci@0/pci@c /SYS/RCSA/PCIE4
pci@380/pci@1/pci@0/pci@a /SYS/RCSA/PCIE9
pci@380/pci@1/pci@0/pci@4 /SYS/RCSA/PCIE10
pci@3c0/pci@1/pci@0/pci@e /SYS/RCSA/PCIE11
pci@3c0/pci@1/pci@0/pci@8 /SYS/RCSA/PCIE12
pci@400/pci@1/pci@0/pci@e /SYS/RCSA/PCIE5
pci@400/pci@1/pci@0/pci@8 /SYS/RCSA/PCIE6
pci@440/pci@1/pci@0/pci@e /SYS/RCSA/PCIE7
pci@440/pci@1/pci@0/pci@8 /SYS/RCSA/PCIE8
pci@480/pci@1/pci@0/pci@a /SYS/RCSA/PCIE13
pci@480/pci@1/pci@0/pci@4 /SYS/RCSA/PCIE14
pci@4c0/pci@1/pci@0/pci@8 /SYS/RCSA/PCIE15
pci@4c0/pci@1/pci@0/pci@4 /SYS/RCSA/PCIE16
pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c /SYS/MB/SASHBA1
pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4 /SYS/RIO/NET2
Added an additional service processor configuration:
# ldm add-spconfig split
# ldm list-spconfig
factory-default
primary
split [current]
And removed many of the resources from the primary domain:
# ldm start-reconf primary
# ldm set-core 4 primary
# ldm set-memory 32G primary
# ldm rm-io pci@340 primary
# ldm rm-io pci@380 primary
# ldm rm-io pci@3c0 primary
# ldm rm-io pci@400 primary
# ldm rm-io pci@440 primary
# ldm rm-io pci@480 primary
# ldm rm-io pci@4c0 primary
# init 6
Needed to add resources to the guest domains:
# ldm add-domain db
# ldm set-core cid=`seq -s"," 48 63` db
# ldm add-memory mblock=0x180000000000:256G db
# ldm add-io pci@480 db
# ldm add-io pci@4c0 db
# ldm add-domain app
# ldm set-core 44 app
# ldm set-memory 704G app
# ldm add-io pci@340 app
# ldm add-io pci@380 app
# ldm add-io pci@3c0 app
# ldm add-io pci@400 app
# ldm add-io pci@440 app
Needed to set up services:
# ldm add-vds primary-vds0 primary
# ldm add-vcc port-range=5000-5100 primary-vcc0 primary
Needed to add a virtual network port for the WebLogic application domain:
# ipadm
NAME CLASS/TYPE STATE UNDER ADDR
lo0 loopback ok -- --
lo0/v4 static ok -- ...
lo0/v6 static ok -- ...
net0 ip ok -- ...
net0/v4 static ok -- xxx.xxx.xxx.xxx/24
net0/v6 addrconf ok -- ....
net0/v6 addrconf ok -- ...
net8 ip ok -- --
net8/v4 static ok -- ...
# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
net1 Ethernet unknown 0 unknown ixgbe1
net0 Ethernet up 1000 full ixgbe0
net8 Ethernet up 10 full usbecm2
# ldm add-vsw net-dev=net0 primary-vsw0 primary
# ldm add-vnet vnet1 primary-vsw0 app
Needed to add a virtual disk to the WebLogic application domain:
# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0t5000CCA02505F874d0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB>
/scsi_vhci/disk@g5000cca02505f874
/dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD0/disk
1. c0t5000CCA02506C468d0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB>
/scsi_vhci/disk@g5000cca02506c468
/dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD1/disk
2. c0t5000CCA025067E5Cd0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB>
/scsi_vhci/disk@g5000cca025067e5c
/dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD2/disk
3. c0t5000CCA02506C258d0 <HITACHI-H106060SDSUN600G-A2B0-558.91GB>
/scsi_vhci/disk@g5000cca02506c258
/dev/chassis/SPARC_T5-4.AK00084038/SYS/SASBP0/HDD3/disk
Specify disk (enter its number): ^C
# ldm add-vdsdev /dev/dsk/c0t5000CCA02506C468d0s2 HDD1@primary-vds0
# ldm add-vdisk HDD1 HDD1@primary-vds0 app
Add some additional spice to the pot:
# ldm set-variable auto-boot\\?=false db
# ldm set-variable auto-boot\\?=false app
# ldm set-var boot-device=HDD1 app
Bind the logical domains:
# ldm bind db
# ldm bind app
At the end of the process, the system is set up like this:
# ldm list -o core,memory,physio
NAME
primary
CORE
CID CPUSET
0 (0, 1, 2, 3, 4, 5, 6, 7)
1 (8, 9, 10, 11, 12, 13, 14, 15)
2 (16, 17, 18, 19, 20, 21, 22, 23)
3 (24, 25, 26, 27, 28, 29, 30, 31)
MEMORY
RA PA SIZE
0x30000000 0x30000000 32G
IO
DEVICE PSEUDONYM OPTIONS
pci@300 pci_0
pci@300/pci@1/pci@0/pci@6 /SYS/RCSA/PCIE1
pci@300/pci@1/pci@0/pci@c /SYS/RCSA/PCIE2
pci@300/pci@1/pci@0/pci@4/pci@0/pci@c /SYS/MB/SASHBA0
pci@300/pci@1/pci@0/pci@4/pci@0/pci@8 /SYS/RIO/NET0
------------------------------------------------------------------------------
NAME
app
CORE
CID CPUSET
4 (32, 33, 34, 35, 36, 37, 38, 39)
5 (40, 41, 42, 43, 44, 45, 46, 47)
6 (48, 49, 50, 51, 52, 53, 54, 55)
7 (56, 57, 58, 59, 60, 61, 62, 63)
8 (64, 65, 66, 67, 68, 69, 70, 71)
9 (72, 73, 74, 75, 76, 77, 78, 79)
10 (80, 81, 82, 83, 84, 85, 86, 87)
11 (88, 89, 90, 91, 92, 93, 94, 95)
12 (96, 97, 98, 99, 100, 101, 102, 103)
13 (104, 105, 106, 107, 108, 109, 110, 111)
14 (112, 113, 114, 115, 116, 117, 118, 119)
15 (120, 121, 122, 123, 124, 125, 126, 127)
16 (128, 129, 130, 131, 132, 133, 134, 135)
17 (136, 137, 138, 139, 140, 141, 142, 143)
18 (144, 145, 146, 147, 148, 149, 150, 151)
19 (152, 153, 154, 155, 156, 157, 158, 159)
20 (160, 161, 162, 163, 164, 165, 166, 167)
21 (168, 169, 170, 171, 172, 173, 174, 175)
22 (176, 177, 178, 179, 180, 181, 182, 183)
23 (184, 185, 186, 187, 188, 189, 190, 191)
24 (192, 193, 194, 195, 196, 197, 198, 199)
25 (200, 201, 202, 203, 204, 205, 206, 207)
26 (208, 209, 210, 211, 212, 213, 214, 215)
27 (216, 217, 218, 219, 220, 221, 222, 223)
28 (224, 225, 226, 227, 228, 229, 230, 231)
29 (232, 233, 234, 235, 236, 237, 238, 239)
30 (240, 241, 242, 243, 244, 245, 246, 247)
31 (248, 249, 250, 251, 252, 253, 254, 255)
32 (256, 257, 258, 259, 260, 261, 262, 263)
33 (264, 265, 266, 267, 268, 269, 270, 271)
34 (272, 273, 274, 275, 276, 277, 278, 279)
35 (280, 281, 282, 283, 284, 285, 286, 287)
36 (288, 289, 290, 291, 292, 293, 294, 295)
37 (296, 297, 298, 299, 300, 301, 302, 303)
38 (304, 305, 306, 307, 308, 309, 310, 311)
39 (312, 313, 314, 315, 316, 317, 318, 319)
40 (320, 321, 322, 323, 324, 325, 326, 327)
41 (328, 329, 330, 331, 332, 333, 334, 335)
42 (336, 337, 338, 339, 340, 341, 342, 343)
43 (344, 345, 346, 347, 348, 349, 350, 351)
44 (352, 353, 354, 355, 356, 357, 358, 359)
45 (360, 361, 362, 363, 364, 365, 366, 367)
46 (368, 369, 370, 371, 372, 373, 374, 375)
47 (376, 377, 378, 379, 380, 381, 382, 383)
MEMORY
RA PA SIZE
0x30000000 0x830000000 192G
0x4000000000 0x80000000000 256G
0x8080000000 0x100000000000 256G
IO
DEVICE PSEUDONYM OPTIONS
pci@340 pci_1
pci@380 pci_2
pci@3c0 pci_3
pci@400 pci_4
pci@440 pci_5
pci@340/pci@1/pci@0/pci@6 /SYS/RCSA/PCIE3
pci@340/pci@1/pci@0/pci@c /SYS/RCSA/PCIE4
pci@380/pci@1/pci@0/pci@a /SYS/RCSA/PCIE9
pci@380/pci@1/pci@0/pci@4 /SYS/RCSA/PCIE10
pci@3c0/pci@1/pci@0/pci@e /SYS/RCSA/PCIE11
pci@3c0/pci@1/pci@0/pci@8 /SYS/RCSA/PCIE12
pci@400/pci@1/pci@0/pci@e /SYS/RCSA/PCIE5
pci@400/pci@1/pci@0/pci@8 /SYS/RCSA/PCIE6
pci@440/pci@1/pci@0/pci@e /SYS/RCSA/PCIE7
pci@440/pci@1/pci@0/pci@8 /SYS/RCSA/PCIE8
------------------------------------------------------------------------------
NAME
db
CORE
CID CPUSET
48 (384, 385, 386, 387, 388, 389, 390, 391)
49 (392, 393, 394, 395, 396, 397, 398, 399)
50 (400, 401, 402, 403, 404, 405, 406, 407)
51 (408, 409, 410, 411, 412, 413, 414, 415)
52 (416, 417, 418, 419, 420, 421, 422, 423)
53 (424, 425, 426, 427, 428, 429, 430, 431)
54 (432, 433, 434, 435, 436, 437, 438, 439)
55 (440, 441, 442, 443, 444, 445, 446, 447)
56 (448, 449, 450, 451, 452, 453, 454, 455)
57 (456, 457, 458, 459, 460, 461, 462, 463)
58 (464, 465, 466, 467, 468, 469, 470, 471)
59 (472, 473, 474, 475, 476, 477, 478, 479)
60 (480, 481, 482, 483, 484, 485, 486, 487)
61 (488, 489, 490, 491, 492, 493, 494, 495)
62 (496, 497, 498, 499, 500, 501, 502, 503)
63 (504, 505, 506, 507, 508, 509, 510, 511)
MEMORY
RA PA SIZE
0x80000000 0x180000000000 256G
IO
DEVICE PSEUDONYM OPTIONS
pci@480 pci_6
pci@4c0 pci_7
pci@480/pci@1/pci@0/pci@a /SYS/RCSA/PCIE13
pci@480/pci@1/pci@0/pci@4 /SYS/RCSA/PCIE14
pci@4c0/pci@1/pci@0/pci@8 /SYS/RCSA/PCIE15
pci@4c0/pci@1/pci@0/pci@4 /SYS/RCSA/PCIE16
pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c /SYS/MB/SASHBA1
pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4 /SYS/RIO/NET2
Start the domains:
# ldm start app
LDom app started
# ldm start db
LDom db started
Make sure to start the vntsd service that was created, above.
# svcs -a | grep ldo
disabled 8:38:38 svc:/ldoms/vntsd:default
online 8:38:58 svc:/ldoms/agents:default
online 8:39:25 svc:/ldoms/ldmd:default
# svcadm enable vntsd
Now use the MAC address to configure the Solaris 11 Automated Installation.
Database Logical Domain |
# telnet localhost
5000 |
{0} ok devalias screen /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@7/display@0 disk7 /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0/disk@p3 disk6 /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0/disk@p2 disk5 /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0/disk@p1 disk4 /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0/disk@p0 scsi1 /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@c/scsi@0 net3 /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4/network@0,1 net2 /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4/network@0 virtual-console /virtual-devices/console@1 name aliases |
{0} ok boot net2 Boot device: /pci@4c0/pci@1/pci@0/pci@c/pci@0/pci@4/network@0 File and args: 1000 Mbps full duplex Link up Requesting Internet Address for xx:xx:xx:xx:xx:xx Requesting Internet Address for xx:xx:xx:xx:xx:xx |
WLS Logical Domain |
# telnet localhost 5001 |
{0} ok devalias hdd1 /virtual-devices@100/channel-devices@200/disk@0 vnet1 /virtual-devices@100/channel-devices@200/network@0 net /virtual-devices@100/channel-devices@200/network@0 disk /virtual-devices@100/channel-devices@200/disk@0 virtual-console /virtual-devices/console@1 name aliases |
{0} ok boot net Boot device: /virtual-devices@100/channel-devices@200/network@0 File and args: Requesting Internet Address for xx:xx:xx:xx:xx:xx Requesting Internet Address for xx:xx:xx:xx:xx:xx |
Repeat the process for the second SPARC T5-4, install Solaris, RAC and WebLogic Cluster, and you are ready to go. Maybe buying a SuperCluster would have been easier.
© Oracle Blogs or respective owner