There is increased adoption of Oracle Solaris 11, and many customers are deploying it on systems that previously ran Solaris 10. That includes older T1-processor based systems like T1000 and T2000. Even though they are old (from 2005) and don't have the performance of current SPARC servers, they are still functional, stable servers that customers continue to operate. One reason to install Solaris 11 on them is that older machines are attractive for testing OS upgrades before updating current, production systems.

Normally this does not present a challenge, because Solaris 11 runs on any T-series or M-series SPARC server. One scenario adds a complication: running Solaris 11 in a control domain on a T1000 or T2000 hosting logical domains.

Solaris 11 pre-installed Oracle VM Server for SPARC incompatible with T1

Unlike Solaris 10, Solaris 11 comes with Oracle VM Server for SPARC preinstalled. The ldomsmanager package contains the logical domains manager for Oracle VM Server for SPARC 2.2, which requires a SPARC T2, T2+, T3, or T4 server. It does not work with T1-processor systems, which are only supported by LDoms Manager 1.2 and earlier.

The following screenshot shows what happens (bold font) if you try to use Oracle VM Server for SPARC 2.x commands in a Solaris 11 control domain. The commands were issued in a control domain on a T2000 that previously ran Solaris 10. We also display the version of the logical domains manager installed in Solaris 11:

root@t2000 psrinfo -vp
The physical processor has 4 virtual processors (0-3)
  UltraSPARC-T1 (chipid 0, clock 1200 MHz)
# prtconf|grep T
SUNW,Sun-Fire-T200
# ldm -V
Failed to connect to logical domain manager: Connection refused
# pkg info ldomsmanager
          Name: system/ldoms/ldomsmanager
       Summary: Logical Domains Manager
   Description: LDoms Manager - Virtualization for SPARC T-Series
      Category: System/Virtualization
         State: Installed
     Publisher: solaris
       Version: 2.2.0.0
 Build Release: 5.11
        Branch: 0.175.0.8.0.3.0
Packaging Date: May 25, 2012 10:20:48 PM 
          Size: 2.86 MB
          FMRI: pkg://solaris/system/ldoms/[email protected],5.11-0.175.0.8.0.3.0:20120525T222048Z

The 2.2 version of the logical domains manager will have to be removed, and 1.2 installed, in order to use this as a control domain.

Preparing to change - create a new boot environment

Before doing anything else, lets create a new boot environment:

# beadm list
BE      Active Mountpoint Space Policy Created          
--      ------ ---------- ----- ------ -------          
solaris NR     /          2.14G static 2012-09-25 10:32 
# beadm create solaris-1 
# beadm activate solaris-1
# beadm list
BE        Active Mountpoint Space Policy Created          
--        ------ ---------- ----- ------ -------          
solaris   N      /          4.82M static 2012-09-25 10:32 
solaris-1 R      -          2.14G static 2012-09-29 11:40 
# init 0

Normally an init 6 to reboot would have been sufficient, but in the next step I reset the system anyway in order to put the system in factory default mode for a "clean" domain configuration.

Preparing to change - reset to factory default

There was a leftover domain configuration on the T2000, so I reset it to the factory install state. Since the ldm command is't working yet, it can't be done from the control domain, so I did it by logging onto to the service processor:

$ ssh -X admin@t2000-sc

Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
Oracle Advanced Lights Out Manager CMT v1.7.9
Please login: 
admin
Please Enter password: ********

sc> showhost
Sun-Fire-T2000 System Firmware 6.7.10  2010/07/14 16:35

Host flash versions:
   OBP 4.30.4.b 2010/07/09 13:48
   Hypervisor 1.7.3.c 2010/07/09 15:14
   POST 4.30.4.b 2010/07/09 14:24 
 
sc> bootmode config="factory-default"
sc> poweroff
Are you sure you want to power off the system [y/n]?  y
SC Alert: SC Request to Power Off Host.
SC Alert: Host system has shut down.
sc> poweron
SC Alert: Host System has Reset

At this point I rebooted into the new Solaris 11 boot environment, and Solaris commands showed it was running on the factory default configuration of a single domain owning all 32 CPUs and 32GB of RAM (that's what it looked like in 2005.)

# psrinfo -vp
The physical processor has 8 cores and 32 virtual processors (0-31)
  The core has 4 virtual processors (0-3)
  The core has 4 virtual processors (4-7)
  The core has 4 virtual processors (8-11)
  The core has 4 virtual processors (12-15)
  The core has 4 virtual processors (16-19)
  The core has 4 virtual processors (20-23)
  The core has 4 virtual processors (24-27)
  The core has 4 virtual processors (28-31)
    UltraSPARC-T1 (chipid 0, clock 1200 MHz)
# prtconf|grep Mem
Memory size: 32640 Megabytes

Note that the older processor has 4 virtual CPUs per core, while current processors have 8 per core.

Remove ldomsmanager 2.2 and install the 1.2 version

The Solaris 11 pkg command is now used to remove the 2.2 version that shipped with Solaris 11:

# pkg uninstall ldomsmanager
            Packages to remove:  1
       Create boot environment: No
Create backup boot environment: No
            Services to change:  2

PHASE                                        ACTIONS
Removal Phase                                130/130 

PHASE                                          ITEMS
Package State Update Phase                       1/1 
Package Cache Update Phase                       1/1
Image State Update Phase                         2/2

Finally, LDoms 1.2 installed via its install script, the same way it was done years ago:

# unzip LDoms-1_2-Integration-10.zip
# cd LDoms-1_2-Integration-10/Install/
# ./install-ldm 
Welcome to the LDoms installer.
You are about to install the Logical Domains Manager package that will
enable you to create, destroy and control other domains on your
system.  Given the capabilities of the LDoms domain manager, you can
now change the security configuration of this Solaris instance using
the Solaris Security Toolkit.
... 
... normal install messages omitted
...

The Solaris Security Toolkit applies to Solaris 10, and cannot be used in Solaris 11 (in which several things hardened by the Toolkit are already hardened by default), so answer b in the choice below:

You are about to install the Logical Domains Manager package that will
enable you to create, destroy and control other domains on your
system.  Given the capabilities of the LDoms domain manager, you can
now change the security configuration of this Solaris instance using
the Solaris Security Toolkit.

Select a security profile from this list:

a) Hardened Solaris configuration for LDoms (recommended)
b) Standard Solaris configuration
c) Your custom-defined Solaris security configuration profile

Enter a, b, or c [a]: b
... other install messages omitted for brevity...

After install I ensure that the necessary services are enabled, and verify the version of the installed LDoms Manager:

# svcs ldmd
STATE          STIME    FMRI
online         22:00:36 svc:/ldoms/ldmd:default
# svcs vntsd
STATE          STIME    FMRI
disabled       Aug_19   svc:/ldoms/vntsd:default
# ldm -V

Logical Domain Manager (v 1.2-debug)
        Hypervisor control protocol v 1.3
        Using Hypervisor MD v 1.1

System PROM:
        Hypervisor      v. 1.7.3.       @(#)Hypervisor 1.7.3.c 2010/07/09 15:14\015
        OpenBoot        v. 4.30.4.      @(#)OBP 4.30.4.b 2010/07/09 13:48

Set up control domain and domain services

At this point we have a functioning LDoms 1.2 environment that can be configured in the usual fashion. One difference is that LDoms 1.2 behavior had 'delayed configuration mode (as expected) during initial configuration before rebooting the control domain. Another minor difference with a Solaris 11 control domain is that you define virtual switches using the 'vanity name' of the network interface, rather than the hardware driver name as in Solaris 10.

# ldm list
------------------------------------------------------------------------------
Notice: the LDom Manager is running in configuration mode. Configuration and
resource information is displayed for the configuration under construction;
not the current active configuration. The configuration being constructed
will only take effect after it is downloaded to the system controller and
the host is reset.
------------------------------------------------------------------------------
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-c--  SP      32    32640M   3.2%  4d 2h 50m 
# ldm add-vdiskserver primary-vds0 primary
# ldm add-vconscon port-range=5000-5100 primary-vcc0 primary
# ldm add-vswitch net-dev=net0 primary-vsw0 primary
# ldm set-mau 2 primary
# ldm set-vcpu 8 primary
# ldm set-memory 4g primary
# ldm add-config initial
# ldm list-spconfig
factory-default
initial [current]

That's it, really. After reboot, we are ready to install guest domains.

Summary - new wine in old bottles

This example shows that (new) Solaris 11 can be installed on (old) T2000 servers and used as a control domain. The main activity is to remove the preinstalled Oracle VM Server for 2.2 and install Logical Domains 1.2 - the last version of LDoms to support T1-processor systems. I tested Solaris 10 and Solaris 11 guest domains running on this server and they worked without any surprises. This is a viable way to get further into Solaris 11 adoption, even on older T-series equipment.