It's been a long time since last I added something here, but having some conversations this last week, I got inspired to update things.

I've been spending a lot of time with Ops Center for managing and installing systems these days.  So, I suspect a number of my upcoming posts will be in that area.

Today, I want to look at how to provision Solaris using Ops Center when your network is not connected to one of the built-in NICs.  We'll talk about how this can work for both Solaris 10 and Solaris 11, since they are pretty similar.  In both cases, WANboot is a key piece of the story.

Here's what I want to do:  I have a Sun Fire T2000 server with a Quad-GbE nxge card installed.  The only network is connected to port 2 on that card rather than the built-in network interfaces.  I want to install Solaris on it across the network, either Solaris 10 or Solaris 11.  I have met with a lot of customers lately who have a similar architecture.  Usually, they have T4-4 servers with the network connected via 10GbE connections.

Add to this mix the fact that I use Ops Center to manage the systems in my lab, so I really would like to add this to Ops Center.  If possible, I would like this to be completely hands free.  I can't quite do that yet. Close, but not quite.

WANBoot or Old-Style NetBoot?

When a system is installed from the network, it needs some help getting the process rolling.  It has to figure out what its network configuration (IP address, gateway, etc.) ought to be.  It needs to figure out what server is going to help it boot and install, and it needs the instructions for the installation.  There are two different ways to bootstrap an installation of Solaris on SPARC across the network.   The old way uses a broadcast of RARP or more recently DHCP to obtain the IP configuration and the rest of the information needed.  The second is to explicitly configure this information in the OBP and use WANBoot for installation

WANBoot has a number of benefits over broadcast-based installation: it is not restricted to a single subnet; it does not require special DHCP configuration or DHCP helpers; it uses standard HTTP and HTTPS protocols which traverse firewalls much more easily than NFS-based package installation.  But, WANBoot is not available on really old hardware and WANBoot requires the use o Flash Archives in Solaris 10.  Still, for many people, this is a great approach.

As it turns out, WANBoot is necessary if you plan to install using a NIC on a card rather than a built-in NIC.

Identifying Which Network Interface to Use

One of the trickiest aspects to this process, and the one that actually requires manual intervention to set up, is identifying how the OBP and Solaris refer to the NIC that we want to use to boot.  The OBP already has device aliases configured for the built-in NICs called net, net0, net1, net2, net3.  The device alias net typically points to net0 so that when you issue the command  "boot net -v install", it uses net0 for the boot.  Our task is to figure out the network instance for the NIC we want to use. 

We will need to get to the OBP console of the system we want to install in order to figure out what the network should be called.  I will presume you know how to get to the ok prompt.  Once there, we have to see what networks the OBP sees and identify which one is associated with our NIC using the OBP command show-nets.

SunOS Release 5.11 Version 11.0 64-bit
Copyright (c) 1983, 2011, Oracle and/or its affiliates. All rights reserved.

{4} ok banner

Sun Fire T200, No Keyboard
Copyright (c) 1998, 2010, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.30.4.b, 32640 MB memory available, Serial #69057548.
Ethernet address 0:14:4f:1d:bc:c, Host ID: 841dbc0c.

{4} ok show-nets
a) /pci@7c0/pci@0/pci@2/network@0,1
b) /pci@7c0/pci@0/pci@2/network@0
c) /pci@780/pci@0/pci@8/network@0,3
d) /pci@780/pci@0/pci@8/network@0,2
e) /pci@780/pci@0/pci@8/network@0,1
f) /pci@780/pci@0/pci@8/network@0
g) /pci@780/pci@0/pci@1/network@0,1
h) /pci@780/pci@0/pci@1/network@0
q) NO SELECTION 
Enter Selection, q to quit: d
/pci@780/pci@0/pci@8/network@0,2 has been selected.
Type ^Y ( Control-Y ) to insert it in the command line. 
e.g. ok nvalias mydev ^Y 
         for creating devalias mydev for /pci@780/pci@0/pci@8/network@0,2
{4} ok devalias
...
net3                     /pci@7c0/pci@0/pci@2/network@0,1
net2                     /pci@7c0/pci@0/pci@2/network@0
net1                     /pci@780/pci@0/pci@1/network@0,1
net0                     /pci@780/pci@0/pci@1/network@0
net                      /pci@780/pci@0/pci@1/network@0
...
name                     aliases

By looking at the devalias and the show-nets output, we can see that our Quad-GbE card must be the device nodes starting with  /pci@780/pci@0/pci@8/network@0.  The cable for our network is plugged into the 3rd slot, so the device address for our network must be /pci@780/pci@0/pci@8/network@0,2.

With that, we can create a device alias for our network interface.  Naming the device alias may take a little bit of trial and error, especially in Solaris 11 where the device alias seems to matter more with the new virtualized network stack. So far in my testing, since this is the "next" network interface to be used, I have found success in naming it net4, even though it's a NIC in the middle of a card that might, by rights, be called net6 (assuming the 0th interface on the card is the next interface identified by Solaris and this is the 3rd interface on the card).  So, we will call it net4.  We need to assign a device alias to it:

{4} ok nvalias net4 /pci@780/pci@0/pci@8/network@0,2
{4} ok devalias
net4                     /pci@780/pci@0/pci@8/network@0,2
...

We also may need to have the MAC for this particular interface, so let's get it, too.  To do this, we go to the device and interrogate its properties.

{4} ok cd /pci@780/pci@0/pci@8/network@0,2
{4} ok .properties
assigned-addresses       82060210 00000000 03000000 00000000 01000000 
                         82060218 00000000 00320000 00000000 00008000 
                         82060220 00000000 00328000 00000000 00008000 
                         82060230 00000000 00600000 00000000 00100000 
local-mac-address        00 21 28 20 42 92 
phy-type                 mif
...

From this, we can see that the MAC for this interface is  00:21:28:20:42:92.  We will need this later.

This is all we need to do at the OBP.  Now, we can configure Ops Center to use this interface.

Network Boot in Solaris 10

Solaris 10 turns out to be a little simpler than Solaris 11 for this sort of a network boot.  Since WANBoot in Solaris 10 fetches a specified

In order to install the system using Ops Center, it is necessary to create a OS Provisioning profile and its corresponding plan.  I am going to presume that you already know how to do this within Ops Center 12c and I will just cover the differences between a regular profile and a profile that can use an alternate interface.

Create a OS Provisioning profile for Solaris 10 as usual.  However, when you specify the network resources for the primary network, click on the name of the NIC, probably GB_0, and rename it to GB_N/netN, where N is the instance number you used previously in creating the device alias.  This is where the trial and error may come into play.  You may need to try a few instance numbers before you, the OBP, and Solaris all agree on the instance number.  Mark this as the boot network.

For Solaris 10, you ought to be able to then apply the OS Provisioning profile to the server and it should install using that interface.  And if you put your cards in the same slots and plug the networks into the same NICs, this profile is reusable across multiple servers.

Why This Works

If you watch the console as Solaris boots during the OSP process, Ops Center is going to look for the device alias netN.  Since WANBoot requires a device alias called just net, Ops Center uses the value of your netN device alias and assigns that device to the net alias.  That means that boot net will automatically use this device.  Very cool!  Here's a trace from the console as Ops Center provisions a server:

Sun Sun Fire T200, No Keyboard
Copyright (c) 1998, 2010, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.30.4.b, 32640 MB memory available, Serial #69057548.
Ethernet address 0:14:4f:1d:bc:c, Host ID: 841dbc0c.

auto-boot? =            false
{0} ok  
{0} ok printenv network-boot-arguments
network-boot-arguments =  host-ip=10.140.204.234,router-ip=10.140.204.1,subnet-mask=255.255.254.0,hostname=atl-sewr-52,client-id=0100144F1DBC0C,file=http://10.140.204.22:5555/cgi-bin/wanboot-cgi
{0} ok
{0} ok devalias net
net                      /pci@780/pci@0/pci@1/network@0
{0} ok devalias net4
net4                     /pci@780/pci@0/pci@8/network@0,2
{0} ok devalias net /pci@780/pci@0/pci@8/network@0,2
{0} ok setenv network-boot-arguments host-ip=10.140.204.234,router-ip=10.140.204.1,subnet-mask=255.255.254.0,hostname=atl-sewr-52,client-id=0100144F1DBC0C,file=http://10.140.204.22:8004/cgi-bin/wanboot-cgi
network-boot-arguments =  host-ip=10.140.204.234,router-ip=10.140.204.1,subnet-mask=255.255.254.0,hostname=atl-sewr-52,client-id=0100144F1DBC0C,file=http://10.140.204.22:8004/cgi-bin/wanboot-cgi
{0} ok
{0} ok boot net - install
Boot device: /pci@780/pci@0/pci@8/network@0,2  File and args: - install
/pci@780/pci@0/pci@8/network@0,2: 1000 Mbps link up
<time unavailable> wanboot info: WAN boot messages->console
<time unavailable> wanboot info: configuring /pci@780/pci@0/pci@8/network@0,2

See what happened?  Ops Center looked for the network device alias called net4 that we specified in the profile, took the value from it, and made it the net device alias for the boot.  Pretty cool!

WANBoot and Solaris 11

Solaris 11 requires an additional step since the Automated Installer in Solaris 11 uses the MAC address of the network to figure out which manifest to use for system installation.  In order to make sure this is available, we have to take an extra step to associate the MAC of the NIC on the card with the host.  So, in addition to creating the device alias like we did above, we also have to declare to Ops Center that the host has this new MAC.

Declaring the NIC

Start out by discovering the hardware as usual.  Once you have discovered it, take a look under the Connectivity tab to see what networks it has discovered.  In the case of this system, it shows the 4 built-in networks, but not the networks on the additional cards.  These are not directly visible to the system controller.

 In order to add the additional network interface to the hardware asset, it is necessary to Declare it.  We will declare that we have a server with this additional NIC, but we will also  specify the existing GB_0 network so that Ops Center can associate the right resources together.  The GB_0 acts as sort of a key to tie our new declaration to the old system already discovered.  Go to the Assets tab, select All Assets, and then in the Actions tab, select Add Asset.  Rather than going through a discovery this time, we will manually declare a new asset.

When we declare it, we will give the hostname, IP address, system model that match those that have already been discovered.  Then, we will declare both GB_0 with its existing MAC and the new GB_4 with its MAC.  Remember that we collected the MAC for GB_4 when we created its device alias.

After you declare the asset, you will see the new NIC in the connectivity tab for the asset.  You will notice that only the NICs you listed when you declared it are seen now.  If you want Ops Center to see all of the existing NICs as well as the additional one, declare them as well.  Add the other GB_1, GB_2, GB_3 links and their MACs just as you did GB_0 and GB_4.

 Installing the OS

 Once you have declared the asset, you can create an OS Provisioning profile for Solaris 11 in the same way that you did for Solaris 10.  The only difference from any other provisioning profile you might have created already is the network to use for installation.  Again, use GB_N/netN where N is the interface number you used for your device alias and in your declaration.

 And away you go.  When the system boots from the network, the automated installer (AI) is able to see which system manifest to use, based on the new MAC that was associated, and the system gets installed.

{0} ok
{0} ok printenv network-boot-arguments
network-boot-arguments =  host-ip=10.140.204.234,router-ip=10.140.204.1,subnet-mask=255.255.254.0,hostname=atl-sewr-52,client-id=01002128204292,file=http://10.140.204.22:5555/cgi-bin/wanboot-cgi
{0} ok
{0} ok devalias net
net                      /pci@780/pci@0/pci@1/network@0
{0} ok devalias net4
net4                     /pci@780/pci@0/pci@8/network@0,2
{0} ok devalias net /pci@780/pci@0/pci@8/network@0,2
{0} ok setenv network-boot-arguments host-ip=10.140.204.234,router-ip=10.140.204.1,subnet-mask=255.255.254.0,hostname=atl-sewr-52,client-id=01002128204292,file=http://10.140.204.22:5555/cgi-bin/wanboot-cgi
network-boot-arguments =  host-ip=10.140.204.234,router-ip=10.140.204.1,subnet-mask=255.255.254.0,hostname=atl-sewr-52,client-id=01002128204292,file=http://10.140.204.22:5555/cgi-bin/wanboot-cgi
{0} ok
{0} ok boot net - install
Boot device: /pci@780/pci@0/pci@8/network@0,2  File and args: - install
/pci@780/pci@0/pci@8/network@0,2: 1000 Mbps link up
<time unavailable> wanboot info: WAN boot messages->console
<time unavailable> wanboot info: configuring /pci@780/pci@0/pci@8/network@0,2
...
SunOS Release 5.11 Version 11.0 64-bit
Copyright (c) 1983, 2011, Oracle and/or its affiliates. All rights reserved.
Remounting root read/write
Probing for device nodes ...
Preparing network image for use
Downloading solaris.zlib
--2012-02-17 15:10:17--  http://10.140.204.22:5555/var/js/AI/sparc//solaris.zlib
Connecting to 10.140.204.22:5555... connected.
HTTP request sent, awaiting response... 200 OK
Length: 126752256 (121M) [text/plain]
Saving to: `/tmp/solaris.zlib'

100%[======================================>] 126,752,256 28.6M/s   in 4.4s   

2012-02-17 15:10:21 (27.3 MB/s) - `/tmp/solaris.zlib' saved [126752256/126752256]

Conclusion

So, why go to all of this trouble?  More and more, I find that customers are wiring their data center to only use higher speed networks - 10GbE only to the hosts.  Some customers are moving aggressively toward consolidated networks combining storage and network on CNA NICs.  All of this means that network-based provisioning cannot rely exclusively on the built-in network interfaces.  So, it's important to be able to provision a system using other than the built-in networks.  Turns out, that this is pretty straight-forward for both Solaris 10 and Solaris 11 and fits into the Ops Center deployment process quite nicely.

Hopefully, you will be able to use this as you build out your own private cloud solutions with Ops Center.