Introduction:

This Oracle Enterprise Manager Ops Center blog entry provides tips for using Ops Center to update Solaris using Live Upgrade on Solaris 10 and Boot Environments on Solaris 11.

Why use Live Upgrade?

• Live Upgrade (LU) can significantly reduce downtime associated with patching
• Live Upgrade avoids dropping to single-user mode for long periods of time during patching
  
• Live Upgrade relies on an Alternate Boot Environment (ABE)/(BE), which is patched while in multi-user mode; thereby allowing normal system operations to continue with the active BE, while the alternate BE is being patched
• Activating an newly patched (A)BE is essentially a reboot; therefore the downtime is ~= reboot
• Admins can easily revert to the prior Boot Environment (BE) as a safeguard / fallback.
  

Why use Ops Center to patch via Live Upgrade, Alternate Boot Environments, and Solaris 11 equivalents?

• All the benefits of Ops Center's extensive patch and package knowledge base can be leveraged on top of Live Upgrade
• Ops Center can orchestrate patching based on Live Upgrade and Solaris 11 features, which all works together to minimize downtime
• Ops Centers advanced inventory and reporting features assurance that each OS is updated to a verifiable, consistent standard, rather than relying on ad-hoc (error prone) procedures and scripts
  
• Ops Center gives admins control over the boot environment specifications or they can let Ops Center decide when a BE is necessary, thereby reducing complexity and lowering the opportunity for user error
  

Preparing to use Live Upgrade-like features in Solaris 11

Requirements and information you should know:

• Global Zone Root file-systems must be separate from Solaris Container / Zone filesystems

• Solaris 11 has features which are similar in concept to Live Upgrade on Solaris 10, but differ greatly in implementation
  Important distinctions:
  • Solaris 11 assumes ZFS root
  • Solaris 11 adds Boot Environments (BE's) as an integrated feature (see beadm)
  • Solaris 11 BE's avoid single-user patching (vs. Solaris 10 w/ ZFS snapshot=ABE).
  • Solaris 11 Image Packaging System (IPS) has hooks for BE creation, as needed
  • Solaris 11 allows pkgs to be installed + upgraded in alternate BE (e.g. instead of the live system) but it is controlled on a per-pkg basis
  • Boot Environments are activated across a reboot; instead of spending long periods installing + upgrading packages in single user mode.
  • Fallback to a prior BE is a function of the BE infrastructure (a la beadm).
  • (Generally) Reboot + BE activation can be much much faster on Solaris 11

Preparing to use Live Upgrade on Solaris 10

Requirements and information you should know:

• Global Zone Root file-systems must be separate from Solaris Container / Zone filesystems

• Live Upgrade Pre-requisite patches must be applied before the first Live Upgrade Alternate Boot Environments are created (see "Pre-requisite Patches" section, below...)
• Solaris 10 Update 6 or newer on ZFS root is the practical starting point for Live Upgrade
• Live Upgrade with ZFS root is far more straight-forward than any scheme based on Alternative Boot Environments in slices or temporarily breaking mirrors
• Use Solaris best practices to upgrade the OS to at least Solaris 10 Update 4 (outside of Ops Center)
• UFS root can (technically) be used, but it is significantly more involved (e.g. discouraged) -- there are many reasons to move to ZFS while going through the process to update to Solaris 10 Update 6 or newer (out side of Ops Center)

• Recommendation: Start with Solaris 10 Update 6 or newer on ZFS root
• Recommendation: Start with Ops Center 12c or newer
• Ops Center 12c can automatically create your ABE's for you, without the need for custom scripts
• Ops Center 12c Update 2 avoids kernel panic on unpatched Solaris 10 update 9 (and older) -- unrelated to Live Upgrade, but more on the issue, below.

NOTE: There is no magic!  If you have systems running Solaris 10 Update 5 or older on UFS root, and you don't know how to get them updated to Solaris 10 on ZFS root, then there are services available from Oracle Advanced Customer Support (ACS), which specialize in this area.

Live Upgrade Pre-requisite Patches (Solaris 10)

Certain Live Upgrade related patches must be present before the first Live Upgrade ABE's are created on Solaris 10.
Use the following MOS Search String to find the “living document” that outlines the required patch minimums, which are necessary before using any Live Upgrade features:

Solaris Live Upgrade Software Patch Requirements

(Click above – the link is valid as of this writing, but search in MOS for the same "Solaris Live Upgrade Software Patch Requirements" string if necessary)

It is a very good idea to check the document periodically and adapt to its contents, accordingly.

IMPORTANT:  In case it wasn't clear in the above document, some direct patching of the active OS, including a reboot, may be required before Live Upgrade can be successfully used the first time.

HINT: You can use Ops Center to determine what to expect for a given system, and to schedule the “pre-patching” during a maintenance window if necessary.

Preparing to use Ops Center

• Discover + Manage (Install + Configure the Ops Center agent in) each Global Zone
• Recommendation:  Begin by using OCDoctor --agent-prereq to determine whether OS meets OC prerequisites (resolve any issues)
• See prior requirements and recommendations w.r.t. starting with Solaris 10 Update 6 or newer on ZFS (or at least Solaris 10 Update 4 on UFS, with caveats)
  
  
• WARNING: Systems running unpatched Solaris 10 update 9 (or older) should run the Ops Center 12c Update 2 agent to avoid a potential kernel panic
• The 12c Update 2 agent will check patch minimums and disable certain process accounting features if the kernel is not sufficiently patched to avoid the panic
• SPARC: 142900-05 Obsoleted by: 142900-06 SunOS 5.10: kernel patch 10 Oracle Solaris on SPARC (32-bit)
• X64: 142901-05 Obsoleted by: 142901-06 SunOS 5.10_x86: kernel patch 10 Oracle Solaris on x86 (32-bit)
• OR
• SPARC: 142909-17 SunOS 5.10: kernel patch 10 Oracle Solaris on SPARC (32-bit)
• X64: 142910-17 SunOS 5.10_x86: kernel patch 10 Oracle Solaris on x86 (32-bit)
• Ops Center 12c (initial release) and 12c Update 1 agent can also be safely used with a workaround (to be performed BEFORE installing the agent):

# mkdir -p /etc/opt/sun/oc

# echo "zstat_exacct_allowed=false" > /etc/opt/sun/oc/zstat.conf

# chmod 755 /etc/opt/sun /etc/opt/sun/oc

# chmod 644 /etc/opt/sun/oc/zstat.conf

# chown -Rh root:sys /etc/opt/sun/oc

NOTE: Remove the above after patching the OS sufficiently, or after upgrading to the 12c Update 2 agent

Using Ops Center to apply Live Upgrade-related Pre-Patches (Solaris 10)
Overview:

• Create an OS Update Profile containing the minimum LU-related pre-patches, based on the Solaris Live Upgrade Software Patch Requirements, previously mentioned.
• SIMULATE the deployment of the LU-related pre-patches
• Observe whether any of the LU-related pre-patches will require a reboot
• The job details for each Global Zone will advise whether a reboot step will be required
• ACTUALLY deploy the LU-related pre-patches, according to your change control process (e.g. if no reboot, maybe okay to do now; vs. must do later because of the reboot).
• You can schedule the job to occur later, during a maintenance window
• Check the job status for each node, resolving any issues found
• Once the LU-related pre-patches are applied, you can Ops Center to patch using Live Upgrade on Solaris 10

Using Ops Center to patch Solaris 10 with LU/ABE's -- the GOODS!
(this is the heart of the tip):

• Create an OS Update Profile containing the patches that make up your standard build
• Use Solaris Baselines when possible
• Add other individual patches as needed
• ACTUALLY deploy the OS Update Profile
• Specify the appropriate Live Upgrade options, e.g.
• Synchronize the active BE to the alternate BE before patching
• Do not activate the BE after patching
• Check the job status for each node, resolving any issues found
• Activate the newly patched BE according to your change control process
• Activate = Reboot to the ABE, making the ABE the new active BE
• Ops Center does not separate LU activate from reboot, so expect a reboot!
• Check the job status for each node, resolving any issues found
  
  
  

Examples (w/Screenshots)

Solaris 10 and Live Upgrade: Auto-Create the Alternate Boot Environment (ZFS root only)

ABE to be created on ZFS with name S10_12_07REC (Example)

Uses built in feature to call “lucreate -n S10_12_07REC” behind scenes if not already present

NOTE: Leave “lucreate” params blank (if you do specify options, the will be appended after -n $ABEName)

Solaris 10 and Live Upgrade: Alternate Boot Environment Creation via Operational Profile (script)

The Alternate Boot Environment is to be created via custom, user-supplied script, which does whatever is needed for the system where Live Upgrade will be used.

Operational Profile, which provides the script to create an ABE:

• Very similar to the automatic case, but with a Script (Operational Profile), which is used to create the ABE
  
• Relies on user-supplied script in the form of an Operational Profile
• Could be used to prepare an ABE based on a UFS root in a slice, or on a separate device (e.g. by breaking a mirror first) – it is up to the script author to do the right thing!
• EXAMPLE: Same result as the ZFS case, but illustrating the Operational Profile (e.g. script) approach to call:

# lucreate -n S10_1207REC

NOTE: OC special variable is $ABEName

Boot Environment Profile, which references the Operational Profile

• Script = Operational Profile on this screen
• Refers to Operational Profile shown in the previous section
• The user-supplied S10_Create_BE Operational Profile will be run
• The Operational Profile must send a non-zero exit code if there is a problem (so that the OS Update job will not proceed)
  

Solaris 10 OS Update Profile (to provide the actual patch specifications)

Solaris 10 Baseline “Recommended” chosen for “Install”

Solaris 10 OS Update Plan (two-steps in this case)

“Create a Boot Environment” + “Update OS” are chosen.

Using Ops Center to patch Solaris 11 with Boot Environments (as needed)

• Create a Solaris 11 OS Update Profile containing the packages that make up your standard build
• ACTUALLY deploy the Solaris 11 OS Update Profile
• BE will be created if needed (or you can stipulate no BE)
• BE name will be auto-generated (if needed), or you may specify a BE name
• Check the job status for each node, resolving any issues found
• Check if a BE was created; if so, activate the new BE
• Activate = Reboot to the BE, making the new BE the active BE
• Ops Center does not separate BE activate from reboot
• NOTE: Not every Solaris 11 OS Update will require a new BE, so a reboot may not be necessary.

Solaris 11: Auto BE Create (as Needed -- let Ops Center decide)

• BE to be created as needed
• BE to be named automatically
  
• Reboot (if necessary) deferred to separate step
  

Solaris 11: OS Profile

Solaris 11 “entire” chosen for a particular SRU

Solaris 11: OS Update Plan (w/BE)

 “Create a Boot Environment” + “Update OS” are chosen.

Summary:

Solaris 10 Live Upgrade, Alternate Boot Environments, and their equivalents on Solaris 11 can be very powerful tools to help minimize the downtime associated with updating your servers.  For very old Solaris, there are some important prerequisites to adhere to, but once the initial preparation is complete, Live Upgrade can be used going forward.  For Solaris 11, the built-in Boot Environment handling is leveraged directly by the Image Packaging System, and the result is a much more straight forward way to patch, and far fewer prerequisites to satisfy in getting there.  Ops Center simplifies using either approach, and helps you improve consistency from system to system, which ultimately helps you improve the overall up-time across all the Solaris systems in your environment.

Please let us know what you think?  Until next time...
\Leon
--

Leon Shaner | Senior IT/Product Architect
Systems Management | Ops Center Engineering @ Oracle

The views expressed on this [blog; Web site] are my own and do not necessarily reflect the views of Oracle.

---

For more information, please go to Oracle Enterprise Manager  web page or  follow us at : 

Twitter | Facebook | YouTube | Linkedin | Newsletter