Where Have All the Composites Gone?

I was just asked to help out with an interesting problem at a customer.  All their composites had disappeared from the EM console, none of them showed as loading in the log files and there was an ominous error message in the logs.

Symptoms

After a server restart the customer noticed that none of his composites were available, they didn’t show in the EM console and in the log files they saw this error message:

SEVERE: WLSFabricKernelInitializer.getCompositeList Error during parsing and processing of deployed-composites.xml file

This indicates some sort of problem when parsing the deployed-composites.xml file.  This is very bad because the deployed-composites.xml file is basically the table of contents that tells SOA Infrastructure what composites to load and where to find them in MDS.  If you can’t read this file you can’t load any composites and your SOA Server now has all the utility of a chocolate teapot.

Verification

We can look at the deployed-composites.xml file from MDS either by connecting JDeveloper to MDS, exporting the file using WLST or exporting the whole soa-infra MDS partition by using EM->SOA->soa-infra->Administration->MDS Configuration.  Exporting via EM is probably the easiest because it then prepares you to fix the problem later.  After exporting the partition to local storage on the SOA Server I then ran an XSLT transform across the file deployed-composites/deployed-composites.xml.

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns="http://www.w3.org/1999/xhtml">
    <xsl:output indent="yes"/>
    <xsl:template match="/">
        <testResult>
            <composite-series>
                <xsl:attribute name="elementCount"><xsl:value-of select="count(deployed-composites/composite-series)"/></xsl:attribute>
                <xsl:attribute name="nameAttributeCount"><xsl:value-of select="count(deployed-composites/composite-series[@name])"/></xsl:attribute>
                <xsl:attribute name="defaultAttributeCount"><xsl:value-of select="count(deployed-composites/composite-series[@default])"/></xsl:attribute>
                <composite-revision>
                    <xsl:attribute name="elementCount"><xsl:value-of select="count(deployed-composites/composite-series/composite-revision)"/></xsl:attribute>
                    <xsl:attribute name="dnAttributeCount"><xsl:value-of select="count(deployed-composites/composite-series/composite-revision[@dn])"/></xsl:attribute>
                    <xsl:attribute name="stateAttributeCount"><xsl:value-of select="count(deployed-composites/composite-series/composite-revision[@state])"/></xsl:attribute>
                    <xsl:attribute name="modeAttributeCount"><xsl:value-of select="count(deployed-composites/composite-series/composite-revision[@mode])"/></xsl:attribute>
                    <xsl:attribute name="locationAttributeCount"><xsl:value-of select="count(deployed-composites/composite-series/composite-revision[@location])"/></xsl:attribute>
                    <composite>
                        <xsl:attribute name="elementCount"><xsl:value-of select="count(deployed-composites/composite-series/composite-revision/composite)"/></xsl:attribute>
                        <xsl:attribute name="dnAttributeCount"><xsl:value-of select="count(deployed-composites/composite-series/composite-revision/composite[@dn])"/></xsl:attribute>
                        <xsl:attribute name="deployedTimeAttributeCount"><xsl:value-of select="count(deployed-composites/composite-series/composite-revision/composite[@deployedTime])"/></xsl:attribute>
                    </composite>
                </composite-revision>
                <xsl:apply-templates select="deployed-composites/composite-series"/>
            </composite-series>
        </testResult>
    </xsl:template>

    <xsl:template match="composite-series">
            <xsl:if test="not(@name) or not(@default) or composite-revision[not(@dn) or not(@state) or not(@mode) or not(@location)]">
                <ErrorNode>
                    <xsl:attribute name="elementPos"><xsl:value-of select="position()"/></xsl:attribute>
                    <xsl:copy-of select="."/>
                </ErrorNode>
            </xsl:if>
    </xsl:template>
</xsl:stylesheet>

The output from this is not pretty but it shows any <composite-series> tags that are missing expected attributes (name and default).  It also shows how many composites are in the file (111) and how many revisions of those composites (115).

<?xml version="1.0" encoding="UTF-8"?>
<testResult xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns="http://www.w3.org/1999/xhtml">
   <composite-series elementCount="111" nameAttributeCount="110" defaultAttributeCount="110">
      <composite-revision elementCount="115" dnAttributeCount="114" stateAttributeCount="115"
                          modeAttributeCount="115"
                          locationAttributeCount="114">
         <composite elementCount="115" dnAttributeCount="114" deployedTimeAttributeCount="115"/>
      </composite-revision>
      <ErrorNode elementPos="82">
         <composite-series xmlns="">
            <composite-revision state="on" mode="active">
               <composite deployedTime="2010-12-15T11:50:16.067+01:00"/>
            </composite-revision>
         </composite-series>
      </ErrorNode>
   </composite-series>
</testResult>

From this I could see that one of the <composite-series> elements (number 82 of 111) seemed to be corrupt.

Having found the problem I now needed to fix it.

Fixing the Problem

The solution was really quite easy.  First for safeties sake I took a backup of the exported MDS partition.  I then edited the deployed-composites/deployed-composites.xml file to remove the offending <composite-series> tag.

Finally I restarted the SOA domain and was rewarded by seeing that the deployed composites were now visible.

Summary

One possible cause of not being able to see deployed composites after a SOA 11g system restart is a corrupt deployed-composites.xml file.  Retrieving this file from MDS, repairing it, and replacing it back into MDS can solve the problem.  This still leaves the problem of how did this file become corrupt!