We’ve known it all along.
The storage for Orchard content items would be much more efficient using a document database than a relational one. Orchard content items are composed of parts that serialize naturally into infoset kinds of documents. Storing them as relational data like we’ve done so far was unnatural and requires
the data for a single item to span multiple tables, related through 1-1 relationships. This means lots of joins
in queries, and a great potential for Select N+1 problems. Document databases, unfortunately, are still a tough sell
in many places that prefer
the more familiar relational model. Being able to x-copy Orchard to hosters has also been a basic constraint
in the design of Orchard. Combine those with
the necessity at
the time to run
in medium trust, and with license compatibility issues, and you’ll find yourself with very few reasonable choices. So we went, a little reluctantly, for relational SQL stores, with
the dream of one day transitioning to document storage. We have played for a while with
the idea of building our own document storage on top of SQL databases, and Sébastien implemented something more than decent along those lines, but we had a better way all along that we didn’t notice until recently…
In Orchard, there are fields, which are named properties that you can add dynamically to a content part. Because they are so dynamic, we have been storing them as XML into a column on
the main content item table. This infoset storage and its associated API are fairly generic, but were only used for fields.
The breakthrough was when Sébastien realized how this existing storage could give us
the advantages of document storage with minimal changes, while continuing to use relational databases as
the substrate. public bool CommercialPrices {
get { return this.Retrieve(p => p.CommercialPrices); }
set { this.Store(p => p.CommercialPrices, value); }
}
This code is very compact and efficient because
the API can infer from
the expression what
the type and name of
the property are. It is then able to do
the proper conversions for you. For this code to work
in a content part, there is no need for a record at all. This is particularly nice for site settings: one query on one table and you get everything you need.
This shows how
the existing infoset solves
the data storage problem, but you still need to query. Well, for those properties that need to be filtered and sorted on, you can still use
the current record-based relational system. This of course continues to work. We do however provide APIs that make it trivial to store into both record properties and
the infoset storage
in one operation:
public double Price {
get { return Retrieve(r => r.Price); }
set { Store(r => r.Price, value); }
}
This code looks strikingly similar to
the non-record case above.
The difference is that it will manage both
the infoset and
the record-based storages.
The call to
the Store method will send
the data
in both places, keeping them
in sync.
The call to
the Retrieve method does something even cooler: if
the property you’re looking for exists
in the infoset, it will return it, but if it doesn’t, it will automatically look into
the record for it. And if that wasn’t cool enough, it will take that value from
the record and store it into
the infoset for
the next time it’s required. This means that your data will start automagically migrating to infoset storage just by virtue of using
the code above instead of
the usual:
public double Price {
get { return Record.Price; }
set { Record.Price = value; }
}
As your users browse
the site, it will get faster and faster as Select N+1 issues will optimize themselves away. If you preferred, you could still have explicit migration code, but it really shouldn’t be necessary most of
the time. If you do already have code using QueryHints to mitigate Select N+1 issues, you might want to reconsider those, as with
the new system, you’ll want to avoid joins that you don’t need for filtering or sorting, further optimizing your queries.
There are some rare cases where
the storage of
the property must be handled differently. Check out this string[] property on SearchSettingsPart for example:
public string[] SearchedFields {
get { return (Retrieve<string>("SearchedFields") ?? "") .Split(new[] {',', ' '}, StringSplitOptions.RemoveEmptyEntries); }
set { Store("SearchedFields", String.Join(", ", value)); }
}
The array of strings is transformed by
the property accessors into and from a comma-separated list stored
in a string.
The Retrieve and Store overloads used
in this case are lower-level versions that explicitly specify
the type and name of
the attribute to retrieve or store.
You may be wondering what this means for code or operations that look directly at
the database tables instead of going through
the new infoset APIs. Even if there is a record,
the infoset version of
the property will win if it exists, so it is necessary to keep
the infoset up-to-date. It’s not very complicated, but definitely something to keep
in mind. Here is what a product record looks like
in Nwazet.Commerce for example:
And here is
the same data
in the infoset:
The infoset is stored
in Orchard_Framework_ContentItemRecord or Orchard_Framework_ContentItemVersionRecord, depending on whether
the content type is versionable or not. A good way to find what you’re looking for is to inspect
the record table first, as it’s usually easier to read, and then get
the item record of
the same id.
Here is
the detailed XML document for this product:
<Data>
<ProductPart Inventory="40" Price="18" Sku="pi-camera-box"
OutOfStockMessage="" AllowBackOrder="false"
Weight="0.2" Size="" ShippingCost="null" IsDigital="false" />
<ProductAttributesPart Attributes="" />
<AutoroutePart DisplayAlias="camera-box" />
<TitlePart Title="Nwazet Pi Camera Box" />
<BodyPart Text="[...]" />
<CommonPart CreatedUtc="2013-09-10T00:39:00Z"
PublishedUtc="2013-09-14T01:07:47Z" />
</Data>
The data is neatly organized under each part. It is easy to see how that document is all you need to know about that content item, all
in one table. If you want to modify that data directly
in the database, you should be careful to do it
in both
the record table and
the infoset
in the content item record.
In this configuration,
the record is now nothing more than an index, and will only be used for sorting and filtering.
Of course, it’s perfectly fine to mix record-backed properties and record-less properties on
the same part. It really depends what you think must be sorted and filtered on.
In turn, this potentially simplifies migrations considerably.
So here it is,
the great shift of Orchard to document storage, something that Orchard has been designed for all along, and that we were able to implement with a satisfying and surprising economy of resources. Expect this code to make its way into
the 1.8 version of Orchard when that’s available.
