We’ve known it all along. The storage for Orchard content items would be much more efficient using a document database than a relational one. Orchard content items are composed of parts that serialize naturally into infoset kinds of documents. Storing them as relational data like we’ve done so far was unnatural and requires the data for a single item
to span multiple tables, related through 1-1 relationships. This means lots of joins in queries, and a great potential for Select N+1 problems. Document databases, unfortunately, are still a tough sell in many places that prefer the more familiar relational model. Being able
to x-copy Orchard
to hosters has also been a basic constraint in the design of Orchard. Combine those with the necessity at the time
to run in medium trust, and with license compatibility issues, and you’ll find yourself with very few reasonable
choices. So we went, a little reluctantly, for relational SQL stores, with the dream of one day transitioning
to document storage. We have played for a while with the idea of building our own document storage on top of SQL databases, and Sébastien implemented something more than decent along those lines, but we had a better way all along that we didn’t notice until recently… In Orchard, there are fields, which are named properties that you can add dynamically
to a content part. Because they are so dynamic, we have been storing them as XML into a column on the main content item table. This infoset storage and its associated API are fairly generic, but were only used for fields. The breakthrough was when Sébastien realized how this existing storage could give us the advantages of document storage with minimal changes, while continuing
to use relational databases as the substrate. public bool CommercialPrices {
get { return this.Retrieve(p => p.CommercialPrices); }
set { this.Store(p => p.CommercialPrices, value); }
}
This code is very compact and efficient because the API can infer from the expression what the type and name of the property are. It is then able
to do the proper conversions for you. For this code
to work in a content part, there is no need for a record at all. This is particularly nice for site settings: one query on one table and you get everything you need.
This shows how the existing infoset solves the data storage problem, but you still need
to query. Well, for those properties that need
to be filtered and sorted on, you can still use the current record-based relational system. This of course continues
to work. We do however provide APIs that make it trivial
to store into both record properties and the infoset storage in one operation:
public double Price {
get { return Retrieve(r => r.Price); }
set { Store(r => r.Price, value); }
}
This code looks strikingly similar
to the non-record case above. The difference is that it will manage both the infoset and the record-based storages. The call
to the Store method will send the data in both places, keeping them in sync.
The call
to the Retrieve method does something even cooler: if the property you’re looking for exists in the infoset, it will return it, but if it doesn’t, it will automatically look into the record for it. And if that wasn’t cool enough, it will take that value from the record and store it into the infoset for the next time it’s required. This means that your data will start automagically migrating
to infoset storage just by virtue of using the code above instead of the usual:
public double Price {
get { return Record.Price; }
set { Record.Price = value; }
}
As your users browse the site, it will get faster and faster as Select N+1 issues will optimize themselves away. If you preferred, you could still have explicit migration code, but it really shouldn’t be necessary most of the time. If you do already have code using QueryHints
to mitigate Select N+1 issues, you might want
to reconsider those, as with the new system, you’ll want
to avoid joins that you don’t need for filtering or sorting, further optimizing your queries.
There are some rare cases where the storage of the property must be handled differently. Check out this string[] property on SearchSettingsPart for example:
public string[] SearchedFields {
get { return (Retrieve<string>("SearchedFields") ?? "") .Split(new[] {',', ' '}, StringSplitOptions.RemoveEmptyEntries); }
set { Store("SearchedFields", String.Join(", ", value)); }
}
The array of strings is transformed by the property accessors into and from a comma-separated list stored in a string. The Retrieve and Store overloads used in this case are lower-level versions that explicitly specify the type and name of the attribute
to retrieve or store.
You may be wondering what this means for code or operations that look directly at the database tables instead of going through the new infoset APIs. Even if there is a record, the infoset version of the property will win if it exists, so it is necessary
to keep the infoset up-to-date. It’s not very complicated, but definitely something
to keep in mind. Here is what a product record looks like in Nwazet.Commerce for example:
And here is the same data in the infoset:
The infoset is stored in Orchard_Framework_ContentItemRecord or Orchard_Framework_ContentItemVersionRecord, depending on whether the content type is versionable or not. A good way
to find what you’re looking for is
to inspect the record table first, as it’s usually easier
to read, and then get the item record of the same id.
Here is the detailed XML document for this product:
<Data>
<ProductPart Inventory="40" Price="18" Sku="pi-camera-box"
OutOfStockMessage="" AllowBackOrder="false"
Weight="0.2" Size="" ShippingCost="null" IsDigital="false" />
<ProductAttributesPart Attributes="" />
<AutoroutePart DisplayAlias="camera-box" />
<TitlePart Title="Nwazet Pi Camera Box" />
<BodyPart Text="[...]" />
<CommonPart CreatedUtc="2013-09-10T00:39:00Z"
PublishedUtc="2013-09-14T01:07:47Z" />
</Data>
The data is neatly organized under each part. It is easy
to see how that document is all you need
to know about that content item, all in one table. If you want
to modify that data directly in the database, you should be careful
to do it in both the record table and the infoset in the content item record.
In this configuration, the record is now nothing more than an index, and will only be used for sorting and filtering.
Of course, it’s perfectly fine
to mix record-backed properties and record-less properties on the same part. It really depends what you think must be sorted and filtered on. In turn, this potentially simplifies migrations considerably.
So here it is, the great shift of Orchard
to document storage, something that Orchard has been designed for all along, and that we were able
to implement with a satisfying and surprising economy of resources. Expect this code
to make its way into the 1.8 version of Orchard when that’s available.
