Thursday, June 02, 2016

Decennial Series #3 – Sitecore.Context is an anti-pattern

And I’ll even attempt to tell you why

If you’re in any way or form following trends for “normal” development (should such a thing exist), you will no doubt have come across one or both of these terms:

  • Inversion of Control
  • Dependency Injection

You might even know what they mean. And why these terms are so popular these days.

Or you might not. But you’re implementing code following these principles anyway. Because your boss, team lead, customer, client or girlfriend demands that you do.

Either way, I’ll just do a very quick explanation of what these mean. If you know enough about these terms to tell me I’m oversimplifying this, this introduction wasn’t meant for you anyway. Skip to the good stuff further down.

IOC and DI in less than 5 minutes

Inversion of Control

Inversion of Control (IoC from here on) is the principle, that nothing in your system should make decisions on exactly which other systems to “talk” to. So your CustomerManager class shouldn’t really get to decide, it wants to talk to a SqlCustomerProvider. Your BasketController shouldn’t really get to decide, everything should be stored in HttpContext.Current.Session.

And why is that?

Because things change. They change over time, as customer requirements change. Someone might later want those BasketItems stored in a back-end Sql table so the Basket isn’t lost between user visits to your site. Someone might not want to pay good money for a Sql Server license to store only 200 customers.

Things change, is my point. And we’re trying to write software to best accommodate that.

But how can I write my BasketController if I can’t store my BasketItems anywhere?

But you can. And in writing it, you do obviously get to decide WHAT you require in order to store and retrieve your BasketItems. You just don’t write the actual implementation. You produce a “blueprint” of the requirements you have.

In C#, we call this an “Interface”.

So in short (simple) summary. Inversion of Control (IoC).

You might have started out like this:

public class BasketController
{
    private readonly BasketStorage _bs;

    public BasketController()
    {
        _bs = new BasketStorage();
    }

    public List<BasketItem> GetBasket(int customerId)
    {
        return _bs.GetBasket(customerId).ToList();
    }

    public void StoreBasket(int customerId, IEnumerable<BasketItem> items)
    {
        _bs.StoreBasket(customerId, items);
    }
}

So what’s the big deal?   Well it’s about control. You might argue; “so what if the client changes her mind, and wants this to come from Sql instead of Session?  I’ll just make a new BasketStorage class, job done”.

And you’re right of course.

But what about me?  the consumer of your API?   or me, the maintenance developer who – 3 years from now – actually want to do both?  Store BasketItems in SessionState – and then persist them to Sql when Session expires?  Why can’t I be your friend?

I’m the consumer of this API, and it’s not unreasonable that I have some form of control over these matters.

So the way we both get happy, is find common ground. With just a little tweaking (and tools like Resharper does like 95% of this work for you anyway; go ask your boss for a license), we can both win. You still get to easily replace one BasketStorage for another, and I get a choice in the matter too.

So we (you, actually) make up the blueprint. The Interface.

public interface IBasketStorage
{
    void StoreBasket(int customerId, IEnumerable<BasketItem> items);
    IEnumerable<BasketItem> GetBasket(int customerId);
}

And you apply this to your existing BasketStorage class.

public class BasketStorage : IBasketStorage
{
    public void StoreBasket(int customerId, IEnumerable<BasketItem> items)
    {
        throw new NotImplementedException();
    }

    public IEnumerable<BasketItem> GetBasket(int customerId)
    {
        throw new NotImplementedException();
    }
}

(and you finish up the methods that I am too lazy to flesh out in this blogpost, naturally)

So far, this is a matter of maybe 20 keystrokes and a few Resharper tricks. And you’re almost done. One last thing.

private readonly IBasketStorage _bs;

public BasketController(IBasketStorage bs)
{
    _bs = bs;
}

And that’s it. You’ve now achieved IoC with Dependency Injection (DI). Well done you.

“But instead of having just 1 place in my code where I created my BasketStorage() class, I am now forced to spread it out everywhere in my codebase!?!?   SURELY that cannot be better?”

And you’re right indeed. Enter: The Service Locator. Notice how I made the person next to you just spill coffee?

How are we for time?  got a couple of minutes before the 5 minutes are up, yea?

Service Locator, also in very short and simple terms.

If you’ve heard the terms IoC and DI, you have absolutely also heard someone – at one point or another – mention “Service Locator is an Anti-Pattern”.

And it can be. Indeed. But let me also tell you…   somewhere in your system, SOMEONE is going to have to instantiate some classes and services at some point. And they are going to need to locate these classes and services. Stay with me on this. We’ll get through this, you and I.

The most direct way to go about, getting these dependencies injected – is to use the Service Locator Pattern. It looks like this:

var basketStorage = DependencyResolver.Current.GetService<IBasketStorage>();
var basketController = new BasketController(basketStorage);

Or the slightly better version:

var basketController = DependencyResolver.Current.GetService<BasketController>();

Notice how, in this last example, we don’t even worry about IBasketStorage any longer. The DependencyResolver is meant to work this out (almost) on it’s own.

“But how does it know?”

Yea, I skipped that didn’t I?  I don’t really want to make this post about, how DI frameworks are configured. In Castle Windsor, it looks like this:

container.Register(Component.For<IBasketStorage>().ImplementedBy<BasketStorage>());

So now, whenever someone requests (registers a dependency for, usually in the class constructor) IBasketStorage, the DependencyResolver knows to instantiate BasketStorage() and send that off to the requesting class.

Just for reference, this form of DI we call “constructor injection” for hopefully obvious reasons.

I think our 5 minutes are up. So let’s address the elephant in the room; “Why is this an Anti-Pattern then?”.

Service Locator and the bad rep

It’s not so bad. Really. Not when used like I just showed here. But that’s not to say, you can’t get in a lot of trouble using SLOC. You absolutely can. Let’s return to the example from before. Imagine I had done it like this:

private readonly IBasketStorage _bs;

public BasketController()
{
    _bs = DependencyResolver.Current.GetService<IBasketStorage>();
}

Same thing, right?  Except now, my BasketController() is nice and easy to instantiate again.

Wrong.

First of all, you’ve now come full circle and achieved exactly nothing. You’ve pushed an interface down on top of BasketStorage, good on you. But to what end?

Dependencies are no longer injected, so DI is out. As for control. Well it’s still somewhat inverted at least – but it’s well hidden from my view. Me, the API consumer/maintenance developer/ravaging psychopath who knows where you live.

So when I, 3 years from now, go… “oh, I’m gonna use this fancy BasketController. Public constructor, no dependencies. I’ll just add 1.000.000 to customerId, and we can use it to store the Wishlish for the customer – I’m a genious. Long weekend, here I come. We’re in an HttpModule with no SessionState, but why should that matter?”. And I go:

var bc = new BasketController();
bc.StoreBasket(1000000+customerId, wishlist);

Yay me. Except. Noooooo. YSOD.

What happened?  Well I didn’t know, invoking BasketController called out using SLOC and procured IBasketStorage. I probably didn’t even know that IBasketStorage required SessionState – but as I said, I didn’t even know about the requirement for it in the first place.

And that, is the problem with the Service Locator.

It hides dependencies. It’s (sort of) IoC but without DI. It’s Prince without The NPG. It’s Bonnie without Clyde.

It’s just not really very good. That’s what I’m saying.

The Good Stuff

I told you this was coming yea?

So as I might have let shine through a little; I do come across zealots from time to time who will turn into an absolute jumblepot of failed arguments and internet quotes whenever I even dare mention the unspeakable Service Locator. “Everybody KNOWS it’s an Anti-Pattern.” (you know who you are).

The irony is, in our day to day work with Sitecore, you (not me) use the Service Locator Pattern ALL THE TIME. Hundreds and hundreds of times every day. Often these very same zealots use it, some of them aren’t even aware.

Take a look at this:

var database = DependencyResolver.Current.GetService<Database>();
var database = Sitecore.Context.Database;

If you think there is anything other than just a semantic difference between those two lines of code, you’ve missed a point somewhere. Start over from the top of this article.

Sitecore.Context is a Service Locator

Actually, I’ll work the font a little and repeat.

Sitecore.Context is a Service Locator

And it has ALL the problems of a Service Locator, too. All of them.

Ever tried calling some of your code – say some Url Generation code – only to find it blows up when called from a Sitecore Publishing Processor?  Sitecore.Context.Site is NOT what you expect is it? 

Or called some Sitecore code from a Unit Test?   I tell you; BOOKS have been written to explain to you why this will blow up. Hint: It’s mostly to do with Sitecore.Context.Database.

Sitecore – in pretty much every corner and crack you look into, has services and functions that rely on other Sitecore services and functions. It is all hidden from view (although some, very far from all, is exposed in the configs), and unless you have lots and lots of Sitecore experience, predicting what calling Sitecore.Context.Database is actually going to need is near impossible.

There is little to no runtime DI in play anywhere; and the config files only allow you to really alter some of the dependencies on a very top level.

And you know what?  this is fine.  Well no it’s not – but it is what it is. If this ever changes, I am almost willing to bet a Dollar, Sitecore will roll out a NEW API to live side-by-side with the existing one. But let’s leave this for now and get to the punchline.

Every time you use Sitecore.Context (or RenderingContext.Current or pretty much any global static in the Sitecore API) – you are using the Service Locator Pattern. And it gives you all the problems that got it labelled as an Anti-Pattern to begin with.

I’m not saying you can avoid it, really. While there are ways to work around most of it; in some cases you just have to cease fire and accept the way things are.

But I’m saying you can:

  • Be aware of it
    • Awareness of a problem is often the first step in minimizing the impact, the problem will have on your project
  • Minimize the use of it
    • Declare your dependencies. Even if you can’t fix all of it; demand “SiteContext context, Database database” in your constructor if you’re going to use them. At least bring back DI.

Over time, this becomes second nature to you. You no longer get mysterious yellow screens because your context is not what you expect. You no longer increase the workload of your poor maintenance programmer by 100-fold, trying to work out what external dependencies your code has (generating a Product Url requires a Site Context, a Context Database, a Context Language, a Context Item and your SSN – seems fair, right? – real story btw, except for the SSN).

Now let’s get back to work.

Wednesday, May 18, 2016

Setup Castle Windsor for Sitecore 8.1 in 5 easy steps

Getting started with Dependency Injection is easier than you think

So I was reading Kevin Obee (@KevinObee) write about Dependency Injection, and how many of you out there wasn’t using it yet. While the post that inspired me addresses Webforms implementations, I also find this lacking in many MVC solutions I come across.

Further; most of the examples you find on the web now – setting up DI for Sitecore – is obsolete as of Sitecore 8.1. These days, it’s much much easier than it used to be.

I’m going to demonstrate using Castle Windsor as my DI container. Personal preference, I’m sure there are similar ways of doing it with AutoFAC, SimpleInjector and whatever else roams about.

So let’s get to it.

Step 1 – Install packages

Start with a blank solution set up for MVC. Or use whatever you have on hand right now. Now get your NuGet going. We’re after the CommonServiceLocator.WindsorAdapter.

Via GUI

step-1_thumb1

or if  you prefer, via the Nuget Package Manager Console.

step-2_thumb2

 

Step 2 – Update packages

Do a few updates. Personal preference here, but registration options have improved a lot since the minimum version of Castle Windsor required for this to work.

step-3_thumb1

 

Step 3 – Write some code

But not a lot. Truly.

step-4_thumb2

IntelliSense… mostly a blessing, but sometimes a bit of a nag.

Step 4 – Line it up

Now we just need to wire this up. Sitecore Best Practice would be, to get this cooked up in the <initialize> pipeline. So let’s do that.

step-5_thumb1

Step 5 – Join us on Slack

Just kidding. You’re done. That’s it. I won’t go into detail on how to actually use Dependency Injection, there are plenty of good tutorials on the web for that.

Meanwhile I suggest you come on the Sitecore Slack Channel. You’ve got plenty of spare time on your hands now, I just showed you how to do a 1 day task in 5 minutes flat ;-)

If you’re feeling doubly lazy, I’ve created a Gist with the two files on display.

Sunday, May 08, 2016

Effective use of Sitecore LinkDatabase

Keep track of your Reference Field references

I’ve often mentioned the Sitecore LinkDatabase, which I’ve found to be a hugely useful resource over the years but it’s getting very little exposure. And given I need to test out the Open Source version of my favourite blogging tool, I figured I might as well hit 2 birds with 1 stone and write a post about it.

What is Sitecore LinkDatabase?

Sitecore LinkDatabase (to save my sanity, hereafter referred to as SLDB) is a simple table that keeps track of Reference Field references. And what are Reference Fields?   Basically anything that stores one or more IDs as the raw field value. So Droplink, Treelist, Image, Internal Link and so on.

SNAGHTML13175c1f

And raw values looking like this:

image

And this is where the SLDB comes into play. What it does, is keep track of those references. It basically stores something to the effect of; “Item {id} on Field {id} keeps a reference to Target Items {id}{id}{id}”.  The reasons this functionality exists, is to support the Sitecore Content Editor so that when you are about to delete an item that might be used elsewhere, it pops a warning.

image

In this case I am not given other choice (as I’m trying to delete a Template in use), but in other cases Sitecore will offer you to remove the references to the Item you’re about to delete, or relink the references to someplace else. As a side note; this, too, is immensely useful functionality if you’re refactoring your content – but that’s a story for a different day.

The nuts and bolts

Currently, the SLDB looks like this:

image

It sits in your “core” database. Be mindful of this if you’re going to be putting it to use – “core” may or may not be available in your Content Delivery environment depending on your setup. Easily fixed, however. Configuration for it sits here:

  <!-- LINK DATABASE -->
  <LinkDatabase type="Sitecore.Data.$(database).$(database)LinkDatabase, Sitecore.Kernel">
    <param connectionStringName="core" />
  </LinkDatabase>

Migrate the “Links” table from “core” to “web” or wherever you please, and you’re good to go.

The day to day is handled by this event handler, you’ll find it on most of the item events.

<handler type="Sitecore.Links.ItemEventHandler, Sitecore.Kernel" method="OnItemCopied" />

And there’s also a setting. Leave it at it’s default if you’re going to be using the SLDB at runtime (if you toggle this “off”, SLDB won’t be kept up to date for “web”).

<setting name="LinkDatabase.UpdateDuringPublish" value="true" />

Caveat: If you’re using the excellent Unicorn; be sure to allow it to update SLDB during sync operations. It is currently off by default. You need version 3.1.5 or later.

The nitty-gritty

Right, so enough. WHY should you care?  Well let me show you.

So I’ve set up a vanilla solution, and defined me some templates. It’s not particularly advanced, but it represents a pretty common IA challenge in our solutions today.

SNAGHTML134587f2

So we have a “News” template. It can exist in several categories, and has 1 Author.

IA challenge #1: We can’t organise “News” in category “folders”, as it is not a 1:1 relationship
IA challenge #2: We need news organised in a deep “folder” structure for SEO and performance

Using a bit of c# code so hacky I would never share it, I end up with 1000 “News” articles, with a random number of categories and so on.

image

I’ve kept the organisation simple, as this will be enough for what I’m trying to demonstrate.

image

So basically a “month” folder. In a real scenario, likely you would go 1 level deeper and include “year”. It won’t matter for this example however.

Now; our common problems with this are:

Problem #1: How do I list all “News” in a specific Category?
Problem #2: How do I list all “News” by a specific Author?

I’m going to pretend we live in a world where indexing this data isn’t an option, and we’re left with just the basic tools available to us.

Solution #1: Use Sitecore Query
Solution #2: Use Sitecore Fast Query
Solution #2: Use SLDB

Let’s see what that looks like. I cook up some rudimentary (and ugly) code:

var db = Factory.GetDatabase("web");
List<Item> authors = db.GetItem("/sitecore/content/Solution Data/Authors").GetChildren().ToArray().ToList();
List<Item> categories = db.GetItem("/sitecore/content/Solution Data/Categories").GetChildren().ToArray().ToList();
ID newsTemplateId = new ID("{5CB0FC6D-DCAA-4A51-8745-D9933F77679A}");
var newsRoot = db.GetItem("/sitecore/content/Home/news");

// Make sure Sitecore is warmed up
newsRoot.Axes.GetDescendants();

Response.Write("<h2>Authors</h2>");
var sw = new Stopwatch();
foreach (var author in authors)
{
    Response.Write($"<strong>{author.Name}</strong><br />");
    sw.Start();
    var referrers = GetAuthorReferrers(newsRoot, author.ID);
    sw.Stop();
    Response.Write(referrers.Length + " articles; Time in Ms: " + sw.ElapsedMilliseconds + "<hr />");
    sw.Reset();
}

And the most important bit, how I query it.

Regular Sitecore Query

Item[] GetAuthorReferrers(Item root, ID authorId)
{
    string query = root.Paths.FullPath + "//*[contains(@Author, '" + authorId + "')]";
    return root.Database.SelectItems(query);
}

I run this a couple of times (I’m not exactly in a stable environment to conduct any real scientific test of this – keep this in mind).

image

I then rewrite it for Sitecore Fast Query.

Sitecore Fast Query

BE AWARE!   Sitecore Fast Query sacrifices functionality to gain performance. Be especially aware of its limitations if your solution is multilingual.

Code now looks like this:

Item[] GetAuthorReferrers(Item root, ID authorId)
{
    string query = "fast:" + root.Paths.FullPath + "//*[@Author = '%" + authorId + "%']";
    return root.Database.SelectItems(query);
}

And an output that comes out like this. (Believe me, I ran this dozens of times).

image

Are we having fun yet?  I can offer up a bit of speculation on what’s going on here, but I’m honestly not completely sure. Mostly because I would never use neither solution to query my data, so my experience is somewhat limited. My best guess would be; to benefit from fast: query, you need more data. I will try this out with a larger dataset a bit further down.

So anyway. Roll on SLDB.

Sitecore LinkDatabase

Drawback of this approach is, that it’s a bit more code heavy. Not by much though.

Item[] GetAuthorReferrers(Item root, Item author)
{
    ID authorFieldId = new ID("{438E45E5-9F85-4705-976E-FC76E563F5EF}");

    var items = new List<Item>();
    ItemLink[] itemLinks = Sitecore.Globals.LinkDatabase.GetItemReferrers(author, false);
    foreach (var il in itemLinks)
    {
        if (il.SourceDatabaseName.Equals(root.Database.Name) && il.SourceFieldID == authorFieldId)
        {
            items.Add(il.GetSourceItem());
        }
    }
    return items.ToArray();
}

And here’s how we end up.

image

Am I managing to convince you yet?   so far we’re doing ok. We’re outperforming Sitecore Query by about 10 to 1. And Sitecore Fast Query by 50 to 1. A pretty decent start I’d say.

The sleight of hand

So I’m left with two problems. I’ve not brought “Category” into play. And I’ve not got a simple output, that fully compares these 3 side by side. I’ve created a Gist with the full source code – it is to long to include here, even by my standards.

Enter, a code rewrite. This time I’m looping through all Categories, then all Authors – finding all News articles that match the Category AND the Author. It completes the setup, and it actually demonstrates one of the slightly more complex situations when using SLDB. Using querying this becomes just another “and” expression.

SLDB code looks like this:

private Item[] GetNewsArticlesSitecoreLinkDatabase(Item newsRoot, Item authorItem, Item categoryItem)
{
    // Note. SLDB version of this actually needs to make 2 lookups. First to find all news articles referencing our author, then all that reference the category.
    // Note #2: Still, my hand is not shaking ;-)

    var authorFieldId = new ID("{438E45E5-9F85-4705-976E-FC76E563F5EF}");
    var categoriesFieldId = new ID("{31D2A6CC-6984-4CA0-BB73-9D39C3B8D0AA}");

    var authorLinks = Globals.LinkDatabase.GetItemReferrers(authorItem, false).Where(al => al.SourceFieldID == authorFieldId).ToList();
    var categoryLinks = Globals.LinkDatabase.GetItemReferrers(categoryItem, false).Where(cl => cl.SourceFieldID == categoriesFieldId).ToList();

    var newsArticles = new List<Item>();
    foreach (var authorIl in authorLinks)
    {
        var categoryIl = categoryLinks.ToList().Find(cl => cl.SourceItemID == authorIl.SourceItemID);
        if (categoryIl != null && categoryIl.SourceDatabaseName == newsRoot.Database.Name)
        {
            newsArticles.Add(categoryIl.GetSourceItem());
        }
    }

    return newsArticles.ToArray();
}

And what do we get?   Well this.

image

It matches up pretty well the results already established. Adding the extra query seems to have added a bit to all of the query execution times – this is completely expected. In relative terms however, I think it is still very clear who the winner is.

Saturday, May 07, 2016

Sitecore Decennial Series #2 - Don't Sitecore Query your content

Sitecore does not query, how you think it queries

As a consultant, I am often brought in to projects and solutions that have - to some extent - gone wrong, or does not quite meet up to expectations. And when it comes to those, absolutely without question, the most common problem I come across, is under-performing solutions.

And let's just clear something up. While you will find a lot of posts out there discussing the speed and performance of Sitecore (Sitecore 8, in particular); these are all referring to the performance of the Sitecore Experience Editor. The client environment. And while there is rarely this much smoke without a fire somewhere, I will say this:

The Sitecore runtime is a very well oiled machine. If it doesn't perform, it means YOU broke it.

You are welcome to quote me on this.

Let me exemplify.






















Pretty gruesome, right?  You'd not blame this on "poor performance of ASP.NET", if you found this in some web application I assume.

Now believe me when I tell you; this horrifying example is actually very mild compared to what I often find hidden deep in some of the Sitecore solutions I visit. Usually tucked away in some "*Helper" or "*Utils" (pet peevees of mine) class, used arbitrarily throughout the entire project, never to have been visited by a developer since it was first put in place.

So anyway. Let's take a look at, how situations like this come to be. I will begin with some fundamentals.

The Sitecore (Sql) Data Store

To try and understand what's going on, we're going to take a bit of a dive into the Sitecore Sql data store. Now before you start; I am very well aware that Sitecore abstracts this layer from view (and rightly so), and under no circumstances should you ever be working on the database layer when doing Sitecore. The structures I am about to show you do change from time to time, and you will end up with a bloody nose if you go down this road.

But we cannot escape the fact, that Sitecore data storage is based on Sql Server (in most cases, actually in every single one I have come across in the past 10 years) - and any data storage mechanism comes with a set of built-in limitations and restrictions. So how Sitecore uses its data storage matters, and it will affect how your solution performs.

[Items]














Without question, the table most central to the Sitecore data store. I imagine the fields are self explanatory. 1 row in this table, for every Item in your Sitecore Database ("master", "web" and so on).

This table holds no other values other than Item.Name. For this, Sitecore looks to 3 other tables. They look almost identical.

[VersionedFields]















[UnversionedFields]



[SharedFields]


If you haven't guessed already; Sitecore decides where to place field values depending on how you check your "Shared" and "Unversioned" checkboxes when defining your Sitecore Templates. I will not go further into this, in this post.

But what does this mean?   Well there's a couple of things to pay close attention to here. The first one being; Sitecore sees its data store as a Tree Structure (d'uh I hear you say, you probably already knew that). But this is important. Notice how [Items] stores Id and ParentId (and a few other Ids we don't care about right now). To go anywhere in the "Tree", Sitecore must therefore traverse up and down this chained list of Ids - starting from the top.

So if I were to do something simple as say; Sitecore.Context.Database.GetItem("/sitecore/content") - what this ends up as, is something like this:

exec sp_executesql N' SELECT [ID] FROM [Items] WHERE [ParentID] = @parentId AND [Name] = @childName',N'@parentId uniqueidentifier,@childName nvarchar(7)',@parentId='11111111-1111-1111-1111-111111111111',@childName=N'content'

(Sitecore caching is in effect, so it was already aware of the parentId)

Which is not so bad; but do remember. Had I queried "/sitecore/content/home/data/articles/2015/february" - this would (uncached) have resulted in 6 queries like the one above, to Sql Server. Yep. Again; this is without considering caching, obviously, or we'd all be in big trouble.

And this is just getting an Item. As we saw above, the field values sit in separate tables. Once you start accessing actual content on the Item, this is what it looks like behind the scenes.

exec sp_executesql N'SELECT [ItemId], [Order], [Version], [Language], [Name], [Value], [FieldId], [MasterID], [ParentID]
                     FROM (
                        SELECT [Id] as [ItemId], 0 as [Order], 0 as [Version], '''' as [Language], [Name], '''' as [Value], [TemplateID] as [FieldId], [MasterID], [ParentID]
                        FROM [Items]

                        UNION ALL                          
                        SELECT [ParentId] as [ItemId], 1 as [Order], 0 as [Version], '''' as [Language], NULL as [Name], '''', NULL, NULL, [Id]
                        FROM [Items] 

                        UNION ALL 
                        SELECT [ItemId], 2 as [Order], 0 AS [Version], '''' as [Language], NULL as [Name], [Value], [FieldId], NULL, NULL
                        FROM [SharedFields] 

                        UNION ALL 
                        SELECT [ItemId], 2 as [Order], 0 AS [Version],       [Language], NULL as [Name], [Value], [FieldId], NULL, NULL
                        FROM [UnversionedFields] 

                        UNION ALL 
                        SELECT [ItemId], 2 as [Order],      [Version],       [Language], NULL as [Name], [Value], [FieldId], NULL, NULL 
                        FROM [VersionedFields]
                     ) as temp  WHERE [ItemId] IN (SELECT [ID] FROM [Items] WITH (nolock)  WHERE [ID] = @itemId) 
                     ORDER BY [ItemId], [Order] ASC, [Language] DESC, [Version] DESC',N'@itemId uniqueidentifier',@itemId='0DE95AE4-41AB-4D01-9EB0-67441B7C2450'

Incidentally; ever had a [Shared Field] that you switched to [Unshared], and now you're not getting the field values you expect?  this query is the reason. When you toggle that checkbox, Sitecore starts a background task that will migrate the field values from [SharedFields] to [UnversionedFields]. It has to, as the above query will otherwise return the first field value it finds in the UNION - which will be the one near the top; [SharedFields]. If this task somehow doesn't complete - your data storage gets left in a bit of a pickle. 

So yea. Sitecore essentially goes and grabs all your field values once you start accessing content on an item. And yes, this IS the efficient approach - a roundtrip to Sql for every field value you access hereafter would be absolutely crippling.

But keeping this in mind; picture this:

Sitecore.Context.Database.SelectItems("/sitecore/content//*[@PageTitle='news']")

Caches aside; you've just instructed Sitecore to recursively go find anything that has /sitecore/content as an ancestor (1 SQL statement for each item, to find them all). And then your request for the PageTitle field will then, for each of the found items, execute the big UNION statement from above so that it can determine the field value for PageTitle.

Yep, you read that right. You may be imagining things like; "but I can construct a SQL statement that finds this more effectively". And you're right, you can. But you'd also be ignoring Sitecore through and through - you might as well not bother using a CMS at all. What you're overlooking (to name a few) are things like Versions (which of the 7 versions of /Home is the active one?), Language, Security, Workflow Status. 

I'll say this as plainly and calmly as I possibly can. Do NOT use Sitecore Query for any code that executes at runtime on your solution. "If it doesn't perform, it means YOU broke it."

When it comes to constructing a more efficient query, you're not the first one to have this thought though; enter Sitecore Fast Query.

Sitecore Fast Query

SFQ is a query mechanism that aims to address some of the limitations of regular Sitecore Query. It does so by compromising exactly some of the things I mention above. So sacrifice a little functionality to gain a little speed. I'm almost tempted to paraphrase Frankling and claim "you shall have neither".

But let's look at it. From the "Using Sitecore Fast Query Cookbook" (how I miss those)
































Now; given everything that I have just shown you - this makes perfect sense. The only supported special attributes - all of those are what you find on the [Items] table. Standard Values are not supported (as these sit on a separate item). I'll leave it to you to speculate why field values need to be surrounded by % signs ;-)

Ok, so let's take a look then. I cook up something new:

Sitecore.Context.Database.SelectItems("fast:/sitecore/content//*[@#Page Title#=\"%article%\"]");

And the resulting Sql looks like this:

exec sp_executesql N'SELECT DISTINCT [i].[ID] [ID], [i].[ParentID] [ParentID] FROM [Items] [i] WITH (NOLOCK)  LEFT OUTER JOIN (SELECT [Fields].* from [Fields] INNER JOIN [Items] ON [Fields].[FieldID] = [Items].[ID] AND lower([Items].[Name]) = ''page title'') [Fields1] ON [i].[ID] = [Fields1].[ItemId] INNER JOIN [Descendants] ON [i].[ID] = [Descendants].[Descendant] INNER JOIN (SELECT DISTINCT [i].[ID] [ID], [i].[ParentID] [ParentID] FROM [Items] [i] WITH (NOLOCK)  INNER JOIN (SELECT DISTINCT [i].[ID] [ID], [i].[ParentID] [ParentID] FROM [Items] [i] WITH (NOLOCK)  WHERE LOWER([i].[Name]) = ''sitecore'' AND [i].[ParentID] = @value1) [a] ON [i].[ParentID] = [a].[ID] WHERE LOWER([i].[Name]) = ''content'') [a] ON [Descendants].[Ancestor] = [a].[ID] WHERE (coalesce([Fields1].[Value], '''') LIKE @value2)',N'@value1 uniqueidentifier,@value2 nvarchar(9)',@value1='00000000-0000-0000-0000-000000000000',@value2=N'%article%'

And a footprint like this.










Ok. So there's no doubt, this is BETTER than our previous situation. But only by a measure, it's not the end-all to our performance woes. I have about 20 (!) items in the vanilla solution I am using to write this blog post by the way. Twenty.

Interesting (unofficial) factoid: Do you know why FAST: query was introduced to begin with?  To solve performance problems in Sitecore Content Editor. Yes. Think about it; opening up any item in CE presents you with a series of fields. Each of these fields may be Treelists and whatnot, and loading up all that related meta-data and presets, populating all the dropdown boxes and so on - well it pretty much killed performance in early Sitecore (5 and 6) releases. And for Content Editor, the limitations of Fast Queries are fine. No doubt they're (often) better than the regular Sitecore Query equivalent. But that still doesn't make them good.

Don't look to caching to save you either. It helps, surely. But even cached results need to be generated at least once; and now you find yourself in a whole heap of other problems instead. When to clear cache?  on publish?  you sure?   So with editors publishing new content every 5 minutes (to test it out - trust me, they do), how much "cache debt" does your solution need to deal with?

Look; we're in my Decennial Series. I'm giving you my hard earned advice. 

Just stay away from any form of querying. There are other and better ways. I'll give you a few pointers; most of what I'm highlighting here is already well covered by blog posts everywhere (far from all of them, mine). Seek them out, please. Please. Next time you feel, you need to query for your content.

What to do instead


Information Architecture

The first and most important thing to keep in mind, is how you organise your content. Try and keep related items close together and organise them in a meaningful deep tree hierarchy. So for, as an example, News Articles, do this:

/sitecore/content/news/2016/04/[your articles here]

Which gives you the option to do:

var root = Sitecore.Context.Database.GetItem("/sitecore/content/news/2016/04");
var newsArticles = root.GetChildren();

And unless you have an extraordinarily "news'y" site, I assure you this will be better than the Query equivalent which might look something like:

var newsArticles = Sitecore.Context.Database.SelectItems("/sitecore/content/news/*[@newsDate > '2016-04-01' & @newsDate < '2016-05-01'])

Don't be overly worried about reading in 10-20 items in a .GetChildren() call; be worried about reading in 2000 items and discarding 1980 of them.

Code Smart(er)

There are often multiple ways of achieving something. This certainly holds true for Sitecore as well. One of the little known gems is the Sitecore LinkDatabase. Ever tried deleting an item, and have Sitecore pop up and tell you this:

















How does Sitecore know this?  And you probably guessed it already. The LinkDatabase. The SLDB is easily worthy of a post of its own, but in essence it's this:

A table maintained by Sitecore, where all reference field references (!) are stored. This allows Sitecore to keep track of inter-Item relations so it can pop this warning to you.

Since we're doing the low-level thing today, here's what it looks like:



















Turns out, this is an excellent tool in many other situations as well. I think my Listing "Related Articles" using LinkDatabase was the first time I blogged about it; and I regularly dust off SLDB and pick it up from my tool shelf even to this day.

A common scenario that often comes up, is "I need to find all News articles, but they're spread out all over my content tree. Silly editors". Well SLDB has a fix for that; although here it is used indirectly.









In case you're wondering; .GetUsageIDs() uses SLDB internally to achieve it's result.

I highly recommend you read up on the Sitecore LinkDatabase. It can really make your life easier in a lot of situations, where you would otherwise feel tempted to start Sitecore Querying.

Index your content

And finally, but also most importantly, use an index for every single Sitecore solution you ever do. I mean it. Seriously.

Want to find all News Articles in a specified date range?  use an index.
Want to find all News Articles in the "Business News" category, sorted by Author? use an index.

Even with all the tricks I have show you here, sooner or later you will come to a situation where the content cannot be organised any differently, where LinkDatabase cannot help you as we're dealing with content and not references - or flat out, we're dealing with so much content that we just have to narrow down the dataset to be able to work with it.

Fortunately, I've already given the answer to these problems. Use an index.

There are quite literally dozens and dozens of blog posts out there, on how to set up and use Sitecore ContentSearch (which is where I recommend you start out). It takes so little effort to get going, it's active out of the box (using Lucene as it's provider), there quite simply is not a single valid excuse not to use it.

I know it's "another thing to learn". This one is not optional however; you really cannot go around it. You may feel you're doing just fine and your Sitecore Queries run fine. Great. Give yourself a nice "It works on my machine" sticker, and then get those indexes going ;-)

Seriously.

Until next time :-)