Intersection() and Except() is too slow with large collections of custom objects

Posted by Theo on Stack Overflow See other posts from Stack Overflow or by Theo
Published on 2013-11-01T20:47:14Z Indexed on 2013/11/01 21:53 UTC
Read the original article Hit count: 163

Filed under:
|
|
|
|

I am importing data from another database.

My process is importing data from a remote DB into a List<DataModel> named remoteData and also importing data from the local DB into a List<DataModel> named localData.

I am then using LINQ to create a list of records that are different so that I can update the local DB to match the data pulled from remote DB. Like this:

var outdatedData = this.localData.Intersect(this.remoteData, new OutdatedDataComparer()).ToList();

I am then using LINQ to create a list of records that no longer exist in remoteData, but do exist in localData, so that I delete them from local database.

Like this:

var oldData = this.localData.Except(this.remoteData, new MatchingDataComparer()).ToList();

I am then using LINQ to do the opposite of the above to add the new data to the local database.

Like this:

var newData = this.remoteData.Except(this.localData, new MatchingDataComparer()).ToList();

Each collection imports about 70k records, and each of the 3 LINQ operation take between 5 - 10 minutes to complete. How can I make this faster?

Here is the object the collections are using:

internal class DataModel
{
        public string Key1{ get; set; }
        public string Key2{ get; set; }

        public string Value1{ get; set; }
        public string Value2{ get; set; }
        public byte? Value3{ get; set; }
}

The comparer used to check for outdated records:

class OutdatedDataComparer : IEqualityComparer<DataModel>
{
    public bool Equals(DataModel x, DataModel y)
    {
        var e =
            string.Equals(x.Key1, y.Key1) &&
            string.Equals(x.Key2, y.Key2) && (
                !string.Equals(x.Value1, y.Value1) ||
                !string.Equals(x.Value2, y.Value2) ||
                x.Value3 != y.Value3
                );
        return e;
    }

    public int GetHashCode(DataModel obj)
    {
        return 0;
    }
}

The comparer used to find old and new records:

internal class MatchingDataComparer : IEqualityComparer<DataModel>
{
    public bool Equals(DataModel x, DataModel y)
    {
        return string.Equals(x.Key1, y.Key1) && string.Equals(x.Key2, y.Key2);
    }

    public int GetHashCode(DataModel obj)
    {
        return 0;
    }
}

© Stack Overflow or respective owner

Related posts about c#

Related posts about Performance