So I was beating my head against a wall for a while before it dawned on me. I have some code that saves a list of names into a text file ala ...

System.IO.File.WriteAllLines(dlg.FileName, this.characterNameMasterList.Distinct().ToArray());

The character names can contain special characters. These names come from the wow armory at www.wowarmory.com There are about 26000 names or so saved in the .txt file.

The names get saved to the .txt file just fine. I wrote another application that reads these names from that .txt file using this code

// download the names from the db
var webNames = this.DownloadNames("character");

// filter names and get ones that need to be added to the db
var localNames = new List<string>(System.IO.File.ReadAllLines(dlg.FileName));
foreach (var name in webNames)
{
    if (localNames.Contains(name.Trim())) localNames.Remove(name);
}

return localNames;

... the code downloads a list of names from my website that are already in the db. Then reads the local .txt file and singles out every name that is not yet in the db so it can later add it. The names that get read from the .txt file also get read just fine with no problems.

The problem comes in when removing names from the localNames list. localNames is a List type. As soon as localNames.Remove(name) gets called any names in the list that had special characters in them would get corrupted and be converted into ? characters. See for screen cap http://yfrog.com/12badcharsp

So i tried doing it another way using ...

 // download the names from web that are already in the db
var webNames = this.DownloadNames("character");

// filter names and get ones that need to be added to the db
var localNames = new List<string>(System.IO.File.ReadAllLines(dlg.FileName));
int index = 0;
while (index < webNames.Count)
{
    var name = webNames[index++];

    var pos = localNames.IndexOf(name.Trim());
    if (pos != -1) localNames.RemoveAt(pos);
}

return localNames;

.. But using localNames.RemoveAt also corrupts the items in the list converting special characters into ?.

So is this a known bug with the List.remove methods? Does anyone know? Has anyone else had this problem? I also used .NET Reflector to disassemble/inspect the list.remove and list.RemoveAt code and it appear to be calling some external Copy function.

Aside from the fact that this is prob not the best way to get a unique list of items from 2 lists am I missing something or should be aware of when using the List.Remove methods ?

I am running windows 7 vs2010 and my app is set for .net 4 (no client profile )