I often come across very impressive mods for PC games that don't have an official editor, other development tools or its source code publicly available. (Take this amazing Multiplayer mod for Just Cause 2, for example.)
How do you go about creating mods for such games?
I'm not talking about replacing the odd texture or 3D model—that sort of thing seems fairly easy given tools to pry them out of game files and put them back in—but more along the lines of adding game behavior. (Tweaking settings files also doesn't count.)

Note that I'm not asking "how to create a mod", I just want to know where to start or where to go to learn.