For educational purposes, I'd like to write an application on a Linux environment that can process keyboard events and draw graphics without huge dependencies like X or SDL. I presume that this must be possible, because X and SDL are just programs themselves, so they must rely on other methods inherent to the environment. Is this understanding correct?

If so, where might I learn to write such a program? My limited experience tells me that it would involve making calls to the kernel, and/or writing to special files; however, I haven't been able to find any tutorials on the matter (I am not even sure what to Google).

Also, in case it is relevant, I am running Debian Squeeze on Virtualbox. I have used a netinst cd without networking, so there isn't much installed on it currently. I will install gcc, but I am hoping I can get by with nothing more.