I am a Unix/Linux/Windows Server noob. So, keep that in mind before you throw your stones at my glass house. :P

I have a Windows Server 2008 R2 machine that's acting as domain controller, Server A. It's also running a DNS server. I have a Red Hat Enterprise Linux Server 5.3, Server B that is intended for mail server.

In order for the mail delivery to happen, I understand that I have to set an MX record on Server A and point it to Server B. Well, I did. I manually added a host name on Server A and pointed to Server B's IP address. Then I added an MX record and pointed it to the host name. That didn't do the trick. After taking the above steps, I used the "dig" command on Server B to lookup the MX record coming back from Server A and it wasn't what I was expecting.

What am I doing wrong here?

I have noticed that... my Windows machines that are joined to the domain (Server A) are listed under the host names. The machines that are not joined to the domain are not list. This is fine, I am not worried about this. What does concern me, do I have to join the Server B to domain in order for Server A to recognize as a valid host and forward the MX properly? If so, some simple steps on how to join Server B to the domain would also help.