Calling managed code from unmanaged win32 assembly dll - crash

Posted by JustGreg on Stack Overflow See other posts from Stack Overflow or by JustGreg
Published on 2012-06-28T10:56:50Z Indexed on 2012/06/29 15:16 UTC
Read the original article Hit count: 198

Filed under:
|
|

I'm developing a serial port dll in win32 assembly (MASM32). It has its own thread checking multiple events and at a specified buffer treshold it'd notify the managed main application by calling a callback function. It just a call with no arguments/return value.

At startup the main application stores the callback function's address by calling a function in the dll:

pCallBackFunction dd 0

SetCallBackPointer proc pcb:DWORD

mov eax, pcb
mov pCallBackFunction, eax
call DWORD ptr pCallBackFunction ; verify it immediately
ret

SetCallBackPointer endp

The upper function immediately calls back the managed application callback routine for verification purposes. It is working fine. However, when I place the call instruction to other functions in the dll it crashes the application. It doesn't matter if the call is in a simple function or in the threadproc of the dll. For example:

OpenPort proc pn:byte,br:dword, inputbuffersize: dword, outputbuffersize:dword, tresholdsize: dword
LOCAL dcb: DCB
LOCAL SerialTimeOuts: COMMTIMEOUTS
call DWORD ptr pCallBackFunction
xor eax, eax
mov al, pn
mov [com_port+3],al

etc. etc.

will crash at call DWORD ptr pCallBackFunction always. Since I call SetCallBackPointer first to store a valid address in pCallBackFunction, it should have a valid address.

My managed app is written in C# and the relevant part is:

public partial class Form1 : Form
{
    public delegate void CallBackDelegate();
    public static CallBackDelegate mydelegate;

    [DllImport("serialport.dll")]
    private static extern void SetCallBackPointer(CallBackDelegate Delegate);

    [DllImport("serialport.dll")]
    public static extern int OpenPort(byte com, uint br, uint inbufsize, uint outbufsize, uint treshsize);



    public Form1() 
    {

        InitializeComponent();



        mydelegate =new CallBackDelegate(CallbackFunction);

        SetCallBackPointer(mydelegate);

        unsafe
        {
            int sysstat;
            int hResult;

            hResult = OpenPort(Convert.ToByte('5'), 9600, 306, 4, 4);
        }
        }

        public static void CallbackFunction()
        {
             MessageBox.Show( "CallBack Function Called by Windows DLL");
        }

The VS debugger reported that the dll had tried to read/write from/to a protected memory address. But when calling SetCallBackPointer there is no such problem. What am I doing wrong here?

Any tips would be great!

© Stack Overflow or respective owner

Related posts about .NET

Related posts about dll