Your VB.NET program must list the functions you wish to call as exportable (so that they are in the EXE's export table). This is a bit tricky in VB.NET (I didn't think it was possible until I found this article): http://www.codeproject.com/KB/vb-interop/interopvb6netcexporthook.aspx
Usually only DLLs have export tables, but there is no reason an EXE can't have one also and be used as if it were a DLL.
Then, you can use LoadLibrary() to open the executable and GetProcAddress() to get pointers to the functions you wish to call. Make sure to pay attention to the article I linked for you.
The function pointer should be declared with the WINAPI calling convention:
Also keep in mind that your exported functions should not take fancy object types as arguments or return them -- keep them to things like integers and characters and char* strings and the like.