yes, things like this can happen. It can go the other way too. Depends on the code.
50% faster is an unusually big difference, though. Did you compile it optmized in both? Its easy to do a debug compile by mistake...
This is possible for some specific programs. On x64, addresses (pointers) and some instructions use more memory; this can adversely affect processor cache usage. Programs that use a very large degree of indirection (many many accesses through pointers/references) are probably more prone to this slowdown when ported to x64.
I routinely have Visual Studio (2019 Community) compile console and Win32 apps as x86 and x64. Most times there is no speed difference between the two. Sometimes the x86 app can be faster, more often the x64 app will be faster when there is a difference in execution speed.
Debug vs. Release can be a major factor, as well as what optimization level is set.
The speed differential is more common with Win32 apps than console ones, but it is still a rare occurrence.
The difference in speed can be really noticeable when the CPU is under a heavy load from other apps/services running.
Another potential bottle-neck is dynamic linking vs. static linking. Dynamic DLL loading can take some time as the library is needed. VS defaults to dynamic linking so the .exe file is smaller than one that is statically linked.
It isn't just VS. MinGW/Code::Blocks can also have execution differential issues.