Use __cdecl rather than __stdcall for icalls on Windows 32-bit

Mono assumes the same calling convention for icalls as the default P/Invoke
calling convention which is __stdcall on Windows. But none of the icalls are
marked with the required __stdcall attribute on Windows so there will be a
mismatch between the caller and callee.

This is the reason why the System.Web.HttpUtilityTest.JavaScriptStringEncode()
test crashes on Windows 32-bit. It generates a call to an icall
(char.ToString() inlines to new string(char,int)) which is assumed to be
__stdcall while it's actually __cdecl. The caller will decrement ESP as if the
callee was __stdcall resulting in ESP decreasing on each iteration. After a
couple of thousand iterations in the loop in that test the stack guard page is
hit and a stack overflow occurs.

diff --git a/mono/metadata/loader.c b/mono/metadata/loader.c
index 9e09748c344ba714c8e4002ed87f1531f690eb2d..0309ce8404e1593cd2f88e5b84af17ac0a61c4f7 100644 (file)
--- a/mono/metadata/loader.c
+++ b/mono/metadata/loader.c
@@ -2434,9 +2434,16 @@ mono_method_signature_checked (MonoMethod *m, MonoError *error)
                mono_error_set_method_load (error, m->klass, m->name, "generic_params table claims method has generic parameters, but signature says it doesn't for method 0x%08x from image %s", idx, img->name);
                return NULL;
        }
-       if (m->iflags & METHOD_IMPL_ATTRIBUTE_INTERNAL_CALL)
+       if (m->iflags & METHOD_IMPL_ATTRIBUTE_INTERNAL_CALL) {
                signature->pinvoke = 1;
-       else if (m->flags & METHOD_ATTRIBUTE_PINVOKE_IMPL) {
+#ifdef HOST_WIN32
+               /*
+                * On Windows the default pinvoke calling convention is STDCALL but
+                * we need CDECL since this is actually an icall.
+                */
+               signature->call_convention = MONO_CALL_C;
+#endif
+       } else if (m->flags & METHOD_ATTRIBUTE_PINVOKE_IMPL) {
                MonoCallConvention conv = (MonoCallConvention)0;
                MonoMethodPInvoke *piinfo = (MonoMethodPInvoke *)m;
                signature->pinvoke = 1;