[io-layer] Correctly detect a PE32+ assembly as managed when starting process

We weren't checking whether a PE file is in PE32 or PE32+ format,
causing us to use the PE32 offsets on the newer format as well.

This means that using Process.Start() on a managed assembly didn't
always work when a PE32+ file was in play. Such a file is created
when targeting x64 e.g. via the -platform:x64 csc.exe/mcs option.

The reason why we probably didn't notice until now is that in an
assembly produced by mcs there happens to be some data at the PE32
CLR header offset even in a PE32+ file, causing the check to "work".

This doesn't apply to csc.exe/roslyn though and so we couldn't
execute those assemblies via Process.Start() even though they're
perfectly managed. Since the .NET Core project.json toolchain
produces x64-targeted assemblies by default more people ran into
the issue trying to run those on Mono: https://github.com/cake-build/cake/issues/1247

Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=44937

PE/COFF spec at https://msdn.microsoft.com/en-us/library/windows/desktop/ms680547(v=vs.85).aspx

diff --git a/mono/io-layer/processes.c b/mono/io-layer/processes.c
index dcce3de93ebda82a4f26ccee44ca7ddbf92d1c4c..fd55cd74314bcad2d4029a28d27bcec1c3fc8b9e 100644 (file)
--- a/mono/io-layer/processes.c
+++ b/mono/io-layer/processes.c
@@ -412,10 +412,10 @@ is_managed_binary (const char *filename)
        off_t new_offset;
        unsigned char buffer[8];
        off_t file_size, optional_header_offset;
-       off_t pe_header_offset;
+       off_t pe_header_offset, clr_header_offset;
        gboolean managed = FALSE;
        int num_read;
-       guint32 first_word, second_word;
+       guint32 first_word, second_word, magic_number;
        
        /* If we are unable to open the file, then we definitely
         * can't say that it is managed. The child mono process
@@ -480,13 +480,34 @@ is_managed_binary (const char *filename)
        if ((num_read != 2) || ((buffer[0] | (buffer[1] << 8)) < 216))
                goto leave;
 
+       optional_header_offset = pe_header_offset + 24;
+
+       /* Read the PE magic number */
+       new_offset = lseek (file, optional_header_offset, SEEK_SET);
+       
+       if (new_offset != optional_header_offset)
+               goto leave;
+
+       num_read = read (file, buffer, 2);
+
+       if (num_read != 2)
+               goto leave;
+
+       magic_number = (buffer[0] | (buffer[1] << 8));
+       
+       if (magic_number == 0x10B)  // PE32
+               clr_header_offset = 208;
+       else if (magic_number == 0x20B)  // PE32+
+               clr_header_offset = 224;
+       else
+               goto leave;
+
        /* Read the CLR header address and size fields. These will be
         * zero if the binary is not managed.
         */
-       optional_header_offset = pe_header_offset + 24;
-       new_offset = lseek (file, optional_header_offset + 208, SEEK_SET);
+       new_offset = lseek (file, optional_header_offset + clr_header_offset, SEEK_SET);
 
-       if (new_offset != optional_header_offset + 208)
+       if (new_offset != optional_header_offset + clr_header_offset)
                goto leave;
 
        num_read = read (file, buffer, 8);