Improve detecting if a handle is a console on Windows.

Use GetConsoleMode. If it succeeds (returns non-zero) then the handle is a
console. If it fails (returns zero) then the handle is not a console.

The old check, (GetFileType (handle) == FILE_TYPE_CHAR) assumes that a handle
of type FILE_TYPE_CHAR is a console. This falsely reports a handle directed to
NUL as a console.

Fixes https://bugzilla.xamarin.com/show_bug.cgi?id=12752

diff --git a/mono/metadata/console-win32.c b/mono/metadata/console-win32.c
index 6b940f3edfb4e85920a526d0cc56a6fa868e273f..7e0d598a2820635795a691bd69ca99f0869e5264 100644 (file)
--- a/mono/metadata/console-win32.c
+++ b/mono/metadata/console-win32.c
@@ -46,7 +46,8 @@ ves_icall_System_ConsoleDriver_Isatty (HANDLE handle)
 {
        MONO_ARCH_SAVE_REGS;
 
-       return (GetFileType (handle) == FILE_TYPE_CHAR);
+       DWORD mode;
+       return GetConsoleMode (handle, &mode) != 0;
 }
 
 MonoBoolean