How to create a Windows program that works both as a GUI and console application

It would be very nice to be able to make Windows applications that can act as either a GUI or console application depending on how they are used (i.e. act as a GUI application if double clicked in Windows Explorer or as a console application if called from a cmd.exe window).

Unfortunately the way Windows work, each exe application has field in the PE header that specifies which subsystem it should run under. This is set in Visual Studio by using one of the linker SUBSYSTEM option (i.e. Windows or Console). The subsystem is used by the Windows kernel to set up the execution environment for the application. If the application is built using SUBSYSTEM/Console then the kernel will connect the application to the parent console and the application’s stdout, stderr and stdin will be redirected to the parent console. If the program is built as a GUI application then the application detaches from the parent console and all output to stdout and stderr are are lost – basically the program runs, but doesn’t output anything to the parent console window.

People have attempted various hacks over the years to solve this problem. One solution proposed was to compile the program as a Windows application and then edit the PE header to mark the program as using the Console subsystem. The downside of this approach is it flashes a Console window on the screen when run as a GUI application that looks pretty unprofessional. The second hack commonly used is to create two separate binaries, for example, a myuselessprogram.com and myuselessprogram.exe. The .com version is built as a Console application while the .exe is built as Windows application. When you run myuselessprogram the Windows probing rule runs the .com version first and if there is nothing on the command line then the .com version calls the .exe Windows version. While both these approaches work, they are to say less than ideal.

A better approach is use the WINAPI AttachConsole function to attach the application to the parent console and then redirect stdout, stdin and stderr back to the parent console. This actually works very well except that when the application exits the parent console can’t detect this and hence release the command prompt. The end result is the parent console just sits there until the user presses the “enter” key.

There is no really elegant solution to this problem, but as applications can simulate the keyboard being used, a simple solution is to call the SendInput API function with the “enter” key just before the application exits. This simulates the user pressing the enter key and hence releases the command prompt.

To show how this approach works I have written a small test application (see below). The main limitations is that AttachConsole is only available on Windows XP and above. It does works under Cygwin which is nice.

Update. I have added a check to make sure that the console window is in focus before sending the enter key. It is a good check to make if you are running your console program in a background script or else you will end up with lots of “enter” key presses in whatever application you actually have in focus – lots of fun if you are working on the documentation while a script runs in the background :)

Update 2. Contrary to what has been posted on Stack Overflow this approach works with STDIN too. I don’t have any need to capture STDIN with my application (just the command line parameters), but all you need to do to capture stdin is treat STDIN as is done for STDOUT (i.e just redirect STDIN to the console) in the attachOutputToConsole function.

Update 3. This is MIT licensed if this is important to you :)

Update 4. Microsoft has broken the old approach I was using in VS2015. I have updated the code to use a different way of attaching STDOUT and STRERR to the parent console. This code works with VS2015 and all earlier compliers I was able to test (VS2013, VS2008, VS2005).

/*

 Copyright (c) 2013, 2016 Daniel Tillett
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions are met:
 
 1. Redistributions of source code must retain the above copyright notice, this
 list of conditions and the following disclaimer.
 2. Redistributions in binary form must reproduce the above copyright notice,
 this list of conditions and the following disclaimer in the documentation
 and/or other materials provided with the distribution.
 
 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
 ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
 WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
 DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
 ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
 (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
 LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
 ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

*/

#define WINVER 0x0501 // Allow use of features specific to Windows XP or later.
#define _WIN32_WINNT 0x0501
#define WIN32_LEAN_AND_MEAN
#include "windows.h"
#include "io.h"
#include "fcntl.h"
#include "stdio.h"
#include "stdlib.h"
#pragma comment(lib, "User32.lib")

// Attach output of application to parent console
static BOOL attachOutputToConsole(void) {
HANDLE consoleHandleOut, consoleHandleError;

if (AttachConsole(ATTACH_PARENT_PROCESS)) {
  // Redirect unbuffered STDOUT to the console
  consoleHandleOut = GetStdHandle(STD_OUTPUT_HANDLE);
  if (consoleHandleOut != INVALID_HANDLE_VALUE) {
    freopen("CONOUT$", "w", stdout);
    setvbuf(stdout, NULL, _IONBF, 0);
    }
  else {
    return FALSE;
    }
  // Redirect unbuffered STDERR to the console
  consoleHandleError = GetStdHandle(STD_ERROR_HANDLE);
  if (consoleHandleError != INVALID_HANDLE_VALUE) {
    freopen("CONOUT$", "w", stderr);
    setvbuf(stderr, NULL, _IONBF, 0);
    }
  else {
    return FALSE;
   }
  return TRUE;
  }
//Not a console application
return FALSE;
}

// Send the "enter" to the console to release the command prompt 
// on the parent console
static void sendEnterKey(void) {
 INPUT ip;
 // Set up a generic keyboard event.
 ip.type = INPUT_KEYBOARD;
 ip.ki.wScan = 0; // hardware scan code for key
 ip.ki.time = 0;
 ip.ki.dwExtraInfo = 0;

 // Send the "Enter" key
 ip.ki.wVk = 0x0D; // virtual-key code for the "Enter" key
 ip.ki.dwFlags = 0; // 0 for key press
 SendInput(1, &ip, sizeof(INPUT));

 // Release the "Enter" key
 ip.ki.dwFlags = KEYEVENTF_KEYUP; // KEYEVENTF_KEYUP for key release
 SendInput(1, &ip, sizeof(INPUT));
}

int WINAPI WinMain(HINSTANCE hInstance, 
HINSTANCE hPrevInstance, 
PSTR lpCmdLine, 
INT nCmdShow) {
int argc = __argc;
char **argv = __argv;
UNREFERENCED_PARAMETER(hInstance);
UNREFERENCED_PARAMETER(hPrevInstance);
UNREFERENCED_PARAMETER(lpCmdLine);
UNREFERENCED_PARAMETER(nCmdShow);
BOOL console;
int i;

//Is the program running as console or GUI application
console = attachOutputToConsole();

if (console) {
    // Print to stdout
    printf("Program running as console application\n");
    for (i = 0; i < argc; i++) {
         printf("argv[%d] %s\n", i, argv[i]);
         }

    // Print to stderr
    fprintf(stderr, "Output to stderr\n");
    }
else {
    MessageBox(NULL, "Program running as Windows GUI application",
    "Windows GUI Application", MB_OK | MB_SETFOREGROUND);
}

// Send "enter" to release application from the console
// This is a hack, but if not used the console doesn't know the application has
// returned. The "enter" key only sent if the console window is in focus.
if (console && (GetConsoleWindow() == GetForegroundWindow())){
    sendEnterKey();
    }
return 0;
}
36 comments on “How to create a Windows program that works both as a GUI and console application
  1. Incredible job! First real solution I’ve even seen.

    All previous “solutions” I’ve seen — and it’s no small number — were “2nd best” solutions that were “ok” but not actually dealing with the issue head-on.

    Very nice to see this solution!

    (I didn’t really think it would work right when I tried it. Then I threw several different scenarios at it and it passed them all! It’s great!)

  2. Nice solution for folks that just want stdout and stderr, but when run as a console app, stdin is not captured from the console, so it’s not true duality. Doesn’t work for my needs. Do you have a way of capturing the console input and sending to stdin for console apps?

  3. I have not tried, but I suspect that it might be possible to get stdin in a similar way to the way I have been getting stdout and stderr. I might play around and see if I can get stdin too.

  4. I’m not sure I understand. If FreeConsole() works like intended there is no need to emulate a key strike. Native function calls have to super-cede gimmick key returns. But to each his own I suppose.

  5. I see the issue now, thank you. As always, the documentation here is clearly lacking and only experience can provide the answer.

  6. Well, in this case I’d say both if I am being truthful.
    “This actually works very well except that when the application exits the parent console can’t detect this and hence release the command prompt.”

    But why exactly? If I read your blog after reading the MSDN I am still unsure of the actual crux of the issue with AttachConsole()/FreeConsole().

    Maybe because AttachConsole() is a misnomer, Red Herring, at best.

    What made it click for me, calling said app with `start /WAIT saidapp.exe` from a cmd.exe prompt, now FreeConsole() works as expected and documented.

  7. Works fine, but if I want to redirect the output to files with:

    myprog.exe > out.txt 2> err.txt

    the files are empty and the output goes to the console

    any idea?

  8. I used your code and get some weird results when compiling with VS2015. When running the a program with this code in a Developer Command Prompt, it fails at the second _open_osfhandle() call for stderr. The function call returns -1 instead of a valid file descriptor. Strange enough the call to _open_osfhandle() succeeds the first time when the file descriptor for stdout is needed.

    On the same machine in a normal command prompt the same program succeeds on both _open_osfhandle() calls.

    Secondly, although in a normal command prompt both the stdout and stderr is capture, the printf example statement doesn’t send any output to the console. The program exits without error though and correctly starts the windowed version of the application if it is started through Explorer or a shortcut.

    Which compiler did you use for testing and any ideas how I can further dig to find the root course of this problem?

  9. Hi Lammert

    I have not tried it with VS2015 so it is quite possible that MS has changed something under the hood. I will take a look myself into this issue.

    Update. Yes Microsoft has change something in VS2015 that breaks the approach I was using to attach to the parent console. I have updated the code and it now works with VS2015.

    Thanks for the heads up on this.

  10. Thanks for your quick reply. I will test the new code and see if it still causes any problems.

    Is this new code backwards compatible with older compilers?

  11. The method in this article and the example code given in the body are more concise than the stackoverflow article mentioned in the comments, but the differences between Windows App and Console App are not well explained. 1) I/O redirection is performed by cmd.exe for Console Apps, but not for Windows Apps. 2) The default behavior for cmd.exe running a Console App is to run it as a child process and wait for it to finish. The default behavior for cmd.exe running a Windows App is to run it without waiting for it to finish. The workaround is to run the Windows App with the command START /WAIT. Running the Windows App with START /WAIT avoids the cmd.exe command prompt being mixed in with the first line of output of the Windows App, and it avoids the need for SendKeys/SendInput at the end of the program.

  12. The code in this article demonstrates that a program compiled in Visual C will send output to stdout after it has been redirected/attached to the parent console. Further testing reveals that the Windows App will send output to stdout without error even if stdout is not attached to the parent console. As a consequence, it is possible to have a Console based jacket routine that runs the Windows App as a child process and explicitly redirects I/O to pipes that are captured and then echoed to the console by the console app. A Visual C method to do this would be to use CreateProcess(). A method that requires a log less coding to run the Windows App as a child process is to run it using cscript.exe and a jacket routine written in VBScript that uses WScript.Shell and the .Exec() method. However, the child process method is prone to mixing StdOut and StdErr output in a different order from the method in this article.

  13. Stefan yes the START /WAIT approach is another solution for avoiding the need for the SendKeys step before exit. The only issues is if the user remembers to do this as it is not something that people expect when using a cmd program.

  14. The only problem I got so far is when I run the program many times from batch script I get a lot of emulated carriage returns when script is done.

  15. Brilliant, thanks so much for posting a solution that also works under VS2015 ( unlike another one I found recently ). Great work.

  16. Thank you so much for your code, It was really helpful as I need to do almost the same since I have GUI and Batch (cmd line) modes.

    However, I also wanted to allow the program to be able to redirect the output to files if needed or allocate console if parent console doesn’t exist (run from a shortcut for example).

    Here is my code for reference:

    static bool attachOutputToConsole() {
    /* requires
    Running app with -batch
    gurantees
    return false if output is redirected to a file
    returns true otherwise by:
    – attaching to the parent console if it exists
    – creating a new console
    which is required to show license and batch messages
    */
    HANDLE outHandle = GetStdHandle(STD_OUTPUT_HANDLE);
    if(!(outHandle = NULL)) {
    if (GetFileType(outHandle) == FILE_TYPE_DISK){
    // if output has been directed to a file, then do nothing
    return false;
    }
    }

    // If output not directed to a file, try attaching to parent console or create a new one
    if(!AttachConsole( ATTACH_PARENT_PROCESS )){
    AllocConsole();
    }

    // redirect stdout, stdin to the console, either new or attached
    freopen(“CONOUT$”, “w”, stdout);
    freopen(“CONOUT$”, “w”, stderr);
    setvbuf(stdout, NULL, _IONBF, 0);
    setvbuf(stderr, NULL, _IONBF, 0);
    return true;
    }

    // Main function
    int PASCAL _tWinMain(a1, a2, a3, a4)
    HINSTANCE a1;
    HINSTANCE a2;
    LPTSTR a3;
    int a4;
    {
    int i = 0;
    bool console = false;

    extern int __argc;
    extern LPTSTR *__targv;

    Argc = __argc;
    Argv = __targv;

    HandleInstance = a1;
    HandlePreviousInstance = a2;
    lpCmdLine = a3;
    nShowCmd = a4;

    // check if it’s run using -batch option, then try to attach it to a console or create one.
    if (Argc >=2){
    for(i=0; i<Argc; i++){
    if(lstrcmp(Argv[i], L"-batch") == 0)
    console = attachOutputToConsole();
    }
    }

    // App code,
    mainInitialize();
    mainStartUp();
    mainRelease();

    if (console && (GetConsoleWindow() == GetForegroundWindow())){
    // just to get the prompt again
    sendEnterKey();
    fclose(stdout);
    fclose(stderr);
    FreeConsole();
    }

    return(ExitStatus);
    }

  17. My solution works also if console window is not currently active, by using GetConsoleWindow I get the handle of the console window, then I simply use PostMessage to send the WM_KEYUP event for the VK_RETURN key:

    h := GetConsoleWindow;
    if IsWindow(h) then
    PostMessage(h, WM_KEYUP, VK_RETURN, 0);

Leave a Reply

Your email address will not be published. Required fields are marked *