A Windows GUI and Console Application in One EXE, Revisited

Penleigh Boyd - Springtime (1921), golden wattle overhanging calm river water

Back in 2013 I wrote a post on making a Windows application work as both a GUI and a console program. The idea was simple: build a Windows-subsystem EXE so that double-clicking it in Windows Explorer doesn’t flash an ugly console window on the screen, then call AttachConsole(ATTACH_PARENT_PROCESS) to wire up the output if someone runs it from a command prompt.

I still like the single-EXE approach (no companion .com binary and no separate launcher), but looking back at my old code there are a few things that were broken or that I would do very differently now. The biggest problem with my 2013 code was that it clobbered redirected input and output, and the worst hack was simulating an Enter key press on exit – which I have now removed altogether.

Here is what was wrong with the original version and how the new version fixes it. The result is still a single Windows-subsystem EXE that can open a GUI, write to a parent console, or read and write to files and pipes when there is no console at all. There is one catch worth stating upfront: attaching to a parent console does not make the command shell wait for the program to finish.

Preserve Redirection Before Attaching to the Console

My original code simply called freopen("CONOUT$", "w", stdout) and freopen("CONOUT$", "w", stderr) after attaching to the parent console. This gets text onto the screen, but it also wipes out any file redirection or pipes the user asked for on the command line. For example, if you ran this command to save the output to a file:

hybrid-demo.exe --help > help.txt 2> errors.txt

Opening CONOUT$ for both streams forced the output straight back onto the console instead of writing it to help.txt and errors.txt. The same thing happened with pipes. Simply checking whether a parent console exists is not enough.

To fix this, you have to save a duplicate of each valid standard handle before calling AttachConsole. The order matters because calling AttachConsole overwrites the process’s standard-handle table (as buried in Microsoft’s GetStdHandle documentation). The new helper checks stdin, stdout and stderr one by one. If a stream already points to a file, a pipe or NUL, it leaves it alone. It only connects a stream to CONIN$ or CONOUT$ if that stream is actually missing and there is a console to connect it to.

This means deliberate output to NUL is preserved rather than treated as a missing handle, and mixed redirection works as expected – you can redirect stdout to a file while stderr still prints to the console.

A Console and an Output Stream Are Not the Same Thing

In the 2013 code, attachOutputToConsole() returned a single BOOL: either you had a console or you didn’t. Unfortunately, that throws away information you actually need.

A process can have no console at all (for example, when launched by a background runner or test script) and still have perfectly good stdout and stderr pipes. Conversely, AttachConsole can succeed while one of the C streams fails to connect. There is also a trap in AttachConsole: if the process is already attached to a console, it fails with ERROR_ACCESS_DENIED, which is very different from having no parent console (ERROR_INVALID_HANDLE).

Instead of returning a boolean, the new helper returns a small state structure with the attachment result and a separate status for each stream. Your application code can then check for what it actually cares about:

const hc_state *io = hc_initialize();
if (io->stream[1].ready) {
    puts("Text output is available");
}

The helper itself doesn’t decide whether to open a GUI. The demo program opens the GUI when started with no arguments, no parent console and no output destination. Passing --gui forces the GUI, while --cli forces command-line mode. If a required output stream is missing in CLI mode, the program exits with an error code rather than popping up a message box in the middle of someone’s background script.

The CRT File Descriptor Trap

Windows has three separate layers for standard I/O: Win32 HANDLE values, C runtime (CRT) integer file descriptors (0, 1 and 2), and stdio FILE* streams (stdin, stdout and stderr). Fixing one layer doesn’t automatically fix the others.

In an MSVC GUI application, there is no console at startup, so _fileno(stdout) can return -2 (_NO_CONSOLE_FILENO), as noted in Microsoft’s _fileno documentation.

An obvious fix is to reopen stdout on NUL and duplicate your saved handle onto that descriptor. When I tested this properly, I hit a nasty bug: stdout could end up bound to file descriptor 4 instead of descriptor 1. Calls to printf() worked and text appeared on the screen, so a naive test passed, but any code calling _write(1, ...) failed completely.

The new code repairs the missing standard descriptors first so that stdin, stdout and stderr are guaranteed to sit on descriptors 0, 1 and 2, and only then connects the saved handles using _open_osfhandle and _dup2. It also updates the Win32 standard handle table to match, so calling WriteFile(GetStdHandle(STD_OUTPUT_HANDLE), ...) goes to the exact same place as printf().

You do have to be careful with handle ownership here. Once _open_osfhandle succeeds, the C runtime owns the underlying Win32 HANDLE – if you close both the CRT file descriptor and the original HANDLE, you will close the same handle twice. The helper duplicates handles explicitly so each layer only closes what it owns.

Call hc_initialize() once at the very start of WinMain, before touching stdio or starting any worker threads. Calling it a second time simply returns the saved state.

Only Disable Buffering on a Real Console

My old helper turned off buffering on stdout using setvbuf(stdout, NULL, _IONBF, 0) as soon as it attached. Having text appear immediately is what you want on an interactive console, but it is a terrible idea when stdout is redirected to a file or a pipe, as it forces a system call on every small write.

The new version only disables buffering when stdout is actually connected to a console. When stdout is redirected to a file or pipe, it keeps normal CRT buffering, and stdin and stderr stay at their defaults. The one thing to watch out for in tests is that if you merge stdout and stderr into the same pipe or file, printf and fprintf(stderr, ...) won’t necessarily appear in the exact order you called them unless you flush stdout first.

Why I Removed the Simulated Enter Key Hack

In my 2013 post I used SendInput to send a fake Enter keystroke just before the program exited so cmd.exe would redraw the command prompt. Even after I added a GetConsoleWindow() == GetForegroundWindow() check to stop it firing Enter keys into whatever document I was typing in while a script ran in the background (my fault), it was still a hack.

What is actually happening is that interactive cmd.exe does not wait for a Windows-subsystem EXE to finish – it launches the process and immediately prints the next prompt while your program is still running. Injecting an Enter keystroke doesn’t make the shell wait; it just stuffs a key press into the input buffer and hopes for the best. Worse still, in modern terminal emulators like Windows Terminal (which use pseudoconsoles), GetConsoleWindow doesn’t even return the visible terminal window in the way old Win32 code expects. Nice work Microsoft.

Rather than faking keystrokes, it is much better to leave the keyboard alone and tell cmd.exe to wait when you run the program interactively:

start "" /b /wait hybrid-demo.exe --cli
start "" /b /wait hybrid-demo.exe --help > help.txt 2> errors.txt
start "" /b /wait hybrid-demo.exe --echo < input.txt > output.txt

The empty quotes "" are required because start treats the first quoted argument as a window title. /b stops it opening a new console window, and /wait makes cmd.exe wait for the program to exit and returns the exit code. Note that inside .bat scripts and PowerShell pipelines the shell often waits when output is piped or redirected anyway (see Microsoft’s start documentation), though /b has its own quirks with Ctrl+C handling that are worth testing for your own program.

That is the fundamental trade-off with a single EXE on Windows: a Windows-subsystem binary avoids the ugly console flash when double-clicked, but AttachConsole alone cannot force every interactive shell to wait for it.

Testing and Building the Code

You can’t really test any of this with mocked API calls, because the bugs live in how the Windows kernel and the MSVC C runtime set up handles at process creation. I wrote a Python test harness that spawns the real EXE with either a hidden private console or no console at all, and checks stdio, raw file descriptors 1 and 2, and Win32 WriteFile across files, pipes, NUL, merged streams and missing standard handles. Console output is read back directly from the console screen buffer so the tests aren’t fooled by pipes.

To build the demo and run the test suite yourself (you will need Python 3.10+ and Visual Studio 2022+ with the C++ desktop tools installed), run these commands from the root of the repository:

python tools/build.py --arch x86
python tests/run_tests.py --exe build/x86/hybrid-demo.exe --keep

This builds build/x86/hybrid-demo.exe (pass x64 instead of x86 for a 64-bit build) and checks the PE header to verify it is a true Windows-subsystem binary.

If you want to use this in your own program, you just need to drop hybrid_console.h and hybrid_console.c into your project. The code is MIT licensed and available in my windows-gui-console GitHub repository. If you find a terminal or C runtime edge case that breaks the descriptor setup, please open an issue or leave a comment below.

Filed under C/C++ Development

Leave a comment

Your email address will not be published. Required fields are marked *