Cleaning up I/O should be the unremarkable part of shutting down a service. In one Go service on Windows, it became the point where progress stopped: a thread entered CancelIoEx and stayed there. The process was still alive, but the work that depended on it was not moving.
I do not use Go every day, so this investigation took me from a familiar cleanup operation into some less familiar territory: Windows I/O cancellation, kernel APCs and the way Go reuses operating-system threads. A rare failure in a small pipe test turned out to be a useful lead for a real production problem.
There are two related cases in this story: concurrent I/O and close on a synchronous pipe, and a thread that issued asynchronous socket I/O before becoming blocked on a synchronous pipe read. Keeping them separate makes both the dump analysis and the eventual upstream changes easier to follow.
When cleanup stops making progress
The first surprise was the documented behavior of CancelIoEx:
The CancelIoEx function does not wait for all canceled operations to complete.
That describes cancellation versus completion; it should not be stretched into a promise that every internal cancellation path is wait-free. A successful cancellation request also does not mean the I/O has finished. Completion can still report success, cancellation or another error, and the associated buffers and OVERLAPPED state must remain valid until completion.
I needed a dump from an affected machine to understand where the wait came from. The original service was difficult to reproduce, so I looked for similar symptoms and found Go issue #64482, which had collected Windows pipe-close timeouts. Its TestPipeIOCloseRace gave me a much smaller workload to investigate.
A small pipe stress program
The workload repeatedly creates a pipe, runs a reader and writer, and closes both ends from a third goroutine after a short delay. This preserves the race from the original experiment; it is a diagnostic workload, not a reliable trigger on every machine. I first observed the hang on Windows 11 ARM64 and later on a virtual x64 system, while my 12th-generation Intel laptop did not reproduce it.
The program below is a standalone Windows adaptation of Go’s pipe-close race test. It uses os.ErrClosed, recognizes Windows broken-pipe results by error code, and accepts positive short reads because a pipe is a byte stream. The small defaults make an initial run manageable; they do not impose a time limit on a blocked system call.
//go:build windows
package main
import (
"errors"
"flag"
"fmt"
"io"
"os"
"runtime"
"sync"
"syscall"
"time"
)
func main() {
workers := flag.Int("workers", 2, "number of concurrent workers")
iterations := flag.Int("iterations", 100, "pipe races per worker")
flag.Parse()
if *workers <= 0 || *iterations <= 0 || flag.NArg() != 0 {
fmt.Fprintln(os.Stderr, "workers and iterations must be positive; no positional arguments")
os.Exit(2)
}
fmt.Printf("%s %s/%s; workers=%d, iterations/worker=%d\n",
runtime.Version(), runtime.GOOS, runtime.GOARCH, *workers, *iterations)
results := make(chan error, *workers)
for i := 0; i < *workers; i++ {
go func(id int) {
for j := 0; j < *iterations; j++ {
if err := runPipeIOCloseRace(); err != nil {
results <- fmt.Errorf("worker %d, iteration %d: %w", id, j, err)
return
}
}
results <- nil
}(i)
}
failed := false
for i := 0; i < *workers; i++ {
if err := <-results; err != nil {
fmt.Fprintln(os.Stderr, err)
failed = true
}
}
if failed {
os.Exit(1)
}
fmt.Println("completed; no hang observed in this run")
}
func expectedWriteError(err error) bool {
return errors.Is(err, os.ErrClosed) ||
errors.Is(err, syscall.ERROR_BROKEN_PIPE) ||
errors.Is(err, syscall.Errno(232)) // ERROR_NO_DATA: pipe is being closed.
}
func writeUntilClosed(w io.Writer) error {
for {
n, err := w.Write([]byte("hi"))
if err != nil {
if expectedWriteError(err) {
return nil
}
return err
}
if n != 2 {
return io.ErrShortWrite
}
}
}
func readUntilClosed(r io.Reader) error {
var buf [2]byte
for {
n, err := r.Read(buf[:])
if err != nil {
if errors.Is(err, io.EOF) || errors.Is(err, os.ErrClosed) {
return nil
}
return err
}
// A pipe is a byte stream: a positive short read is valid.
if n == 0 {
return io.ErrNoProgress
}
}
}
func runPipeIOCloseRace() error {
r, w, err := os.Pipe()
if err != nil {
return err
}
var wg sync.WaitGroup
var once sync.Once
var firstErr error
fail := func(err error) {
if err != nil {
once.Do(func() { firstErr = err })
}
}
wg.Add(3)
go func() {
defer wg.Done()
fail(writeUntilClosed(w))
}()
go func() {
defer wg.Done()
fail(readUntilClosed(r))
}()
go func() {
defer wg.Done()
time.Sleep(time.Millisecond)
fail(r.Close())
fail(w.Close())
}()
wg.Wait()
return firstErr
}Save the listing as hangtest_windows.go. In a disposable Windows test VM, after taking a checkpoint and arranging an external watchdog and dump collection, build it and start with a small workload:
go version
go build -o hangtest.exe hangtest_windows.go
.\hangtest.exe -workers 2 -iterations 100The original workload was 10 workers with 100,000 iterations each: 1,000,000 pipe races in total. To request that scale, use .\hangtest.exe -workers 10 -iterations 100000. A finite iteration count is not a timeout: if a close blocks, the worker and main goroutine can still wait indefinitely. An in-process timer or context does not automatically interrupt that kernel wait.
Unexpected I/O errors produce diagnostics and a nonzero exit status. “No hang observed” only describes that run; it is not proof that a toolchain or Windows build is unaffected. Changing the harness and its workload also changes timing, so this listing is not a byte-for-byte reconstruction of the executable used for the old dumps.
What the dump analysis showed
The investigation with Microsoft support is recorded in my public July 2025 follow-up. In the pipe-only case, one thread was waiting in a synchronous pipe write. Another was inside CancelIoEx on a related pipe endpoint. The support explanation identified a cancellation APC that could not make progress while normal kernel APC delivery was disabled on the thread in the synchronous wait.
These are related pipe file objects, not evidence that both calls used the identical handle or file object. The excerpt below retains the original addresses, offsets and anonymized process names. It does not record the exact Windows build, Go toolchain or full dump needed to reproduce the kernel analysis independently.
Pipe write and concurrent cancellation
The writer is waiting in nt!IopWaitForSynchronousIoEvent; the file object is marked Synchronous IO:
Process Thread CID UserTime KernelTime ContextSwitches Wait Reason Time State
hangtest.exe (ffffa202a18f0080) ffffa2029d6c3080 2d38.1cd0 63ms 344ms 3380 Executive 9m:35.406 Waiting
Irp List:
IRP File Driver Owning Process
ffffa202a47f2b00 (null) Npfs hangtest.exe
# Child-SP Return Call Site
0 ffffe881ff1f7e70 fffff8067f02a4d0 nt!KiSwapContext+0x76
1 ffffe881ff1f7fb0 fffff8067f0299ff nt!KiSwapThread+0x500
2 ffffe881ff1f8060 fffff8067f0292a3 nt!KiCommitThreadWait+0x14f
3 ffffe881ff1f8100 fffff8067f1f1744 nt!KeWaitForSingleObject+0x233
4 ffffe881ff1f81f0 fffff8067f406d96 nt!IopWaitForSynchronousIoEvent+0x50
5 (Inline) ---------------- nt!IopWaitForSynchronousIo+0x23
6 ffffe881ff1f8230 fffff8067f3cfdb5 nt!IopSynchronousServiceTail+0x466
7 ffffe881ff1f82d0 fffff8067f487ec0 nt!IopWriteFile+0x23d
8 ffffe881ff1f83d0 fffff8067f212005 nt!NtWriteFile+0xd0
9 ffffe881ff1f8450 00007fffdb70d5f4 nt!KiSystemServiceCopyEnd+0x25
a 00000055e4fff638 0000000000000000 0x7fffdb70d5f4
Irp Details: ffffa202a47f2b00
Thread Process Frame Count
============================= ===========
ffffa2029d6c3080 hangtest.exe 2
Irp Stack Frame(s)
# Driver Major Minor Dispatch Routine Flg Ctrl Status Device File
=== ================ ===== ===== ================ === ==== ======= ================ ================
->2 \FileSystem\Npfs WRITE 0 IRP_MJ_WRITE 0 1 Pending ffffa202987985f0 ffffa2029f495670
================================================================
ffffa2029f495670
Related File Object: 0xffffa2029f494d10
Device Object: 0xffffa202987985f0 \FileSystem\Npfs
Vpb is NULL
Flags: 0x40082
Synchronous IO
Named Pipe
Handle CreatedThe cancellation thread is waiting below nt!NtCancelIoFileEx, in nt!IopCancelIrpsInThreadList:
FsContext: 0xffffc40ddc15a8c0 FsContext2: 0xffffc40ddc6bb291
Private Cache Map: 0x00000001
CurrentByteOffset: 0
Process Thread CID UserTime KernelTime ContextSwitches Wait Reason Time State
hangtest.exe (ffffa202a18f0080) ffffa2029f0d7080 2d38.241c 31ms 391ms 4151 Executive 9m:35.406 Waiting
# Child-SP Return Call Site
0 ffffe881fedd8fe0 fffff8067f02a4d0 nt!KiSwapContext+0x76
1 ffffe881fedd9120 fffff8067f0299ff nt!KiSwapThread+0x500
2 ffffe881fedd91d0 fffff8067f0292a3 nt!KiCommitThreadWait+0x14f
3 ffffe881fedd9270 fffff8067f4a72dd nt!KeWaitForSingleObject+0x233
4 ffffe881fedd9360 fffff8067f5218dc nt!IopCancelIrpsInThreadList+0x125
5 ffffe881fedd93b0 fffff8067f4a70d6 nt!IopCancelIrpsInThreadListForCurrentProcess+0xc4
6 ffffe881fedd9470 fffff8067f212005 nt!NtCancelIoFileEx+0xc6
7 ffffe881fedd94c0 00007fffdb70e724 nt!KiSystemServiceCopyEnd+0x25
8 00000055e45ffc88 0000000000000000 0x7fffdb70e724
This thread has been waiting 9m:35.406 on a kernel component request
===============================================================
0xffffa2029f494d10
Related File Object: 0xffffa2029fb0dd60
Device Object: 0xffffa202987985f0 \FileSystem\Npfs
Vpb is NULL
Event signalled
Flags: 0x40082
Synchronous IO
Named Pipe
Handle CreatedA socket operation and a synchronous pipe read
The production case involved a different combination. The support account describes a thread that had issued asynchronous socket I/O and later blocked in a synchronous pipe read. Cancellation from another thread targeted the AFD endpoint, but depended on an APC executing on the thread now waiting in the pipe operation. It was not the blocked thread trying to call CancelIoEx itself.
Process Thread CID UserTime KernelTime ContextSwitches Wait Reason Time State
reducted_executable_name.exe (ffffaf85feb300c0) ffffaf8602c90080 e88.2b24 57s.703 24s.281 549327 Executive 3h:25:43.718 Waiting
Irp List:
IRP File Driver Owning Process
ffffaf8600e76640 (null) Npfs reducted_executable_name.exe
# Child-SP Return Call Site
0 ffffd005e29d4570 fffff8025d7071d7 nt!KiSwapContext+0x76
1 ffffd005e29d46b0 fffff8025d706d49 nt!KiSwapThread+0x297
2 ffffd005e29d4770 fffff8025d705ad0 nt!KiCommitThreadWait+0x549
3 ffffd005e29d4810 fffff8025dca1548 nt!KeWaitForSingleObject+0x520
4 (Inline) ---------------- nt!IopWaitForSynchronousIo+0x3e
5 ffffd005e29d48e0 fffff8025dc9ffc8 nt!IopSynchronousServiceTail+0x258
6 ffffd005e29d4990 fffff8025d877cc5 nt!NtReadFile+0x688
7 ffffd005e29d4a90 00007fff4e820094 nt!KiSystemServiceCopyEnd+0x25
8 000000002de1fbe8 00007fff4a9aa747 ntdll!ZwReadFile+0x14
9 000000002de1fbf0 000000000047337e KERNELBASE!ReadFile+0x77
a 000000002de1fc70 000000000000056c reducted_executable_name+0x7337e
b 000000002de1fc78 000000c000573ec0 0x56c
c 000000002de1fc80 0000000000008000 0xc000573ec0
d 000000002de1fc88 000000c001701d54 0x8000
e 000000002de1fc90 0000000000000000 0xc001701d54
Irp Details: ffffaf8600e76640
Thread Process Frame Count
=========================== ===========
ffffaf8602c90080 reducted_executable_name.exe 2
Irp Stack Frame(s)
# Driver Major Minor Dispatch Routine Flg Ctrl Status Device File
=== ================ ===== ===== ================ === ==== ======= ================ ================
->2 \FileSystem\Npfs READ 0 IRP_MJ_READ 0 1 Pending ffffaf85f7da2c00 ffffaf86107074b0
==============================================================
ffffaf86107074b0
Related File Object: 0xffffaf8602a1f6a0
Device Object: 0xffffaf85f7da2c00 \FileSystem\Npfs
Vpb is NULL
Flags: 0x40082
Synchronous IO
Named Pipe
Handle Created
File Object is currently busy and has 0 waiters.
Process Thread CID UserTime KernelTime ContextSwitches Wait Reason Time State
reducted_executable_name.exe (ffffaf85feb300c0) ffffaf8602d06080 e88.2840 53s.969 24s.203 539612 Executive 3h:27:07.468 Waiting
# Child-SP Return Call Site
0 ffffd005e2b02630 fffff8025d7071d7 nt!KiSwapContext+0x76
1 ffffd005e2b02770 fffff8025d706d49 nt!KiSwapThread+0x297
2 ffffd005e2b02830 fffff8025d705ad0 nt!KiCommitThreadWait+0x549
3 ffffd005e2b028d0 fffff8025dd06e87 nt!KeWaitForSingleObject+0x520
4 ffffd005e2b029a0 fffff8025dd6b377 nt!IopCancelIrpsInThreadList+0x11f
5 ffffd005e2b029f0 fffff8025dd06cd2 nt!IopCancelIrpsInThreadListForCurrentProcess+0xc3
6 ffffd005e2b02ab0 fffff8025d877cc5 nt!NtCancelIoFileEx+0xc2
7 ffffd005e2b02b00 00007fff4e8211c4 nt!KiSystemServiceCopyEnd+0x25
8 000000002e01fc28 00007fff4a9f9ef0 ntdll!ZwCancelIoFileEx+0x14
9 000000002e01fc30 000000000047337e KERNELBASE!CancelIoEx+0x10
a 000000002e01fc70 000000002e01fc88 reducted_executable_name+0x7337e
0xffffaf86`11746570
\Endpoint
Device Object: 0xffffaf85f7db0b20 \Driver\AFD
Vpb is NULL
Flags: 0x6040400
Queue Irps to Thread
Handle Created
Skip Completion Port Queueing On Success
Skip Set Of File Object Event On CompletionThis explains the two reported incidents; the stack names alone do not prove that every CancelIoEx wait has that cause. A live investigation still needs the relevant handles, pending IRPs, issuing threads, symbols and exact binary versions. The public report and later Go implementation provide context, not a substitute for those details.
What changed upstream
Go 1.26 was discussed as a possible target in 2025, but that discussion did not establish which release would contain the fix. In the September 2026 comment, the maintainer distinguished the pipe-only hang addressed by the first proposed change from the socket-plus-pipe case still under investigation.
The subsequent synchronous-pipe isolation change was reverted after a double-close regression. A corrected restoration was merged into Go’s development branch on 19 October 2026 and closed #64482. It coordinates cancellation with dedicated synchronous-I/O workers rather than simply removing pipe cancellation.
A development-branch fix and a released toolchain are different things. The Go 1.27.3 and Go 1.26.10 sources still have the earlier pipe-close path and do not contain that restoration. The release history is the place to check later availability; an issue being closed is not sufficient evidence that an installed binary includes the change.
What this means for a Windows service
The API distinction matters when you own the native I/O layer. CancelSynchronousIo takes a handle to the issuing thread, whereas CancelIoEx takes a file handle and can target a particular overlapped request. Cancellation can race with completion; neither API makes lifetime management or completion handling disappear.
That is not a drop-in replacement for application code using os.File.Close. Go does expose runtime.LockOSThread, but locking a goroutine to its current thread does not make that thread fresh or remove I/O it issued earlier. A correct native implementation needs controlled thread ownership, handle lifetime and completion/cancellation coordination. The upstream restoration addresses those concerns inside the I/O implementation.
- Record the exact Go version, Windows build and architecture of an affected process. Retain a matching binary and symbols with the dump; a timeout stack is a lead, not a complete diagnosis.
- Check the implementation and release containing the relevant fix before attributing a result to an upgrade. Rebuild and exercise the actual service workload on its target Windows environment.
- Keep lifecycle ownership clear: know which component starts, cancels and waits for each operation. Do not patch out
CancelIoExor switch APIs solely because a small stress test stops hanging; that can break other close/cancellation paths. - If a service must recover from this kind of stall, consider process isolation and externally supervised recovery where the workload permits it. A context deadline or another goroutine waiting on close is not, by itself, a remedy for a blocked native call.
The useful lesson is how an ordinary cleanup operation crossed an abstraction boundary. Goroutines made the application straightforward, but pending I/O and cancellation still belonged to Windows threads and kernel objects. The flaky pipe test gave us a way into that boundary and enough evidence to turn an elusive production hang into an upstream change.