Unhelpful IP Helper: A Handle Leak Story

After my post about a CancelIoEx bug, I decided to cover another defect in Windows system components, this time in IP Helper (the part of the Windows API responsible for network statistics and connection tables).

Among other things, this API lets you map packets intercepted at the network level to local processes. You’d think it’s a battle-tested mechanism running under the hood of many tools and network filters. But while testing WireSock Secure Connect in process-based split tunneling mode, we hit a leak that can accumulate a large number of process handles within minutes in our test.

The investigation started with a message in our WireSock Telegram support group: one user noticed abnormally fast growth in the number of open process handles. The problem reproduced reliably under load and disappeared if we switched to IP-based filtering. That was the first clue that eventually led us to a bug in IP Helper’s implementation.

Special thanks to @dno5iq, who spotted the issue, reversed GetOwnerModuleFromPidAndInfo, and helped confirm the defect in its implementation.

Getting to the root cause

We debugged it right in chat, as a mini-investigation. It quickly became clear that handle leaks occurred only in process-based split tunneling; with IP-based filtering (AllowedIPs/DisallowedIPs), everything worked fine. That narrowed the search and let us focus on the code that resolves a process name by PID.

In WireSock, mapping an intercepted packet to a local process is done via the connection tables you can obtain through IP Helper. We use that API to derive the executable module name from a PID.

In the original resolver, WireSock did not open process handles directly for this. However, the IP Helper functions GetOwnerModuleFromTcpEntry, GetOwnerModuleFromTcp6Entry, GetOwnerModuleFromUdpEntry, and GetOwnerModuleFromUdp6Entry can internally open a handle to the owning process to retrieve module info. Those handles aren’t returned to the caller; Windows is supposed to close them internally. Under load, something clearly went wrong.

In the Windows build we examined, all of these functions called the same internal routine – GetOwnerModuleFromPidAndInfo. After decompiling it, we found that it indeed opens a process handle and then performs two checks:

1. Whether the handle was obtained successfully.
2. Whether GetModuleFileNameExW succeeds in retrieving the path to the executable.

The pseudocode below summarizes the failure path we observed in that build. It is not a public IP Helper implementation or a claim about every Windows version.

If the first check passes (e.g., the process has technically exited but still lingers thanks to remaining references) and the second fails, the function returns without closing the previously opened handle. Under load, that turns into a steadily growing leak.

Simplified pseudocode

HANDLE hProcess = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, FALSE, pid);

if (hProcess) {
    // Try to obtain the process module path
    if (!GetModuleFileNameExW(hProcess, NULL, pathBuffer, bufferSize)) {
        // This observed error path skips CloseHandle and leaks this handle
        return ERROR;
    }
    // Continue processing the process name
    CloseHandle(hProcess);
}

Our workaround

Because the leak was inside the IP Helper path we used, I moved the process-name lookup into code where we control handle lifetime. This workaround does not depend on whether a particular Windows build has since changed.

I rewrote the process resolver so we no longer rely on GetOwnerModuleFromPidAndInfo to obtain the process name. Now:

• We open and close handles explicitly in our code.
• We added request caching to speed up frequent lookups.
• Performance in our scenario is actually better than with the original IP Helper path.

The source for the new resolver is in the ProxiFyre repository and is used in both ProxiFyre and WireSock Secure Connect. In our high-churn test scenarios, this removed the observed handle growth and improved responsiveness. The performance result depends on workload and is not a benchmark for every application.

The fix has been included in WireSock Secure Connect since v2.4.18. Early builds and discussion are available in our Telegram support group: https://t.me/wiresock.

Takeaways

Even system APIs that developers rely on can harbor defects that show up only under specific conditions. Finding them requires reproducible load tests, isolating suspicious areas, and, if necessary, digging into internal implementations.

In this case, we sidestepped the bug by replacing the problematic call with our own implementation, eliminating the leak and making the application faster and more reliable.