Understanding Netlink Process Connector Output

This post is the latest in a series of posts about tracking processes on Linux which looks at the fine details of various APIs.

Here is some sample output from the Netlink Process Connector API on Linux. It contains an odd looking fork¹:

EventType=FORK

ParentPid=100
ParentTgid=99

ChildPid=200
ChildTgid=199

Take a moment to think about what is going on here and who gave birth to who.

At first glance, this fork looks impossible. The child doesn’t share a thread-group (tgid) with the parent, so it looks like a new process was created and not a new thread. However, if a new process was created then we should have a process with only one thread and therefore ChildPid should equal ChildTgid.

After a brief look at kernel sources, it turns out that the ParentPid and ParentTgid refer to the new process’ current parent which isn’t necessarily the process which called fork and birthed a child. In this case, we’re looking at a pretty standard report on a new thread’s creation by clone(CLONE_THREAD). Threads have the same parent process as the thread which created them. Something like this happened:

1. Process A (pid 99, tgid 99) created a new thread B
2. Thread B (pid 100, tgid 99) created a new process C
3. Process C (pid 199, tgid 199) created a new thread D
4. D has pid 200, tgid 199. This is expected because it is a thread in C’s process group. Likewise, it’s parent is thread B in process A. Therefore ParentPid=100 and ParentTgid=99.

Tracking processes accurately is one of those things that seem easy at first but turn out to be full of odd edge cases. If you liked this post, you should also read my post on a strange bug where the netlink process connector “forgot” to send events or my comparison of all the methods of tracking running processes on linux.

Work with me

Does this sort of thing interest you?

I started as a low-level engineer, but today I’m the co-founder and CEO of Robusta.dev and we’re hiring! I still do the occasional deep technical dive, as well as building a world-class team of excellent engineers and a product used by hundreds of companies.

If you join our team, you’ll work closely with me and be a core part of the founding team. Email [email protected] and mention you came from this post. We’re hiring in Israel as well as remote.