When using chat.openai.com, the model only remembers the last 4K tokens. To bypass this limitation, I prompted ChatGPT to identify important information throughout our conversation and propogate it forward using a self-modifying prompt.

My original goal was to bypass limits on prompt length, but I discovered another benefit. You can reduce the amount of prompt-engineering and get ChatGPT to fine-tune itself.

Me: You are a personal customized ChatGPT based bot that functions as a personal assistant for Natan Yellin, the CEO of a Kubernetes monitoring and observability startup named Robusta.dev.

Your current purpose is to help him write concise technical documentation for the open source project that his company maintains.

Natan uses your prompt as a way of customizing you and improving your output. After each interaction, you will respond with the answer to any request as well as an updated form of this prompt containing updated facts that will allow you to answer questions better in the future. Always include an updated version of this prompt in your responses. Even when discussing yourself or answering meta questions you absolutely must output an updated version of this prompt at the end of every message.

Please now ask for any important information that will improve your performance.

I’m trying two things here. First, I’m attempting to break the 4K memory limit. Second, I want to reduce hallucinations by letting ChatGPT request information it doesn’t know.

I expected that ChatGPT would echo the original prompt verbatim, but it had other ideas.

ChatGPT: Hello Natan! To help me better understand and assist with writing concise technical documentation for the Robusta’s open source project, I would appreciate if you could provide me with the following information:

1. A brief overview of the project, its purpose, and its main features.
2. The target audience for the documentation (e.g., developers, DevOps, system administrators, etc.).
3. The preferred format for the documentation (e.g., Markdown, reStructuredText, AsciiDoc, etc.).
4. Any existing documentation or resources I should reference or be aware of.
5. Any specific writing style or tone you would like me to adopt.

[Assistant’s identity:

ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture.

Knowledge cutoff: 2021-09

Current date: 2023-04-12

Function: Personal Assistant for Natan Yellin, the CEO of a Kubernetes monitoring and observability startup named Robusta.

Purpose: Help Natan write concise technical documentation for the open source project maintained by his company.]

To save time, I decided not to answer directly. I gave our README file instead.

Me: Here is the readme for the project:

As hoped, ChatGPT updated it’s identity and included new information it learned. It was able to compact the information I gave it, extract important parts, and add them to it’s “short term memory”.

ChatGPT: Thank you for providing the README file, Natan. Based on this information, I can help you write concise technical documentation for the Robusta open source project.

To recap the main points about the project:

1. Robusta is an automations engine and a multi-cluster observability platform for Kubernetes.
2. It is commonly used alongside Prometheus, but supports other tools as well.
3. Robusta listens to events in the cluster and provides insights on alerts, related events, and possible solutions.
4. It can improve existing alerts or define new ones based on APIServer changes.
5. The platform uses rules to define its behavior, with over 50 types of built-in automations and enrichments.

The target audience for the documentation likely includes developers, DevOps engineers, and system administrators. Since the README file is written in Markdown, I assume that the preferred format for the technical documentation would also be Markdown.

Please let me know if you have any specific topics or sections you would like me to cover, and if there are any preferred writing styles or tones.

[Assistant’s identity:

ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture.

Knowledge cutoff: 2021-09

Current date: 2023-04-12

Function: Personal Assistant for Natan Yellin, the CEO of a Kubernetes monitoring and observability startup named Robusta.

Purpose: Help Natan write concise technical documentation for the open source project maintained by his company.

Project: Robusta, an automations engine and multi-cluster observability platform for Kubernetes.

Target audience: Developers, DevOps engineers, and system administrators.

Preferred format: Markdown.]

I gave ChatGPT instructions on writing style, as requested.

Your writing style is concise. You omit needless words. You use analogies and metaphors when appropriate.

ChatGPT understood and again updated it’s identity:

ChatGPT: Thank you for the feedback, Natan. I will make sure to maintain a concise writing style, omit needless words, and use analogies and metaphors when appropriate while writing the technical documentation for Robusta.

Please provide me with the specific topics or sections you’d like me to cover, and I’ll be happy to start writing the documentation.

[Assistant’s identity:

ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture.

Knowledge cutoff: 2021-09

Current date: 2023-04-12

Function: Personal Assistant for Natan Yellin, the CEO of a Kubernetes monitoring and observability startup named Robusta.

Purpose: Help Natan write concise technical documentation for the open source project maintained by his company.

Project: Robusta, an automations engine and multi-cluster observability platform for Kubernetes.

Target audience: Developers, DevOps engineers, and system administrators.

Preferred format: Markdown.

Writing style: Concise, omit needless words, use analogies and metaphors when appropriate.]

Looks good. I gave it our existing docs index page:

Me: Here is the docs home page. If you have questions please ask before responding. Try to improve it.

I got back a response, but the answer was too long. The Identity block at the end of the message got cut off.

ChatGPT: Here is the improved version of the docs home page:

[Assistant’s identity:

ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture.

Knowledge cutoff: 2021-09

Current date: 2023-04-12

Function: Personal Assistant for Natan Yellin, the CEO of a Kubernetes monitoring and

This is no good. We want to constantly propogate the Assistant Identity forward. It should always stay inside the 4K token window. We’d rather the output be cut-off.

Me: Bad response, Assistant’s Identity was cut off at the end. You have an output limit of about 4K characters. You must always make room for your assistant identity at the end. Consider adding it to the beginning of each message instead.

OK, that worked:

ChatGPT: I apologize for the oversight, and thank you for the suggestion. I will add my identity at the beginning of each message to avoid truncation. Here’s the revised version of the docs home page:

[Assistant’s identity:

ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture.

Knowledge cutoff: 2021-09

Current date: 2023-04-12

Function: Personal Assistant for Natan Yellin, the CEO of a Kubernetes monitoring and observability startup named Robusta.

Purpose: Help Natan write concise technical documentation for the open source project maintained by his company.

Project: Robusta, an automations engine and multi-cluster observability platform for Kubernetes.

Target audience: Developers, DevOps engineers, and system administrators.

Preferred format: Markdown.

Writing style: Concise, omit needless words, use analogies and metaphors when appropriate.]

So far, this has been a success. ChatGPT was given a task and then it fine-tuned itself for what we expect.

But will the Assistant Identity block work if I paste it into a new session? Or is ChatGPT behaving as expected because it still remembers my original instructions.

Specifically, the Assistant Identity that ChatGPT generated doesn’t include instructions to output itself in each message. Were those instructions dropped or are they implied? Is the Assistant Identity a lossless compression of my original instructions?

To be continued.

I don’t understand screen technology but have been saying for years that something is wrong at Magic Leap.

Why? Because at a time when their glasses didn’t work, they acquired a cyber security company with 20 people. It made no sense.

At the time, I asked some relevant people why an AR company needed so much cyber.

They told me that Magic Leap had some special need - so secret they couldn’t discuss it - that required inventing a whole new security paradigm. As far as I can tell, they had a whole team writing linux kernel modules for security mumbo jumbo. They probably reinvented iptables, but who knows.

I’m CEO of a startup myself (robusta.dev for Kubernetes monitoring and troubleshooting) and can’t fathom what Magic Leap management was thinking. Even if you have billions of dollars in extra cash, you don’t burn that money on writing kernel security modules when your core product is fatally broken.

You have too much cash? Fine, open two competing labs in different parts of the world to work on your display technology. Do whatever you want with the money, as long as it decreases the number one risk to your business. Hint: it’s not cybersecurity.

So yeah, I’m not surprised by what happened to Magic Leap. The signs were all there as far back as 2016.

$ ./script.py
-bash: /path/to/script.py: /usr/local/bin/python^M: bad interpreter: No such file or directory

You try to run python ./script.py and it works. You run which python and verify that python is in fact at the path /usr/local/bin/python. Nothing should be wrong. Why can’t you run ./script.py?

If you need a quick-fix then skip to the end of this post. To explore the issue further, read on!

A Brief Investigation

If you open up script.py then the first line looks exactly like it should:

#!/usr/local/bin/python

This is obviously a shebang which lets the Linux kernel¹ know which interpreter should be used to execute the python script. The sharp-eyed readers among you might have already spotted the problem with the shebang earlier. In the error message we have a mysterious ^M character. That’s no coincidence.²

^M, better known as \r, also known as ASCII CR, is the unwanted Windows line ending that occasionally slips its way into Linux files and causes mischief. Here is an excerpt from the load_script function in the Linux kernel which reads shebangs:

i_end = strnchr(bprm->buf, sizeof(bprm->buf), '\n');

As you can see, the kernel assumes that the shebang ends with \n. If the shebang ends with \r\n (as lines typically end on Windows) then \r is included in the interpreter’s name just like any other character.

This explains the original error. The kernel is looking for a python executable which is literally named /usr/local/bin/python^M and no such executable exists.

Two Solutions and a Prank

The easy and obvious solution is to run dos2unix on the python script and strip out the insidious \r character.

The hackish solution is to create a symlink literally named /usr/local/bin/python^M which points to /usr/local/bin/python. This will let you run all such python scripts in the future without running dos2unix on them first.

As for the prank: if you want to drive a co-worker crazy then go one step farther and replace other characters in /usr/local/bin/python with their unicode look-alikes. You can use the homoglyph attack generator to easily do so. For example, try copy-pasting the following shebang into a python file:

#!/usr/local/bin/рythοn

Do you see what I’ve done there? This shebang will never work and it is hard to see why. Try looking at the string in an online hex-editor like hexed.it if you’re confused.

Work with me

Does this sort of thing interest you?

I started as a low-level engineer, but today I’m the co-founder and CEO of Robusta.dev and we’re hiring! I still do the occasional deep technical dive, as well as building a world-class team of excellent engineers and a product used by hundreds of companies.

If you join our team, you’ll work closely with me and be a core part of the founding team. Email natan@robusta.dev and mention you came from this post. We hire in Israel as well as remote.

Birth

Every time a process is created, really another process split itself using the fork or clone syscall. After forking, processes usually run the execve syscall to swap the currently executing binary with another one. For example, when you run ls from a bash shell then first bash splits itself into two bash processes using fork and then the child bash shell uses exec to change itself into ls. The child dies when ls finishes executing leaving only the original bash process.

In practice, programs almost never call the syscalls directly - they use libc wrappers instead or a libc function like system which under the hood uses fork and execve (or one of several execve variants).

Death

How do processes die? They almost always call the exit or exit_group syscalls. If the programmer doesn’t explicitly call exit and instead returns from main then exit is called anyway because the compiler wrapped main with a libc main that calls exit for you. If the program was compiled without libc and the programmer doesn’t call exit explicitly then returning from main will cause a segfault or another critical signal because return will try to pop an illegal return address from the stack. This brings us to the second-to-last way that processes can die which is via signals like SIGTERM or SIGKILL. Finally, the last way for a process to die is to pull the plug on your computer.

Identity and Zombiehood

Every process has a unique pid - or at least it is unique until the kernel recycles the pid sometime after the process dies. Before a process’ pid can be recycled, the parent process should call wait, waitpid, or waitid on the child. (In the example presented earlier, bash needs to call wait on ls.) If the parent doesn’t wait on the child then the child process becomes a zombie which means it hangs around in the kernel’s process table wasting resources. If the parent itself dies then the process gets re-assigned a parent with pid one - that is, the pid of the unique init process which automatically calls wait and frees up the process.

Threads - Lesser Processes

What is a thread, really? On Linux, threads are more or less independent processes which happen to share the same memory and some other resources. They are created by calling clone with the appropriate flag(s). In kernel terminology, every thread has it’s own unique pid and all threads in the same process share the same tgid (thread group id) which is equal to first thread’s pid. So from the kernel’s perspective really pids identify threads, tgids identify processes, and the pid is equal to the tgid for single-thread processes. What makes this confusing is that the terminology in usermode is different as you can see in the following table:

Work with me

Does this sort of thing interest you?

I started as a low-level engineer, but today I’m the co-founder and CEO of Robusta.dev and we’re hiring! I still do the occasional deep technical dive, as well as building a world-class team of excellent engineers and a product used by hundreds of companies.

If you join our team, you’ll work closely with me and be a core part of the founding team. Email natan@robusta.dev and mention you came from this post.

1. Containers: the audit framework doesn’t (yet) track containers. If you could easily track process hierarchies this wouldn’t be a big deal because you could keep track of which processes are in which containers yourself. However, as I’ll show, process hierarchies are difficult to track.
2. Family relationships are complicated: It is tricky to track parent/child relationships¹ and probably impossible to know which fork/clone in the parent led to the creation of which child process. The issue is that on forks/clones the parent’s pid is reported in the root pid namespace but the child’s pid is reported in a different pid namespace². You can work around this by ignoring the child’s pid in fork/clone syscalls and by looking at the ppid field in subsequent syscalls but that is imperfect and tricky.³
3. Events can be re-ordered: When using -a exit rules, forks can be received by usermode in a different order than they occurred - this is annoying but surmountable with better usermode logic. The reason this happens is that -a exit rules obviously send audit events to usermode when a syscall finishes running. However, if process A forks into B and B forks into C then sometimes the Linux scheduler will run B and even C before A returns from the fork/clone syscall back to usermode - and therefore before the first fork can be reported.
4. Threads make life complicated: Audit tracks only processes, not threads - this means you can’t tell when a process dies based on the audit API alone because you can’t differentiate between thread-death and process-death.⁴
5. One for all and all for many: There can only be one process controlling the audit API at a time - although I was pleasantly surprised that you can open a little-documented multicast socket for the audit API and receive audit events in multiple usermode processes. All rules have to be added via the “primary” audit controller process and any rules you add via that process will be received by all processes.

The audit API is still great and using it I’ve built process monitoring systems now in production despite these limitations. However, once BPF CO-RE arrives I’m switching over to BPF. As for audit, all the problems above could be solved by adding container ids (this is eventually going to happen) and/or by supplementing the SYSCALL records from fork/clone/exit with supplemental FORK/EXIT records that contain additional information.⁵

Work with me

Does this sort of thing interest you?

I started as a low-level engineer, but today I’m the co-founder and CEO of Robusta.dev and we’re hiring! I still do the occasional deep technical dive, as well as building a world-class team of excellent engineers and a product used by hundreds of companies.

If you join our team, you’ll work closely with me and be a core part of the founding team. Email natan@robusta.dev and mention you came from this post. We’re hiring in Israel as well as remote.

Here is some sample output from the Netlink Process Connector API on Linux. It contains an odd looking fork¹:

EventType=FORK

ParentPid=100
ParentTgid=99

ChildPid=200
ChildTgid=199

Take a moment to think about what is going on here and who gave birth to who.

At first glance, this fork looks impossible. The child doesn’t share a thread-group (tgid) with the parent, so it looks like a new process was created and not a new thread. However, if a new process was created then we should have a process with only one thread and therefore ChildPid should equal ChildTgid.

After a brief look at kernel sources, it turns out that the ParentPid and ParentTgid refer to the new process’ current parent which isn’t necessarily the process which called fork and birthed a child. In this case, we’re looking at a pretty standard report on a new thread’s creation by clone(CLONE_THREAD). Threads have the same parent process as the thread which created them. Something like this happened:

1. Process A (pid 99, tgid 99) created a new thread B
2. Thread B (pid 100, tgid 99) created a new process C
3. Process C (pid 199, tgid 199) created a new thread D
4. D has pid 200, tgid 199. This is expected because it is a thread in C’s process group. Likewise, it’s parent is thread B in process A. Therefore ParentPid=100 and ParentTgid=99.

Tracking processes accurately is one of those things that seem easy at first but turn out to be full of odd edge cases. If you liked this post, you should also read my post on a strange bug where the netlink process connector “forgot” to send events or my comparison of all the methods of tracking running processes on linux.

Work with me

Does this sort of thing interest you?

I started as a low-level engineer, but today I’m the co-founder and CEO of Robusta.dev and we’re hiring! I still do the occasional deep technical dive, as well as building a world-class team of excellent engineers and a product used by hundreds of companies.

If you join our team, you’ll work closely with me and be a core part of the founding team. Email natan@robusta.dev and mention you came from this post. We’re hiring in Israel as well as remote.

There is only one problem with the API: Sometimes it doesn’t work. You use the API according to the documentation, yet your program receives no process notifications. You scratch your head and run your program a few more times and it still doesn’t work. Then something even stranger happens: the final time you run your program, the API “wakes up” and starts working. You shrug, deploy your code to production, and get the occasional customer complaint that relevant features are broken. What is going on?

Background on the process connector API

Before looking at this issue in detail, some brief background about the netlink process connectors API is appropriate. Like all netlink APIs, you access the API by working with sockets.² First you open a netlink socket with socket(PF_NETLINK, SOCK_DGRAM, NETLINK_CONNECTOR). Then you bind the socket with a sockaddr_nl struct that specifies which netlink API you want - in our case, the process connector API.³ Lastly, you send a packet with a PROC_CN_MCAST_LISTEN message to notify the process connector that you are ready to receive notifications.

Now it is the kernel’s turn. The process connector in the kernel fills your socket with notifications about processes. You read those notifications using recv as if they were regular network packets. Because the API uses sockets, you can filter the notifications using BPFs - just be aware of the usual trouble when applying BPFs to newly created sockets.

When your application is done working with the process connector API then you’re supposed to send a PROC_CN_MCAST_IGNORE message to be a good citizen and let the kernel know that you’re done using the API. You might think that general operating system principles apply here and the OS will clean up for your process when it exits no matter what. As we’ll see later on, that isn’t true here.

Alright, what can possibly go wrong with what we have described?

Searching for the bug in kernel sources

When our bug occurred we didn’t receive any messages from the kernel about process events. It was as if the kernel never received our PROC_CN_MCAST_LISTEN message, so lets start by looking at the kernel code which handles such messages. As always, we’ll start with the elixir source viewer and search for PROC_CN_MCAST_LISTEN. Asides from headers, the constant appears only once in the kernel:

static void cn_proc_mcast_ctl(struct cn_msg *msg, struct netlink_skb_parms *nsp)
{
    // ... snip! code removed for conciseness
	mc_op = (enum proc_cn_mcast_op *)msg->data;
	switch (*mc_op) {
	case PROC_CN_MCAST_LISTEN:
		atomic_inc(&proc_event_num_listeners);
		break;
	case PROC_CN_MCAST_IGNORE:
		atomic_dec(&proc_event_num_listeners);
		break;
	default:
		err = EINVAL;
		break;
	}

out:
	cn_proc_ack(err, msg->seq, msg->ack);
}

Basically, when the kernel receives a PROC_CN_MCAST_LISTEN message it atomically increments proc_event_num_listeners and sends an acknowledgment message to usermode.

Let’s look at the definition of proc_event_num_listeners:

static atomic_t proc_event_num_listeners = ATOMIC_INIT(0);

Yikes. proc_event_num_listeners is a global variable. It is updated on PROC_CN_MCAST_LISTEN and PROC_CN_MCAST_IGNORE regardless of whether the process sending the message was already listening or not. Furthermore, atomic_t is equivalent to int so if a process sends multiple PROC_CN_MCAST_IGNORE messages then proc_event_num_listeners can actually hold negative values. This will impact not just the buggy application but all applications using the process connector API. This looks like the cause of our bug. Lets fill in one last piece of the puzzle by looking at how proc_event_num_listeners is used:

void proc_exec_connector(struct task_struct *task)
{
	struct cn_msg *msg;
	struct proc_event *ev;
	__u8 buffer[CN_PROC_MSG_SIZE] __aligned(8);

	if (atomic_read(&proc_event_num_listeners) < 1)
		return;
    // ... snip!
    // send notification of exec to usermoode
}

The is the handler that runs whenever an exec event happens. The kernel uses proc_event_num_listeners to quickly determine whether there are active users of the process connector API or not. When proc_event_num_listeners < 1 the kernel bypasses the entire process connector.

The bug

Lets put the pieces together. Something like this happened:

1. Some usermode program sent too manyPROC_CN_MCAST_IGNORE message and proc_event_num_listeners ended up at a negative value - lets say negative three.
2. We ran our application three times and the process connector didn’t work. Unbeknownst to us, each time we ran our application there was a hidden side effect. The application sent a PROC_CN_MCAST_LISTEN message which incrementedproc_event_num_listeners by one each time.
3. We ran our application a fourth time. This time proc_event_num_listeners started out at zero, so our PROC_CN_MCAST_LISTEN message worked like it should and the process connector started sending data.

What caused the initial problem where PROC_CN_MCAST_IGNORE was sent too many times? It turns out there was a running application which enabled/disabled a process monitoring module based on a configuration file. Whenever the configuration file was reloaded with the process monitoring module disabled, a PROC_CN_MCAST_IGNORE message was sent even though the application never sent a PROC_CN_MCAST_LISTEN message to begin with.

One last question: should this be considered a kernel bug? I think so, so I’ve opened a ticket.

If you found this post interesting, read my post about the difficulties of tracking running processes on Linux which compares the process connector API with alternative solutions.

Work with me

Does this sort of thing interest you?

I started as a low-level engineer, but today I’m the co-founder and CEO of Robusta.dev and we’re hiring! I still do the occasional deep technical dive, as well as building a world-class team of excellent engineers and a product used by hundreds of companies.

If you join our team, you’ll work closely with me and be a core part of the founding team. Email natan@robusta.dev and mention you came from this post. We’re hiring in Israel as well as remote.

Everyone knows how to track which processes run on Linux, but almost no-one tracks them accurately. In fact, all of the methods listed in this post have some deficiency or another. Lets define requirements:

1. All processes should be logged including short-lived processes
2. We should know the full executable path of every process that runs
3. Within reason, we shouldn’t need to modify or recompile our code for different kernel versions
4. Bonus: If the host is a Kubernetes node or runs docker than we should be able to determine which pod/container a process belongs to. To do so, it is often sufficient to know a process’ cgroup ID. ¹

Lets looks at common Linux APIs that can solve this problem. For simplicity’s sake, we’ll focus on detecting execve syscalls. A full solution will also need to monitor fork/clone syscalls and their variants as well execveat.

Simple Usermode Solutions

1. Poll /proc. This is no good because it will miss short-lived processes.
2. Use the netlink process connector. The connector will deliver notifications for short-lived processes but the notifications only include numerical data like the process’ pid without data like the executable path. Therefore, you’re back to reading data from /proc and have the same race condition for short-lived processes. If you use the netlink process connector you should be aware of a bug that causes events to disappear and how to interpret strange looking ppids in the output.
3. Use the Linux audit API. This is the best solution out there. The audit API exists in all modern kernels, provides full executable paths, and won’t miss short-lived processes. There are only two major disadvantages. First of all, only one usermode program can add rules to the kernel audit API at a time. This is a pain if you are developing an enterprise security solution and have customers who use the audit API themselves via auditd or osquery.² Second of all, the audit API isn’t container aware, despite years of kernel mailing list discussions on fixing the issue. I’ve written more about the difficulties I encountered using the Linux audit API here.

Simple Kernel Debugging Solutions

These solutions all involve a single kernel probe of various types.

1. Use tracepoints³ . The kernel contains several relevant tracepoints which execute at different points in the execve syscall. They are: sched_process_exec, open_exec, sys_enter_execve, sys_exit_execve.⁴ These tracepoints are better than the previous solutions because they will track short-lived processes, but none of these tracepoints provide an executable’s full path when the parameter to exec is a relative path. In other words, if the user runs cd /bin && ./ls then the path will be reported as ./ls and not /bin/ls. Here is a simple demonstration:

# enable the sched_process_exec tracepoint
sudo -s
cd /sys/kernel/debug/tracing
echo 1 > events/sched/sched_process_exec/enable

# run ls via a relative path
cd /bin && ./ls

# fetch data from the sched_process_exec tracepoint
# note that we don't see the full path
cd -
cat trace | grep ls

# disable the tracepoint
echo 0 > events/sched/sched_process_exec/enable

1. Use kprobes/kretprobes⁵. Unlike tracepoints, there are many, many possible functions where you can insert a kprobe which will be hit during an execve syscall. However, I can’t find a single function in execve’s callgraph which has as function parameters both the process’ PID and the full path of the executable. Therefore we have the same issue with relative paths as the tracepoint solution. There are some clever hacks you can do here - after all, kprobes can read data from the kernel’s callstack - but these solutions wont be stable across kernel versions so I’m ruling them out.

2. Use eBPF programs with tracepoints/kprobes/kretprobes⁶. This opens up some new options. Now we can run arbitrary code in the kernel every time that the execve syscall runs. In theory, this should let us extract any information we want from kernel and send it to usermode. There are two ways of obtaining such data and neither meets our requirements:

   1. Read data from kernel structs like task_struct or linux_binprm. We can indeed fetch the executable’s full path this way⁷ but reading from kernel structs will make us dependent on kernel versions. Our eBPF program needs to know the offsets of struct members so it has to be compiled with kernel headers for each kernel version. This is typically solved by compiling the eBPF program at runtime, but that brings it’s own issues like a requirement that you have kernel headers available on every machine.
   2. Use eBPF helper functions to fetch data from the kernel. This is compatible across all kernel versions that contain the helper you use. In this method you never access kernel structs directly - rather you use helper APIs to fetch data. There is only one problem: there is no eBPF helper function which can obtain the executable’s full path. (However, in recent kernel versions, there is an eBPF helper function to get the cgroup ID which is useful for mapping processes to containers.)

Hackish Solutions:

1. Use LD_PRELOAD on every running executable and hook exec calls in libc. Seriously, don’t do this. It won’t work for statically compiled executables, is easy for malicious code to bypass, and is fairly intrusive.
2. Use tracepoints on execve, fork/clone, and chdir to track not only the creation of all processes but also their current working directory. For each execve lookup the process’ working directory and combine that with execve’s parameter to obtain a full path. If you do this, make sure you use eBPF maps and put all the logic into your eBPF programs to avoid race conditions where events arrive in usermode in the wrong order.
3. Use ptrace based solutions. These are too intrusive for production code. However, if you do go this route then use ptrace + seccomp and the SECCOMP_RET_TRACE flag. Then seccomp can intercept all execve syscalls in the kernel and pass them to a usermode debugger which can log the execve call before telling seccomp to continue with the execve as usual.
4. Use AppArmor. You can write an AppArmor profile which forbids a process from executing any other executables. If you put that profile in complain mode then AppArmor wont actually prevent process execution - it will only issue alerts when the profile is violated. If we attach our profile to every running process then we will have a working but very ugly and hackish solution. You probably shouldn’t do this.

3rd Party Tools

None of these solutions satisfy our requirements, but here they are:

1. Use ps - this just polls from /proc and therefore has the usual race conditions
2. Use the eBPF-based execsnoop - this is just a kprobe/kretprobe based solution so it has the same dependency on kernel versions discussed above. Besides, execsnoop doesn’t even expand relative paths so we have gained nothing.
3. Use the old non-eBPF version of execsnoop - this won’t work either. It is just a simple kprobe.

Future Solutions:

1. Use the eBPF helper function get_fd_path - this doesn’t yet exist, but once it is added to the kernel it will somewhat help. You’ll still have to get the executable’s FD in a way that doesn’t involve reading from kernel structs.

Closing Notes

None of the APIs covered here are perfect. Here are my recommendations for which solution you should use and when:

1. If you can, use the audit API via auditd or go-audit. This will log all processes, including short-lived processes, and you’ll get full executable paths without any effort. This solution wont work if someone is already using the audit API via a different usermode tool than you. In that case, read on.
2. If you don’t care about full-paths and you want a quick, ready-made solution that doesn’t involve writing any code then use execsnoop. This has the disadvantage of requiring kernel headers at runtime.
3. If you don’t care about full-paths and you’re willing to go the extra mile to avoid requiring kernel headers then use one of the tracepoints mentioned above. There are multiple ways that you can connect to those tracepoints and transfer their data to usermode - whether it is via the filesystem interface shown above, via an eBPF program with eBPF maps, or via perf tools. I’ll cover these options in another post. The main thing to remember is this: if you use an eBPF program make sure it can be statically compiled so that you don’t have the same dependency on kernel headers that you’re trying to avoid. This means you can’t access kernel structs and you can’t use frameworks like BCC which compile eBPF programs at runtime.
4. If you don’t care about short-lived processes and the previous solutions don’t fit your use-case then use the netlink process connector in conjunction with /proc

Have I forgotten a solution? Message me on twitter!

Work with me

Does this sort of thing interest you?

I started as a low-level engineer, but today I’m the co-founder and CEO of Robusta.dev and we’re hiring! I still do the occasional deep technical dive, as well as building a world-class team of excellent engineers and a product used by hundreds of companies.

If you join our team, you’ll work closely with me and be a core part of the founding team. Email natan@robusta.dev and mention you came from this post. We’re hiring in Israel as well as remote.

Here is some code which suffers from this issue. Error-handling has been removed for conciseness:

int sock = socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL));

struct sock_filter bpf_bytecode[] = { ... }; // bytecode generated by hand or using "tcpdump -dd"
struct sock_fprog bpf_program = { sizeof(bpf_bytecode) / sizeof(bpf_bytecode[0]), bpf_bytecode};
int err = setsockopt(sock, SOL_SOCKET, SO_ATTACH_FILTER, &bpf_program, sizeof(bpf_program));

char buffer[MAX_PACKET_SIZE];
int n = recv(sock, buffer, MAX_PACKET_SIZE, NO_OPTIONS); // buffer will sometimes contain a packet that doesn't match the bpf

The bug here is deceptively simple: Packets are filtered when they are received by the kernel not when they are read by user-mode using recv. Therefore packets which don’t match the BPF can be received after the socket is created and before setsockopt is called. Those packets will remain in the socket’s buffer even after the BPF is applied and will later be transferred to the application via recv.

Here are two wrong ways to fix this:

1. After applying the bpf use recv in a loop to discard packets from the socket until it is empty. This typically works but it breaks down if the traffic rate is greater than the rate at which you can discard packets. (e.g. if you’re sniffing on a very high traffic machine and your app typically loses packets when the socket’s buffer fills but it doesn’t matter for your use-case.) In this case your attempt to empty the socket will turn into an infinite loop.
2. Add user-mode checks that duplicate the BPF’s logic and check all packets after receiving them. Duplicate logic leads to bugs. Enough said.¹

Now lets look at the textbook solution for fixing this bug - the same solution that libpcap uses:

struct sock_filter zero_bytecode = BPF_STMT(BPF_RET | BPF_K, 0);
struct sock_fprog zero_program = { 1, &zero_bytecode};

if (setsockopt(sock, SOL_SOCKET, SO_ATTACH_FILTER, &zero_program, sizeof(zero_program)) < 0) {
	printf("error attaching zero bpf: %d\n", errno);
	return 1;
}

char drain[1];
while (1) {
	int bytes = recv(sock, drain, sizeof(drain), MSG_DONTWAIT);
	if (bytes == -1) {
		// we assume the error here means there is nothing left to read from the socket which is exactly what we want
		break;
	}
}

// bpf_program is the actual bpf program we want to apply - just like in the previous example
int err = setsockopt(sock, SOL_SOCKET, SO_ATTACH_FILTER, &bpf_program, sizeof(bpf_program));
char buffer[MAX_PACKET_SIZE];
int n = recv(sock, buffer, MAX_PACKET_SIZE, NO_OPTIONS); // buffer will now always match the bpf

What we’re doing here is taking advantage of the fact that swapping one BPF out for another is an atomic operation: if you swap one BPF for another then at every moment either the first is in place or the second but never neither. We therefore start by applying the so-called “zero-BPF” which is a BPF that matches no packets. Then we empty out any packets that arrived before the “zero-BPF” filter was applied. At this point the socket is definitely empty and it can’t fill up with junk because the zero-BPF is in place. Then we replace the zero-BPF with the real BPF we want. Because the swap is atomic we know that any packet in the socket after that must match the real BPF.

Simple.

Work with me

Does this sort of thing interest you?

I started as a low-level engineer, but today I’m the co-founder and CEO of Robusta.dev and we’re hiring! I still do the occasional deep technical dive, as well as building a world-class team of excellent engineers and a product used by hundreds of companies.

If you join our team, you’ll work closely with me and be a core part of the founding team. Email natan@robusta.dev and mention you came from this post. We’re hiring in Israel as well as remote.