Dealing with zombie processes in containers

zombie container init golang

3 min read | by Jordi Prats

When we run a process in a container, it becomes the init process. This means that it is responsible for reaping any child processes that exit. If it doesn't do this, they become zombies.

Having some zombies are not a problem, but if you have too many of them, you can run out of resources (PIDs, memory, disk space...). This is what is going to cause issues in your host system: the zombies won't be a problem, but the resources they consume will.

A zombie process is a process that has completed execution but still has an entry in the process table. This occurs for child processes, where the entry is still needed to allow the parent process to read its exit status. Every time a process exits, it will become a zombie until the parent process reads the exit status but we rarely see them because the parent process handles the reaping by retrieving the process exit code (via the wait system call) It's implementation can be either via a handler for the SIGCHLD signal, or continuously checking for child processes that have exited.

A zombie process should not be confused with an orphan process: An orphan process is a process that is still executing, but whose parent has died. The orphan process is then adopted by the init process, which will wait for it to complete and then reap it.

In a container, the init process is the process that is started when the container starts: Since it might not have the code to reap zombies, orphan processes are going to become zombies.

We can prevent this from happening by using bash as the main process in the container: Bash happens to have a process reaper, so running a command under bash -c is going to protect us from this problem.

On the other hand, we can add the relevant code to the main process to reap zombies. Here's how we could do it using Golang.

We are going to use a goroutine that will use syscall.Wait4 with the syscall.WNOHANG to determine which zombie processes to reap, but we could implement the same using a SIGCHLD handler.

var (
  wg     sync.WaitGroup
  ctx    context.Context
  cancel context.CancelFunc
)

func reapZombies(ctx context.Context, wg *sync.WaitGroup) {
  for {
    var status syscall.WaitStatus

    // Wait for orphaned zombie process
    pid, _ := syscall.Wait4(-1, &status, syscall.WNOHANG, nil)

    if pid <= 0 {
      // no child waiting
      time.Sleep(1 * time.Second)
    } else {
      // a child was reaped
      continue
    }

    select {
    case <-ctx.Done():
      wg.Done()
      return
    default:
    }
  }
}

We don't need this code if we are not the init process, so we can conditionally start the reaper based on the current PID:

func StartReaper() {
  currentPid := os.Getpid()

  if currentPid == 1 {
    ctx, cancel = context.WithCancel(context.Background())
    wg.Add(1)
    go reapZombies(ctx, &wg)
  }
}

Finally, we'll need to cleanup the goroutine when the process exits by using the defer statement to a stop function:

func StopReaper() {
  // Signal zombie goroutine to stop
  cancel()
  // wait for it to release waitgroup
  wg.Wait()
}

So, we can just add the following to the main function:

func main() {

  (...)

  // zombie reaper
  reaper.StartReaper()
  defer reaper.StopReaper()

  (...)
}

Posted on 02/10/2024

Categories