We have a C++ shared library that uses ZeroC's Ice library for RPC and unless we shut down Ice's runtime, we've observed child processes hanging on random mutexes.
The Ice runtime starts threads, has many internal mutexes and keeps open file descriptors to servers.
Additionally, we have a few of mutexes of our own to protect our internal state.
Our shared library is used by hundreds of internal applications so we don't have control over when
the process calls fork(), so we need a way to safely shutdown Ice and lock our mutexes while
the process forks.
Reading
the POSIX standard on pthread_atfork() on handling mutexes and internal state:
Alternatively, some libraries might have been able to supply just a child routine that reinitializes
the mutexes in
the library and all associated states to some known value (for example, what it was when
the image was originally executed). This approach is not possible, though, because implementations are allowed to fail *_init() and *_destroy() calls for mutexes and locks if
the mutex or lock is still locked. In this case,
the child routine is not able to reinitialize
the mutexes and locks.
On Linux,
the this test C program returns EPERM from pthread_mutex_unlock() in
the child pthread_atfork() handler. Linux requires adding _NP to
the PTHREAD_MUTEX_ERRORCHECK macro for it to compile.
This program is linked from this good thread.
Given that it's technically not safe or legal to unlock or destroy a mutex in
the child, I'm thinking it's better to have pointers to mutexes and then have
the child make new pthread_mutex_t on
the heap and leave
the parent's mutexes alone, thereby having a small memory leak.
The only issue is how to reinitialize
the state of
the library and I'm thinking of reseting a pthread_once_t. Maybe because POSIX has an initializer for pthread_once_t that it can be reset to its initial state.
#include <pthread.h>
#include <stdlib.h>
#include <string.h>
static pthread_once_t once_control = PTHREAD_ONCE_INIT;
static pthread_mutex_t *mutex_ptr = 0;
static
void
setup_new_mutex()
{
mutex_ptr = malloc(sizeof(*mutex_ptr));
pthread_mutex_init(mutex_ptr, 0);
}
static
void
prepare()
{
pthread_mutex_lock(mutex_ptr);
}
static
void
parent()
{
pthread_mutex_unlock(mutex_ptr);
}
static
void
child()
{
// Reset
the once control.
pthread_once_t once = PTHREAD_ONCE_INIT;
memcpy(&once_control, &once, sizeof(once_control));
setup_new_mutex();
}
static
void
init()
{
setup_new_mutex();
pthread_atfork(&prepare, &parent, &child);
}
int
my_library_call(int arg)
{
pthread_once(&once_control, &init);
pthread_mutex_lock(mutex_ptr);
// Do something here that requires
the lock.
int result = 2*arg;
pthread_mutex_unlock(mutex_ptr);
return result;
}
In
the above sample in
the child() I only reset
the pthread_once_t by making a copy of a fresh pthread_once_t initialized with PTHREAD_ONCE_INIT. A new pthread_mutex_t is only created when
the library function is invoked in
the child process.
This is hacky but maybe
the best way of dealing with this skirting
the standards. If
the pthread_once_t contains a mutex then
the system must have a way of initializing it from its PTHREAD_ONCE_INIT state. If it contains a pointer to a mutex allocated on
the heap than it'll be forced to allocate a new one and set
the address in
the pthread_once_t. I'm hoping it doesn't use
the address of
the pthread_once_t for anything special which would defeat this.
Searching
comp.programming.threads group for pthread_atfork() shows a lot of good discussion and how little
the POSIX standards really provides to solve this problem.
There's also
the issue that one should only call async-signal-safe functions from pthread_atfork() handlers, and it appears
the most important one is
the child handler, where only a memcpy() is done.
Does this work? Is there a better way of dealing with
the requirements of our shared library?
