How to use a custom-built jemalloc with BIND on FreeBSD

Intro

jemalloc is a userspace memory management library which avoids memory fragmentation, performs well in multithreaded programs, and offers profiling. It has been integrated in the libc on FreeBSD. Whenever you call malloc() or free() in your program on FreeBSD, you talk to functions provided by the jemalloc library. Unfortunatelly, jemalloc as part of the libc on FreeBSD is compiled without support for memory profiling. You can proof this by starting any program with given MALLOC_CONF environment variable, e.g. from csh:

% env MALLOC_CONF="prof:true" ls /foobar
<jemalloc>: Invalid conf pair: prof:true
ls: /foobar: No such file or directory

Thus, we need to build a custom version of jemalloc.

Compilation

First, install these programs as they are needed to download and to build a vanilla copy of jemalloc:
```
% pkg install git gmake autoconf automake
```
Fetch jemalloc from Github:

% git clone https://github.com/jemalloc/jemalloc.git
% cd jemalloc

Checkout your desired version, e.g. latest 5.3:
```
% git checkout 5.3.0
```
To avoid any interference with the native jemalloc compiled into FreeBSD's libc, edit the file src/jemalloc.c and advance to lines 997 and 999:
```
                    JEMALLOC_CPREFIX"MALLOC_CONF"
#else
                    "MALLOC_CONF"
```
Replace both strings "MALLOC_CONF" by "MALLOX_CONF", e.g.
```
                    JEMALLOC_CPREFIX"MALLOX_CONF"
#else
                    "MALLOX_CONF"
```
This enables us to use the environment variable "MALLOX_CONF" to pass any option to our custom-built library only.
Now generate the GNU-style makefiles:
```
% sh autogen.sh --enable-prof --disable-cxx --prefix=/usr/local/jemalloc530
```
If you plan to link any program written in C++ against this library, remove the --disable-cxx option given to autogen.sh.
Now compile and install it:
```
% gmake
% gmake install
```
FreeBSD allows you to instruct the run-time linker to load arbitrary shared libraries when a program requires a certain dynamic object. This is configured in /etc/libmap.conf, which in turn includes any file found in /usr/local/etc/libmap.d. Thus, to allow a program to load our custom-build version of jemalloc:
```
% echo "libjemalloc.so.2 /usr/local/jemalloc530/lib/libjemalloc.so.2" > /usr/local/etc/libmap.d/jemalloc.conf
```
Of course, if you do not take care, libmap.conf allows you to render your operating system to an unusable state, which will require you to reboot in rescue mode most likely.

BIND

This section describes how to build BIND which uses our custom version of jemalloc. This is normally not needed on FreeBSD, since jemalloc is part of libc.

Configure the Bind 9.20 port:
```
% cd /usr/ports/dns/bind920
% make config
```
I chose just these options:
- IDN
- TCP_FASTOPEN
- STATS_XML
- GSSAPI_NONE
Now advance to the step where the ports system has just run the configure script of the Bind source code, i.e. right before any actual compilation will take place:
```
% make configure
```

Now call the configure script manually and have it pick up our custom-built jemalloc library. Both the option --enable-jemalloc and the environment variable PKG_CONFIG_PATH are crucial:

% env PKG_CONFIG_PATH=/usr/local/jemalloc530/lib/pkgconfig ./configure --enable-dnsrps --localstatedir=/var --sysconfdir=/usr/local/etc/namedb --with-openssl=/usr --with-readline=libedit --disable-tracing --disable-dnstap --disable-fixed-rrset --disable-geoip --without-maxminddb --without-gssapi --with-libidn2=/usr/local --disable-largefile --without-lmdb --disable-querytrace --without-json-c --with-libxml2 --enable-tcp-fastopen --prefix=/usr/local --mandir=/usr/local/share/man --disable-silent-rules --infodir=/usr/local/share/info/ --build=amd64-portbld-freebsd14.3 --with-jemalloc

Of course, any other option given to the configure script should reflect the options you set when calling make config for the port.

Now compile and install Bind:

% cd ../../
% make
% make install clean

Now you may start named from the command line, and provide any memory profiling options in the MALLOX_CONF environment variable:

% env MALLOX_CONF="prof:true,prof_prefix=/var/tmp/named_prof" /usr/local/sbin/named -t -u bind -c /usr/local/etc/namedb/named.conf