FreeBSD, ps, and processes with long argument lists

or: Why does ps ax just show [mysqld] instead of its full pathname and arguments?

I was logged into a FreeBSD box as non-root user and stumpled upon the output of ps ax (lines are truncated as I use a 120 characters terminal width): 

...
81976  ??  Ss        0:00.01 /bin/sh /usr/local/percona/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --use
82107  ??  S         0:00.12 [mysqld]
...
Process 82107 was missing its full pathname and arguments. Even if mysqld had called setproctitle(), the output would have been in a different format, e.g.: 
...
35454  ??  Ss        1:30.38 postgres: writer process    (postgres)
...
When called as user root or mysql, ps ax revealed the pathname and arguments: 
...
81976  ??  Is        0:00.01 /bin/sh /usr/local/percona/bin/mysqld_safe --defaults-extra-file=/var/db/mysql/my.cnf --use
82107  ??  I         0:00.22 /usr/local/percona/bin/mysqld --defaults-extra-file=/var/db/mysql/my.cnf --basedir=/usr/loc
...
My first thought was that Percona (a MySQL clone) is doing something nasty here. So I wrote a tiny shell script: 
#!/bin/sh
sleep 60
As user root, I started it with the same argument list that was given to mysqld: 
./test.sh --defaults-extra-file=/var/db/mysql/my.cnf --basedir=/usr/local/percona --datadir=/var/db/mysql --plugin-dir=/usr/local/percona/lib/mysql/plugin --user=mysql --log-error=/var/db/mysql/fbsd.intra.ogris.net.err --pid-file=/var/db/mysql/fbsd.intra.ogris.net.pid
While the script was sleeping, I logged in as normal user and called ps again: 
...
82942   2  S+        0:00.00 [sh]
82943   2  S+        0:00.00 sleep 30
...
I removed some command line arguments and called test.sh again: 
./test.sh --defaults-extra-file=/var/db/mysql/my.cnf --basedir=/usr/local/percona --datadir=/var/db/mysql --plugin-dir=/usr/local/percona/lib/mysql/plugin --user=mysql
Then, as unprivileged user, ps showed: 
...
83024   2  S+        0:00.00 /bin/sh ./test.sh --defaults-extra-file=/var/db/mysql/my.cnf --basedir=/usr/local/percona -
83025   2  S+        0:00.00 sleep 30
...
Obviously, the length of the command line is crucial. As ps ax always shows the full pathname and all arguments when called as user root, it might be a permission issue, too. So I poked around in the FreeBSD source code, mainly bin/ps, sys/kern, sys/sys (include headers), and lib/libkvm. Interestingly, FreeBSD caches a process'es command line options. From sys/sys/proc.h: 
struct proc {
        ...
        struct sysentvec *p_sysent;     /* (b) Syscall dispatch info. */
        ...
};
That struct sysentvec is defined in sys/sys/sysent.h: 
struct sysentvec {
        ...
        vm_offset_t     sv_psstrings;   /* PS_STRINGS */
        ...
};
The kern.ps_arg_cache_limit sysctl dictates how many characters of a process'es command line are cached: 
[fjo@fbsd ~]$ sysctl kern.ps_arg_cache_limit
kern.ps_arg_cache_limit: 256
The default value of kern.ps_arg_cache_limit is PAGE_SIZE/16, or 4096/16=256 on i386 and amd64.
ps uses libkvm to enumerate all running processes and their command line arguments. If the arguments of a process exceeds the cached entry, libkvm tries to read /proc/$PID/mem (where $PID denotes the process id of that particular process). /proc/$PID/mem is a virtual file which gives access to the memory area of a process and thus to its command line parameters. Naturally, this file may only be read by the user who runs the process, and root, but no one else. You can verify this: As user root, call the above shell with parameters that exceed 256 characters. While it's running, start truss ps ax as unprivileged user.