docs/Troubleshooting.md
Profiler allocates 8 kB perf_event buffer for each thread of the target process.
The above error may appear if the total size of perf_event buffers (8 * threads kB)
exceeds locked memory limit. This limit is comprised of ulimit -l plus
the value of kernel.perf_event_mlock_kb sysctl multiplied by the number of CPU cores.
For example, on a 16-core machine, ulimit -l 65536 and kernel.perf_event_mlock_kb=516
is enough for profiling (65536 + 516*16) / 8 = 9224 threads.
If an application has more threads, increase one of the above limits, or native stacks
will not be collected for some threads.
A privileged process is not subject to the locked memory limit.
Due to limitation of HotSpot Dynamic Attach mechanism, the profiler must be run
by exactly the same user (and group) as the owner of target JVM process.
If profiler is run by a different user, it will try to automatically change
current user and group. This will likely succeed for root, but not for
other users, resulting in the above error.
The profiler cannot establish communication with the target JVM through UNIX domain socket. Usually this happens in one of the following cases:
Attach socket /tmp/.java_pidNNN has been deleted. It is a common
practice to clean /tmp automatically with some scheduled script.
Configure the cleanup software to exclude .java_pid* files from deletion.
lsof -p PID | grep java_pid. If it lists a socket file, but the file does not exist, then this is exactly
the described problem.JVM is started with -XX:+DisableAttachMechanism option.
/tmp directory of Java process is not physically the same directory
as /tmp of your shell, because Java is running in a container or in
chroot environment. asprof attempts to solve this automatically,
but it might lack the required permissions to do so.
strace asprof PID jcmdJVM is busy and cannot reach a safepoint. For instance, JVM is in the middle of long-running garbage collection.
kill -3 PID. Healthy JVM process should print
a thread dump and heap info in its console.The connection with the target JVM has been established, but JVM is unable to load profiler shared library.
Make sure the user of JVM process has permissions to access libasyncProfiler.so by exactly the same absolute path.
For more information see #78.
perf_event_open() syscall has failed. Typical reasons include:
/proc/sys/kernel/perf_event_paranoid is set to restricted mode (>=2).perf_event_open API in a container.For permissions-related reasons (such as 1 and 2), using --fdtransfer while running the profiler
as a privileged user may solve the issue.
If changing the configuration is not possible, you may fall back to
-e ctimer profiling mode. It is similar to cpu mode, but does not
require perf_events support. As a drawback, there will be no kernel
stack traces.
The OpenJDK debug symbols are required for allocation profiling for applications developed with JDK prior to 11. See Installing Debug Symbols for more details. If the error message persists after a successful installation of the debug symbols, it is possible that the JDK was upgraded when installing the debug symbols. In this case, profiling any Java process which had started prior to the installation will continue to display this message, since the process had loaded the older version of the JDK which lacked debug symbols. Restarting the affected Java processes should resolve the issue.
JVM shared library does not export gHotSpotVMStructs* symbols -
apparently this is not a HotSpot JVM. Sometimes the same message
can be also caused by an incorrectly built JDK
(see #218).
In these cases installing JDK debug symbols may solve the problem.
Async-profiler was unable to parse non-Java function names because of
the corrupted contents in /proc/[pid]/maps. The problem is known to
occur in a container when running Ubuntu with Linux kernel 5.x.
This is the OS bug, see https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1843018.
Output file is written by the target JVM process, not by the profiler script.
Make sure the path specified in -f option is correct and is accessible by the JVM.
No Java stacks will be collected if -XX:MaxJavaStackTraceDepth is zero
or negative. The exception is --cstack vm mode, which does not take
MaxJavaStackTraceDepth into account.
Too short profiling interval may cause continuous interruption of heavy
system calls like clone(), so that it will never complete;
see #97.
The workaround is simply to increase the interval.
When agent is not loaded at JVM startup (by using -agentpath option) it is
highly recommended to use -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints JVM flags.
Without those flags the profiler will still work correctly but results might be
less accurate. For example, without -XX:+DebugNonSafepoints there is a high chance
that simple inlined methods will not appear in the profile. When the agent is attached at runtime,
CompiledMethodLoad JVMTI event enables debug info, but only for methods compiled after attaching.
On most Linux systems, perf_events captures call stacks with a maximum depth
of 127 frames. On recent Linux kernels, this can be configured using
sysctl kernel.perf_event_max_stack or by writing to the
/proc/sys/kernel/perf_event_max_stack file.
You will not see the non-Java frames preceding the Java frames on the
stack, unless --cstack vmx is specified.
For example, if start_thread called JavaMain and then your Java
code started running, you will not see the first two frames in the resulting
stack. On the other hand, you will see non-Java frames (user and kernel)
invoked by your Java code.
macOS profiling is limited to user space code only.