Diagnosing and reporting critical errors

It is essential to gather as much as possible details about an issue once it appears. For this purpose, we provide utility scripts that generate such issue reports by collecting data from various log files, JVM etc. Using those issue reports helps us to investigate and provide an appropriate solution as quickly as possible.

Logs

GraphDB uses slf4j for logging through the Logback implementation (the Sesame facilities for log configuration discovery are no longer used). Instead, the whole distribution has a central place for the logback.xml configuration file in GraphDB-HOME/conf/logback.xml. If you use the war file setup, you can provide the log file location through a system parameter or we will pick it up from the generated war file.

Note

Check the the Logback configuration location rules for more information.

On startup, GraphDB logs the logback configuration file location:

[INFO ] 2016-03-17 17:29:31,657 [main | ROOT] Using 'file:/opt/graphdb-ee/conf/logback.xml' as logback's configuration file for graphdb

Setting up the root logger

The default ROOT logger is set to INFO. You can change it in several ways:

  • Edit the logback.xml configuration file.

    Note

    You don’t have to restart the database as it will check the file for changes every 30 seconds and will reconfigure the logger.

  • Change the log level through the logback JMX configurator. For more information see the Logback manual chapter 10.

  • Start each component with graphdb.logger.root.level set to your desired root logging level. For example:

bin/graphdb -Dgraphdb.logger.root.level=WARN

Logs location

By default, all database components and tools log in GraphDB-HOME/logs, when run from the bin folder. If you setup GraphDB by deploying .war files into a stand-alone servlet container, the following rules apply:

  1. To log in a specified directory, set the logDestinationDirectory system property.
  2. If GraphDB is run in Tomcat, the logs can be find in ${catalina.base}/logs/graphdb.
  3. If GraphDB is run in Jetty, the logs can be find in ${jetty.base}/logs/graphdb.
  4. Otherwise, all logs are in the logs subdirectory of the current working directory for the process.

Log files

Different things are logged in different files. This should make it easier to follow what is going on in different parts of the system

  • http-log.log - contains the HTTP communication between the master and the workers.
  • query-log.log - contains all queries that were sent to the database. The format is machine readable and allows us to replay the queries when debugging a problem.
  • main.log - contains all messages coming from the main part of the engine.

Report script

The Report script gathers valuable information for tracing and reproducing an issue.

Note

The report script is designed to collect logs only when GraphDB is run in Tomcat!

Note

If the issue is related to the cluster, an information must be provided for every worker and master.

Tip

If possible, run the report script while the issue is taking place.

report is a bash script that accepts a Tomcat’s pid, automatically gathers most of the necessary information and produces a bug report. It can be found in the bin folder in the GraphDB distribution.

To start the script run:

./report <tomcat's-pid>

The reproduced issue report contains following information:

✅ Logs files.
✅ jstack <pid> from Tomcat.
✅ jmap -histo <pid> from Tomcat.
✅ jstat -gcutil <pid> 1000 100 - (optional) it makes 100 gc snapshots one every second. A much better option is to give GC logs.
❌ If possible, start the JVM with the following GC logs parameters:
-XX:NumberOfGCLogFiles=5 -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCCause -XX:+PrintGCDetails
  -XX:-PrintTenuringDistribution -XX:+UseCompressedOops -XX:+UseGCLogFileRotation -Xloggc:<file>
then change the file destination to something that is writeable and provide these log files.
✅ tail -10000 from the syslog (on most machines, this is located in /var/log/syslog).
✅ If anything is touched in the Tomcat’s conf directory, it must be provided as is.
❌ If this is an out of memory issue, then start java with -XX:+HeapDumpOnOutOfMemoryError and try to reproduce. Then, provide the heap dump, generated by the JVM.
❌ If this is a cluster, check that there is NTP on the machines (i.e., their time is synchonised).
❌ If the issue is reproduced on a client dataset, it will help us to have access to it.
✅ Output from bin/versions.sh in Tomcat.
✅ The version of graphdb.
❌ Logs from the client. We need this to check for errors from the Sesame client.

Hint

Legend:
✅ Information gathered by the script.
❌ Information that is not supported by the script and you need to gather it separately.

Requirements

  • bash installed;
  • run the script with either the Tomcat’s user or a user that has equal read/write permissions.

Example

Before running the script, you might need to give it executable permission chmod +x report

Without parameters

If no parameters are given, a help message will be displayed:

$ report
The program accepts a single argument, tomcat's pid
Usage: ./report <tomcat's-pid>

Real run

You need the Tomcat’s pid. On most setups you can easily check the Tomcat’s pid with jps

$ jps | grep Bootstrap
32053 Bootstrap

you can see that Tomcat is running with pid 32053. Now run the report script:

$ ./report 32053
Picked up _JAVA_OPTIONS: -Dawt.useSystemAAFontSettings=lcd -Dswing.aatext=true
Found tomcat home as /home/myname/Downloads/apache-tomcat-7.0.55
Found aduna base as /home/myname/.graphdb
Did you change files in tomcat's conf directory?[y|n] (n): y
Getting tomcat version information
Getting graphdb version information
Collecting tomcat runtime parameters
Copying tomcat logs from /home/myname/Downloads/apache-tomcat-7.0.55/logs
Copying aduna logs from /home/myname/.graphdb/logs
Waiting for jstat to finish
You can find the collected data in 32053-data

first the script asks you if this is a default Tomcat configuration: Did you change files in Tomcat's conf directory?[y|n] y will copy the Tomcat config for you. It will also tell you where to find the produced diagnostic files (You can find the collected data in 32053-data), which varies from pid to pid. When reporting an issue, just zip this directory and send it to us.