Stardog could not be initialized: Cannot list, system closed.

Created by Steve Place, Modified on Wed, Aug 7 at 5:17 PM by Steve Place

This article applies to Kubernetes users. If you're getting this error and you're not running Stardog in Kubernetes, please open a support ticket with your stardog.log and a description of what you were doing when this error occurred.

If you get an error message from your pod logs that looks like this:

Stardog could not be initialized: Cannot list, system closed.
java.lang.IllegalArgumentException: Cannot list, system closed.
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
    at com.complexible.stardog.AbstractSystemInfo.list(AbstractSystemInfo.java:127)
    at com.complexible.stardog.RdfSystemInfo.list(RdfSystemInfo.java:108)
    at com.complexible.stardog.StardogKernel.handleUnknownDatabaseDirectories(StardogKernel.java:2648)
    at com.complexible.stardog.StardogKernel.start(StardogKernel.java:2590)
    at com.complexible.stardog.StardogKernel.initialize(StardogKernel.java:840)
    at com.complexible.stardog.Kernel.initialize(Kernel.java:133)
    at com.complexible.stardog.Stardog.initKernel(Stardog.java:270)
    at com.complexible.stardog.Stardog.<init>(Stardog.java:257)
    at com.complexible.stardog.Stardog$StardogBuilder.create(Stardog.java:693)
    at com.complexible.stardog.cli.impl.ServerStart.call(ServerStart.java:190)
    at com.complexible.stardog.cli.impl.ServerStart.call(ServerStart.java:43)
    at com.complexible.stardog.cli.CLIBase.execute(CLIBase.java:56)
    at com.complexible.stardog.cli.admin.CLI.main(CLI.java:168)
Cannot list, system closed.
Waiting for running tasks to complete....done. Executor service has been shut down.
Stardog server X.Y.Z shutdown on [date].

This error is typically caused by a process sending the signal to shutdown Stardog because it's taking too long to come online.

Troubleshooting steps

The first step is to modify startupProbe (or add it if you don't have one) in statefulset.yaml. Our recommendations for its properties can be found in our open-source helm charts here, and our recommendations for its values can be found here here.

Setting the startupProbe sets an amount of time for Kubernetes to wait for Stardog to start before it (Kubernetes) sends the shutdown signal.

Common causes of Stardog taking a long time to start

Computing indexes

One cause of Stardog taking a long time to start is because Stardog is indexing literals. Stardog indexes literals when full-text search (FTS) is enabled (search.enabled=true). While that process occurs, you will see messages like this in your stardog.log:

INFO  2024-08-07 15:52:14,534 [Stardog.Executor-22] com.complexible.stardog.index.Index:printInternal(324): Indexing triples: 30% complete in 00:06.083 (1315.9K triples/sec)

If Stardog is taking a long time to start because you enabled FTS on one or more databases, you have two options:

Set your startupProbe to a high enough timeout to allow the indexes to compute. The default value in our helm charts is likely sufficient for this, but your server will be unusable until the indexes finish computing.
Perform the following sequence:
1. Start Stardog in safe mode.
2. Offline the databases with FTS enabled. stardog-admin db offline -- yourDb
3. Turn off FTS on all databases. stardog-admin metadata set -o search.enabled=false -- yourDb
4. Stop Stardog and start it again normally.
5. Turn on FTS for the databases you offlined. stardog-admin metadata set -o search.enabled=true -- yourDb
6. Online the databases you offlined. stardog-admin db online -- yourDb
  1. The onlining process won't occur until the indexes finish computing, so the databases will be unusable until that finishes.

Insufficient memory or CPU

This error can also be caused by Stardog trying to use more memory or CPU than Kubernetes is allowing it to use.

To see how to increase CPU, see this article: https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/

To see how to increase memory, see this article: https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/

The latter article has a section called "Exceed a container's memory limit" that walks through checking if a container was killed because it was using to much memory.