How do I troubleshoot a broken pipe error?

Broken pipes occur when the server cuts off communication with the client before the request between the server and client can complete. This can happen for many different of reasons, with one example being a load balancer or proxy terminating the connection between client and server after too much idle time has elapsed.

Some Stardog operations (such as db create or db restore) continue on the server side after the connection between server and client is severed. Once the connection is killed, you won't get a success/failure message for that operation from the client, so you'll have to monitor the server instead. You can do this by examining stardog.log or checking to see if the change you're expecting has been made (e.g., if the database has actually been created).

Below are some common causes of broken pipe errors and what to send to Support when you encounter a broken pipe error. See the appendix for a discussion of broken pipe errors vs. timeouts.

Common causes of broken pipes

Prometheus

If the stacktrace (in stardog.log) for your broken pipe error includes the following:

    at io.prometheus.client.exporter.common.TextFormat.writeEscapedHelp(TextFormat.java:69) ~[simpleclient_common-0.6.0.jar:?]  
    at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:26) ~[simpleclient_common-0.6.0.jar:?]  
    at com.complexible.stardog.protocols.http.server.StatusService.getPrometheusMetrics(StatusService.java:335) ~[stardog-protocols-http-server-10.1.0.jar:?]  
    at com.complexible.stardog.protocols.http.server.StatusService.prometheus(StatusService.java:235) ~[stardog-protocols-http-server-10.1.0.jar:?]

Network interruptions are a common cause of broken pipe errors. 

As an example, if your infrastructure is using an AWS Classic Load Balancer, it will timeout idle connections between client and server after a default of 60 seconds. It will even terminate connections between a client and server using TCP keep-alive because data is not sent in the payload. You can increase the idle timeout to a max of 4000 seconds, per the AWS documentation.

By increasing the idle timeout, you will maximize the amount of time that can elapse before the load balancer breaks the connection between Stardog and its client.

Other load balancers or proxies typically have similar settings and may need to be investigated and adjusted accordingly.

Heavy load 

If Stardog is under heavy load, it may be unable to respond to requests in time, which can cause a broken pipe error.

What to send to Support

If you encounter a broken pipe error, send the following in a support ticket:

1. The stardog.log containing this error and the stacktrace that follows it.
2. The output of stardog-admin ps list when you're running an operation that results in a broken pipe error.
3. A description of what actions you're taking when you get the broken pipe error.

Appendix

Broken pipes vs. timeouts

Broken Pipe

A broken pipe error occurs when one end of a communication channel (like a socket or a pipe) tries to write data, but the other end closes the connection. This typically happens when:

• The process on the receiving end has terminated.
• The socket connection has been closed unexpectedly.

Timeout

A timeout error occurs when an operation takes longer than the allowed or expected time limit. This can happen in various contexts, such as:

• Network communications: When a connection attempt or read/write operation doesn't complete within a specified time period.
• Database queries: When a query takes too long to execute.
• Function execution: When a function exceeds its execution time limit.

Timeouts are designed to prevent indefinite waiting and to allow programs to handle situations where operations take longer than expected. They can be set for both network connections and other operations where waiting indefinitely would be undesirable.

Stardog has configurable timeout parameters for read queries, update queries, and databases.