How do I troubleshoot a broken pipe error?

Created by Steve Place, Modified on Tue, Jul 23 at 12:51 PM by Steve Place

Broken pipes occur when the server cuts off communication with the client before the request between the server and client can complete. This can happen for many different of reasons, with one example being a load balancer or proxy terminating the connection between client and server after too much idle time has elapsed.


Some Stardog operations (such as db create or db restore) continue on the server side after the connection between server and client is severed. Once the connection is killed, you won't get a success/failure message for that operation from the client, so you'll have to monitor the server instead. You can do this by examining stardog.log or checking to see if the change you're expecting has been made (e.g., if the database has actually been created).


Below are some common causes of broken pipe errors and what to send to Support when you encounter a broken pipe error. See the appendix for a discussion of broken pipe errors vs. timeouts.


Common causes of broken pipes


Prometheus


If the stacktrace (in stardog.log) for your broken pipe error includes the following:

    at io.prometheus.client.exporter.common.TextFormat.writeEscapedHelp(TextFormat.java:69) ~[simpleclient_common-0.6.0.jar:?]  
    at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:26) ~[simpleclient_common-0.6.0.jar:?]  
    at com.complexible.stardog.protocols.http.server.StatusService.getPrometheusMetrics(StatusService.java:335) ~[stardog-protocols-http-server-10.1.0.jar:?]  
    at com.complexible.stardog.protocols.http.server.StatusService.prometheus(StatusService.java:235) ~[stardog-protocols-http-server-10.1.0.jar:?]

then the broken pipe is caused by Stardog's Prometheus endpoint trying to send more metrics than can be sent while the connection is open. If this is the case, you'll need to filter the metrics you request to only include the ones you need (to limit the amount of data sent while the connection is open).


You can see how to do that here.


Network issues 


Network interruptions are a common cause of broken pipe errors. 


As an example, if your infrastructure is using an AWS Classic Load Balancer, it will timeout idle connections between client and server after a default of 60 seconds. It will even terminate connections between a client and server using TCP keep-alive because data is not sent in the payload. You can increase the idle timeout to a max of 4000 seconds, per the AWS documentation.


By increasing the idle timeout, you will maximize the amount of time that can elapse before the load balancer breaks the connection between Stardog and its client.


Other load balancers or proxies typically have similar settings and may need to be investigated and adjusted accordingly.


Heavy load 


If Stardog is under heavy load, it may be unable to respond to requests in time, which can cause a broken pipe error.


What to send to Support


If you encounter a broken pipe error, send the following in a support ticket:

  1. The stardog.log containing this error and the stacktrace that follows it.
  2. The output of stardog-admin ps list when you're running an operation that results in a broken pipe error.
  3. A description of what actions you're taking when you get the broken pipe error.


Appendix


Broken pipes vs. timeouts

Broken Pipe

A broken pipe error occurs when one end of a communication channel (like a socket or a pipe) tries to write data, but the other end closes the connection. This typically happens when:

  • The process on the receiving end has terminated.
  • The socket connection has been closed unexpectedly.

Timeout

A timeout error occurs when an operation takes longer than the allowed or expected time limit. This can happen in various contexts, such as:

  • Network communications: When a connection attempt or read/write operation doesn't complete within a specified time period.
  • Database queries: When a query takes too long to execute.
  • Function execution: When a function exceeds its execution time limit.

Timeouts are designed to prevent indefinite waiting and to allow programs to handle situations where operations take longer than expected. They can be set for both network connections and other operations where waiting indefinitely would be undesirable.


Stardog has configurable timeout parameters for read queries, update queries, and databases.

Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Thank you for your feedback

Let us know how can we improve this article!

Select at least one of the reasons
CAPTCHA verification is required.

Feedback sent

We appreciate your effort and will try to fix the article