Connection reset by peer

Overview

"Connection reset by peer" is a common networking issue that occurs when the connection between the client (your browser, the CLI, wherever you send commands from) and the server (Stardog) is unexpectedly closed by the server or an intermediary network device, like a load balancer. It means that the server has forcibly closed the connection while the client was still trying to communicate.

This error typically happens in long-running operations (e.g., database optimize) where the server or network device is configured with timeout settings. If the client takes too long to send data or keep the connection alive, the server or intermediary device may assume the connection has become idle and terminate it. When the client tries to send more data after this, it receives the "Connection reset by peer" error.

Some common causes of this error include:

• Inactivity or idle time during a long-running process.
• Misconfigured network devices that close connections after a set timeout period.
• Network or firewall policies that drop long-lived connections.

Specific instances of this problem and solutions

Load balancer timeout

In cloud environments, like Azure Kubernetes Service (AKS), this issue can arise when a Stardog operation takes longer than the load balancer's timeout threshold. When an operation takes longer than the threshold, the load balancer resets the connection, resulting in the error.

One way to fix this is to send keep-alive packets so the load balancer does not terminate the connection. [1] Kubernetes has keep-alive settings, but those only change the keep-alive timer without enabling it. To enable keep-alives, we need to modify the socket that establishes and maintains the connection between the client and Stardog.

We can do that with the following Python script:

# Define a custom HTTPAdapter to modify the default socket options
class KeepAliveAdapter(HTTPAdapter):
    def init_poolmanager(self, *args, **kwargs):
        # Initialize the pool manager and set keepalive options for new connections
        kwargs['socket_options'] = HTTPConnection.default_socket_options + [
            (socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),  # Enable keepalive
            (socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 150),  # Set idle time to 150 seconds
            # Optional settings for more control over keepalive:
            # (socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 60),  # Set interval between keepalive packets
            # (socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 5),    # Set number of keepalive probes before connection is closed
        ]
        super(KeepAliveAdapter, self).init_poolmanager(*args, **kwargs)

# Create a session with the custom adapter
session = requests.Session() # use this `session` object to issue Stardog commands rather than a `requests` object
adapter = KeepAliveAdapter()
session.mount("http://", adapter)
session.mount("https://", adapter)

# below is the rest of the code for your script

[1] This assumes your load balancer/proxy respects TCP keep-alive. Some, like AWS classic load balancers, do not. In this case, the only solution is to set the idle timeout to the max value, knowing Stardog received the command, and monitoring to see when the command finishes. You can do this monitoring by watching your stardog.log, running stardog-admin ps list, checking to see if your action had the intended effect (e.g., there is now data in your database), etc.

Other causes

If your instance of this issue isn't covered by this article, open a Support Ticket with a description of what you were doing when you got "Connection reset by peer" and a diagnostics report.