Skip to main content
Feedback

Debugging hung runs and connection timeouts

A Data Flow run that connects over a Secure Shell (SSH) tunnel to a database such as PostgreSQL can get stuck with these symptoms:

  • The run stays in Running indefinitely, with 0 bytes transferred and no output in the logs.
  • Canceling and manually rerunning the same Data Flow succeeds within minutes.
  • The issue typically affects only a small fraction of runs, clustered within a short window, such as your account's daily schedule tick.

These runs are commonly called zombie runs. In most cases, the root cause is on your network path, not in Data Integration. The checks in this guide let your infrastructure or database administration team confirm the cause and apply a fix without opening a support case.

Which layer is failing

A database connection that uses an SSH tunnel has two distinct points of failure. Understanding which one applies to your case determines which fix to apply.

SSH tunnel layer (bastion)

Either the bastion never establishes the SSH tunnel, or the connection between the bastion and the database silently drops. The ssh_tunnel_timeout_sec and ssh_tunnel_keepalive_sec connection settings cover this layer.

Database session layer

The tunnel is healthy, but the database accepts the socket and never responds. For PostgreSQL sources, the session-level connect_timeout setting and Transmission Control Protocol (TCP) keepalives on the extraction path cover this layer.

note

If your bastion is refusing or throttling new SSH connections, neither of the two protections above triggers. The connection never gets far enough to be instrumented. The checks in the next section are for this specific case.

Check for SSH connection throttling on the bastion

Run the following read-only commands on the bastion, in order, to determine whether it is throttling new SSH connections during your scheduled run window.

  1. Check the configured limits:

    sudo sshd -T | grep -iE 'maxstartups|maxsessions'
    grep -iE '^(MaxStartups|MaxSessions|LoginGraceTime)' /etc/ssh/sshd_config
  2. Check the number of live concurrent SSH connections:

    ss -tn state established '( sport = :22 )' | wc -l
  3. Watch the connection count during the scheduled burst window:

    watch -n 2 "ss -tn state established '( sport = :22 )' | wc -l"
  4. Check the SSH logs for evidence of past throttling, using the command for your operating system:

    • Debian or Ubuntu:

      sudo grep -i 'maxstartups' /var/log/auth.log*
    • RHEL, Amazon Linux, or CentOS:

      sudo grep -i 'maxstartups' /var/log/secure*
    • Journald-based systems, scoped to the incident window:

      sudo journalctl -u sshd --since "<start>" --until "<end>" | grep -iE 'maxstartups|refused|too many'

A line such as sshd[nnnn]: error: beginning MaxStartups throttling confirms that the bastion randomly refused new connections at that time.

Remediation: raise the bastion's MaxStartups limit

OpenSSH's default MaxStartups value is 10:30:60. This means the bastion accepts the first 10 unauthenticated connections freely, then randomly refuses a rising percentage of new connections, up to a hard ceiling of 60 simultaneous unauthenticated connections. An account that opens several hundred tunnels within a few-minute scheduled window can trip this limit.

  1. Validate the current configuration before making changes:

    sudo sshd -t
  2. Open /etc/ssh/sshd_config in a text editor and update the following values:

    MaxStartups 30:50:200      # OpenSSH default is 10:30:60
    MaxSessions 20
  3. Reload the SSH daemon to apply the change:

    sudo systemctl reload sshd

    If systemctl is not available, use sudo service sshd reload instead.

Other causes to rule out

If the bastion checks come back clean, also check for:

  • PostgreSQL connection drops, restarts, failovers, or idle and statement timeouts around the time of the hang.
  • Network or firewall idle timeouts between the bastion and the database.
  • Bastion load, configuration, or network changes around the time of the hang.

What to send Support if the checks come back clean

If none of the checks above explain the hang, open a support case and include:

  • The Data Flow name, environment, run ID, and hang timestamp (UTC).
  • The output of the sshd -T and log-grep commands from this guide for the incident window.
  • Whether a manual rerun of the same Data Flow succeeded.
On this Page