megacolorboy

Abdush Shakoor's Weblog

Writings, experiments & ideas.

Check connectivity from a Node.js application pod

I've deployed quite a few Node.js applications over the past three months. Sometimes I just want to do a basic connectivity check from the application's container without installing another tool.

Since Node.js is already there, why not use it?

For a hostname lookup using the container's operating-system resolver, try this:

kubectl exec web-7c9d6f8b5d-k2m4p -n staging -c app -- node -e "require('node:dns').lookup('example.com', { all: true }, (error, addresses) => { if (error) { console.error(error); process.exitCode = 1; return; } console.log(addresses); })"

Replace the pod name, namespace, container name, and hostname with your own. An illustrative result might look like this:

[ { address: '192.0.2.10', family: 4 } ]

That address is only an example, not the actual address of example.com. Also, dns.lookup() uses the system's name-resolution facilities, which may include /etc/hosts; it isn't necessarily a direct DNS query. The Node.js DNS documentation explains the distinction.

To go a step further and make an HTTPS request, use a Node.js runtime with built-in fetch and AbortSignal.timeout support, such as Node.js 18 or later:

kubectl exec web-7c9d6f8b5d-k2m4p -n staging -c app -- node -e "fetch('https://example.com', { signal: AbortSignal.timeout(10000) }).then(response => { console.log('HTTP status:', response.status); if (!response.ok) process.exitCode = 1; }).catch(error => { console.error(error); process.exitCode = 1; })"

An example response is:

HTTP status: 200

This checks more than name resolution: the request must also connect and complete TLS negotiation before receiving an HTTP response. A 401 or 403 still shows that an HTTP server responded, even though access wasn't granted. See the Node.js fetch documentation.

These commands start a new Node.js process, so they won't automatically reproduce proxy settings or custom HTTP clients configured inside your running application.

Hope you found this tip useful!

Test port connectivity using curl's Telnet support

Honestly, I discovered this for the first time recently. I had always used telnet to check whether I could connect to a port, but I never knew curl could do this too.

Here's a quick example:

curl -v --connect-timeout 5 --max-time 10 telnet://mail.example.com:25

Replace the hostname and port with the service you're testing. The connection timeout gives it five seconds to establish a connection, while --max-time limits the whole session to ten seconds.

An illustrative excerpt from a reachable SMTP server might look like this:

*   Trying 192.0.2.25:25...
* Connected to mail.example.com (192.0.2.25) port 25
220 mail.example.com ESMTP Postfix

The Connected line confirms that the TCP connection opened. In this example, the SMTP banner also tells us that the server responded. Other services might stay silent until you send something.

Don't be surprised if the command eventually reports a timeout: the TCP connection may have succeeded and then remained open until the ten-second limit. Read the connection messages instead of relying only on the final exit code.

This doesn't verify authentication, TLS, or the health of an entire application. It's just a handy first check when troubleshooting connectivity.

If your curl build reports that Telnet isn't supported, run curl --version and check its protocol list. The curl tutorial includes more about its Telnet support.

Hope you found this useful!

Find out why a Kubernetes pod keeps crashing

If you've been deploying applications to Kubernetes for a while, you've probably come across CrashLoopBackOff. Seeing it is annoying, but the useful part is finding out what happened just before the container stopped.

Here's how I would start:

kubectl get pod web-7c9d6f8b5d-k2m4p -n staging

Replace the pod name and namespace with your own. For example, the output might be:

NAME                  READY   STATUS             RESTARTS      AGE
web-7c9d6f8b5d-k2m4p   0/1     CrashLoopBackOff   5 (34s ago)   8m

That shows the symptom. To investigate it, get the pod's details:

kubectl describe pod web-7c9d6f8b5d-k2m4p -n staging

Look at the container's state, last termination reason, exit code, and the events near the bottom. An illustrative excerpt could look like this:

State:          Waiting
  Reason:       CrashLoopBackOff
Last State:     Terminated
  Reason:       Error
  Exit Code:    1

Next, check the application logs:

kubectl logs web-7c9d6f8b5d-k2m4p -n staging -c app --tail=100

If the container has already restarted, the previous instance's logs can be more useful:

kubectl logs web-7c9d6f8b5d-k2m4p -n staging -c app --previous --tail=100

Here, app is the container name. The --previous flag retrieves logs from its previous terminated instance when available; it isn't a history of every restart. See the kubectl logs reference.

For example, your application might have logged:

Error: Missing required environment variable DATABASE_URL

Now you have something specific to investigate: the application's configuration.

If you see ImagePullBackOff or ErrImagePull instead, the image couldn't be pulled, so application logs may not exist yet. Start with the events from describe and check the image name, tag, registry access, and image-pull credentials. The Kubernetes debugging guide covers these checks.

Hope you found this tip useful!

Send a test email from Kubernetes using curl

When troubleshooting email delivery, I sometimes want to send a test message from inside the cluster. This helps narrow down whether the SMTP server is reachable from that environment.

For a quick test, you can start a temporary pod with curl installed:

kubectl run smtp-test -n staging --rm -it --restart=Never --image=curlimages/curl --command -- sh

Replace staging with your namespace. This creates a separate pod. --command selects the shell, and --restart=Never prevents the container from restarting when it exits. During a normal attached session, --rm cleans up the pod when you're done. See the kubectl run reference.

Inside the shell, send a message through your internal SMTP relay:

curl -v --connect-timeout 5 --max-time 30 \
  --url smtp://mail.example.com:25 \
  --mail-from 'sender@example.com' \
  --mail-rcpt 'recipient@example.com' \
  --crlf --upload-file - <<'EOF'
From: sender@example.com
To: recipient@example.com
Subject: Test email from Kubernetes

This is a test message.
EOF

Replace the hostname and addresses with real ones before running it. This example assumes an internal relay that allows unauthenticated, unencrypted SMTP from the test pod.

For an authenticated server using STARTTLS on port 587, use these options in the same command:

--url smtp://mail.example.com:587 --ssl-reqd --user 'smtp-user'

With only a username supplied, curl prompts for the password. For implicit TLS, use smtps://mail.example.com:465 instead. These options are described in the curl manual.

An illustrative excerpt after uploading the message might look like this:

< 250 2.0.0 Ok: queued as A1B2C3D4
> QUIT
< 221 2.0.0 Bye

That means the relay accepted the message, not that it has reached the recipient's inbox. Check delivery logs if it never arrives.

Also, a separate test pod may have different labels and network policies from your application. Treat this as a test from that pod, rather than proof that the application has identical access.

Type exit to finish. If the session was interrupted and the pod remains, remove it with kubectl delete pod smtp-test -n staging.

Hope you found this useful!

Restart Kubernetes deployments using kubectl

I usually restart application pods using Argo CD or Rancher. That's convenient for one or two applications, but when there are several deployments to restart, I'd rather do it from the terminal.

First, check the current cluster context and the deployments in your namespace:

kubectl config current-context
kubectl get deployments -n staging

Then trigger a restart for one deployment:

kubectl rollout restart deployment/web -n staging

An example response is:

deployment.apps/web restarted

Replace web and staging with your deployment and namespace. The response confirms that the restart was requested, so I also watch its progress:

kubectl rollout status deployment/web -n staging --timeout=120s

Once complete, you might see:

deployment "web" successfully rolled out

To restart all deployments in that namespace:

kubectl rollout restart deployment -n staging

Or narrow it down using a label that exists on your deployments:

kubectl rollout restart deployment -n staging -l app=web

One distinction worth remembering: this replaces the pods managed by the selected deployments. It doesn't restart every kind of pod in the namespace. Replacement behavior follows each deployment's strategy, so availability still depends on its configuration and available capacity.

You can find more examples in the kubectl rollout reference.

Hope you found this useful!

Connect to MySQL using a Kubernetes debug container

What if you wanted to connect to a database server from an application's pod, but the application image didn't include a MySQL client?

You can add an ephemeral container with the client and run it directly. Here's a quick way to do it:

kubectl debug -it web-7c9d6f8b5d-k2m4p -n staging \
  --image=mysql:8.4 -- \
  mysql --host=mysql.staging.svc.cluster.local --port=3306 --user=app_reader --password

Replace the pod, namespace, database hostname, and username with your own values. This example uses the MySQL image to run its client; it doesn't start a database server.

The client prompts you for a password. Once connected, you can run a small query:

SELECT CURRENT_USER(), DATABASE();

For example:

+----------------+------------+
| CURRENT_USER() | DATABASE() |
+----------------+------------+
| app_reader@%   | NULL       |
+----------------+------------+
1 row in set (0.00 sec)

Here, NULL means no default database has been selected. To select one and list its tables:

USE app_db;
SHOW TABLES;

The debug container shares the pod's network, but it doesn't automatically inherit the application's environment variables, credentials, or mounted certificates. Supply the connection settings yourself, including your database's required TLS options.

A successful connection verifies this client's network access and authentication. Your application may still have a different configuration.

Type exit to leave the MySQL client. The ephemeral container stops, although its record remains attached to the pod. You also need permission to add debug containers; see the Kubernetes debugging guide.

Hope you found this tip useful!

Debug pod networking with ephemeral containers

Ever needed to check whether your application pod could reach a database or an HTTP endpoint, only to find that it didn't have curl, telnet, or any other useful tools installed?

Your first instinct might be to install the missing packages inside the application container. But there's a handy alternative: an ephemeral debug container.

This adds a separate container to the existing pod, sharing its network namespace. You can investigate connectivity without adding packages to the application's image. It still uses resources in the pod, so it isn't completely without impact.

I found an image called Netshoot that includes tools such as curl, dig, and nc. Here's how you can open a debugging shell:

kubectl debug -it web-7c9d6f8b5d-k2m4p -n staging --image=nicolaka/netshoot -- /bin/bash

Replace the pod name and namespace with your own. Your cluster must allow ephemeral containers and the chosen image.

After startup messages, an illustrative shell prompt might look like this:

web-7c9d6f8b5d-k2m4p:~#

Inside that shell, try resolving a database service or checking an HTTP endpoint:

dig +short mysql.staging.svc.cluster.local
curl -I --connect-timeout 5 --max-time 10 https://example.com

An example DNS response would be:

10.96.42.18

Use your own service hostname; this example assumes the cluster's DNS domain is cluster.local.

If you also need to inspect another container's processes, add --target=app to the debug command, replacing app with that container's name. This depends on runtime support and isn't needed just to share the pod's network.

When you're done, type exit. The debug container stops, but its entry remains in the pod until the pod is removed. It isn't a regular sidecar, and it doesn't automatically restart. The ephemeral container documentation explains these lifecycle restrictions.

Pretty cool, huh?

Copy files to and from Kubernetes pods

Every now and then, I need to copy files into an application pod. One use case for me is moving uploaded assets while migrating an application from staging to production.

For this, you can use kubectl cp. The remote path uses a namespace and pod name, so there's no SSH username or server address involved.

Here's an example of copying a local file into a container:

kubectl cp ./logo.png staging/web-7c9d6f8b5d-k2m4p:/var/www/html/public/logo.png -c app

Replace staging, the pod name, app, and the paths with your own values. The destination directory must exist and be writable by the container's user.

You can also do the reverse and copy a file into your current directory:

kubectl cp staging/web-7c9d6f8b5d-k2m4p:/var/www/html/public/logo.png ./logo.png -c app

For a directory of uploaded assets, I would copy it locally first, then into the destination pod:

kubectl cp staging/web-7c9d6f8b5d-k2m4p:/var/www/html/public/uploads ./uploads -c app
kubectl cp ./uploads production/web-5f7c8d9b6-m3q8r:/var/www/html/public/ -c app

Use a fresh local destination and check the target directory before copying, as existing files can be overwritten.

One detail that's easy to miss: kubectl cp needs tar inside the container. Without it, the copy fails. The kubectl cp reference also covers cases that need a more explicit tar-based copy.

Also, think about where these files will live. Files written into a container's writable layer won't survive its replacement. For uploaded assets that need to persist, use the application's persistent volume or shared storage, and coordinate the copy if uploads are still changing.

Hope you found this useful!