megacolorboy

Abdush Shakoor's Weblog

Writings, experiments & ideas.

Connect to MySQL using a Kubernetes debug container

What if you wanted to connect to a database server from an application's pod, but the application image didn't include a MySQL client?

You can add an ephemeral container with the client and run it directly. Here's a quick way to do it:

kubectl debug -it web-7c9d6f8b5d-k2m4p -n staging \
  --image=mysql:8.4 -- \
  mysql --host=mysql.staging.svc.cluster.local --port=3306 --user=app_reader --password

Replace the pod, namespace, database hostname, and username with your own values. This example uses the MySQL image to run its client; it doesn't start a database server.

The client prompts you for a password. Once connected, you can run a small query:

SELECT CURRENT_USER(), DATABASE();

For example:

+----------------+------------+
| CURRENT_USER() | DATABASE() |
+----------------+------------+
| app_reader@%   | NULL       |
+----------------+------------+
1 row in set (0.00 sec)

Here, NULL means no default database has been selected. To select one and list its tables:

USE app_db;
SHOW TABLES;

The debug container shares the pod's network, but it doesn't automatically inherit the application's environment variables, credentials, or mounted certificates. Supply the connection settings yourself, including your database's required TLS options.

A successful connection verifies this client's network access and authentication. Your application may still have a different configuration.

Type exit to leave the MySQL client. The ephemeral container stops, although its record remains attached to the pod. You also need permission to add debug containers; see the Kubernetes debugging guide.

Hope you found this tip useful!

Debug pod networking with ephemeral containers

Ever needed to check whether your application pod could reach a database or an HTTP endpoint, only to find that it didn't have curl, telnet, or any other useful tools installed?

Your first instinct might be to install the missing packages inside the application container. But there's a handy alternative: an ephemeral debug container.

This adds a separate container to the existing pod, sharing its network namespace. You can investigate connectivity without adding packages to the application's image. It still uses resources in the pod, so it isn't completely without impact.

I found an image called Netshoot that includes tools such as curl, dig, and nc. Here's how you can open a debugging shell:

kubectl debug -it web-7c9d6f8b5d-k2m4p -n staging --image=nicolaka/netshoot -- /bin/bash

Replace the pod name and namespace with your own. Your cluster must allow ephemeral containers and the chosen image.

After startup messages, an illustrative shell prompt might look like this:

web-7c9d6f8b5d-k2m4p:~#

Inside that shell, try resolving a database service or checking an HTTP endpoint:

dig +short mysql.staging.svc.cluster.local
curl -I --connect-timeout 5 --max-time 10 https://example.com

An example DNS response would be:

10.96.42.18

Use your own service hostname; this example assumes the cluster's DNS domain is cluster.local.

If you also need to inspect another container's processes, add --target=app to the debug command, replacing app with that container's name. This depends on runtime support and isn't needed just to share the pod's network.

When you're done, type exit. The debug container stops, but its entry remains in the pod until the pod is removed. It isn't a regular sidecar, and it doesn't automatically restart. The ephemeral container documentation explains these lifecycle restrictions.

Pretty cool, huh?

Copy files to and from Kubernetes pods

Every now and then, I need to copy files into an application pod. One use case for me is moving uploaded assets while migrating an application from staging to production.

For this, you can use kubectl cp. The remote path uses a namespace and pod name, so there's no SSH username or server address involved.

Here's an example of copying a local file into a container:

kubectl cp ./logo.png staging/web-7c9d6f8b5d-k2m4p:/var/www/html/public/logo.png -c app

Replace staging, the pod name, app, and the paths with your own values. The destination directory must exist and be writable by the container's user.

You can also do the reverse and copy a file into your current directory:

kubectl cp staging/web-7c9d6f8b5d-k2m4p:/var/www/html/public/logo.png ./logo.png -c app

For a directory of uploaded assets, I would copy it locally first, then into the destination pod:

kubectl cp staging/web-7c9d6f8b5d-k2m4p:/var/www/html/public/uploads ./uploads -c app
kubectl cp ./uploads production/web-5f7c8d9b6-m3q8r:/var/www/html/public/ -c app

Use a fresh local destination and check the target directory before copying, as existing files can be overwritten.

One detail that's easy to miss: kubectl cp needs tar inside the container. Without it, the copy fails. The kubectl cp reference also covers cases that need a more explicit tar-based copy.

Also, think about where these files will live. Files written into a container's writable layer won't survive its replacement. For uploaded assets that need to persist, use the application's persistent volume or shared storage, and coordinate the copy if uploads are still changing.

Hope you found this useful!

Find a Kubernetes pod and open a shell inside it

At my new job as a Senior DevOps engineer, I have learnt a lot in a short amount of time. One of the first things I got comfortable with was finding an application's pod and opening a shell inside its container.

This is quite useful when you want to check file permissions or explore what is actually running inside your application container.

First, get the pods in your namespace:

kubectl get pods -n staging

For example, you might see:

NAME                   READY   STATUS    RESTARTS   AGE
web-7c9d6f8b5d-k2m4p    1/1     Running   0          2h
worker-6b8f4d7c9-x5n2q  1/1     Running   0          2h

Here, staging is an example namespace. Replace it and the pod names with your own throughout this post. Also, make sure kubectl is connected to the intended cluster.

Once you have the pod name, open a shell:

kubectl exec -it web-7c9d6f8b5d-k2m4p -n staging -- /bin/sh

The -i flag keeps standard input open, and -t allocates a terminal. The -- separates kubectl's options from the command you want to run inside the container.

If the pod has multiple containers, choose one explicitly:

kubectl exec -it web-7c9d6f8b5d-k2m4p -n staging -c app -- /bin/sh

The shell depends on the image. /bin/sh is common, while /bin/bash only works if Bash is installed. Some minimal images have no shell at all; in that case, a debug container can help.

Type exit when you're done. This closes your shell session; it doesn't stop the application.

Hope you found this tip useful!

Enable password asterisks for sudo on Fedora

After switching from Linux Mint to Fedora, one small difference I noticed was the password prompt behavior in the terminal. On Mint, sudo showed asterisks while typing the password. On Fedora, it stayed blank.

Some people prefer the blank prompt, but I like having visual feedback when typing, especially if I am not fully sure whether I missed a key.

If you want the same behavior on Fedora, you can enable sudo password feedback with a small sudoers change.

This note applies to sudo prompts in the terminal, not graphical login screens.

1. Open the sudoers file safely

Do not edit /etc/sudoers directly with a normal text editor. Use visudo so syntax errors are caught before the file is saved:

sudo visudo

2. Enable password feedback

Add this line:

Defaults pwfeedback

If you already have other Defaults lines such as Defaults env_reset, leave them as they are. Just add Defaults pwfeedback as a separate line.

3. Save and test

Save the file and exit the editor. Then run a sudo command again, for example:

sudo -k
sudo true

When prompted for your password, you should now see asterisks as you type. This setting makes password length visible on screen. If that matters in your environment, leave the default behavior in place.

Hope you found this article useful!

Clean removal of Linux from a Windows dual-boot machine

Recently, I purchased a ThinkPad X1 Carbon Gen 13 to replace my old Dell XPS 13 9360. It came with Windows 11, and instead of wiping it, I decided to set up a dual-boot system with Linux. I still need Windows for .NET development, but I use Linux most of the time.

I eventually settled on Fedora, but I tried several distributions and desktop environments before getting there. Each time I removed Linux, I needed to clean up the old partitions and boot entries properly so they would not interfere with the next installation.

If you are doing the same kind of distro-hopping on a UEFI system, this is the cleanup process that worked for me.

This note covers UEFI systems only.

If you want to remove Linux from a Windows dual-boot setup, there are two things to clean up:

  1. Delete the Linux partitions.
  2. Remove the Linux bootloader files and firmware entries.

Before making changes, make sure Windows is working normally and back up anything you want to keep from your Linux installation.

1. Delete the Linux partitions

Open Windows Disk Management and identify the partitions that belong to Linux. These are usually the Linux root partition, swap partition, and optionally a separate /home partition.

Do not delete:

  • The Windows partition
  • The EFI System Partition
  • The Windows recovery partition

Delete only the Linux partitions, then either leave the space unallocated or extend your Windows partition into it.

2. Remove Linux bootloader files from the EFI partition

Open Command Prompt as Administrator and assign a drive letter to the EFI System Partition:

diskpart
list vol
select vol <EFI volume number>
assign letter=S
exit

Then remove the Linux bootloader directories from the EFI partition:

S:
cd EFI
dir
rmdir /s /q <linux-folder>

Only delete the directory that matches your Linux installation. Common names include fedora, ubuntu, debian, and similar distro-specific folders.

Do not delete Microsoft.

3. Remove stale UEFI firmware boot entries

List the firmware boot entries:

bcdedit /enum firmware

Look for entries that reference your removed Linux installation, such as descriptions containing Fedora, ubuntu, GRUB, or a path that points to the distro's EFI loader.

Delete only the matching Linux entry:

bcdedit /delete {xxxx-xxxx-xxxx-xxxx}

Replace {xxxx-xxxx-xxxx-xxxx} with the actual identifier from the previous command.

4. Reboot and verify

Reboot the machine and confirm that the Linux boot entry is gone from the firmware boot menu and that Windows starts normally.

Hope you found this article useful!

Why Hangfire recurring jobs should always have stable IDs?

Recently, I learned that not giving explicit IDs to Hangfire recurring jobs can silently create duplicates.

When you call RecurringJob.AddOrUpdate(...) without a pre-defined ID, Hangfire auto-generates one based on the method expression. That sounds fine—until you deploy across multiple nodes or refactor your code.

In such cases, Hangfire may generate a new derived ID, while the old recurring job continues running in the background. The result? Duplicate jobs executing on schedule, often goes unnoticed.

Simple fix

Make sure that you always pass a clear, human-readable ID:

RecurringJob.AddOrUpdate<ICustomBackgroundServiceManager>(
    "payments:reconcile-3h", // Human-readable ID
    s => s.ResendPaymentReportToVendorHourly(),
    Cron.Hourly(3)
);

Next, ensure that overlaps are impossible in a multi-node setup by adding this attribute to your job method:

[DisableConcurrentExecution(timeoutInSeconds: 3600)]
public Task ResendPaymentReportToVendorHourly() { /* ... */ }

This uses a distributed lock backed by Hangfire storage—so even with multiple nodes, only one execution runs at a time. If you’re wondering whether this locks the job for the full hour: it doesn’t. The timeout exists for crash-safety, not throttling.

After doing this, recurring jobs become much easier to reason about, identify, pause, or delete in a multi-node environment.

Hope you found this tip useful!

Running Laravel queues with Supervisor on Ubuntu

Laravel queues aren’t new to me, but I realized I’d never really written down how I usually set them up on a fresh Ubuntu server.

Whenever I need queue workers to run continuously — and survive restarts or crashes — I almost always reach for Supervisor. It’s simple, boring, and gets the job done.

First, install it on your server:

sudo apt update
sudo apt install supervisor

Then create a small config file for the queue worker:

sudo vim /etc/supervisor/conf.d/laravel-worker.conf

This is the setup I generally start with:

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/project/artisan queue:work --sleep=3 --tries=3 --max-time=3600
autostart=true
autorestart=true
user=www-data
numprocs=2
redirect_stderr=true
stdout_logfile=/var/www/project/storage/logs/laravel-worker.log
stopwaitsecs=600

A few quick notes on what this does:

  • It runs queue:work instead of queue:listen, so the worker stays in memory
  • Two worker processes are started in parallel (numprocs=2)
  • If the queue is empty, the worker sleeps for a few seconds instead of spinning
  • Failed jobs are retried a limited number of times
  • The worker is force-restarted every hour to avoid memory leaks
  • If a worker crashes, Supervisor brings it back up automatically

Once the file is saved, reload Supervisor’s config:

sudo supervisorctl reread
sudo supervisorctl update

Start the workers:

sudo supervisorctl start laravel-worker:*

And check their status:

sudo supervisorctl status

Whenever I deploy new code or change environment variables, I just restart them:

sudo supervisorctl restart laravel-worker:*

Logs end up in Laravel’s storage/logs, which is usually the first place I look when something feels off.

This setup has been reliable for me across multiple projects. No dashboards, no extra moving parts — just queue workers quietly doing their job in the background.

Hope you found this useful!