James Hibbard

Erase Sensitive Files from Git History with git-filter-repo

2025-06-23T00:00:00+00:00

Ever accidentally commit a password to Git? You’re not the only one.

Last weekend, while working on a Rails project, I accidentally added my database.yml file to the repo and pushed it to GitHub. It contained my local MySQL password. No public exposure thankfully, but it still nagged at me. I didn’t want that file, or its contents, lingering in my Git history.

After a bit of research, I found a quick and effective way to scrub it from history: a tool called git filter-repo. In this post, I’ll show you how to use it to remove committed secrets, credentials, or just a file that shouldn’t be there.

Why `git rm` or Reverting a Commit Isn’t Enough

Your first thought might be to delete the file with git rm or roll back the commit. But that won’t erase the damage.

Git is built to remember everything. Even if you remove the file from the latest commit, it remains in earlier ones. Anyone with access to your repo can still retrieve it—or see it in the commit history.

That’s not just inconvenient. It’s a security risk. Even if the credentials are no longer valid, their presence in your history could expose internal architecture or violate compliance policies.

To get rid of it for good, you need to rewrite history.

Meet `git filter-repo`: A Cleaner Solution

git filter-repo is a fast, flexible, and actively maintained tool for rewriting Git history. It lets you permanently remove files, directories, commits, authors—anything you want to scrub from your repository’s timeline.

It’s also the officially recommended replacement for git filter-branch, the older built-in command that’s now widely considered obsolete due to its slowness and complexity.

You might’ve also come across bfg-repo-cleaner, which is still a valid option. It’s reliable and works well for simple tasks like deleting large files or purging secrets. But it does come with a few caveats (as detailed here) and it runs on Java, so you’ll need that installed on your system.

Installation

If you’re on Ubuntu or any Debian-based system, the easiest way to install git-filter-repo is via apt:

sudo apt install git-filter-repo

For macOS users, Homebrew has you covered:

brew install git-filter-repo

Alternatively, git-filter-repo is available as a Python package via PyPI, which works cross-platform:

pip install git-filter-repo

You can also install it manually by downloading the script directly, placing it somewhere in your $PATH, and ensuring it’s executable.

For other methods—including installing from source, using a Makefile, or additional Windows-specific instructions—refer to the official INSTALL.md on GitHub.

Absolutely, My Lord. Here’s the revised section—clearer, more explanatory, and grounded in what each step actually does, with a proper link back to the official recommendation:

Step-by-Step: Removing a File from Git History

With git filter-repo installed, here’s how to permanently remove a file from your Git history—such as my accidentally committed config/database.yml.

There are two main approaches: the canonical method, recommended by the tool’s author, and a pragmatic shortcut that works fine if you’re confident and know the risks.

⚠️ Warning: This process rewrites Git history and cannot be undone. Make a backup before you begin. Clone the repo elsewhere, zip up your project folder—anything. I’m not responsible if you wipe out your repository. Proceed with caution.

Option 1: The Canonical Way (Fresh Clone)

This is the approach recommended by the tool’s official documentation, and it’s the safest way to proceed.

1. Create a fresh mirror of your repository

git clone --mirror https://github.com/your-user/your-repo.git
cd your-repo.git

This creates a bare clone with all branches, tags, and references—essentially the full repo history—without any working files. It’s the clean slate git filter-repo expects.

2. Run the filter to delete the file

git filter-repo --path config/database.yml --invert-paths

This tells git filter-repo to remove (--invert-paths) every instance of the file from all history. See Filtering based on paths in user manual for more details

3. Push the cleaned history

git push --force

Force-push is required because the rewritten history no longer matches the remote.

Option 2: The Pragmatic Shortcut (Local Repo)

If you don’t want to clone again and are okay rewriting history in place:

git filter-repo --path config/database.yml --invert-paths --force
git push --force

This does exactly the same thing, but runs against your current working repository. The --force flag is required because git filter-repo will otherwise abort when it detects that your repo isn’t freshly cloned.

By taking this route, you bypass the safety net. Your .git directory will be rewritten in place. Any mistakes, and you may not be able to recover unless you’ve backed it up or can reclone it. So proceed only if you’re confident you’ve got a clean working state—or don’t mind burning it all down.

Either way, the result is the same: the file is scrubbed from every commit it ever touched. It’s no longer in your Git history. Not even in the shadows.

What to Do Next: Rotate Keys and Notify Your Team

Even though the file is now gone from Git history, its contents may have been exposed during the time it was accessible. If it contained any secrets—like API keys, database passwords, or tokens—you should treat them as compromised and rotate them. Especially if the exposed data could grant access to anything internet-facing, don’t take chances. You don’t necessarily need to rotate everything, but rotate anything that could plausibly be used externally. Better to be overly cautious than to discover abuse later.

It’s also wise to review your access logs. If the repository was public or shared with others during the time the file was present, see if there were any unexpected clones or forks. Even if it’s unlikely that someone accessed the secret, assuming the worst puts you on safer ground.

Let your team know what happened. Anyone who cloned or forked the repository before the cleanup will still have a copy of the secret in their local history. Ask them to delete their clone or re-clone the repository entirely. You’ll want everyone working from the same cleaned state moving forward.

Avoiding This in the Future

The best defense against this kind of thing is a proactive setup that makes it harder to slip up again.

First, use a properly configured .gitignore file to keep sensitive files—like config/database.yml, .env, or credential stores—out of version control from the start. This is your first and simplest safeguard.

Instead of hardcoding secrets, store them in environment variables or use a dedicated secrets manager which let you separate configuration from code.

You can also set up Git pre-commit hooks to block bad commits. Frameworks like pre-commit make this easy to manage across projects, letting you run linters, formatters, or custom scripts automatically before each commit.

For secret detection specifically, git-secrets scans your changes for known patterns (like AWS keys) and aborts the commit if something leaks through.

Conclusion: Clean Repos, Clean Conscience

In the end, scrubbing my database.yml from history was quick and painless and git filter-repo did exactly what it promised.

But if you’ve ever exposed something critical—an API key, a password, anything live, you’ll know the sinking feeling. The good news? With the right tools, you can clean up the mess and move on with confidence.

I hope this post helped—if you have any questions or thoughts, feel free to drop them in the comments below.

Deploying a Rails App with Apache & MySQL on Ubuntu Server

2025-06-17T00:00:00+00:00

Getting a Rails application running on a remote server involves a fair bit of setup. Recently, I’ve had to go through the process a couple of times—provisioning a virtual machine, installing and configuring Apache, MySQL, Ruby, and Rails, and finally deploying the app. To make future deployments easier (and to have something to refer back to), I decided to document the steps.

In this tutorial, I’ll walk through how to deploy a Rails application (the one built in this guide) to a virtual private server running Ubuntu. In my case, I’m using a VirtualBox VM, but the same instructions apply to any internet-accessible server that supports SSH. The goal is to get the app up and running and accessible over the web.

Let’s begin.

Server Setup Prerequisites

To follow along with this tutorial, you’ll need access to a machine running an Ubuntu-based system (I’m using Ubuntu Server 24.04) with SSH enabled. I’ll be working with a VirtualBox VM, but the same steps apply to any server you can SSH into—whether that’s a DigitalOcean droplet or something else entirely.

If you need help setting up a VirtualBox instance, I’ve written a separate tutorial that covers the full process, including how to enable SSH access. You can find it here. This guide assumes that your server is already configured and accessible via SSH.

For reference, I’ve started VirtualBox in headless mode as described in that tutorial, and I’ve SSHed into the server using a key I previously copied over. Here’s what that looks like in my terminal:

Unless stated otherwise, all commands in this tutorial are to be run on the server.

In addition to the above, you’ll also need a GitHub account and a Unix-like environment to run commands such as scp. If you’re on Windows, the Windows Subsystem for Linux (WSL) is a great option.

But Why Not Kamal or Docker?

Before we dive in, a quick word about Kamal. It’s the official deployment tool for Rails, backed by 37signals and built on Docker. It’s a great fit for apps that need more infrastructure—things like scaling, orchestration, remote builds, and zero-downtime deploys.

That said, it’s not what I’m using here. I already had this VPS setup in place, and it works. I know how to manage it, and if something goes wrong, I can SSH in and fix it. Docker adds a level of abstraction that I don’t find helpful for small, self-hosted apps. Kamal simplifies a lot, but it still depends on a containerization workflow that feels heavier than necessary for my needs.

If you’re curious about Kamal, there are plenty of tutorials online. But if you’d rather stick to a more traditional setup, keep reading—we’ll get your app deployed without any containers.

Installing the LAMP Stack

Before we can run our Rails app, we need a working web server and database. The easiest way to get there on Ubuntu is by installing the LAMP stack—short for Linux, Apache, MySQL, and PHP. While Rails doesn’t need PHP, the lamp-server^ meta-package pulls in Apache and MySQL along with some useful defaults, and it’s a quick way to get a web server and database up and running.

To install it, run:

sudo apt install lamp-server^

The caret (^) at the end tells apt to install the full task as defined by Ubuntu’s tasksel system, not just a single package. This installs Apache, MySQL, PHP, and various modules.

Enabling Password Access for MySQL

By default, MySQL on modern Ubuntu versions uses socket-based authentication for the root user. But Rails expects password-based access when connecting via a database.yml file. To fix this, run the following:

sudo mysql
ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'password';
FLUSH PRIVILEGES;
EXIT;

This switches the root user back to traditional password-based login using mysql_native_password, sets the password to “password” (you should change this to your actual password), and reloads the permissions.

To confirm the stack installed correctly, check the versions like so:

jim@grunt:~$ apache2 -v
Server version: Apache/2.4.58 (Ubuntu)
Server built:   2025-04-03T14:36:49

jim@grunt:~$ mysql --version
mysql  Ver 8.0.42-0ubuntu0.24.04.1 for Linux on x86_64 ((Ubuntu))

jim@grunt:~$ php -v
PHP 8.3.6 (cli) (built: Mar 19 2025 10:08:38) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.3.6, Copyright (c) Zend Technologies
    with Zend OPcache v8.3.6, Copyright (c), by Zend Technologies

This gives us a complete, ready-to-use web server and database setup with minimal effort.

Securing MySQL

Once MySQL is installed, it’s a good idea to run the built-in security script to harden the setup:

sudo mysql_secure_installation

When prompted, I answered as follows, but feel free to tailor this to your needs:

Set up VALIDATE PASSWORD plugin? — No
Change the root password? — No (we already did this earlier)
Remove anonymous users? — Yes
Disallow root login remotely? — Yes
Remove test database and access to it? — Yes
Reload privilege tables now? — Yes

This strips out a few defaults that aren’t needed and helps lock things down a bit.

Installing phpMyAdmin

Although not strictly necessary, I find it helpful to have a web interface for inspecting and managing the database. To install phpMyAdmin, run:

sudo apt install phpmyadmin

When prompted:

Choose apache2 as the web server to configure automatically
Select Yes to configure the database with dbconfig-common
When asked for the MySQL application password for phpmyadmin, you can either set a password or leave it blank. Leaving it blank will generate a random password for the internal phpmyadmin MySQL user, which is fine for most setups—you’ll still log in to phpMyAdmin using your regular MySQL user credentials (e.g. root).

When installing phpMyAdmin, the package installer will present a series of screens in a text-based interface. You can use the arrow keys or the Tab key to move between options (e.g. between Apache2, lighttpd, and the <OK> button). Press Space to toggle selections on or off, and Enter to confirm your choices and continue.

Once the installation is complete, visiting http://your-server-ip/ in a browser will show the Apache2 default page, confirming that Apache is running correctly. To access phpMyAdmin, go to http://your-server-ip/phpmyadmin.

To log in, use the root MySQL user and the password you configured earlier.

Install Git, rbenv, and Ruby

Git is already installed by default on Ubuntu Server 24.04. To confirm this, run:

git -v

If it’s not installed, you can add it with:

sudo apt install git

Either way, it’s a good idea to set your global Git config:

git config --global color.ui true
git config --global user.name "Your Name"
git config --global user.email "you@example.com"

To manage Ruby versions, we’ll use rbenv. It allows you to install and switch between different versions of Ruby on a per-user basis—ideal for avoiding version conflicts.

Start by cloning the rbenv repository and initializing it:

git clone https://github.com/rbenv/rbenv.git ~/.rbenv
~/.rbenv/bin/rbenv init
source ~/.bashrc
type rbenv

If the installation was successful, that final command should output:

rbenv is a function

Then install the ruby-build plugin to enable installing Ruby versions:

git clone https://github.com/rbenv/ruby-build.git "$(rbenv root)"/plugins/ruby-build

Install the required libraries for building Ruby:

sudo apt install build-essential libffi-dev libmysqlclient-dev libreadline-dev libssh-dev libssl-dev libyaml-dev zlib1g-dev

Now install Ruby 3.4.4 and set it as the global default:

rbenv install 3.4.4
rbenv global 3.4.4

Depending on how much RAM your server has, this might take a while to run. When it has, check that Ruby is installed correctly:

ruby -v
# ruby 3.4.4 (2025-05-14 revision a38531fd3f) +PRISM [x86_64-linux]

Install JavaScript Runtime (Optional)

As of Rails 8, you no longer need a JavaScript runtime if you’re using import maps, which are now the default. This setup loads JavaScript directly in the browser, without requiring Node.js, NPM, or a bundler.

However, if you’re planning to use a bundler like Vite or esbuild, you’ll still need Node.js to compile assets. We won’t be using Vite in this tutorial, but here’s how to install Node anyway—just in case your app setup requires it.

The easiest way to manage Node versions is with nvm the Node Version Manager. It allows you to install and switch between Node.js versions easily, much like rbenv does for Ruby.

To install and configure nvm:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash
source ~/.bashrc

Then install the latest LTS version of Node.js and set it as default:

nvm install --lts
nvm alias default lts/*

Verify the installation:

node -v
# v22.16.0

With Ruby and Node in place, the system is now ready for Rails.

Install and Configure Passenger

To serve a Rails app through Apache, we’ll use Passenger—an application server that integrates directly with Apache and makes deploying Ruby apps much simpler. Passenger handles process management and routes incoming web traffic to the Rails app behind the scenes.

We’ll follow the official installation instructions for Ubuntu 24.04 (“Noble”).

First, add the Passenger APT repository and install the module:

sudo apt install -y dirmngr gnupg apt-transport-https ca-certificates curl
curl https://oss-binaries.phusionpassenger.com/auto-software-signing-gpg-key.txt | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/phusion.gpg >/dev/null
sudo sh -c 'echo deb https://oss-binaries.phusionpassenger.com/apt/passenger noble main > /etc/apt/sources.list.d/passenger.list'
sudo apt update
sudo apt install libapache2-mod-passenger

Once Passenger is installed, we need to tell it to use the correct Ruby version (the one installed via rbenv):

sudo nano /etc/apache2/mods-enabled/passenger.conf

Find this line:

PassengerDefaultRuby /usr/bin/passenger_free_ruby

…and replace it with this:

PassengerDefaultRuby /home/jim/.rbenv/shims/ruby

💡 Adjust the path to match your actual user and environment.

A Quick Note on Nano

When you run a command like sudo nano /etc/apache2/mods-enabled/passenger.conf, it opens the file in nano, a simple terminal-based text editor that comes preinstalled on most Linux systems.

Nano displays the contents of the file in the main area, with a helpful list of shortcuts at the bottom of the screen. You can move around using the arrow keys.

To save and exit:

Press Ctrl + X to begin exiting
Press Y when prompted to save changes
Press Return to confirm the file name

If you’re unfamiliar or get stuck, this short guide provides a handy reference. But don’t worry—nano is beginner-friendly and hard to mess up.

Enable the Module and Restart Apache

Now enable the Passenger module and restart Apache:

sudo a2enmod passenger
sudo apache2ctl restart

If you see a warning like:

Could not reliably determine the server's fully qualified domain name

edit the Apache config:

sudo nano /etc/apache2/apache2.conf

…and add this to the bottom:

ServerName localhost

Then restart Apache again.

Verify the Installation

To confirm everything installed correctly, run:

sudo /usr/bin/passenger-config validate-install

This will open an interactive menu asking what you’d like to validate. Use the arrow keys to move up and down, Space to select “Passenger itself” and Enter to proceed.

You should see output ending with:

Everything looks good. :-)

This indicates that Passenger is correctly installed and ready to go.

Next, check if Passenger’s core processes are running:

sudo /usr/sbin/passenger-memory-stats

You should see Apache processes as well as Passenger processes. For example:

Version: 6.0.27
Date   : 2025-06-13 20:49:28 +0000

---------- Apache processes ----------
PID    PPID   VMSize    Private  Name
--------------------------------------
42450  1      290.5 MB  2.7 MB   /usr/sbin/apache2 -k start
...

----- Passenger processes -----
PID    VMSize    Private  Name
-------------------------------
42690  358.7 MB  2.4 MB   Passenger watchdog
42694  977.2 MB  3.9 MB   Passenger core
...

Finally, check that Passenger itself is responsive:

sudo passenger-status

You should see output similar to:

jim@grunt:~$ sudo passenger-status
Version : 6.0.27
Date    : 2025-06-13 20:51:45 +0000
Instance: JtJFm3v1 (Apache/2.4.58 (Ubuntu) Phusion_Passenger/6.0.27)

----------- General information -----------
Max pool size : 6
App groups    : 0
Processes     : 0
Requests in top-level queue : 0

----------- Application groups -----------

All of this confirms that Apache and Passenger are working together and ready to serve your Rails application.

Set Up an Apache Virtual Host

To serve your Rails app, we’ll define a virtual host—a block of configuration that tells Apache where your app lives and how to handle requests to it.

Within this block, the ServerName directive tells Apache which requests should be routed to this site. If you’re accessing the server by IP address, ServerName can be anything you like—Apache will simply use the first matching config. But if you’re using a hostname (like app.mycompany.com), then that name must match what appears in your browser and be resolvable—either via DNS or your system’s /etc/hosts file.

For this tutorial, we’ll set the host name to devise.local as it reflects the functionality of the Rails app we’re deploying. You can change this to anything you like—just be sure it matches your access method. The folder path /var/www/devise-demo/current/public should remain structured as shown, but the devise-demo portion can be renamed to suit your project.

Configure the Site

First, create the folder structure Apache will use:

sudo mkdir -p /var/www/devise-demo
sudo chown -R jim:www-data /var/www/devise-demo
sudo chmod -R 775 /var/www/devise-demo

This sets up the directory for your deployed Rails app. You can change devise-demo to suit your project name, and replace jim with your actual system username.

Next, create a new site definition file:

sudo nano /etc/apache2/sites-available/devise.local.conf

Add the following:

<VirtualHost *:80>
  ServerName devise.local
  DocumentRoot /var/www/devise-demo/current/public

  <Directory /var/www/devise-demo/current/public>
    AllowOverride all
    Options -MultiViews
    Require all granted
  </Directory>

  <Location /phpmyadmin>
    PassengerEnabled off
  </Location>
</VirtualHost>

Once the site configuration is in place, you need to tell Apache to use it:

sudo a2ensite devise.local.conf

This command enables the devise.local.conf virtual host by creating a symlink in /etc/apache2/sites-enabled/. Apache will now consider this configuration when handling incoming requests.

Next, disable the default site:

sudo a2dissite 000-default.conf

Apache ships with a default site configuration (000-default.conf) that catches all requests not matched by another config. Disabling it ensures your new virtual host (devise.local.conf) takes precedence and avoids any unexpected routing issues.

Finally, reload Apache to apply the changes:

sudo systemctl reload apache2

This gracefully restarts Apache, reloading the configuration without interrupting existing connections. After this, your server will respond to requests for devise.local as defined in your new virtual host.

Add Hostname Resolution (On the Host Machine)

💡 This step is only necessary if you’re running a local VirtualBox VM and want a readable hostname instead of using the raw IP. If you’re deploying to a remote server, you can skip this and use the actual IP or domain name directly.

To make devise.local resolve to your guest VM, edit your host’s /etc/hosts file:

sudo nano /etc/hosts

Add a line like:

192.168.178.51 devise.local

Replace 192.168.178.51 with the IP address of your VirtualBox guest.

Now, visiting http://devise.local in your browser should connect to the server, but you’ll likely see a 404 Not Found error for now—that’s because we haven’t deployed the Rails app yet. Apache is already configured to serve it, and we’ll upload the app shortly.

Create the Production Databases

Rails 8 introduces four logical databases in production by default: primary, cache, queue, and cable (see: discuss.rubyonrails.org). These correspond to the entries in your config/database.yml. Even if you’re not actively using cache, queue, or cable features, Rails will expect them to exist—or throw errors.

To satisfy that, you’ll need to create all four production databases and grant access to a single MySQL user.

Run:

mysql -uroot -p

Then execute:

CREATE DATABASE store_production;
CREATE DATABASE store_production_cache;
CREATE DATABASE store_production_queue;
CREATE DATABASE store_production_cable;

This setup ensures all of Rails 8’s expected database connections exist. In future iterations—especially for production-scale apps—you may choose to consolidate some of these (e.g. using a single DB for cache/queue) or isolate them entirely via environment-specific deploys or host-based separation. But for now, this covers the out-of-the-box structure of Rails 8.

If you visit http://devise.local/phpmyadmin/, log in as the MySQL root user, and navigate to the databases tab, you should see all four databases listed as expected.

Deploy the App Using Capistrano

Before we can deploy our Rails app to a remote server, we need a reliable way to automate the process. This is where Capistrano comes in.

What is Capistrano?

Capistrano is a remote server automation and deployment tool written in Ruby. It connects to your server via SSH, clones your code from a Git repository, runs deployment scripts, and sets up the app directory structure. This means that instead of uploading files manually or copying things over with scp, you can deploy your app with a single command.

It also supports things like rollbacks, restarting services, and running database migrations remotely. It’s a mature, battle-tested tool, widely used in the Rails ecosystem.

What We’ll Be Using

For this tutorial, we’ll be deploying a Rails app that uses Devise for authentication and CanCanCan for authorization.

The source code is available on GitHub at github.com/jameshibbard/authentication-with-devise-and-cancancan, and we’ll be working from the mysql branch. As the name suggests, this version of the app is configured to use MySQL.

If you’d like to see how the app was built, there’s a full walkthrough available on my website.

Rather than deploying from my repository (which you can’t push to), you’ll fork it to your own GitHub account—giving you full control and the ability to configure and deploy the app yourself.

The following steps (1–6) should all be carried out on your local machine.

1. Fork the Repository

Make sure you’re signed into your GitHub account, then:

Visit https://github.com/jameshibbard/authentication-with-devise-and-cancancan
Click the Fork button in the top menu bar
Choose your GitHub account

Next, ensure you have an SSH key set up with GitHub. This is required for Capistrano to deploy your app. If you haven’t already done so, follow GitHub’s official guide: Adding a new SSH key to your GitHub account

Once set up, clone your new repository using the SSH URL:

git clone git@github.com:<your-username>/authentication-with-devise-and-cancancan.git devise-demo
cd devise-demo
git checkout -b mysql origin/mysql

💡 Replace <your-username> with your actual GitHub username.

You now have a local copy of the repo configured to use MySQL.

2. Check GitHub Access (Optional)

To confirm that Capistrano will be able to pull your code, you can run:

git ls-remote git@github.com:<your-username>/authentication-with-devise-and-cancancan.git

If it lists a set of HEAD and refs, you’re good to go.

3. Add Capistrano to the App

To deploy the app, we’ll now integrate Capistrano into the project.

First, open the Gemfile and add the following gems inside the :development group:

group :development do
  gem 'capistrano', require: false
  gem 'capistrano-rails', require: false
  gem 'capistrano-passenger', require: false
  gem 'capistrano-bundler', require: false
  gem 'capistrano-rbenv', require: false
end

Then run:

bundle install

This installs the Capistrano tooling needed for Rails deployments, as well as plugins to handle Bundler, rbenv, Passenger, and Rails-specific tasks like asset precompilation and database migration.

4. Initialize Capistrano

Now initialize Capistrano in your Rails app:

cap install

This generates a Capfile in the project root, along with configuration files located at config/deploy.rb, config/deploy/staging.rb and config/deploy/production.rb.

💡 If this command fails due to a version mismatch or “command not found” error, prepending bundle exec can resolve the issue:
bundle exec cap install
This guarantees you’re running the Capistrano version defined in your project, and avoids conflicts.

Next, open the Capfile and alter it like so:

# Load DSL and set up stages
require 'capistrano/setup'

# Include default deployment tasks
require 'capistrano/deploy'

# Use Git for source control
require 'capistrano/scm/git'
install_plugin Capistrano::SCM::Git

# Load standard Capistrano integrations
require 'capistrano/rails'
require 'capistrano/passenger'
require 'capistrano/rbenv'

# Load custom tasks from `lib/capistrano/tasks` if you have any defined
Dir.glob('lib/capistrano/tasks/*.rake').each { |r| import r }

This section loads the plugins Capistrano needs to work with our setup. The Git SCM plugin enables Capistrano to fetch the code from GitHub, while the Rails and Passenger plugins add tasks for database migrations, asset compilation, and restarting the app.

5. Configure Your Deploy Settings

Next, open config/deploy.rb and update the key settings that tell Capistrano how and where to deploy your app. This includes your app name, GitHub repo, deployment directory, and some shared resources that should persist across releases.

Here are the key settings you’ll want to define.

lock "~> 3.19.2"

set :application, "devise-demo"
set :repo_url, "git@github.com:<your-username>/authentication-with-devise-and-cancancan.git"
set :branch, "mysql"

# Default deploy_to directory is /var/www/my_app_name
set :deploy_to, "/var/www/devise-demo"

# Default value for :linked_files is []
append :linked_files, "config/database.yml", "config/credentials.yml.enc"

# Default value for linked_dirs is []
append :linked_dirs, "log", "tmp/pids", "tmp/cache", "tmp/sockets", "public/system"

# Default value for keep_releases is 5
set :keep_releases, 5

# Specify your Ruby version and rbenv type
set :rbenv_type, :user
set :rbenv_ruby, '3.4.4'

💡 Replace <your-username> with your GitHub username.

The linked_files section ensures that sensitive files like database.yml and credentials.yml.enc remain shared across deployments. The linked_dirs setting keeps folders like logs and uploaded files intact between releases. And finally, specifying your Ruby version ensures the correct environment is used on the server.

If you’re curious, Capistrano includes a number of other optional settings as comments in this file—feel free to explore or customize them as needed.

Next, open config/deploy/production.rb and define the production server:

server "devise.local", user: "jim", roles: %w{app db web}

Replace "devise.local" with your server’s domain name or IP address—just make sure it resolves correctly (e.g. via DNS or your /etc/hosts file). Also replace "jim" with your actual SSH username.

The roles tell Capistrano what responsibilities this server has. In our case, it’s handling the application code (app), the database (db), and web requests (web), all on one box.

6. Verify the Setup

To confirm everything is wired up correctly, run the following command from the root of your Rails project:

cap production doctor

This performs a dry run and checks whether Capistrano can access your server, pull your code from GitHub, and apply the deployment settings. If everything is configured properly, you’ll see output broken into several sections:

Environment – Displays your Ruby version, Rubygems, and the Capistrano command path.
Gems – Lists the Capistrano-related gems detected in your project.
Variables – Shows the values of all relevant deployment settings.
Servers – Confirms which server(s) are defined and the roles assigned.

If all of that appears and no errors are thrown, then your deployment setup is ready to go.

Prepare the Server for Deployment

Before Capistrano can successfully deploy the app, we need to create a few files it expects to find in the shared directory on the remote server.

SSH into the production machine and prepare the necessary structure:

sudo mkdir -p /var/www/devise-demo/shared/config
sudo chown -R jim:www-data /var/www/devise-demo

This sets the ownership of the /var/www/devise-demo directory so that your user (jim) owns the files and the group is www-data, which allows the web server to access them.

Don’t forget to replace jim and devise-demo with your actual Linux username and project name to match your server’s setup.

Add `database.yml`

Capistrano will symlink database.yml from the shared config directory. Create it like so:

nano /var/www/devise-demo/shared/config/database.yml

Paste in the complete database.yml you are already using locally, including the production: section with the primary, cache, queue, and cable databases.

Notice that we are using Rails.application.credentials to fetch the database username and password—this keeps sensitive data out of source control. We’ll define those credentials next.

Define Your Rails Master Key and Encrypted Credentials

Your Rails app relies on config/master.key to decrypt config/credentials.yml.enc, where sensitive data like your secret_key_base and database credentials are stored securely.

If you cloned the repo and both config/master.key and config/credentials.yml.enc are missing (which they will be), you’ll need to generate fresh ones.

Run this command locally in the cloned repo:

EDITOR=nano bin/rails credentials:edit

This will:

Create a new config/master.key
Create (or overwrite) config/credentials.yml.enc
Decrypt credentials.yml.enc and open it in nano
Auto-populate it with a new secret_key_base

💡 You can specify a different editor by setting the EDITOR environment variable. For example, use EDITOR=subl to open the file in Sublime Text.

While editing this file, add your production database credentials:

production:
  database_user: your-user
  database_password: your-password-here

Replace your-user and your-password with your actual MySQL user and password. If you have been following along with the tutorial, the MySQL user will be root.

As these files contain sensitive information, you should be sure to keep the key out of the repo. Check that your .gitignore (located in the procject root) includes:

/config/master.key

And add it if it doesn’t.

Upload Encrypted Credentials to the Server

Once you’ve added your production database credentials locally, you’ll need to upload the encrypted credentials file so Rails can access it during deployment.

From your project root (on your local machine), run:

scp config/credentials.yml.enc your-user@your-server-ip:/var/www/devise-demo/shared/config/

💡 Replace your-user and your-server-ip with your actual SSH username and server address.

If you don’t have the scp command available (e.g. on Windows), you can use WinSCP to upload the file via a graphical interface.

Provide Rails with the Master Key

Rails needs the master.key to decrypt the credentials. The recommended approach is to set it as an environment variable on the server.

Open your shell config (e.g. ~/.bashrc):

nano ~/.bashrc

At the very top of the file, add the following line (replacing your-masterkey):

export RAILS_MASTER_KEY="your-masterkey"

⚠️ Place this at the very top, or it may be ignored.

Save and close the file, then apply the changes:

source ~/.bashrc

To check that this set correctly across sessions, log out of the server and log back in. Then run:

printenv | grep RAILS_MASTER_KEY

Optional: Upload `master.key` Instead

If the environment variable approach doesn’t work—Capistrano can be finicky—you can instead upload the master.key directly.

On your local machine run:

scp config/master.key your-user@your-server-ip:/var/www/devise-demo/shared/config/

Ensure your deploy.rb includes the following:

append :linked_files, "config/database.yml", "config/credentials.yml.enc", "config/master.key"

And ensure that you have removed the environment variable from your shell config.

Both methods are valid, but the envv ar approach is preferred, as it avoids potential symlink or permissions issues during deploys.

Run the First Deploy

Now that everything is set up, it’s time to do the exciting bit—let’s get the app live!

First commit your changes and push them to GitHub.

git add --all
git commit -m "Added capistrano"
git push origin mysql

Then from your project root, run:

cap production deploy

This kicks off the Capistrano deploy process. You should see it stepping through various tasks—checking versions, uploading files, compiling assets, and restarting the app. Be aware that on the first deploy this might take a while.

Here’s a peek at what the output might look like:

If all goes well, your app is now live on the server.

Open your browser and head to http://devise.local (or whatever your server name is in your /etc/hosts file). If everything worked, you should see your shiny new Rails app up and running.

Congratulations—you’ve deployed a Rails 8 app with Capistrano and Passenger! 🎉

⚠️ If you don’t see your shiny new Passenger app in the browser, don’t panic—just keep reading. In the next step, I’llgive you some tips on how to debug what’s gone wrong.

Troubleshooting

Assuming you’ve followed all the steps correctly, your app should now be up and running. Luckily for me, when I opened the browser to check everything was working, I was greeted not by my shiny new Rails app, but by a wonderfully vague message: “Something went wrong.”

This actually turned out to be a good thing—because it gives me the chance to show you how to debug this kind of issue.

The error message means we’re not even reaching the Rails app—Passenger is failing before the app boots—so instead of checking the Rails production.log, we turn to the Apache error log (/var/log/apache2/error.log), where Passenger logs these kinds of startup issues.

Opening the Apache log:

sudo cat /var/log/apache2/error.log

I saw the following:

The application encountered the following error: You have already activated stringio 3.1.2, but your Gemfile requires stringio 3.1.7. Since stringio is a default gem, you can either remove your dependency on it or try updating to a newer version of bundler that supports stringio as a default gem. (Gem::LoadError)

A bit of Googling revealed that this is caused by Passenger preloading Bundler too eagerly, which can lead to conflicts with Ruby’s default gems such stringio.

To fix it, I opened up my Apache config:

sudo nano /etc/apache2/sites-available/devise.local.conf

Then inside the <VirtualHost> block I added PassengerPreloadBundler on:

<VirtualHost *:80>
  ServerName devise.local
  DocumentRoot /var/www/devise-demo/current/public
  PassengerPreloadBundler on

  <Directory /var/www/devise-demo/current/public>
    AllowOverride all
    Options -MultiViews
    Require all granted
  </Directory>

  <Location /phpmyadmin>
    PassengerEnabled off
  </Location>
</VirtualHost>

Then I restarted Apache:

sudo systemctl restart apache2

After that, I refreshed the browser—and boom, everything worked as expected.

Finishing Touches

At this point, you can access the app, but if you try to log in, or create a user, nothing happens. The app will just reload.

This only applies if you’re testing locally over plain HTTP (e.g. Using VirtualBox). If you’re deploying to a proper server with HTTPS configured, you shouldn’t run into this issue.

Looking in the browser’s console will give us a clue as to what is happening. You will see the following error:

Cookie "_store_session" has been rejected because a non-HTTPS cookie can't be set as "secure".

Rails is complaining that it’s trying to set a secure cookie—which only works over HTTPS—but your app is being served over plain HTTP.

Disabling SSL Enforcement (For Local Testing Only)

To resolve this, open config/environments/production.rb locally and disable SSL enforcement.

Set the following to false:

# Assume all access to the app is happening through a SSL-terminating reverse proxy.
config.assume_ssl = false

# Force all access to the app over SSL, use Strict-Transport-Security, and use secure cookies.
config.force_ssl = false

This tells Rails not to expect HTTPS and not to mark cookies as secure, allowing login to function normally over HTTP. You can read more about these options here and here.

Of course, in a real production environment, SSL should be enabled, and these settings should remain true to protect user data. We’re only disabling them here to get things working locally. If your app is already running over HTTPS, this won’t be an issue. But if you’re testing on something like VirtualBox without SSL, this workaround helps you move forward.

Logging “Fix” (Rails 7.1+)

As of Rails 7.1, the default logging behavior was changed to log to STDOUT rather than directly to a file like log/production.log. This is great for containerized or cloud environments where STDOUT is captured by a logging system—but it’s not especially helpful when running on a regular server with Passenger, where STDOUT often goes nowhere useful.

To restore the traditional file logging while keeping the new default, you can configure Rails to broadcast logs to both STDOUT and production.log:

In your local config/environments/production.rb, change:

config.logger = ActiveSupport::TaggedLogging.logger(STDOUT)

to:

config.logger = ActiveSupport::BroadcastLogger.new(
  ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new($stdout,              formatter: Logger::Formatter.new)),
  ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new("log/production.log", formatter: Logger::Formatter.new))
)

This gives you the best of both worlds: Rails continues to stream logs to STDOUT (in case you later want to capture them via systemd or similar), but also writes to production.log, so you can see what’s happening right now.

Seeding the Database

Before testing everything out, let’s add some sample data to the production database. This ensures you have a user account to log in with.

First, hop onto the server and start the Rails console in production mode:

cd /var/www/devise-demo/current
RAILS_ENV=production bundle exec rake db:seed

Once done, head to your app in the browser and log in using one of the test accounts. If everything’s set up correctly, you should land on the homepage with access to the full app.

Running the Rails Console

If you need to peek into your live app to tweak some data, or debug a production issue directly, you can run the Rails console on the server in the same way that we ran that rake task.

Navigate to your app’s current directory (which always points to the latest release):

cd /var/www/devise-demo/current

Then start the console in production mode:

RAILS_ENV=production bundle exec rails console

You’re now inside your live app. Tread carefully!

Enabling SSL with Let’s Encrypt & Certbot

Once your app is live to the world, you’ll want to serve it securely over HTTPS. The recommended way to do this is with Let’s Encrypt, a free, automated, and open certificate authority.

To simplify the process, Let’s Encrypt recommends using Certbot, a tool maintained by the Electronic Frontier Foundation (EFF). Certbot can automatically request, install, and renew certificates, and even configure your web server for HTTPS.

The general process involves:

Ensuring your domain points to your server.
Confirming your web server (like Apache) is properly configured.
Running Certbot, which will request a certificate and update your server configuration.

We won’t go into the details here, as they vary by operating system and web server setup. Instead, head over to the official resources and follow the steps tailored to your platform:

These guides will walk you through everything you need to secure your app with confidence.

Conclusion

Phew! This has turned into quite a long post!

We started out with a blank server and gradually provisioned it to serve a Rails app. From installing dependencies and setting up Capistrano, to managing secrets and configuring Apache with Passenger, we pulled in our app and got it running in the browser—production-ready and humming along.

Thanks so much for reading. If you have any questions, hit a snag, or just want to share your experience, I’d be happy to help out in the comments below.

Authentication with Devise and CanCanCan in Rails 8

2025-06-02T00:00:00+00:00

This is a tutorial on how to set up authentication (verifying who you are) and authorization (what you’re allowed to do) in a Ruby on Rails app using Devise and CanCanCan. We’ll use Rails 8 and Ruby 3.4, and build everything locally—no external auth services required.

The code for this tutorial is on GitHub: https://github.com/hibbard-eu/authentication-with-devise-and-cancancan

Note: this post was updated on 7^th June, 2025 to use the latest versions of Ruby, Rails, Devise, and CanCanCan..

The Scenario

The app we’ll be building is a store. In order for people to use the store, they’ll need to register an account. The store will also have sellers (otherwise it would be a rubbish store) and an admin.

This means we’ll need the following resources: Item, User, Role.

Here’s a UML diagram showing how they relate to one another:

Note that users can have a maximum of one role. The permissions for each of these users will break down as follows:

Unregistered users are redirected to the sign up page
Registered users can view items
Sellers can view items, create items, as well as update and destroy any items that belong to them
Admins can perform any CRUD operation on any resource

So let’s get started.

Installing the Dependencies

To follow along you will need Ruby and Rails installed on your machine.

For Ruby the exact steps you follow will depend upon your system. If in doubt you can consult the official Ruby install guide.

If you are using a Unix-like system, I recommend rbenv. rbenv is a Ruby version manager and is extremely useful for switching between multiple Ruby versions on the same machine. With rbenv installed, pulling in the latest version of Ruby is as simple as:

rbenv install 3.4.4
rbenv global 3.4.4

Then install Rails globally for this Ruby version:

gem install rails

For more info, see the official Rails installation guide.

You can check the version of each like so:

$ ruby -v
ruby 3.4.4 (2025-05-14 revision a38531fd3f) +PRISM [x86_64-linux]

$ rails -v
Rails 8.0.2

Generating the Project Files

Let’s start by creating a new Rails app. We will skip using the Jbuilder gem, as we won’t be rendering any JSON responses.

rails new store --skip-jbuilder

Rails 8 uses Import Maps by default, which means you don’t need Node.js or any JavaScript bundler to get started.

Once the project is created, change into the store directory and optionally perform an initial git commit:

cd store
git add --all
git commit -m "Initial commit"

Now generate the basic scaffolding:

rails g scaffold role name:string description:string
rails g scaffold user name:string role:belongs_to
rails g scaffold item name:string description:text 'price:decimal{5,2}', user:belongs_to

This order ensures foreign keys are created after their referenced tables — required when using MySQL.

Here price:decimal{5,2} sets up a decimal field with a precision of 5 and a scale of 2. This allows values from -999.99 to 999.99.

The user:belongs_to defines a foreign key relationship, meaning each item will be associated with a specific user.

Finally, run the migrations with the following command:

rake db:migrate

Changes to Rails Scaffolding in Rails 7+

Rails 7 fundamentally changed how scaffold-generated views are structured. Older versions used simple tables in full-page templates. Rails 7 ditched this for a more modular approach with partials and div-based layouts, driven by Turbo Frames and Hotwire.

This means, for example, that index pages no longer use <table> rows but instead render each item via partials, often wrapped in divs. While this works better with Turbo, it broke expectations for many developers used to tabular views.

If you prefer the old approach, you can still create your own table-based views or override the scaffold templates. These discussions go into the rationale and community response:

Ruby on Rails Discussions – Rails 7 - scaffolds? - table view?
Reddit – Why did Rails 7 kill scaffolding?
Reddit – Scaffold Index no longer generating HTML Tables

Editing the Boilerplate

Now we can tailor the files that Rails has generated to suit our needs.

Start by removing the following line from the top of the index and show templates for the Item, Role and User resources. These are in the app/views folder.

- <p style="color: green"><%= notice %></p>

In items/_item.html.erb change:

“User” to “Seller”
<%= item.price %> to <%= "%.2f" % item.price %>
item.user_id to item.user.name.

In items/_form.html.erb remove the complete user_id field (including the surrounding div tags).

- <div>
-   <%= form.label :user_id, style: "display: block" %>
-   <%= form.text_field :user_id %>
- </div>

In users/_user.html.erb change user.role_id to user.role.name

In users/_form.html.erb change <%= form.text_field :role_id %> to:

<%= collection_select(
  :user, :role_id, Role.all, :id, :name, { prompt: true }
) %>

At this point if you start up Puma (rails s) and visit http://localhost:3000/items, http://localhost:3000/roles, or http://localhost:3000/users, you can see that our basic scaffolding is working (albeit without any data)

Adding Some Styles

As you can see, the default Rails scaffolding comes with no styling at all—just raw HTML elements stacked on a blank page. To improve the appearance of our app, copy the CSS styles from the project repository and paste them into app/assets/stylesheets/application.css.

These styles are a simple way to smarten up the interface and make things easier to work with during development. They’re not designed for production, but they provide a clean starting point until you’re ready to implement a proper design.

Authentication with Devise

Now let’s turn our attention to Devise and the authentication logic.

Devise is a full-featured authentication solution for Rails based on Warden. One of the things I like about it most (aside from the ease of use) is that it is built on a modularity concept. This makes it easy to include only those features you need in your application.

Why Not Use Built-in Rails Authentication?

Rails 8 ships with a built-in authentication generator that gives developers full control over the auth stack. It’s ideal if you prefer to manage your own logic and don’t mind writing some boilerplate.

That said, Devise remains the better choice for most real-world apps. It provides a robust, out-of-the-box solution with minimal setup, and comes packed with features like user confirmation, session timeouts, and OAuth support—saving time and reducing complexity.

If you’d like to explore the built-in option, see the official guide: Securing Your App with the Default Authentication Generator (Rails 8 Unpacked)

Installing And Configuring Devise

To get started with Devise, add it to our project’s Gemfile:

bundle add devise

Then run Devise’s installation generator:

rails g devise:install

This command generates a couple of files: an initializer and a locale file that contains all of the messages that Devise needs to display.

As per the post installation message, we’ll need to make a couple of alterations to our config files.

Add the following line to the bottom of config/environments/development.rb:

config.action_mailer.default_url_options = { host: "localhost", port: 3000 }

And set a root route in /config/routes.rb

root "items#index"

We’re going to use the User model to handle authentication and Devise provides a generator for doing just that:

rails g devise User
rake db:migrate

Devise won’t override our current User model – it will simply add a bunch of attributes to it.

Now let’s update our application layout to include a header and navigation. In /app/views/layouts/application.html.erb replace:

<body>
  <%= yield %>
</body>

With the following:

<body>
  <div class="container">
    <header class="site-header">
      <div class="branding">
        <%= link_to "MyStore", root_path %>
      </div>
        <div class="header-right">
        <nav class="user-nav">
          <% if user_signed_in? %>
            Signed in as <%= current_user.email %> |
            <%= link_to "Edit profile", edit_user_registration_path %> |
            <%= link_to "Sign out", destroy_user_session_path, data: { "turbo-method": :delete } %>
          <% else %>
            <%= link_to "Sign up", new_user_registration_path %> or
            <%= link_to "sign in", new_user_session_path %>
          <% end %>
        </nav>

        <nav class="admin-nav">
          <% if user_signed_in? %>
            <%= link_to "Items", items_path %> |
            <%= link_to "Users", users_path %> |
            <%= link_to "Roles", roles_path %>
          <% end %>
        </nav>
      </div>
    </header>

    <% flash.each do |name, msg| %>
      <%= content_tag :div, msg, class: "flash #{name}" %>
    <% end %>

    <main>
      <%= yield %>
    </main>
  </div>
</body>

The new layout includes a top navigation bar showing the current user’s authentication status. If the user is signed in, it displays their email address along with links to edit their profile or sign out. If not signed in, it shows links to sign up or log in. Flash messages are displayed directly beneath the header.

We’ve also added an admin navigation bar with links to Items, Users, and Roles. This is primarily for convenience during development, allowing quick access across key resources. It’s currently visible to all users, but we’ll later restrict it to admins only.

Next, add the following line to the top of ItemsController, RolesController and UsersController:

before_action :authenticate_user!

This will ensure that the user is logged in before being able to access any of these resources.

Now, populate your database with some initial records. Alter db/seeds.rb thus:

r1 = Role.create({ name: "Regular", description: "Can read items" })
r2 = Role.create({ name: "Seller", description: "Can read and create items. Can update and destroy own items" })
r3 = Role.create({ name: "Admin", description: "Can perform any CRUD operation on any resource" })

u1 = User.create({ name: "Sally", email: "sally@example.com", password: "aaaaaaaa", password_confirmation: "aaaaaaaa", role_id: r1.id })
u2 = User.create({ name: "Sue", email: "sue@example.com", password: "aaaaaaaa", password_confirmation: "aaaaaaaa", role_id: r2.id })
u3 = User.create({ name: "Kev", email: "kev@example.com", password: "aaaaaaaa", password_confirmation: "aaaaaaaa", role_id: r2.id })
u4 = User.create({ name: "Jack", email: "jack@example.com", password: "aaaaaaaa", password_confirmation: "aaaaaaaa", role_id: r3.id })

i1 = Item.create({ name: "Rayban Sunglasses", description: "Stylish shades", price: 99.99, user_id: u2.id })
i2 = Item.create({ name: "Gucci watch", description: "Expensive timepiece", price: 199.99, user_id: u2.id })
i3 = Item.create({ name: "Henri Lloyd Pullover", description: "Classy knitwear", price: 299.99, user_id: u3.id })
i4 = Item.create({ name: "Porsche socks", description: "Cosy footwear", price: 399.99, user_id: u3.id })

Then run:

rake db:seed

Finally, we can restart the Rails server and log in using the email address and password of one of the users we defined in the seeds file.

Notice that if we are logged out and try and access any of the protected resources, we are redirected to a log in page with the message “You need to sign in or sign up before continuing.”

Exciting times, huh?

A Bit More About Devise

As I mentioned briefly above, Devise is based on a modularity concept. To expand on that, take a peek at the User model. You should see:

class User < ApplicationRecord
  # Include default devise modules. Others available are:
  # :confirmable, :lockable, :timeoutable, :trackable and :omniauthable
  devise :database_authenticatable, :registerable,
         :recoverable, :rememberable, :validatable
  belongs_to :role
end

In its default configuration, Devise comes with five of ten modules enabled.

You can see these in action in our app – for example Rememberable remembers the user’s authentication in a saved cookie (embodied by the checkbox on the login form that says “Remember me”).

If you didn’t require this functionality, simply comment out :rememberable and it won’t be included.

I would encourage you to take the time to read through what all of the modules do. They are listed on the project’s readme

Now, to get our app working properly, we need to make a few tweaks.

Adding an Admin Namespace

Let’s start by namespacing the CRUD interface. This is necessary as otherwise the user registration routes can conflict with the user management routes. Alter config/routes.rb like so:

devise_for :users
scope "/admin" do
  resources :users
end

resources :items
resources :roles

Then, in the User model, we can add the missing association (app/models/user.rb):

has_many :items, dependent: :destroy

We are specifying dependent: :destroy as if we delete a seller, it also makes sense to delete the items they are selling.

And in the ItemsController, make sure that a user is assosciated with each item before it is saved:

def create
  @item = Item.new(item_params)
  @item.user = current_user
  ...
end

Once you have restarted the server, you will be able to (kinda) manage users at http://localhost:3000/admin/users. I write kinda, as you’ll not yet be able to create new users. We’ll get to that a little later.

Customizing Devise

If you click on the “sign up” or “edit profile” links, you’ll notice that the user’s name attribute is missing. It would be nice for users to specify their names when registering, so let’s fix that.

Devise comes with many built-in views. To customize these pages, we must transfer the Devise view files into our project so we can modify them. Luckily, Devise has a generator to do just that:

rails g devise:views

This will copy across a bunch of templates to the app/views/devise directory.

Change into this folder, then into the registrations sub-folder. Locate the new.html.erb and edit.html.erb files and after this line:

<%= render "devise/shared/error_messages", resource: resource %>`:

Add the following:

<div class="field">
  <%= f.label :name %><br />
  <%= f.text_field :name, autofocus: true %>
</div>

And remove the autofocus: true attribute from the email field.

This adds the name field to the registration and edit forms. But if you fill them out at this point, you will notice that Devise doesn’t save your changes.

The reason for this is that the extra parameter you are passing to the controller (name in this case) needs to be expressly permitted.

You can do this as follows in the ApplicationController:

class ApplicationController < ActionController::Base
  before_action :configure_permitted_parameters, if: :devise_controller?

  # Only allow modern browsers supporting webp images, web push, badges, import maps, CSS nesting, and CSS :has.
  allow_browser versions: :modern

  protected

  def configure_permitted_parameters
    devise_parameter_sanitizer.permit(:sign_up, keys: [ :name ])
    devise_parameter_sanitizer.permit(:account_update, keys: [ :name ])
  end
end

Now Devise will accept and save the name field when users register or edit their account.

We also want to assign a default role to new users. Rather than requiring the user to select one, we can assign it automatically using a callback. Update the User model like so:

class User < ApplicationRecord
  # Include default devise modules. Others available are:
  # :confirmable, :lockable, :timeoutable, :trackable and :omniauthable
  devise :database_authenticatable, :registerable,
         :recoverable, :rememberable, :validatable

  belongs_to :role
  has_many :items, dependent: :destroy
  before_validation :assign_role
  validates :name, presence: true

  def assign_role
    self.role = Role.find_by name: "Regular" if role.nil?
  end
end

This ensures that any new user is automatically given the “Regular” role if no other role is set.

Notice how we are assigning the role in a before_validation callback. One might consider using before_create or before_save instead, but since Rails 5, belongs_to enforces presence by default, and those callbacks fire after validations have run — too late to meet the requirement. Using optional: true would suppress the error, but would contradict the application’s intent, as every user should have a role.

Take a moment at this point to start up the app again and make sure that everything is working. You should be able to specify your name when signing up as a new user.

Creating New Users

Next we’re going to give the admin the power to create new users via the admin interface, as well as to edit any of the existing users’ attributes.

First off, let’s add the user’s email address to the views.

app/views/users/_user.html.erb:

<p>
  <strong>Email:</strong>
  <%= user.email %>
</p>

app/views/users/_form.html.erb:

<div class="field">
  <%= form.label :email %>
  <%= form.text_field :email %>
</div>

In the form partial we can also add password and confirmation fields:

<div class="field">
  <%= form.label :password %>
  <%= form.password_field :password, placeholder: "Leave blank if unchanged" %>
</div>

<div class="field">
  <%= form.label :password_confirmation %>
  <%= form.password_field :password_confirmation %>
</div>

Now things get a little complicated. So as to be able to create a new user via our admin interface (http://localhost:3000/admin/users/new), we need to permit the attributes that Devise requires:

/app/controllers/users_controller.rb

def user_params
  params.expect(user: [
    :name,
    :email,
    :role_id,
    :password,
    :password_confirmation
  ])
end

That works, but if we try and edit an existing user (e.g. http://localhost:3000/admin/users/1/edit), then we are prompted to set a new password every time we want to update something. This is not ideal.

To get around this, we can take advantage of Devise’s update_without_password method.

Alter the update method in UsersController as shown (making sure to include the protected method needs_password?):

def update
  successfully_updated = if needs_password?(@user, user_params)
    @user.update(user_params)
  else
    @user.update_without_password(user_params)
  end

  if successfully_updated
    redirect_to @user, notice: "User was successfully updated."
  else
    render :edit, status: :unprocessable_entity
  end
end

private

def needs_password?(_user, params)
  params[:password].present?
end

And with that we can now create, update and delete users.

Enabling the Trackable Module

Before we finish looking at Devise and authentication, let’s enable the trackable module, to give us a little more information on our users.

In /app/models/users.rb alter the code like so:

class User < ApplicationRecord
  # Include default devise modules. Others available are:
  # :confirmable, :lockable, :timeoutable and :omniauthable
  devise :database_authenticatable, :registerable,
         :recoverable, :rememberable, :validatable, :trackable
  ...
end

Create a new migration:

rails g migration AddDeviseTrackableColumnsToUsers

This will create a new file in the db/migrate folder. Alter it like so:

class AddDeviseTrackableColumnsToUsers < ActiveRecord::Migration[8.0]
  def change
    change_table :users do |t|
      t.integer  :sign_in_count, default: 0, null: false
      t.datetime :current_sign_in_at
      t.datetime :last_sign_in_at
      t.string   :current_sign_in_ip
      t.string   :last_sign_in_ip
    end
  end
end

Then run the migration with rake db:migrate. As you can see from the migration file, this will add several columns to the user table.

Now, add the following to /app/views/users/show.html.erb:

<div class="user-meta">
  <p><strong>Joined on:</strong> <%= @joined_on %></p>
  <p><strong>Last logged in on:</strong> <%= @last_login %></p>
  <p><strong>No. times logged in:</strong> <%= @user.sign_in_count %></p>
</div>

And the logic to the UsersController:

def show
  @joined_on = @user.created_at.to_formatted_s(:short)

  if @user.current_sign_in_at
    @last_login = @user.current_sign_in_at.to_formatted_s(:short)
  else
    @last_login = "never"
  end
end

Also, to show us which users are associated with a particular role, add the association to the Role model.

app/models/role.rb:

has_many :users, dependent: :restrict_with_exception

The restrict_with_exception option will cause an ActiveRecord::DeleteRestrictionError exception to be raised if you try to delete a Role record, but it has associated User records.

Next, add the following to app/views/roles/show.html.erb:

<div class="user-meta">
  <p>
    <strong>Associated users:</strong>
    <%= @associated_users %>
  </p>
</div>

And the logic to the RolesController:

def show
  if @role.users.empty?
    @associated_users = "None"
  else
    @associated_users = @role.users.map(&:name).join(", ")
  end
end

And there we go. In not very much code we have implemented a robust authentication solution for our app, as well as building a simple interface to administer users.

Take a moment to restart the app, have a play with what we’ve got so far and assure yourself that it is working.

Authorization With CanCanCan

CanCanCan is an authorization library for Ruby on Rails that restricts which resources a given user is allowed to access. It is the continuation of the now defunct cancan project started by Ryan Bates (of Railscast fame).

To install it, run:

bundle add cancancan

CanCanCan expects a current_user method to exist in the controller, which we have thanks to Devise.

It also expects user permissions to be defined in an Ability class, for which it includes a generator:

rails g cancan:ability

This will generate a file at app/models/ability.rb. Edit it like so:

class Ability
  include CanCan::Ability

  def initialize(user)
    if user.admin?
      can :manage, :all
    else
      can :read, :all
    end
  end
end

As you can see, abilities are defined with the method can. This method takes two parameters: the first is the action that we want to perform and the second is the model class that the action applies to. In the above example we are permitting the admin to perform any CRUD action on any resource within our app and restricting all other users to just being able to read.

We don’t care about the permissions of those users who have not yet logged in (you remember they should just be directed to the sign in page), but if we did—for example, if they could view items—you would need to initialize the user variable to something sensible. Otherwise you would end up checking for nil values all over the place.

def initialize(user)
  user ||= User.new # guest user

  if user.admin?
    ...
  elsif user.seller?
    ...
  elsif user.regular?
    ...
  else
    # guest permissions
  end
ens

Next we need to define an admin? method in our User model.

app/models/user.rb:

def admin?
  role&.name == "Admin"
end

And in the UsersController we need to specify which users are authorized to do what. You can do this on a per action basis using the authorize! method, which in turn performs the can? check and raises an exception if needed.

For example to check if a given user can edit an item:

def edit
  authorize! :edit, @item
end

However, repeating this check across every action would quickly become tedious. Fortunately, if your controller follows RESTful conventions, CanCanCan provides a shortcut: load_and_authorize_resource.

This method serves two purposes. First, it automatically loads the relevant resource into an instance variable based on the controller name and action—for example, loading the correct Item into @item in an ItemsController. Second, it performs an authorization check against the current user using the rules defined in your Ability class.

In short, it reduces boilerplate by handling both resource loading and permission checks for you.

Place it at the top of your ItemsController like so:

class ItemsController < ApplicationController
  before_action :authenticate_user!
  load_and_authorize_resource
  ...
end

As this method loads the necessary resource for us based on the action we are performing, we can also remove any lines of code that set the instance variable in each action.

In the case of our ItemsController that would mean removing the following lines:

class ItemsController < ApplicationController
-  before_action :set_item, only: [:show, :edit, :update, :destroy]

  def new
-   @item = Item.new
  end

  def create
-    @item = Item.new(item_params)
    ...
  end

  private
-  def set_item
-    @item = Item.find(params[:id])
-  end

end

If you now restart the server and access http://localhost:3000/items as a regular user or as a seller, you will be in read only mode. As an admin, you will still be able to create, edit and delete records.

Now we need to repeat the above steps for our other two resources:

app/controllers/users_controller.rb

class UsersController < ApplicationController
-  before_action :set_user, only: [:show, :edit, :update, :destroy]
  before_action :authenticate_user!
+  load_and_authorize_resource

  def new
-    @user = User.new
  end

  def create
-    @user = User.new(user_params)
    ...
  end

  private
-  def set_user
-    @user = User.find(params[:id])
-  end
end

app/controllers/roles_controller.rb

class RolesController < ApplicationController
-  before_action :set_role, only: [:show, :edit, :update, :destroy]
  before_action :authenticate_user!
+  load_and_authorize_resource

  def new
-    @role = Role.new
  end

  def create
-    @role = Role.new(role_params)
    ...
  end

  private
-  def set_role
-    @role = Role.find(params[:id])
-  end
end

This ensures that regular users and sellers are also in read-only mode when accessing Roles and Users.

Better Error Handling For CanCanCan

Next, we can ensure that our users see a nicer error page than the one with the AccessDenied exception they currently see when attempting to access something they are not authorized to.

We can do this using a method called rescue_from that we can place in our ApplicationController. We’ll pass it a block inside of which we’ll make the application show a flash error message and redirect to the home page.

app/controllers/application_controller.rb

rescue_from CanCan::AccessDenied do
  flash[:error] = "Access denied!"
  redirect_to root_url
end

Now all that remains to do is to set the permissions of regular users and sellers to something sensible.

Let’s define two more methods to check the user’s current role:

app/models/user.rb

def seller?
  role&.name == "Seller"
end

def regular?
  role&.name == "Regular"
end

Then it’s a matter of updating the abilities in the Ability class.

app/models/ability.rb

if user.admin?
  can :manage, :all
elsif user.seller?
  can :read, Item
  can :create, Item
  can :update, Item do |item|
    item.try(:user) == user
  end
  can :destroy, Item do |item|
    item.try(:user) == user
  end
elsif user.regular?
  can :read, Item
end

The permissions for admins and regular users should be straight forward.

In the case of sellers however, things become a little more complicated when dealing with the update and destroy actions—remember that sellers should only be able to update and delete their own items.

Here we pass can a block which will receive the instance of the model we’re checking. The block should return true or false depending on whether the action should be allowed, or not. Within the block we’ll use Rails’ try method to check that the item’s user attribute is the current user. Using try covers the eventuality that the item is nil and will prevent an exception being raised.

Finally, so that regular users and sellers don’t see any links to actions they are not permitted to perform, alter app/views/items/show.html.erb like so:

<% if can? :update, @item %>
  <%= link_to "Edit this item", edit_item_path(@item) %> |
<% end %>
<%= link_to "Back to items", items_path %>
<% if can? :destroy, @item %>
  <%=
    button_to "Destroy this item", @item,
    data: { turbo_method: :delete, turbo_confirm: "Are you sure?" }
  %>
<% end %>

And app/views/items/index.html.erb like so:

<% if can? :create, Item %>
  <%= link_to 'New Item', new_item_path %>
<% end %>

The Finishing Touches

Before we end, let’s add a couple of finishing touches. First, let’s adjust the admin navigation so that it is only displayed for the admin.

<nav class="admin-nav">
  <% if @current_user&.admin? %>
    <%= link_to "Items", items_path %> |
    <%= link_to "Users", users_path %> |
    <%= link_to "Roles", roles_path %>
  <% end %>
</nav>

Next, we can enhance our flash messages by making them fade out and disappear after a short delay. In Rails 8, the idiomatic way to handle this is with Stimulus, which is included by default via Importmap.

First, generate a new Stimulus controller:

rails g stimulus flash

This will create a file at app/javascript/controllers/flash_controller.js. Edit it to contain:

import { Controller } from '@hotwired/stimulus';

export default class extends Controller {
  connect() {
    setTimeout(() => {
      this.element.classList.add('opacity-0');
      setTimeout(() => this.element.remove(), 1500);
    }, 3500);
  }
}

Next, ensure the flash messages are rendered with the appropriate data-controller attribute.

In app/views/layouts/application.html.erb:

<% flash.each do |name, msg| %>
  <%= content_tag :div, msg,
      class: "flash #{name}", data: { controller: "flash" } %>
<% end %>

This will automatically fade out and remove flash messages after 3.5 seconds.

Note that the necessary transition styles were included earlier when we copied the CSS from the project repository. These utility classes handle the fade-out animation applied by the Stimulus controller. If you skipped that step, be sure to add them manually or copy them from below.

.flash {
  ...
  opacity: 1;
  transition: opacity 1.5s ease;
}

.flash.opacity-0 {
  opacity: 0;
}

Finally, the very last thing I want to examine (promise!) is a slightly lesser documented feature of Devise. You remember that we initially wanted users who are not logged in to be redirected to the sign-in page? Well, wouldn’t it be nicer if we directed them to a welcome page with a link to either register or sign in?

To do this we need a WelcomeController and a corresponding view. These can be generated with the following command:

rails g controller welcome index

Now edit the newly generated view template at app/views/welcome/index.html.erb:

<h1>Welcome to MyStore™</h1>
<h2>Supplying unnecessary objects to discerning procrastinators since 2020</h2>
<p>
  At MyStore™, we specialise in products you didn’t ask for, solving problems you don’t really have. From elegantly useless kitchen gadgets to beautifully confusing tech accessories, we’re here to complicate your life with style.
</p>
<p>
  Our mission? To ensure your shelves are full, your drawers are cluttered, and your bank account just slightly emptier than it was five minutes ago. Because nothing says satisfaction like a parcel you forgot you ordered.
</p>
<p>
  We understand your needs. Or at least we pretend to. That’s why we’ve created a shopping experience that’s slick, efficient, and only mildly intrusive. You click, we ship. You regret, we thrive.
</p>
<p>
  So settle in, scroll aimlessly, and add with abandon. MyStore™: because owning things is easier than finding meaning.
</p>

Once that is in place, alter your config/routes.rb to take advantage of Devise’s authenticated route thus:

authenticated :user do
  # Authenticated users see this
  root "items#index", as: :authenticated_root
end

## Everyone else sees this
root "welcome#index"

This gives us:

Rails.application.routes.draw do
  devise_for :users

  authenticated :user do
    root "items#index", as: :authenticated_root
  end
  root "welcome#index"

  scope "/admin" do
    resources :users
  end

  resources :items
  resources :roles

  get "up" => "rails/health#show", as: :rails_health_check
end

Restart your server and we’re done! We have now tidied up the navigation, we have disappearing flash messages and any users who are not logged in will be directed to the front-facing page of our app.

In case you missed it at the beginning, the code for this tutorial is on GitHub: https://github.com/hibbard-eu/authentication-with-devise-and-cancancan

This was quite a long post — I hope it proves useful for people. If you have any questions or comments, I’d be glad to hear them below.

How to Install Ubuntu Server on VirtualBox

2025-05-17T00:00:00+00:00

In this post I’ll show you how to install Ubuntu Server 24.04 LTS (Jammy Jellyfish) on Oracle’s VirtualBox. I’ll also demonstrate how to connect to the Ubuntu instance via SSH, as well as how to run VirtualBox in headless mode.

Let’s get started!

Note: this post was updated on 17^th May, 2025 to use the latest versions of Ubuntu Server and VirtualBox.

What Is VirtualBox?

VirtualBox is a software virtualization package that you can install on your operating system (just as you would a normal program). It supports the creation and management of virtual machines into which you can install a second operating system.

In VirtualBox terminology, the operating system on which you install VirtualBox (i.e. your regular OS) is called the host. The operating system you install within VirtualBox (i.e. inside the virtual machine) is called the guest.

For this tutorial, I’ll be using Linux Mint 22.1 as the host OS, but there’s no reason you couldn’t use a different Linux distro, or macOS, or Windows (if you’re so inclined).

Install VirtualBox

The first thing to do is to get VirtualBox installed. The process varies slightly depending on your host system, but Oracle provides platform-specific installers that are straightforward to use. If you encounter issues or have special requirements, consult the User Guide, especially these sections:

On most Linux distributions, you can install VirtualBox directly via your package manager or software center. For example, on Linux Mint, simply searching for “VirtualBox” will pull in the appropriate package from the official repositories. This is what I did, and it installed version 7.0.16 — slightly behind the latest version (7.1.8) available from the VirtualBox website.

Download Ubuntu Server

The next thing to do is to grab a copy of Ubuntu Server. You can do this from their download page. Hit the green download button and this will download a 3.2GB ISO file to your PC.

At the time of writing the current LTS version is Ubuntu Server 24.04 and this is what I’ll be using. It’s supported until April 2029.

Create a New Virtual Machine

Start up VirtualBox. This should open the VirtualBox Manager, the interface from which you will administer all of your virtual machines.

Click on New (in the top right of the VirtualBox Manager) to proceed.

Give your virtual machine a name — the two dropdown menus should update automatically based on your input.

Next, select the ISO image. To do this, click the ISO Image dropdown (which initially shows <not selected>), then choose Other…. A file picker will open. Navigate to the file you downloaded earlier — it should be called ubuntu-24.04.2-live-server-amd64.iso. Select it and confirm (e.g. click Open).

Also, tick the box labeled Skip Unattended Installation. This gives you full visibility into the Ubuntu setup process and allows for more fine-grained control over how the system is installed.

Once done, click Next.

The wizard will now prompt you to choose how much memory (RAM), in megabytes, to allocate to the virtual machine. I went with 2GB (2048MB).

You’ll also be asked to select the number of processor cores to assign — I chose 2.

Having made your selection, click Next.

On the next screen you’ll be prompted to add a virtual hard disk to the new machine. Make sure Create a Virtual Hard Disk Now is selected.

On this same screen, you can also set the size of the virtual hard disk — I left it at the default 25GB. If you leave the Pre-allocate Full Size box unchecked, the disk will grow dynamically as needed rather than occupying the full 25GB immediately.

Once you’ve configured everything, click Next to continue.

Finally, you’ll see a summary of the configuration you have chosen. Assuming everything is correct, click Finish.

The hard disk should now be created and you should find yourself back in the VirtualBox Manager. You should see your newly created virtual machine listed on the left.

Install Ubuntu Server in the Virtual Machine

In this section, we’ll walk through installing Ubuntu Server as a guest operating system inside VirtualBox, running on your host machine.

Before we get started, a quick note for those of you using a 4K monitor: the VirtualBox window will likely appear incredibly small, making it nearly impossible to read anything. To fix this, open VirtualBox, go to File > Preferences > Display, and set the Scale Factor to 200% (or whatever looks best on your screen). This setting will make the VirtualBox interface much more usable.

Now, with your virtual machine selected, press Start. Once the machine boots, you’ll see a screen offering the option to Try or Install Ubuntu Server. The first option — Install Ubuntu Server — will already be selected. Click anywhere inside the VM window to focus it.

Note the first time you click inside the VM, VirtualBox will display a message explaining that you’ve clicked into the virtual machine and that it is capturing your keyboard and mouse input. This is normal. Simply tick the box that says Do not show this message again, then click Capture.

You will also notice a semi-transparent sidebar on the right side of the VM window. This sidebar cannot be permanently disabled (source). However, you can manage it in two ways:

Click the speech bubble icon with an exclamation mark at the top of the sidebar to temporarily hide it.
For each message, click the speech bubble with a line through it to permanently suppress that specific message.

Once you’re done dismissing or ignoring those messages, press Return (or wait a few seconds) to move into the Ubuntu welcome screen. From there, the installation process will begin.

Navigating the Install Screen

Once the Ubuntu installer loads, you’ll interact with it entirely via the keyboard.

Use Tab to move forward through the available options, and Shift+Tab to move backwards. Press Space to select or deselect checkboxes, and Return to confirm a choice or activate a highlighted button.

These keys are all you’ll need to navigate and complete the installation process in the terminal-based interface. You’ll first use them on the welcome screen to select your language and begin the setup.

The Welcome Screen

Here you should select your preferred language. I’m using English.

Keyboard Configuration

Here you should select a keyboard layout. As I’m using a German keyboard, I asked Ubuntu to detect my layout, which it did with a couple of simple questions.

Type of Install

Next you should choose between the default install that contains a curated set of packages and a minimized version, which has been customized to have a small runtime footprint. Choose the default option.

The Search for third-party drivers option scans your system for hardware that may benefit from proprietary drivers (e.g. graphics, Wi-Fi, or input devices) and installs them if available. I left it unchecked.

Network Connections

Here Ubuntu will attempt to configure the standard network interface. Normally you can just accept the default and select Done.

Configure Proxy

If your system requires a proxy to connect to the internet (mine doesn’t), enter its details in the next dialogue. Then select Done.

Configure Ubuntu Archive Mirror

If you wish to use an alternative mirror for Ubuntu, you can enter the details here. Otherwise accept the default mirror by selecting Done. I accepted the default.

Guided Storage Configuration

The installer can guide you through partitioning an entire disk or, if you prefer, you can do it manually. If you choose to partition an entire disk you will still have a chance to review and modify the results before Ubuntu is installed. I selected Use an entire disk.

You can optionally instruct the installer to set up the disk as an LVM group, as well as to encrypt it using LUKS. I chose to go with the LVM setup, as LVM offers a number of benifits, such as allowing easier backups of a running server. You can read more about LVM here: What is LVM and what is it used for?

Storage Configuration Summary

The next screen summarizes the choices you made in the previous step. If you are happy with everything, select Done.

As this is a “destructive action”. I was asked to confirm my choice with Continue.

Profile Setup

Here you are required to enter:

Your (real) name
Your server’s name
Your username
Password

Fill these details out as you see fit.

Upgrade to Ubuntu Pro

At this point, you’re given the option to enable Ubuntu Pro, a free subscription for personal use that extends security updates to a broader set of packages and adds support for various compliance standards.

I chose to skip this step. You can always enable Ubuntu Pro later using the pro attach command.

SSH Setup

Here we have a chance to install the OpenSSH server package. We’ll need this to connect to the virtual machine via SSH later on, so ensure that you select it.

You also have the opportunity to import your SSH keys from GitHub or Launchpad. I selected No for this option.

Featured Server Snaps

Here you can select from a list of popular snaps to install on your system. Snaps are self-contained software packages that work across a range of Linux distributions. I didn’t select any.

Install and reboot

And that’s it, the installaler will now install Ubuntu 24.04. Once it is finished you should select Reboot Now

Ubuntu will ask you to remove the installation medium and press Enter. You can remove the disk via Devices > Optical Drives > Remove disk from virtual drive. You will need to put a check mark next to ubuntu-22.04-live-server-amd64.iso if it is not selected already.

If for some reason this option is grayed out (it was for me), just hit Return

Up and Running with SSH

Once your virtual machine has rebooted and you have logged in, you’ll probably notice that some packages can be updated.

Let’s fix that:

sudo apt update
sudo apt upgrade

Now let’s double check that SSH is installed (it should be if you selected the option Install OpenSSH server during instalation).

jim@grunt:~$ ssh
usage: ssh [-46AaCfGgKkMNnqsTtVvXxYy] [-B bind_interface] [-b bind_address]
           [-c cipher_spec] [-D [bind_address:]port] [-E log_file]
           [-e escape_char] [-F configfile] [-I pkcs11] [-i identity_file]
           [-J destination] [-L address] [-l login_name] [-m mac_spec]
           [-O ctl_cmd] [-o option] [-P tag] [-p port] [-R address]
           [-S ctl_path] [-W host:port] [-w local_tun[:remote_tun]]
           destination [command [argument ...]]
       ssh [-Q query_option]

If you get a “command not found” error, you can install it with:

sudo apt-get install openssh-server

The next step is to give our Ubuntu server an IP address on our local network. To do this, power off the virtual machine using sudo poweroff or Machine > ACPI Shutdown.

Then, in VirtualBox Manager, make sure your machine is selected and click Settings. Click on Network on the left and change the setting Adapter 1 > NAT to Bridged Adapter and click OK.

Start up your virtual machine, then enter ip address (in the guest) and note the IP address assigned to your main network adapter. In my case this was 192.168.178.49.

Note: it is also possible to stick with the original NAT interface and SSH into the guest using port forwarding. You can read more about that here. You can find information on all of the VBox network settings in this comprehensive guide.

Starting and Stopping VirtualBox in Headless Mode

You might have noticed, working with the VirtualBox Manager and the guest OS is a bit of a pain. If you’re going to continue doing this, you should at least install the guest additions, as well as enable clipboard support.

There is a slightly nicer way however — you can start and stop the virtual machine using the VBoxManage command from your terminal.

To power on:

VBoxManage startvm "Ubuntu Server 24.04" --type headless

And to power off:

VBoxManage controlvm "Ubuntu Server 24.04" poweroff

Where “Ubuntu Server 24.04” is whatever you called your virtual machine (the name it has in the VirtualBox Manager GUI).

Connecting to the Ubuntu Server

First, make sure the virtual machine is shut down. If it’s still running, open the terminal inside Ubuntu and run sudo poweroff.

Once it’s powered down, we’ll start it again in headless mode:

VBoxManage startvm "Ubuntu Server 24.04" --type headless

And then connect to it via SSH.

Note: The following commands should be run on your host.

On most *nix systems, the SSH client software should be part of the default installation. If you don’t have it available, you should be able to grab it from the repos, like so:

sudo apt install openssh-client

or just hit DuckDuckGo.

Then (ensuring that you replace “jim” and the IP address with your corresponding values) you can connect like so:

ssh jim@192.168.178.49

This will give you a warning that the host’s authenticity cannot be established and ask you if you want to continue connecting. Answer “yes”.

Next, it will prompt you for your password. Enter it and you will be connected to your Ubuntu server from your host OS.

For Windows Users

If you’re running Windows you’ll need to install a SSH client such as PuTTY.

When PuTTY starts, a window titled PuTTY Configuration should open. This window has a configuration pane on the left, a Host Name field and other options in the middle, and a pane for saving session profiles in the lower right area.

For simple use, all you need to do is to enter the IP address of the host you want to connect to in the Host Name field and click Open.

Generate and Install a SSH Key Pair

SSH keys offer a secure manner of logging into a server without the need of a password.

In a nutshell, this depends upon you generating a public and a private SSH key pair. The private key is kept on your PC (and should be guarded carefully). The public key is copied over to the server you wish to connect to.

SSH keys are a complex subject and as such, out of the scope of this tutorial. If you’d like to find out more, I recommend looking for a dedicated tutorial (such as this one).

Generate the Keys

On *nix systems (Windows users see the next section), you can generate your key pair with the following command:

ssh-keygen -o -b 4096 -t rsa

The -o option instructs ssh-keygen to store the private key in the new OpenSSH format instead of the old (and more compatible PEM format). This is advisable, as the new OpenSSH format has an increased resistance to brute-force password cracking.

The -b option is used to set the key length to 4096 bits instead of the default 1024 bits for security reasons.

In the following dialogue you will be required to answer a couple of questions:

Where to save the newly generated key pair
Which passphrase to use

Here you can accept the default location and leave the passphrase blank by pressing Return.

ssh-keygen will then output a summary of what it has done:

Generating public/private rsa key pair.
Enter file in which to save the key (/home/jim/.ssh/id_rsa):
Created directory '/home/jim/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/jim/.ssh/id_rsa.
Your public key has been saved in /home/jim/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:sx5uJeVdH/cT/1+GxsSWYzmjf5hUaE33f/e57EbqBfY jim@fitz
The key's randomart image is:
+---[RSA 4096]----+
|                 |
|                o|
|               +o|
|          .  .+==|
|        So . =@oB|
|        .oo o*+BB|
|        oo  ..*EX|
|       o..   +=+=|
|       .o   ..+=+|
+----[SHA256]-----+

Copy the Public Key to the Ubuntu Server

To copy the public key to the Ubuntu server use:

ssh-copy-id -i ~/.ssh/id_rsa.pub jim@192.168.178.49

Where ~/.ssh/id_rsa.pub is the path to your public key, taken from the output above. And where jim@192.168.178.49 should be altered to reflect your details.

The command will run and you should be asked for your server password. Enter it, then attempt to log into the server like so:

ssh jim@192.168.178.49

This time you should be in without a password.

For Windows Users

You should be able to use a tool like PuTTYgen to achieve the same thing. Here is a tutorial on using PuTTYgen to create a new key pair for authentication.

You will have a little more leg work when it comes to copying the key to the server, where you will need to add the public key to a ~/.ssh/authorized_keys file.

You can do that like so (on the guest):

cd
mkdir .ssh
cd .ssh
nano authorized_keys

This will create the appropriate file, then open the nano editor into which you can copy your newly generated public key.

When you’re done, press Ctrl + X to save your changes and exit nano.

Conclusion

This has been quite a long post, but by the end of it you should have a working installation of Ubuntu Server running on VirtualBox that you can connect to from your host operating system via SSH.

If you have any questions or feedback, I’d be glad to hear your from you in the comments.

Fixing Cinnamon Gjs-CRITICAL Errors in Linux Mint After Upgrade

2025-01-19T00:00:00+00:00

Yesterday, I upgraded from Linux Mint 22 to 22.1 Xia using the Update Manager. The process was smooth enough, but having upgraded my .xsession-errors file started filling up with Gjs-CRITICAL errors.

Here’s how I solved the problem.

Quick Fix

For those of you looking for a quick fix, the problem was my workspace switcher applet. I stopped it filling .xsession-errors with junk by altering the display() method on the WorkspaceOsd class in /usr/share/cinnamon/js/ui/workspaceOsd.js to return immediately:

var WorkspaceOsd = GObject.registerClass(
  class WorkspaceOsd extends Clutter.Actor {
    ...
    display(activeWorkspaceIndex, workspaceName) {
      return; // Add this line.
      ...
    }
  }
);

This effectively disabled the Workspace OSD (On-Screen Display), preventing it from rendering or animating during workspace switches.

If that solves your problem, great! If you’d like to understand more about what the problem was and what that line does, read on.

So What is `.xsession-errors`?

.xsession-errors is a log file typically located in your home directory (~/.xsession-errors). It captures output from graphical applications and the X session, including warnings, errors, and debugging messages. If you are looking for this file but cannot see it, you may need to have Linux display hidden files (Ctrl + h).

While this might sound like a useful file to have, occasionally a rogue program might write lots and lots of errors to it, causing the file to balloon to many gigabytes in size. This can be inconvenient at best, or crash your operating system at worst.

You can read about how to set up an early warning system to stop this happening here: How to Prevent the .xsession-errors File From Growing Too Large.

The problem in my case was caused by my workspace switcher applet.

Every time I switched workspace, it was writing ca. 2000 lines of errors to my .xsession-errors file. Most of them looked like this:

(cinnamon:3211): Gjs-CRITICAL **: 12:20:15.112: Object .Gjs_ui_workspaceOsd_WorkspaceOsd (0x5e6cf4bdd290), has been already disposed — impossible to access it. This might be caused by the object having been destroyed from C code using something such as destroy(), dispose(), or remove() vfuncs.
== Stack trace for context 0x5e6cf3617600 ==
#0   7ffd3805b250 b   /usr/share/cinnamon/js/ui/environment.js:156 (201847d7c150 @ 15)
#1   7ffd3805b360 b   self-hosted:221 (201847d70970 @ 267)
#2   7ffd3805bac0 b   /usr/share/cinnamon/js/ui/environment.js:156 (201847d70fb0 @ 625)
#3   7ffd3805c210 b   /usr/share/cinnamon/js/ui/environment.js:360 (201847d7c9c0 @ 23)
#4   5e6cf48b7118 i   /usr/share/cinnamon/js/ui/workspaceOsd.js:124 (251492f0f0b0 @ 108)

Not good!

As one can see from the stack trace, the problem occurs on line 124 in /usr/share/cinnamon/js/ui/workspaceOsd.js

GLib.source_remove(this._timeoutId);

This line removes a previously set timeout, which is used to control the disappearance of the Workspace OSD. The workspace OSD is a visual indicator that appears briefly when switching workspaces, showing the active workspace name or number.

To understand how this code is being triggered, I traced the method containing it. Line 124 is part of the _onTimeout() method:

_onTimeout() {
  GLib.source_remove(this._timeoutId);
  this._timeoutId = 0;
  this.ease({
    opacity: 0.0,
    duration: ANIMATION_TIME,
    mode: Clutter.AnimationMode.EASE_OUT_QUAD,
    onComplete: () => this.destroy(),
  });
  return GLib.SOURCE_REMOVE;
}

This method is called when the timeout expires, and as seen here, it destroys the WorkspaceOsd object. This aligns with the error message about accessing an already disposed object.

Searching the file revealed that _onTimeout() is invoked in the display() method via this line:

this._timeoutId = GLib.timeout_add(GLib.PRIORITY_DEFAULT, DISPLAY_TIMEOUT, this._onTimeout.bind(this));

This is where the timeout is set. The display() method is responsible for initiating the animation and lifecycle of the Workspace OSD, making it the root cause of the issue. Thus, modifying display() to return immediately prevents the problematic chain of events.

var WorkspaceOsd = GObject.registerClass(
  class WorkspaceOsd extends Clutter.Actor {
    ...
    display(activeWorkspaceIndex, workspaceName) {
      return; // Add this line.
      ...
    }
  }
);

You can edit the file in an editor of your choice. I used Sublime Text, but in the worst case, nano would also suffice.

sudo nano /usr/share/cinnamon/js/ui/workspaceOsd.js

Restart Cinnamon

Once the change had been made I needed to restart Cinnamon.

You can do this by pressing Alt + F2, then typing “r” (for restart ) into the Run a Command dialog box that appears.

Alternatively, you can type the following into the terminal:

cinnamon --replace &

Or log out and log back in again.

I hope this helps someone. If you have any questions, you can hit me up in the comments below and I will do my best to help.

Export Thunderbird Accounts to Mobile When Stuck on an Older ESR Release

2024-11-01T00:00:00+00:00

Mozilla recently launched Thunderbird Mobile for Android, and I was eager to give it a try. However, I am running Linux Mint 21.3, and installing Thunderbird from the repositories means you’re stuck with version 115.x, which doesn’t include the “Export to Mobile” feature. This is only available as of Thunderbird Desktop 128.4.0 or Thunderbird Beta 132 and newer.

Manually setting up my many email accounts on mobile wasn’t something I wanted to deal with. Instead, I downloaded a newer portable version of Thunderbird, used that to export my settings to the mobile app, and then reverted to my distro’s ESR release. This post explains how you can do the same.

Note: While I am using Linux Mint, these instructions can be adapted for other operating systems. The steps may differ slightly based on your OS.

1. Downloading a Newer Thunderbird Version

First, visit the Thunderbird website and download a newer version that includes the “Export to Mobile” functionality. I selected the Thunderbird Extended Support Release (default option), which currently provides version 128.4.3.

Clicking Download will save a file named thunderbird-128.4.3esr.tar.bz2 to your Downloads folder. Right-click the file and choose Extract Here.

2. Locating and Preparing Your Profile

Next, locate the folder where Thunderbird stores its profiles. On Linux Mint, this is typically found at home/<user-name>/.thunderbird. Open this folder and note the subfolders it contains. Also, make a copy of installs.ini and profiles.ini.

You’ll also need to know where your current Thunderbird profile is stored. Unless you’ve changed it, it should be in the same folder. However, mine is located at /home/jim/files/Thunderbird, which I find handy for backup purposes. If you’re unsure of your profile’s location, open Thunderbird and navigate to Help > Troubleshooting Information > Profile Directory > Open Directory.

3. Launching the New Version

Ensure that no other versions of Thunderbird are running, then launch the newly downloaded version. You can do this in the folder you just extracted, by either double-clicking the thunderbird executable or running ./thunderbird from a terminal.

When the newer version of Thunderbird starts, it will create a new profile folder in home/<user-name>/.thunderbird. For example, mine was named ee3snioc.default-esr. Once this new profile folder is created, close Thunderbird. Then, rename the new profile folder by appending --orig to its name. I renamed mine to ee3snioc.default-esr--orig.

IMPORTANT: Before proceeding any further, back up your profile folder. Skipping this step may lead to a “You have launched an older version of Thunderbird” error when reverting to your original ESR version.

4. Creating a Symlink to Use Your Existing Profile

To use your existing profile with the new version, you need to create a symlink pointing Thunderbird to your current profile folder.

To create the symlink, open a terminal and use the following syntax: ln -s /path/to/your/profile /path/to/new/profile

In my case, the command was: ln -s /home/jim/files/Thunderbird /home/jim/.thunderbird/c381xert.default-esr.

With this done, restart the newer Thunderbird version. It will use your existing profile, and you can proceed with exporting your settings to the mobile app.

5. Exporting Settings to Thunderbird Mobile

To find the export button, click: ≡ > Tools > Export for Mobile, then follow the instructions on the Mozilla site to complete the process.

6. Cleaning Up and Reverting to the ESR Version

Once everything has been set up on your mobile, close the newly downloaded version of Thunderbird. Then, delete the tar file, the extracted folder, and the original profile folder it created. Also, delete the symlink you made and your profile folder, as well as installs.ini and profiles.ini.

Finally, restore your profile folder, installs.ini, and profiles.ini from the backups you made earlier. Open Thunderbird, and you’ll be back on your distro’s ESR release as though nothing ever happened.

I hope this tip helps someone. If you have any questions, leave a comment below, and I’ll be happy to answer them.

Sorting Selected Taxonomy Terms Alphabetically in ACF’s Select2 Field

2024-07-19T00:00:00+00:00

When using Advanced Custom Fields (ACF) to manage taxonomy terms in a WordPress custom post type, the selected terms often appear in the order they were added rather than alphabetically.

This can be frustrating, especially when dealing with a long list of terms. Without a clear order, finding the right term at a glance can become a little tricky.

Luckily, we can hook into ACF’s filters to modify how the terms are loaded and ensure they are sorted alphabetically.

Click here if you want to jump straight to the solution

Keyword Order in the Post Editor

Before writing any code, let’s take a second to understand the problem.

Imagine we have a Research Papers custom post type and a Keyword taxonomy. Each paper is assigned multiple keywords using an ACF taxonomy field. These keywords help categorize papers based on topics, making it easier to filter and search.

However, when selecting keywords in the WordPress post editor, the order in which they appear is not alphabetical—they show up in the order they were added.

What we really want are the selected keywords to appear alphabetically. This is a more structured approach which makes it easier to find and manage terms, especially as the number of associated keywords grows.

So how can we acheive this?

Hooking into ACF’s Load Filter

In order to sort the keywords, we first need to hook into acf/load_value, a filter that allows us to modify the value of an ACF field before it is loaded in the editor.

Let’s start by adding the following to our functions.php file:

function sort_keywords_alphabetically( $value, $post_id, $field ) {
  error_log($value);

  return $value;
}

// Replace 'keywords' with the ACF field name assigned to your taxonomy.
add_filter(
  'acf/load_value/name=keywords',
  'sort_keywords_alphabetically',
  10, 3
);

Here we are inspect the data we are dealing with before making any modifications. The function logs $value, which represents the currently selected terms in the taxonomy field.

If you’re not sure how to use error_log in WordPress, you can check this Stack Overflow answer.

The add_filter() function attaches our custom function to ACF’s data loading process. The third argument (10) defines the filter’s priority—higher numbers run later, although 10 is the default. The fourth argument (3) tells WordPress to pass three parameters ($value, $post_id, $field) to our function, allowing us to inspect and modify the field’s value.

As we can see from the output, ACF provides an array of term IDs rather than term objects.

Array
(
  [0] => 296
  [1] => 297
  [2] => 293
  [3] => 288
  [4] => 295
  [5] => 289
  [6] => 291
  [7] => 294
  [8] => 290
  [9] => 292
)

Before we can sort the terms alphabetically by name, we need to retrieve their full details.

Change the function like so:

function sort_keywords_alphabetically( $value, $post_id, $field ) {
  $terms = get_the_terms( $post_id, 'keyword' );
  error_log(print_r($terms, true));

  return $value;
}

Now the function uses WordPress’s get_the_terms() to fetch the terms associated with the post. It then logs them using error_log().

Array
(
  [0] => WP_Term Object
    (
      [term_id] => 288
      [name] => Adaptive Neural Networks
      [slug] => adaptive-neural-networks
      [term_group] => 0
      [term_taxonomy_id] => 288
      [taxonomy] => keyword
      [description] =>
      [parent] => 0
      [count] => 1
      [filter] => raw
    )
    ...
)

Sorting the Selected Keywords

With the data structure understood, we can now sort the keywords alphabetically before they are displayed in the post editor.

Change the function like so:

function sort_keywords_alphabetically( $value, $post_id, $field ) {
  $terms = get_the_terms( $post_id, 'keyword' );

  usort( $terms, fn( $a, $b ) => strcmp( $a->name, $b->name ) );

  return array_map( fn( $term ) => $term->term_id, $terms );
}

We are now using usort() to arrange the terms alphabetically by their name property. Once sorted, we extract the term IDs using array_map() and return them. This ensures that the data is in the format that ACF is expecting.

The keywords should now be displayed in alphabetical order within the post editor.

But before we call it a day, we should check whether the field being processed is keywords and whether it has a value:

if ( 'keywords' !== $field['name'] || empty( $value ) ) {
  return $value;
}

This prevents the function from running on other fields and avoids unnecessary processing when no terms are selected. If the field isn’t keywords or the value is empty, we simply return it as is.

Final Code Implementation

Here’s the final version of our function. This ensures that the selected taxonomy terms in ACF taxonomy fields are sorted into in alphabetical order before they appear in the post editor.

function sort_keywords_alphabetically( $value, $post_id, $field ) {
  if ( 'keywords' !== $field['name'] || empty( $value ) ) {
    return $value;
  }

  $terms = get_the_terms( $post_id, 'keyword' );

  usort( $terms, fn( $a, $b ) => strcmp( $a->name, $b->name ) );

  return array_map( fn( $term ) => $term->term_id, $terms );
}

// Replace 'keywords' with the ACF field name assigned to your taxonomy.
add_filter(
  'acf/load_value/name=keywords',
  'sort_keywords_alphabetically',
  10, 3
);

By hooking into the acf/load_value filter, we retrieve the selected terms, sort them by name, and return the ordered list of term IDs.

And that’s it. I hope this post helps someone (even if it’s only me when I next revisit my code).

If you have any questions, please leave a comment below and I’ll do my best to help.

Automating Weekly MySQL Backups via Email

2023-06-21T00:00:00+00:00

Manually backing up databases can be a frustrating and time-consuming task. It often involves logging into servers, managing credentials, transferring files, and keeping track of schedules to avoid missed backups.

In this post, I’ll walk you through the steps I took to create an automated solution for weekly MySQL backups. It involves setting up a cron job to run the backup at regular intervals, then emailing it to myself for easy access.

In this guide I will be using Ubuntu Server 22.04 LTS, however it shouldn’t be too hard to adapt these instructions for other operating systems.

Configuring mysqldump for Password-Free Access

The first step is to programatically generate a backup of the MySQL database. We can use mysqldump for this purpose.

To give it a try, log into your server and run:

mysqldump -u <db-user> -p <db-name> > test_export.sql

This will prompt you to enter your password, then will create a test_export.sql file containing the specified database, in the same directory you ran the command.

Obviously we are not going to enter the password every time, meaning we need to save it somewhere our script will be able to access it.

We could hardcode the password into the script we are about to create, but a more flexible solution is to use a .my.cnf file in our home directory. This file is a user-specific configuration file that allows MySQL tools to securely access credentials without requiring them to be entered manually or exposed in scripts.

You can check if this file exists on your system like so:

test -f ~/.my.cnf && echo "File exists" || echo "File does not exist"

Next, open the file in your editor. If the file doesn’t exist, this will create it:

cd
nano .my.cnf

The cd command navigates to the user’s home directory. For the rest of this guide, I will assume we are working from there.

Now enter:

[client]
user=your_username
password=your_password

Then press Ctrl+O to save the file, followed by Enter, and Ctrl+X to exit the editor.

Once created, ensure the file is secure by setting its permissions to allow only the owner to read it:

chmod 600 .my.cnf

MySQL tools will now automatically use these credentials when executed.

You can try it out to make sure it works. Enter:

mysql

This should land you in the MySQL command-line client without prompting for a username or password. If it doesn’t work, double-check the [client] section in your .my.cnf file and ensure the file permissions are set correctly.

Creating a MySQL Backup Script

Next we need to write a shell script to run the mysqldump command.

nano db-backup.sh

Then, adjusting the first two variables to suit your needs, enter:

#!/bin/bash

# Variables
DB_NAME="your_database_name"
BACKUP_DIR="/path/to/backup"
BACKUP_FILE="$BACKUP_DIR/backup_$(date +%Y%m%d%H%M%S).sql"

# Create backup directory if it doesn't exist
mkdir -p "$BACKUP_DIR"

# Run mysqldump
mysqldump --defaults-file=~/.my.cnf "$DB_NAME" > "$BACKUP_FILE"

# Check if backup was successful
if [ $? -eq 0 ]; then
  echo "Backup successful: $BACKUP_FILE"
else
  echo "Backup failed"
  exit 1
fi

I have added comments to the script to show what each part does. Worthy of note however, is the check to determine whether the mysqldump command was successful by evaluating its exit status ($?). This ensures that we’ll be notified if something goes wrong.

The backup is saved in the directory specified by BACKUP_DIR, which is created automatically if it doesn’t exist. The script generates a timestamped filename for each backup, keeping them organized and easy to manage.

Save the script as db-backup.sh and make it executable:

chmod +x db-backup.sh

Then run it to make sure that it is working as expected:

./db-backup.sh

If everything has gone according to plan, navigate to where you are storing the files and run the ls command. You should see something like:

jim@server:~/db$ ll
backup_20241221075046.sql

Sending the Backup via Email

Once the backup is created, you can email it to yourself using mailx. With a little luck, this may already be installed and configured on your server.

You can test this by sending a simple email (swap out the email address with your own):

echo "This is a test email" | mailx -s "Test Email" your_email@example.com

If you receive the email, mailx is ready to use. If not, you’ll need to install and configure it first.

Setting Up Postfix to Work with Gmail and mailx

The following steps will guide you through installing and configuring Postfix (a Mail Transfer Agent) to work seamlessly with mailx and Gmail. You will need a Gmail account to follow along.

1. Install Postfix and mailx

Begin by installing Postfix and mailutils (which includes mailx):

sudo apt update && sudo apt install mailutils

2. Select the Configuration Type

During installation, choose Internet Site when prompted to set the mail server configuration type. When asked for a domain name, you can enter localhost unless you have a proper domain you wish to use.

3. Edit the Postfix Configuration

Open the Postfix configuration file:

sudo nano /etc/postfix/main.cf

Add or modify the following lines:

relayhost = [smtp.gmail.com]:587
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous

Save and close the file.

4. Obtain an App Password from Gmail

To use Gmail with Postfix, you must generate an app-specific password. This requires enabling two-factor authentication on your Gmail account.

Go to the Security section of your Google account.
Under How you sign in to Google, select 2-Step Verification.
At the bottom of the page, select App Passwords.
Create a new password and note it down for the next step.

If you don’t see the App Passwords option, you can also try the direct link.

5. Create the SASL Password File

Create the file that will store your Gmail credentials:

sudo nano /etc/postfix/sasl_passwd

Add the following line, replacing the email address and password with your Gmail email and app password:

[smtp.gmail.com]:587 your_email@gmail.com:your_app_password

Secure the file and postmap it to create a hashed version:

sudo chmod 600 /etc/postfix/sasl_passwd
sudo postmap /etc/postfix/sasl_passwd

This converts the plain text file into a secure format that Postfix can use for lookups.

6. Restart Postfix

Restart the Postfix service to apply the changes:

sudo systemctl restart postfix

7. Test Email Sending

To test your configuration, send a test email. Replace the email address with the address you wish to send to:

echo "This is a test email" | mailx -s "Test Email" your_email@gmail.com

If everything is configured correctly, you should receive the email. If not, check the logs for debugging:

sudo tail -f /var/log/mail.log

This is a quick overview of setting up Postfix with Gmail. For a more detailed guide, refer to this extensive tutorial on Dev.to.

With mailx now working, we can update the backup script to include email functionality.

First, let’s install sharutils, which includes the uuencode utility used for encoding files for email attachments.

sudo apt update && sudo apt install sharutils

Here is the updated script:

#!/bin/bash

# Variables
DB_NAME="your_database_name"
BACKUP_DIR="/path/to/backup"
BACKUP_FILE="$BACKUP_DIR/backup_$(date +%Y%m%d%H%M%S).sql"
COMPRESSED_FILE="$BACKUP_FILE.gz"

EMAIL="your_email@example.com"
SUBJECT="Database Backup: $(date +%Y-%m-%d)"
BODY="Attached is the database backup."

# Create backup directory if it doesn't exist
mkdir -p "$BACKUP_DIR"

# Run mysqldump
mysqldump --defaults-file=~/.my.cnf "$DB_NAME" > "$BACKUP_FILE"

# Check if backup was successful
if [ $? -eq 0 ]; then
  echo "Backup successful: $BACKUP_FILE"

  # Compress the backup
  gzip "$BACKUP_FILE"
  echo "Backup compressed: $COMPRESSED_FILE"

  # Send email with compressed backup attached
  echo "$BODY" | uuencode "$COMPRESSED_FILE" "$(basename "$COMPRESSED_FILE")" | mailx -s "$SUBJECT" "$EMAIL"
  echo "Email sent to $EMAIL"
else
  echo "Backup failed"
  exit 1
fi

As you can see we have made a couple of changes:

The backup file is now compressed using gzip before being emailed. This reduces the file size, making it faster to send and easier to store.
The script uses uuencode to encode the compressed backup and attach it to the email.
Timestamped filenames are used for both the uncompressed and compressed backups.

If you run the script at this point, you should receive an email with a compressed backup attached.

Automating with Cron

Finally, we want to automate the script to run once a week.

Open the crontab editor by running:

crontab -e

If it your first time running this, you might be asked to select an editor. Choose nano.

Then add the following line to schedule the script to run every Monday at midnight:

0 0 * * 1 /path/to/db-backup.sh

Be sure to replace /path/to/db-backup.sh with the actual path, before saving the file and exiting the editor.

You can confirm the job is scheduled correctly by listing the crontab:

crontab -l

If you want to run the conjob at a different time, or different frequency, you can consult this cron expression generator.

Conclusion

And that’s a wrap!

In this guide, I’ve demonstrated how to automate MySQL backups by running a backup at a regular interval, and emailing the resultant backup files to an aribtary address.

By implementing these steps and scheduling the process with cron, you can maintain regular backups effortlessly and safeguard against data loss.

If you have any questions, or are stuck on a particular part of the tutorial, hit me up in the comments below and I’ll do my best to help.

How to Create a CRUD App with Rails and React

2022-04-01T00:00:00+00:00

Most web applications need to persist data in one form or other. When working with a server-side language, this is normally a straightforward task. However when you add a front-end JavaScript framework to the mix, things start to get a bit trickier.

In this tutorial I am going to demonstrate how to build a JSON API using Ruby on Rails and then code a fully-functional React frontend to interact with the API. The app we’ll be building is an event manager, which will let you create and manage a list of academic events.

The app will showcase basic CRUD functionality and will add a couple of extra features, such as a datepicker and search.

This is what the finished product will look like.

You can find the complete code for the tutorial on GitHub.

This post is also available in Japanese.

Prerequisites

To follow along, you’ll need both Ruby and Node installed on your system. For Ruby you can either go here and download the official binaries for your system, or use a version manager such as rbenv.

The same goes for Node. You can either go here and download the official binaries for your system, or use a version manager such as nvm.

In both cases I would encourage people to use a version manager. They are easy to set up and make managing multiple versions of Node/Ruby a breeze. They also help negate permissions problems, meaning you don’t end up having to install gems/packages with admin rights.

For this tutorial I’ll be using Ruby version 3.1 and Node version 16 (the latest LTS). My operating system is Linux Mint, so any terminal related commands will be tailored towards ‘nix.

Tech Stack

When building an app like this, there are many ways to accomplish the same goal. This section gives you an overview of the libraries I have used and the tech choices I have made.

I am using the following libraries:

I have used SQLite as a database, as this requires least setup and is what Rails uses as a default for a new app.

I am using esbuild to bundle the React app, although I will also show how to set up Shakapacker, the successor to Webpacker.

To reflect current trends in the React community, this tutorial will use hooks and function components, not class-based components. If you would like to read an earlier version of this tutorial which uses class-based components, you can find that here.

Otherwise, I have tried to keep packages and dependencies to a minimum. For example, I have used npm as a package manager (as opposed to installing Yarn) and am making any Ajax requests using the Fetch API, as opposed to using a package such as Axios.

Finally, it is worth mentioning that the React app will live within the app/javascript folder of the Rails app. It would be possible to make the Rails app and the React app separate projects, but for an app of this size, I prefer to keep one within the other.

Choosing a Bundler

Before we issue the command to create a new Rails application, we have to decide how we are going to configure Rails to deal with our JavaScript.

There has been a lot happening in this space recently and one exciting new development is import maps. This is the default in Rails 7.

As the name suggests, this feature lets you import JavaScript modules directly from a CDN (for example) and map these imports to versioned/digested files. This in turn enables you to build JavaScript applications without the need for a transpilation or bundling step.

Unfortunately, when building a React app, this approach isn’t perfect, as you will need to compile the JSX. This is where JavaScript Bundling for Rails comes in. This is a gem that allows you to use either esbuild, rollup.js, or webpack to bundle your JavaScript, then deliver it via the asset pipeline. Of these three bundling options, the Rails community seems most enthusiastic about esbuild, so that is what I will be going with here.

I will however, also demonstrate how to set things up with Shakapacker. This is the successor to the now retired Webpacker gem. It provides a wrapper around the webpack build system, a standard webpack configuration and a reasonable set of defaults.

There are a couple of differences between the two solutions. esbuild is considerably more lightweight, but not as fully featured as Shakapacker. For example it offers no hot module replacement and code splitting is still a work in progress. Transforming ES6+ syntax to ES5 is not supported in esbuild, whereas with Shakapacker (which uses Babel), it is.

If you would like to read more about what JavaScript in Rails currently looks like, check out the following videos and blog post, all by by DHH (the creator of Rails):

Creating a New Rails App

First, let’s install Rails and check the version number:

gem install rails
rails -v
=> 7.0.2.3

Then, choose a bundler (esbuild or Shakapacker) and follow the instructions below.

esbuild

Follow this section if you want a lightweight bundling solution, with no hot module replacement and no transformation of ES6+ syntax.

Create a new Rails project like so:

rails new event-manager -j esbuild

Once the installer has run, change into the app directory and install React.

cd event-manager
npm i react react-dom

If you open the package.json file in the project’s root, you’ll see that there is an npm script to build the app. Let’s add a second one to watch the app/javascript directory for changes and to rebundle everything when any are detected.

"scripts": {
  "build": "esbuild app/javascript/*.* --bundle --sourcemap --outdir=app/assets/builds --loader:.js=jsx",
  "watch": "esbuild app/javascript/*.* --watch --bundle --outdir=app/assets/builds --loader:.js=jsx"
}

Notice that we have used --loader:.js=jsx to tell esbuild to allow JSX syntax in .js files. The alternative here is to give your JSX files a .jsx extension.

Finally edit the Procfile.dev file in the project root:

web: bin/rails server -p 3000
js: npm run watch

Skip the next section and proceed to Creating a Hello World React App.

Shakapacker

Follow this section if you want a fully featured bundler, with hot module replacement and transformation of ES6+ syntax via Babel.

Create a new Rails project like so:

rails new event-manager --skip-javascript

Change into the newly created directory and add Shakapacker to your Gemfile:

bundle add shakapacker --strict

Like Webpacker, Shakapacker relies on yarn, so make sure you have that available:

npm i -g yarn

Then run the following commands:

./bin/bundle install
./bin/rails webpacker:install
yarn add react react-dom @babel/preset-react

Note: there is a bug in the latest version of Shakapacker which results in it not being able to find the Webpacker configuration file when you run rails webpacker:install. If you encounter this, you can pin Shakapacker to its previous version like so: bundle add shakapacker --version "6.2.1" --strict.

Update package.json to add the @babel/preset-react:

"babel": {
  "presets": [
    "./node_modules/shakapacker/package/babel/preset.js",
    "@babel/preset-react"
  ]
},

Creating a Hello World React App

Generate a site controller with an index action. This is where the React app will be served from:

rails g controller site index

Replace the contents of app/views/site/index.html.erb with the following:

<div id="root"></div>

Next, create an App component for our React application inside of a components folder in the app/javascript directory:

mkdir app/javascript/components
touch app/javascript/components/App.js

Add the following code to app/javascript/application.js:

import React from 'react';
import { createRoot } from 'react-dom/client';
import HelloMessage from './components/App';

const container = document.getElementById('root');
const root = createRoot(container);

document.addEventListener('DOMContentLoaded', () => {
  root.render(<HelloMessage name="World" />);
});

This imports a HelloMessage component and renders it in the div element we created above.

Add the following code to app/javascript/components/App.js:

import React from 'react';
const HelloMessage = ({ name }) => <h1>Hello, {name}!</h1>;
export default HelloMessage;

Finally, add a root route to the config/routes.rb file:

Rails.application.routes.draw do
  root to: 'site#index'
end

If you are using esbuild, run: ./bin/dev from the project route.

If you are using Shakapacker, kick off the Rails server in one terminal and the webpack dev server in another:

rails s
./bin/webpacker-dev-server

Then hit http://localhost:3000/. You should see our React app displaying a “Hello, World!” message. 🎉

Troubleshooting SQLite

Depending on your operating system, you may need to install some additional libraries for Rails to interface with SQLite correctly.

On macOS, it seems that SQLite is preinstalled, so no action is necessary.

On Linux you may need to install the SQLite 3 development files:

sudo apt-get install libsqlite3-dev

On Windows you will likely need the SQLite binaries in your path, as suggested in this Stack Overflow answer.

If you run into any difficulties, try searching for the error message.

Hot Module Replacement

As far as following along with this tutorial is concerned, the main difference between the two bundlers is what happens when you make changes to your project files.

In the case of esbuild, it will rebundle everything, but in order to see the changes you need to refresh the browser manually. It doesn’t seem like esbuild has any plans to support HMR, but it does (kinda) work with live reloading. If you’re interested in that, check out Live reloading with Ruby on Rails and esbuild.

Shakapacker on the other hand, comes with live reloading (i.e. an automatic page refresh) out of the box. It can also do Hot Module Replacement (HMR) whereby it automatically updates only that part of the page that changed while preserving your app’s state. This is definitely very convenient, but comes at the cost of a whole bunch of dependencies.

Before deciding on which bundler to use, take a second to weigh up your choices and decide which approach is best for you and the app that you are building.

Enable HMR for Shakapacker

To enable HMR for a React app when using Shakapacker, a little extra configuration is needed. If you are using esbuild, you can skip this section and go on to Building the API.

First off, hop into config/webpacker.yml set hmr is set to true.

Then alter config/webpack/webpack.config.js like so:

const { webpackConfig, inliningCss } = require('shakapacker');
const ReactRefreshWebpackPlugin = require('@pmmmwh/react-refresh-webpack-plugin');
const isDevelopment = process.env.NODE_ENV !== 'production';

if (isDevelopment && inliningCss) {
  webpackConfig.plugins.push(
    new ReactRefreshWebpackPlugin({
      overlay: {
        sockPort: webpackConfig.devServer.port,
      },
    })
  );
}

module.exports = webpackConfig;

Install the react-refresh package, as well as @pmmmwh/react-refresh-webpack-plugin:

yarn add --dev react-refresh @pmmmwh/react-refresh-webpack-plugin

Finally, delete the Babel configuration from package.json:

- "babel": {
-   "presets": [
-     "./node_modules/shakapacker/package/babel/preset.js",
-     "@babel/preset-react"
-   ]
- },

Then create a babel.config.js file in the root of project and add the following:

module.exports = function (api) {
  const defaultConfigFunc = require('shakapacker/package/babel/preset.js')
  const resultConfig = defaultConfigFunc(api)
  const isDevelopmentEnv = api.env('development')
  const isProductionEnv = api.env('production')
  const isTestEnv = api.env('test')

  const changesOnDefault = {
    presets: [
      [
        '@babel/preset-react',
        {
          development: isDevelopmentEnv || isTestEnv,
          useBuiltIns: true
        }
      ]
    ].filter(Boolean),
    plugins: [
      isProductionEnv && ['babel-plugin-transform-react-remove-prop-types',
        {
          removeImport: true
        }
      ],
      process.env.WEBPACK_SERVE && 'react-refresh/babel'
    ].filter(Boolean),
  }

  resultConfig.presets = [...resultConfig.presets, ...changesOnDefault.presets]
  resultConfig.plugins = [...resultConfig.plugins, ...changesOnDefault.plugins ]

  return resultConfig
}

Restart the server, refresh the browser, and with that, HMR for your React app is enabled. 🚀

Building the API

Let’s start off by generating an Event model:

rails g model Event \
event_type:string \
event_date:date \
title:text \
speaker:string \
host:string \
published:boolean

Migrate the database:

rake db:migrate

Next, seed the model with some test data. You can do this by creating a db/seeds/events.json file and adding the contents from the corresponding file in the project repo.

Then in db/seeds.rb, add:

json = ActiveSupport::JSON.decode(File.read('db/seeds/events.json'))
json.each do |record|
  Event.create!(record)
end

And run rake db:seed. Start up the rails console with rails c and confirm that you have some data:

rails c
Loading development environment (Rails 7.0.2.3)
irb(main):001:0> Event.all.count
   (1.7ms)  SELECT sqlite_version(*)
  Event Count (0.2ms)  SELECT COUNT(*) FROM "events"
=> 6

Controllers

In the next step, we’ll create an Events controller to respond to incoming requests to our API. We’ll put the controller in its own folder, as we’re going to namespace it. This will keep our code nice and organized and allow us to create our own set of routes for the API.

mkdir app/controllers/api
touch app/controllers/api/events_controller.rb

Add the following code to app/controllers/api/events_controller.rb:

class Api::EventsController < ApplicationController
  before_action :set_event, only: %i[show update destroy]

  def index
    @events = Event.all
    render json: @events
  end

  def show
    render json: @event
  end

  def create
    @event = Event.new(event_params)

    if @event.save
      render json: @event, status: :created
    else
      render json: @event.errors, status: :unprocessable_entity
    end
  end

  def update
    if @event.update(event_params)
      render json: @event, status: :ok
    else
      render json: @event.errors, status: :unprocessable_entity
    end
  end

  def destroy
    @event.destroy
  end

  private

  def set_event
    @event = Event.find(params[:id])
  end

  def event_params
    params.require(:event).permit(
      :id,
      :event_type,
      :event_date,
      :title,
      :speaker,
      :host,
      :published,
      :created_at,
      :updated_at
    )
  end
end

This is a basic set of controller methods to make up the CRUD functionality of our API. Hopefully the code is easy enough to follow, as I don’t want to go into it in much depth here. If you are new to Rails and would like to find out more about API building, check out How to Create a Rails Backend API.

The final thing we need to do regarding controllers is to change the forgery protection method in app/controllers/application_controller.rb:

class ApplicationController < ActionController::Base
  protect_from_forgery with: :null_session
end

The reason this is necessary is that Rails has a built in mechanism to protect against cross site request forgery (CSRF) attacks. By default this sees Rails generate a unique token and validate its authenticity with each POST PUT PATCH DELETE request. If the token is missing, Rails will throw an exception.

However, as we are building a single-page app, we will only have a fresh token upon first render, which means we will need to alter this behavior. The above code ensures that if no CSRF token is provided, Rails will respond with an empty session, which will prevent any other scripts from using our authenticated session to do bad things

If you’d like to read more about this, check out:

Routes

Finally let’s fix up the routes in config/routes.rb. The routing for the controller has to consider the fact that it’s within the Api namespace. We’ll do this using the namespace method.

Rails.application.routes.draw do
  root to: 'site#index'

  namespace :api do
    resources :events, only: %i[index show create destroy update]
  end
end

At this point if you can hit the various endpoints ( e.g. http://localhost:3000/api/events) and interact with the API.

You might also like to test the API with Postman. Here’s how you would create a new event.

Set the request type to POST, the URL to http://localhost:3000/api/events, the Headers to Content-Type: application/json and under Body > raw enter:

{
  "event": {
    "event_type": "Colloquium",
    "event_date": "2022-07-21",
    "title": "Investigating the Battle of Hastings",
    "speaker": "Sarah Croix",
    "host": "Jim Bradbury",
    "published": false
  }
}

Then hit send and you should see a response similar to that below.

You could also accomplish the same thing using curl:

curl --location --request POST 'http://localhost:3000/api/events' \
--header 'Content-Type: application/json' \
--data-raw '{
  "event": {
    "event_type": "Colloquium",
    "event_date": "2022-07-21",
    "title": "Investigating the Battle of Hastings",
    "speaker": "Sarah Croix",
    "host": "Jim Bradbury",
    "published": false
  }
}'

Before moving on, check either the Rails console or http://localhost:3000/api/events to satisfy yourself that the event has been created.

Scaffolding the Event Manager

Next we need to think about how to structure our app’s UI. We’ll start off with an <Editor> component which will contain the following child components:

A <Header> component to display our app’s title
An <EventList> component to display a list of events
An <Event> component to display individual events
An <EventForm> component to allow us to edit and create events

The whole thing will look like this:

Fetching Events

Let’s start off by creating the files we will need in this section:

touch app/javascript/components/{Editor.js,Header.js,EventList.js}

Please note that from now on I won’t give the full path of the React components. They are all located in app/javascript/components

Next, install React’s prop-types package. This package will allow us to to document the intended types of component properties and also to make sure any values passed are of the correct data type.

npm i prop-types

If you are using Shakapacker, remember to use Yarn to install your dependencies. In this case, the command would be yarn add prop-types.

Alter app/javascript/application.js thus:

import React, { StrictMode } from 'react';
import { createRoot } from 'react-dom/client';
import App from './components/App';

const container = document.getElementById('root');
const root = createRoot(container);

document.addEventListener('DOMContentLoaded', () => {
  root.render(
    <StrictMode>
      <App />
    </StrictMode>
  );
});

When using esbuild, Rails will add imports for “@hotwired/turbo-rails” and “./controllers”. These relate to Turbo and Stimulus, which together form the core of Hotwire. They are not relevant to our React application and I would recommend just leaving them as they are.

You will also notice that we are wrapping the <App> component in a <StrictMode> component. This does not render any visible UI, rather it is a helper component that activates additional checks and warnings for its descendants while in development mode. You can read more about it here.

Data Fetching With React Hooks

Now we can get on to building the React app. Let’s start off in App.js where we will require and render our <Editor> component.

import React from 'react';
import Editor from './Editor';

const App = () => <Editor />;

export default App;

Next, add the following code to Editor.js:

import React, { useState, useEffect } from 'react';
import Header from './Header';
import EventList from './EventList';

const Editor = () => {
  const [events, setEvents] = useState([]);
  const [isLoading, setIsLoading] = useState(true);
  const [isError, setIsError] = useState(false);

  useEffect(() => {
    const fetchData = async () => {
      try {
        const response = await window.fetch('/api/events');
        if (!response.ok) throw Error(response.statusText);
        const data = await response.json();
        setEvents(data);
      } catch (error) {
        setIsError(true);
        console.error(error);
      }

      setIsLoading(false);
    };

    fetchData();
  }, []);

  return (
    <>
      <Header />
      {isError && <p>Something went wrong. Check the console.</p>}

      {isLoading ? <p>Loading...</p> : <EventList events={events} />}
    </>
  );
};

export default Editor;

There is a bit more going on here, so let’s break it down. Essentially we want to contact our API, grab a list of events and pass them to the <EventList> component, so that it can display them on the page.

We start off by employing the useState hook to declare three variables in state (events, isLoading and isError), as well as functions to set the values of these variables. We also assign them some initial values.

Next comes a useEffect hook to handle our data fetching. As we are passing it an empty array as a second argument, this will run once when the component is rendered. This functions similarly to componentDidMount in a class-based component.

Inside the useEffect hook, we declare a fetchData function, which uses the Fetch API to hit the /api/events endpoint. Assuming this returns a valid JSON response (a list of events), we save that to the events variable in state.

The data fetching happens within a try... catch block, so that we can handle any errors that might occur. Note that the Promise returned from fetch() won’t reject according to HTTP error status, even if the response is 404 or 500. This is why we have to inspect the response’s ok property and catch any errors manually. You can read more about that here: Handling Failed HTTP Responses With fetch().

Once the data fetching has completed, we set the isLoading variable to false.

If you want to check the loading effect, add a sleep 5 to the index method in the EventsController.

The last thing we do inside the hook is to invoke the fetchData function. We need a separate function here, as we cannot mark the callback function we pass to the useEffect hook as being async.

Finally, we return some JSX. This consists of the <Header> component we will declare shortly, then either an error message, a loading message, or the <EventList> component to which we pass a list of events. The <Editor> component works out which of these to render based on the value of the isError and isLoading variables we declared previously.

If you are new to using hooks in React, I suggest checking out this article over on SitePoint to get up to speed. If you would like to dive into data fetching in React using hooks, I recommend this tutorial by Robin Wieruch.

In Header.js:

import React from 'react';

const Header = () => (
  <header>
    <h1>Event Manager</h1>
  </header>
);

export default Header;

Nothing exciting going on here. We’re just rendering a header element.

In EventList.js add the following:

import React from 'react';
import PropTypes from 'prop-types';

const EventList = ({ events }) => {
  const renderEvents = (eventArray) => {
    eventArray.sort((a, b) => new Date(b.event_date) - new Date(a.event_date));

    return eventArray.map((event) => (
      <li key={event.id}>
        {event.event_date}
        {' - '}
        {event.event_type}
      </li>
    ));
  };

  return (
    <section>
      <h2>Events</h2>
      <ul>{renderEvents(events)}</ul>
    </section>
  );
};

EventList.propTypes = {
  events: PropTypes.arrayOf(PropTypes.shape({
    id: PropTypes.number,
    event_type: PropTypes.string,
    event_date: PropTypes.string,
    title: PropTypes.string,
    speaker: PropTypes.string,
    host: PropTypes.string,
    published: PropTypes.bool,
  })).isRequired,
};

export default EventList;

This component receives an array of event objects as props (events) and is responsible for displaying them as an ordered list. This happens in the renderEvents method which sorts the array by date in descending order, then renders a list item for each event.

Note that we have also implemented some prop validation to ensure that the events prop is an array of objects and that each object in that array has a certain set of properties, each of a certain type. We are specifying that the events prop and all of the object properties are required. Consequently, while in development mode, an error will be thrown if anything is missing or of the incorrect type.

If you now visit http://localhost:3000 you should see a list of events displayed. Exciting, huh?

Adding Some Development Tooling

Now that we’re writing some JavaScript, it’s a good time to install a couple of tools to aid our development process and to ensure the quality of our code.

ESLint

ESLint is a tool for identifying common errors and problematic patterns in JavaScript code. As regards code quality, this is one of the most useful tools in a JavaScript developer’s toolbox. You can install it like so:

npm i -D eslint

Then add the Airbnb config to the project. This provides a set of rules corresponding to the Airbnb JavaScript Style Guide:

npm i -D eslint-config-airbnb

If you are using Yarn, this will be yarn add --dev eslint eslint-config-airbnb.

Next, find out what the remaining dependencies are:

npm info "eslint-config-airbnb@latest" peerDependencies

Outputs:

{
  eslint: '^7.32.0 || ^8.2.0',
  'eslint-plugin-import': '^2.25.3',
  'eslint-plugin-jsx-a11y': '^6.5.1',
  'eslint-plugin-react': '^7.28.0',
  'eslint-plugin-react-hooks': '^4.3.0'
}

Add the final four packages to the devDependencies section of package.json and run npm i (or yarn install) to pull them in.

"devDependencies": {
  ...
  "eslint-plugin-import": "^2.25.3",
  "eslint-plugin-jsx-a11y": "^6.5.1",
  "eslint-plugin-react": "^7.28.0",
  "eslint-plugin-react-hooks": "^4.3.0"
}

Create an .eslintrc.js file in the project root and add:

module.exports = {
  root: true,
  extends: ['airbnb', 'airbnb/hooks'],
  rules: {
    'react/jsx-filename-extension': [1, { extensions: ['.js', '.jsx'] }],
    'react/function-component-definition': [
      1,
      { namedComponents: 'arrow-function' },
    ],
    'no-console': 0,
    'no-alert': 0,
  },
};

This will tell ESLint to use the Airbnb ruleset we just installed and to enable the linting rules for React hooks . It will also allow files with a js ending to contain JSX, switch off warnings for console and alert statements and allow us to use arrow function syntax for our function components.

If you would prefer to enforce a different function type for function components, you can read about how to configure that rule here.

You can run ESLint from the terminal:

./node_modules/.bin/eslint app/javascript

Or as an npm script:

"scripts": {
  "lint": "eslint app/javascript"
},

But for the best results, you’ll probably want to integrate it into your editor. I’m using Sublime Text 3 with SublimeLinter, SublimeLinter-eslint and SublimeJsPrettier to great effect. I also use eslint-config-prettier, which turns off any linting rules that might conflict with Prettier.

If you’re more of a VS Code person, you might like this video by Wes Bos: ESLint + Prettier + VS Code — The Perfect Setup.

React Developer Tools

While we are looking at tooling, you might also like to take a minute to check out React’s Developer Tools. These let you inspect the React component hierarchy, including component props and state and are available as a browser extension (for Chrome and Firefox), and as a standalone app.

You won’t need the React Developer Tools overly much if you follow along with this tutorial verbatim, but as soon as you start to try things out and deviate from what I am doing, they will be extremely helpful in understanding what is going on under the hood.

Displaying an Event

Next, let’s make the events list clickable, so that when a user selects an event, its details are displayed on the screen. For this we’re going to need React router, which will change the URL to reflect the current event and provide us with an outlet for our event information.

React Router

You can install React Router like so:

npm i react-router-dom@6

Or like so if you are using Yarn:

yarn add react-router-dom@6

As you can see, we are using the latest version of React Router (version 6). You should be aware that this library underwent a major rewrite between versions 5 and 6. If you would like to take a closer look at working with React Router 6, I recommend this tutorial by Robin Wieruch.

To begin, let’s sort out the routes in config/routes.rb:

Rails.application.routes.draw do
  root to: redirect('/events')

  get 'events', to: 'site#index'
  get 'events/new', to: 'site#index'
  get 'events/:id', to: 'site#index'
  get 'events/:id/edit', to: 'site#index'

  namespace :api do
    resources :events, only: %i[index show create destroy update]
  end
end

In the first line we’re pointing our root route to http://localhost:3000/events, this is purely for aesthetic reasons. However in the four lines that follow, you can see that we are informing Rails about the routes we will be using in our React application. This is important, as otherwise if a user requested any of these routes directly (by refreshing the page, for example), Rails would know nothing about them and would respond with a 404. Doing things this way means that Rails can simply serve our React app and let it work out which view to display.

Now let’s add the router to app/javascript/application.js:

import React, { StrictMode } from 'react';
import { createRoot } from 'react-dom/client';
import { BrowserRouter } from 'react-router-dom';
import App from './components/App';

const container = document.getElementById('root');
const root = createRoot(container);

document.addEventListener('DOMContentLoaded', () => {
  root.render(
    <StrictMode>
      <BrowserRouter>
        <App />
      </BrowserRouter>
    </StrictMode>
  );
});

This wraps the app in a <BrowserRouter> component, that uses the HTML5 history API to keep the UI in sync with the URL.

A small change is necessary in App.js:

import React from 'react';
import { Routes, Route } from 'react-router-dom';
import Editor from './Editor';

const App = () => (
  <Routes>
    <Route path="events/*" element={<Editor />} />
  </Routes>
);

export default App;

Instead of rendering our <Editor> component directly, we will now use a <Route> component to render it whenever the browser’s URL begins with events/.

To make the event display in the correct place, we need to use a further route. In the <Editor> component:

...
import { Routes, Route } from 'react-router-dom';
import Event from './Event';

const Editor = () => {
  ...
  return (
    <>
      <Header />
      {isError && <p>Something went wrong. Check the console.</p>}
      {isLoading ? (
        <p>Loading...</p>
      ) : (
        <>
          <EventList events={events} />

          <Routes>
            <Route path=":id" element={<Event events={events} />} />
          </Routes>
        </>
      )}
    </>
  );
};

export default Editor;

As in the <App> component, we are using a <Route> component, whose path we set to :id. This is known as a dynamic segment which will match the ID of the current event.

This means that given a URL such as http://localhost:3000/events/7, the following will happen:

application.js will render our <App> component, wrapped in a <BrowserRouter>
In the <App> component, the <Route> component will match the events/ part of the URL and render the <Editor> component.
In the <Editor> component, the <Route> component will match the remainder of the URL (i.e. 7) and render the <Event> component, passing it the list of events we fetched previously.
Inside the <Event> component, the :id section of the URL will be available to our code.

Take a moment to ensure you understand everything that is going on here, then move on to the next section.

The `<Event>` Component

Next, we’ll need an <Event> component to display the event.

touch app/javascript/components/Event.js

Then add:

import React from 'react';
import PropTypes from 'prop-types';
import { useParams } from 'react-router-dom';

const Event = ({ events }) => {
  const { id } = useParams();
  const event = events.find((e) => e.id === Number(id));

  return (
    <>
      <h2>
        {event.event_date}
        {' - '}
        {event.event_type}
      </h2>
      <ul>
        <li>
          <strong>Type:</strong> {event.event_type}
        </li>
        <li>
          <strong>Date:</strong> {event.event_date}
        </li>
        <li>
          <strong>Title:</strong> {event.title}
        </li>
        <li>
          <strong>Speaker:</strong> {event.speaker}
        </li>
        <li>
          <strong>Host:</strong> {event.host}
        </li>
        <li>
          <strong>Published:</strong> {event.published ? 'yes' : 'no'}
        </li>
      </ul>
    </>
  );
};

Event.propTypes = {
  events: PropTypes.arrayOf(
    PropTypes.shape({
      id: PropTypes.number.isRequired,
      event_type: PropTypes.string.isRequired,
      event_date: PropTypes.string.isRequired,
      title: PropTypes.string.isRequired,
      speaker: PropTypes.string.isRequired,
      host: PropTypes.string.isRequired,
      published: PropTypes.bool.isRequired,
    })
  ).isRequired,
};

export default Event;

Here we are importing the useParams hook from React Router. This hook gives us access to an object containing the dynamic params from the current URL that were matched by the <Route path> (:id in our case).

We then grab the ID of the current event using destructuring assignment and use that ID to filter the list of events we passed as props, and find the event we want to display.

I would have preferred to only pass the component the event it needs to display (as opposed to all of them), but the way that React Router now works, made this rather tricky. I did also consider sticking all of the events into Context, but that would have increased the complexity of an already long tutorial and would have been overkill in this case. Nonetheless, be aware that this is an option open to you.

The rest of the code is hopefully easy enough to understand, with some JSX being returned and the same prop validation happening as before.

Making Events Clickable

Finally, let’s make the list of events in <EventList> clickable. When clicked, they should navigate to /events/:id.

import { Link } from 'react-router-dom';

...

const renderEvents = (eventArray) => {
  eventArray.sort((a, b) => new Date(b.event_date) - new Date(a.event_date));

  return eventArray.map((event) => (
    <li key={event.id}>
      <Link to={`/events/${event.id}`}>
        {event.event_date}
        {' - '}
        {event.event_type}
      </Link>
    </li>
  ));
};

Here, we are making use of React router’s <Link> component to create the navigation around our application.

And now when you click on a link, the correct event should display.

Adding Some Styling

The app looks pretty ugly right now, so let’s brighten it up a little. Create a file named App.css:

touch app/javascript/components/App.css

And add the following styles:

body, html, div, blockquote, img, label, p, h1, h2, h3, h4, h5, h6, pre, ul, ol, li, dl, dt, dd, form, a, fieldset, input, th, td {
  margin: 0;
  padding: 0;
}

ul, ol {
  list-style: none;
}

body {
  font-family: Roboto;
  font-size: 16px;
  line-height: 28px;
}

header {
  background: #f57011;
  height: 60px;
}

header h1, header h1 a{
  display: inline-block;
  font-family: "Maven Pro";
  font-size: 28px;
  font-weight: 500;
  color: white;
  padding: 14px 5%;
  text-decoration: none;
}

header h1:hover {
  text-decoration: underline;
}

.grid {
  display: grid;
  grid-gap: 50px;
  grid-template-columns: minmax(250px, 20%) auto;
  margin: 25px auto;
  width: 90%;
  height: calc(100vh - 145px);
}

.eventList {
  background: #f6f6f6;
  padding: 16px;
}

.eventList h2 {
  font-size: 20px;
  padding: 8px 6px 10px;
}

.eventContainer {
  font-size: 15px;
  line-height: 35px;
}

.eventContainer h2 {
  margin-bottom: 10px;
}

.eventList li:hover, a.active {
  background: #f8e5ce;
}

.eventList a {
  display: block;
  color: black;
  text-decoration: none;
  border-bottom: 1px solid #dddddd;
  padding: 8px 6px 10px;
  outline: 0;
}

.eventList h2 > a {
  color: #236fff;
  font-size: 15px;
  float: right;
  font-weight: normal;
  border-bottom: none;
  padding: 0px;
}

.eventForm {
  margin-top: 15px;
}

label > strong {
  display: inline-block;
  vertical-align: top;
  text-align: right;
  width: 100px;
  margin-right: 6px;
  font-size: 15px;
}

input, textarea {
  padding: 2px 0 3px 3px;
  width: 400px;
  margin-bottom: 15px;
  box-sizing: border-box;
}

input[type="checkbox"] {
  width: 13px;
}

button[type="submit"] {
  background: #f57011;
  border: none;
  padding: 5px 25px 8px;
  font-weight: 500;
  color: white;
  cursor: pointer;
  margin: 10px 0 0 106px;
}

.errors {
  border: 1px solid red;
  border-radius: 5px;
  margin: 20px 0 35px 0;
  width: 513px;
}

.errors h3 {
  background: red;
  color: white;
  padding: 10px;
  font-size: 15px;
}

.errors ul li {
  list-style-type: none;
  margin: 0;
  padding: 8px 0 8px 10px;
  border-top: solid 1px pink;
  font-size: 12px;
  font-weight: 0.9;
}

button.delete {
  background: none !important;
  border: none;
  padding: 0 !important;
  margin-left: 10px;
  cursor: pointer;
  color: #236fff;
  font-size: 15px;
  font-weight: normal;
  text-decoration: none;
}

button.delete:hover {
  text-decoration: underline;
}

h2 a {
  color: #236fff;
  font-size: 15px;
  font-weight: normal;
  margin: 3px 12px 0 12px;
  text-decoration: none;
}

h2 a:hover {
  text-decoration: underline;
}

.form-actions a {
  color: #236fff;
  font-size: 15px;
  margin: 3px 12px 0 12px;
  text-decoration: none;
}

.form-actions a:hover {
  text-decoration: underline;
}

input.search {
  width: 92%;
  margin: 15px 2px;
  padding: 4px 0 6px 6px;
}

.loading {
  height: calc(100vh - 60px);
  display: grid;
  justify-content: center;
  align-content: center;
}

Please note that these are all of the styles we will need in the app. Listing them all in one go is intended to keep the article a tad shorter.

Here, we’re using a small custom reset and the goodness of CSS grid for our layout. If you’re unfamiliar with CSS grid, there’s a good tutorial here: A Beginners Guide to CSS Grid Layout

Import our styles in App.js:

import './App.css';

Next, alter the markup in the <Editor> component:

return (
  <>
    <Header />
    <div className="grid">
      {isError && <p>Something went wrong. Check the console.</p>}
      {isLoading ? (
        <p className='loading'>Loading...</p>
      ) : (
        <>
          <EventList events={events} />

          <Routes>
            <Route path=":id" element={<Event events={events} />} />
          </Routes>
        </>
      )}
    </div>
  </>
);

The <EventList> component:

return (
  <section className="eventList">
    <h2>Events</h2>
    <ul>{renderEvents(events)}</ul>
  </section>
);

And the <Event> component:

return (
  <div className="eventContainer">
    <h2> ... </h2>
    <ul> ... </ul>
  </div>
);

Open app/views/layouts/application.html.erb and add a couple of custom fonts:

<head>
  <title>EventManager</title>
  <meta name="viewport" content="width=device-width,initial-scale=1">
  <%= csrf_meta_tags %>
  <%= csp_meta_tag %>

  <link rel="preconnect" href="https://fonts.googleapis.com">
  <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
  <link href="https://fonts.googleapis.com/css2?family=Maven+Pro:wght@400;500;700&family=Roboto:ital,wght@0,300;0,400;0,700;1,400&display=swap" rel="stylesheet">

  ...
</head>

Bundler Specific Configuration

Depending on the bundler you are using, some extra configuration is necessary.

esbuild

If you are using esbuild, you need to delete app/assets/stylesheets/application.css from the project, as otherwise the next time you start the Rails server you will see the following warning:

ActionView::Template::Error (Multiple files with the same output path cannot be linked (“application.css”)

You could alternatively choose to not create an App.css file and add all of the CSS to app/assets/stylesheets.css for Rails’ asset pipeline to handle. It doesn’t make a big difference in the case of this tutorial, but I for your average React app, it makes more sense for the CSS to live alongside the components it is styling.

Shakapacker

If you are using Shakapacker, you’ll need to install some more packages, so that it can handle CSS files:

yarn add css-loader style-loader mini-css-extract-plugin css-minimizer-webpack-plugin

Then restart the server for good measure.

See the Shakapacker docs for more details: https://github.com/shakacode/shakapacker#css

Either way, now everything should be styled nicely.

Highlighting the Selected Event

Before we move on, let’s make one final tweak and add some styling to the selected event in the list of events. This makes it easier for the user to see which event they are currently viewing at a glance.

All we have to do here is to swap the <Link> component, for a <NavLink> component in the <EventList> component.

import { Link, NavLink } from 'react-router-dom';

const EventList = ({ events }) => {
  const renderEvents = (eventArray) => {
    eventArray.sort((a, b) => new Date(b.event_date) - new Date(a.event_date));

    return eventArray.map((event) => (
      <li key={event.id}>
        <NavLink to={`/events/${event.id}`}>
          {event.event_date}
          {' - '}
          {event.event_type}
        </NavLink>
      </li>
    ));
  };

  ...
};

A <NavLink> is a is a special kind of <Link> that will add an “active” class to the currently active link, which we can now target through our CSS

Creating an Event

So far we have the Read functionality of our CRUD app. Now let’s add the ability to create an event.

Start off in the <Editor> component:

...
import EventForm from './EventForm';

const Editor = () => {
  ...

  return (
    <>
      <Header />
      <div className="grid">
        {isError && <p>Something went wrong. Check the console.</p>}
        {isLoading ? (
          <p>Loading...</p>
        ) : (
          <>
            <EventList events={events} />

            <Routes>
              <Route path="new" element={<EventForm />} />
              <Route path=":id" element={<Event events={events} />} />
            </Routes>
          </>
        )}
      </div>
    </>
  );
}

Here, we’ve added a new <Route> component, whose path property is set to “new”. When this matches the current URL (i.e. /events/new), it will render an <EventForm> component, which will contain our form for adding (and later editing) events.

Next, let’s add a link to display the form in the <EventList> component.

const EventList = ({ events }) => {
  const renderEvents = (eventArray) => { ... };
  ...
  return (
    <section className="eventList">
      <h2>
        Events
        <Link to="/events/new">New Event</Link>
      </h2>
      <ul>{renderEvents(events)}</ul>
    </section>
  );
};

Now, let’s create the <EventForm> component:

touch app/javascript/components/EventForm.js

And add the following content:

import React from 'react';

const EventForm = () => {
  const handleSubmit = (e) => {
    e.preventDefault();
    console.log('Submitted');
  };

  return (
    <section>
      <h2>New Event</h2>
      <form className="eventForm" onSubmit={handleSubmit}>
        <div>
          <label htmlFor="event_type">
            <strong>Type:</strong>
            <input type="text" id="event_type" name="event_type" />
          </label>
        </div>
        <div>
          <label htmlFor="event_date">
            <strong>Date:</strong>
            <input type="text" id="event_date" name="event_date" />
          </label>
        </div>
        <div>
          <label htmlFor="title">
            <strong>Title:</strong>
            <textarea cols="30" rows="10" id="title" name="title" />
          </label>
        </div>
        <div>
          <label htmlFor="speaker">
            <strong>Speakers:</strong>
            <input type="text" id="speaker" name="speaker" />
          </label>
        </div>
        <div>
          <label htmlFor="host">
            <strong>Hosts:</strong>
            <input type="text" id="host" name="host" />
          </label>
        </div>
        <div>
          <label htmlFor="published">
            <strong>Publish:</strong>
            <input type="checkbox" id="published" name="published" />
          </label>
        </div>
        <div className="form-actions">
          <button type="submit">Save</button>
        </div>
      </form>
    </section>
  );
};

export default EventForm;

At this point the form should appear and when you click Save, it should log “Submitted” to the console.

Form Validation

Now, let’s add in some validation to make sure all of the fields (apart from published) are filled out. All of the action will take place in the <EventForm> component.

Change the code in that file, so that it looks like so:

import React, { useState } from 'react';

const EventForm = () => {
  const [event, setEvent] = useState({
    event_type: '',
    event_date: '',
    title: '',
    speaker: '',
    host: '',
    published: false,
  });

  const [formErrors, setFormErrors] = useState({});

  const handleInputChange = (e) => {
    const { target } = e;
    const { name } = target;
    const value = target.type === 'checkbox' ? target.checked : target.value;

    setEvent({ ...event, [name]: value });
  };

  const validateEvent = () => {
    const errors = {};

    if (event.event_type === '') {
      errors.event_type = 'You must enter an event type';
    }

    if (event.event_date === '') {
      errors.event_date = 'You must enter a valid date';
    }

    if (event.title === '') {
      errors.title = 'You must enter a title';
    }

    if (event.speaker === '') {
      errors.speaker = 'You must enter at least one speaker';
    }

    if (event.host === '') {
      errors.host = 'You must enter at least one host';
    }

    return errors;
  };

  const isEmptyObject = (obj) => Object.keys(obj).length === 0;

  const renderErrors = () => {
    if (isEmptyObject(formErrors)) {
      return null;
    }

    return (
      <div className="errors">
        <h3>The following errors prohibited the event from being saved:</h3>
        <ul>
          {Object.values(formErrors).map((formError) => (
            <li key={formError}>{formError}</li>
          ))}
        </ul>
      </div>
    );
  };

  const handleSubmit = (e) => {
    e.preventDefault();
    const errors = validateEvent(event);

    if (!isEmptyObject(errors)) {
      setFormErrors(errors);
    } else {
      console.log(event);
    }
  };

  return (
    <section>
      {renderErrors()}

      <h2>New Event</h2>
      <form className="eventForm" onSubmit={handleSubmit}>
        <div>
          <label htmlFor="event_type">
            <strong>Type:</strong>
            <input
              type="text"
              id="event_type"
              name="event_type"
              onChange={handleInputChange}
            />
          </label>
        </div>
        <div>
          <label htmlFor="event_date">
            <strong>Date:</strong>
            <input
              type="text"
              id="event_date"
              name="event_date"
              onChange={handleInputChange}
            />
          </label>
        </div>
        <div>
          <label htmlFor="title">
            <strong>Title:</strong>
            <textarea
              cols="30"
              rows="10"
              id="title"
              name="title"
              onChange={handleInputChange}
            />
          </label>
        </div>
        <div>
          <label htmlFor="speaker">
            <strong>Speakers:</strong>
            <input
              type="text"
              id="speaker"
              name="speaker"
              onChange={handleInputChange}
            />
          </label>
        </div>
        <div>
          <label htmlFor="host">
            <strong>Hosts:</strong>
            <input
              type="text"
              id="host"
              name="host"
              onChange={handleInputChange}
            />
          </label>
        </div>
        <div>
          <label htmlFor="published">
            <strong>Publish:</strong>
            <input
              type="checkbox"
              id="published"
              name="published"
              onChange={handleInputChange}
            />
          </label>
        </div>
        <div className="form-actions">
          <button type="submit">Save</button>
        </div>
      </form>
    </section>
  );
};

export default EventForm;

We start off by defining two variables in state: event and formErrors. The event variable is initialized as an object with some sensible defaults and formErrors is initialized as an empty object.

Next comes a handleInputChange function. We are going to make all of the fields in our form controlled inputs, which is to say React will responsible for maintaining and setting their state. The handleInputChange function will be called whenever the user changes the values of any of the fields and it will update the event object, so that it mirrors what has been entered into the form. Be aware of the square bracket notation that allows us to use a variable (name) as an object key.

After that we have a couple of helper functions: validateEvent and isEmptyObject. The first of these runs a bunch of checks on the event object and returns an object containing any errors, whereas the second returns true or false depending on whether the object it is passed has any properties or not.

We then have a renderErrors function which returns null if the formErrors object is empty, or otherwise some JSX representing a warning that the form could not be saved, as well as a list of errors.

And finally, we have updated our handleSubmit function to validate the user’s input (and check that each field has a value) and either display an error message if anything is missing, or to log the valid event to the console. We have also updated the JSX slightly and added an onChange property to all our form inputs.

Creating Some Helper Functions

By now our <EventForm> component is growing pretty large and it would be a good idea to split the more generic functions into a file of their own. Initially validateEvent and isEmptyObject are good candidates, as these could conceivably be used elsewhere in the app. This approach will also make the helper functions easier to test, as they are decoupled from React.

Let’s create a new file for these functions:

mkdir app/javascript/helpers
touch app/javascript/helpers/helpers.js

Now add the following code to helpers.js, making sure to remove validateEvent and isEmptyObject from the <EventForm> component:

export const isEmptyObject = obj => Object.keys(obj).length === 0;

export const validateEvent = (event) => {
  const errors = {};

  if (event.event_type === '') {
    errors.event_type = 'You must enter an event type';
  }

  if (event.event_date === '') {
    errors.event_date = 'You must enter a valid date';
  }

  if (event.title === '') {
    errors.title = 'You must enter a title';
  }

  if (event.speaker === '') {
    errors.speaker = 'You must enter at least one speaker';
  }

  if (event.host === '') {
    errors.host = 'You must enter at least one host';
  }

  return errors;
}

And import them into the <EventForm> component like so:

import { isEmptyObject, validateEvent } from '../helpers/helpers';

Now when you attempt to submit a form which is not properly filled out, you should see some nicely formatted errors.

Making the Date Field a Datepicker

The next thing to do is to wire up our date field as a datepicker. For this we’ll use Pikaday.

First, we need to install the library from npm:

npm i pikaday

yarn add pikaday

Then in the <EventForm> component, alter the React import, then import Pikaday like so:

import React, { useState, useRef, useEffect } from 'react';
import Pikaday from 'pikaday';
import 'pikaday/css/pikaday.css';

At the top of the component add:

const EventForm = () => {
  const [event, setEvent] = useState({ ... });
  const [formErrors, setFormErrors] = useState({});

  // new line
  const dateInput = useRef(null);
  ...
}

And change the date field like so:

<div>
  <label htmlFor="event_date">
    <strong>Date:</strong>
    <input
      type="text"
      id="event_date"
      name="event_date"
      ref={dateInput}
      autoComplete="off"
    />
  </label>
</div>

As you can see, we are using the useRef hook to create a reference to the date input field, so we can access it elsewhere in the code.

Next, we need to add a useEffect hook to initialize the datepicker when the component is mounted.

useEffect(() => {
  const p = new Pikaday({
    field: dateInput.current,
    onSelect: (date) => {
      const formattedDate = formatDate(date);
      dateInput.current.value = formattedDate;
      updateEvent('event_date', formattedDate);
    },
  });

  // Return a cleanup function.
  // React will call this prior to unmounting.
  return () => p.destroy();
}, []);

Thanks to our ref, the field property of the configuration object that we are passing to Pikaday’s constructor, points to DOM element we want to turn into a datepicker. The onSelect method determines what will happen when the user selects a date. In this case, the date is formatted into a YYYY-MM-DD string and the event object we are holding in state is updated.

We can write the formatDate function as a helper method in app/javascript/helpers/helpers.js. This receives a Date object and returns a YYYY-MM-DD string.

export const formatDate = (d) => {
  const YYYY = d.getFullYear();
  const MM = `0${d.getMonth() + 1}`.slice(-2);
  const DD = `0${d.getDate()}`.slice(-2);

  return `${YYYY}-${MM}-${DD}`;
};

Don’t forget to import it in the <EventForm> component:

import { formatDate, isEmptyObject, validateEvent } from '../helpers/helpers';

We can declare the updateEvent method in the <EventForm> component:

const updateEvent = (key, value) => {
  setEvent((prevEvent) => ({ ...prevEvent, [key]: value }));
};

Notice that we are calling the setEvent function slightly differently in that we are passing it a function as an argument. This callback function receives the previous value of event, which we spread into a new object, updating the key/value pair that has changed. This object is returned as the new value of event.

We need to do it like this — and not setEvent({ ...event, [key]: value }) — as otherwise, inside the onSelect method, event will point to its initial value (i.e. an empty object). This is because when it is declared, onSelect forms a closure over event and captures an incorrect value. You can read more about this here: Be Aware of Stale Closures when Using React Hooks.

Alternatively, we could add event to the useEffect dependency array, which would solve the problem. This would however, mean that we are creating a new datepicker every time the user types a character into our form, which is not ideal. There’s more on this in this GitHub issue.

Finally, let’s update our handleInputChange function to use this new method:

const handleInputChange = (e) => {
  const { target } = e;
  const { name } = target;
  const value = target.type === 'checkbox' ? target.checked : target.value;

  updateEvent(name, value);
};

And that’s it. We now have a datepicker.

Warning in Webpack Console

If you are following along using Shakapacker you will see a warning in the console at this point. If you are using esbuild, skip to the next section.

The warning is:

WARNING in ./node_modules/pikaday/pikaday.js 15:23-40
Module not found: Error: Can't resolve 'moment' ...

This is caused by the fact that Pikaday has made moment an optional dependency — if it is available, Pikaday requires it, otherwise it doesn’t. Unfortunately, this causes webpack to throw the above error. There is quite a lengthy issue looking at why that is here: https://github.com/webpack/webpack/issues/196.

The Pikaday maintainer doesn’t regard this as problematic and his advice is to ignore the warning. If we are going to do that, then it would be better to turn off webpack’s full-screen error overlay in config/webpacker.yml:

client:
  # Should we show a full-screen overlay in the browser
  # when there are compiler errors or warnings?
  overlay: false

Alternatively, you can get rid of the error by commenting out the Moment requires in node_modules/pikaday/pikaday.js:

(function (root, factory)
{
    'use strict';

    var moment;
    if (typeof exports === 'object') {
        // CommonJS module
        // Load moment.js as an optional dependency
        // try { moment = require('moment'); } catch (e) {}
        module.exports = factory(moment);
    } else if (typeof define === 'function' && define.amd) {
        // AMD. Register as an anonymous module.
        define(function (req)
        {
            // Load moment.js as an optional dependency
            var id = 'moment';
            // try { moment = req(id); } catch (e) {}
            return factory(moment);
        });
    } else {
        root.Pikaday = factory(root.moment);
    }
}(this, function (moment)

Why Pikaday?

So why are we using Pikaday? In addition to the ugly warning Shakapacker users are seeing, there doesn’t seem to be much active development happening on the project. Its GitHub repo has a lot of open issues, with a couple of them calling for the project to be marked as unmaintained.

Nonetheless, Pikaday is downloaded over 1 million times each month. The current version is stable, used in production and is adequate for our use case. I also wanted to demonstrate how to include a third party library in a React application.

If you would rather use a component-based solution, I recommend React Date Picker. It’s simple to set up and seems to be under active development. Otherwise you could try any of the other solutions out there.

Ref.:

Saving an Event

Currently, if you save a valid event, the event is logged to the console and nothing else happens. To actually save it to the database, we’re going to pass a callback function to our <EventForm> component, that can be called in the context of its parent.

In the <Editor> component:

import { Routes, Route, useNavigate } from 'react-router-dom';
...

const Editor = () => {
  ...
  const navigate = useNavigate();

  useEffect(() => { ... }, []);

  const addEvent = async (newEvent) => {
    try {
      const response = await window.fetch('/api/events', {
        method: 'POST',
        body: JSON.stringify(newEvent),
        headers: {
          Accept: 'application/json',
          'Content-Type': 'application/json',
        },
      });
      if (!response.ok) throw Error(response.statusText);

      const savedEvent = await response.json();
      const newEvents = [...events, savedEvent];
      setEvents(newEvents);
      window.alert('Event Added!');
      navigate(`/events/${savedEvent.id}`);
    } catch (error) {
      console.error(error);
    }
  };

  return (
    <>
      ...
        <Routes>
          <Route path="new" element={<EventForm onSave={addEvent} />} />
          <Route path=":id" element={<Event events={events} />} />
        </Routes>
      ...
    </>
  );
};

As you can see, we have defined an addEvent method, which receives a newEvent object and then fires off a request to our API to create a new event using that data. If the request is successful, it will add the newly created event to the array of events that are being held in state and the UI will update accordingly. It will also use the navigate function, which is made available to us by the useNavigate hook, to change the URL to that of the newly created event.

In the case that the API request is not successful (e.g. network problems, server responds with an error code etc), the error is logged to the console.

Note also that we are passing the addEvent method into the EventForm component as a callback. Now, all we’ve got to do is call it in our <EventForm> component:

import PropTypes from 'prop-types';
...

const EventForm = ({ onSave }) => {
  ...

  const handleSubmit = (e) => {
    e.preventDefault();
    const errors = validateEvent(event);

    if (!isEmptyObject(errors)) {
      setFormErrors(errors);
    } else {
      onSave(event);
    }
  };

  return ( ... );
};

export default EventForm;

EventForm.propTypes = {
  onSave: PropTypes.func.isRequired,
};

Now, when you attempt to save an event to the database, you should get an alert pop up, informing you that the save was successful. If you’re following along, I’d encourage you to give this a try and satisfy yourself that everything is working before continuing.

Deleting Events

Now, if you’re anything like me, you will have created a bunch of silly events while following along with this tutorial. Let’s add a delete button so that we can nuke them. 💥

As with adding an event, we’ll want to declare a method to delete an event in our <Editor> component, then pass it to our <Event> component as a prop.

First the method:

const Editor = () => {
  ...

  const addEvent = async (newEvent) => { ... };

  const deleteEvent = async (eventId) => {
    const sure = window.confirm('Are you sure?');

    if (sure) {
      try {
        const response = await window.fetch(`/api/events/${eventId}`, {
          method: 'DELETE',
        });

        if (!response.ok) throw Error(response.statusText);

        window.alert('Event Deleted!');
        navigate('/events');
        setEvents(events.filter(event => event.id !== eventId));
      } catch (error) {
        console.error(error);
      }
    }
  };

  return ( ... );
};

In our deleteEvent method, we ask the user for confirmation that they really want to delete the event via a confirm dialogue. If the user is sure, we send a DELETE request to our API and once a successful response comes back, we inform the user that the event has been deleted, redirect the user to /events and remove the deleted event from state. As with the addEvent method, if the request is unsuccessful, the error is logged to the console.

Next, pass the deleteEvent callback to the <Event> component:

<Routes>
  <Route path="new" element={<EventForm onSave={addEvent} />} />
  <Route path=":id" element={<Event events={events} onDelete={deleteEvent} />} />
</Routes>

Now, in the <Event> component we can create a button to delete the event. We’re not using a <Link> component here (as we do for creating a new event), as this is not a hypertext link that can be followed or be crawled. To remain consistent with the styling though, I have styled it to look like a link in the CSS above.

const Event = ({ events, onDelete }) => {
  ...

  return (
    <div className="eventContainer">
      <h2>
        {event.event_date}
        {' - '}
        {event.event_type}
        <button
          className="delete"
          type="button"
          onClick={() => onDelete(event.id)}
        >
          Delete
        </button>
      </h2>
      ...
    </div>
  );
};

Event.propTypes = {
  events: PropTypes.arrayOf(
    PropTypes.shape({ ... })
  ).isRequired,
  onDelete: PropTypes.func.isRequired,
};

And now we can delete events.

Adding Flash Messages

Alerts are all well and good to tell the user that something happened, but they don’t look very pretty. Let’s add flash message functionality instead, using the React-Toastify library.

npm i react-toastify

Or with Yarn:

yarn add react-toastify

We’ll stick this functionality in its own helper file, notifications.js.

touch app/javascript/helpers/notifications.js

Then add:

import { toast, Flip } from 'react-toastify';
import 'react-toastify/dist/ReactToastify.css';

const defaults = {
  position: 'top-right',
  autoClose: 5000,
  hideProgressBar: true,
  closeOnClick: true,
  pauseOnHover: true,
  draggable: true,
  progress: undefined,
  transition: Flip,
};

export const success = (message, options = {}) => {
  toast.success(message, Object.assign(defaults, options));
};

export const info = (message, options = {}) => {
  toast.info(message, Object.assign(defaults, options));
};

export const warn = (message, options = {}) => {
  toast.warn(message, Object.assign(defaults, options));
};

export const error = (message, options = {}) => {
  toast.error(message, Object.assign(defaults, options));
};

Now we have a centralized place to set some sensible defaults and can reduce the boilerplate when calling the flash messages. React-Toastify is highly configurable, so if you’d like to see what else the library can do, head on over to their playground and experiment with the notifications until you find a style you like. Also be sure to check out the different transition effects, as these are configurable, too.

Next, include the library in the <App> component:

import { ToastContainer } from 'react-toastify';

...

const App = () => (
  <>
    <Routes>
      <Route path="events/*" element={<Editor />} />
    </Routes>
    <ToastContainer />
  </>
);

Here we are adding a ToastContainer to our app, for the flash messages (or toast messages, I guess I should say) to display.

Then alter the <Editor> component to replace our alerts:

import { success } from '../helpers/notifications';
...
const addEvent = async (newEvent) => {
  try {
    ...
-   window.alert('Event Added!);
+   success('Event Added!');
    ...
  } catch (error) {
    console.error(error);
  }
};

const deleteEvent = async (eventId) => {
  const sure = window.confirm('Are you sure?');

  if (sure) {
    try {
      ...
-   window.alert('Event Deleted!);
+   success('Event Deleted!');
    ...
    } catch (error) {
      console.error(error);
    }
  }
};

While we’re at it we can move the error handling into a helper method, too. In app/javascript/helpers/helpers.js:

import { error } from './notifications';
...
export const handleAjaxError = (err) => {
  error('Something went wrong');
  console.error(err);
};

And in the <Editor> component:

import { handleAjaxError } from '../helpers/helpers';

And replace the three occurrences of:

console.error(error);

With:

handleAjaxError(error);

The following lines can also be removed from the component, as they are no longer needed.

- const [isError, setIsError] = useState(false);
- setIsError(true);
- {isError && <p>Something went wrong. Check the console.</p>}

Now, when you create or delete an event, you should get a nicely styled flash message.

Note that if you are testing the error message functionality (e.g. by making a typo in an endpoint), you will see two flash messages telling you that something has gone wrong, where you might have expected one. The reason for this is the use of StrictMode, which renders components twice (in development only) so as to detect any problems with your code and warn you accordingly.

Updating an Event

The final piece of our CRUD functionality to add is the ability to update an event. To avoid duplication we’re going to reuse our <EventForm> component. If we pass it a list of events, it should grab the event ID from the URL, find the correct event from the list, then prepopulate the form fields with the correct values. We will also define an updateEvent function that will allow us to perform a different action when the user hits the Save button, depending on whether we are creating an event, or updating an existing one.

Let’s start by adding the Edit link to the <Event> component. It’s fine to make this a link, as it will change the URL:

import { useParams, Link } from 'react-router-dom';
...
<h2>
  {event.event_date}
  {' - '}
  {event.event_type}
  <Link to={`/events/${event.id}/edit`}>Edit</Link>
  <button
    className="delete"
    type="button"
    onClick={() => onDelete(event.id)}
  >
    Delete
  </button>
</h2>

In the <Editor> component, let’s add the updateEvent function :

const updateEvent = async (updatedEvent) => {
  try {
    const response = await window.fetch(
      `/api/events/${updatedEvent.id}`,
      {
        method: 'PUT',
        body: JSON.stringify(updatedEvent),
        headers: {
          Accept: 'application/json',
          'Content-Type': 'application/json',
        },
      }
    );

    if (!response.ok) throw Error(response.statusText);

    const newEvents = events;
    const idx = newEvents.findIndex((event) => event.id === updatedEvent.id);
    newEvents[idx] = updatedEvent;
    setEvents(newEvents);

    success('Event Updated!');
    navigate(`/events/${updatedEvent.id}`);
  } catch (error) {
    handleAjaxError(error);
  }
};

Hopefully, this is starting to feel a little familiar by now. Here we send a PUT request to the API with the updated event data. If we get a good response back (i.e. no error) then we create a newEvents variable, setting it to the current value of events. We then determine the index of the updated event inside the newEvents array and swap out the old event with the new one. Finally, we update the value of events in state, before showing the user a success message and programatically navigating to the new event.

We go through the trouble of creating an additional variable, as it is good practice not to mutate state directly.

Next, let’s declare a new route for editing events which will render our <EventForm> component:

<Routes>
  <Route
    path=":id"
    element={<Event events={events} onDelete={deleteEvent} />}
  />
  <Route
    path=":id/edit"
    element={<EventForm events={events} onSave={updateEvent} />}
  />
  <Route path="new" element={<EventForm onSave={addEvent} />} />
</Routes>

In the <EventForm> component, we need to make sure that the form fields are populated with the correct values, whenever we pass it a list of events.

Remove the following code from the component:

- const [event, setEvent] = useState({
-   event_type: '',
-   event_date: '',
-   title: '',
-   speaker: '',
-   host: '',
-   published: false,
- });

Then add the following:

import { useParams } from 'react-router-dom';
...

const EventForm = ({ events, onSave }) => {
  const { id } = useParams();

  const defaults = {
    event_type: '',
    event_date: '',
    title: '',
    speaker: '',
    host: '',
    published: false,
  }

  const currEvent = id? events.find((e) => e.id === Number(id)) : {};
  const initialEventState = { ...defaults, ...currEvent }
  const [event, setEvent] = useState(initialEventState);
  ...
}

Here we are using React Router’s useParams hook to retrieve the ID of the current event from the URL. This will either be an integer or, when the form is being used to create a new event, undefined. We then declare some sensible defaults for our event fields.

Next we check the value of the id variable. If it is undefined (we are creating a new event), we set currEvent to be an empty object. Otherwise, we filter our array of events to find the event we are updating, and set the value of currEvent to that.

We then merge defaults and currEvent into a new variable called initialEventState before declaring an event property in state and initializing it with the value of initialEventState.

This has the effect of event being initialized with some sensible defaults, or the value of the event we wish to edit.

This might seem like a bit of a convoluted way to do things and ideally I would have liked to pass the <EventForm> component only the event it needed to display. However, since upgrading to React router 6, I have been unable to find a way to reference the :id property in the parent component while keeping the routes as they are. If anyone has a suggestion as to how to do this, I’d be more than happy to hear it in the comments bellow.

Next, we need to make sure the form is initialized with the correct values from event and we need to update the component’s prop validation:

return (
  <div>
    <h2>New Event</h2>
    {renderErrors()}

    <form className="eventForm" onSubmit={handleSubmit}>
      <div>
        <label htmlFor="event_type">
          <strong>Type:</strong>
          <input
            type="text"
            id="event_type"
            name="event_type"
            onChange={handleInputChange}
            value={event.event_type}
          />
        </label>
      </div>
      <div>
        <label htmlFor="event_date">
          <strong>Date:</strong>
          <input
            type="text"
            id="event_date"
            name="event_date"
            ref={dateInput}
            autoComplete="off"
            value={event.event_date}
            onChange={handleInputChange}
          />
        </label>
      </div>
      <div>
        <label htmlFor="title">
          <strong>Title:</strong>
          <textarea
            cols="30"
            rows="10"
            id="title"
            name="title"
            onChange={handleInputChange}
            value={event.title}
          />
        </label>
      </div>
      <div>
        <label htmlFor="speaker">
          <strong>Speakers:</strong>
          <input
            type="text"
            id="speaker"
            name="speaker"
            onChange={handleInputChange}
            value={event.speaker}
          />
        </label>
      </div>
      <div>
        <label htmlFor="host">
          <strong>Hosts:</strong>
          <input
            type="text"
            id="host"
            name="host"
            onChange={handleInputChange}
            value={event.host}
          />
        </label>
      </div>
      <div>
        <label htmlFor="published">
          <strong>Publish:</strong>
          <input
            type="checkbox"
            id="published"
            name="published"
            onChange={handleInputChange}
            checked={event.published}
          />
        </label>
      </div>
      <div className="form-actions">
        <button type="submit">Save</button>
      </div>
    </form>
  </div>
);

...

EventForm.propTypes = {
  events: PropTypes.arrayOf(
    PropTypes.shape({
      id: PropTypes.number.isRequired,
      event_type: PropTypes.string.isRequired,
      event_date: PropTypes.string.isRequired,
      title: PropTypes.string.isRequired,
      speaker: PropTypes.string.isRequired,
      host: PropTypes.string.isRequired,
      published: PropTypes.bool.isRequired,
    })
  ),
  onSave: PropTypes.func.isRequired,
};

EventForm.defaultProps = {
  events: [],
};

We also need to pass the datepicker a toString function in its initial configuration, so that the date is formatted properly when the event date is added to the form:

useEffect(() => {
  const p = new Pikaday({
    field: dateInput.current,
+   toString: date => formatDate(date),
    onSelect: (date) => { ... },
  });

  // Return a cleanup function.
  // React will call this prior to unmounting.
  return () => p.destroy();
}, []);

And finally, we need to add another useEffect hook to ensure that the fields are cleared when a user is editing an event, then clicks New Event. This would have been accomplished by using the componentWillReceiveProps lifecycle method in a class-based component.

useEffect(() => {
  setEvent(initialEventState);
}, [events]);

Notice that we are specifying the prop that we want to watch for changes in the dependency array.

And that’s that. We now have all of our CRUD functionality. Open the app and satisfy yourself that everything is working before moving on.

Dealing With the ESLint Warning

If you installed ESLint earlier, you’ll notice a warning if at this point we lint our code.

So what is going on here?

Well, what the warning is telling us is that our second useEffect hook is using a value which could potentially change between renders. It’s complaining about initialEventState which is defined like so:

const defaults = { ... };
const currEvent = id? events.find((e) => e.id === Number(id)) : {};
const initialEventState = { ...defaults, ...currEvent }

The culprit is currEvent whose value is dependent on both id and events.

Now we could just switch this warning off, like so:

// eslint-disable-next-line react-hooks/exhaustive-deps
useEffect(() => {
  setEvent(initialEventState);
}, [events]);

But the React team explicitly advise against doing this.

If we follow the advice in the warning above and move initialState inside the dependency array like so:

useEffect(() => {
  setEvent(initialEventState);
}, [events, initialEventState]);

It just gives us a different warning:

The 'initialEventState' object makes the dependencies of useEffect Hook (at line 58) change on every render.
To fix this, wrap the initialization of 'initialEventState' in its own useMemo() Hook.

And, as a nasty side-effect, we can no-longer use our form, as the form reverts to its initial state after every key stroke.

To fix this up, we can employ a useCallback hook, like so:

Delete the following code from the component:

- const defaults = {
-   event_type: '',
-   event_date: '',
-   title: '',
-   speaker: '',
-   host: '',
-   published: false,
- }
- const currEvent = id? events.find((e) => e.id === Number(id)) : {};
- const initialEventState = { ...defaults, ...currEvent }

And replace it with the following:

const initialEventState = useCallback(
  () => {
    const defaults = {
      event_type: '',
      event_date: '',
      title: '',
      speaker: '',
      host: '',
      published: false,
    };

    const currEvent = id ?
      events.find((e) => e.id === Number(id)) :
      {};

    return { ...defaults, ...currEvent }
  },
  [events, id]
);

We also need to import useCallback at the top of the component:

import React, { useState, useRef, useEffect, useCallback } from 'react';

The useCallback hook returns a memoized version of a function, which won’t change on every render unless its own dependencies (events and id) also change. This is exactly what we want.

You can read more about this here: Is it safe to omit functions from the list of dependencies?

Adding Some Form Tweaks

Next, let’s add a Cancel button to the form (in case the user changes their mind whilst editing or creating an event). We’ll also change the title of the form to reflect which action they are performing. And while we’re at it, we’ll improve the validation for our date field — at the moment it just checks if the user has entered a value.

In the <EventForm> component:

import { useParams, Link } from 'react-router-dom';

const EventForm = ({ events, onSave }) => {
  ...
  const cancelURL = event.id ? `/events/${event.id}` : '/events';
  const title = event.id ? `${event.event_date} - ${event.event_type}` : 'New Event';

  return (
    <div>
      <h2>{title}</h2>
      {renderErrors()}
      <form className="eventForm" onSubmit={handleSubmit}>
        ...
        <div className="form-actions">
          <button type="submit">Save</button>
          <Link to={cancelURL}>Cancel</Link>
        </div>
      </form>
    </div>
  );
};

And the date validation in helpers.js:

const isValidDate = dateObj => !Number.isNaN(Date.parse(dateObj));

export const validateEvent = (event) => {
  ...

  if (!isValidDate(event.event_date)) {
    errors.event_date = 'You must enter a valid date';
  }

 ...

  return errors;
};

Finally, let’s make the header a link, that returns us to the main view. In the <Header> component:

import { Link } from 'react-router-dom';

const Header = () => (
  <header>
    <Link to='/events/'>
      <h1>Event Manager</h1>
    </Link>
  </header>
);

Adding Search

It’d be nice to add search functionality to the events list. Luckily this is not complicated as we are holding all the events in state.

Let’s start off by adding a search input in our <EventList> component:

const EventList = ({ events }) => {
  ...
  return (
    <section className="eventList">
      <h2>
        Events
        <Link to="/events/new">New Event</Link>
      </h2>

      <input
        className="search"
        placeholder="Search"
        type="text"
        ref={searchInput}
        onKeyUp={updateSearchTerm}
      />

      <ul>{renderEvents(events)}</ul>
    </section>
  );
};

Notice that we have added a ref to the input element, so that we can reference it elsewhere in our component. Now let’s create that ref and declare a searchTerm property in state.

import React, { useState, useRef } from 'react';
...

const EventList = ({ events }) => {
  const [searchTerm, setSearchTerm] = useState('');
  const searchInput = useRef(null);

  const updateSearchTerm = () => {
    setSearchTerm(searchInput.current.value);
  };
  ...
}

We’re also creating a updateSearchTerm method which will be called every time a key press is registered on the search field.

The list of events is rendered in the renderEvents method. Let’s apply a filter to our event list, so that only events matching the search criteria are displayed. Replace the entire renderEvents function with the following:

const renderEvents = (eventArray) =>
  eventArray
    .filter((el) => matchSearchTerm(el))
    .sort((a, b) => new Date(b.event_date) - new Date(a.event_date))
    .map((event) => (
      <li key={event.id}>
        <NavLink to={`/events/${event.id}`}>
          {event.event_date}
          {' - '}
          {event.event_type}
        </NavLink>
      </li>
    ));

And finally, we need the matchSearchTerm method:

const EventList = ({ events }) => {
  ...

  const matchSearchTerm = (obj) => {
    const { id, published, created_at, updated_at, ...rest } = obj;
    return Object.values(rest).some(
      (value) => value.toLowerCase().indexOf(searchTerm.toLowerCase()) > -1
    );
  };

  const renderEvents = (eventArray) => ... ;

  return ( ... );
};

Here, we are excluding some database fields that were returned by the original Ajax call, but which we are not interested in filtering by. Then, we are returning only those events which contain our search term in any of the relevant fields.

Et voilà! We have rudimentary search.

Adding a 404 Component

The last thing we will do is add a component to render when an event is not found. This might be useful if a user has bookmarked an event which has since been deleted.

First, create the new component:

touch app/javascript/components/EventNotFound.js

And add the content. Nothing exciting here:

import React from 'react';

const EventNotFound = () => <p>Event not found!</p>;

export default EventNotFound;

Now we need to update the <Event> component:

import EventNotFound from './EventNotFound';

const Event = ({ events, onDelete }) => {
  ...

  if (!event) return <EventNotFound />;

  return ( ... );
};

And finally, the <EventForm> component:

import EventNotFound from './EventNotFound';

const EventForm = ({ events, onSave }) => {
  ...

  if (id && !event.id) return <EventNotFound />;

  return ( ... );
};

Now if the user attempts to view or edit a non-existent event, they will be shown a 404 component.

Conclusion

And that’s everything. Well done if you’ve made it up to here. You should now have a fully functioning React/Rails CRUD app, as well as a reasonable overview of all of the moving parts involved in building something like this. I hope this tutorial has left you with a good jumping off point for creating a similar SPA of your own.

If you’re looking to continue honing your skills, there are a several further steps you can consider. You could, for example deploy the app to Heroku, add authentication, or port the backend to use Node.

Either way, I’d be glad to hear your comments in the discussion below, and don’t forget, the full source code for this tutorial is available on GitHub.

How to Prevent the .xsession-errors File From Growing Too Large

2022-02-06T00:00:00+00:00

Sometimes, without warning, your .xsession-errors file will grow to an enormous size. This can be inconvenient at best, or crash your operating system at worst.

In this post I explain how you can set up a cron job to automatically watch the file and get notified if it exceeds a certain size.

My operating system of choice is Linux Mint, but with a little tweaking this should work on most distros.

If you’d like to skip straight to the solution, click here.

What Is This File and Why Does It Get So Huge?

Many Linux distributions use something called the X Window System (also known as X11, or just X) to display graphical user interfaces. The .xsession-errors file is where X logs errors that occur within any of its client processes. These processes are regular GUI programs that you launch, such as a media player, image viewer, or file explorer.

The .xsession-errors file is located in your home directory. The dot at the beginning of the file name denotes that it is hidden by default, so you will have to make sure you can view hidden files (e.g. by pressing Ctrl + H in a file explorer) before accessing it.

The reason that it can grow so big, is that a large amount of applications can write to it and occasionally, one of these applications will go a little haywire and dump the same entry thousands of times over.

For example, this happened to me recently, when my .xsession-errors file grew to over 300GB and was packed full of entries like this:

[00007f600009b310] vdpau_chroma filter error: video mixer attributes failure: An invalid handle value was provided.
[00007f600009b310] vdpau_chroma filter error: video mixer rendering failure: An invalid handle value was provided.
[00007f600009b310] vdpau_chroma filter error: video mixer features failure: An invalid handle value was provided.

It turns out this was being caused by VLC.

Every time you reboot your system, the current .xsession-errors file is renamed to .xsession-errors.old and an empty .xsession-errors file will be created. However, if you don’t often reboot your computer, this isn’t very helpful and other measures are called for.

How to Automatically Watch the Size of the `.xsession-errors` File

There are various posts around the internet that recommend truncating this file whenever it gets too big, or disabling writes on it altogether. And while that’s fine, I would prefer to get notified instead (via a system notification), so that I can check which application is misbehaving and address the root of the problem.

Doing anything automatically in regular intervals on a Linux system sounds like a job for cron, so let’s look at that first.

Setting up a Cron Job

Cron is a command-line utility and you can use it to schedule jobs (commands or shell scripts) to run periodically at fixed times, dates, or intervals. You can access it by typing crontab -e into a terminal, which will then open the crontab file for the current user in an editor (you will be asked to select an editor if you are running this for the first time).

Enter the following line at the bottom of the file:

* * * * * XDG_RUNTIME_DIR=/run/user/$(id -u) notify-send 'Hey' 'Hello, World!'

Save the file and exit. You should now recieve a “Hello, World!” system notification every minute. Nice, huh?

Satisfy yourself that this command works, then let’s take a look at what’s going on.

The above line breaks down as follows:

<frequency> <command-to-execute>

The frequency part is * * * * *. As we can read in the screenshot above, to define the time you can provide concrete values for minute, hour, day of the month, month, and day of the week. Or we can use * in these fields for ‘any’. An asterisk in every field means run the given command every minute.

If you’d like to run a command at a different interval, you can use Cron Helper to figure out the correct syntax.

The command part is notify-send 'Hey' 'Hello, World!'. This uses the current desktop environment’s notification system to create a notification and is called thus: notify-send 'Title' 'Message'. You can also run this command straight from the command line to see it in action.

To use this command with cron however, we need to specify a XDG_RUNTIME_DIR environment variable. This tells notify-send where to find a user-specific directory in which to store small temporary files.

The value we set the XDG_RUNTIME_DIR variable to is /run/user/$(id -u), where $(id -u) returns the current user id as a number. This usually starts at 1000, meaning the whole path will be something like /run/user/1000.

Note that the $( ... ) syntax is used for command substitution, which allows the output of a command to replace the command itself.

Checking the Size of `.xsession-errors`

The next thing to look at is how to check the size of the .xsession-errors file.

We can do that with the disk usage command:

du -k ~/.xsession-errors

The -k switch ensures that the output is in kilobytes. This will output something like:

240  /home/jim/.xsession-errors

As we are only interested in the first number (not the path), we can grab this using the awk command:

du -k ~/.xsession-errors | awk '{ print $1 }'

This outputs the size of .xsession-errors in kilobytes:

Finally, we can use the test command to check whether the file size is greater than a certain threshold. For example 1MB:

test $(du -k ~/.xsession-errors | awk '{ print $1 }') -gt 1024

Notice that once again we are using the command substitution syntax ($( ... )), to allow the output of both commands to replace the commands themselves.

This gives us something like this:

test 240 -gt 1024

The test command provides no output, but returns an exit status of 0 for “true” (test successful) and 1 for “false” (test failed). We can use this to our advantage in the next section.

You can also check this using the $? variable which represents the exit status of the previous command.

test 1 -gt 2; echo $?
1

test 1 -gt 0; echo $?
0

Only Trigger Notification if File Size Exceeded

The final piece of the puzzle is to ensure that our notification is only triggered if the size of .xsession-errors is above a certain threshold. We can do this with the && operator, like so:

<frequency> <command-1> && <command-2>

The right side of && will only be evaluated if the exit status of the left side is zero (i.e. true).

This gives us the following as our final solution:

*/15 * * * * [ $(du -k .xsession-errors | awk '{ print $1 }') -gt 1024 ] && XDG_RUNTIME_DIR=/run/user/$(id -u) notify-send 'Hey' '.xsession errors is getting too big!'

Note that I am using square brackets as an alias for the test command and that I have set the cron job to run every 15 minutes (which is plenty). It is also not necessary to specify the complete path of the .xsession-errors file, as the cron job will be run in the user’s home directory.

If the file size exceeds whichever threshold you have specified, a notification will be triggered every 15 minutes until the problem is addressed.

For added convenience, I have also set up an alias in my shell, so that when I type x, the .xsession-errors file is opened in my editor.

Conclusion

In this post I have shown how to set up a cron job to automatically monitor the size of the .xsession-errors file and to trigger a notification if the file size becomes too big. I hope people find this useful. If you have any questions or comments, let me know below.

James Hibbard

Erase Sensitive Files from Git History with git-filter-repo

Why git rm or Reverting a Commit Isn’t Enough

Meet git filter-repo: A Cleaner Solution

Installation

Step-by-Step: Removing a File from Git History

Option 1: The Canonical Way (Fresh Clone)

1. Create a fresh mirror of your repository

2. Run the filter to delete the file

3. Push the cleaned history

Option 2: The Pragmatic Shortcut (Local Repo)

What to Do Next: Rotate Keys and Notify Your Team

Avoiding This in the Future

Conclusion: Clean Repos, Clean Conscience

Deploying a Rails App with Apache & MySQL on Ubuntu Server

Server Setup Prerequisites

But Why Not Kamal or Docker?

Installing the LAMP Stack

Enabling Password Access for MySQL

Securing MySQL

Installing phpMyAdmin

Install Git, rbenv, and Ruby

Install JavaScript Runtime (Optional)

Install and Configure Passenger

A Quick Note on Nano

Enable the Module and Restart Apache

Verify the Installation

Set Up an Apache Virtual Host

Configure the Site

Add Hostname Resolution (On the Host Machine)

Create the Production Databases

Deploy the App Using Capistrano

What is Capistrano?

What We’ll Be Using

1. Fork the Repository

2. Check GitHub Access (Optional)

3. Add Capistrano to the App

4. Initialize Capistrano

5. Configure Your Deploy Settings

6. Verify the Setup

Prepare the Server for Deployment

Add database.yml

Define Your Rails Master Key and Encrypted Credentials

Upload Encrypted Credentials to the Server

Provide Rails with the Master Key

Optional: Upload master.key Instead

Run the First Deploy

Troubleshooting

Finishing Touches

Disabling SSL Enforcement (For Local Testing Only)

Logging “Fix” (Rails 7.1+)

Seeding the Database

Running the Rails Console

Enabling SSL with Let’s Encrypt & Certbot

Conclusion

Authentication with Devise and CanCanCan in Rails 8

The Scenario

Installing the Dependencies

Generating the Project Files

Changes to Rails Scaffolding in Rails 7+

Editing the Boilerplate

Adding Some Styles

Authentication with Devise

Why Not Use Built-in Rails Authentication?

Installing And Configuring Devise

A Bit More About Devise

Adding an Admin Namespace

Customizing Devise

Creating New Users

Enabling the Trackable Module

Authorization With CanCanCan

Better Error Handling For CanCanCan

The Finishing Touches

How to Install Ubuntu Server on VirtualBox

What Is VirtualBox?

Install VirtualBox

Download Ubuntu Server

Create a New Virtual Machine

Install Ubuntu Server in the Virtual Machine

Navigating the Install Screen

Why `git rm` or Reverting a Commit Isn’t Enough

Meet `git filter-repo`: A Cleaner Solution

Add `database.yml`

Optional: Upload `master.key` Instead

So What is `.xsession-errors`?