LearnBasic Automation: Batch Processing & Lockfiles

64 / 69 0%

Lesson 64 of 69

5 mins read

Basic Automation: Batch Processing & Lockfiles

In Plain English

Automating background tasks and batch pipelines requires defensive engineering: preventing overlapping concurrent job runs with lockfiles (`mkdir` or `flock`), logging timestamped events, and gracefully handling missing files.

The Mental Model

"A lockfile is like the "Occupied" sign on an airplane restroom door: it prevents a second automation job from walking in while the first job is still working."

Deep Dive: How It Works

Atomic Lock with `mkdir`: `if ! mkdir /tmp/job.lock 2>/dev/null; then echo "Job already running"; exit 1; fi`. `mkdir` is an atomic OS system call.

flock Utility: `( flock -n 200 || exit 1; ... ) 200>/var/lock/myjob.lock` provides kernel-level file descriptor locking.

Batch File Processing: Iterating over file globs, transforming assets, and appending results atomically.

Syntax Blueprint

LOCKDIR="/tmp/myjob.lock"
if ! mkdir "$LOCKDIR" 2>/dev/null; then
  echo "Job already running."
  exit 1
fi
trap "rm -rf '$LOCKDIR'" EXIT

# Run batch job here...

Use atomic mkdir locks with an EXIT trap to ensure robust single-instance automation.

Core Rules to Remember

Atomic Lock Acquisition: mkdir succeeds or fails atomically at the filesystem kernel level, preventing race conditions.

Trap Guarantees Unlock: `trap "rm -rf $LOCK"` ensures locks are released even if the script crashes or is killed.

Flock File Descriptor: flock automatically releases the lock when the process file descriptor closes upon exit.

Common Beginner Traps & How to Fix Them

Mistake:Using `touch lockfile` and `if [ -f lockfile ]` to implement concurrency locking.

Why it happens: There is a time-of-check to time-of-use (TOCTOU) race condition between checking and creating.

How to fix: Use `mkdir` or `flock` which combine testing and creation into one atomic system call.

Pro Tip: Add PID to lock folders (`echo $$ > /tmp/job.lock/pid`) so you can inspect which process owns a stale lock.

Live Interactive Example

Hit Run Code to see it live

Atomic Lockfile Pattern for Safe Batch Automation

Git 2.45 / CLI

Output Console

Click "Run Code" to view the rendered output.

How it works: The script safely acquired the atomic lock, ran batch tasks, and released the lock via trap.

Your Turn: Micro Challenge

No pressure! Edit the starter code below and test your solution with instant feedback.

Micro Exercise

Simulate Atomic Lock Acquisition

Create a lock with `mkdir /tmp/test_lock.lock`.
Echo "Lock acquired".
Clean up with `rmdir /tmp/test_lock.lock` and echo "Lock released".

Sandbox Output

Click "Run & Check" to test your solution.

Finished reading and practicing?

Mark this lesson as completed to update your course progress.