Pipes (|), Pipefail & Process Pipelines

In Plain English

The Pipe (`|`) operator embodies the core Unix philosophy: write small programs that do one thing well, and connect them together to solve complex problems. A pipe connects the standard output (`stdout`) of the left command directly to the standard input (`stdin`) of the right command in memory.

The Mental Model

"A pipe is like a water conduit between two machines: the water produced by machine 1 flows directly into the intake valve of machine 2 with zero buckets (temporary disk files) needed in between."

Deep Dive: How It Works

Kernel Buffering: Pipes operate via a 64KB kernel buffer in RAM without writing intermediate data to disk.

Parallel Execution: Commands in a pipeline run concurrently in separate subshells; the second command starts processing data the instant the first begins streaming.

The `tee` Command: `cmd1 | tee file.log | cmd2` splits the stream, writing to disk while simultaneously piping forward.

Syntax Blueprint

cat /var/log/nginx/access.log | grep " 500 " | cut -d" " -f1 | sort | uniq -c | sort -nr | head -n 5

# Split stream with tee:
./build.sh | tee build.log | grep -E "(ERROR|WARN)"

Pipes connect command output streams; tee duplicates streams to both file and screen.

Core Rules to Remember

Zero Disk Overhead: Pipelines stream multi-gigabyte datasets directly through RAM buffers.

Concurrent Streaming: Downstream commands start filtering immediately without waiting for upstream to finish.

The tee Utility: Splits output so you can watch live logs on screen while archiving to disk.

Common Beginner Traps & How to Fix Them

Mistake:By default, `$?` in a pipeline only reports the exit code of the LAST command in the chain.

Why it happens: If `broken_command | grep "foo"` runs, `$?` reports grep's status (0), masking the upstream crash.

How to fix: Enable `set -o pipefail` at the start of your script and inspect `${PIPESTATUS[@]}`.

Pro Tip: Inspect `${PIPESTATUS[@]}` immediately after a pipeline to see an array of every individual command's exit code.

Live Interactive Example

Hit Run Code to see it live

Multi-Stage Analytics Pipeline with tee

Git 2.45 / CLI

Output Console

Click "Run Code" to view the rendered output.

How it works: The pipeline stripped comments, uppercased characters, sorted, logged with tee, and counted.

Your Turn: Micro Challenge

No pressure! Edit the starter code below and test your solution with instant feedback.

Micro Exercise

Build a Pipeline with grep and wc

Echo three lines ("apple", "banana", "apricot") and pipe into `grep "^a" | wc -l`.

Sandbox Output

Click "Run & Check" to test your solution.

Finished reading and practicing?

Mark this lesson as completed to update your course progress.