Python Pipes (|), Pipefail & Process Pipelines
The Pipe (`|`) operator embodies the core Unix philosophy: write small programs that do one thing well, and connect them together to solve complex problems. A pipe connects the standard output (`stdout`) of the left command directly to the standard input (`stdin`) of the right command in memory.
"A pipe is like a water conduit between two machines: the water produced by machine 1 flows directly into the intake valve of machine 2 with zero buckets (temporary disk files) needed in between."
Deep Dive: How It Works
Kernel Buffering: Pipes operate via a 64KB kernel buffer in RAM without writing intermediate data to disk.
Parallel Execution: Commands in a pipeline run concurrently in separate subshells; the second command starts processing data the instant the first begins streaming.
The `tee` Command: `cmd1 | tee file.log | cmd2` splits the stream, writing to disk while simultaneously piping forward.
Syntax Blueprint
cat /var/log/nginx/access.log | grep " 500 " | cut -d" " -f1 | sort | uniq -c | sort -nr | head -n 5 # Split stream with tee: ./build.sh | tee build.log | grep -E "(ERROR|WARN)"
Pipes connect command output streams; tee duplicates streams to both file and screen.
Core Rules to Remember
Common Beginner Traps & How to Fix Them
By default, `$?` in a pipeline only reports the exit code of the LAST command in the chain.Why it happens: If `broken_command | grep "foo"` runs, `$?` reports grep's status (0), masking the upstream crash.
How to fix: Enable `set -o pipefail` at the start of your script and inspect `${PIPESTATUS[@]}`.
Live Interactive Example
Hit Run Code to see it liveYour Turn: Micro Challenge
No pressure! Edit the starter code below and test your solution with instant feedback.
Build a Pipeline with grep and wc
- Echo three lines ("apple", "banana", "apricot") and pipe into `grep "^a" | wc -l`.
Finished reading and practicing?
Mark this lesson as completed to update your course progress.