Lesson 4 of 725 minutesHands-on

Search and filter with pipes

Each Linux command does one small job. The clever part is that you can join them, feeding the output of one straight into the next. That’s how people answer questions like “which address is hammering my website?” in a single line, on files far too big to open. You’ll do exactly that in this lesson, on a short practice log.

In this lesson

Search inside files with grep
Join commands together with a pipe
Count and rank the lines in a log, and save the result

You’ll need: the terminal and your practice folder. Everything else is copied in below.

The scenario

You run a small website and something feels off. The server keeps a log: one line for every request, showing who asked for which page and how it went. A real log can have millions of lines, so you can’t read it. You have to ask it questions.

Try it yourself · part 1

Create the practice log

Go to your practice folder. Then copy this whole block, paste it into the terminal and press Enter. It writes twelve lines to a file called access.log:

cd ~/practice

cat > access.log <<'EOF'
203.0.113.7 GET /index.html 200
198.51.100.23 GET /index.html 200
203.0.113.7 GET /about.html 200
192.0.2.44 GET /wp-login.php 404
203.0.113.7 GET /contact.html 200
192.0.2.44 POST /wp-login.php 404
198.51.100.23 GET /contact.html 200
192.0.2.44 GET /admin.php 404
192.0.2.44 GET /.env 404
203.0.113.7 GET /index.html 200
192.0.2.44 POST /xmlrpc.php 404
198.51.100.90 GET /index.html 200
EOF

Count the lines. wc means “word count”, and -l asks for lines:
wc -l access.log
```
12 access.log
```

Each line has an address, a request type, a page and a status code, where 200 means “OK” and 404 means “not found”. The addresses come from ranges reserved for examples, so none of them is a real visitor.

grep: find the lines you want

grep prints only the lines that contain your text. It’s probably the command you’ll use most.

Try it yourself · part 2

Ask the log some questions

Which requests failed?

grep 404 access.log

192.0.2.44 GET /wp-login.php 404
192.0.2.44 POST /wp-login.php 404
192.0.2.44 GET /admin.php 404
192.0.2.44 GET /.env 404
192.0.2.44 POST /xmlrpc.php 404

Just count them, with -c:
grep -c 404 access.log
Ignore capitals with -i. This still finds it:
grep -i WP-LOGIN access.log
Flip the search with -v, which shows every line that doesn’t match:
grep -v 404 access.log

The pipe: joining commands

The | character is called a pipe. It takes what one command would have printed and hands it to the next command as its input. You can chain as many as you like, so you build the answer one step at a time, and check each step as you go.

Try it yourself · part 3

Find out who’s knocking

The question: which address made the most requests? Add one piece at a time, using ↑ to bring the previous line back.

Pull out only the first column. cut splits each line at a separator (-d' ' means a space) and keeps one field (-f1):
cut -d' ' -f1 access.log
Sort them, so the same addresses end up next to each other:
cut -d' ' -f1 access.log | sort
Count the repeats with uniq -c:
cut -d' ' -f1 access.log | sort | uniq -c
Sort again, by that count: -n means by number and -r means biggest first:
cut -d' ' -f1 access.log | sort | uniq -c | sort -rn
```
      5 192.0.2.44
      4 203.0.113.7
      2 198.51.100.23
      1 198.51.100.90
```

Now look back at what 192.0.2.44 asked for: /wp-login.php, /admin.php, /.env and /xmlrpc.php. Those aren’t pages anyone reads. They’re the login and settings files automated scanners try on every website, hoping to find one left open. Every one of them failed with a 404. On a real server, you’d read that as noise to keep an eye on, not a break-in.

Why sort before uniq? uniq only merges repeats that sit next to each other. Without the sort, the same address gets counted in several separate groups and your totals are wrong. It’s the classic pipe mistake, and it doesn’t give an error: just a wrong answer.

Try it yourself · part 4

Save the answer, then use it on your own system

Pipes work with the > from the last lesson, so you can keep your results:
grep 404 access.log > errors.txt
wc -l errors.txt
Which pages get the most requests? Same recipe, but a different column:
cut -d' ' -f3 access.log | sort | uniq -c | sort -rn | head -n 3
Pipes work on any output, not only on files. How many settings files in /etc have “conf” in their name?
ls /etc | grep -c conf
And your shell remembers what you’ve typed. Which of your commands used grep?
history | grep grep

Going further: who owns those addresses?

A count tells you which address to look at, but not whose it is. Paste a log into our free Bulk IP Lookup and it pulls out every address, counts how often each appears, and shows who owns it and where it’s registered. If you try it with this practice log, it will mark all four addresses as Documentation, because they’re reserved for examples. On a real log you’d see real networks.

Command	Job in a pipe
`grep`	Keep the lines that match
`cut`	Keep certain columns
`sort`	Put lines in order
`uniq -c`	Merge adjacent repeats and count them
`head`, `tail`	Keep the first or last few lines
`wc -l`	Count the lines

Key takeaways

grep keeps the lines containing your text; -i, -c and -v change how.
A pipe, |, feeds one command’s output into the next.
Build a pipe one step at a time and check each step.
sort | uniq -c | sort -rn ranks anything by how often it appears.
Requests for /wp-login.php and /.env are usually automated scanners trying their luck.

Check yourself

Three questions. Pick one answer each and you’ll see why.

1. What does the | in ls /etc | grep conf do?

A pipe connects the two commands: ls produces the list, and grep keeps only the lines containing conf.

2. Why does uniq -c usually need sort before it?

3. Your log shows one address asking for /.env and /wp-login.php, all with 404. What’s the most likely explanation?

Next: why some of those commands get “Permission denied”, and what to do about it.

This lesson is part of the free course Linux and the Command Line, licensed under CC BY-NC-SA 4.0: you may share and adapt it for non-commercial use, with credit to ExamineIP and a link to the original.