Tame Your Files Using Pipe Command

Overview

Can use "Unix pipes" when checking access logs or making new files from other files.
Why have to use pipes?

• No additional software required → good for making common manual
• Using other tools can make mistake → miss in copy&paste when using Excel

In Access Logs

You may want to know how many users access to your site.

1. Sample access logs are as below.

2. Then can use pipes as below.

grep -v "HOME" : Exclude access from HOME(ex. redirect access from application itself)
cut -d' ' -f1,7,11,12 : Considered space as a field delimiter then print fields in 1, 7, 11, 12
sort | uniq -c : Sorting then count the unique rows. Have to "sort" before "uniq" because "uniq" only check unique in consecutive values.
sort -nr : sort by number in reverse order

In CSV Files

Business team give you an csv files that including delivery information then said "We want to give free delivery to users who live in tokyo and one's delivery fee is 300yen". So you have to update DB by CSV files.

1. Here is delivery information.

2. We need user's id for updating delivery fee.

sort -n : Sort by number(user id)
grep "tokyo" : Who live in tokyo
awk -F',' '{if ($5 == 300) print $1}' : Delimiter is ',' for it is CSV. Then find values which delivery_fee($5) is 300yen. sed 's/"//g' : Delete ' " ' just in case for "id" is String value.

3. Then can make update query from id list.

Reference

https://d2.naver.com/helloworld/3585246
https://thenewstack.io/tutorial-tame-your-access-log-with-unix-pipes/
https://askubuntu.com/questions/685000/tail-a-log-file-but-show-only-specific-lines/685031
https://stackoverflow.com/questions/8692597/read-snort-log-via-pipe-or-stream
https://codinginfinite.com/data-visualizing-csv-format-chart-using-python-matplotlib/
https://bconnelly.net/posts/working_with_csvs_on_the_command_line/