File tools

Copy many small files efficiently

Run command in directory containing files

tar cf - . | pv | (cd /destination; tar xf -)

Count files in a directory recursively

find <dir> -type f | wc -l

Size of a directory recursively

du -sh

Find files in a directory recursively

grep -Riw "string-to-search-here" /path-to-search-here

Export changes to /etc/exports

exportfs -ar

No need to restart nfs-server

Monitoring tools

IOSTAT CPU and disk activity

iostat -c -d -x -t -m /dev/md0 /dev/sda /dev/sdb

The -c gives the CPU, -d gives devices, -x extended report. List as many or as few /dev as required.

Watching (continuously updated monitoring)

watch -n 2 'command'

The -n specifies the update interval in secs. The command should be enclosed in 'quotes' if it takes an argument.

Watching multiple values

watch -n 2 'du -s /mnt/raid/rmcache && find /mnt/piraid/ -type f | wc -l && iostat -h'

Separate the commands with &&

Networking

Mount a share as another user

This cannot be done by directly using the mount command. The correct method is to add the mount to /etc/fstab then mount from the cmd line.

in fstab:

192.168.1.9:/mnt/raid/rmcache /var/www/html/rmcache nfs rw,user,noauto 0 0

on the command line:

sudo -u www-data mount /var/www/html/rmcache

THIS WILL NOT WORK:

sudo -u www-data mount 192.168.1.9:/mnt/raid/rmcache /var/www/html/rmcache

It will produce the error "only root can do that"

Remount a share re-writable

mount -o remount,rw /partition/identifier /mount/point

Scan local network for devices IPv4

arp-scan --interface=eth0 --localnet

Scan local network for devices IPv6

This scans the local network for IPv6 addresses and outputs the addresses (only) to a file. Replace eth0 with the network adapter you're connecting through.

ping6 -I eth0 -c 2 ff02::1 | grep DUP | awk '{print substr($4, 1, length($4)-1)}' > ipv6_hosts.txt

With the list of addresses, a port scan on each address can be carried out by running:

nmap -6 eth0 -iL ipv6_hosts.txt

This takes each address from the previous list and processes them one at a time for open ports.

Mount CIFS (Windows Share)

apt-get install cifs-utils

mount -t cifs -o username=<user>,password=<password> //WIN_SHARE_IP/<share> /mnt/win_share

SQL tools

Select a database

USE <dbname>;

Search a table for a string by column

SELECT * FROM <tablename> WHERE <columname> LIKE '%somestring%';

Note that the % are essential and that the search string is case sensitive

Show a table schema

SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME='<tablename>';

Repair a database with .MYI/MYD suffix and mysqld offline

Steps:

1. Stop mysql service

systemctl stop mysql

2. CD to directory containing the database (table files)

cd /var/lib/mysql/drupal

3. Issue the repair command

nohup myisamchk -fpqqr --tmpdir=/mnt/piraid/mysqltmp searchindex &

4. Restart the mysql server

systemctl start mysql

This can take a long time (24 hours). The site will be offline while it happens. The cause of the table crash was dropping caches to free RAM. The DB server apparently ran out of free inodes while they were being cleared. The process can use 2x 3x the original table and index files - i.e. GB of space for a large table. The temp directory needs to have plenty of space an needs to be reliable if a network drive.

Repair a database table while mysqld is running

This method is now my preferred method. It leaves the database online and fixes the index without duplicating the entire table

Enter the mysql interpreter from root:

mysql

Issue SQL command:

USE <database name>;

Check a table for errors (optional - may take 2 hours):

CHECK TABLE 'tablename' FAST QUICK;

Repair the table (24 hours):

REPAIR TABLE 'tablename' QUICK;

Export parts of a database table to file

The syntax for exporting from joined tables (i.e. records from one table where other columns for those records are present in another table) is this:

SELECT column_name, another_column, etc_column FROM table1_name, joined_table2 WHERE table1_column =  table2_column INTO OUTFILE '/path/to/outfile.txt';

This is the command to export the current page revisions from mediawiki sites. The output directory seems to need to be owned my user mysql to be writable:

SELECT page_title, rev_timestamp, rev_sha1, page_id FROM page, revision WHERE page.page_id = revision.rev_page INTO OUTFILE '/var/lib/mysql/drupal/dump.txt';

Bash tools

Running a process in the background

nohup <command> &

Output is directed to nohup.out

Process can be retrieved from the background by

fg

TMux

Tmux is a virtual terminal manager. It creates sessions, each of which can have multiple arrangeable windows. The individual sessions can survive disconnects and can be resumed on other devices. It is considered to be superior to screen. This is a list of the commands [1]:

tmux shortcuts & cheatsheet

start new:

tmux

start new with session name:

tmux new -s myname

attach:

tmux a  #  (or at, or attach)

attach to named:

tmux a -t myname

list sessions:

tmux ls

kill session:

tmux kill-session -t myname

Kill all the tmux sessions:

tmux ls | grep : | cut -d. -f1 | awk '{print substr($1, 0, length($1)-1)}' | xargs kill

In tmux, hit the prefix ctrl+b (my modified prefix is ctrl+a) and then:

List all shortcuts

to see all the shortcuts keys in tmux simply use the bind-key ? in my case that would be CTRL-B ?

Sessions

:new<CR>  new session
s  list sessions
$  name session

Windows (tabs)

c  create window
w  list windows
n  next window
p  previous window
f  find window
,  name window
&  kill window

Panes (splits)

%  vertical split
"  horizontal split

o  swap panes
q  show pane numbers
x  kill pane
+  break pane into window (e.g. to select text by mouse to copy)
-  restore pane from window
⍽  space - toggle between layouts
<prefix> q (Show pane numbers, when the numbers show up type the key to goto that pane)
<prefix> { (Move the current pane left)
<prefix> } (Move the current pane right)
<prefix> z toggle pane zoom

User and group tools

List users

cat /etc/passwd

List groups

cat /etc/group

List user's group membership

id <username>

Change group id

groupmod -g 600 group01

This modifies the GID of the group group01 and changes it from the current value to 600.

Change user id

usermod -u 900 -g 600 user01

This modifies the user id of user01 and changes it from the current value to 900. It also changes the group membership to agree with the previous example.

Caveats (important)

1. If there are multiple users in the group “group01”, after changing the GID of the group you will have to modify the other users as well along with the user01 as shown above.

2. Once you have changed the UID and GID, you will have to change the permissions of the files owned by the user/group as well. But the chown command also resets the SETUID and SETGID of the files, so you will need to manually change the permissions of these files later on. To find such files:

find / -uid 900 -perm /6000 -ls
find / -gid 900 -perm /6000 -ls

3. To find the files owned by user01 and group01 and to change their permissions:

find / -uid 800 -exec chown -v -h 900 '{}' \;
find / -gid 700 -exec chgrp -v 600 '{}' \;

The -h option is used to change the permissions of symbolic links as well.

Add a group (with a specific group id)

groupadd -g 112 mysql

NGINX

List nginx status

nginx -t

Apache2

Reload apache2 configuration without restarting the server

service apache2 reload