Useful Linux commands

Tags: Linux
Last update: Mar 2018

1. Compare binary files within the console and highlight those areas which differ. Vim and Colordiff have to be installed before.

diff -y <(xxd file.one) <(xxd file.two) | colordiff

2. Using a harddisk device for storing encrypted data:

sudo cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha512 -y /dev/sdX
sudo cryptsetup luksOpen /dev/sdX usb-backup
sudo mkfs.btrfs -L backupharddisk /dev/mapper/backup
sudo mount /dev/mapper/backup /usb-backup

3. Mirroring with rsync from /source to /destination. Compression is great when backing up over the Internet.

rsync --compress --compress-level=9 --human-readable -av --progress --delete --log-file=/path/to/log.file /source /destination

4. Wipe a file with zeros:

shred -fvzun 0 file

5. Wipe a directory recursively with zeros:

srm -rvs directory

6. Backup of harddisks into compressed image file. Due to its multithreading capability pigz is much faster than gzip during compression.

sudo dd if=/dev/sdX status=progress bs=1M | pigz > harddisk.img.gz

7. Restore this prior zipped image file and write it directly to disk:

sudo gunzip -c image.gz | sudo dd of=/dev/sdX status=progress bs=1M

8. Recover a Linux Soft-RAID md/5:

sudo mdadm --assemble --scan
sudo cryptsetup luksOpen /dev/md/5 temp
sudo mount /dev/mapper/temp /your_path

9a. Mount a Linux Soft-RAID manually:

sudo mdadm --examine /dev/sdX
sudo mdadm -A -R /dev/md9 /dev/sdX
sudo mount /dev/md9 /openedRaidPartition

9b. And close it again:

sudo umount /openedRaidPartition
sudo mdadm -S /dev/md9

10. List all loadable kernel modules:

find /lib/modules/$(uname -r) -type f -name \*.ko

11. Template for configuring a fresh manjaro xfce installation:

#update system
sudo pacman -Syyu
 
#install zshell
sh -c "$(curl -fsSL https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"
sudo pacman -S powerline-fonts
 
#install virtualbox
sudo pacman -S virtualbox
 
#copy fonts which you need into /usr/share/fonts and run
sudo fc-cache
 
#fixing the Manjaro Linux XFCE text shadow problem on the desktop
xfconf-query -c xfce4-desktop -p /desktop-icons/center-text -n -t bool -s false

12. Nice to have Manjaro packages:

gufw (firewall gui)
chromium (browser)
synapse (launcher)
psensor (temperature)
ether (pendrive tool)
file-roller
p7zip zip unzip unrar
tilda (background terminal)

13. Laravel development quickstart:

git clone project.git
cd project
composer install
composer update
cp .env.example .env
php artisan key:generate
php artisan serve --port=8000

14. Restricting sFTP:

Add to /etc/ssh/sshd_config:

Subsystem sftp internal-sftp
 
  Match Group sftpusers
  ChrootDirectory %h
  ForceCommand internal-sftp
  AllowTcpForwarding no
  X11Forwarding no

Adding user and change configuration:

useradd newuser
usermod -s /bin/false newuser
chown root:root /home/newuser
chmod 755 /home/newuser
mkdir /home/newuser/writablefolder
chown newuser:newuser /home/newuser/writeablefolder
chmod 755 /home/newuser/writeablefolder
 
groupadd sftpusers
usermod -G sftpusers newuser
 
service sshd restart
service vsftpd restart

15. Replacing a broken harddisk (/dev/sda) with two partitions with RAID1 (md0 + md1):

#after having replaced the harddisk; copying MBR partition table
sfdisk -d /dev/sdb | sfdisk /dev/sda
#adding partitions to existing RAID-array md0 and md1:
sudo mdadm /dev/md0 -a /dev/sda1
sudo mdadm /dev/md1 -a /dev/sda2
#reinstalling grub on both disks again, leave options empty, select two disks (/dev/sda + /dev/sdb)
sudo dpkg-reconfigure grub-pc

16. Rsync command which doesn’t copy everything every time while preserving times:

rsync -avP source destination
#a = archive mode
#v = verbose
#P = show progress

17. Bash: iterate through all files within a specific folder and count its files

for i in /folderpath /folderpath ; do 
    echo -n $i": " ; 
    (find "$i" -type f | wc -l) ; 
done

18. copying KVM to PROXMOX

# converting image
qemu-img convert -f qcow2 -O raw image.qcow2 image.img

# creating VM in proxmox
qm create 120 --bootdisk scsi0

# importing disk to someStorage (as named in "pvesm status")
# importdisk adds the image as unused disk
qm importdisk 120 someImage.img someStorage

# attaching the image to the VM. still needs to be marked as boot device by hand
qm set 120 --scsi0 someStorage:vm-120-disk-0

19. repair a degraded ZFS pool/raid1 in PROXMOX

# "zpool status" output during normal operation

  pool: rpool
 state: ONLINE
  scan: none requested
config:

	NAME                                             STATE     READ WRITE CKSUM
	rpool                                            ONLINE       0     0     0
	  mirror-0                                       ONLINE       0     0     0
	    ata-SanDisk_SDSSDH3_2T00_193732800276-part3  ONLINE       0     0     0
	    ata-SanDisk_SDSSDH32000G_192970802134-part3  ONLINE       0     0     0

errors: No known data errors

# "zpool status" of degraded state if we unplug /dev/sdb manually

  pool: rpool
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
	invalid.  Sufficient replicas exist for the pool to continue
	functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: none requested
config:

	NAME                                             STATE     READ WRITE CKSUM
	rpool                                            DEGRADED     0     0     0
	  mirror-0                                       DEGRADED     0     0     0
	    ata-SanDisk_SDSSDH3_2T00_193732800276-part3  ONLINE       0     0     0
	    ata-SanDisk_SDSSDH32000G_192970802134-part3  UNAVAIL      3   212     0

errors: No known data errors

We see that the disk with ID ata-SanDisk_SDSSDH32000G_192970802134-part3 is not available any more. We have to do the following steps:

• shutdown the server
• plug in a brand new replacement disk
• boot the server up again

You’ll see now another status:

  pool: rpool
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
	invalid.  Sufficient replicas exist for the pool to continue
	functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: none requested
config:

	NAME                                             STATE     READ WRITE CKSUM
	rpool                                            DEGRADED     0     0     0
	  mirror-0                                       DEGRADED     0     0     0
	    ata-SanDisk_SDSSDH3_2T00_193732800276-part3  ONLINE       0     0     0
	    14464946144281260868                         UNAVAIL      0     0     0  was /dev/disk/by-id/ata-SanDisk_SDSSDH32000G_192970802134-part3

We need to prepare the new drive and replace it:

# copy partition table from good /dev/sda to new /dev/sdb
sgdisk -R /dev/sdb /dev/sda
sgdisk -G /dev/sdb

# copy the bios and efi boot partitions
dd if=/dev/sda1 of=/dev/sdb1 bs=512
dd if=/dev/sda2 of=/dev/sdb2 bs=512

# replace command
zpool replace rpool ata-SanDisk_SDSSDH32000G_192970802134-part3 /dev/sdb3

Keep in mind: Never use the detach command to remove a disk from a pool, because by adding it again you might be not careful enough to add it in mirror function. The default behaviour is to add a disk in striping mode (raid0)! You’ll have to backup everything and to reinstall a mirrored array again from scratch. Useful commands are:

# show raid status
zpool status

# show all your disk IDs
ls -l /dev/disk/by-id/