Skip to content

unix-tips.md

Latest commit

 

History

History
555 lines (387 loc) · 11.8 KB

unix-tips.md

File metadata and controls

555 lines (387 loc) · 11.8 KB

Useful Unix Commands and Notes

This is a compilation of commands that I have found particularly useful in a pinch. 👍

If you want to get an explanation for a command, see --> https://explainshell.com/

Table of Contents

Introduction

Start by learing safe shell scripting best practices.

Every Day Commands

find commands

  • find all files that were modified less that 2 days ago
find . -mtime -2
  • find all files owned by root
find . -uid `id -u root`
  • find all files 28 days old or newer
find . -name '*.log' -mtime -28
  • find/replace with sed
find . -type f -exec sed -i 's/bad/good/g' {} \;
  • find and remove files
find . -type f -exec rm {} \;
  • remove all files except for those that match a pattern
sudo find . -not -name 'FileA*' -type f -exec rm {} \;
  • remove all files except for those that match a list
find ./ -maxdepth 1 -not \( -name 'A*' -or -name 'B*' \)
  • test network speeds by ping the Google Public DNS IP (8.8.8.8)
ping 8.8.8.8

sed commands

  • multiple replacements in one command
sed -i -e 's/string1/val1/' -e '/s/string2/val2/' file.txt
  • delete first 100 lines of a file
sed -i '1,100d' filename

test

Some file test flags. See also the wikipedia article

  • -e -- file exists
  • -f -- file is a regular file
  • -d -- file is a directory
  • ! -- negate

Examples:

if test ! -s "$1"
 then
   echo $1 does not exist or is empty.
 fi

cron

*    *    *    *    *  command to be executed
┬    ┬    ┬    ┬    ┬
│    │    │    │    │
│    │    │    │    │
│    │    │    │    └───── day of week (0 - 6) (0 or 6 are Sunday to Saturday, or use names)
│    │    │    └────────── month (1 - 12)
│    │    └─────────────── day of month (1 - 31)
│    └──────────────────── hour (0 - 23)
└───────────────────────── min (0 - 59)

(borrowed from http://ricostacruz.com/cheatsheets/cron.html)

misc commands

  • load a property file with periods

  • see memory information on a host in MB

free -m

or

cat /proc/meminfo
  • what's my IP? (Use ifconfig.co to get your external facing IP)
curl ifconfig.co
  • find/replace one liner in perl
perl -p -i -e 's/oldstring/newstring/g' `find ./ -name *.html`
  • traverse symlinks for a file and give the actual location
readlink -f /usr/bin/java
  • read a password
read -es password  #reads the next line into $password and masks user input
  • simple replacement for dos2unix (which is not standard)
tr -d '\r' < dosfile > unixfile
  • rename *.bak to *
rename 's/\.bak$//' *.bak
  • find a uid or gid for a user
id -u username
id -g username
  • set http_proxy env variable
http_proxy=http://username:password@hostname:port;
export (or set on Windows) $http_proxy
  • list groups (this searches through naming directories as well as /etc/group)
getent group|grep <group name/>
  • use getent to retrieve a service user name
getent passwd|grep maven|awk -F\: '{print $1}'
  • system calls include child processes
strace -f /bin/true
  • run a simple HTTP server using python (2.7+)
python -m SimpleHTTPServer [port]
TEST="remove.importantfile.txt"
echo ${TEST#*.}
  • cut the end of a value stored in a variable
TEST="importantfile.txt.remove"
echo ${TEST%.*}
  • use pygmentize to generate an HTML output of a diff file
pygmentize -l diff -f html -O full -o diff.html diff.txt
  • pass env variables using sudo
sudo -E bash -c 'echo $HTTP_PROXY'
  • capture command execution in a variable -- this example is useful for a date format string (man date for formats)
timestamp=$(date '+%Y%m%d%H%M')
  • bash shell default to env value or use positional argument
jboss_user=${jboss_user:-"$1"}
  • tar up a directory but exclude logs
tar cvfz mydir.tgz mydir --exclude mydir/logs
  • list a directory sorted by size reversed
ls -lSr
  • check a unique file size (better than md5sum)
sha1sum {file_name}
  • human readable file sizes including summary of first level directories
du -h --max-depth=1 --all
  • interactive tool that shows file sizes
ncdu -x /
  • view configuration of services in /etc/rc[0-9]/. The chkconfig can be used to administer all start up services.
sudo chkconfig --list
  • generate a random 15 character password
head -c 500 /dev/urandom | tr -dc a-zA-Z0-9\.\^\*\? | head -c 15; echo
  • download Java SDK stuff
curl -s http://get.sdkman.io | bash
sdk install kotlin
ps -o nlwp <pid>
  • change your default shell
chsh -s /bin/zsh

file permissions (chmod)

# Permission rwx
7 full 111
6 read and write 110
5 read and execute 101
4 read only 100
3 write and execute 011
2 write only 010
1 execute only 001
0 none 000

Networking

SSL

keytool -list -keystore file.pfx -storetype PKCS12
  • Convert the .pfx to a PEM .cer -- this is a prereq to use a Java PFX cert in curl
openssl pkcs12 -in certificate.pfx -out certificate.cer -nodes

TCP commands

  • find what process is associated with a TCP port
$ /usr/sbin/lsof -i :8380
  • see what ports are listening
sudo lsof -i -P -n | grep LISTEN
  • See all TCP ports
$ /usr/sbin/lsof -Pnl +M -i4
  • ping Google DNS -- this is very useful for checking network reliability
ping -t 8.8.8.8
  • test connectivity to a TCP or UPD port

You can start up a port to listen on:

nc -l <port to listen on>

You can connect to an active port using:

nc -z <IP or host> <port> (e.g., nc - z dev1-app1 1234)
nc -z -v -u <IP or host> <port> (test UDP port)
sudo iptables -A OUTPUT -p tcp --dport 25 -j REJECT

curl commands

  • run curl with the .cer
curl -k https://{https_url} -E ~/certificate.cer:{cert_password}
  • run curl with the http proxy and authentication
curl -O some_url -x <host/>:<port/> -U <user/>:<pass/>

DNS

SSH commands

set up SSH keys

  1. generate keys: ssh-keygen -t rsa
  2. copy the public key: ssh-copy-id user@host

SSH config

  • turn off GSSAPIAuthentication
echo "GSSAPIAuthentication no" >> ~/.ssh/config
  • turn off StrictHostKeyChecking in ssh
ssh -o StrictHostKeyChecking=no <user>@<host> -i <priv_key_loc>
  • restart sshd
service sshd restart
  • to troubleshoot SSH connections, check /var/log/secure

System

sudo mount 1.2.3.4:/volume1/myvolume /local/folder
  • set system time
ntpdate 127.0.0.1 (replace with NTP server IP)
  • change default shell to zsh (on OS X)
chsh -s $(which zsh)

Benchmarking

  • I/O performance benchmarks
hdparm -tT /dev/sda1

Arithmetic

#!/bin/bash

OFFSET=${1}

echo "$((OFFSET + 9990))"

Bash command arguments

Here is an example of a script that can take one or more arguments. The shift 1 command is used to remove the first argument from the list of arguments.

while [[ $# -gt 0 ]]; do
  case "$1" in
    -b)
      BACKUP="true"
      shift 1
      ;;
    --no-dump)
      NO_DUMP="true"
      shift 1
      ;;
    --help)
      usage
      ;;
    *)
      echo "Unknown argument: $1"
      usage
  esac
done

Aliases

AWS specific

alias myip='curl -w "\n" http://169.254.169.254/latest/meta-data/local-ipv4'

References

Reading List

List of other useful Unix/Linux cheatsheets or tutorials: