Below are linux commands I thought were worth saving to later reference (ongoing project).

find

grep

wget

download an entire website

  • wget --random-wait -r -p -e robots=off -U mozilla http://www.example.com

generate random fact

  • wget randomfunfacts.com -O - 2>/dev/null | grep \ | sed "s;^.*\(.*\).*$;\1;"

crontab

$ crontab -e

backup everynight at 10pm

0 22 * * * rsync -av --delete /Directory1/ /Directory2/

download google image datasets

$googleimagesdownload -k "cooling logo" -s medium -l 20 -o /home/casey/up/uplaw/resources/data/ -i trademarks/cooling

narrow by time range

$googleimagesdownload -k "baseball game" -s medium -wr '{"time_min":"09/01/2018","time_max":"09/30/2018"}' -l 500 -o fastai/courses/dl1/data/baseballcricket -i train/baseball -cd ~/chromedriver

find

delete all files recursive ending in .svg

$find . -name "*.svg" -type f --delete

find number of files in each sub dir

$find . -maxdepth 1 -mindepth 1 -type d -exec sh -c 'echo "{} : $(find "{}" -type f | wc -l)" file(s)' \;

find duplicate files based on size then hash (add --delete to delete).

$find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate

graphicl tree of subdirectories

$ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/ /' -e 's/-/|/'

nmap

show all open ports on local network in range 0-255 (gets all machines on localhost)

nmap -oG - 10.0.0.0-255 -p 22 -vv > SCAN

$cat SCAN | grep Up | awk -F " " '{print $2}' > SCAN2

$nmap -iL SCAN2

gets ip address/domain name

$nslookup uplaw.us

rdfind

$ rdfind -dryrun true /home/user

$ rdfind -makehardlinks true /home/user

$ rdfind -deleteduplicates true /home/user

rsync

c = checksum; a= recursive; v=verbose;

$rsync -cav –delete -e ssh /Directory1/ geek@192.168.235.137:/Directory2/

$zip /ZippedFiles/archive.zip /Directory1/ && rsync -av --delete /ZippedFiles/ /Directory2/

ssh tunneling to run jupyter server on remote host (then go to url localhost:8000?token=23tqgr)

$ssh -L 8000:localhost:8888 sammy@your_server_ip

port forward 22 on home desktop, then use ssh tunneling to connect to home elasticsearch on remote server (on remote server, use 8000 instead of 9200 for es port)

$ssh -L 8000:localhost:9200 casey@98.204.53.222

wireshark

track by mac address

eth.src==9C:30:5B:4C:46:7D

Compression--Tar

  1. Backup directory

tar pczvf /root/etc.tar.gz /etc

tar pczf bulk.tar.gz ./bulk

This command creates the archive myarchive.tar.gz which contains the files and folders from the path /home/till/mydocuments. The command line options explained:

[p] This option stand for "preserve", it instructs tar to store details on file owner and file permissions in the archive.

[c] Stands for create. This option is mandatory when a file is created.

[z] The z option enables gzip compression.

[f] The file option tells tar to create an archive file. Tar will send the output to stdout if this option is omitted.

  1. Create tar archive file

tar -cvf tecmint-14-09-12.tar /home/tecmint/

  1. Create tar.gz archive file

tar cvzf MyImages-14-09-12.tar.gz /home/MyImages

  1. Create tar.bz2 Archive File

tar cvfj Phpfiles-org.tar.bz2 /home/php

OR

tar cvfj Phpfiles-org.tar.tbz /home/php

OR

tar cvfj Phpfiles-org.tar.tb2 /home/php

  1. Untar tar Archive File

Untar files in Current Directory

tar -xvf public_html-14-09-12.tar

Untar files in specified Directory

tar -xvf public_html-14-09-12.tar -C /home/public_html/videos/

  1. Uncompress tar.gz Archive File

tar -xvf thumbnails-14-09-12.tar.gz

  1. Uncompress tar.bz2 Archive File

tar -xvf videos-14-09-12.tar.bz2

  1. List Content of tar Archive File

tar -tvf uploadprogress.tar

  1. List Content tar.gz Archive File

tar -tvf staging.tecmint.com.tar.gz

  1. List Content tar.bz2 Archive File

tar -tvf Phpfiles-org.tar.bz2

//replace all string in .c

$find /home/user/directory -name *.c -exec sed -i "s/hello world/hello/g" {} \;

//replace university with resource in all files in root

$find ./ -type f -exec sed -i -e 's/university/resource/g' {} \;

//remove all .pyc file recurssively

find . -name *.pyc -delete

-----WGET

wget -r -l1 -H -nd -A mp3 -e robots=off http://example/url

This will download all files of the type specified after "-A" from a website. Here is a breakdown of the options:

-r turns on recursion and downloads all links on page

-l1 goes only one level of links into the page(this is really important when using -r)

-H spans domains meaning it will download links to sites that don't have the same domain

-nd means put all the downloads in the current directory instead of making all the directories in the path

-A mp3 filters to only download links that are mp3s(this can be a comma separated list of different file formats to search for multiple types)

-e robots=off just means to ignore the robots.txt file which stops programs like wget from crashing the site... sorry http://example/url lol..

wget -r -l1 -H -e robots=off http://www.themesindustry.com/html/xeone/demos/

SCP

scp root@159.89.80.196:/home/xxx/xxx/11.jpeg /Users/username/Desktop/ //Copy files on service to local dir: You must be in local bash terminal to conduct this command, not when you are in ssh!

scp /Users/username/Desktop/11.jpeg username@domain:/home/xxx/xxx //copy files in local dir to remote service : also you must be in local bash terminal

DOCKER COMMANDS

$docker-compose -f local-linux.yml -p uplaw build

$docker-compose -f local-linux.yml -p uplaw up -d

$docker-compose -f local-linux.yml -p uplaw logs --tail=500 backend

//get inside the container

docker-compose -f local-linux.yml -p uplaw exec es bash

//get in container without getting into container

$docker-compose -f local-linux.yml -p uplaw exec backend mvn dependency:resolve -Dclassifer=jar -DexcludeTransitive=false

$docker-compose -f local-linux.yml -p uplaw exec backend mvn dependency:resolve-plugins -DexcludeTransitive=false

//compiles app

$docker-compose -f local-linux.yml -p uplaw exec backend mvn clean package

//

$docker-compose -f local-linux.yml -p uplaw restart backend

docker-compose -f local-linux.yml -p uplaw exec backend /entrypoint.sh

docker-compose -f local-linux.yml -p uplaw exec backend curl "localhost:9200/_cat/indices?v"

docker-compose -f local-linux.yml -p uplaw exec backend curl 'http://localhost:18080/service?name=WikimediaCrawler&action=START'

DOCKER COMMANDS

/opt/env/backend/src/src/main/java/com/uplaw/crawling

docker-compose -f crawlers.yml -p uplaw build

docker-compose -f crawlers.yml -p uplaw up -d

docker-compose -f crawlers.yml -p uplaw exec backend mvn clean package

docker-compose -f crawlers.yml -p uplaw restart backend

docker-compose -f crawlers.yml -p uplaw exec backend curl "localhost:8080/crawler?name=innovation.columbia.edu&action=START"

docker-compose -f crawlers.yml -p uplaw exec backend curl "http://localhost:8080/crawler?name=arkansas.technologypublisher.com&action=START"

PERMISSIONS

sudo chown www-data:www-data -R *

sudo find . -type d -exec chmod 755 {} \;

sudo find . -type f -exec chmod 644 {} \;

docker-compose -f crawlers.yml -p uplaw logs --tail=500 backend

GREP COMMANDS

$grep -rnw '/path/to/somewhere/' -e 'pattern'

grep -rnw '/path/to/somewhere/' -e 'pattern'

-r or -R is recursive,

-n is line number, and

-w stands for match the whole word.

-l (lower-case L) can be added to just give the file name of matching files.

Along with these, --exclude, --include, --exclude-dir flags could be used for efficient searching:

This will only search through those files which have .c or .h extensions:

grep --include=*.{c,h} -rnw '/path/to/somewhere/' -e "pattern"

This will exclude searching all the files ending with .o extension:

grep --exclude=*.o -rnw '/path/to/somewhere/' -e "pattern"

For directories it's possible to exclude a particular directory(ies) through --exclude-dir parameter. For example, this will exclude the dirs dir1/, dir2/ and all of them matching *.dst/:

grep --exclude-dir={dir1,dir2,*.dst} -rnw '/path/to/somewhere/' -e "pattern"

This works very well for me, to achieve almost the same purpose like yours.

For more options check man grep.

replace string

grep -rnwl "." -e "package main.java.com." | xargs sed -i 's/"package main.java.com."/"package com."/g'

grep --include=*.{php,html} -rlw kBase . | xargs sed -i 's/kBase/Silq/g'

grep --include=*.{php,html} -rliw Jango . | xargs sed -i 's/Jango/UpLaw/Ig' //note i and I = ignore case

FIND

$find / -type f -name "" //find filename

$find / -type f -iname "filename*" //find matching filename

$find . -type f -readable ! -executable -size 1033c //find properties human readable, 1033 bytes size, not executable

$$ find / -user bandit7 -group bandit6 -size 33c 2>/dev/null

SCREEN

screen -r

detach screen:

ctrl + a | d

resume detached screen session

ctrl + r

ICONV (convert encodings)

Converting -f (from) latin1 (ISO-8859-1)

-t (to) standard UTF_8

iconv -f ISO-8859-1 -t UTF-8 < input.txt > output.txt

WARGAMES COMMANDS

$cat data.txt | grep "millionth" //The password for the next level is stored in the file data.txt next to the word millionth

$cat data.txt | sort | uniq -u //The password for the next level is stored in the file data.txt and is the only line of text that occurs only once

$ strings data.txt | grep "=" //The password for the next level is stored in the file data.txt in one of the few human-readable strings, beginning with several ‘=’ characters.

$ cat data.txt | base64 --decode //The password for the next level is stored in the file data.txt, which contains base64 encoded data

$ cat data.txt | tr '[A-Za-z]' '[N-ZA-Mn-za-m]' //The password for the next level is stored in the file data.txt, where all lowercase (a-z) and uppercase (A-Z) letters have been rotated by 13 positions

$ zcat in_file > out_file //gzip decompress

$ bzip2 -d file //bzip2 decompress

$tar xvf file //tar decompress

ELASTIC CURL COMMANDS

curl -XPUT '104.196.173.168:9200/*/_settings' -H 'Content-Type: application/json' -d '{ "index" : { "max_result_window" : 500000 } }'

  1. Put mapping from json file

curl -XPUT 'localhost:9200/ll' -H 'Content-Type: application/json' -d @clusters-mappings.json

put date into document

$curl -XPUT 'localhost:9200/my_index/_doc/1?pretty' -H 'Content-Type: application/json' -d'

{ "date": "2015-01-01" }

'

sort by date

$curl -XGET '35.196.6.48:9200/law_library/_search?pretty' -H 'Content-Type: application/json' -d'

{

"sort": { "date": "asc"}

}

'

curl -X GET "10.0.0.104:9200/_cat/templates?v&s=order:desc,index_patterns"

results to csv

curl -X GET 'http://10.0.0.104:9200/opinion/opinion/_search?from=0&size=0' -d '

{"from": 0,

"size": 0,

"facets": {

"resource_uri": {

"global": true,

"terms": {

"order": "count",

"size": 20,

"all_terms": true,

"field": "resource_uri"

}

}

},

"sort": [

{

"_score": "desc"

}

],

"query": {

"filtered": {

"query": {

"match_all": {}

}

}

}

}' | jq -r '.facets["resource_uri"].terms[] | [.term, .count] | @csv'

nmap

$sudo nmap -A -O -sS -p- -oX file localhost

cd

TCPDUMP

First The Basics

Breaking down the Tcpdump Command Line

The following command uses common parameters often seen when wielding the tcpdump scalpel.

:~$ sudo tcpdump -i eth0 -nn -s0 -v port 80

-i : Select interface that the capture is to take place on, this will often be an ethernet card or wireless adapter but could also be a vlan or something more unusual. Not always required if there is only one network adapter.

-nn : A single (n) will not resolve hostnames. A double (nn) will not resolve hostnames or ports. This is handy for not only viewing the IP / port numbers but also when capturing a large amount of data, as the name resolution will slow down the capture.

-s0 : Snap length, is the size of the packet to capture. -s0 will set the size to unlimited - use this if you want to capture all the traffic. Needed if you want to pull binaries / files from network traffic.

-v : Verbose, using (-v) or (-vv) increases the amount of detail shown in the output, often showing more protocol specific information.

port 80 : this is a common port filter to capture only traffic on port 80, that is of course usually HTTP.

Display ASCII text

Adding -A to the command line will have the output include the ascii strings from the capture. This allows easy reading and the ability to parse the output using grep or other commands. Another option that shows both hexadecimal output and ASCII is the -X option.

:~$ sudo tcpdump -A -s0 port 80

Capture on Protocol

Filter on UDP traffic. Another way to specify this is to use protocol 17 that is udp. These two commands will produce the same result. The equivalent of the tcp filter is protocol 6.

:~$ sudo tcpdump -i eth0 udp

:~$ sudo tcpdump -i eth0 proto 17

Capture Hosts based on IP address

Using the host filter will capture traffic going to (destination) and from (source) the IP address.

:~$ sudo tcpdump -i eth0 host 10.10.1.1

Alternatively capture only packets going one way using src or dst.

:~$ sudo tcpdump -i eth0 dst 10.10.1.20

Write a capture file

Writing a standard pcap file is a common command option. Writing a capture file to disk allows the file to be opened in Wireshark or other packet analysis tools.

:~$ sudo tcpdump -i eth0 -s0 -w test.pcap

Line Buffered Mode

Without the option to force line (-l) buffered (or packet buffered -C) mode you will not always get the expected response when piping the tcpdump output to another command such as grep. By using this option the output is sent immediately to the piped command giving an immediate response when troubleshooting.

:~$ sudo tcpdump -i eth0 -s0 -l port 80 | grep 'Server:'

Previous PostPost with a slider and lightbox
Next PostPost with YouTube Video
Comments (2)
John Doe
Posted at 15:32h, 06 December Reply

Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.

John Doe
Posted at 15:32h, 06 December Reply

It is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout. The point of using Lorem Ipsum is that it has a more-or-less normal

John Doe
Posted at 15:32h, 06 December Reply

There are many variations of passages of Lorem Ipsum available, but the majority have suffered alteration in some form, by injected humour, or randomised words which don't look even slightly believable. If you are going to use a passage of Lorem Ipsum, you need to be sure there isn't anything embarrassing hidden in the middle of text.

John Doe
Posted at 15:32h, 06 December Reply

The standard chunk of Lorem Ipsum used since the 1500s is reproduced below for those interested. Sections 1.10.32 and 1.10.33 from "de Finibus Bonorum et Malorum" by Cicero are also reproduced in their exact original form, accompanied by English versions from the 1914 translation by H. Rackham.

Leave a Comment