User:Laurier Rochon/prototyping/varlogmessages: Difference between revisions

From XPUB & Lens-Based wiki
No edit summary
No edit summary
 
(18 intermediate revisions by the same user not shown)
Line 1: Line 1:
My eternal journey through packet capturing, wireless cracking and TCP/IP bending.
open-wrt, DD-wrt, shell scripts and other nice things.
== Nov 26 2011 ==
I haz all your packets, APPLE fanboyz.
[[File:ssp1.jpg]]
Some files SCPed and ready to get inspected...
[[File:ssp2.jpg]]
* Also added today a reboot script (crontabbed it) every 6 hours. Somehow the TCPDump command hangs after some 20+ hours of operation, and it's a bit hard to debug. Could write to log file, but I prefer having clean working system anyways. Rebooting only takes ~30 seconds, so not a huge loss out of 6 hours (0.13% of "downtime" is manageable!)
== Nov 22 2011 ==
* To connect FROM router TO another machine /w script, make sure the pub key is in ~/.ssh/known_hosts and NOT .ssh/known_hosts (what a waste of time...). Thankfully, this file doesn't get wiped upon reboot.
* And a very simple startup script
<source lang="bash">
#!/bin/sh /etc/rc.common
START=99 #this is the order you want to load with init. had it at 50 and wouldn't work...
start() {
        mkdir -p /tmp/captures
        sh /scripts/scraper > /dev/null 2>&1 & #just a really long tcpdump command
        sh /scripts/manager > /dev/null 2>&1 & #check periodically if the files are getting too big, if so scp to PZI server
                                                             
}                                           
                                             
stop() {
        pkill tcpdump       
        rm /tmp/captures/* 
        echo "Deleted files in /tmp/captures folder, stopped tcpdump"
}                                                                   
                                                                   
boot(){                                                             
        start
}
</source>
== Nov 20 2011 ==
OpenWrt now starting to actually do nasty stuff...
* limited tcpdump to TCP port 80 traffic for now...
[[File:ss3.jpg]]
* TOP LEFT : on the router, tmp folder, making sure files get gzipped properly, then SCPed then deleted right away (very limited space...)
* TOP RIGHT : checking on PZI server to make sure all goes well, files piling up
* BOTTOM LEFT : tcpdump command, reading packets from wireless interface, writing to tmp folder
* BOTTOM RIGHT : manager script that looks for "full" files to rename, gzip, scp, gunzip on server, tcpdump -r and collect the garbage. Bash script below -
<source lang="bash">
while [ "true" ]
do
  for i in 0 1 2 3 4
  do
    if [ -f /tmp/captures/capturefile$i ]
    then
      F=$(cat /tmp/captures/capturefile$i | wc -c)
      if [ $F -gt 1000000  ]
      then
        echo compressing, transferring file...
        #actually doing this on the PZI server now, wayyyyy faster and easier. took out -n for DNS name resolving.
        #tcpdump -XX -aer /tmp/captures/capturefile$i > /tmp/captures/tmptcpfile
        #echo "done reading"
        Q=$(date +"%Y%m%d%H%M%S")
        mv /tmp/captures/capturefile$i /tmp/captures/$Q
        gzip /tmp/captures/$Q
        echo "gzipped"
        echo "starting SCP"
        scp -i /etc/dropbear/dropbear_rsa_host_key /tmp/captures/$Q.gz lrochon@pzwart3.wdka.hro.nl:public_html/captures
        echo "SCP done"
        ssh -i /etc/dropbear/dropbear_rsa_host_key lrochon@pzwart3.wdka.hro.nl "cd public_html/captures; gunzip $Q.gz; tcpdump -XX -Aer $Q > _$Q; rm $Q; exit;"
        echo "sshed into pzi, done gunzip"
        rm /tmp/captures/$Q.gz
        echo "removed file"
      fi
    fi
  sleep 3
  echo waiting...
  done
done
</source>
== Nov 18 2011 ==
Some data viz eye candy by Ryoji Lkeda
* http://www.youtube.com/watch?v=XL06WpbGoZY&feature=related
* http://www.youtube.com/watch?v=SUrRH8hdQtg
* http://www.youtube.com/watch?v=fQi8FiYPLHo&feature=related
* http://www.youtube.com/watch?v=lk_38sywJ6U&feature=related
* http://www.youtube.com/watch?v=ZVlrG8-i7zo&feature=related
* http://www.youtube.com/watch?v=k3J4d4RbeWc&feature=related
* http://www.ryojiikeda.com/project/datamatics/
== Nov 17 2011 ==
Today
* Used the infamous AWK to create a nice 1-liner which formats the iwlist of wireless signals
<source lang="bash">
sudo iwlist wlan0 scanning | awk -F '[ :=]+' '/Qual/ {printf $3 "\t";} /SSI/{printf $3} /WEP/{printf "WEP"} /SSI/{printf "\n"}' | ./order.sh
</source>
Turns out it comes on openWrt! woohoo - no need to install it...
Which basically pipes the result of "iwlist" (wireless scan) into awk, which cuts it up in pieces (split) on certain delimiters (:,=) and then checks for quality, SSID and if encryption is on, then orders upon those criteria - using another pipe into "order.sh" (below).
<source lang="bash">
#!/bin/bash
declare -a ARRAY
exec 10<&0
let count=0
while read LINE; do
    ARRAY[$count]=$LINE
    ((count++))
done
#echo Number of elements: ${#ARRAY[@]}
readarray -t sorted < <(for a in "${ARRAY[@]}"; do echo "$a"; done | sort -g -r)
for a in "${sorted[@]}"; do echo "$a"; done
exec 0<&10 10<&-
</source>
Which results in something like this...
<source lang="text">
70/70 off "FREE_PUBLIC_WIFI"
43/70 on "singel162wgb"
30/70 on "Singelsluis"
25/70 on "Singel162wna"
24/70 on "x00x00"
22/70 on "Thomson0EEFA9"
22/70 on "Sitecom6E2660"
21/70 on "jackie"
20/70 on "Techno"
14/70 on "UPC0050484"
11/70 on "REDTEX2"
11/70 on "airblok"
8/70 on "Thuis"
7/70 on "SitecomB0A46C"
6/70 on "Home_Net"
2/70 on "UPC0039700"
</source>
* did more bash scripting stuff. creating startup scripts for the router in /etc/init.d and more
* tcpdump takes up very little memory, and basically no CPU, yay! (below, the contents of "top")
<source lang="text">
Mem: 13820K used, 504K free, 0K shrd, 636K buff, 4584K cached
CPU:  0% usr  1% sys  0% nic  97% idle  0% io  0% irq  0% sirq
Load average: 0.30 0.25 0.10 2/21 864
  PID  PPID USER    STAT  VSZ %MEM %CPU COMMAND
  817    1 root    S      676  5%  1% /usr/bin/luci-bwc -d
  833  789 root    S    1184  8%  1% /usr/sbin/dropbear -p 22 -P /var/run/
  864  834 root    R    1408  10%  0% top
  452    1 root    S    4440  31%  0% tcpdump -w /tmp/captures/capturefile
  834  833 root    S    1420  10%  0% -ash
  148    1 root    S    1420  10%  0% syslogd -C16
    1    0 root    S    1408  10%  0% init     
  119    1 root    S    1408  10%  0% init     
  116    1 root    S    1404  10%  0% /bin/sh /etc/init.d/rcS S boot
  150    1 root    S    1400  10%  0% klogd
  118  116 root    S    1400  10%  0% logger -s -p 6 -t sysinit
  789    1 root    S    1120  8%  0% /usr/sbin/dropbear -p 22 -P /var/run/
  797    1 root    S    1116  8%  0% /usr/sbin/uhttpd -f -h /www -r OpenWr
  811    1 nobody  S      860  6%  0% /usr/sbin/dnsmasq -D -y -Z -b -E -s l
    8    1 root    SW      0  0%  0% [mtdblockd]
    3    1 root    SWN      0  0%  0% [ksoftirqd_CPU0]
    4    1 root    SW      0  0%  0% [kswapd]
    2    1 root    SW      0  0%  0% [keventd]
    5    1 root    SW      0  0%  0% [bdflush]
    6    1 root    SW      0  0%  0% [kupdated]
</source>
== Nov 16 2011 ==
Some discoveries from today
* note to self : everything that I add in the /tmp folder of openWrt is WIPED OUT upon reboot.
* 1MB of raw packets scraped from tcpdump > ~300-400k after being read back into tcpdump (-r) > ~50-60K gzipped = PROFIT!
* lsof is 1.5megs - fuser is 5K, but relies on Perl , which is 15Megs...pft. All this just to check on open files?
'''SSH into your router /w rsa keys'''
<source lang="bash">
ssh-keygen -t dsa
scp ~/.ssh/id_dsa.pub root@192.168.1.1:/tmp
cd /etc/dropbear
cat /tmp/id_*.pub >> authorized_keys
chmod 0600 authorized_keys
ssh root@192.168.1.1
</source>
'''SSH <FROM> your router <INTO> another machine'''
<source lang="bash">
dropbearkey -y -f /etc/dropbear_rsa_host_key | grep ssh-rsa > /tmp/dropbear_rsa_host_key.pub #this dropbear thingy is the magic part
#you might have to / or not add "root@$router_hostname" at the end of that file if it's missing
scp root@192.168.1.10:/tmp/dropbear_rsa_host_key.pub ~/.ssh/ #will ask for PW. copy the file onto machine you need to SSH into
cat /tmp/dropbear_rsa_host_key.pub >> ~/.ssh/authorized_keys # same thing here
</source>
and then you can ssh/scp back into your computer, another remote computer, another router, etc.
== Nov 15 2011 ==
== Nov 15 2011 ==


Latest revision as of 15:07, 12 January 2012

My eternal journey through packet capturing, wireless cracking and TCP/IP bending.

open-wrt, DD-wrt, shell scripts and other nice things.

Nov 26 2011

I haz all your packets, APPLE fanboyz.

Ssp1.jpg

Some files SCPed and ready to get inspected...

Ssp2.jpg

  • Also added today a reboot script (crontabbed it) every 6 hours. Somehow the TCPDump command hangs after some 20+ hours of operation, and it's a bit hard to debug. Could write to log file, but I prefer having clean working system anyways. Rebooting only takes ~30 seconds, so not a huge loss out of 6 hours (0.13% of "downtime" is manageable!)


Nov 22 2011

  • To connect FROM router TO another machine /w script, make sure the pub key is in ~/.ssh/known_hosts and NOT .ssh/known_hosts (what a waste of time...). Thankfully, this file doesn't get wiped upon reboot.
  • And a very simple startup script
#!/bin/sh /etc/rc.common

START=99 #this is the order you want to load with init. had it at 50 and wouldn't work...

start() {
        mkdir -p /tmp/captures
        sh /scripts/scraper > /dev/null 2>&1 & #just a really long tcpdump command
        sh /scripts/manager > /dev/null 2>&1 & #check periodically if the files are getting too big, if so scp to PZI server
                                                              
}                                             
                                              
stop() {
        pkill tcpdump        
        rm /tmp/captures/*   
        echo "Deleted files in /tmp/captures folder, stopped tcpdump"
}                                                                    
                                                                     
boot(){                                                              
        start
}


Nov 20 2011

OpenWrt now starting to actually do nasty stuff...

  • limited tcpdump to TCP port 80 traffic for now...


Ss3.jpg

  • TOP LEFT : on the router, tmp folder, making sure files get gzipped properly, then SCPed then deleted right away (very limited space...)
  • TOP RIGHT : checking on PZI server to make sure all goes well, files piling up
  • BOTTOM LEFT : tcpdump command, reading packets from wireless interface, writing to tmp folder
  • BOTTOM RIGHT : manager script that looks for "full" files to rename, gzip, scp, gunzip on server, tcpdump -r and collect the garbage. Bash script below -
while [ "true" ]

do

  for i in 0 1 2 3 4
  do

    if [ -f /tmp/captures/capturefile$i ]
    then
       F=$(cat /tmp/captures/capturefile$i | wc -c)
       if [ $F -gt 1000000  ]
       then
         echo compressing, transferring file...
         #actually doing this on the PZI server now, wayyyyy faster and easier. took out -n for DNS name resolving.
         #tcpdump -XX -aer /tmp/captures/capturefile$i > /tmp/captures/tmptcpfile
         #echo "done reading"

         Q=$(date +"%Y%m%d%H%M%S")
         mv /tmp/captures/capturefile$i /tmp/captures/$Q
         gzip /tmp/captures/$Q
         echo "gzipped"

         echo "starting SCP"
         scp -i /etc/dropbear/dropbear_rsa_host_key /tmp/captures/$Q.gz lrochon@pzwart3.wdka.hro.nl:public_html/captures
         echo "SCP done"

         ssh -i /etc/dropbear/dropbear_rsa_host_key lrochon@pzwart3.wdka.hro.nl "cd public_html/captures; gunzip $Q.gz; tcpdump -XX -Aer $Q > _$Q; rm $Q; exit;"
         echo "sshed into pzi, done gunzip"

         rm /tmp/captures/$Q.gz
         echo "removed file"
       fi
     fi
  sleep 3
  echo waiting...
  done
done

Nov 18 2011

Some data viz eye candy by Ryoji Lkeda


Nov 17 2011

Today

  • Used the infamous AWK to create a nice 1-liner which formats the iwlist of wireless signals
sudo iwlist wlan0 scanning | awk -F '[ :=]+' '/Qual/ {printf $3 "\t";} /SSI/{printf $3} /WEP/{printf "WEP"} /SSI/{printf "\n"}' | ./order.sh

Turns out it comes on openWrt! woohoo - no need to install it...

Which basically pipes the result of "iwlist" (wireless scan) into awk, which cuts it up in pieces (split) on certain delimiters (:,=) and then checks for quality, SSID and if encryption is on, then orders upon those criteria - using another pipe into "order.sh" (below). 

#!/bin/bash
declare -a ARRAY
exec 10<&0
let count=0

while read LINE; do
    ARRAY[$count]=$LINE
    ((count++))
done

#echo Number of elements: ${#ARRAY[@]}
readarray -t sorted < <(for a in "${ARRAY[@]}"; do echo "$a"; done | sort -g -r)
for a in "${sorted[@]}"; do echo "$a"; done
exec 0<&10 10<&-


Which results in something like this... 

70/70	off	"FREE_PUBLIC_WIFI"
43/70	on	"singel162wgb"
30/70	on	"Singelsluis"
25/70	on	"Singel162wna"
24/70	on	"x00x00"
22/70	on	"Thomson0EEFA9"
22/70	on	"Sitecom6E2660"
21/70	on	"jackie"
20/70	on	"Techno"
14/70	on	"UPC0050484"
11/70	on	"REDTEX2"
11/70	on	"airblok"
8/70	on	"Thuis"
7/70	on	"SitecomB0A46C"
6/70	on	"Home_Net"
2/70	on	"UPC0039700"


  • did more bash scripting stuff. creating startup scripts for the router in /etc/init.d and more
  • tcpdump takes up very little memory, and basically no CPU, yay! (below, the contents of "top")
Mem: 13820K used, 504K free, 0K shrd, 636K buff, 4584K cached
CPU:   0% usr   1% sys   0% nic  97% idle   0% io   0% irq   0% sirq
Load average: 0.30 0.25 0.10 2/21 864
  PID  PPID USER     STAT   VSZ %MEM %CPU COMMAND
  817     1 root     S      676   5%   1% /usr/bin/luci-bwc -d 
  833   789 root     S     1184   8%   1% /usr/sbin/dropbear -p 22 -P /var/run/
  864   834 root     R     1408  10%   0% top 
  452     1 root     S     4440  31%   0% tcpdump -w /tmp/captures/capturefile
  834   833 root     S     1420  10%   0% -ash 
  148     1 root     S     1420  10%   0% syslogd -C16 
    1     0 root     S     1408  10%   0% init       
  119     1 root     S     1408  10%   0% init       
  116     1 root     S     1404  10%   0% /bin/sh /etc/init.d/rcS S boot 
  150     1 root     S     1400  10%   0% klogd 
  118   116 root     S     1400  10%   0% logger -s -p 6 -t sysinit 
  789     1 root     S     1120   8%   0% /usr/sbin/dropbear -p 22 -P /var/run/
  797     1 root     S     1116   8%   0% /usr/sbin/uhttpd -f -h /www -r OpenWr
  811     1 nobody   S      860   6%   0% /usr/sbin/dnsmasq -D -y -Z -b -E -s l
    8     1 root     SW       0   0%   0% [mtdblockd]
    3     1 root     SWN      0   0%   0% [ksoftirqd_CPU0]
    4     1 root     SW       0   0%   0% [kswapd]
    2     1 root     SW       0   0%   0% [keventd]
    5     1 root     SW       0   0%   0% [bdflush]
    6     1 root     SW       0   0%   0% [kupdated]

Nov 16 2011

Some discoveries from today

  • note to self : everything that I add in the /tmp folder of openWrt is WIPED OUT upon reboot.
  • 1MB of raw packets scraped from tcpdump > ~300-400k after being read back into tcpdump (-r) > ~50-60K gzipped = PROFIT!
  • lsof is 1.5megs - fuser is 5K, but relies on Perl , which is 15Megs...pft. All this just to check on open files?


SSH into your router /w rsa keys 

ssh-keygen -t dsa
scp ~/.ssh/id_dsa.pub root@192.168.1.1:/tmp
cd /etc/dropbear
cat /tmp/id_*.pub >> authorized_keys
chmod 0600 authorized_keys
ssh root@192.168.1.1

SSH <FROM> your router <INTO> another machine 

dropbearkey -y -f /etc/dropbear_rsa_host_key | grep ssh-rsa > /tmp/dropbear_rsa_host_key.pub #this dropbear thingy is the magic part
#you might have to / or not add "root@$router_hostname" at the end of that file if it's missing
scp root@192.168.1.10:/tmp/dropbear_rsa_host_key.pub ~/.ssh/ #will ask for PW. copy the file onto machine you need to SSH into
cat /tmp/dropbear_rsa_host_key.pub >> ~/.ssh/authorized_keys # same thing here

and then you can ssh/scp back into your computer, another remote computer, another router, etc.

Nov 15 2011

Some wget schtuff 

wget -o /dev/null --post-data "var1=DFKSDFLSJFSLKJFS" http://www.YOURURL.com/file.php

Posting data to a webpage using wget! basically the --post-data is what makes it possible, and -o just suppresses output. fancy!

Nov 13 2011

The magic command to make the -C option work in TCPDUMP 

tcpdump -w capturefile -C 1 -tt -v -s 0 -W 3
  • -C is millions of bytes, so 1 is a single MB
  • -W is rotate 3 files and start overwriting the #1 file if you've reached the limit. This maxes out to 3MB of packets, and avoid memory problems. I will probably make this even less in time.


Nov 12 2011

Some stuff that I can see when you connect to my open network, and you're not SSLed

I can infer :

  • your ip address
  • website urls/addresses your are checking out
  • your cookies
  • the time/date of your connection
  • type of requests
  • a bunch of stuff about the sites/ips you are accessing (server type, charset, encoding, expiry of cached items, etc.)
  • your type of computer
  • your browser
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip
Vary: Accept-Encoding
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
Set-Cookie: expEAPID=00000; expires=Sun, 13-Nov-2011 22:09:18 GMT; path=/
Set-Cookie: hl_upm=Nn2Ysa9T4Cqie71CJ0BJAQ2dl+sJdqq8EU242ghkODRO2IkiNYVxHYyIPWFN+6wqfsvxGC/9jrVBX928g/PtId9XJDSkLwiJ8Im2YjeI24XXxWtfKkjWFgc9Yav3yYuRNoqggHdVGBEPuF+rxbrMexjiU69envcu8Y4nl+6v5FEzXgGFpF65EFS/ZnGMS1CNrpkag4pXQ9xn1GUlebDS4KsI0fGUrmWdYsYZ5D3ixtq7u4oGj8HkEESArT6rNLxIlchMPUpGe/n5ot3uLg18u8mK2hafL/Hbejp8nexXmlUAx7wNHQE9jWviiPWQt0DV; expires=Tue, 11-Nov-2014 22:09:18 GMT; path=/
X-Powered-By: ASP.NET
p3p: CP="ALL DSP COR CUR ADMo DEVo PSAo PSDo IVDi OUR STP PRE"
Date: Sat, 12 Nov 2011 22:09:17 GMT
Connection: close
Content-Length: 171


Host: extras.expedia.com
Connection: keep-alive
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1
Accept: 
03:33:55.707141 IP (tos 0x0, ttl 64, id 34647, offset 0, flags [DF], proto TCP (6), length 821)
    192.168.1.80.49813 > 209.235.221.100.www: Flags [P.], cksum 0xdebd (correct), seq 1369:2138, ack 1, win 14600, options [nop,nop,TS val 396284 ecr 820465701], length 769
E..5.W@.@.?#...P...d...Pq....._:..9........
....0.P%*/*
Referer: http://www.expedia.ca/Flights-Search?c=81da3b4b-fa6c-4ad6-90c0-85eb0680061b&
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: s_vi=[CS]v1|275F76E685010720-600001156055591E[CE]; UID=2022018174|0|0; U9Z5=3Lu9nSNzg7HLr4Cf1D4DszGoQyp2VhN7B4X8zISY4mPMYbGAwx7Gb_A; expEAPID=0000; hl_upm=Nn2Ysa9T4Cqie71CJ0BJAQ2dl+sJdqq8EU242ghkODRO2IkiNYVxHYyIPWFN+6wqfsvxGC/9jrVBX928g/PtIWvbHBSJ0UHbBfQl1hvELszPXpQ7noYDsJlI6gfxmbpusNW6RXkpfOCE3FBP4yjN62ERPvDbEVdHRh3iWHvlZKe+OU7GblMzSkAU+y8E3tfu7eURIbBUa+yuVFpFA0vc2GKF8puH2Ntqvw5a9TwZ1WOLHA69KGZt/oXjFXK4qxOnDBFzUxMcc7+os6N4QJqZXOoPc4oihgL+bu2lvS9risbpRlyAWCnqTH0D/gLppMVL; hl_ubm=uKjVtH2uMJkYHxNp2xd/i9ghFhQsIBk98RpJ41rmDQtykvzGxwuI5WjQGN4RgkyM


..'GET /hphotos-ak-snc7/295091_10150336609875660_507750659_10156333_1388848_s.jpg HTTP/1.1
Host: photos-e.ak.fbcdn.net
Connection: keep-alive
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1
Accept: */*
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
  • Managed to revive a bricked router recently
  • Managed to brick another router I had working fine yesterday...meh.

openWrt shows me many nice things...

Openwrt1.jpg


Just like your average linux box...but you have about 1000K free to operate things (on a classic WRT54GL anyways). Choose your packages wisely!

Openwrt2.jpg


And it also sports a nice GUI, so I can monitor you in pink colors

Openwrt3.jpg


The end for now

Openwrt4.jpg

Nov 11 2011

Hardware harvesting

New router /w USB connector on its way...

Got my hands on TWO WRT54G's to do some flashing, testing and whatnot. I can actually try to create an AP with one and then attempt to connect using the other...hmm. Also got a 50m ethernet cable to get things going.

Wrts.jpg

And then one of these two devices would be talking to my Arduino hypothetically through my Ethernet Shield mounted onto the arduino

Arduino ethernet.jpg

Retrieved from "https://pzwiki.wdka.nl/mw-mediadesign/index.php?title=User:Laurier_Rochon/prototyping/varlogmessages&oldid=22402"