User:Laurier Rochon/prototyping/varlogmessages: Difference between revisions

Revision as of 14:11, 26 November 2011

Nov 26 2011

I haz all your packets.

Some files SCPed and ready to get inspected...

Also added today a reboot script (crontabbed it) every 6 hours. Somehow the TCPDump command hangs after some 20+ hours of operation, and it's a bit hard to debug. Could write to log file, but I prefer having clean working system anyways. Rebooting only takes ~30 seconds, so not a huge loss out of 6 hours (0.13% of "downtime" is manageable!)

Nov 22 2011

To connect FROM router TO another machine /w script, make sure the pub key is in ~/.ssh/known_hosts and NOT .ssh/known_hosts (what a waste of time...). Thankfully, this file doesn't get wiped upon reboot.
And a very simple startup script

#!/bin/sh /etc/rc.common

START=99 #this is the order you want to load with init. had it at 50 and wouldn't work...

start() {
        mkdir -p /tmp/captures
        sh /scripts/scraper > /dev/null 2>&1 & #just a really long tcpdump command
        sh /scripts/manager > /dev/null 2>&1 & #check periodically if the files are getting too big, if so scp to PZI server
                                                              
}                                             
                                              
stop() {
        pkill tcpdump        
        rm /tmp/captures/*   
        echo "Deleted files in /tmp/captures folder, stopped tcpdump"
}                                                                    
                                                                     
boot(){                                                              
        start
}

Nov 20 2011

OpenWrt now starting to actually do nasty stuff...

limited tcpdump to TCP port 80 traffic for now...

TOP LEFT : on the router, tmp folder, making sure files get gzipped properly, then SCPed then deleted right away (very limited space...)
TOP RIGHT : checking on PZI server to make sure all goes well, files piling up
BOTTOM LEFT : tcpdump command, reading packets from wireless interface, writing to tmp folder
BOTTOM RIGHT : manager script that looks for "full" files to rename, gzip, scp, gunzip on server, tcpdump -r and collect the garbage. Bash script below -

while [ "true" ]

do

  for i in 0 1 2 3 4
  do

    if [ -f /tmp/captures/capturefile$i ]
    then
       F=$(cat /tmp/captures/capturefile$i | wc -c)
       if [ $F -gt 1000000  ]
       then
         echo compressing, transferring file...
         #actually doing this on the PZI server now, wayyyyy faster and easier. took out -n for DNS name resolving.
         #tcpdump -XX -aer /tmp/captures/capturefile$i > /tmp/captures/tmptcpfile
         #echo "done reading"

         Q=$(date +"%Y%m%d%H%M%S")
         mv /tmp/captures/capturefile$i /tmp/captures/$Q
         gzip /tmp/captures/$Q
         echo "gzipped"

         echo "starting SCP"
         scp -i /etc/dropbear/dropbear_rsa_host_key /tmp/captures/$Q.gz lrochon@pzwart3.wdka.hro.nl:public_html/captures
         echo "SCP done"

         ssh -i /etc/dropbear/dropbear_rsa_host_key lrochon@pzwart3.wdka.hro.nl "cd public_html/captures; gunzip $Q.gz; tcpdump -XX -Aer $Q > _$Q; rm $Q; exit;"
         echo "sshed into pzi, done gunzip"

         rm /tmp/captures/$Q.gz
         echo "removed file"
       fi
     fi
  sleep 3
  echo waiting...
  done
done

Nov 18 2011

Some data viz eye candy by Ryoji Lkeda

Nov 17 2011

Today

Used the infamous AWK to create a nice 1-liner which formats the iwlist of wireless signals

sudo iwlist wlan0 scanning | awk -F '[ :=]+' '/Qual/ {printf $3 "\t";} /SSI/{printf $3} /WEP/{printf "WEP"} /SSI/{printf "\n"}' | ./order.sh

Turns out it comes on openWrt! woohoo - no need to install it...

Which basically pipes the result of "iwlist" (wireless scan) into awk, which cuts it up in pieces (split) on certain delimiters (:,=) and then checks for quality, SSID and if encryption is on, then orders upon those criteria - using another pipe into "order.sh" (below).

#!/bin/bash
declare -a ARRAY
exec 10<&0
let count=0

while read LINE; do
    ARRAY[$count]=$LINE
    ((count++))
done

#echo Number of elements: ${#ARRAY[@]}
readarray -t sorted < <(for a in "${ARRAY[@]}"; do echo "$a"; done | sort -g -r)
for a in "${sorted[@]}"; do echo "$a"; done
exec 0<&10 10<&-

Which results in something like this...

70/70	off	"FREE_PUBLIC_WIFI"
43/70	on	"singel162wgb"
30/70	on	"Singelsluis"
25/70	on	"Singel162wna"
24/70	on	"x00x00"
22/70	on	"Thomson0EEFA9"
22/70	on	"Sitecom6E2660"
21/70	on	"jackie"
20/70	on	"Techno"
14/70	on	"UPC0050484"
11/70	on	"REDTEX2"
11/70	on	"airblok"
8/70	on	"Thuis"
7/70	on	"SitecomB0A46C"
6/70	on	"Home_Net"
2/70	on	"UPC0039700"

did more bash scripting stuff. creating startup scripts for the router in /etc/init.d and more
tcpdump takes up very little memory, and basically no CPU, yay! (below, the contents of "top")

Mem: 13820K used, 504K free, 0K shrd, 636K buff, 4584K cached
CPU:   0% usr   1% sys   0% nic  97% idle   0% io   0% irq   0% sirq
Load average: 0.30 0.25 0.10 2/21 864
  PID  PPID USER     STAT   VSZ %MEM %CPU COMMAND
  817     1 root     S      676   5%   1% /usr/bin/luci-bwc -d 
  833   789 root     S     1184   8%   1% /usr/sbin/dropbear -p 22 -P /var/run/
  864   834 root     R     1408  10%   0% top 
  452     1 root     S     4440  31%   0% tcpdump -w /tmp/captures/capturefile
  834   833 root     S     1420  10%   0% -ash 
  148     1 root     S     1420  10%   0% syslogd -C16 
    1     0 root     S     1408  10%   0% init       
  119     1 root     S     1408  10%   0% init       
  116     1 root     S     1404  10%   0% /bin/sh /etc/init.d/rcS S boot 
  150     1 root     S     1400  10%   0% klogd 
  118   116 root     S     1400  10%   0% logger -s -p 6 -t sysinit 
  789     1 root     S     1120   8%   0% /usr/sbin/dropbear -p 22 -P /var/run/
  797     1 root     S     1116   8%   0% /usr/sbin/uhttpd -f -h /www -r OpenWr
  811     1 nobody   S      860   6%   0% /usr/sbin/dnsmasq -D -y -Z -b -E -s l
    8     1 root     SW       0   0%   0% [mtdblockd]
    3     1 root     SWN      0   0%   0% [ksoftirqd_CPU0]
    4     1 root     SW       0   0%   0% [kswapd]
    2     1 root     SW       0   0%   0% [keventd]
    5     1 root     SW       0   0%   0% [bdflush]
    6     1 root     SW       0   0%   0% [kupdated]

Nov 16 2011

Some discoveries from today

note to self : everything that I add in the /tmp folder of openWrt is WIPED OUT upon reboot.
1MB of raw packets scraped from tcpdump > ~300-400k after being read back into tcpdump (-r) > ~50-60K gzipped = PROFIT!
lsof is 1.5megs - fuser is 5K, but relies on Perl , which is 15Megs...pft. All this just to check on open files?

SSH into your router /w rsa keys

ssh-keygen -t dsa
scp ~/.ssh/id_dsa.pub root@192.168.1.1:/tmp
cd /etc/dropbear
cat /tmp/id_*.pub >> authorized_keys
chmod 0600 authorized_keys
ssh root@192.168.1.1

SSH <FROM> your router <INTO> another machine

dropbearkey -y -f /etc/dropbear_rsa_host_key | grep ssh-rsa > /tmp/dropbear_rsa_host_key.pub #this dropbear thingy is the magic part
#you might have to / or not add "root@$router_hostname" at the end of that file if it's missing
scp root@192.168.1.10:/tmp/dropbear_rsa_host_key.pub ~/.ssh/ #will ask for PW. copy the file onto machine you need to SSH into
cat /tmp/dropbear_rsa_host_key.pub >> ~/.ssh/authorized_keys # same thing here

and then you can ssh/scp back into your computer, another remote computer, another router, etc.

Nov 15 2011

Some wget schtuff

wget -o /dev/null --post-data "var1=DFKSDFLSJFSLKJFS" http://www.YOURURL.com/file.php

Posting data to a webpage using wget! basically the --post-data is what makes it possible, and -o just suppresses output. fancy!

Nov 13 2011

The magic command to make the -C option work in TCPDUMP

tcpdump -w capturefile -C 1 -tt -v -s 0 -W 3

-C is millions of bytes, so 1 is a single MB
-W is rotate 3 files and start overwriting the #1 file if you've reached the limit. This maxes out to 3MB of packets, and avoid memory problems. I will probably make this even less in time.

Nov 12 2011

Some stuff that I can see when you connect to my open network, and you're not SSLed

I can infer :

your ip address
website urls/addresses your are checking out
your cookies
the time/date of your connection
type of requests
a bunch of stuff about the sites/ips you are accessing (server type, charset, encoding, expiry of cached items, etc.)
your type of computer
your browser

Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip
Vary: Accept-Encoding
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
Set-Cookie: expEAPID=00000; expires=Sun, 13-Nov-2011 22:09:18 GMT; path=/
Set-Cookie: hl_upm=Nn2Ysa9T4Cqie71CJ0BJAQ2dl+sJdqq8EU242ghkODRO2IkiNYVxHYyIPWFN+6wqfsvxGC/9jrVBX928g/PtId9XJDSkLwiJ8Im2YjeI24XXxWtfKkjWFgc9Yav3yYuRNoqggHdVGBEPuF+rxbrMexjiU69envcu8Y4nl+6v5FEzXgGFpF65EFS/ZnGMS1CNrpkag4pXQ9xn1GUlebDS4KsI0fGUrmWdYsYZ5D3ixtq7u4oGj8HkEESArT6rNLxIlchMPUpGe/n5ot3uLg18u8mK2hafL/Hbejp8nexXmlUAx7wNHQE9jWviiPWQt0DV; expires=Tue, 11-Nov-2014 22:09:18 GMT; path=/
X-Powered-By: ASP.NET
p3p: CP="ALL DSP COR CUR ADMo DEVo PSAo PSDo IVDi OUR STP PRE"
Date: Sat, 12 Nov 2011 22:09:17 GMT
Connection: close
Content-Length: 171


Host: extras.expedia.com
Connection: keep-alive
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1
Accept: 
03:33:55.707141 IP (tos 0x0, ttl 64, id 34647, offset 0, flags [DF], proto TCP (6), length 821)
    192.168.1.80.49813 > 209.235.221.100.www: Flags [P.], cksum 0xdebd (correct), seq 1369:2138, ack 1, win 14600, options [nop,nop,TS val 396284 ecr 820465701], length 769
E..5.W@.@.?#...P...d...Pq....._:..9........
....0.P%*/*
Referer: http://www.expedia.ca/Flights-Search?c=81da3b4b-fa6c-4ad6-90c0-85eb0680061b&
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: s_vi=[CS]v1|275F76E685010720-600001156055591E[CE]; UID=2022018174|0|0; U9Z5=3Lu9nSNzg7HLr4Cf1D4DszGoQyp2VhN7B4X8zISY4mPMYbGAwx7Gb_A; expEAPID=0000; hl_upm=Nn2Ysa9T4Cqie71CJ0BJAQ2dl+sJdqq8EU242ghkODRO2IkiNYVxHYyIPWFN+6wqfsvxGC/9jrVBX928g/PtIWvbHBSJ0UHbBfQl1hvELszPXpQ7noYDsJlI6gfxmbpusNW6RXkpfOCE3FBP4yjN62ERPvDbEVdHRh3iWHvlZKe+OU7GblMzSkAU+y8E3tfu7eURIbBUa+yuVFpFA0vc2GKF8puH2Ntqvw5a9TwZ1WOLHA69KGZt/oXjFXK4qxOnDBFzUxMcc7+os6N4QJqZXOoPc4oihgL+bu2lvS9risbpRlyAWCnqTH0D/gLppMVL; hl_ubm=uKjVtH2uMJkYHxNp2xd/i9ghFhQsIBk98RpJ41rmDQtykvzGxwuI5WjQGN4RgkyM


..'GET /hphotos-ak-snc7/295091_10150336609875660_507750659_10156333_1388848_s.jpg HTTP/1.1
Host: photos-e.ak.fbcdn.net
Connection: keep-alive
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1
Accept: */*
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3

Managed to revive a bricked router recently
Managed to brick another router I had working fine yesterday...meh.

openWrt shows me many nice things...

Just like your average linux box...but you have about 1000K free to operate things (on a classic WRT54GL anyways). Choose your packages wisely!

And it also sports a nice GUI, so I can monitor you in pink colors

The end for now

Nov 11 2011

Hardware harvesting

New router /w USB connector on its way...

Got my hands on TWO WRT54G's to do some flashing, testing and whatnot. I can actually try to create an AP with one and then attempt to connect using the other...hmm. Also got a 50m ethernet cable to get things going.

And then one of these two devices would be talking to my Arduino hypothetically through my Ethernet Shield mounted onto the arduino

@@ Line 10: / Line 10: @@
 [[File:ssp2.jpg]]
+* Also added today a reboot script (crontabbed it) every 6 hours. Somehow the TCPDump command hangs after some 20+ hours of operation, and it's a bit hard to debug. Could write to log file, but I prefer having clean working system anyways. Rebooting only takes ~30 seconds, so not a huge loss out of 6 hours (0.13% of "downtime" is manageable!)
 == Nov 22 2011 ==