Get the Wikipedia database!

Wikipedia, possibly world’s largest encyclopedia. It has over 2 million articles. I sometimes shudder at the thought that someday (maybe) if (for some reason) Wikipedia shuts down then what will happen to vast amount of data we have now! Surely, this hears like a dooms day, which we assume it will never ever happen. I am born collector. I love to collect anything which is very dear to my heart and is valuable. So, I was wondering today can I keep a backup copy of Wikipedia? If yes then how BIG would be the database?

As it turned out after some googling and more re-googling that there are more than one ways for this and the size of Wikipedia’s database (all articles, templates, image descriptions, and primary meta-pages, but no images) is 3.2 GB (compressed size – archive as on Jan 2008). Below are the various methods to get the database.

  1. Data dumps (link). These are raw data dumps of Wikipedia. You have options of various kinds of downloads. You can download the wiki in any particular language, or a particular subset of the database (e.g. only the titles of articles or only the abstracts of articles). This is the link to English Wikipedia’s Jan 2008 backup (3.2 GB download size).
  2. Wikipedia on DVD (download link) – only 422MB uncompressed size. This is the best way to download Wikipedia. This is the DVD version of Wikipedia, but has only selected articles and is quite outdated. I would sincerely urge Wikipedia to keep this updated. The DVD consists of a nice GUI provided by the software – Kiwix.
  3. Misc downloads (link). For example, Commons Picture of the year archive (link), MediaWiki – The website software Wikipedia.org uses itself (link).
  4. To download all the pictures of Wikipedia read here (to download the all image torrents click here). Note: that many of these pictures could be copyrighted. So, if you take full assume all liability for the use of any images. The download size could be as big as hundreds of GBs. The best way to download the pictures is download a subset of them. If you want to download only the pictures that are referenced by the XML file you download from here, then use Wikix.
  5. You can also download all XMLs for English Wiki from here.

So, happy life and stay informed.

magic_shutdown: Shutdown computer remotely using Magic Packet

The title says it all. If you don’t know what a Magic Packet is then read here. Below is the code of a shell script (I call magic_shutdown) that uses tcpdump to listen for Magic Packets. When it receives on then the shell script verifies its content to make sure that this packet was meant for this computer. Note this script doesn’t require that your network interface card support Wake-on-LAN. It has no special hardware dependencies.

All-in-all, this script lets you create create a shortcut which will act as remote power button for your computer. Press it to turn the computer ON. Press it again to turn the computer OFF.

Get magic_shutdown‘s code
License: GNU Public License version 3.

[code lang=”bash”]#!/bin/bash
#Author:AppleGrew
#License:GPL version 3

listenPort=9
interface="eth0"

#Forking to daemonize…
if [[ "$2" != "forked" ]]
then
echo "Forking $0…"
"$0" "$1" forked &
echo "Forked."
exit 0
fi

#Creating pid file
ppid=$$
echo $ppid >"$1"

echo "Started"
mac=`ifconfig "$interface"|head -n1|sed -e ‘s/.*HWaddr \([0-9:a-fA-F]*\)/\1/g’ -e ‘s/://g’`
pckt_expect=`echo "ff ff ff ff ff ff $mac $mac $mac $mac $mac $mac $mac $mac"|sed ‘s/ //g’|tr ‘A-Z’ ‘a-z’`
while `true`
do
pckt_data=`tcpdump -i "$interface" -x -c 1 udp port ${listenPort}`
if [[ $? != 0 ]]
then
echo "tcpdump returned error."
exit 1
fi

pckt_data=`echo "$pckt_data" | \
grep ‘0x[0-9]*:’| \
tr ‘A-Z’ ‘a-z’| \
sed ‘s/[ \t]//g’| \
sed ‘s/0x[0-9]*:\([0-9a-f]*\)/\1/g’| \
tr -d ‘\n\r’| \
cut -c 57-`
if [[ "$pckt_data" == "$pckt_expect" ]]
then
echo "Matched! Received Magic packet shutting down…"
shutdown -P now #Not recommended
#For Gnome #Doesn’t work dunno why?
#dbus-send –session –dest=org.gnome.PowerManager \
# –type=method_call –print-reply –reply-timeout=2000 \
# /org/gnome/PowerManager org.gnome.PowerManager.Shutdown
#For KDE 3.5
#dcop `dcop|grep power-manager` power-manager shutdown
exit 0
fi
done

echo "EXITED"
exit 0[/code]

Below is the startup script that must be used to launch magic_shutdown script.

Get launch_magic_shutdown‘s code
License: GNU Public License version 3.

[code lang=”bash”]#!/bin/bash
#Author:AppleGrew
#License:GPL version 3

SCRIPT="/opt/magic_shutdown"
PID_FILE="/var/run/magic_shutdown.pid"
case "$1" in
start)
test -f "$PID_FILE" && echo "Already Running…" && exit 1
"$SCRIPT" "$PID_FILE"
echo "Started"
;;
stop)
pid=`cat "$PID_FILE"`

tcpPid=`pgrep -P $pid tcpdump`
kill -9 $pid
kill -2 $tcpPid
if [ -f "$PID_FILE" ] && ! ps -p $pid >/dev/null
then
rm -f "$PID_FILE"
else
echo "Failed to delete pid file. Maybe its already deleted."
fi
echo "Stopped"
;;
esac[/code]

Installations:-
Assuming that you have downloaded the above two codes into your home directory. Now run the following commands.

sudo cp launch_magic_shutdown /etc/init.d
sudo chmod a+x /etc/init.d/launch_magic_shutdown
cd /etc/rc2.d
sudo ln -s ../init.d/launch_magic_shutdown S99launch_magic_shutdown

sudo cp magic_shutdown /opt/magic_shutdown
sudo chmod a+x /opt/magic_shutdown

Hope this helps. Report in the comments section if you encounter any problem.


Update: Stuart’s Script
Stuart (see comment section) posted his version of magic_shutdown script. If the above one doesn’t work then maybe you should give this one a try.
[code lang=”bash”]#!/bin/sh
#Author: Stuart
#Original Author:AppleGrew
#License:GPL version 3

#Forking to daemonize…
if [[ "$2" != "forked" ]]
then
echo "Forking $0…"
"$0" "$1" forked &
echo "Forked."
exit 0
fi

#Creating pid file
ppid=$$
echo $ppid >"$1"

echo "Started"
interface=`route -n | grep "^0.0.0.0" | awk -F " " ‘{print $8}’`
mac=`ifconfig "$interface"|head -n1|sed -e ‘s/.*HWaddr \([0-9:a-fA-F]*\)/\1/g’ -e ‘s/://g’`
pckt_expect=`echo "$mac $mac $mac $mac $mac $mac $mac $mac $mac $mac $mac $mac $mac $mac $mac $mac"|sed ‘s/ //g’|tr ‘A-Z’ ‘a-z’`
while `true`
do
pckt_data=`tcpdump -i "$interface" -s 0 -x -c 1 \( \(ether dst "$mac" and not ip and not arp and not rarp\) or \(udp port 9\) \)`
if [[ $? != 0 ]]
then
echo "tcpdump returned error."
exit 1
fi
pckt_data=`echo "$pckt_data" | \
grep ‘0x[0-9]*:’| \
tr ‘A-Z’ ‘a-z’| \
sed ‘s/[ \t]//g’| \
sed ‘s/0x[0-9]*:\([0-9a-f]*\)/\1/g’| \
tr -d ‘\n\r’ | \
awk -F "ffffffffffff" ‘{print $2}’`
if [[ "$pckt_data" == "$pckt_expect" ]]
then
echo "Matched! Received Magic packet shutting down…"
rm -f $1
/sbin/poweroff
exit 0
fi
done

echo "EXITED"
exit 0[/code]

Turn on (power on) your computer remotely

I was scanning the net today for ways to turn on any computer remotely. The solution I stumbled upon was Wake-On-LAN (WOL). (Read more) What’s more good, is that most of the computers sold today support this feature.

To know if your system supports this feature or not run the following command

sudo ethtool eth0

You will get output similar to the one below.

Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Advertised auto-negotiation: Yes
Speed: 100Mb/s
Duplex: Full
Port: MII
PHYAD: 32
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
Link detected: yes

In the above please substitute eth0 with the network interface card’s name you want to listen for “Magic Packets“. The Supports Wake-on gives the methods your card supports to wake your computer up. In the output above, this card supports waking on any physical activity, on unicast message, on multicast message, on broadcast message and “Magic Packets” respectively. A value of d for Supports Wake-on means that your card doesn’t support WOL. (For the full list of meanings for these letters see ethtool‘s manpage under the wol portion.) In the output above Wake-on is g, which means that WOL for eth0 is enabled and is set to listen for “Magic Packets” only.

Useful links:-

  1. On Wkipedia, has steps to enable WOL via Windows.
  2. How-to for Ubuntu users.
  3. Very useful resource on WOL.
  4. Shutdown remote computers using Magic Packets.

I am a Nerd!!!

A few months back I came across a blog post “The Nerd Handbook” (read here). While reading it I felt as if it was about me. I always thought about myself to be very different, but now I know, I am just one of the many nerds. Its so intriguing that the author of the post is such an observant man. After reading his post I am trying to get rid of some of the negative traits of nerds. So, wish me luck.

autoWallChange: Automatically changes wallpaper

About the script
This is a KDE (may not work in KDE 4) only shell script which changes wallpaper every few minutes as set by you. KDE too has a built-in feature to change wallpaper but it has few limitations:-

  1. It doesn’t show pictures within the sub-directories of the given path.
  2. It applies the same picture position setting (i.e. Scala&Crop, Tile, etc.) to all the pictures.

This script allows you to apply different picture position settings to pictures from different directories. e.g. if you use the following settings

picDir=”/home/geek/Pictures”
shapeDir=”Rectangle Square Tall”
shapeMode=”8 8 5″

Then pictures from /home/geek/Pictures/Rectangle and /home/geek/Pictures/Square will be shown using Scale & Crop position setting, and pictures from /home/geek/Pictures/Tall will be Titled Maxpect. Following is the table for which number stands for what position setting.

  • 1 -> Centered
  • 2 -> Titled
  • 3 -> Center Titled
  • 4 -> Centered Maxpect
  • 5 -> Titled Maxpect
  • 6 -> Scaled
  • 7 -> Centered Auto Fit
  • 8 -> Scale & Crop

This script is best used with my iflickrDown script (see here).

The code
License: GNU Public License version 3.

[code lang=”bash”]#!/bin/bash
#Coded by AppleGrew
#License: GPL version 3

#############################################
############Alternative way to set###########
##########the following variables############
#Create another shell script and export it #
#from there. #
#===============SAMPLE=======================
# #!/bin/bash
# export picDir="/home/geek/Pictures" #Donot put ‘/’ at the end
# export shapeDir="Square Rectangle Tall"
# export shapeMode="8 8 5"
# export changeWallIntv=2 #in mins
#
# ./autoWallChange >/tmp/autoWall.log & #Launching the main program.
############Configs##########################
picDir="" #Donot put ‘/’ at the end
shapeDir=""
shapeMode=""
changeWallIntv=1 #in mins
#############################################

tmpFile="/tmp/autoWallChange.tmp"
lstFile="/tmp/autoWallChange.lst"
fileLen=0
factor=0

shapeDir=( $shapeDir )
shapeMode=( $shapeMode )

echo "${shapeDir[@]}"
echo "${shapeMode[@]}"

function refreshFileList(){
local DIR_PATH=$1
local sM=$2
echo "Refreshing content of ‘$DIR_PATH’, shape mode is $sM"

if [ -a "$DIR_PATH" ] && [ -d "$DIR_PATH" ] && [ -w "$DIR_PATH" ]
then
for f in `find "$DIR_PATH" -type f -print | sed ‘s/ /|/g’`
do
matchFound=0
for fl in `cat "$tmpFile"`
do
fl=`echo "$fl"|sed -e ‘s/\([^>]*\)>\([^>]*\)/\1\n\2/g’|head -n 1|tail -n 1`
[[ "$fl" == "$f" ]] && matchFound=1 && break
done
[[ $matchFound == 0 ]] && printf "%s>%d\n" $f $sM >>"$tmpFile" && let ‘fileLen++’
done
let ‘factor=(fileLen-1)*100000000/32767’
fi
echo "Refreshed"
}

printf "">"$tmpFile" #Creating new empty file.
i=0
for shape in ${shapeDir[@]}
do
test -a "$picDir/$shape" && find -L "$picDir/$shape" -type f -print > "$lstFile.$i"
test -a "$picDir/$shape" && refreshFileList "$picDir/$shape" ${shapeMode[$i]}
let ‘i++’
done

shownPicsLine=0
lastL=""
totShown=0
RANDOM=$$
while true
do
let ‘l=(factor*RANDOM+100000000)/100000000’
test $totShown -ge $fileLen && lastL="" && totShown=0
while echo "$lastL" | grep -o "\"$l\"" > /dev/null
do
let "l++"
test $l -gt $fileLen && l=1
done
lastL="${lastL}\"$l\""
let "totShown++"

pfileL=`cat "$tmpFile"|head -n $l|tail -n 1`
pfilePath=`echo "$pfileL"|sed -e ‘s/\([^>]*\)>\([^>]*\)/\1\n\2/g’|head -n 1|tail -n 1|sed ‘s/|/ /g’`
pfileMode=`echo "$pfileL"|sed -e ‘s/\([^>]*\)>\([^>]*\)/\1\n\2/g’|head -n 2|tail -n 1`

if [ $totShown -ge $fileLen ]
then
lastL=""
totShown=0
fi

[ -f "$pfilePath" ] && dcop kdesktop KBackgroundIface setWallpaper "$pfilePath" "$pfileMode"

i=0
for shape in ${shapeDir[@]}
do
if test -a "$picDir/$shape" && ! find -L "$picDir/$shape" -type f -print | diff "$lstFile.$i" ->/dev/null
then
echo "’$picDir/$shape’ content changed"
refreshFileList "$picDir/$shape" ${shapeMode[$i]}
find -L "$picDir/$shape" -type f -print > "$lstFile.$i"
fi
let ‘i++’
done
echo "sleeping"
sleep "${changeWallIntv}m"
done<[/code]