CInk version 2 finally released!

Finally CInk version 2 has been released.

Some key new things

  • New website with complete API documentation and guides on how you can use CInk JS code.
  • Finally released the full source code of CInk renderer and compiler. License – GPL v3.
  • CInk finally supports all the features of original CFDG, including Paths.
  • CInk has introduced support for texts which extends the capabilities of CFDG considerably. Check out the cool demo – Neon Letters. To learn more about it see – “Text transforms” section here.

Last but not the least, you can post comments on CInk website. The comments section is at the bottom of each page.

Goto Cink website – cink.applegrew.com.

Create your own Cyberoam client in Python!

This is a very old post. Reposting it since while migrating it from blogger.com to here it got lost. Found it recently. Not sure how relevant or correct is this now.


Cyberthon updated to version 2.5 by Vinit and Siddhartha Sahu:-
Get the latest code from https://github.com/siddharthasahu/Cyberthon-enhanced.
Cyberthon updated to version 1.2:-
Added a new feature in this version. Now Cyberthon won’t exit ever, once started, even when it can’t connect to Cyberoam server. If liked its old behavior then set never_quit to False
Code fixed now. Sorry, I didn’t notice before (for 2 months since posting it here) that the code here was not working. I think it is fixed now. The problem was that the tabs were being expaded here differently. So, I just ran expand cyberthon.py command to expand all tabs in it beforehand.

My college uses the software Cyberoam for controlling the bandwidth and timing allocated to us for surfing the internet. If you don’t know what is it then visit here. So, to access the internet we all need to install the client software and login via that. But I don’t like to install it because it feels like a college spy on my computer, you may find it irrational but I can’t help being a paranoid. I have always used the web login page of Cyberoam but it is not a very good solution, first because I will always need to keep the browser open. (The Cyberoam page refreshes at a specific time periodically. If it fails to do so then you get logged out.) Second problem is that on my Linux platform if I work from console mode then I can’t login because no browser can run without GUI support (I know of Lynx but I haven’t been able to use it for it.)

Recently I thought of investigating the working of the web login page of Cyberoam. After bringing up the login page of Cyberoam I changed the value of attribute method of form tag to GET from POST. So after filling-up the username and password when I hit enter key, there it was all the details that are sent from my browser to Cyberoam server in my browser address bar. I thought of simulating the POST action using a Python code. It required only a pinch of Python to do this. I tried the following Python code to simulate that and it worked, as it should. You too can try the following code and login to Cyberoam.

Code:

[code lang=”python” wraplines=”false”]
import urllib
urllib.urlopen("http://192.168.5.10:8090/corporate/servlet/CyberoamHTTPClient","mode=191&isAccessDenied=null&url=null&message=&username=YourUserName&password=YourPassword&saveinfo=saveinfo&login=Login")[/code]

Be sure to substitute YourUserName and YourPasswordfor your username and password respectively. The above will get you logged in but won’t keep you logged it because you need to re-post the login data at every (usually) 3 minutes. So, I got down to work again. Below is quick and dirty shell script to get around this problem. Note the above code has been distorted and compressed into single line and passed via piping to Python in script below.

Code:

[code lang=”bash” wraplines=”false”]
while [[ 1 == 1 ]]
do
printf "%s\n%s\n" "import urllib" "urllib.urlopen(\"http://192.168.5.10:8090/corporate/servlet/CyberoamHTTPClient\",\"mode=191&isAccessDenied=null&url=null&message=&username=YourUserName&password=YourPassword&saveinfo=saveinfo&login=Login\")" |python
sleep 3m
done[/code]

Again in the above code do replace YourUserName and YourPassword for your actual username and password respectively. Well now it was working great. 🙂 As usual I was again not satisfied because how will I know if I have logged in successfully or not? The only way for that was by parsing the returned page after logging-in for the message. I noticed that returned page’s source code contained…

[code lang=”html”][/code]

Please notice the part message. The value after that is the message I was seeking. So, all I needed is to parse this page for the value of attribute src of the tag frame. Also, note this very string also contains some more very useful values, viz. – loginstatus and liverequesttime. I noticed that value of loginstatus becomes true when logged in otherwise it remains false, and the value of liverequesttimegiave the time (in seconds) in which to re-post the login data.

The HTML parsing class in the Python script (which I wrote) below is based on the code from the great HTML parsing tutorial located here. Note the script uses the zenity command to display GUI dialog boxes. If you do not have this installed or simply don’t want to display GUI dialog boxes then start the script with -nogui argument. If you want the script to remain completely silent then use the -silent argument. I call this script Cyberthon (=Cyberoam+Python) 😉

Requirement(s):-
1) Python 2.5 (may work with previous versions, but not tested)
2) Zenity (needed to show the dialog boxes, use -nogui switch if you want the messages to appear on the console.)
Code:

[code lang=”python” wraplines=”false”]#!/usr/bin/python
#!/usr/bin/python

#Program Name: Cyberthon (Python Cyberoam Client)
#Coder AppleGrew
#License GPL
#Version 1.2
cyberroamIP = "192.168.5.10" #The IP of the Cyberoam site.
cyberroamPort = "8090" #Set to "" if not using.
username = "your_username" #Your username
passwordFile = "/home/you/.passwd" #Path file containing a single string, your password.
sleeptime = 0 #in minutes or set to 0, it will then parse this value from the cyberoam returned page dynamically.
never_quit = True #Once started cyberthon will never, even when the cyberoam server cannot be connected.

import sys

silent = False
nogui = False
for arg in sys.argv:
if "-silent" == arg:
silent = True
if "-nogui" == arg:
nogui = True

#Parsing and logging in too.
import sgmllib

class MyCyberroamParser(sgmllib.SGMLParser):
"A simple parser class."

def parse(self, s):
"Parse the given string ‘s’."
self.feed(s)
self.close()

def __init__(self, verbose=0):
"Initialise an object, passing ‘verbose’ to the superclass."

sgmllib.SGMLParser.__init__(self, verbose)
self.required_entities = [‘message’,’loginstatus’,’liverequesttime’]
self.frames_attr = []
self.in_required_entity = False
self.current_entity = ""
self.entity_values = {}

def do_frame(self, attributes):
for name, value in attributes:
if name == "src":
self.frames_attr.append(value)

def unknown_entityref(self,ref):
self.current_entity = ref
if ref in self.required_entities:
self.in_required_entity=True

def handle_data(self, data):
"Try to get the value of entity &message. Used in 2nd pass of parsing."

if self.in_required_entity:
self.entity_values[self.current_entity] = data[1:] #To remove the preceeding =
self.in_required_entity = False

def get_src(self,index=-1):
"Return the list of src targets."
if index == -1:
return self.frames_attr
else:
return self.frames_attr[index]

import urllib, sgmllib,time,commands,os

pf = open(passwordFile)
passwd = pf.readline()
pf.close()
if passwd[-1] == ‘\n’: #Removing terminating newline character.
passwd = passwd[:-1]

cyberroamAddress = cyberroamIP
if cyberroamPort != "":
cyberroamAddress = cyberroamAddress+":"+cyberroamPort

sec2sleep = 60*sleeptime
lastmsg = ""
msgChanged = True
lastMsgWasFailMsg = False
sec2sleepOnError = 6
while True:
try:
# Logging in and fetching the Cyberroam login page.
f = urllib.urlopen("http://"+cyberroamAddress+"/corporate/servlet/CyberoamHTTPClient","mode=191&isAccessDenied=null&url=null&message=&username="+username+"&password="+passwd+"&saveinfo=saveinfo&login=Login")
sec2sleepOnError = 6
except IOError, (errno, strerror):
if not silent:
print "Connection to Cyberoam server timed out. Error(%s): %s" % (errno, strerror)

if sec2sleepOnError > 30:
if not silent:
if nogui:
print "Quitting program."
else:
if never_quit:
if not lastMsgWasFailMsg:
os.popen(‘zenity –info –text="Failed to connect to server, but I am NOT quitting." –title="Cyberthon" >/dev/null’)
lastMsgWasFailMsg = True
else:
commands.getoutput(‘zenity –info –text="Could not connect to the server. Quitting program." –title="Cyberthon"’)
if not never_quit:
sys.exit(1)
else:
sec2sleepOnError = 6

if not silent:
print "Retrying in %s seconds" % sec2sleepOnError

time.sleep(sec2sleepOnError)
sec2sleepOnError = sec2sleepOnError*2
continue

s = f.read()

# Try and process the page.
# The class should have been defined first, remember.
myparser = MyCyberroamParser()
myparser.parse(s)

# Get the the src targets. It contains the status message. And then parse it again for entity &message.
qindex = myparser.get_src(1).index(‘?’)
srcstr = myparser.get_src(1)[:qindex+1]+’&’+myparser.get_src(1)[qindex+1:]

myparser.parse(srcstr)

message = myparser.entity_values[‘message’]
if lastmsg != message or lastMsgWasFailMsg:
lastmsg = message
msgChanged = True
lastMsgWasFailMsg = False

if (not silent) and msgChanged:
msgChanged = False

msg=”
i=0
while i < len(message): #Converting hex nos. to characters. t=message[i] if message[i]==’%’: no=int(message[i+1:i+3],16) t=chr(no) i=i+2 msg=msg+t i=i+1 message = "" for x in msg:#Changing all + to space. if x == ‘+’: x = " " message=message+x if nogui: print message else: os.popen(‘zenity –info –text="From Cyberoam: ‘+message+’" –title="Cyberthon" >/dev/null’)

if myparser.entity_values[‘loginstatus’].lower()!="true":
break;

if sleeptime==0:
sec2sleep = int(myparser.entity_values[‘liverequesttime’])
time.sleep(sec2sleep)
[/code]

Please copy the file very carefully. A single extra space here and there may throw Python haywire, making it throw up all sorts of nasty syntax errors. Paste the contents of this code in a file and name it – Cyberthon.py.

Migrate from Apache to Nginx: The new guide

`AG_BACKQUOTE_TURN_ON`

If you are here then most probably, like me you too want to migrate from Apache to Nginx. Well I have already migrated and I am loving it! You can get a quick recap of Apache vs Nginx comparison here. Via this blog post I will share and explain my Nginx conf, in the hope that this can prove helpful for you. I have assumed that you have Nginx and Php-FPM installed. You can read how to install Ngnix from here – http://goo.gl/tq6vT. I installed it from its source. You can install PHP-FPM as per the guide at – http://goo.gl/Fx9hJ. My repo had PHP-FPM so I was saved from this trouble.

The net is littered with loads of blog posts on this topic, but most of them are either out of date or make contradictory suggestions. I have scoured the net and cooked up my own Nginx config based on many helpful blog posts on them. My config has been progressively tweaked as per my needs. I now feel that it is good enough to be shared.

To see if my Nginx config suites your needs, you first need to understand my use case. I have four domains hosted on the same server – applegrew.com, cink.applegrew.com, fb.applegrew.com and this blog.

  • applegrew.com – This serves some static HTML, few XMLs and dynamic web pages (PHP).
  • cink.applegrew.com – This is my Chrome Experiment site and has no dynamic web pages, only static files like HTMLs, images, text files, etc.
  • fb.applegrew.com – I added this one recently to host my Facebook Apps. So, naturally this has dynamic web pages, coded in PHP. Since, FB has mandated that from 1 Oct, 2011, all FB Apps must be accessible via HTTPS, so, this domain is configured to be accessible via both HTTP and HTTPS. The Nginx config for this takes care of setting the PHP parameter `$_SERVER[‘HTTPS’]` when HTTPS is used.
  • blog.applegrew.com– Configuring Nginx for this blog was no easy task. This blog is powered by WordPress. If you too have a WordPress blog then you MUST install the following WP plugins for performance. The Nginx config that I have shared assumes that these plugins are installed and takes full advantage of it.

    Must have WordPress plugins:-

    • WP Super Cache– Excellent plugin which generates static HTML files for your blog. Nobody updates their blog every minute. It’s not a Twitter. So, why generate that same page again and again for every user who visits you blog? The solution is to cache the generate page. Later when any user visits your blog then that user will be served the cached page. This saves a ton of overhead. Particularly when you are using Nginx, since, we run PHP and Nginx processes separately. So we can configure Nginx to serve the generated file, if present, and completely bypass PHP. This plugin is smart enough to refresh the cache when you make a new post or update it.

      Tip: Install this plugin after you have finalized your sites design, else you will have to manually clean the cache to make the site changes available.

    • WP Minify– This plugin strips out all the JS and CSS links from your blog and then combines them generate a unified CSS and JS. The result is cut down on the number of requests to your server for additional CSS and JS files. This plugin also minifies the combined CSS and JS files, which produces a much smaller file.

      Tip: If you install a new plugin after you install this one, and if that is not working, then try clearing the cache of this plugin. Since, it is possible that the new plugin will try to put some new CSS or JS which might get stripped out but not cached in the combined file.

Now its time for the configs.


nginx.conf

[code lang=”cink”]
user apache apache; #The uid and gid of the nginx process
worker_processes 4; #Number of worker processes that needs to be created.

error_log /var/log/error-n.log;

pid /usr/local/nginx/logs/nginx.pid;

events {
worker_connections 1000;
}

http {
include mime.types; #Includes a config file which is available with ngix’s default installation.
index index.html index.htm index.php index.shtml;

log_format main ‘$remote_addr – $remote_user [$time_local] "$request" ‘
‘$status $body_bytes_sent "$http_referer" ‘
‘"$http_user_agent" "$http_x_forwarded_for"’;

sendfile on;
keepalive_timeout 5;

gzip on;

# Sets the default type to text/html so that gzipped content is served
# as html, instead of raw uninterpreted data.
#default_type text/html;

server {#If someone tries to access the url http://applegrew.com/xxx then
#this will redirect him to http://www.applegrew.com/xxx.
server_name applegrew.com;
rewrite ^ http://www.applegrew.com$request_uri? permanent;
}

server {#The config for www.applegrew.com
server_name www.applegrew.com;

access_log /var/www/applegrew.com/access-n.log main; #Where access log will be written for this domain.
error_log /var/www/applegrew.com/error-n.log;

root /var/www/applegrew.com/html; #The document root for this domain.

location ~ /admin/ { deny all; } #Denies access to some www.applegrew.com/admin/ url
location ~ /private/ { deny all; }

include cacheCommon.conf; #This caches common static files. This config is given later in this post.
include drop.conf; #This config is given later in this post.
include php.conf; #Configures PHP access for this domain. This config is given later in this post.
include err.conf; #Some common custom error messages I show. This config is given later in this post.
}

server {#Config to serve HTTP traffic.
server_name fb.applegrew.com;

access_log /var/www/fb.applegrew.com/access.log main;
error_log /var/www/fb.applegrew.com/error.log;

root /var/www/fb.applegrew.com/html;

include cacheCommon.conf;
include php.conf;
include drop.conf;
include err.conf;
}

server {//Config to serve HTTPS traffic.
listen 443;
server_name fb.applegrew.com;

ssl on;
ssl_certificate /var/ssl/fb.applegrew.com.crt; #See http://goo.gl/mvHo7 to know how to create crt file.
ssl_certificate_key /var/ssl/fb_applegrew_com.key;

access_log /var/www/fb.applegrew.com/access.log main;
error_log /var/www/fb.applegrew.com/error.log;

root /var/www/fb.applegrew.com/html;

include cacheCommon.conf;
include phpssl.conf; #Notice the difference. This is not php.conf. This config will be provided later in this post.
include drop.conf;
include err.conf;
}

server {
server_name blog.applegrew.com;

access_log /var/www/blog.applegrew.com/access-n.log main;
error_log /var/www/blog.applegrew.com/error-n.log;

root /var/www/blog.applegrew.com/html;

#If tgz file mathcing the request already exists then that will be sent, skipping on the fly compression by nginx.
gzip_static on;

location / {
# does the requested file exist exactly as it is? if yes, serve it and stop here
if (-f $request_filename) { break; }

# sets some variables to help test for the existence of a cached copy of the request
set $supercache_file ”;
set $supercache_uri $request_uri;

# IF the request is a post, has a query attached, or a cookie
# then don’t serve the cache (ie: users logged in, or posting comments)
if ($request_method = POST) { set $supercache_uri ”; }
if ($query_string) { set $supercache_uri ”; }
if ($http_cookie ~* "comment_author_|wordpress|wp-postpass_" ) {
set $supercache_uri ”;
}

# if the supercache_uri variable hasn’t been blanked by this point, attempt
# to set the name of the destination to the possible cache file
if ($supercache_uri ~ ^(.+)$) {
set $supercache_file /wp-content/cache/supercache/$http_host/$1index.html;
}

# If a cache file of that name exists, serve it directly
if (-f $document_root$supercache_file) { rewrite ^ $supercache_file break; }

# Otherwise send the request back to index.php for further processing
if (!-e $request_filename) { rewrite . /index.php last; }

#try_files $uri $uri/ /index.php;
}
location ~ /wp-config\.php { deny all; }
location ~ /wp-content/bte-wb/.*\..* { deny all; }

include cacheCommon.conf;
include drop.conf;
include php.conf;
include err.conf;

#Let wordpress show its own error pages.
fastcgi_intercept_errors off;
}

server {
server_name cink.applegrew.com;

access_log /var/www/cink.applegrew.com/access-n.log main;
error_log /var/www/cink.applegrew.com/error-n.log;

root /var/www/cink.applegrew.com/html;

include cacheCommon.conf;
include drop.conf;
include err.conf;
}

server {#If none the above matched then maybe the url was accessed, (say) via the IP directly. We then show applegrew.com.
listen 80 default;
server_name _;
access_log /var/www/applegrew.com/access.log-n main;

server_name_in_redirect off;

rewrite ^ http://www.applegrew.com$request_uri? permanent;
include err.conf;
}
}
[/code]


cacheCommon.conf

[code lang=”cink”]
#Asks browsers to cache files with extension ico, css, gif, jpg, jpeg, png, txt and xml.
location ~* \.(?:ico|css|js|gif|jpe?g|png|txt|xml)$ {
# Some basic cache-control for static files to be sent to the browser
expires max;
add_header Pragma public;
add_header Cache-Control "public, must-revalidate, proxy-revalidate";
}
[/code]


drop.conf

[code lang=”cink”]
location = /favicon.ico { access_log off; log_not_found off; } #Don’t log this.
location ~ /\. { access_log off; log_not_found off; deny all; } #Block . (dot) files access
#Don’t log and deny access to files which end with ~, as these are usually backup files.
location ~ ~$ { access_log off; log_not_found off; deny all; }
[/code]


err.conf

[code lang=”cink”]
error_page 500 502 503 504 /50x.html;
error_page 403 404 /404.html; # Yes for 403 too we show 404 error, just to mislead.
location = /50x.html {
root /home/webadmin/err/;
}
location = /404.html {
root /home/webadmin/err/;
}
[/code]


php.conf

[code lang=”cink”]
location ~ \.php { #All requests that end with .php are directed to PHP process.
include phpparams.conf; #This file is described later in this post.
}
[/code]


phpssl.conf

[code lang=”cink”]
location ~ \.php {#This the same as php.conf but adds few ssl specific configs.
fastcgi_param HTTPS on; #This sets $_SERVER[‘HTTPS’] to ‘on’.
fastcgi_param SSL_PROTOCOL $ssl_protocol; #This sets the $_SERVER[‘SSL_PROTOCOL’].
fastcgi_param SSL_CIPHER $ssl_cipher; #This sets the $_SERVER[‘SSL_CIPHER’].
fastcgi_param SSL_SESSION_ID $ssl_session_id; #This sets the $_SERVER[‘SSL_SESSION_ID’].
fastcgi_param SSL_CLIENT_VERIFY $ssl_client_verify; #This sets the $_SERVER[‘SSL_CLIENT_VERIFY’].

include phpparams.conf;
}
[/code]
We need to set the `$_SESSION` ourselves since unlike mod_php (in Apache), Php-Fpm is not embedded in Nginx and it doesn’t have these information available to it unless we set it. The above config doesn’t set all the flags a script might expect, but the usual ones. If need to set some more then go to Nginx HttpSslModule’s Built-in variables section.


phpparams.conf

[code lang=”cink”]
#PHP FastCGI
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;

fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param REQUEST_URI $request_uri;
fastcgi_param DOCUMENT_URI $document_uri;
fastcgi_param DOCUMENT_ROOT $document_root;
fastcgi_param SERVER_PROTOCOL $server_protocol;

fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx;

fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;

fastcgi_connect_timeout 60;
fastcgi_send_timeout 180;
fastcgi_read_timeout 180;
fastcgi_buffer_size 128k;
fastcgi_buffers 4 256k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;

fastcgi_pass unix:/usr/local/nginx/logs/php5-fpm.sock; #I have configured both Php-Fpm and Nginx to communicate via file sockets.
[/code]


/etc/php-fpm.conf

[code]
include=/etc/php-fpm.d/*.conf
pid = /var/run/php-fpm/php-fpm.pid
error_log = /var/log/php-fpm/error.log
log_level = error
[/code]


/etc/php-fpm.d/www.conf

[code]
[www]
listen = /usr/local/nginx/logs/php5-fpm.sock
listen.allowed_clients = 127.0.0.1
user = apache
group = apache

pm = dynamic
pm.max_children = 6; #This can be increased on 512MB RAM. For 256MB you ca use 2.
pm.start_servers = 3; #This can be increased. For 256MB you can use 1.
pm.min_spare_servers = 3; #This can be increased. For 256MB you can use 1.
pm.max_spare_servers = 5; #This can be increased. For 256MB you can use 1.
pm.max_requests = 500
slowlog = /var/log/php-fpm/www-slow.log
php_admin_value[error_log] = /var/log/php-fpm/www-error.log #All PHP errors will go into this.
php_admin_flag[log_errors] = on
[/code]
Note: The settings above are indicative. You need to experiment with different settings on your system. I have a MySql DB too running on the same system. In my case the minimal settings for 256MB RAM too cause problem. The PHP process used to choke after 4-5 days of running. So, finally I was forced to increase server RAM to 512MB.


/etc/init.d/nginxd

I wrote this shell script to start stop nginx as a service on my CentOS server.
[code lang=”bash”]
#!/bin/bash
# chkconfig: 235 85 15
# description: The Nginx Server is an efficient and extensible \
# server implementing the current HTTP standards.

cmd=/usr/local/nginx/sbin/nginx #Change this to match your Nginx installation path.
start() {
pgrep ‘nginx$’ > /dev/null
if (( $? != 0 ))
then
echo ‘Staring nginx’
$cmd
RETVAL=$?
if (( $RETVAL == 0 ))
then
echo ‘Started successfully’
fi
else
echo ‘Nginx already running’
RETVAL=0
fi
}

RETVAL=0
case "$1" in
start)
start
;;
stop)
echo ‘Shutting down Nginx quickly’
$cmd -s stop
RETVAL=$?
;;
quit)
echo ‘Gracefully shutting down Nginx’
$cmd -s quit
RETVAL=$?
;;
restart)
echo ‘Stopping Nginx’
$cmd -s stop
start
;;
reload)
echo ‘Reloading cofig’
$cmd -s reload
RETVAL=$?
;;
reopen)
echo ‘Reopening log files’
$cmd -s reopen
RETVAL=$?
;;
help)
$cmd -?
RETVAL=$?
;;
test)
echo ‘Test config’
$cmd -t
RETVAL=$?
;;
*)
echo $"Usage: nginx {start|stop|quit|restart|reload|reopen|help|test}"
echo "stop – quick shutdown"
echo "quit – graceful shutdown"
echo "reload – close workers, load config, start new workers"
echo "reopen – reopen log files"
echo "test – only tests the config"
RETVAL=3
esac

exit $RETVAL
[/code]
You can install the above by copying the nginxd file to /etc/init.d then run
`sudo /sbin/chkconfig nginxd –add`
`sudo /sbin/chkconfig nginxd on`

You can the give commands to the script by
`sudo /sbin/service nginxd command here`


Well, I hope this post been helpful.