Posts Tagged ‘Linux’

常用命令

March 4th, 2014

Linux

  • find /home -iname “*.log” -exec rm {} \;

    Delete file recursively by extension.

Windows

linux batch rename files

October 28th, 2011

Use this command to rename a set of files:

for file in * ; do mv $file `echo $file | sed 's/OLD_STRRING/NEW_STRING/'` ; done

And here is a batch file:
——————
batch_rename.sh
——————

#!/bin/bash

if [ $# -ne 3 ]; then
        echo "Usage: $0 'FilePattern' SearchString ReplaceString";
        echo "Eg: $0 'r-*.sh' 5 6";
else
        for file in $1 ; do mv $file `echo $file | sed s/$2/$3/` ; done
fi

If the 2nd or 3rd parameter contains space, use ‘ ‘ to surround the parameter.

mysqlpp connection timeout problem

October 3rd, 2011

If you have a application having long time connection with mysql using mysqlpp. You may at first think of using Connection::connected() to determine whether the connection is still valid. But actually this doesn’t work if the connection gets timeout.

After checking the reference, I found out Connection::ping() would be a good way to determine timeout.

Here is a piece of test code:

#include <iostream>
#include <string>
#include <mysql++.h>

using namespace std;

int main() {
	string db_schema("DB_SCHEMA");
	string db_host("DB_HOST");
	string db_user("DB_USER");
	string db_pass("DB_PASS");
	string qry_str("SELECT MAX(user_id) FROM users");

	mysqlpp::Connection cn;
	cn.connect(db_schema.c_str(), db_host.c_str(), db_user.c_str(), db_pass.c_str());
	mysqlpp::Query qry = cn.query(qry_str);
	mysqlpp::StoreQueryResult res = qry.store();
	cout << "frist query result: " << res[0][0] << endl;

	string tmp;
	cout << "Wait for a period of time to let the connection timeout, and then input whatever a string to continue: " << endl;
	cin >> tmp;

	cout << "ping result: " << cn.ping() << endl;
	cout << "connected: " << cn.connected() << endl;
	
	if(!cn.ping())
		cn.connect(db_schema.c_str(), db_host.c_str(), db_user.c_str(), db_pass.c_str());
	mysqlpp::Query qry2 = cn.query(qry_str);
	res = qry2.store();
	cout << "second query result: " << res[0][0] << endl;

	cn.disconnect();
	
	return 0;	
}

But of course, I’m still new to mysqlpp. There may be a better way to do this. Please comment below if you have a better way.

NOTE: Here is how to change mysql connection idle timeout.

UPDATE: Just found another way to handle the timeout problem: use Connection::set_option(new mysqlpp::ReconnectOption(true));. This is much more elegant.
Here is a piece of sample code:

mysqlpp::Connection cn;
cn.set_option(new mysqlpp::ReconnectOption(true));
cn.connect(db_schema.c_str(), db_host.c_str(), db_user.c_str(), db_pass.c_str());

bash script for killing all mysql processes / connections

September 29th, 2011

Here is a small bash script for killing all mysql processes (or connections).

The basic idea is to select all process IDs into a tmp file (each process ID a line), and then read the file line by line and kill the process, finally remove the tmp file.

NOTE:
This bash only works for MYSQL >= 5.1.7. Becuase before 5.1.7 process list is not stored in the table INFORMATION_SCHEMA.PROCESSLIST.

USAGE:

  • either by providing the MYSQL root password as a parameter to the bash as:
    user> mysql_kill_all.sh MysqlRootPass
  • or by editing the password the line mysqlpass=$1 as:
    mysqlpass=MysqlRootPass and then excute the bash without parameter:
    user> mysql_kill_all.sh.

    Of course, you can also edit the other variables in the bash file.

——————————-
SOURCE FILE: mysql_kill_all.sh
——————————-

#!/bin/bash

# mysql host/user/pass who has the privilege to read schema INFORMATION_SCHEMA
mysqlhost=localhost
mysqluser=root
mysqlpass=$1
# tmp file path
tmpprocesslistfile=/tmp/mysql_processlist.txt

# select all process IDs into the tmp file. each line has a process id.
echo "SELECT id FROM INFORMATION_SCHEMA.PROCESSLIST INTO OUTFILE '$tmpprocesslistfile';" | mysql -h$mysqlhost -u$mysqluser -p$mysqlpass INFORMATION_SCHEMA

# read the file line by line and kill the process
while read qryid
do
echo "kill $qryid;" | mysql -h$mysqlhost -u$mysqluser -p$mysqlpass
done < "$tmpprocesslistfile"

# remove the tmp file
echo 'All connections have been killed.';
echo 'Please enter sudo password to remove the tmp file if necessary.';
sudo rm $tmpprocesslistfile

System V message queue example (msgget, msgctl, msgsnd, msgrcv)

September 21st, 2011

Found a very good example of message queue showing how to use msgget, msgctl, msgsnd, and msgrcv at http://www.dps.uibk.ac.at/~tf/lehre/ss04old/bs/tutorials/prozesse-syscalls/29.htm. Here I list the code with tiny fix and some of my comments.

The linux man page of msgsnd & msgrcv can be found here: http://linux.die.net/man/2/msgrcv.

———————————————-
msg_queue.c
———————————————-

#include<string.h>
#include<time.h>
#include<sys/ipc.h>
#include<sys/msg.h>
#include<sys/wait.h>
#include<sys/errno.h>
     
extern int errno;       // error NO.
#define MSGPERM 0600    // msg queue permission
#define MSGTXTLEN 128   // msg text length

int msgqid, rc;
int done;

struct msg_buf {
  long mtype;
  char mtext[MSGTXTLEN];
} msg;

int main(int argc,char **argv)
{
  // create a message queue. If here you get a invalid msgid and use it in msgsnd() or msgrcg(), an Invalid Argument error will be returned.
  msgqid = msgget(IPC_PRIVATE, MSGPERM|IPC_CREAT|IPC_EXCL);
  if (msgqid < 0) {
    perror(strerror(errno));
    printf("failed to create message queue with msgqid = %d\n", msgqid);
    return 1;
  }
  printf("message queue %d created\n",msgqid);
  
  // message to send
  msg.mtype = 1; // set the type of message
  sprintf (msg.mtext, "%s\n", "a text msg..."); /* setting the right time format by means of ctime() */

  // send the message to queue
  rc = msgsnd(msgqid, &msg, sizeof(msg.mtext), 0); // the last param can be: 0, IPC_NOWAIT, MSG_NOERROR, or IPC_NOWAIT|MSG_NOERROR.
  if (rc < 0) {
    perror( strerror(errno) );
    printf("msgsnd failed, rc = %d\n", rc);
    return 1;
  }

  // read the message from queue
  rc = msgrcv(msgqid, &msg, sizeof(msg.mtext), 0, 0); 
  if (rc < 0) {
    perror( strerror(errno) );
    printf("msgrcv failed, rc=%d\n", rc);
    return 1;
  } 
  printf("received msg: %s\n", msg.mtext);

  // remove the queue
  rc=msgctl(msgqid,IPC_RMID,NULL);
  if (rc < 0) {
    perror( strerror(errno) );
    printf("msgctl (return queue) failed, rc=%d\n", rc);
    return 1;
  }
  printf("message queue %d is gone\n",msgqid);

  return 0;
}

————– Update 14/Dec/2011 ——————
A note about the struct msg_buf. From: “The Linux Programmer’s Guide” http://tldp.org/LDP/lpg/node30.html.

The ability to assign a given message a type, essentially gives you the capability to multiplex messages on a single queue. For instance, client processes could be assigned a magic number, which could be used as the message type for messages sent from a server process. The server itself could use some other number, which clients could use to send messages to it. In another scenario, an application could mark error messages as having a message type of 1, request messages could be type 2, etc. The possibilities are endless.

On another note, do not be misled by the almost too-descriptive name assigned to the message data element (mtext). This field is not restricted to holding only arrays of characters, but any data, in any form. The field itself is actually completely arbitrary, since this structure gets redefined by the application programmer. Consider this redefinition:


struct my_msgbuf {
long mtype; /* Message type */
long request_id; /* Request identifier */
struct client info; /* Client information structure */
};

Here we see the message type, as before, but the remainder of the structure has been replaced by two other elements, one of which is another structure! This is the beauty of message queues. The kernel makes no translations of data whatsoever. Any information can be sent.

command line network tools on linux

July 29th, 2011

netstat

  • netstat -tulpan:
  • netstat -tl
  • netstat -anp –tcp –udp: will show all open tcp and udp connections.

lsof

  • lsof -i
  • lsof -i | grep 80
  • lsof -i | grep apache2

iftop

display bandwidth usage on an interface by host

Install & use mysql++ on linux(debian squeeze, ubuntu)

July 28th, 2011

The key to use mysql++ on debian/ubuntu system is to find where the mysql++ .h head files and the .so binary files.

Firrst install mysqlclient and mysql++ on debian squeeze:

root# apt-get install libmysqlclient-dev libmysqld-dev libmysql++-dev libmysql++-doc libmysql++3

This will install the mysql head files in /usr/include/mysql, mysql++ head files in /usr/include/mysql, and libmysqlpp.so and libmysqlclient.so in /usr/lib. These are all the files we need.

Now let us use the mysql++ library.

Save the simple code below as test.cpp

#include <iostream>
#include <mysql++.h>

int main() {

mysqlpp::String s(“hello, world”);

std::cout << s << std::endl;

return 0;

}

And save the makefile as Makefile:

CXX := g++
CXXFLAGS := -I/usr/include/mysql -I/usr/include/mysql++

LDFLAGS := -L/usr/lib -lmysqlpp -lmysqlclient -lnsl -lz -lm
EXECUTABLE := main

all: test
clean:        rm -f $(EXECUTABLE) *.o

Finally, excute

make all

The compile result would be test.

compress & decompress files under Linux

February 24th, 2011

.zip

compress:

zip -r xxx.zip dir

decompress:

unzip xxx.zip

.rar

compress:

rar a xxx.rar dir

decompress:

unrar e xxx.rar

.bz2

compress:

bzip2 file

decompress:

bunzip2 xxx.bz2 (NOTE: use the switch -k to keep the .bz2 file)

.gz

compress:

gzip file (NOTE: -r, --recursive   operate recursively on directories)

decompress:

gunzip xxx.gz

tar.bz2

compress:

tar -cjf xxx.tar.bz2 dir

decompress:

tar -jxvf xxx.tar.bz2

tar.gz

compress:

tar -czf xxx.tar.gz dir

decompress:

tar -zxvf xxx.tar.gz

set up postfix on debian

July 13th, 2010

Suppose you have 2 sites: aaa.com & bbb.com. And you would like to set aaa.com as a local domain, and bbb.com as a virtual domain. That means emails of user@aaa.com will go to the system user’s personal mail dir while user@bbb.com has no actual corresponding system user. Details about local domain & virtual domain, please see paragraph “Virtual domains versus local domains” at http://workaround.org/ispmail/etch.

part 1: configure local domain aaa.com

  • install postfix
    apt-get install postfix
  • config postfix
    add the domain to config files using psotconf:
    postconf -e "myorigin = aaa.com"

    add hostname:
    postconf -e "myhostname=aaa.com"
  • Reload Postfix Server:
    postfix reload

Now you can try to send a email via telnet.
telnet localhost 25
You should see:

Trying 127.0.0.1…
Connected to localhost.
Escape character is ‘^]’.
220 aaa.com ESMTP Postfix (Debian/GNU)

Now send a test mail to yourself:

mail from:<test@testsite.com>
rcpt to: <user@aaa.com>
data
To: user@aaa.com
From: test@testsite.com
Subject: First test mail to yourself
Damn it! It worked! Hahahahaha…
Bye…

End the input with an enter, then a dot “.”, and an enter again.
<enter>
.
<enter>

And then quit:
quit

Now use mail to check your email.

part 2: configure virtual domain bbb.com

  • create a user and take his home dir as the virtual server’s base dir.
    #> groupadd -g 5000 vmail
    #> useradd -g vmail -u 5000 vmail -d /home/vmail -m
  • set the virtual_uid_maps and virtual_gid_maps to these IDs:
    #> postconf -e virtual_uid_maps=static:5000
    #> postconf -e virtual_gid_maps=static:5000
  • add bbb.com as a virtual domain into virtual_mailbox_domains
    #> postconf -e virtual_mailbox_domains=bbb.com
  • set the base dir
    #> postconf -e virtual_mailbox_base=/var/spool/vmail
  • reload postfix
    #> postfix reload;
  • add virtual email addresses
    • tell which file the email addresses are stored in
      #> postconf -e virtual_mailbox_maps=hash:/etc/postfix/vmailhost
    • create the file and add some addresses in there.
      #> vi /etc/postfix/vmailhost
      with content like this:
      service@bbb.com bbb.com/service/
      webmaster@bbb.com bbb.com/webmaster/

      So, the mail files for service will actually be stored in /home/vmail/bbb.com/service, and mail files for webmaster will be in /home/vmail/bbb.com/webmaster
    • Now, map the file into DB
      #> postmap /etc/postfix/vmailhost
  • Probably you also need to create the dirs for these 2 addresses
    mkdir /home/vmail/bbb.com/service/bbc.com
    for service@bbb.com
    #> mkdir /home/vmail/bbb.com/service/bbc.com/service
    #> cd /home/vmail/bbb.com/service/bbc.com/service
    #> mkdir new curr tmp
    And do the same for webmaster@bbb.com
  • change the owner and permissions if you created this folders manually and with ROOT privilege:
    #> chown -R vmail:vmail /home/vmail/bbb.com/
    #> chmod -R 700 /home/vmail/bbb.com/

OK, now you can send mail via telnet to have a test.

part 3: configure alias for service@bbb.com

If you want to forward received mails of service@bbb.com, you can use a alias to forward to your another email address, let’s say service.bbb@gmail.com.

  • tell where is the alias configuration file
    #> postconf -e "alias_maps = hash:/etc/postfix/aliases"
    postfix reload;
  • add forward rules in the file:
    #> vi /etc/postfix/aliases
    with the conetent:
    service@bbb.com service.bbb@gmail.com
    and map it:
    #> postmap /etc/postfix/aliases

Note

  • Every configuration done by postconf, you can actually do it by editing /etc/postfix/main.cf directly.
  • Every time you change mail.cf with command postmap, do not forget to reload postfix with: #> postfix reload.
  • Every time you change a db file, do not forget to re-postmap it with: #> postmap /etc/postfix/xxxxx.
  • If you want to set both sites into virtual server, only set localhost as your destination and both sites into virtual domain.

Important debug method

First, all check the log which should be /var/log/mail.log, no matter you want to find the cause of the failure or to check that your email is sent/received successfully.

Second, use mutt to try to send/receive emails

Important references

domain

———————– v1.1: 27/08/2010 ————————

How to add SASL auth

atually a complete instruction for install postfix on debian
official postfix manual

———————– v1.2: 27/08/2010 ————————

How to set up SPF

http://www.howtoforge.com/postfix_spf

———————– v1.3: 03/09/2010 ————————

How to set up domain keys & dkim

Setting up domain keys with dkfilter:

Note: as the dkfilter author has said, dkfilter is deprecated now and DKIMproxy is recommended.

Follow this link : http://bliki.rimuhosting.com/space/knowledgebase/linux/mail/dkfilter+setup except the settings in file /etc/postfix/master.cf should be:

pickup    fifo  n       -       n       60      1       pickup
    -o content_filter=dksign:[127.0.0.1]:10027

And, you may also need to add these as said in http://jason.long.name/dkfilter/:

#
# modify the default submission service to specify a content filter
# and restrict it to local clients and SASL authenticated clients only
#
submission  inet  n     -       n       -       -       smtpd
    -o smtpd_etrn_restrictions=reject
    -o smtpd_sasl_auth_enable=yes
    -o content_filter=dksign:[127.0.0.1]:10027
    -o receive_override_options=no_address_mappings
    -o smtpd_recipient_restrictions=permit_mynetworks,permit_sasl_authenticated,reject

#
# specify the location of the DomainKeys signing filter
#
dksign    unix  -       -       n       -       10      smtp
    -o smtp_send_xforward_command=yes
    -o smtp_discard_ehlo_keywords=8bitmime

#
# service for accepting messages FROM the DomainKeys signing filter
#
127.0.0.1:10028 inet  n  -      n       -       10      smtpd
    -o content_filter=
    -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks
    -o smtpd_helo_restrictions=
    -o smtpd_client_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o mynetworks=127.0.0.0/8
    -o smtpd_authorized_xforward_hosts=127.0.0.0/8

How to generate key:

perl -MMIME::Base64 -e 'print encode_base64("cihangcihangpassword");'

How to test:

telnet localhost 25
ehlo localhost
auth plain Y2loYW5nAGNpaGFuZwB4TDM4MjVNdjZ1

Setting up Domain Keys and/or DKIM with DKIMproxy:

didn’t try. see here: Mail-DKIM and DKIMproxy.

setting java environment under ubuntu

May 20th, 2010

you can set all the environment vars in /etc/environment.

for example:

PATH=”/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games”
JAVA_HOME=”/usr/lib/jvm/java-6-sun/jre”
CLASSPATH=”.:$JAVA_HOME/lib:/home/xxxxxx/other-packages/lucene-3.0.1″

UPDATE:

use bash> source /etc/environment to refresh the system variables.