Archive

Archive for November, 2012

join – file 1 is not in sorted order (Error)

November 30, 2012 Leave a comment

Problem:
I had two sorted files that I needed to join but join kept on giving me the following error
join: file 1 is not in sorted order.

user@computer:~$less file1
1,0
2,0
3,0
4,0
5,0
6,0
7,0
8,0
9,0
10,0
user@computer:~$less file2
1,1701
2,59
3,6
4,1
5,1
user@computer:~$join -t"," -1 1 -2 1 -a 1 -e'0' -o '1.1,2.2' file1 file2
1,1701
2,59
3,6
4,1
5,1
6,0
7,0
8,0
9,0
join: file 1 is not in sorted order
10,0

Solution:
join –nocheck-order

user@computer:~$join --nocheck-order -t"," -1 1 -2 1 -a 1 -e'0' -o '1.1,2.2' file1 file2
1,1701
2,59
3,6
4,1
5,1
6,0
7,0
8,0
9,0
10,0

Source:
info join

NB:
This is the version of join I was using. The –nocheck-order option may not be available on earlier versions. You might not even get the error with older versions.

join (GNU coreutils) 8.5
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later .
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Haertel.

Categories: bash

sed – Convert to upper case

November 28, 2012 1 comment

Problem:
I need to covert text to UPPER CASE using sed.

Solution:

user@computer:~$ echo "first,second,third" |\
sed 's/\([a-z]\+\),/\U\1,/'
FIRST,second,third

Source:
http://nixcraft.com/shell-scripting/15862-sed-convert-text-lower-upper-case.html

Categories: bash, sed

How to package Your Python Code

November 24, 2012 Leave a comment

Interesting read from python weekly newsletter.

How To Package Your Python Code
This tutorial aims to put forth an opinionated and specific pattern to make trouble-free packages for community use. It doesn’t describe the only way of doing things, merely one specific approach that works well.

Source:
http://www.pythonweekly.com/

Categories: Interesting, python

bash – Mimic MySQL GROUP BY function with awk

November 20, 2012 1 comment

Problem:
I needed to get GROUP BY results and I did not have the time to wait to restore the mysqldump and run the commands. I had like over 360 zipped files to work with. Restoring each database and running the query even in a script was just too long a process. I opted to just extract the data and use awk to mimic GROUP BY.

Solution:
I used sed to extract the desired results from the mysql dump files. Then took the relevant date fields and used awk to do the group count.

Here is the SELECT…GROUP BY using mysql

$ mysql -u <user> -p<password> 
-e"SELECT date_format(timestamp,'%Y-%m-%d') as 'Date',count(distinct(<field>)) FROM <table>
WHERE <search criteria>
GROUP BY date_format(timestamp,'%Y-%m-%d')
ORDER BY date_format(timestamp,'%Y-%m-%d')" | sed 's/\t/,/g'
2012-10-10,4
2012-10-11,21
2012-10-12,19
2012-10-13,8
2012-10-14,22
2012-10-15,7
2012-10-16,13

Here is the same but now using awk. After extracting the data from the dumps I end up with the following

$ head -n 3 <file-with-extracted-content>
2012-10-01 06:08:54|377772327281
2012-10-01 06:08:55|371202225977
2012-10-01 06:08:55|061712877272
$ 
$ awk -v FS=" |\|" '{print $1, $3}' <file-with-extracted-content> | 
sort -u |  
awk -v OFS=, '{a[$1]+=1}END{for (i in a) print i,a[i]}' | 
sort -t, -k1
2012-10-10,4
2012-10-11,21
2012-10-12,19
2012-10-13,8
2012-10-14,22
2012-10-15,7

Categories: awk, bash, MySQL

bash – Generating Random Passwords

November 13, 2012 Leave a comment

Problem:
Someone asked me to write them a script to generate random passwords based on words. They normally have to do this periodically for more than 10 accounts. The passwords are just random words separated by spaces.

Solution:
I used pwgen and /usr/share/dict/words

#! /bin/bash
# Password Generator - password_generator.sh
# Create simple word based passwords
# Usage - bash password_generator.sh [number of words]

no_of_words="$1"

# Get the first letter of each word
word=$(pwgen 30 | head)

echo $word | sed 's|\([a-z]\)|\1\n|g' | sort -u |\
sed '/^$/d' | sort -R | head -n ${no_of_words:=5} |\
while read letter
do
    #echo $letter
    # Pick 7 letter words from dictionary
    egrep  "^${letter}[a-z]{6}$" /usr/share/dict/words | sort -R | head -n 1
done > secret_message

sed ':mylbl;N;s/\n/ /;t mylbl' secret_message


Sample output

$ bash password_generator.sh 4
suckles veering hallway entails
$ 
$ bash password_generator.sh 4
humaner aptness ethnics privacy
$ 
$ bash password_generator.sh 4
staffer incised adjured orchids
$ 
$ bash password_generator.sh 4
abjured parsley insists boggled
$ 
$ bash password_generator.sh 2
exiling grouper
$ 
$ bash password_generator.sh 2
emetics impress
$ 
$ bash password_generator.sh 6
extorts prefect abstain legible upgrade tensest
$ 
$ bash password_generator.sh
yeshiva infuses gabbing aphelia kissing
$ 
$ bash password_generator.sh
prudery chapels alarmed hansoms skaters
$ 
$ bash password_generator.sh 8
horizon outgrow knocker zippers approve engrave cremate munched
$ 

Source:
Got some tips from here
http://www.linuxjournal.com/article/8919

Categories: bash