Archive for category Programming
Load Python in 32bit mode
Posted by hvera1981 in Programming on March 10, 2011
VERSIONER_PYTHON_PREFER_32_BIT=yes python
Convert Canon CR2 raw images to JPEG images
Posted by hvera1981 in Linux, Programming on March 10, 2011
#!/bin/bash for i in *.cr2; do dcraw -c -q 0 -w -H 5 -b 8 $i | cjpeg -quality 100 > $i.jpg; done Tip from http://snipt.net
The Goertzel Algorithm
Posted by hvera1981 in Programming on October 27, 2010
The Goertzel algorithm can perform tone detection using much less CPU horsepower than the Fast Fourier Transform, but many engineers have never heard of it. This article attempts to change that.
Most engineers are familiar with the Fast Fourier Transform (FFT) and would have little trouble using a “canned” FFT routine to detect one or more tones in an audio signal. What many don’t know, however, is that if you only need to detect a few frequencies, a much faster method is available. It’s called the Goertzel algorithm.
Tone detection
Many applications require tone detection, such as:
- DTMF (touch tone) decoding
- Call progress (dial tone, busy, and so on) decoding
- Frequency response measurements (sending a tone while simultaneously reading back the result)-if you do this for a range of frequencies, the resulting frequency response curve can be informative. For example, the frequency response curve of a telephone line tells you if any load coils (inductors) are present on that line.
Although dedicated ICs exist for the applications above, implementing these functions in software costs less. Unfortunately, many embedded systems don’t have the horsepower to perform continuous real-time FFTs. That’s where the Goertzel algorithm comes in.
In this article, I describe what I call a basic Goertzel and an optimized Goertzel.
The basic Goertzel gives you real and imaginary frequency components as a regular Discrete Fourier Transform (DFT) or FFT would. If you need them, magnitude and phase can then be computed from the real/imaginary pair.
The optimized Goertzel is even faster (and simpler) than the basic Goertzel, but doesn’t give you the real and imaginary frequency components. Instead, it gives you the relative magnitude squared. You can take the square root of this result to get the relative magnitude (if needed), but there’s no way to obtain the phase.
In this short article, I won’t try to explain the theoretical background of the algorithm. I do give some links at the end where you can find more detailed explanations. I can tell you that the algorithm works well, having used it in all of the tone detection applications previously listed (and others).
A basic Goertzel
First a quick overview of the algorithm: some intermediate processing is done in every sample. The actual tone detection occurs every Nth sample. (I’ll talk more about N in a minute.)
As with the FFT, you work with blocks of samples. However, that doesn’t mean you have to process the data in blocks. The numerical processing is short enough to be done in the very interrupt service routine (ISR) that is gathering the samples (if you’re getting an interrupt per sample). Or, if you’re getting buffers of samples, you can go ahead and process them a batch at a time.
Before you can do the actual Goertzel, you must do some preliminary calculations:
- Decide on the sampling rate.
- Choose the block size, N.
- Precompute one cosine and one sine term.
- Precompute one coefficient.
These can all be precomputed once and then hardcoded in your program, saving RAM and ROM space; or you can compute them on-the-fly.
Sampling rate
Your sampling rate may already be determined by the application. For example, in telecom applications, it’s common to use a sampling rate of 8kHz (8,000 samples per second). Alternatively, your analog-to-digital converter (or CODEC) may be running from an external clock or crystal over which you have no control.
If you can choose the sampling rate, the usual Nyquist rules apply: the sampling rate will have to be at least twice your highest frequency of interest. I say “at least” because if you are detecting multiple frequencies, it’s possible that an even higher sampling frequency will give better results. What you really want is for every frequency of interest to be an integer factor of the sampling rate.
Block size
Goertzel block size N is like the number of points in an equivalent FFT. It controls the frequency resolution (also called bin width). For example, if your sampling rate is 8kHz and N is 100 samples, then your bin width is 80Hz.
This would steer you towards making N as high as possible, to get the highest frequency resolution. The catch is that the higher N gets, the longer it takes to detect each tone, simply because you have to wait longer for all the samples to come in. For example, at 8kHz sampling, it will take 100ms for 800 samples to be accumulated. If you’re trying to detect tones of short duration, you will have to use compatible values of N.
The third factor influencing your choice of N is the relationship between the sampling rate and the target frequencies. Ideally you want the frequencies to be centered in their respective bins. In other words, you want the target frequencies to be integer multiples of sample_rate/N.
The good news is that, unlike the FFT, N doesn’t have to be a power of two.
Precomputed constants
Once you’ve selected your sampling rate and block size, it’s a simple five-step process to compute the constants you’ll need during processing:
w = (2*π/N)*k
cosine = cos w
sine = sin w
coeff = 2 * cosine
For the per-sample processing you’re going to need three variables. Let’s call them Q0, Q1, and Q2.
Q1 is just the value of Q0 last time. Q2 is just the value of Q0 two times ago (or Q1 last time).
Q1 and Q2 must be initialized to zero at the beginning of each block of samples. For every sample, you need to run the following three equations:
Q0 = coeff * Q1 – Q2 + sample
Q2 = Q1
Q1 = Q0
After running the per-sample equations N times, it’s time to see if the tone is present or not.
real = (Q1 – Q2 * cosine)
imag = (Q2 * sine)
magnitude2 = real2 + imag2
A simple threshold test of the magnitude will tell you if the tone was present or not. Reset Q2 and Q1 to zero and start the next block.
An optimized Goertzel
The optimized Goertzel requires less computation than the basic one, at the expense of phase information.
The per-sample processing is the same, but the end of block processing is different. Instead of computing real and imaginary components, and then converting those into relative magnitude squared, you directly compute the following:
magnitude2 = Q12 + Q22-Q1*Q2*coeff
This is the form of Goertzel I’ve used most often, and it was the first one I learned about.
Pulling it all together
Listing 1 shows a short demo program I wrote to enable you to test-drive the algorithm. The code was written and tested using the freely available DJGPP C/C++ compiler. You can modify the #defines near the top of the file to try out different values of N, sampling_rate, and target_frequency.
The program does two demonstrations. In the first one, both forms of the Goertzel algorithm are used to compute relative magnitude squared and relative magnitude for three different synthesized signals: one below the target_frequency, one equal to the target_frequency, and one above the target_frequency.
You’ll notice that the results are nearly identical, regardless of which form of the Goertzel algorithm is used. You’ll also notice that the detector values peak near the target frequency.
In the second demonstration, a simulated frequency sweep is run, and the results of just the basic Goertzel are shown. Again, you’ll notice a clear peak in the detector output near the target frequency. Figure 1 shows the output of the code in Listing 1.
Figure 1: Demo output
For SAMPLING_RATE = 8000.000000 N = 205 and FREQUENCY = 941.000000, k = 24 and coeff = 1.482867
For test frequency 691.000000:
real = -360.392059 imag = -45.871609
Relative magnitude squared = 131986.640625
Relative magnitude = 363.299652
Relative magnitude squared = 131986.640625
Relative magnitude = 363.299652
For test frequency 941.000000:
real = -3727.528076 imag = -9286.238281
Relative magnitude squared = 100128688.000000
Relative magnitude = 10006.432617
Relative magnitude squared = 100128680.000000
Relative magnitude = 10006.431641
For test frequency 1191.000000:
real = 424.038116 imag = -346.308716
Relative magnitude squared = 299738.062500
Relative magnitude = 547.483398
Relative magnitude squared = 299738.062500
Relative magnitude = 547.483398
| Freq= | 641.0 | rel mag^2= | 146697.87500 | rel mag= | 383.01160 |
| Freq= | 656.0 | rel mag^2= | 63684.62109 | rel mag= | 252.35812 |
| Freq= | 671.0 | rel mag^2= | 96753.92188 | rel mag= | 311.05292 |
| Freq= | 686.0 | rel mag^2= | 166669.90625 | rel mag= | 408.25226 |
| Freq= | 701.0 | rel mag^2= | 5414.02002 | rel mag= | 73.58002 |
| Freq= | 716.0 | rel mag^2= | 258318.37500 | rel mag= | 508.25031 |
| Freq= | 731.0 | rel mag^2= | 178329.68750 | rel mag= | 422.29099 |
| Freq= | 746.0 | rel mag^2= | 71271.88281 | rel mag= | 266.96796 |
| Freq= | 761.0 | rel mag^2= | 437814.90625 | rel mag= | 661.67584 |
| Freq= | 776.0 | rel mag^2= | 81901.81250 | rel mag= | 286.18494 |
| Freq= | 791.0 | rel mag^2= | 468060.31250 | rel mag= | 684.14935 |
| Freq= | 806.0 | rel mag^2= | 623345.56250 | rel mag= | 789.52234 |
| Freq= | 821.0 | rel mag^2= | 18701.58984 | rel mag= | 136.75375 |
| Freq= | 836.0 | rel mag^2= | 1434181.62500 | rel mag= | 1197.57324 |
| Freq= | 851.0 | rel mag^2= | 694211.75000 | rel mag= | 833.19373 |
| Freq= | 866.0 | rel mag^2= | 1120359.50000 | rel mag= | 1058.47034 |
| Freq= | 881.0 | rel mag^2= | 4626623.00000 | rel mag= | 2150.95874 |
| Freq= | 896.0 | rel mag^2= | 160420.43750 | rel mag= | 400.52521 |
| Freq= | 911.0 | rel mag^2= | 19374364.00000 | rel mag= | 4401.63184 |
| Freq= | 926.0 | rel mag^2= | 81229848.00000 | rel mag= | 9012.76074 |
| Freq= | 941.0 | rel mag^2= | 100128688.00000 | rel mag= | 10006.43262 |
| Freq= | 956.0 | rel mag^2= | 43694608.00000 | rel mag= | 6610.18994 |
| Freq= | 971.0 | rel mag^2= | 1793435.75000 | rel mag= | 1339.19226 |
| Freq= | 986.0 | rel mag^2= | 3519388.50000 | rel mag= | 1876.00330 |
| Freq= | 1001.0 | rel mag^2= | 3318844.50000 | rel mag= | 1821.76965 |
| Freq= | 1016.0 | rel mag^2= | 27707.98828 | rel mag= | 166.45717 |
| Freq= | 1031.0 | rel mag^2= | 1749922.62500 | rel mag= | 1322.84644 |
| Freq= | 1046.0 | rel mag^2= | 478859.28125 | rel mag= | 691.99658 |
| Freq= | 1061.0 | rel mag^2= | 284255.81250 | rel mag= | 533.15643 |
| Freq= | 1076.0 | rel mag^2= | 898392.93750 | rel mag= | 947.83594 |
| Freq= | 1091.0 | rel mag^2= | 11303.36035 | rel mag= | 106.31726 |
| Freq= | 1106.0 | rel mag^2= | 420975.65625 | rel mag= | 648.82635 |
| Freq= | 1121.0 | rel mag^2= | 325753.78125 | rel mag= | 570.74841 |
| Freq= | 1136.0 | rel mag^2= | 36595.78906 | rel mag= | 191.30026 |
| Freq= | 1151.0 | rel mag^2= | 410926.06250 | rel mag= | 641.03516 |
| Freq= | 1166.0 | rel mag^2= | 45246.58594 | rel mag= | 212.71245 |
| Freq= | 1181.0 | rel mag^2= | 119967.59375 | rel mag= | 346.36337 |
| Freq= | 1196.0 | rel mag^2= | 250361.39062 | rel mag= | 500.36127 |
| Freq= | 1211.0 | rel mag^2= | 1758.44263 | rel mag= | 41.93379 |
| Freq= | 1226.0 | rel mag^2= | 190195.57812 | rel mag= | 436.11417 |
| Freq= | 1241.0 | rel mag^2= | 74192.23438 | rel mag= | 272.38251 |
To avoid false detections in production code, you will probably want to qualify the raw detector readings by using a debouncing technique, such as requiring multiple detections in a row before reporting a tone’s presence to the user.
As you can see, the Goertzel algorithm deserves to be added to your signal processing toolbox.
Kevin Banks has been developing embedded systems for 19 years, as a consultant and as an employee at companies including SCI, TxPORT, DiscoveryCom, and Nokia. Currently he is back in consulting mode. He can be reached at kbanks@hiwaay.net.
References
Here are some links to other resources you may find useful:
www.numerix-dsp.com/goertzel.html
ptolemy.eecs.berkeley.edu/papers/96/dtmf_ict/www/node3.html
www.analogdevices.com/library/dspManuals/Using_ADSP-2100_Vol1_books.html
www.analogdevices.com/library/dspManuals/pdf/Volume1/Chapter_14.pdf
www.analogdevices.com/library/dspManuals/Using_ADSP-2100_Vol2_books.html
www.analogdevices.com/library/dspManuals/pdf/2100Chapter_8.pdf
Return to the September 2002 Table of Contents
Read the original at:
http://web.archive.org/web/20070926222355/http://www.embedded.com/story/OEG20020819S0057
Unlock Huawei modem
Posted by hvera1981 in Linux, Programming on August 9, 2010
#!/usr/bin/env python
#
# Copyright 2010: dogbert <dogber1 at gmail dot com>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
#
import hashlib, getopt, sys
def info():
print “huawei-unlock.py: calculates unlock codes for Huawei modems”
print “Copyright (c) 2010 dogbert <dogber1 at gmail dot com>”
print “For information and updates, visit http://dogber1.blogspot.com”
print “”
def usage():
print “Options: -i*, –imei=* IMEI of the device”
print “”
print “Example: huawei-unlock.py –imei=351234567891239″
print “”
def testIMEIChecksum(digits):
_sum = 0
alt = False
for d in reversed(digits):
assert 0 <= d <= 9
if alt:
d *= 2
if d > 9:
d -= 9
_sum += d
alt = not alt
return (_sum % 10) == 0
def checkIMEI(imei):
digits = []
if len(imei) != 15:
print “IMEI too short/long”
return False
for i in imei:
digits.append(int(i))
if not testIMEIChecksum(digits):
print “IMEI checksum invalid”
return False
return True
def getCode(imei, salt):
digest = hashlib.md5((imei+salt).lower()).digest()
code = 0
for i in range(0,4):
code += (ord(digest[i])^ord(digest[4+i])^ord(digest[8+i])^ord(digest[12+i])) << (3-i)*8
code &= 0x1ffffff
code |= 0×2000000
return code
def main():
info()
try:
opts, args = getopt.getopt(sys.argv[1:], “hi:”, ["help", "imei="])
except getopt.GetoptError, err:
print str(err) # will print something like “option -a not recognized”
usage()
sys.exit(2)
imei = “”
interactive = False
for o, a in opts:
if o in (“-i”, “–imei”):
imei = a
elif o in (“-h”, “–help”):
usage()
sys.exit()
if imei == “”:
imei = raw_input(“Please enter the IMEI of the device: “)
interactive = True
if checkIMEI(imei):
print “Unlock Code: %d” % (getCode(imei, hashlib.md5(“hwe620datacard”).hexdigest()[8:24]))
print “Flash Code: %d” % (getCode(imei, hashlib.md5(“e630upgrade”).hexdigest()[8:24]))
print “done.”
if interactive:
raw_input()
if __name__ == “__main__”:
main()
#EOF
thanks to http://dogber1.blogspot.com/2010/03/unlocker-for-huawei-modems.html
Remove duplicate files based on md5 hash
Posted by hvera1981 in Linux, Programming on July 27, 2010
OUTF=rem-duplicates.sh; echo “#! /bin/sh” > $OUTF; find “$@” -type f -exec md5sum {} \; | sort –key=1,32 | uniq -w 32 -d –all-repeated=separate | sed -r ‘s/^[0-9a-f]*( )*//;s/([^a-zA-Z0-9./_-])/\\\1/g;s/(.+)/#rm \1/’ >> $OUTF; chmod a+x $OUTF; ls -l $OUTF
http://elonen.iki.fi/code/misc-notes/remove-duplicate-files/
DotProject : Project ID is invalid. Unable to display.
Posted by hvera1981 in Programming on May 7, 2010
Correction:
modules/projects/view.php
71c71
< $q->addWhere(‘t1.task_id = t1.task_parent’);
—
> // $q->addWhere(‘t1.task_id = t1.task_parent’);
Writing an Android Twitter Client with Python
Posted by hvera1981 in Linux, Programming on June 20, 2009
import android # core Android routines
import httplib # for talking to web servers
import urllib # to format our status update nicely
import base64 # to encode our username and password for Basic authentication
print ‘Cool, we\’re running!’ # sorry, had to do this…
# get an instance of Android
droid = android.Android()
# where are we posting data to?
twitterhost = ‘twitter.com’
uri = ‘/statuses/update.xml?’
# get our status update from user
statusmsg = droid.getInput(‘Twitter Update’,'Whatcha Doin?’)
#extract the “textual” portion of the response
statusdata = “%(result)s” % statusmsg
# uncomment the next line to display the message to the terminal screen
#print statusdata
# uncomment the next line to display a notification to the user
#droid.makeToast(statusdata)
# clean up the data so it can be sent as the ‘querystring’
statusupdate = urllib.urlencode({‘status’:statusdata})
# setup your username and password here…
username = ‘yourusername’
password = ‘yourpassword’
credentials = username + “:” + password
# uncomment the next line to see what credentials you are using….
#print credentials
encodedcredentials = base64.encodestring(credentials)
# connect to server
h = httplib.HTTP(twitterhost)
# build url we want to request
fullurl = uri + statusupdate
#uncomment the next line to see the url printed
#print fullurl
# POST our data. Twitter requires status updates to be POSTed
h.putrequest(‘POST’,uri + statusupdate)
# setup the authentication header
h.putheader(‘Authorization’,'Basic ‘ + base64.encodestring(credentials))
# setup the user agent
h.putheader(‘user-agent’,'Android-Scripting-Engine-Python’)
# we’re done with the headers….
h.endheaders()
# get the response from the request
returncode,returnmsg,headers = h.getreply()
# should compare the returncode to 200 for a good response, etc.
#display whatever the results are….
f = h.getfile()
print f.read()
Source http://code.google.com
