Merging pdf files in Python with pyPDF

Here is a very simple python script that marges two pdf files, using the pyPDF library.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import os.path
import pyPdf
 
pdfOne = "C:\\a.pdf"
pdfTwo = "C:\\b.pdf"
 
merged = "C:\\c.pdf"
 
if os.path.exists(pdfOne) and os.path.exists(pdfTwo): 
 
	output = pyPdf.PdfFileWriter()
 
	pdfOne = pyPdf.PdfFileReader(open(pdfOne, "rb"))
	for page in range(pdfOne.getNumPages()):
		output.addPage(pdfOne.getPage(page))
 
	pdfTwo = pyPdf.PdfFileReader(open(pdfTwo, "rb"))
	for page in range(pdfTwo.getNumPages()):
		output.addPage(pdfTwo.getPage(page))
 
	outputStream = open(merged, "wb")
	output.write(outputStream)
	outputStream.close()

PIR sensor + Arduino + Processing + Skype…

This post is about a simple and interesting combination of software and hardware.

Last week, I picked up a couple of PIR sensors from Ebay.

pir2_w500

PIR sensors are (usually) easy to interface with an arduino board.
Red: 5V, Black: Ground, Yellow: Analog In. Could it be easier than this?

PIR_w500

After a bit of experimentation, I found that the board sends a pulse as soon as it detects some sort of movement. Also, it sends a pulse at the very beginning, when it sort of “boots”.

My ultimate aim was to start Skype as soon as movement was detected.

I loaded the standard firmata sketch on my arduino, see: http://arduino.cc/en/Reference/Firmata. Basically, the arduino simply sends all the analog port readings to Processing.

The following Processing code is simply a proof of concept. The sketch starts, looks for an arduino at the second serial port of my PC, waits for 5 secs for the PIR to send its first pulse and then, if movement is detected calls a number, using Skype.

At any moment, in the middle of the screen, you see the value of the PIR sensor.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
import cc.arduino.*;
import java.io.*;
import processing.serial.*;
 
int analogPin = 5;
int myV; // PIR sensor value
 
PFont font;
Arduino arduino;
 
//////////////////////////////////////////////////////////////////////////////
void setup(){
  size(400, 300);
  font = createFont("ArialUnicodeMS-12.vlw",22, false); 
 
  //my board is the 2nd serial device, hence Arduino.list()[1]
  println(arduino.list());
  arduino = new Arduino(this, Arduino.list()[1], 57600);
}
 
//////////////////////////////////////////////////////////////////////////////
void draw(){
    background(0);
    fill(255);
 
  // the PIR sensor is sending out HIGH signal in the first 5 secs
  // so we will ignore all the early signals
  if(millis() < 5000){
 
  }
 
  else{
    // myV is the scaled value of the PIR sensor
    myV = arduino.analogRead(analogPin);
 
    textFont(font, 22); 
    textAlign(CENTER,CENTER);
    text("PIR sensor value: " + myV,width/2,height/2);
    text("Active for " + round(millis()/1000) + " seconds",width/2,height*3/4);
 
    callme();
  }
}
 
//////////////////////////////////////////////////////////////////////////////
void callme(){
 
  // you should be logged in on Skype and have some credit
  // if your are dialling an external line
  String mySkype = "C:\\Program Files\\Skype\\Phone\\skype.exe /nosplash /callto:+44798000000";
 
  if(myV>100){
 
    print(" MyV: " + myV + " ");
 
    try {
      String line;
      Process p = Runtime.getRuntime().exec(mySkype);
      p.waitFor();
      System.out.println("EXIT: " + p.exitValue());
    }
 
    catch (Exception err) {
      err.printStackTrace();
    }
 
      delay(3000); 
  }
 
 
 
}
//////////////////////////////////////////////////////////////////////////////

Here is a very short video of this test in action:

PIR Sensor – Test from Arkadian.Eu on Vimeo.

Arduino-based keyboard for Windows

Today, on the arduino.cc forums, a user had the following problem: he wanted to start video in quicktime by pressing a custom button on an arduino. Basically, the arduino board should act as a simple keyboard for the computer.

I found this problem interesting and I ended up solving it. Here is the link to the forum.

Basically this problem has three parts.
1. An arduino sketch which, when the button is pressed, sends a character on the serial.
2. A serial-to-keystrokes program, which takes the character from the serial connection and converts it to a real keystroke
3. A program on the windows machine that translates the keystroke to whatever we want to do.

Sounds tricky? It’s actually easier than it sounds.

1. The Arduino Sketch
This is quite straight forward. The simple arduino keyboard circuit looks like this:

arduino-keyboard

Here is a modified version of the Button.pde example that comes with the latest Arduino release. I have only added the lines required for the serial connection and a very long (2 sec) delay after the key is pressed; I used this 2 sec delay to keep things simple, but you may want to adapt the longer and more appropriate “Debounce.pde” sketch (also available as an example in the latest Arduino release) which takes care of the debouncing more efficiently.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
const int buttonPin = 2;     
const int ledPin =  13;      
int buttonState = 0;         
 
void setup() {
  Serial.begin(9600); 
  pinMode(ledPin, OUTPUT);      
  pinMode(buttonPin, INPUT);     
}
 
void loop(){
  buttonState = digitalRead(buttonPin);
  if (buttonState == HIGH) {     
    digitalWrite(ledPin, HIGH);  
    Serial.print("a");
    delay(2000);
  }  
  else {
    digitalWrite(ledPin, LOW); 
  }
}

2. From Serial to Keystrokes
I found a very nice program, AAC Keys, which converts any characters the arduino sends to the serial port to actual keystrokes. This is available for both windows and Mac and it’s free to download; the download is a single executable file, which, when activated, stays in your system tray and monitors the serial port. The only thing you need to configure here is the port and the Baud rate:
aac_keys

Note that if AAC Keys is activated and configured, the arduino environment won’t be able to communicate with the arduino board, as the port cannot be shared.

At this stage, with the AAC Keys activated and configured, pressing the button on the arduino will create the character “a” on the computer.

3. Windows Shortcuts
What we need to do now is to somehow start playing a video with Quicktime. To play a video with QuickTime, all we need to do is to run the following command in a DOS window:

"C:\Program Files\QuickTime\QuickTimePlayer.exe" "C:\your video path\Video_A.avi"

If the video doesn’t start directly, then go to QuickTime >> Preferences >> Player Preferences… and select “automatically play movies when opened”.

My favourite windows program for these tasks is AutoHotKey. When AutoHotKey is installed, textfiles with the .AHK extension are executed by AutoHotKey.

The following AHK script, when executed, will stay in the system tray waiting for the “a” key (either from the keyboard or from the arduino board) to be pressed.

1
2
3
a:: 
Run "C:\Program Files\QuickTime\QuickTimePlayer.exe" "C:\your video path\Video_A.avi"
return

…and that’s it! A simple (1-key) Arduino keyboard, which, when pressed, launches the video we want in quicktime.

There are many ways to extend this idea. More keys can be added (many can be added on a single analog port with resistors), and, with a bluetooth adaptor on the arduino board and the computer, we could even design a bluetooth keyboard for the computer! :)

simpleDateFormat in Processing

I spent about an hour, trying to figure out how to use the simpleDateFormat java library in Processing… It seemed so straight forward, yet I couldn’t get it to work…

Well, it seems that the only thing I had to add is a try/catch.

So, this example works:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
import java.text.*;
Date myDate;
 
void setup(){
SimpleDateFormat myDateFormat = new SimpleDateFormat("yyyyMMdd");
 
try {
myDate = myDateFormat.parse("20001010");
} catch (Exception e) {
  println("Unable to parse date stamp");
} 
 
println(myDate);
}

but this one does not, even though there is no error to catch…

1
2
3
4
5
6
7
8
import java.text.*;
Date myDate;
 
void setup(){
SimpleDateFormat myDateFormat = new SimpleDateFormat("yyyyMMdd");
myDate = myDateFormat.parse("20001010");
println(myDate);
}

I’m not sure why this is the case. If anyone does, please let me know!

Parsing files with Processing…

In the following basic example, I’m using Processing to parse a 2GB textfile.

The textfile I’m using (content.rdf.u8) is from the DMOZ.org project. You can download the compressed file here.

As mentioned earlier, the size of the textfile, content.rdf.u8, is approx 2GB. For a sample of the contents and structure of the file, follow this link. For a file of this size, it’s important to use BufferedReader. Read here for more on BufferedReader.

The following piece of code, reads the textfile and converts it to another textfile of the format: URL|Title|Description|Category:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
import java.util.regex.*;
 
String fileName = dataPath("C:\\Users\\Arkadian\\Downloads\\content.rdf.u8") ;
String outputFile = dataPath("D:\\dmoz\\data.txt") ;
 
String sep = "|";
String outputLine = "";
String L = "";
String t = "";
 
PrintWriter output;
output = createWriter(outputFile);
 
try
{
  BufferedReader file = new BufferedReader (new FileReader (fileName)) ;
 
  while (file.ready()){ 
 
    L = file.readLine(); 
    //println(L); //for debuging only...
 
    String[] m1 = match(L, "<ExternalPage about=");
    String[] m2 = match(L, "<d:Title>");
    String[] m3 = match(L, "<d:Description>");
    String[] m4 = match(L, "<topic>");
    String[] m5 = match(L, "</ExternalPage>");
 
    // Start of an external page & URL
    if(m1 != null){
      t = L.replaceAll("<ExternalPage about=\"","");
      t = t.replaceAll("\">","");
      outputLine = t;
    }
 
    // Title
    if(m2 != null){
      t = L.replaceAll("  <d:Title>","");
      t = t.replaceAll("</d:Title>","");
      outputLine = outputLine + sep + t;
    }   
 
    // Description
    if(m3 != null){
      t = L.replaceAll("  <d:Description>","");
      t = t.replaceAll("</d:Description>","");
      outputLine = outputLine + sep + t;
    }   
 
    // Topic
    if(m4 != null){
      t = L.replaceAll("  <topic>","");
      t = t.replaceAll("</topic>","");
      outputLine = outputLine + sep + t;
    }    
 
    // End of External page
    if(m5 != null){
      println(outputLine);
      output.println(outputLine);
      output.flush();
    }    
  }
}
 
catch (Exception e){
  println ("Error" + e) ;
}
 
output.close();
exit();

Pretty simple, yet effective…

Processing, MySQL and RAM tables…

I’m currently working on a simulation model in Processing, which creates in every cycle, 350,000 lines of temporary data in chunks of 1,000 lines.

I decided to save the temporary data in a MySQL table, as this allows me to easily add an extra query at the end that summarises everything and saves the final results on a different table.

As the 350,000 lines need only to be stored temporarily, I’m using the Memory storage engine, instead of the default (MyISAM). The advantage of the Memory storage engine for temporary storage is that it saves the data on the RAM rather than the hard drive, is (or should be) faster and when you reboot the machine, the temp data is gone. The structure of the table is stored on the hard drive, so you can use the table again and again.

Here is a test sketch I wrote to get an estimate of how long it takes to create 350,000 records.

If you want to try this, you need to download and install the SQLibrary (http://bezier.de/processing/libs/sql/). You also need to install a mySQL database locally. If you don’t have one installed already, I suggest you go to apachefriends.org and download the latest version of XAMPP.

First, to create a simple table in RAM, run the following mysql snippet:

1
2
3
4
5
6
CREATE TABLE IF NOT EXISTS `arkadian_test` (
  `myID` int(11) NOT NULL AUTO_INCREMENT,
  `myRecord` int(15) NOT NULL,
  `myTimestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (`myID`)
) ENGINE=MEMORY  DEFAULT CHARSET=latin1;

This is a simple table, that is stored in ram (ENGINE=MEMORY).

The following sketch creates 350,000 records in chunks of 1,000 records.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
 
/* //////////////////////////////////////////////////////////////////////
Date: 1 Aug 2009
From: Arkadian.eu
Desc: A quick sketch that creates 350K records in MySQL
*///////////////////////////////////////////////////////////////////////
 
import de.bezier.data.sql.*;
MySQL mx;
 
public int maxRecords = 350000;
public int currentRecord = 0;
 
// adjust these for your configuration
public String user     = "root";
public String pass     = ""; //blank
public String database = "test";
public String dbhost = "localhost";
/////////////////////////////////////////////////////////////////////////
void setup() {
  size(100, 100);
  mx = new MySQL( this, dbhost, database, user, pass );
  while(mx.connect()==false){delay(1000);}
  mx.execute("TRUNCATE TABLE  `arkadian_test`"); 
}
/////////////////////////////////////////////////////////////////////////
void draw() {
 if(currentRecord<maxRecords){
    // let's check to see if the connection is still up
    while(mx.connect()==false){
      mx = new MySQL( this, "localhost", database, user, pass );
    }
 
    if ( mx.connect()==true ){  
 
      for(int i=0; i<1000; i++){
        currentRecord++;
        mx.execute("INSERT INTO arkadian_test(myRecord) VALUES ('" + currentRecord+ "')"); 
      }
 
    }
 
  }
 
  else{
    mx.close();
    exit();
  }
}
/////////////////////////////////////////////////////////////////////////

The above sketch takes about 45-50 seconds to run on my 2GHz Intel Core2 Duo laptop with 3GB RAM.

To see how long it takes on your machine, just look at the timestamps of the first and last record on your database.

For more information on MySQL tables on RAM, visit this site.