Data Conversion Utility

# description: Data conversion utility
# The input file must have three columns: date, time & amplitude
# Line Example: 09/20/2018 08:23:56 0.1765432
#
# The output file will have two columns: time(sec) & amplitude
# The time will start at zero.

pump_data_convert.py

– Tom Irvine

Python CSV File Reading and Writing

The CSV (Comma Separated Values) format is a widely-accepted format for spreadsheet programs. Python has the capability to read and write csv data via the module:

import csv

Here is a demonstration script: csv_read_write.py

Here is the data file for the demonstration script: input_data.csv

See also:

Python Standard Library CSV

Tom Irvine

Python Lambda Functions

Introduction

The lambda keyword creates a small anonymous function or procedure.

It creates a function object the same way that the def keyword does, but with the following rules:

1. The lambda construct does not have a return statement.
2. The body can contain only a single expression.
3. The expression may yield a value.
4. The expression may be a conditional statement which returns a value.

Lambda may also be used to create anonymous procedures for use in GUI callbacks.

Lambda Examples

Here is a simple example of a lambda function.

g = lambda x: x**2
print g(8)

The result is: 64

Here is an example using two arguments:

g=lambda x,y: x+y
print g(2,3)

The result is: 5

A lambda function may be used inside a regular function.

def transform(n):

return lambda x: x + n
f = transform(3)
print f(4)

The result is: 7

Map function

A lambda function may be used with a map function to process a sequence of values.

g = lambda x: x**2
temp=(1, 2, 3, 4)
f = map(g, temp)
print f

The result is: [1, 4, 9, 16]

Reduce function

The function reduce(func, seq) continually applies the function func() to the sequence seq. It returns a single value.
The following sum a list of numbers.

reduce(lambda x,y: x+y, [47,11,42,13])

Result: 113

These statements find the maximum in a list.

f = lambda a,b: a if (a > b) else b
reduce(f, [47,11,42,102,13])

Result: 102

Calculate the sum of the numbers from 1 to 100:

reduce(lambda x, y: x+y, range(1,101))

Result: 5050

* * *

Tom Irvine

Python & Gnuplot

Gnuplot is a portable command-line driven graphing utility for Linux, OS/2, MS Windows, OSX, VMS, and many other platforms. The source code is copyrighted but freely distributed (i.e., you don’t have to pay for it). It was originally created to allow scientists and students to visualize mathematical functions and data interactively, but has grown to support many non-interactive uses such as web scripting

Gnuplot can be opened as a subprocess within a Python script.

The follow lines of code are required for the case of the sample program running in Windows or Linux:

from subprocess import Popen, PIPE

import platform

if platform.system() == ‘Windows’ or platform.system() == ‘Linux’:

if platform.system() == ‘Windows’:

gnuplot = r’C:\Program Files (x86)\gnuplot\bin\pgnuplot’
plot=Popen([gnuplot,’-persist’],stdin=PIPE,stdout=PIPE,stderr=PIPE)
if platform.system() == ‘Linux’:

plot=Popen([‘gnuplot’,’-persist’],stdin=PIPE,stdout=PIPE,stderr=PIPE)
# do some plot commands common to Windows & Linux

Here is the sample script which generates a gnuplot file, saves it to a folder, then loads it into gnuplot.

Windows & Linux version: python_gnuplot_demo.py
Here is the data file used in the demonstration: drop.txt

See also:

Gnuplot Blog Entry

Talking to gnuplot by pipes

Tom Irvine

Python: reading two columns of numerical data with an arbitrary number of header & blank lines

Here is a method.

The input data may be either space, tab, or comma delimited.

##########################################################################

import os
import re

from numpy import array,zeros,concatenate,linspace,std
from sys import stdin

import matplotlib.pyplot as plt

########################################################################

def start():
print(” “)
print(“Entpath =stdin.readline()
file_path = input_file_path.rstrip(‘\n’)
if os.path.exists(file_path):
print “This file exists”
print ” ”
infile = open(file_path,”rb”)
lines = infile.readlines()
infile.close()
if not os.path.exists(file_path):
print “This file doesn’t exist”
start()
return lines

########################################################################

def lines_two_columns():
#
# http://flockhart.virtualave.net/RBIF0100/regexp.html
#
lines=start()
a = []
b = []
num=0
for line in lines:
#
if re.search(r”(\d+)”, line): # matches a digit
iflag=0
else:
iflag=1 # did not find digit
#
if re.search(r”#”, line):
iflag=1
#
if iflag==0:
line=line.lower()
if re.search(r”([a-d])([f-z])”, line): # ignore header lines
iflag=1
else:
line = line.replace(“,”,” “)
col1,col2=line.split()
a.append(float(col1))
b.append(float(col2))
num=num+1
return a,b,num

##########################################################################

a,b,num =lines_two_columns()

* * *
Tom Irvine

Cython: Fourier Transform

This project demonstrates wrapping C/C++ functions in Cython. It also shows how to pass numpy arrays to C/C++ arrays and vice versa.

There are potential execution speed advantages to this mixed language approach.
The sample source code uses this approach to calculate a Fourier transform from a time history signal.
The scripts on this page require the utility module tompy.py

The C/C++ source code and its header file are: fourier_ccode.cpp & fourier_ccode.h

The Cython file and its setup script are: fourier_trans.pyx & ft_setup.py

The Cython & C/C++ files are compiled in command-line via: python ft_setup.py build_ext –inplace

The main Python file is: fourier_c.py
This project was developed using the reference material at: Wrapping Cython

Tom Irvine

Cython: Runge-Kutta

The follow set of scripts calculate the response of a specific single-degree-of-freedom system to an arbitrary base input time history, via the Runge-Kutta fourth order method.

The scripts also demonstrate the passing of numpy arrays to the function, which processes the the data and then returns the resulting arrays to the main Python script.

The scripts on this page require the utility module tompy.py

The function is defined in: sdof_rk4.pyx

The setup file is: sdof_rk4_setup.py

The function is complied in command-line mode via: python sdof_rk4_setup.py build_ext –inplace
The main Python script which imports the function is: arbit_rk4_c.py

Tom Irvine

Cython

Introduction

Cython is a compiler which compiles Python-like code files to C code. The resulting functions can then be imported into other Python scripts.

This method can be used to increase the execution speed of a Python script, particularly if the script uses for-loops.

The Python-like *.pyx script which provides the function must have have static C data types in order for the main Python script to gain full speed performance.

Installation

Cython can be installed on a Ubuntu system using the “Ubuntu Software Center.”

It is also bundled with the Enthought Python Distribution [EPD] and with Sage Notebook.

Steps

Using Cython consists of these steps:

1. Write a .pyx source file for the function

2. Write a setup.py file using the disutils function

3. Compile the source code in command-line mode using: python setup.py build_ext –inplace

4. Import the function into the main Python script and then run it
There are a few alternative compiler methods, but disutils is the most reliable from the author’s experience.

Examples

Cython: Runge-Kutta

Cython: Fourier transform

Tom Irvine