Augustina's Technological Blog

Technology, Perl, Linux, and a Woman's perspective on the FOSS community

Archive for the ‘Programming’ Category

Getting Started with Devel::NYTProf

leave a comment »

I went to OSCON 2010 and saw Tim Bunce’s talk on Devel::NYTProf. For my local Perl User Group (SPUG) I decided to do a talk on how to get started with NYTProf and to provide an overview of the benefits of code profiling. This entry is a summary of the talk that I gave.

What is Devel::NYTProf?

Devel::NYTProf is a source code profiler. It is a tool that gives you another perspective on how your code is functioning. Typically people use NYTProf to find places in their code that could be optimized to run faster, however it can also be used to better understand what is happening when your program executes.

Why should I use Devel::NYTProf?

If you’re working with unfamiliar code, NYTProf provides a high level overview on what parts you need to pay the most attention to. It breaks the source code down line by line and reports execution times for each block and subroutine. NYTProf highlights hotspots in your code which helps to identify priority areas that need improvement. NYTProf provides visualization tools that are not only useful for communicating to other folks who might not be as familiar with the source code, but are also useful for right-brained artsy-fartsy visually-oriented people like myself ūüėČ

How do I use Devel::NYTProf?

1. Install the module Devel::NYTProf (I used CPAN)

Run the following command:

perl -d:NYTProf my-perl.pl

2. Set options to customize behavior

Command line:

 -addpid=1

Environment variable:

export NYTPROF=file=/tmp/nytprof.out

(See the POD for more info about custom options)

3. Format output

nytprofhtml

When you run NYTProf, it creates an output file in the same location where you ran it (unless you’ve specified something different in the options). To read the output, format it using NYTProf’s formatters. These are discussed in the POD.

If you are using the HTML formatter, by default it creates a folder called nytprof in the same directory you run it from. Open the index.html  file contained in that directory to view the formatted profile.

How do I read the output?

Once you’ve formatted the output into something human-readable, making sense of what’s being reported is the next step. Besides an HTML layout, there are visualizations of the code in Graphiz and a Treemap based on subroutine frequency and inclusive time.

The key terms to be aware of when profiling are Inclusive Time and Exclusive Time. Inclusive Time is the total time to run a subroutine including any additional subroutines being called within the subroutine. This is also referred to as the Cumulative Time in other profilers. Exclusive Time is the total time spent in this subroutine excluding calls to any other subroutines.

Another important thing to note about the reported times is that there are no hard and fast rules for what is considered the ideal time for any subroutine. Everything is relative. Look at things in red to see if the reported time makes sense given the nature of the data and function being performed. The hotspots only indicate what’s taken the most time in the program, it does not mean that something is necessarily inefficient.

Statement Profiling

NYTProf reports on blocks of code in addition to subroutines. It measures the time between entering one perl statement and entering the next.

Perl makes heavy use of additional system calls like regexes, operating sytem functions, and user input. NYTProf intercepts perl opcodes and reports them as subroutines. This allows you to evaluate whether a block of code was inefficient due to something within your code or if the system call is what’s slowing you down. If there is an alternative method to using that system call then you can see how much an improvement it makes by implementing it and re-running the profiler.

Use the -slowops=N option to customize how NYTProf reports opcodes.

Subroutine Profiling

NYTProf determines subroutine durations by measuring the time between entering a subroutine and leaving it. As discussed earlier, it accumulates both Inclusive vs Exclusive times for each subroutine. It also reports how frequently a subroutine was called both for the whole program and in a given location. The profile reports  separate values for each location where a subroutine was called.

Recursive Subroutines

NYTProf differentiates regular subroutines from recursive ones. It reports the Inclusive time for outermost call and also indicates that the subroutine in recursive. It also reports the maximum recursion depth.

Application Profiling

NYTProf is a fairly sophisticated profiler in that it can support threads and forks. For complex applications, it records extra info in a data file, including:

  • Filename
  • Line ranges of subroutines
  • Detects forks

It also supports Apache Profiling with mod_perl hooks. See CPAN module notes for more info on how to hook into Apache.

A note about optimization

Optimization is not always necessary. You really have to ask yourself if the amount of time to execute something makes sense given the amount of work. If the task is pretty hairy, chances are you’re better off finding a different area to optimize.

A lot of small changes add up. If you can make a few small optimizations to subroutines that are called the most frequently, you can shave seconds off of your times.

Devel::NYTProf just finds the hotspots (kind of like SQL Explain). It really doesn’t know anything about how your program is supposed to work, it just gives you a bunch of interesting statistics ūüôā

More Info

Tim Bunce’s Devel::NYTProf Talk

Devel::NYTProf POD

 

Written by missaugustina

February 5, 2011 at 5:15 am

Posted in Perl, Programming

An Introduction to Object-Oriented Perl

with 2 comments

I’ve been using Perl for a few years now and I am finally getting to a place where I’m working with more complex problems. Traditionally I’ve written little utility scripts or handlers for small specific tasks. In my current position, one of my roles is writing ad-hoc scripts for bulk reporting and bulk updates to a large number of database records for client requests. As I’m passing some of these duties on to my coworkers, I realized I needed a template to ensure consistency and to make their lives easier. I’ve also been tasked with a larger project for which I’ve decided the solution is to make a single Object-Oriented API in our code base that can be called by many different sources.

My original training is in Java/C++ and Object-Oriented Design. I wanted to approach both of these problems using my OO background, but I was having trouble grasping how OO really works in Perl. I’ve scoured the internet and poured through books, and none of them could really answer my questions. Finally, I sat down with someone recently who answered my “why why why” questions (and who also proofread this entry for accuracy! Thanks Dave!!). I decided I should document this here in hopes that it might help others who are in the same conundrum.

This article will be most useful if you already understand OO and you understand Perl but you just want to know from a bottom up perspective how OO works in Perl.

Key Concepts

Unlike languages like C++ and Java where Object-Orientation was considered during the language development, in Perl it was more of an afterthought.

Bless

Bless is a built-in Perl function that turns any reference into an object. While generally it’s good practice to use hash references, there are many differing schools of thought on Perl OO, including inside-out objects. These techniques are beyond the scope of this particular article¬†(bad OO pun haha)¬†so I won’t discuss them here.

There are 2 key differences between references and blessed references (aka objects). An object contains an OBJECT flag telling the interpreter that its an object. An object also contains a string with the name of its class so the interpreter can look up methods called on the object that are defined in its class.

When you instantiate an object in your executable Perl code, the Perl interpreter stores the class name in the object data.¬†When a method is called on the object instance, Perl attempts to resolve the method at runtime. Using the class name, the interpreter looks for the method in the class definition. If it does not find the method, it looks in @ISA to see if the class is a child and if the method is defined in the parent class. If it does not find the method defined for the class, the interpreter throws a “Can’t locate object method” exception.

Here are examples using Devel::Peek to view what a reference and an object looks like:

First examine what a hash reference looks like:

perl -MDevel::Peek -wle 'my $not_an_object = {}; print Dump($not_an_object)'

Output:

SV = RV(0x90e492c) at 0x90e4920
REFCNT = 1
FLAGS = (PADMY,ROK)
RV = 0x90e4880
SV = PVHV(0x90ef340) at 0x90e4880
REFCNT = 1
FLAGS = (SHAREKEYS)
ARRAY = 0x0
KEYS = 0
FILL = 0
MAX = 7
RITER = -1
EITER = 0x0

Now examine what an blessed hash reference (aka object) looks like:

perl -MDevel::Peek -wle 'my $object = bless {}, "main"; print Dump($object)'

Output:

SV = RV(0x979d92c) at 0x979d920
REFCNT = 1
FLAGS = (PADMY,ROK)
RV = 0x979d880
SV = PVHV(0x97a8340) at 0x979d880
REFCNT = 1
FLAGS = (OBJECT,SHAREKEYS)
STASH = 0x979d770 "main"
ARRAY = 0x0
KEYS = 0
FILL = 0
MAX = 7
RITER = -1
EITER = 0x0

Notice the second FLAGS entry and the STASH entry. STASH contains both the string of the class name and the address where the string is stored in memory. FLAGS contains the value OBJECT which tells the Perl interpreter that $object is an object.

Intrinsic vs. Explict Methods

In Perl, a method does not¬†intrinsically¬†know what object or class it’s called on. When a method is called, the Perl interpreter either passes a reference to that object or the class name as the first argument. When writing methods that take arguments, the first argument passed in by the interpreter is the object reference or the class name depending on whether the method was called by an object or by the class itself. For instance, the new() constructor is called on the class so the first argument the interpreter passes in is a string containing the class name. For a method you define called by an object instance, the¬†interpreter¬†passes in a reference to that object.

While it’s good practice to call the first argument¬†$self or $me, you can call it whatever you want.

Here’s are two examples of handling arguments in methods:

sub my_method {
my $self = shift;
my ($arg1, $arg2) = @_;
}

sub my_method {
my ($self, $arg1, $arg2) = @_;
}

Here’s an example of calling the method, for this example the class is “MyClass”.

my $my_class = MyClass->new();

$my_class->my_method($arg1, $arg2);

When calling a class method without an object instance, the first argument is a string representing the name of the class.

MyClass->my_method($arg1, $arg2);

Class Methods vs. Object Methods

In Perl, you don’t have to instantiate an object to call its class methods, so you have to provide your own enforcement if this is a possible risk.

When a method is called by a class, the first argument passed is a string containing the name of the class. When a method is called by an object, the first argument is a reference to the object. To enforce calling methods by object, check that the first argument passed is a reference.


die "Please call this method on an object instance." if (! ref $self);

Unenforced Privacy

There are no private methods in Perl except by convention. To designate a method as private, the general practice is to begin the name with an _ character. While this does not prevent use of this method in other contexts, it makes your code and your intentions easier to read.

sub _my_private_method {}

sub my_public_method {}

Data Attributes

One of the advantages to using a blessed hash ref for your object is that you can manage basic data attributes without using accessor methods or having to maintain a long list of variables. I believe accessor methods should be used under two¬†circumstances: 1.¬†when they add some value or additional functionality or enforceability with regards to data types and data values, and 2. when a data attribute needs to be modified external to the class definition. When dealing with simple values used internal to the class definition, maintaining a long list of variables and getter/setter methods is overkill. This is my opinion and I’m sure there are plenty of others out there that differ, so do what works best for the problem you are trying to solve. I’m just presenting an alternative that was shown to me that I’ve found to be pretty handy.


$self->{NAME} = $name;

Instead of having to create and maintain a get/set_name method, your object can access it’s own “name” with $myobject->{NAME}.

Garbage Collection

Once an object goes out of scope, the Perl interpreter calls DESTROY on it prior to dereferencing the memory. Unless your class has specific requirements for what needs to happen when the object is destroyed, you don’t need to implement the destroy method.

To implement the DESTROY method, define a method called DESTROY in all caps.

Perl does not automatically call DESTROY on super classes. To DESTROY a super class, you must explicitly call $self->SUPER::DESTROY.

How To

As stated in the previous section, there are many philosophies about how to do Object Oriented Perl. The intent of this article is provide basic instructions for getting started with writing a simple class and to explain some of the Perl internals. The reason for this is because I was having a hard time finding the information I needed and up-to-date examples. Once you’ve gotten a general idea of the basics, it’s worthwhile to explore different OO Perl philosophies.

1. Create a class file ending in .pm

For this example, create a file called MyClass.pm and save it where Perl can find it.

Define a new() method to bless your passed in reference and assign it to the class. Define methods as appropriate and return 1 at the end.


package MyClass;
use strict;
use warnings;

sub new {
# first argument is the class name
my ($class, %args) = @_;

# create an anonymous hash ref called $self
# bless $self to set it as obj with class name $class
# return obj ref
my $self = bless {}, $class;

# call a method on $self to set an initial value
if (defined $args{name})
{
$self->set_name($args{name});
# alternatively we can set the name using
# $self->{NAME} = $name;
} else {
die "Please specify a name for the $class instance.";
}

# return the obj instance
return $self;
}

sub set_name {
my ($self, $name) = @_;

if (! ref $self) # if called as a class method, $self is a string, not a ref
{
die "Please call this method on an object instance.";

} else {
# assign class data attributes to keys within the $self hash
$self->{NAME} = "My name is " . $name;
}
}

sub print_name {
my ($self) = @_;
print $self->{NAME}, "\n";
}

1;

2. Create a Perl executable file ending in .pl. Instantiate your class by creating a variable and using the new() method. Call a method on your class to see if it works!


#!/usr/bin/perl -w
use strict;
use MyClass;

# create an object by instantiating MyClass
my $object = MyClass->new(name => 'Eugene');

# call an object method on MyClass
$object->print_name();

# calling a class method returns an error
MyClass->set_name('Marsha');

Written by missaugustina

June 6, 2010 at 10:31 pm

Posted in Perl, Programming

Managing Ebooks on the Kindle

with one comment

DefectiveByDesign and other Free Speech advocacy groups have been targeting the Kindle for its DRM limited content. This was something I grappled with when I bought my Kindle last year. Of course Amazon wants Kindle users to buy books from their store, but limiting ebook content really limits the usefulness of the Kindle! What if an ebook I want isn’t available through Amazon, but is available in another format through another seller? What about checking out ebooks from the library?

I initially was interested in ebook readers because I use the bus as my main mode of transportation. Also I love to read but I tend to forget to return books to the library and my collection of used books keeps growing. Living in the city, I don’t have a lot of space. If I could have all of my (non-picture) books as ebooks I would be very satisfied.

The great thing about the Kindle is that I’ve been reading A LOT more and it’s a fun device to use.

The not-so-great thing is that Amazon limits the way you can access ebooks DRM’ed by other providers. Fortunately there are ways around these limitations!

Download Free Ebooks in Mobi Format

There are non-DRM books available that the Kindle can read and it’s really easy to do. I use my SD card because I think cables are annoying. Insert the SD card into your computer and create a new folder called “documents”.

There are several free ebook sites on the web, in particular Project Gutenburg and Mobipocket Free Books. Pick a title and download it in “mobi” format. Save it to the “documents” folder you created on your SD card.

Insert the SD card into the Kindle. When you start your Kindle up, you will see the titles you downloaded as “new” items.

Convert Digital Files to Mobi Format

Kindle offers a service that will convert other ebook formats (HTML, PDF, etc) to Mobi format. This service is $.10 USD per book. Alternatively you can convert it yourself. Either use MobiPocket Reader to import the file (it is saved under “My eBooks” in your Documents folder) or download MobiPocket Creator. Note that there may be additional challenges with DRM’ed ebooks. Once converted, copy the files into the “documents” folder either on your Kindle or on your SD card.

Register Your Kindle as a MobiPocket Reader Device

The final workaround involves registering your Kindle as a MobiPocket Reader device. Not only can you read Kindle ebooks on your computer, but you can transfer them to other devices, and even transfer ebooks DRM’ed by other providers to your Kindle. For more detailed instructions on converting the Kindle files to MobiPocket format, see this article.

What you’ll need: A Kindle, an SD Card (or USB Cable to transfer files between your Kindle and your computer), ActivePython 2.x, MobiPocket Reader, Python Scripts (see below for where to get them)

Step 1) Get the Serial Number of your Kindle.

This is under the back cover on the label above the battery. The “Serial No.” is listed under the first barcode.

Step 2) Copy the ebook file from the Kindle to your computer.

Insert the SD card into your Kindle. Open the Content Manager and select a book on your list. Choose the option to save it to the SD card. Alternatively you can download files from your Kindle profile in Amazon to your computer.

Turn off the Kindle and insert the SD card into your computer. Your book is in a folder called “downloads”. I have a folder I created in “Documents” called “Ebooks”.

Step 3) Install Python and download the scripts.

Install Python 2.6 if you haven’t already. Note that the scripts are not compatible with Python 3.x. On Windows, I recommend ActivePython. Next download a small suite of Python scripts from the linked article above, under “Step 4”. Please let me know if the archive is unavailable so I can update this article.

When you extract the archive, you can place the scripts in the Tools\scripts directory of your Python folder and update the system path variable. I placed mine in the same folder I save my ebook files to for convenience. You need to run the scripts from the same location as the files you copied from your Kindle.

Step 4) Create the Mobi file

Open up a command line interface. On Windows type “cmd” either in “Run” (XP) or in the search dialogue of the Windows menu (Vista).

Navigate to the folder containing the files you copied from your Kindle:

cd Documents\Ebooks

Type the following command, replacing yourserialnumber with your serial number:

python kindlepid.py yourserialnumber

The script returns a 10-digit PID as the last item on the printed line. The third to last digit of the PID is an asterisk(*). Make a note of the PID.

Step 5) Open the book in Mobipocket Reader.

Install Mobipocket Reader if you haven’t already done so.

To register your Kindle as a device in Mobipocket Reader, select Reading Devices from the navigation on the left. Click the button located at the upper center part of the window to add your Kindle. I named my Kindle “Kindle” and chose “Other” for device type. Enter your PID in the field that asks for a MobiPocket ID. To open Kindle files in Mobi, change the extension of the “.azw” file to “.mobi”. Select Open or Import to view the file in Mobipocket Reader.

Double click the .mobi file you created to open it in Mobipocket Reader. Use Mobipocket Reader to register other supported devices so you can transfer your ebooks to them!

Written by missaugustina

August 26, 2009 at 6:37 am

Installing Flex On Ubuntu

with one comment

Install Eclipse

Do not install Eclipse from the Ubuntu repositories.¬† Navigate to http://www.eclipse.org/downloads/ and select Europa on the left-hand menu.¬† Download the Eclipse for Java download. Extract the .tar.gz file and move it to the /usr/lib/ directory. Create a symbolic link in /usr/bin to launch Eclipse.¬†¬† “sudo ln -s /usr/lib/eclipse eclipse”¬† Also you can add a menu item under Programming to your applications menu.¬† Launch Eclipse to make sure the installation went smoothly.

Configure Sun Java

Follow this HOWTO: http://ubuntuforums.org/showthread.php?t=201378
Or follow the instructions to use your JDK of choice here – https://help.ubuntu.com/community/EclipseIDE

Install Flex
Once you’ve configured Eclipse, install the Flex plugin for Linux – http://labs.adobe.com/technologies/flex/flexbuilder_linux/
Download the bin file and from the command line, chmod+x to make it executable.¬† Run it from the command line by cd’ing to the directory where it’s downloaded and run it by prefacing the downloaded bin file with ./ .¬† Example: sudo ./flexbuilder_linux_install_a4_081408.bin

Be sure to run this install script with root privileges because the installation will need to access folders that will require it.¬† During the installation you’ll be prompted for 2 different directories, one for the Flex install and another for the Eclipse root folder.¬† I chose the /opt directory for Flex installation.¬† On Ubuntu, if you installed Eclipse via apt the Eclipse root folder will be at¬† /usr/lib/eclipse .¬† Do not use the defaults the installation script is prompting.

Create a Project

Launch Eclipse and select File > New > Other…

A dialog should pop up with a list of wizards.  Expand the folder for Flex Builder and select your project type.

Note that I was unable to get this working with Eclipse 3.4 .  If you have a version of Eclipse installed from the Ubuntu repositories I highly recommend you completely remove it and use the packages provided by Eclipse.org.

Written by missaugustina

March 11, 2009 at 7:43 pm

FileMaker ODBC Fetch Forward Error

leave a comment »

After scouring the internet for this mysterious “Fetch Forward” error I sometimes get when migrating FileMaker records to other database formats via ODBC and Perl DBI, I discovered what causes it.

The error is seemingly random and goes like this:
[FileMaker][ODBC FileMaker Pro driver][FileMaker Pro]Unknown error (SQL-HY000)
[FileMaker][ODBC FileMaker Pro driver]An attempt to fetch forward has failed for table: TABLENAME (SQL-HY000)(DBD: st_fetc/SQLFetch err=-1) at myperlscript.pl line 123.

This occurs when FileMaker drops the ODBC connection or is unable to respond in a timely manner.

Solutions are as follows:
1) Set up your access script in a while loop so that when it aborts due to an error it will resume after x amount of time.
2) Minimize the number of indexes (if you had to recover a corrupted database check for indexes in your field definitions, FileMaker likes to create them for you).
3) Minimize the number of calculated fields and summary fields.  Only use what is absolutely necessary.

Written by missaugustina

October 29, 2007 at 1:03 pm

Posted in Perl, Programming

Python Game Development on Linux

leave a comment »

I’m currently curious about game programming (and artwork).¬† While I know C++ I am also interested in exploring viable Python alternatives.¬† Python is an easy to use language that would enable rapid prototyping, and I think it’s easier to read and maintain in the long run.

The options for Python game development I found are as follows:  <a href=http://www.pygame.org>pygame</a>, <a href=http://www.crystalspace3d.org/main/CELstart>CrystalSpace3d CELstart</a>, and <a href=http://www.ogre3d.org/wiki/index.php/PyOgre>PyOgre</a>.

Pygame is mainly targeted at 2D animation which is ideal for simpler games or just starting out.

PyOgre’s instructions indicate it will take 3-4 hours possibly over several days to compile and install on Linux (but they have a self extracting exe for windows developers).

CrystalSpace3d CELstart can be downloaded (and updated) via apt for Debian users  and has support for Blender via Blender2crystal (also available via apt).

The easiest one to get started with is definitely CrystalSpace3d.¬† Do not, I repeat, do not install ANYTHING before following the HOWTO.¬† If you already have CrystalSpace3d installed you might want to uninstall it then reinstall everything through the Blender2crystal-sdk.¬† I had a dependency problem and that’s how I resolved it.

There is a great HOWTO here for Debian packages:
<a href=http://b2cs.delcorp.org/index.php/Debian_Packages>http://b2cs.delcorp.org/index.php/Debian_Packages</a&gt;

You can download some CEL examples from the CrystalSpace 3d site.¬† To run the examples, type “celstart xxx.celzip” from the directory where you downloaded the zip file.¬† Press ESC to exit.

Blender is a little quirky… no window frames or borders, and it’s full screen right away.¬† fun.. ūüėõ¬† To make it usable I turned on auto-hide for the top and bottom toolbar space area.

My research for game development in Python on Linux has only just begun, but at least I’ve got the tools configured (I hope..).

Written by missaugustina

April 10, 2007 at 6:03 pm

Inserting Line Breaks Into Text

with 2 comments

This will force a line break into every 10th character.

$count = 0;
$_ = '1PATonthebackbackpatonthefoot2PAT3PAT4PAT';

s{
.
}{
if ( (++$count %10) == 0) {
$& . "\n";
} else {
$&;
}
}gex;

print;
print "\n"

$_ can also be a specified scalar

$count = 0;
$string = '1PATonthebackbackpatonthefoot2PAT3PAT4PAT';
#$_ = $string; < ---another option

$string =~ s{
\S
}{
if ( (++$count %10) == 0) {
$& . "\n";
} else {
$&;
}
}gex;

print $string;
print "\n"

This was courtesy of the Perl Cookbook!

Written by missaugustina

April 6, 2007 at 6:04 pm

Posted in Perl, Programming