Getting Started with Devel::NYTProf

I went to OSCON 2010 and saw Tim Bunce’s talk on Devel::NYTProf. For my local Perl User Group (SPUG) I decided to do a talk on how to get started with NYTProf and to provide an overview of the benefits of code profiling. This entry is a summary of the talk that I gave.

What is Devel::NYTProf?

Devel::NYTProf is a source code profiler. It is a tool that gives you another perspective on how your code is functioning. Typically people use NYTProf to find places in their code that could be optimized to run faster, however it can also be used to better understand what is happening when your program executes.

Why should I use Devel::NYTProf?

If you’re working with unfamiliar code, NYTProf provides a high level overview on what parts you need to pay the most attention to. It breaks the source code down line by line and reports execution times for each block and subroutine. NYTProf highlights hotspots in your code which helps to identify priority areas that need improvement. NYTProf provides visualization tools that are not only useful for communicating to other folks who might not be as familiar with the source code, but are also useful for right-brained artsy-fartsy visually-oriented people like myself 😉

How do I use Devel::NYTProf?

1. Install the module Devel::NYTProf (I used CPAN)

Run the following command:

perl -d:NYTProf my-perl.pl

2. Set options to customize behavior

Command line:

 -addpid=1

Environment variable:

export NYTPROF=file=/tmp/nytprof.out

(See the POD for more info about custom options)

3. Format output

nytprofhtml

When you run NYTProf, it creates an output file in the same location where you ran it (unless you’ve specified something different in the options). To read the output, format it using NYTProf’s formatters. These are discussed in the POD.

If you are using the HTML formatter, by default it creates a folder called nytprof in the same directory you run it from. Open the index.html  file contained in that directory to view the formatted profile.

How do I read the output?

Once you’ve formatted the output into something human-readable, making sense of what’s being reported is the next step. Besides an HTML layout, there are visualizations of the code in Graphiz and a Treemap based on subroutine frequency and inclusive time.

The key terms to be aware of when profiling are Inclusive Time and Exclusive Time. Inclusive Time is the total time to run a subroutine including any additional subroutines being called within the subroutine. This is also referred to as the Cumulative Time in other profilers. Exclusive Time is the total time spent in this subroutine excluding calls to any other subroutines.

Another important thing to note about the reported times is that there are no hard and fast rules for what is considered the ideal time for any subroutine. Everything is relative. Look at things in red to see if the reported time makes sense given the nature of the data and function being performed. The hotspots only indicate what’s taken the most time in the program, it does not mean that something is necessarily inefficient.

Statement Profiling

NYTProf reports on blocks of code in addition to subroutines. It measures the time between entering one perl statement and entering the next.

Perl makes heavy use of additional system calls like regexes, operating sytem functions, and user input. NYTProf intercepts perl opcodes and reports them as subroutines. This allows you to evaluate whether a block of code was inefficient due to something within your code or if the system call is what’s slowing you down. If there is an alternative method to using that system call then you can see how much an improvement it makes by implementing it and re-running the profiler.

Use the -slowops=N option to customize how NYTProf reports opcodes.

Subroutine Profiling

NYTProf determines subroutine durations by measuring the time between entering a subroutine and leaving it. As discussed earlier, it accumulates both Inclusive vs Exclusive times for each subroutine. It also reports how frequently a subroutine was called both for the whole program and in a given location. The profile reports  separate values for each location where a subroutine was called.

Recursive Subroutines

NYTProf differentiates regular subroutines from recursive ones. It reports the Inclusive time for outermost call and also indicates that the subroutine in recursive. It also reports the maximum recursion depth.

Application Profiling

NYTProf is a fairly sophisticated profiler in that it can support threads and forks. For complex applications, it records extra info in a data file, including:

• Filename
• Line ranges of subroutines
• Detects forks

It also supports Apache Profiling with mod_perl hooks. See CPAN module notes for more info on how to hook into Apache.

A note about optimization

Optimization is not always necessary. You really have to ask yourself if the amount of time to execute something makes sense given the amount of work. If the task is pretty hairy, chances are you’re better off finding a different area to optimize.

A lot of small changes add up. If you can make a few small optimizations to subroutines that are called the most frequently, you can shave seconds off of your times.

Devel::NYTProf just finds the hotspots (kind of like SQL Explain). It really doesn’t know anything about how your program is supposed to work, it just gives you a bunch of interesting statistics 🙂

More Info

– Tim Bunce’s Devel::NYTProf Talk

– Devel::NYTProf POD

 

Advertisement

An Introduction to Object-Oriented Perl

I’ve been using Perl for a few years now and I am finally getting to a place where I’m working with more complex problems. Traditionally I’ve written little utility scripts or handlers for small specific tasks. In my current position, one of my roles is writing ad-hoc scripts for bulk reporting and bulk updates to a large number of database records for client requests. As I’m passing some of these duties on to my coworkers, I realized I needed a template to ensure consistency and to make their lives easier. I’ve also been tasked with a larger project for which I’ve decided the solution is to make a single Object-Oriented API in our code base that can be called by many different sources.

My original training is in Java/C++ and Object-Oriented Design. I wanted to approach both of these problems using my OO background, but I was having trouble grasping how OO really works in Perl. I’ve scoured the internet and poured through books, and none of them could really answer my questions. Finally, I sat down with someone recently who answered my “why why why” questions (and who also proofread this entry for accuracy! Thanks Dave!!). I decided I should document this here in hopes that it might help others who are in the same conundrum.

This article will be most useful if you already understand OO and you understand Perl but you just want to know from a bottom up perspective how OO works in Perl.

Key Concepts

Unlike languages like C++ and Java where Object-Orientation was considered during the language development, in Perl it was more of an afterthought.

Bless

Bless is a built-in Perl function that turns any reference into an object. While generally it’s good practice to use hash references, there are many differing schools of thought on Perl OO, including inside-out objects. These techniques are beyond the scope of this particular article (bad OO pun haha) so I won’t discuss them here.

There are 2 key differences between references and blessed references (aka objects). An object contains an OBJECT flag telling the interpreter that its an object. An object also contains a string with the name of its class so the interpreter can look up methods called on the object that are defined in its class.

When you instantiate an object in your executable Perl code, the Perl interpreter stores the class name in the object data. When a method is called on the object instance, Perl attempts to resolve the method at runtime. Using the class name, the interpreter looks for the method in the class definition. If it does not find the method, it looks in @ISA to see if the class is a child and if the method is defined in the parent class. If it does not find the method defined for the class, the interpreter throws a “Can’t locate object method” exception.

Here are examples using Devel::Peek to view what a reference and an object looks like:

First examine what a hash reference looks like:

perl -MDevel::Peek -wle 'my $not_an_object = {}; print Dump($not_an_object)'


Output:

SV = RV(0x90e492c) at 0x90e4920
REFCNT = 1
FLAGS = (PADMY,ROK)
RV = 0x90e4880
SV = PVHV(0x90ef340) at 0x90e4880
REFCNT = 1
FLAGS = (SHAREKEYS)
ARRAY = 0x0
KEYS = 0
FILL = 0
MAX = 7
RITER = -1
EITER = 0x0


Now examine what an blessed hash reference (aka object) looks like:

perl -MDevel::Peek -wle 'my $object = bless {}, "main"; print Dump($object)'


Output:

SV = RV(0x979d92c) at 0x979d920
REFCNT = 1
FLAGS = (PADMY,ROK)
RV = 0x979d880
SV = PVHV(0x97a8340) at 0x979d880
REFCNT = 1
FLAGS = (OBJECT,SHAREKEYS)
STASH = 0x979d770 "main"
ARRAY = 0x0
KEYS = 0
FILL = 0
MAX = 7
RITER = -1
EITER = 0x0


Notice the second FLAGS entry and the STASH entry. STASH contains both the string of the class name and the address where the string is stored in memory. FLAGS contains the value OBJECT which tells the Perl interpreter that $object is an object.

Intrinsic vs. Explict Methods

In Perl, a method does not intrinsically know what object or class it’s called on. When a method is called, the Perl interpreter either passes a reference to that object or the class name as the first argument. When writing methods that take arguments, the first argument passed in by the interpreter is the object reference or the class name depending on whether the method was called by an object or by the class itself. For instance, the new() constructor is called on the class so the first argument the interpreter passes in is a string containing the class name. For a method you define called by an object instance, the interpreter passes in a reference to that object.

While it’s good practice to call the first argument $self or $me, you can call it whatever you want.

Here’s are two examples of handling arguments in methods:

sub my_method {
my $self = shift;
my ($arg1, $arg2) = @_;
}

sub my_method {
my ($self, $arg1, $arg2) = @_;
}

Here’s an example of calling the method, for this example the class is “MyClass”.

my $my_class = MyClass->new();

$my_class->my_method($arg1, $arg2);

When calling a class method without an object instance, the first argument is a string representing the name of the class.

MyClass->my_method($arg1, $arg2);

Class Methods vs. Object Methods

In Perl, you don’t have to instantiate an object to call its class methods, so you have to provide your own enforcement if this is a possible risk.

When a method is called by a class, the first argument passed is a string containing the name of the class. When a method is called by an object, the first argument is a reference to the object. To enforce calling methods by object, check that the first argument passed is a reference.


die "Please call this method on an object instance." if (! ref $self);


Unenforced Privacy

There are no private methods in Perl except by convention. To designate a method as private, the general practice is to begin the name with an _ character. While this does not prevent use of this method in other contexts, it makes your code and your intentions easier to read.

sub _my_private_method {}

sub my_public_method {}

Data Attributes

One of the advantages to using a blessed hash ref for your object is that you can manage basic data attributes without using accessor methods or having to maintain a long list of variables. I believe accessor methods should be used under two circumstances: 1. when they add some value or additional functionality or enforceability with regards to data types and data values, and 2. when a data attribute needs to be modified external to the class definition. When dealing with simple values used internal to the class definition, maintaining a long list of variables and getter/setter methods is overkill. This is my opinion and I’m sure there are plenty of others out there that differ, so do what works best for the problem you are trying to solve. I’m just presenting an alternative that was shown to me that I’ve found to be pretty handy.


$self->{NAME} = $name;


Instead of having to create and maintain a get/set_name method, your object can access it’s own “name” with $myobject->{NAME}.

Garbage Collection

Once an object goes out of scope, the Perl interpreter calls DESTROY on it prior to dereferencing the memory. Unless your class has specific requirements for what needs to happen when the object is destroyed, you don’t need to implement the destroy method.

To implement the DESTROY method, define a method called DESTROY in all caps.

Perl does not automatically call DESTROY on super classes. To DESTROY a super class, you must explicitly call $self->SUPER::DESTROY.

How To

As stated in the previous section, there are many philosophies about how to do Object Oriented Perl. The intent of this article is provide basic instructions for getting started with writing a simple class and to explain some of the Perl internals. The reason for this is because I was having a hard time finding the information I needed and up-to-date examples. Once you’ve gotten a general idea of the basics, it’s worthwhile to explore different OO Perl philosophies.

1. Create a class file ending in .pm

For this example, create a file called MyClass.pm and save it where Perl can find it.

Define a new() method to bless your passed in reference and assign it to the class. Define methods as appropriate and return 1 at the end.


package MyClass;
use strict;
use warnings;

sub new {
# first argument is the class name
my ($class, %args) = @_;

# create an anonymous hash ref called $self
# bless $self to set it as obj with class name $class
# return obj ref
my $self = bless {}, $class;

# call a method on $self to set an initial value
if (defined $args{name})
{
$self->set_name($args{name});
# alternatively we can set the name using
# $self->{NAME} = $name;
} else {
die "Please specify a name for the $class instance.";
}

# return the obj instance
return $self;
}

sub set_name {
my ($self, $name) = @_;

if (! ref $self) # if called as a class method, $self is a string, not a ref
{
die "Please call this method on an object instance.";

} else {
# assign class data attributes to keys within the $self hash
$self->{NAME} = "My name is " . $name;
}
}

sub print_name {
my ($self) = @_;
print $self->{NAME}, "\n";
}

1;


2. Create a Perl executable file ending in .pl. Instantiate your class by creating a variable and using the new() method. Call a method on your class to see if it works!


#!/usr/bin/perl -w
use strict;
use MyClass;

# create an object by instantiating MyClass
my $object = MyClass->new(name => 'Eugene');

# call an object method on MyClass
$object->print_name();

# calling a class method returns an error
MyClass->set_name('Marsha');


Managing Ebooks on the Kindle

DefectiveByDesign and other Free Speech advocacy groups have been targeting the Kindle for its DRM limited content. This was something I grappled with when I bought my Kindle last year. Of course Amazon wants Kindle users to buy books from their store, but limiting ebook content really limits the usefulness of the Kindle! What if an ebook I want isn’t available through Amazon, but is available in another format through another seller? What about checking out ebooks from the library?

I initially was interested in ebook readers because I use the bus as my main mode of transportation. Also I love to read but I tend to forget to return books to the library and my collection of used books keeps growing. Living in the city, I don’t have a lot of space. If I could have all of my (non-picture) books as ebooks I would be very satisfied.

The great thing about the Kindle is that I’ve been reading A LOT more and it’s a fun device to use.

The not-so-great thing is that Amazon limits the way you can access ebooks DRM’ed by other providers. Fortunately there are ways around these limitations!

Download Free Ebooks in Mobi Format

There are non-DRM books available that the Kindle can read and it’s really easy to do. I use my SD card because I think cables are annoying. Insert the SD card into your computer and create a new folder called “documents”.

There are several free ebook sites on the web, in particular Project Gutenburg and Mobipocket Free Books. Pick a title and download it in “mobi” format. Save it to the “documents” folder you created on your SD card.

Insert the SD card into the Kindle. When you start your Kindle up, you will see the titles you downloaded as “new” items.

Convert Digital Files to Mobi Format

Kindle offers a service that will convert other ebook formats (HTML, PDF, etc) to Mobi format. This service is $.10 USD per book. Alternatively you can convert it yourself. Either use MobiPocket Reader to import the file (it is saved under “My eBooks” in your Documents folder) or download MobiPocket Creator. Note that there may be additional challenges with DRM’ed ebooks. Once converted, copy the files into the “documents” folder either on your Kindle or on your SD card.

Register Your Kindle as a MobiPocket Reader Device

The final workaround involves registering your Kindle as a MobiPocket Reader device. Not only can you read Kindle ebooks on your computer, but you can transfer them to other devices, and even transfer ebooks DRM’ed by other providers to your Kindle. For more detailed instructions on converting the Kindle files to MobiPocket format, see this article.

What you’ll need: A Kindle, an SD Card (or USB Cable to transfer files between your Kindle and your computer), ActivePython 2.x, MobiPocket Reader, Python Scripts (see below for where to get them)

Step 1) Get the Serial Number of your Kindle.

This is under the back cover on the label above the battery. The “Serial No.” is listed under the first barcode.

Step 2) Copy the ebook file from the Kindle to your computer.

Insert the SD card into your Kindle. Open the Content Manager and select a book on your list. Choose the option to save it to the SD card. Alternatively you can download files from your Kindle profile in Amazon to your computer.

Turn off the Kindle and insert the SD card into your computer. Your book is in a folder called “downloads”. I have a folder I created in “Documents” called “Ebooks”.

Step 3) Install Python and download the scripts.

Install Python 2.6 if you haven’t already. Note that the scripts are not compatible with Python 3.x. On Windows, I recommend ActivePython. Next download a small suite of Python scripts from the linked article above, under “Step 4”. Please let me know if the archive is unavailable so I can update this article.

When you extract the archive, you can place the scripts in the Tools\scripts directory of your Python folder and update the system path variable. I placed mine in the same folder I save my ebook files to for convenience. You need to run the scripts from the same location as the files you copied from your Kindle.

Step 4) Create the Mobi file

Open up a command line interface. On Windows type “cmd” either in “Run” (XP) or in the search dialogue of the Windows menu (Vista).

Navigate to the folder containing the files you copied from your Kindle:

cd Documents\Ebooks

Type the following command, replacing yourserialnumber with your serial number:

python kindlepid.py yourserialnumber

The script returns a 10-digit PID as the last item on the printed line. The third to last digit of the PID is an asterisk(*). Make a note of the PID.

Step 5) Open the book in Mobipocket Reader.

Install Mobipocket Reader if you haven’t already done so.

To register your Kindle as a device in Mobipocket Reader, select Reading Devices from the navigation on the left. Click the button located at the upper center part of the window to add your Kindle. I named my Kindle “Kindle” and chose “Other” for device type. Enter your PID in the field that asks for a MobiPocket ID. To open Kindle files in Mobi, change the extension of the “.azw” file to “.mobi”. Select Open or Import to view the file in Mobipocket Reader.

Double click the .mobi file you created to open it in Mobipocket Reader. Use Mobipocket Reader to register other supported devices so you can transfer your ebooks to them!

My First Day with the Samsung Reclaim

I’ve been using a Motorola Razr for the past 3 years and I’ve been less than satisfied. My former employer provided me with a Samsung Blackjack running Windows Mobile, so I had no reason to upgrade my personal phone. When I was laid off in June, I had to give up my Blackjack.

Based on my experiences with both the Razr and the Blackjack, I created a list of features I wanted in my new phone (in order of importance):

1. Qwerty keyboard
2. Text messaging and social networking focus
3. GPS / Google maps
4. Web browsing at least as good as the Blackjack running Windows Mobile

Deal-breaking features: Decent battery life and responsive OS (minimal lag/slowness).

I originally considered the Palm Pre because it covered most of those things and a lot more. I hesitated because of the battery life issue. I didn’t feel the price of the Palm was worth it because of the battery life issue and because it’s a 1.0 piece of hardware. Furthermore, I don’t need all the bells an whistles the Pre has to offer. While it’s “cool” it’s not worth the price tag to me when there are mid-tier phones that can fit my needs. Finally, I would have to change my current plan to accommodate the Pre.

I saw an advertisement on Instructables for the Samsung Reclaim. I had never heard of this phone nor seen it before, so I started reading about it. The Samsung Reclaim’s plastic casing is made from corn-based bio-plastic material (40%) and Sprint claims the phone is overall made from 80% recycled materials. I was under the impression that meant the phone also had parts made from recycled plastic, but they may be counting the packaging. I checked on both the Sprint and Samsung websites and I was unable to find a whitepaper outlining exactly what the recycled content of the phone is. Regardless, I liked the look of the phone and I like the direction Sprint is going with it’s eco-conscious image.

I went to the Sprint store to try the phone out and I was pleased. The Samsung Reclaim comes in both a bright green and a bright blue. Both are great colors, however I liked the green better. I wanted to test the QWERTY keyboard for usability and although small, it is very accurate. I have small fingers and am comfortable with small buttons. The phone slides out easily and I can hold it without accidentally pressing any other buttons (unlike the Razr). Because I felt the phone meets my basic requirements, I purchased the Samsung Reclaim.

The OS is Sprint’s own operating system that is similar to the Pre’s WebOS in look and feel. The button navigation and labels are similar to my Razr so I could easily figure out how to move around. Other aspects had the same labels and organization as the Blackjack so things I had figured out there also applied to this phone. I haven’t read the manual yet and already I’ve changed my screen pattern and ringtones, set up twitter and facebook, set up email and instant messaging, updated my contacts using bluetooth from my old phone, set up Google applications, taken photos including a panoramic, made phone calls, and sent text messages. For me, this phone is super easy to use.

While the Reclaim is not a smartphone like the Pre (in fact Gizmodo coined the term “dumbphone”), the Reclaim has enough applications and features to sufficiently meet my requirements. The up front cost was $100 plus tax and the mail-in rebate takes 6-8 weeks to process. I am very pleased with this phone and will write a further update once I’ve used it awhile to discuss how well it does at meeting the requirements I’ve listed above.

Defcon 17 Schedule Posted (+overview of Defcon 16)

I’m not really a “hacker” in the network security sense, nor am I really into network security.  I went to Defcon last year because a coworker who was planning to attend convinced me I would have fun.  I have other friends who attended in the past and enjoyed it.  I mentioned it to a few friends, they were interested, so we went!

I discovered that Defcon is not just for Internet Security Geeks!  There are different types of “hackers” not all of them computer related.  Defcon hosts a Lockpicking lounge and a Hardware Hacking lounge in addition to a variety of talks covering a variety of subjects, not all internet security related.

Ada Lovelace Day Post

Ada Lovelace is known as one of the first computer scientists. She developed a language and initial functions for the “analytical engine” Charles Babbage envisioned in the 1800’s. Ada Lovelace Day is an international day of blogging to draw attention to women excelling in technology. Several ladies have posted about famous female Computer Scientists that inspired them either to turn to Computer Science or as they were studying it. I guess I could list the usual Grace Hopper, Mary Lou Jepsen, Anita Borg……. but the truth is I didn’t really know much about any of these people until after I’d already graduated and was already on the path so to speak. So instead I’ll talk about how I strive to be a role model for other women 🙂

Installing Flex On Ubuntu

Install Eclipse

Do not install Eclipse from the Ubuntu repositories.  Navigate to http://www.eclipse.org/downloads/ and select Europa on the left-hand menu.  Download the Eclipse for Java download. Extract the .tar.gz file and move it to the /usr/lib/ directory. Create a symbolic link in /usr/bin to launch Eclipse.   “sudo ln -s /usr/lib/eclipse eclipse”  Also you can add a menu item under Programming to your applications menu.  Launch Eclipse to make sure the installation went smoothly.

Configure Sun Java

Follow this HOWTO: http://ubuntuforums.org/showthread.php?t=201378
Or follow the instructions to use your JDK of choice here – https://help.ubuntu.com/community/EclipseIDE

Install Flex
Once you’ve configured Eclipse, install the Flex plugin for Linux – http://labs.adobe.com/technologies/flex/flexbuilder_linux/
Download the bin file and from the command line, chmod+x to make it executable.  Run it from the command line by cd’ing to the directory where it’s downloaded and run it by prefacing the downloaded bin file with ./ .  Example: sudo ./flexbuilder_linux_install_a4_081408.bin

Be sure to run this install script with root privileges because the installation will need to access folders that will require it.  During the installation you’ll be prompted for 2 different directories, one for the Flex install and another for the Eclipse root folder.  I chose the /opt directory for Flex installation.  On Ubuntu, if you installed Eclipse via apt the Eclipse root folder will be at  /usr/lib/eclipse .  Do not use the defaults the installation script is prompting.

Create a Project

Launch Eclipse and select File > New > Other…

A dialog should pop up with a list of wizards.  Expand the folder for Flex Builder and select your project type.

Note that I was unable to get this working with Eclipse 3.4 .  If you have a version of Eclipse installed from the Ubuntu repositories I highly recommend you completely remove it and use the packages provided by Eclipse.org.

Vmware Workstation on Ubuntu Feisty 7.04

Installing VMWare workstation was somewhat painless on Ubuntu.  I only had access to an RPM package, so I had to download Alien in order to convert it to .deb.

I searched for “alien” in Synaptic and installed it through there.  Once I’d done that, I opened up my terminal, went to the location where I’d saved the RPM, and typed “sudo alien –scripts vmware….rpm” (insert the filename where the ….’s are, the suffix is .rpm).

Once I had my .deb package, I double clicked it to launch the package installer.  Everything installed properly.  Next I opened up a terminal window and typed “vmware” since I didn’t see it in the menu.  I was prompted to run a configuration Perl script.  At this point, depending on what you want to do with VMWare you might want to download the PDF manual: http://www.vmware.com/support/pubs/ws_pubs.html

After running through that (I just used the defaults for most things), I found VMWare in the Applications menu under “System Tools”.  VMWare was unable to start because it couldn’t find the EULA.  I dug around and found a link to a .gz file that must not have gotten decompressed properly.  I uncompressed it, and ran VMWare again, this time successfully.

VMWare requires you to have a copy of whatever OS it is you want to install, in my case I was installing the OEM version of Windows XP that came with my machine.  I selected to create a new Virtual Machine, Windows XP Professional, 8 gig.  I inserted my installation CD and clicked to “run” my Virtual Machine.  At this point I was prompted to enter my VMWare serial number.  I did, and the Virtual Machine started reading the CD.

The UI is very helpful and intuitive.  I didn’t have to consult the manual for anything.

Gutsy Gibbon.. to upgrade or not?

I upgraded my home computer but I did not upgrade my work machine.

My home machine is a bare bones Asus system with a P4 processor and an Nvidia 6800 GPU.  Major changes were Compiz Fusion (I had been running Beryl and had to re-set everything up) and VMWare workstation.  VMWare workstation needed to be reconfigured and wouldn’t run properly.  The issue is addressed here.

Using the linked utility I was able to get things up and running on the home system, but without official VMWare support for 7.10 I’m not going to be upgrading my work computer just yet.

Reconfiguring Compiz Fusion was a hassle.  I don’t have my cool splashy water effects anymore 😦  However, now I can add more sides to my cube, and I did finally figure out how to make my firey windows multi-colored again.

The upgrade process itself was SUPER EASY.  Just click the upgrade button on the software updates screen.  You’ll want to hang around for about the first 10-15 minutes to take care of any dialogues.  Also be sure to update everything in 7.04 before running the upgrade.

FileMaker ODBC Fetch Forward Error

After scouring the internet for this mysterious “Fetch Forward” error I sometimes get when migrating FileMaker records to other database formats via ODBC and Perl DBI, I discovered what causes it.

The error is seemingly random and goes like this:
[FileMaker][ODBC FileMaker Pro driver][FileMaker Pro]Unknown error (SQL-HY000)
[FileMaker][ODBC FileMaker Pro driver]An attempt to fetch forward has failed for table: TABLENAME (SQL-HY000)(DBD: st_fetc/SQLFetch err=-1) at myperlscript.pl line 123.

This occurs when FileMaker drops the ODBC connection or is unable to respond in a timely manner.

Solutions are as follows:
1) Set up your access script in a while loop so that when it aborts due to an error it will resume after x amount of time.
2) Minimize the number of indexes (if you had to recover a corrupted database check for indexes in your field definitions, FileMaker likes to create them for you).
3) Minimize the number of calculated fields and summary fields.  Only use what is absolutely necessary.