Yuval Baror’s Blog

The natural process of erosion affects not only the organic elements of the Earth, but also the hardware we use, the code we write, and the databases we keep. When it comes to maintaining databases, there are several things that can be done to fight the degradation in query run time, and the increase in disk usage. There is one such action that is especially useful in PostgreSQL DBs and this is the command REINDEX INDEX. This command works on a single index and rebuilds it from scratch. Along the way it frees up any unnecessary disk space used by the index, and improves the response time of all queries using the index.

How much can this help? The answer is: a lot. In one example I witnessed reindexing an index improved the run time of a query using that index from several minutes to a few seconds. In another example, reindexing all of the indexes in a DB reduced the disk space used by that DB by over 40%.

So how can it be done? Simple:
REINDEX INDEX <index_name>;

Now assume that you want to reindex all of the indexes related to a list of tables in your database, all you need to do is create a script that extracts the names of the indexes in your DB that are related to these tables and then run the REINDEX command on each of them. Here is an example bash script that does just that:

#!/bin/bash

database=mydb
indexes=$(psql -q -t -A -P pager=off -c ”
select
c.relname
from
pg_catalog.pg_class c
join pg_catalog.pg_namespace n on n.oid        = c.relnamespace
join pg_catalog.pg_index i     on i.indexrelid = c.oid
join pg_catalog.pg_class t     on i.indrelid   = t.oid
where
c.relkind = ‘i’
and n.nspname not in (’pg_catalog’, ‘pg_toast’)
and pg_catalog.pg_table_is_visible(c.oid)
and t.relname in (’table1′, ‘table2′, ‘table3′)” $database)

for index in $indexes
do
echo `date “+%y-%m-%d %H:%M:%S”`” – Reindexing index $index”
psql -c “REINDEX INDEX $index” $database
done

echo `date “+%y-%m-%d %H:%M:%S”`” – Done”

One of the most annoying parts of web development is making sure your website looks the same on all browsers. A few days ago I noticed that on one of my websites the fonts are different in IE and in Firefox. I tried everything I could find on the web to make the fonts look the same – setting the font-family, font-size, font-weight, font-style, font-face, and every other aspect of the font. But the fonts still looked different.

Then I read about ClearType – Microsoft’s subpixel rendering technology that is turned off by default in Windows XP, but turned on by default in IE7 and IE8. This means that when using Windows XP, the same font will be rendered differently when viewed in IE and in Firefox.

To enable ClearType go to the Control Panel, then select Display, select the Appearance tab, and click on the Effects button. The second drop-down is labeled “Use the following method to smooth edges of screen fonts“. Change the selection from “Standard” to “ClearType“. Now any web page you load will be rendered using ClearType.

Enabling ClearType will eliminate the difference between the fonts in the different browsers. However, this will also change the way fonts are rendered throughout the whole operating system. To date there is no way to enable ClearType only in specific applications.

Here are some samples of pages before enabling ClearType and after enabling it:

Before

IE8 - Before ClearType

Chrome - Before ClearType

Firefox - Before ClearType

IE6 - Before ClearType

_____________________________________

After

IE8 - After ClearType (No change)

Chrome - After ClearType

Firefox - After ClearType

IE6 - After ClearType

The Bugzilla GUI doesn’t support adding a new version to all products. I used to add new versions to each product one by one. This got very tedious and annoying, especially as the number of products grew. I searched online for solutions provided by others and found only one thread on the topic on Nabble and on the Bugzilla support forum. In these threads the only suggested solutions were to use SQL on the Bugzilla DB, or to use custom fields instead of the built-in version field.

None of these solutions are satisfactory in my opinion, so I decided to create a better one. I created it in the form of a Perl script using the Bugzilla Perl API. The script can be run on the machine on which Bugzilla in installed and is really easy to use. Just put it in the installation directory of Bugzilla and run:

AddBugzillaVersion.pl <version_name>

Here is the script itself:

AddBugzillaVersion.pl

Notes:

1. The script must be in the Bugzilla installation directory in order to work properly. Alternatively – you can put the script anywhere on the machine and add the Bugzilla installation directory into the Perl library path.
2. The script uses the Bugzilla API, which requires several Perl modules to be installed. These are usually installed by the install-module.pl script, but if any are missing – they can be installed from CPAN. Instructions on installing CPAN modules can be found here.

The other day I wanted to create a simple bash script that would receive the name of a process and print out its pid. I wanted to run ps and then use awk to filter out the correct process and print out its pid. So I created the following bash script (called getPIDOf):

#! /bin/bash

command=”ps -o pid,command -C $1″
for (( i=1; i<=$#; i+=1 )); do
curval=${!i}
command=”$command | grep \”$curval\”"
done
command=”$command | awk ‘!/awk/ && !/getPIDOf/ {print \$1}’”

echo $command
$command

I was quite pleased with myself that I had even added in the option to specify multiple identifiers for the process. But when I ran the script I got the following output:

yuval@obt:$ ./getPIDOf nc
ps -o pid,command -C nc | grep “nc” | awk ‘!/awk/ && !/getPIDOf/ {print $1}’
ERROR: Garbage option.
********* simple selection *********  ********* selection by list *********
-A all processes                      -C by command name
-N negate selection                   -G by real group ID (supports names)
-a all w/ tty except session leaders  -U by real user ID (supports names)
-d all except session leaders         -g by session OR by effective group name
-e all processes                      -p by process ID
T  all processes on this terminal     -s processes in the sessions given
a  all w/ tty, including other users  -t by tty
g  OBSOLETE — DO NOT USE             -u by effective user ID (supports names)
r  only running processes             U  processes for specified users
x  processes w/o controlling ttys     t  by tty
*********** output format **********  *********** long options ***********
-o,o user-defined  -f full            –Group –User –pid –cols –ppid
-j,j job control   s  signal          –group –user –sid –rows –info
-O,O preloaded -o  v  virtual memory  –cumulative –format –deselect
-l,l long          u  user-oriented   –sort –tty –forest –version
-F   extra full    X  registers       –heading –no-heading –context
********* misc options *********
-V,V  show version      L  list format codes  f  ASCII art forest
-m,m,-L,-T,H  threads   S  children in sum    -y change -l format
-M,Z  security data     c  true command name  -c scheduling class
-w,w  wide output       n  numeric WCHAN,UID  -H process hierarchy

I thought that I must have something wrong with the escaping. But when I ran the command that was printed out, it worked fine:

yuval@obt:$ ps -o pid,command -C nc | grep “nc” | awk ‘!/awk/ && !/getPIDOf/ {print $1}’
892

I even tried the following simplified script:

#! /bin/bash

command=”ps -o pid,command -C $1″
command2=”awk ‘!/awk/ && !/getPIDOf/ {print \$1}’”

echo “$command | $command2″
$command | $command2

And got the following output:

yuval@obt:$ ./getPIDOf nc
ps -o pid,command -C nc | awk ‘!/awk/ && !/getPIDOf/ {print $1}’
awk: 1: unexpected character ”’

By this time I was quite puzzled. My simple getPIDOf script had become full of strange behaviors. So I decided to take the quick route out and use Perl instead of awk.

Here is my final version of the getPIDOf script:

#!/usr/bin/perl -w
use strict;

die “Usage: $0 <identifier1> [identifier2] …\n” unless (@ARGV);

my $command = “/bin/ps -o pid,command -C \”$ARGV[0]\”";
my $output = `$command`;
foreach my $line (split(/\n/, $output))
{
if ($line =~ /^\s*(\d+)\s+(.+?)$/)
{
my $pid = $1;
my $command = $2;
my $found_mismatch = 0;
foreach my $filter (@ARGV)
{
if ($command !~ /$filter/)
{
$found_mismatch = 1;
last;
}
}

if ($found_mismatch){next;
}

print “$pid\n”;
}
}

And the output, as desired, is:

yuval@obt:$ ./getPIDOf nc
892

My previous posts on this topic part 1 and part 2 were posted about half a year ago. Since then SWIG version 1.3.40 has been released and it includes several enhancements in the C# wrapper for std::vector and std::map based on the work that I did, but with several great improvements implemented by William Fulton, the person in charge of the C# wrappers in SWIG.

Since version 1.3.40 wrapping std::vector and std::map in C# is extremely simple. I’ll give a quick overview:

Wrapping std::vector

Lets assume you have a C++ class called MyClass with a method std::vector<int> GetIntVector(); You want to wrap MyClass from  C++ to C# and have the GetIntVector method return an IList<int> in C#. In order to do this, your swig interface file (the .i file) should look something like this:

/* File : MyProject.i */
%module MyProject

%{
#include “MyClass.h”
%}

%include “std_vector.i”
%include “MyClass.h”

%template(Int_Vector) std::vector<int>;

The C# class Int_Vector will implement the IList<int> interface as desired.

Now lets make things a bit trickier by introducing a new class called MyItem and lets add a method to MyClass with the following signature: std::vector<MyItem> GetItems(); In this case you’ll want the C# method to return an IList<MyItem>. One important issue that should be mentioned here is that any class being used in the IList<> generic must overload the == operator. Otherwise, it is only possible to use the IEnumerable<> generic interface.

So if MyItem doesn’t overload the == operator, and you’re content with having the C# method return an IEnumerable<MyItem>, then your swig interface file will look something like this:

/* File : MyProject.i */
%module MyProject

%{
#include “MyClass.h”
#include “MyItem.h”
%}

%include “std_vector.i”
%include “MyClass.h”
%include “MyItem.h”

%template(Int_Vector) std::vector<int>;
%template(Item_Vector) std::vector<MyItem>;

The C# class Item_Vector will implement the IEnumerable<MyItem> interface.

If, however, MyItem overloads the == operator, then you can have the C# Item_Vector implement the IList<MyItem> interface. You’ll need to tell swig that your class can be used with the IList<> interface by using the SWIG_STD_VECTOR_ENHANCED macro. Your swig interface file will look something like this:

/* File : MyProject.i */
%module MyProject

%{
#include “MyClass.h”
#include “MyItem.h”
%}

%include “std_vector.i”
%include “MyClass.h”
%include “MyItem.h”

%template(Int_Vector) std::vector<int>;
SWIG_STD_VECTOR_ENHANCED(MyItem)
%template(Item_Vector) std::vector<MyItem>;

Now the C# class Item_Vector will implement the IList<MyItem> interface.

Wrapping std::map

Thanks to some nifty features added to the SWIG core, it is now possible to wrap C++ std::maps into C# IDictionary<> generic classes very easily.

Lets assume you have a C++ class called MyClass with a method std::map<std::string, int> GetMap(); You want to wrap MyClass from  C++ to C# and have the GetMap method return an IDictionary<string,int> in C#. In order to do this, your swig interface file should look something like this:

/* File : MyProject.i */
%module MyProject

%{
#include “MyClass.h”
%}

%include “std_string.i”
%include “std_vector.i”
%include “MyClass.h”

%template(String_Int_Map) std::map<std::string, int>;

The C# class String_Int_Map will implement the IDictionary<string,int> interface as desired. It’s that simple. No need for any specialization macros or anything.

A few days ago I was trying to set up an offline installation of the FirePHPCore PEAR package on a computer that isn’t connected to the internet. This task wasn’t as easy as I thought it would be.

I downloaded the package (on a computer that has an internet connection) using:

1. pear channel-discover pear.firephp.org
2. pear download firephp/FirePHPCore

This gave me a file called FirePHPCore-0.3.1.tgz. Quite pleased with myself I transferred this file to the target machine and ran:

pear install -O FirePHPCore-0.3.1.tgz

This gave me the following error:

Parsing of package.xml from file “/tmp/pear/cache/package.xml” failed
Cannot download non-local package “FirePHPCore-0.3.1.tgz”
Package “FirePHPCore-0.3.1.tgz” is not valid
install failed

That can’t be good. I tried to fix this problem in several ways. I ended up with the following solution:

1. Extract the contents of the tar file using: tar xzf FirePHPCore-0.3.1.tgz. This creates a file called package.xml and a directory called FirePHPCore-0.3.1 with some files in it.
2. Edit the package.xml file and change the contents of the “channel” tag (line 4) from pear.firephp.org to pear.php.net
3. Recreate the tgz file using: tar czf FirePHPCore-0.3.1.tgz package.xml FirePHPCore-0.3.1/*
4. You are now the proud owner of a FirePHPCore-0.3.1.tgz file that can be installed offline.

Running pear install -O FirePHPCore-0.3.1.tgz on the new tgz file now results in:

install ok: channel://pear.php.net/FirePHPCore-0.3.1

Much better.

The Adaptive Communication Environment (ACE)  framework is an extensive set of cross-platform tools in C++. In this post I will describe in detail the steps required to set up the ACE framework on Windows using Visual Studio, and to set up a project using the ACE framework. The process is not complicated, but there are several pitfalls that one should be aware of.

Here we go!

Note: Steps 1-6 are partially covered in the ACE Build Guide. I give them here with some extra hints and tips.

1. Download the ACE framework from here. We will be using the full version of ACE.zip (last line under “Latest Micro Release Kit”).
2. Unzip this file into the directory where you want your ACE files to be located. I put my ACE files under C:\Programming\C++. Unzipping the file creates an ACE_Wrappers directory (C:\Programming\C++\ACE_Wrappers in my case). We’ll call this directory the ACE_ROOT directory.
3. Go to the ACE_ROOT\ace directory (C:\Programming\C++\ACE_Wrappers\ace in my case) and create a file called “config.h”. Edit this file with notepad or any other text editor and put the following text in the file:
   #include “ace/config-win32.h”
4. In the ACE_ROOT directory there are several solutions for different versions of Visual Studio and for different types of libraries. In this tutorial I will be using Visual C++ 2008 Express Edition (which is also called vc9) and building ACE as a static library. So the solution I  need is ‘ACE_wrappers_vc_9_Static.sln’. Select the solution that fits your needs.
5. Select your configuration (Debug or Release) and build the ACE project in your solution. In my case this is ‘ACE_vc9_Static’.
   
6. After a few minutes of building, you should have the ACE library file in your ACE_ROOT\lib directory. I built using the Debug configuration, so the file that was created is c:\Programming\C++\ACE_Wrappers\lib\ACEsd.lib
7. We are now ready to create our first ACE project! Create an empty C++ solution and then edit the project properties.
8. In “Properties”–”C/C++”–”General”–”Additional Include Directories” add the ACE_ROOT directory.
   

9. In “Properties”–”C/C++”–”Preprocessor”–”Preprocessor Definitions” set the following:
   ACE_AS_STATIC_LIBS
   _CRT_SECURE_NO_WARNINGS
   WIN32

   
   

10. In “Properties”–”C/C++”–”Code Generation”–”Runtime Library” set to Multi-Threaded debug (/MTd) for the Debug configuration and Multi-Threaded (/MT) for the Release configuration.
    
    
11. In “Properties”–”Linker”–”General”–”Additional Library Directories” add the ACE_ROOT\lib directory.
    
12. In “Properties”–”Linker”–”Input”–”Additional Dependencies” add ACEsd.lib for the Debug configuration and ACEs.lib for the Release configuration.
    
    
13. Your project is now configured to work with ACE. You can create your project code and start using the ACE framework. Following is a simple example program that runs a thread to perform some work.
    ACE Test Program

That’s all folks!

Perl modules are sometimes located in non-standard directories. There are several ways to use these modules:

• Add the directory to the PERL5LIB environment variable.
• Run the script using the perl -l.
• Within the script ‘use lib‘ with the directory name.

These methods are quite common and handle most of the cases perfectly.

For example:

#!/usr/bin/perl -w
use strict;

use lib (”/home/users/yuval/perl/modules”);
use MyModule;

…

This works fine as long as you know that you need to search in /home/users/yuval/perl/modules to find MyModule.

But what happens if you don’t know the location of the module ahead of time?
None of the methods mentioned above can handle this situation. Consider the following code:

#!/usr/bin/perl -w
use strict;

my $user = GetCoolestUser();
use lib (”/home/users/$user/perl/modules”);
use MyModule;

sub GetCoolestUser()
{
return “yuval”;
}

…

Running this code gives a bunch of really scary compilation errors.

Yes – compilation errors. Yes – in Perl. What gives you ask?

The reason is that the Perl compiler tries to locate all of the modules that are going to be loaded before running the script. It does this before values are given to any variables, and before any functions are evaluated. Therefore the $user variable is undefined when it is evaluated in the parameters to the ‘use lib‘ statement, and then MyModule isn’t found in any of the directories that Perl searches.

There are two ways around this:

1. Push the directory name directly into the @INC array in the BEGIN function

#!/usr/bin/perl -w
use strict;

BEGIN
{
sub GetCoolestUser()
{
return “yuval”;
}
my $user = GetCoolestUser();
push(@INC, “/home/users/$user/perl/modules”);
}
use MyModule;

…

You may notice that this code doesn’t use the elegant ‘use lib‘ but rather the tasteless pushing of the directory into the @INC array. Also – the GetCoolestUser function has to be defined within the BEGIN block in order to be used.

2. Use the eval function

#!/usr/bin/perl -w
use strict;

my $user = GetCoolestUser();
eval “use lib qw(/home/users/$user/perl/modules);
use MyModule;”;
if ($@) {
die “Error loading module MyModule:\n$@\n”;
}

sub GetCoolestUser()
{
return “yuval”;
}

…

This method “tricks” Perl compiler and allows loading the module from any location you wish. This method is stronger than the previous method mentioned because the module loading occurs during run time and the location can be calculated dynamically during run time as well.

I’ve just added a new problem visualization. The visualization is of the 8-Puzzle – a classic game in which you have a 3 by 3 board with tiles numbered 1 to 8, and one missing tile. Your goal is to bring the board to a state in which the tiles are ordered according to their numbers, and the empty spot is in the bottom right-hand corner. The only movements allowed are taking a tile that is adjacent to the empty spot and sliding it (horizontally or vertically) into the empty spot.

The visualizations added implement the DFS algorithm, the greedy algorithm and the A* algorithm.

You can find the visualization here, along with detailed information about the algorithms.