Programming

Meant to publish this earlier, but it’s just been sitting in a tab in my browser for a day now. Anyways, John Resig (hm, familiar domain name / real name combination there…) has a post about Orto from Japan, a compiler that transformers Java bytecode to JavaScript statements. The end result is Java applications that do not depend on Java, but can be run with just JavaScript available in a browser. Not quite sure what I’d want to use this for, but on a scale for awesome, this ranks pretty high! John’s page has screenshots and links to several more interesting sources about Orto and is well worth the read.

Marco has a neat list up with five different features about PHP5 which people are still not quite catching on to, and I agree with every single item that made it to the list. I’ve been using SimpleXML myself, and except for a few cryptic issues regarding namespaces and iterators (SimpleXML does quite a bit of magic..) it’s a breeze to work with. For simple .. XML .. parsing, it’s ingenious.

For PDO, we’ve already moved all our projects to PDO, and it’s been my preferred method of accessing databases for at least a year and a half already. Great stuff. We’ve also used the json module for quite some time, which neatly ties into jQuery, mootools and other JavaScript APIs. I still haven’t used SPL that much, but that might change soon. Anyways, a good read for anyone who still live in the PHP4 world..

Mikko is quite active with the development of the Imagick-extension in PHP, possibly the best thing to hit PHP since it’s birth over ten years ago. There’s nothing like the Imagick-extension to make you realize how utter crap the GD extension really is :-) Anyways: Mikko has a new post up about how to use an image as a fill pattern in Imagick under PHP. Well worth a read!

You should also check out all the other interesting Imagick related posts Mikko has made.

I’m not sure how much sugar Christer got into his system today, but he’s been completely on fire with his blog posts. This one shows how to write a custom Zend_Form_Decorator_Label, which he wrote after someone asked a question about how to do something in particular on the Zend MVC mail list. That guy is amazing.

XRPC_Server is a simple as possible XML-RPC server component I wrote for a project a year or two ago, and is a good alternative if you want to try to stay away from large frameworks or complex components. The component is license under the MIT License, so you’re pretty much free to do whatever you want with it. All patches are welcome! The server requires PHP5 and reflection enabled.

Usage is as simple as requiring the php-file into your “gateway” page (the URL you’ll be calling from your XML-RPC clients), and then creating the server object with the functions you want to expose as the argument to the constructor:

A simple file illustrating the usage is also included, XRPC_Server_Test.php.

You can see the source code of the server at XRPC_Server.phps.

The class can be downloaded from this site: XRPC_Server.tar.bz2

Christer has obviously been having a good day today too, as he’s actually made yet ANOTHER blog post about something related to Zend Framework: Using Objects with Zend_View’s PartialLoop Helper. This time he’s examining why some methods suddenly has disappeared in his PartialLoop. I’m a bit surprised his father hasn’t helped him with his view scripts, tho. It’s probably just something he’s saying to don’t appear too incompetent.

«Estimate» stumbled across this awesome page with different string metric algorithms earlier today. Here you’ll find descriptions and implementations of Hamming distance, Levenshtein distance, Needleman-Wunch distance, Smith-Waterman distance and dozens other. Invaluable if you’re ever going to need to compare strings against each other and need some way to measure their similiarity.

I finally found out why Christer had been so quiet all day: he’s obviously been writing the largest post seen in the history of blogs. His introduction to Translating Zend Form Error Messages is enormous and a giant of a beast, and will give a thorough introduction to the concept of using Zend_Translate together with Zend_Form to use resource files to present an user interface in several localized versions.

While working with the facets for our search result earlier today, I came across the need to limit the search against solr on one specific field in addition to our regular search string (which we run against several fields with different weights). The situation was something like this:

• Lastname
• AggregateSearchField
• AggregatePhoneticSearchField

We do the searches against the AggregateSearchField and the AggregatePhoneticSearchField, where we weight the exact match higher than the phonetic matches. This ensure that the more specific matches are ranked higher than those that are merely similiar. We do this for several different field groupings, but that’s not revelant for this post, so let’s just assume that these are the three fields relevant. We search against two of them, and uses Lastname as a facet / navigator field to allow users to get more specific with their search.

However, while users should be allowed to get more specific with their search when selecting one of the facets, it should not change their regular search. And since the dismax handler will search through all the allowed field for a given value, you cannot just append Lastname:facetValue to the search string and be done with it (dismax does not support fielded searches through the regular query). After a bit of searching through our friends over at Google, I finally stumbled across the solution (which I of course should have seen on the Solr wiki): use the fq-parameter. This allows you to submit a “Filter Query“ in your request, which will be used to further filter your existing query through another set of queries. This fits very neatly in with keeping your original query and then appending filter queries for each facet limitation that gets set.

A small code example for Solrj: (filterQueries is a HashMap<String, String> which contains the facets; filterQueries.put(“Lastname”, “Smith”) will add a limitation on the field “Lastname” being “Smith” (you might want to escape “-s in the facet values)):

So now we can just parse the query string for valid facet limitations, and set the fields in the filterQueries HashMap accordingly. As we already have a list of facet fields to include, this is a simple as iterating that list and checking for the parameters in the request variables.

A great thank you to Mike Klaas in the dismax and Plone thread indexed by nabble.com that sent me in the right direction.

High Scalability has a neat post today highlighting a recent presentation given by Jeff Dean from Google at this year’s Data-Intensive Computing Symposium. The presentation named “Handling Large Datasets at Google: Current Systems and Future Directions“ (video (hosted by Yahoo!) ) (slides) dips into quite an amount of issues and thoughts about what it takes to run something handling petabytes of data. the video of the presentation

I’ll leave you with a quite interesting list shown in slide 8 (of 58) under the title of “Typical first year for a new cluster“:

• ~0.5 overheating (power down most machines in <5 mins, ~1-2 days to recover)
• ~1 PDU failure (~500-1000 machines suddenly disappear, ~6 hours to come back)
• ~1 rack-move (plenty of warning, ~500-1000 machines powered down, ~6 hours)
• ~1 network rewiring (rolling ~5% of machines down over 2-day span)
• ~20 rack failures (40-80 machines instantly disappear, 1-6 hours to get back)
• ~5 racks go wonky (40-80 machines see 50% packet loss)
• ~8 network maintenances (4 might cause ~30-minute random connectivity losses)
• ~12 router reloads (takes out DNS and external vips for a couple minutes)
• ~3 router failures (have to immediately pull traffic for an hour)
• ~dozens of minor 30-second blips for dns
• ~1000 individual machine failures
• ~thousands of hard drive failures

This year’s version of the unconference PHP Vikinger is taking place 21st of June in Skien, Norway. Derick has just opened up the registration which involves using high tech methods such as an E-mail-client and writing your name and other relevant information. One thing’s for sure, I, Eirik and Christer are heading out, and hopefully we’ll get a few more friends to join in.. and that includes YOU!

This year’s PHP-related projects for Google’s Summer of Code was announced today. Interesting projects include the unicode-implementation for PHP6, improvements to PECL, optimizations, Xdebug and several others. In short words: yay!

I especially find the project for creating a LLVM-extension for Zend interesting.

As Solrj – The Java Interface for Solr – is slated for being released together with Solr 1.3, it’s time to take a closer look! Solrj is the preferred, easiest way of talking to a Solr server from Java (unless you’re using Embedded Solr). This way you get everything in a neat little package, and can avoid parsing and working with XML etc directly. Everything is tucked neatly away under a few classes, and since the web generally lacks a good example of how to use SolrJ, I’m going to share a small class I wrote for testing the data we were indexing at work. As Solr 1.2 is the currently most recent version available at apache.org, you’ll have to take a look at the Apache Solr Nightly Builds website and download the latest version. The documentation is also contained in the archive, so if you’re going to do any serious solrj development, this is the place to do it.

Oh well, enough of that, let’s cut to the chase. We start by creating a CommonsHttpSolrServer instance, which we provide with the URL of our Solr server as the only argument in the constructor. You may also provide your own parsers, but I’ll leave that for those who need it. I don’t. By default your Solr-installation is running on port 8080 and under the solr directory, but you’ll have to accomodate your own setup here. I’ve included the complete source file for download.

The next thing we’re going to do is to actually create the query we’re about to ask the Solr server about, and this means building a SolrQuery object. We simply instanciate the object and then start to set the query values to what we’re looking for. The setQueryType call can be dropped to use the default QueryType-handler, but as we currently use dismax, this is what I’ve used here. You can then also turn on Facet-ing (to create navigators/facets) and add the fields you want for those.
Then we simply query the server by calling server.query, which takes our parameters, build the query URL, sends it to the server and parses the response for us.This result can then be fetched by calling .getResults(); on the QueryResponse object; qr.We then output the information fetched in the query. You can change this to print all fields or other stuff, but as this is a simple application for searching a database of names, we just collect the first and last name of each entry and print them out. Before we do that, we print a small header containing information about the query, such as the number of elements found and which element we started on. After this we output the facets and their information, just so you can see how you’d go about fetching this information from Solr too:
And there you have it, a very simple application to just test the interface against Solr. You’ll need to add all the jar-files from the lib/-directory in the solrj archive (and from the solr library itself) to compile and run the example.

Download: SolrjTest.java

While working with a view of a collection of documents returned from Solr using Solrj earlier today, I was attempting to write out the number of documents found in the search. In pure Java code you’d just request this by just calling .getNumFound() on the SolrDocumentList containing your documents, which whould also mean that they should be available through EL in JSTL by calling ${solrDocumentList.numFound} (which in turn calls getNumFound() in the SolrDocumentList object). The code in question was as simple as:

Which resulted in this error message, which kind of came as a surprise:

java.lang.NumberFormatException: For input string: “numFound” at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48) at java.lang.Integer.parseInt(Integer.java:447) at java.lang.Integer.parseInt(Integer.java:497)

After digging around a bit and reading the error message yet again, it suddenly hit me: $solrDocumentList was being interpreted and casted to a List, and as such, EL expected an index into the List instead of my call to a function. I’ve not been working with JSTL for too long, so I thought a bit about how to solve this. One solution would be to do the calls in the Action and then just map them to separate variables in the template, but this wasn’t really as pretty as it could be. Instead I wrote a simple wrapper around the SolrDocumentList, which is not a list in itself, but exposes all the elements through it’s getDocumentList-method. That way we can access it in the template by calling ${solrDocumentList.documentList…}.

I’ve included the simple, simple wrapper here. It should be expanded with access to Facet fields etc, but this should be a simple indicator of my suggested solution.

Any comments and updates are of course as always welcome.

My good friend Christer has written a simple tutorial on how to write a custom validator for a Zend_Form_Element. If you’ve ever laid your hands on Zend_Form, you’ll want to have a look at this for a short and concise introduction to the topic. He’ll show you how to create a “repeat the password”-field by creating a custom validator and hooking it onto the original password field. Neat stuff.