PyQuery brings jQuery selector syntax to Python. If you’re already familiar with jQuery selectors you’ll be right at home. Even if you don’t its easy enough to pick up.

Here’s a quick example:

from pyquery import PyQuery as pq

d = pq(url="http://www.straw-dogs.co.uk")
# d is now like the jQuery $

for post in d('div.post'):
	p = pq(post)
	print p('h2').text()

Get it here.

PyQuery is also available to install via “pip”.

…Ruby. One thing I envy in Ruby is the simplicity of so many of their libraries – both core and 3rd party. Python’s urllib2 never really felt natural but here is a solution. Requests: “HTTP for humans”.

An example will do it justice:

r = requests.get('http://www.straw-dogs.co.uk')
r.content

Wow – clean, simple and quick.

Add to that the awesome logo and you have a winner.

Everyone Has a Mac

As a web developer/coder I used Windows and Linux (mainly Debian, Arch & Mint) for everything. I switched to a Macbook Pro in 2008 (the first unibody model) while at home.

When I first got it, I loved it. It was the best thing since getting my first computer as a kid. It made web development easier and all the major scripting languages I used were installed by default.

But now – 3 years later I’m having doubts and here’s why:

Cross platform GUI development is bad

If you want to create something in Objective C you’re fine and there’s bindings for all the major languages. However, if you want to create, say, a GTK app or QT app then you’re screwed. Especially if you want to do them in Ruby or Python. Want to use Python’s wxWidgets? ok – as long as you use 32bit mode. But this will cause problems when you package it for distribution. GTK – you’ll need the buggy, messy X11 server running on the host system. A GTK app on Windows and Linux ‘just work’ – to steal a favourite Apple phrase.

MacPorts is a Mess

I didn’t realise at first until I started getting numerous version conflicts between packages. MacPorts tries to make sure dependencies are kept in check but whenever I ran an update I hit numerous snags about active ports which required me to manually activate and deactivate things. There’s actually an FAQ entry about this but only for Perl. However, it doesn’t just affect Perl. I’ve had it happen with Python, Ruby, Lua, Perl and numerous others I’ve probably forgotten.

Lesson learnt – don’t fall into the trap of using MacPorts. You WILL regret it. It will take a while for you to notice the problems but they’ll come, especially if you want to do anything different and during upgrades.

The MacPorts alternatives aren’t much better

Although removing Macports solved a lot of problems there were still issues installing some libraries for Python and Ruby. The library would often recommend Macports because that was the easiest way to install it. Otherwise they’d provide difficult custom setup instructions that would invariably break numerous other things on your system. The whole phrase – “it just works” rapidly starts to lose its credibility after 2 hours trying to setup a Python library.

Too many versions and conflicts

There’s been so many times I’ve had the existing base installed software conflict with a MacPorts package or a binary install. I couldn’t understand why my dev web server wasn’t working. I was editing config files but nothing happened. After an hour of digging I discovered MacPorts had installed Apache as a dependency of some obscure library and I had that running instead of my usual web server – which died when it couldn’t use port 80. I’ve had the same issues with MySQL. I had the binary installed but it conflicted with a MacPorts install. I’ve got about 4 versions of Python and although virtualenv can make life easier its still a pain if you want to package anything. At one point I ran “python” on the terminal and it was version 2.5.5. Odd – I thought – Lion is meant to be 2.7. That explained that when I ran “easy_install xxx” I couldn’t access it on the Python shell. Because easy_install was installing to version 2.7 of “site-packages”. Not 2.5. Turns out “python” was pointing to the wrong version. Mac NEEDS the other Python versions for various internal things so you can’t get rid of them.

Of course I’m sure some will comment that I’m an idiot. I should have been more careful, more thoughtful, etc. However, what happened to: “it just works”? The fact is I was able to develop on a Linux box for years without having to obsessively keep track of versions and installs.

Its too similar to Linux/Unix – Some simple apps just don’t work

As an extension to the comments on GUI frameworks I mentioned I wanted to check out Deluge (a torrent client). Its built in Python so surely it’ll run? Oh yes – if you jump through numerous hoops, sacrifice a goat, pray to Beelzebub and eat a few virgins. The gods may allow you to run it fully. I wanted to use Deluge mainly to tinker with plugin development for it. I eventually used a Linux VM instead. The fact is, OSX is similar to Linux/Unix in many ways. Which is why so many Linux apps will run on it with very little change. But thats actually a hinderance. Its close but there’s just enough differences to make it really awkward to install many things. Unless you use Macports but as I’ve already pointed out that is a poisoned chalice destined to murder your sanity in the future.

Fuse/FTPFS

I’ve found it very useful to use FTPFS which enables me to link a local directory to a remote FTP location. So I can effectively edit remote code like it was local. Yes I’m well aware using FTP is frowned upon but we don’t always get a choice.

This was very easy on Leopard. Its extremely easy on Linux. However, on Snow Leopard MacFUSE broke. There’s a few new libraries trying to fill the gaping hole but none of them support FTPFS (last I checked a month ago).

I still love my Mac but I’m under no illusions that if I want to do anything remotely off the beaten path its a nightmare. I find myself switching to Linux VM’s or Linux servers to do coding work now. Because more often than not – Linux “just works”.

It details a few paths people can take to go form Python beginner to expert. Its one of the most insightful answers/postings I’ve come across on SO. Here’s the summary – link at the end:

Discover list comprehensions

Discover generators

Incorporate map, reduce, filter, iter, range, xrange often into your code

Discover Decorators

Write recursive functions, a lot

Discover itertools and functools

Read Real World Haskell

Rewrite all your old Python code with tons of higher order functions, recursion, and whatnot.

Annoy your cubicle mates every time they present you with a Python class. Claim it could be “better”

Implemented as a dictionary plus some functions. Embrace functional programming.

Rediscover the Strategy pattern and then all those things from imperative code you tried so hard to forget after Haskell.

Find a balance.

I really don’t agree with the Haskell bit – I hate Haskell more than cancer. Obviously my ‘reasoning’ is well thought out and completely unbiased by my bastard of a lecturer at university who had anger management issues and a fundamental lack of people skills. I hate Haskell.

You may also find this post I did previously which lists a number of places you can get practice coding small apps.

Full post at Stack Overflow.

The quickie backstory:

• I suggested Python is better than Ruby.
• They argued using two example pictures – see below.

Rubyists

I Said:

The Ruby pic: The guy on the right has his eyebrow raised as if to say: “I’m fucking amazing”. The rest of them are all geeks who are using Ruby as their way to be cool in a world that rejected them at high school age for being geeks. Little do they realise (maybe deep down they do) but the only people who may think they’re cool is a small fringe group of other Ruby-ists and beginner coders. The other Ruby-ists will be consumed by an undertone of competitiveness. The Python coders will think they’re rubbish fanboys. The hot girl who rejected them at high school will think they’re just like any other geek.

Pythonistas

I Said:

The good thing about the Python coders is they’re not bothered. They’re just not even trying. They’re so past giving a shit about being cool that they’ve all grown ugly facial hair.

This piercing psychological insight into the programmers of Python and Ruby is just the tip of the iceberg. For $200 I’ll furnish you with a graph showing my facts in a nice way that will go well in a professional report.

It looks like I’ve found one – and awesome it is too!

wtop is by far the best I’ve come across of the options available and I’m not just saying that because its Python based. It nicely formats based on a variety of rules you give to it.

I plan on using it to monitor when certain IP’s access my server then hooking it upto an SMS API to text me when it happens. You can use it for monitoring of dodgy behaviour such as accesses to Paypal folders and such.

Some important points though:

1. Make sure your log format is configured properly. You need to add the %D option which the default “combined” format doesn’t have.
2. After changing the formats and making sure the wtop config matches your new Apache log format you may need to delete or archive your old ‘badly’ formatted logs. As wtop won’t cope well with the new/old format mix.
3. Install it – don’t think it’ll work straight off the bat.
4. Read the instructions.
5. To make sure you’ve got it working run a very basic command with it that does no filtering.

   logrep -o 'class,max(bytes),avg(msec)' access.log

6. More info on using logrep for monitoring your Apache logs is here.

Some More Examples…

• Show all 404 page errors in log with a count:
  

  logrep -m grep -o 'status,count(*),url' -f 'status=404' access.log

• Show visits of a certain IP to the homepage of the site:
  

  logrep -f 'ip~192.29.29,class=home' -o hour,minute,ip,url access.log

It uses Java – of a sort. More specifically it uses the Dalvik VM which is a register based as opposed to stack based VM and has many of its core classes based on Apache Harmony.

(Did I sound like I knew what that meant? Because I didn’t. I was paraphrasing Wikipedia – I love Wikipedia’s ability to make you sound like an expert at anything)

That’s why I hate it – the Java part. I hate Java. I’ve not developed in Java since university and have conducted a mental cleansing of all my painful memories of the language. As such I am a desperate ‘newbie’ on this platform. Scrabbling round like some lost raccoon on the North York Moors (there are no raccons in England so it would be VERY lost).

ASE comes to rescue the day but first….

Why ASE? Android programs are ‘easy” enough to write.

This is in the wiki docs of ASE. This question gets asked – apparently quite frenquently.

All I can say is: “Holy mother of jesus-bloody-ballsing-christ”.

What I find even more shocking is the answer…

That’s very true. Android’s development environment does make life pretty easy.

PRETTY EASY?!?!

WHAT?!?!

Are you a masochist??

It took an hour just to get it all up & running. Getting Eclipse working how its meant to with the right plugins, setting up emulator images, downloading tutorials and the multitude of other faffing tasks.

If you’ve gotten used to scripting languages like Python, Ruby or PHP then programming Java is akin to stabbing your eyeballs repeatedly with a rusty fork. Its rammed full of verbose bullshit, silly configs, insane amounts of setup, strict typing, misery, self abuse, pain, kitten torture, puppy genocide, etc, etc.

For example try taking a look at the Android Dev tutorial. Part 1 will consume 6hrs of your life. After which you will have created a useless ‘notepad app’ that doesn’t allow you to edit or delete notes. Only create a new one with hard coded contents and a name. 6 hours and I could have built Athens Online – an online city of dreams and Greek pornography. What? Everything online is porn.

Anyway…

Java Can Come Later

The Android Scripting Environment is a breath of fresh air though. This was released in May/June and allows development of quick, funky scripts in Python, Lua or BeanShell (*hiss*, *spit*). Ruby and Javascript are to follow soon.

If you fancy developing the scripts on your PC and running them on the phone you’ll need a tool available in the Android SDK called ADB (Android Debug Bridge). Though if you’re following the ASE docs don’t forget you need to turn on USB Debugging on the phone to enable it.

What It Can’t Do – Yet

I love ASE and I love the German gent who made it even more – even though I know only his name. I’m sure that he’s a great man for only a great man would create ASE.

Okay – I’ll stop now.

ASE can’t do the following at the moment:

• Compile in to APK/packages for selling or distributing on Market or elsewhere.
• Its restricted to a subset of the Android API made available by an RPC interface.
• It can’t cure cancer.

I’ve got high hopes for ASE and its making my life more fun. If you’re a Python/Lua/scripting sort then you’ll love it too.

Django Debug Toolbar

This looks very sexy – almost as sexy as DjangoPony.  Its a debug toolbar that was inspired by the PHP Symphony project.  Hold on now, don’t run away just cos I mentioned PHP.  This is pure Python goodness and its damn useful too.  It provides a load of debug info such as headers, exec time, SQL queries, variables, etc.

Its a MiddleWare app you can install into your Django installation.  Methinks I’ll give it a go at the weekend and report back.

Get it here.