Posts Tagged ‘web development’

I’ve spent the last 4 days working on the problem of how to make a screenshot of a URL on a Linux server with no display. Furthermore, I need it to be programmatic, so I could call it from a web app when needed. Additionally, I wanted some architecture that could conceivably lend itself to threading in the future, so that we could scale the thing.

Essentially, all the solutions I found did this:

  1. Create a headless X server with something like Xvfb.
  2. Open some sort of browser on that in-memory X window.
  3. Screenshot the contents of that browser.
  4. Render that screenshot content out to some image format.

There are several ways to do this with names like html2image, khtml2png, webkit2png, etc. but they all had issues with either my platform or my need to not pay for it 🙂

That’s when I found the work of Roland Tapken. His script and explanation were the solution I needed. It made nice screenshots, had the configuration options I needed (screen size, scaling image, destruction of X server, etc.). I thank him deeply for his work and his sharing it with the world.

The only problem off the bat was that it requires at least 2 packages that were not installed for me on Ubuntu 7.10 (Gutsy). In fact, they aren’t available in the versions I needed until the most recent Ubuntu 8.10 (Intrepid). Those packages are libqt4-webkit and python-qt4. Even on Intrepid (at least on my slicehost slice), I needed to install these and their dependencies. You’ll need to make sure to get libqt4-core as well. The libqt4-webkit didn’t seem to require it according to aptitude.

I had a small problem with the script as far as passing arguments into the Xvfb server. For that reason, I’m sharing my version of the code. However, all kudos go to Roland. His blog post on this subject provides a nice overview of the thought process and rationale behind this code. If you’d like a little more on the how and why, check it out. He also provides a simpler version if you’d just like to test out the Qt WebKit stuff and see how it works.

In summary, to use this code:

  1. Read Roland’s blog post.
  2. Install the necessary libraries: libqt4-core, libqt4-webkit, python-qt4, xvfb.
  3. Grab my code here. Change the extension to py.
  4. To run, use a command like this: python webkit2png.py -x -o cnn.png –debug http://www.cnn.com

If you have questions, comments, or code to add, please post in the comments or pingback this post. I will try to update once I have a good threading model built out around this functionality.

Advertisements

A long time ago, longer than I’d like to admit, I started on a freelance project. I can’t say too much about what it does, but I would like share a bit about the development process I went through.

When I started the project, I was beginning to get back into web development after a couple of years away from the work during which I was teaching. So, I had some catching up to do and rusty skills needed to be dusted off and honed.

The project is somewhat ambitious with lots of GIS work and a little bit of social networking thrown in. It is a big project and means a lot to me, both financially and as a way to learn and really sharpen up my skills. It’s been a pleasure to work on as it is finally coming around to fruition.

All that said, I jumped in with both feet and starting coding in PHP, with some bad habits such as HTML all over my PHP. I know enough CSS to be trouble, so there was lots of that, too. I was building with some eye toward a previous incarnation of the app, so there were some design decisions made then that we have long since abandoned.

Many months late, I produced a passable version. It didn’t look like much, but it worked. When the fellow I was doing the work for asked me how scalable and maintainable it was, I had to do a lot of soul searching. I had become friends with this client and I knew deep inside that what I had built would work up to a point, but that it wasn’t that extensible and I was unsure how some of the GIS bits would scale.

I owned up to some of what I saw as problems with what I had built. I was building something to be a an alpha/proof-of-concept when what the client wanted was a solid beta product. I had misunderstood and probably gotten in over my head.

It was then that my client and I had a heart-to-heart and decided we would start from scratch with a platform that was extensible, maintainable, and offered some niceties that no homegrown PHP app could easily do. That platform was Python/Django/PostgreSQL.

With PostgreSQL, we got the PostGIS libraries which overjoy me day in and day out. With Python, we got clean, maintainable code. With Django, we got a robust framework that makes it easy to add new features without massive rewrites. Django also gave us fantastic code/design separation and it was just plain fun to learn.

These great visions in mind, I set about to not only learn the Django framework, but to also learn Python. Luckily, my new full-time job was giving me plenty of PostgreSQL experience, though no PostGIS (yet!), as well as, giving me the ability to work with Django and Python on the new features we are building at work. So, I wasn’t totally in over my head.

Every piece of the app that I write now is a pleasure to code. When I’m “done,” I can hand over these neat, clean HTML templates to a designer to spruce up with little fear of confusion. The database access stuff, especially with GeoDjango in tow, has become a breeze. Everything I used to hate about web development has been made a tiny little bit of the work. The vast majority of my time is spent thinking about scalability and how the feature actually works rather than building another script to handle form post input. It has simply been a joy all the way around.

We are nearing the end of the application work on the new Django version of the app. There’s lots more to do and lots of ideas for the next version. The difference between this Django version of the app and the PHP version is night and day. It might look similar from a UI perspective, but it is vastly fewer lines of code (not including Django itself), another developer could easily pick it up and improve on it or add to it, bugs are much, much easier to track down, and it was actually fun.

When all is said and done, I will, of course, point you to the app and will maybe do some more detailed comparisons between the two with some code. One thing to keep in mind is that the difference is not just between a PHP and Django app, but between the developer I was and the developer I am becoming. Sure, I’m no rock star, but I’m getting better at this every day.

I’d like to give a big thanks to my client for his faith and reassurance as I tackled this daunting task. A great client and an open, honest relationship go a long way. I’ve so far past any deadline we ever set on this project, but my client sees the vision and understands why we’re doing what we’re doing.