Webbots, Spiders, and Screen Scrapers A Guide to Developing Internet Agents with PHP/CURL Michael Schrenk

Amazon.com: $22.19

US Kindle Edition: $26.19

Amazon.co.uk: £24.56

UK Kindle Edition: £16.30

I was initially surprised to discover this title. “Webbots, Spiders, and Screen Scrapers” is a niche topic which the majority of web developers never consider. Michael Schrenk apparently does little else and his enthusiasm is evident throughout the book.

While browsers are great, they are general purpose tools which display web pages and have no concept of the underlying information. Schrenk will inspire you to use your web skills in a different context and achieve a lot given a small investment. It certainly inspired me to write a small PHP program to check several thousand pages for holiday availability — it saved hours of hunting.

The 362 pages are split into four main sections. The first introduces fundamental concepts such as PHP’s cURL functions, fopen and fget. The author provides a downloadable library which is used to simplify most of the example code.

Part two provides a number of sample projects such as link verifiers, image capture bots, FTP downloaders and email analysers. The chapters are concise but provide enough information to get you started.

Part three covers more advanced topics such as spiders, procurement, SSL, authentication, cookies, scheduling (in Windows) and browser macros. Part four describes larger considerations such as proxy servers, fault tolerance, redirection handling and legal implications. The author gives good suggestions and useful pointers. There’s little practical code, but that’s not unexpected given the wide scope of the topics.

Finally, there are a number of useful appendices including a good cURL reference, email to SMS gateways, HTTP and NNTP status codes. The 18-page index illustrates just how many topics Schrenk had to cover.

There are a number of issues which concerned me…

The author dislikes regular expressions. He admits it’s controversial but states they are hard to use and don’t show information context. My opinion: if regular expressions are useful for anything, it’s web page parsing. I accept they can be difficult to comprehend but it’s possible to create a series of simpler expressions if that becomes an issue. I’m not convinced by the ‘lack of context’ argument either; a single regular expression can extract titles and associated data.

There is also a chapter dedicated to reverse-engineering HTML forms. It recommends using an online form parser provided by the developer but makes no mention of HTTP analysis tools such as Fiddler, Firebug, HttpFox or Live HTTP Headers which make the process immeasurably easier.

I was also surprised that hashes or checksums weren’t described in the chapters about data storage. File compression is covered, but there may be no need to store or compress a file if a previously-generated checksum indicates it hasn’t changed.

Finally, my biggest concern is the use of the author’s own libraries — which he states are not particularly elegant. The samples describe how to use his code rather than the lower-level PHP and cURL instruction sets. That’s a shame; the book covers a heavy-duty topic and is aimed at an advanced audience but Schrenk abstracts the technicalities away from curious developers.

“Webbots, Spiders, and Screen Scrapers” is well-written and easy to read. Schrenk will encourage you to look at the web as a data resource and inspire you to write useful code which saves time and money. It’s ideal for those new to the subject, but is a little too lightweight for experienced developers.

Note for Amazon: you really should look at your prices for paper and Kindle versions in the US!

The Book of Ruby A Hands-On Guide for the Adventurous Huw Collingbourne

Amazon.com: $26.50

US Kindle Edition: $29.02

Amazon.co.uk: £19.60

UK Kindle Edition: £17.64

I’m a total Ruby newbie. Having developed in PHP, .NET and JavaScript for more than a decade, it’ll take a lot to persuade me to drop my favoured languages. But Ruby interests me — perhaps because its developers are earning astronomical salaries?

“The Book of Ruby” is aimed at novices like me. It’s 373 pages and published by No Starch Press so it has a lot to live up to following their excellent Book of CSS3.

I should point out that The Book of Ruby is not an introduction to programming. The reader will benefit from some development knowledge, but there’s no need to be familiar with Ruby syntax or concepts.

The chapters are logically ordered and start easily with variable types, classes, loops, conditional statements, and exception handling before moving on to more advanced topics such as modules, mixins, threads and dynamic programming. The writing style is friendly, chatty and easy to read. Most chapters end with a “Digging Deeper” section which provides useful advanced information.

It’s difficult to fault the first half of the book, but the final chapters are more challenging. This was possibly my limited experience with Ruby, but I found several sections difficult to follow. I suspect most developers would benefit by reading it from start to finish: it’s not necessarily a book you can dip into.

A few aspects struck me as odd. For example, most books of this type dedicate a chapter to installation of the language on various OS or server combinations. Collingbourne give us a single paragraph and a couple of links. Perhaps Ruby is easy to install so that’s all we need? So is MySQL, yet he provides a four-page appendix about installing the database.

Despite the niggles, “The Book of Ruby” is an excellent introduction to the language. I’ve seen a few negative comments from Ruby experts but the majority of points seem minor and, besides, the book isn’t aimed at them.

Have I been persuaded to drop PHP? Not quite, but I’m no longer afraid of the Ruby or daunted by its syntax. If you’re looking for a good introduction to the language “The Book of Ruby” should be on your buying list.

The Book of CSS3 A Developer’s Guide to the Future of Web Design Peter Gasston

Amazon.com: $20.87

Amazon.co.uk: £14.06

If you’ve been avoiding CSS3, perhaps it’s time to re-evaluate that policy. If Peter Gasston’s book doesn’t tempt you to delve into the murky waters, nothing will.

“The Book of CSS3″ is the culmination of 5 years effort writing about the technology. Unlike other titles, this isn’t a book about Cascading Style Sheets with a little CSS3 tagged on — it’s a complete overview of the latest techniques. It won’t teach you CSS, but it gives existing web developers an insight into new possibilities.

The CSS3 specifications are in flux but the majority of browsers support some of the properties — abet with vendor prefixes. The book was completed shortly before the release of IE9 and Firefox 4, but those browsers are included in the support summary provided for every module.

Peter eases the reader in gently with a little history (were you aware that work on CSS3
started in 1998?) and a light-hearted rant about HTML5 confusion caused by the media. The remainder of the 278 pages covers specific modules; those with the most reliable and consistent support come first.

Chapter 2 starts with media queries: the basis of responsive design for mobile and desktop devices. Peter recommends creating your basic mobile site first then applying larger layouts and assets for desktop browsers. It’s a sensible approach which is gaining more traction in the industry.

Chapters 3 and 4 describe CSS3 selectors, pseudo-classes and pseudo-elements. These can cause confusion, but Peter adequately covers the basics and more advanced techniques. It’s a section you’ll return to again and again.

Chapter 5 is dedicated to web fonts. If you rarely consider the implications of fonts on the web, Peter delves into the issues, problems and workarounds in fine detail. A bulletproof CSS3 syntax is provided for the widest possible browser support.

Text effects and typographic styles are described in chapter 6. If you were expecting an easy-going section about text shadows, you’ll be amazed to find great information about resizing, wrapping, kerning, rendering, and smoothing.

Chapter 7 is dedicated to multiple columns: a concept CSS developers have struggled with for many years. Fortunately, CSS3 will make our lives easier; Peter describes how the Mozilla and webkit browsers support the standard and the differences you’ll encounter.

Midway through the book, chapter 8 explains background images and other decorative properties. Multiple backgrounds, repeating, resizing, clipping, masks and related topics are succinctly explained.

Chapter 9 reveals border and box effects such as rounded corners, border images, and drop shadows. Chapters 10 and 11 follow this with the new colour definition methods, gradients and opacity.

If you’ve forgotten everything you learnt about trigonometry, chapter 12 will bring it flooding back as Peter covers 2D transformations. Rotating, scaling and skewing soon give way to scary concepts such as matrix transformations.

Chapter 13 introduces transitions and animations. Peter shares my view that they’ve blurred the distinction between presentation (defined in CSS) and behaviour (JavaScript). But let’s not worry too much — they’re fun!

Chapter 14 will strain your brain as Peter describes 3D transformations and geometric manipulation concepts. It’s early days for the techniques and the properties are only supported by the latest webkit browsers. I doubt anyone will mind if you skip a few pages!

Flexible box layouts are described in chapter 15. Peter admits that implementation remains patchy, but it provides a solution to several common website layout problems.

Chapter 16 describes the CSS3 template layout: a module which is so new, no browser offers native support! In essence, it allows developers to resurrect some useful table layout concepts without resorting to HTML tables. Peter provides information about a JavaScript shim which allows you to use the techniques today.

Finally, chapter 17 discusses the future of CSS with some great ideas which could become commonplace within a few years. This is followed by appendices containing browser support information, online resources and the all-important index.

The book benefits from Peter Gasston’s natural writing style. It’s informal, relaxed, well written and attractively laid out. Complex technical details are explained concisely without relying on jargon or glossing over details.

“The Book of CSS3″ is one of the best technology books I’ve read. I’d recommend it to any web developer who’s itching to experiment with the new techniques. At this price, it’s an absolute bargain.

Eloquent JavaScript A Modern Introduction to Programming Marijn Haverbeke

Amazon.com: $17.97

Amazon.co.uk: £13.92

On first glance, you might expect “Eloquent JavaScript” to be just another code book. There are few diagrams; just 196 pages of text and code samples. Don’t let that put you off.

Despite being the most-used programming language on the planet, JavaScript is the most misunderstood. Desktop and server-side developers used to sneer at JavaScript but it’s finally being recognised as a flexible language which can be used to build full-scale applications.

The author, Marijn Haverbeke, admits to despising JavaScript but came to love its liberal style and flexibility. Eloquent JavaScript has been written for those who are new to programming or perhaps taking their first steps in the language. There are 12 chapters:

1. A good introduction to variable value types and control flow statements. Nothing too scary for programming novices.
2. Functions. While many development books might keep this simple, Haverbeke delves into more advanced topics such as closures and recursion. I suspect it might be a little confusing for some, but they’re succinctly explained.
3. Data structures such as arrays and object literals. An amusing example which extracts cat information from emails is provided.
4. Error handling. A useful chapter since many experienced developers never consider JavaScript errors. Exceptions and automated tests are explained.
5. Functional programming. Stories such as the split-pea soup recipe and the semi-autobiographical “recluse writing a technical book” are used to explain concepts such as abstraction.
6. Object Orientated Programming. This is a long chapter which delves into the intricacies of JavaScript objects and prototypal inheritance. Good example code is provided which creates a terrarium simulation with animated insects.
7. Modularity. The author explains how modules can be used to separate code.
8. Regular expressions. A solid and concise explanation is provided.
9. Web programming crash course. The chapter provides a brief overview of browsers, forms and associated technologies. Leaving this toward the end of the book may seem odd, but it’s logical.
10. The Document Object Model and
11. Events are two areas which confuse the most skilled browser developers. The concepts are well covered and the author uses the new techniques to convert the terrarium code into a working version of Sokoban!
12. HTTP requests are explained with reference to Ajax using both XML and JSON message formats.

Eloquent JavaScript is well written, and uses a chatty, informal style which is easy to read. The stories and code samples are genuinely amusing without becoming silly.

My only reservation concerns the audience. Is JavaScript a good first language to learn? Most new developers will be writing it in their browser — a fairly hostile and unforgiving programming environment. I also suspect the OOP chapter, although well explained, could terrify new developers.

Overall, Eloquent JavaScript is a great book if you’re learning JavaScript. More experienced developers are likely to skip a few chapters, but even they’ll still find something of interest. Recommended.

Google Analytics Understanding Visitor Behaviour By Justin Cutroni

Amazon.com: $19.79

Amazon.co.uk: £11.70

Google Analytics revolutionised the web tracking industry. Before it’s launch in 2005, organisations could spend thousands on log analysis software. Few systems offered (near) real-time reporting or in-depth visitor information. Amazingly, Analytics is free and almost everyone uses it. It doesn’t matter whether you have a single-page hobbyist site or a large-scale corporate operation: Google Analytics is the tracking tool of choice.

Few programmers have an interest in conversion analysis or marketing reports. Many add the basic tracking code and think no further about it. Similarly, few marketing experts have the technical knowledge to exploit behavioural analysis. “Google Analytics” from O’Reilly is a 190-page book which could help those from either discipline.

The first couple of chapters introduce the process of measurement, analysis, change and creating an implementation plan for your company. The reader is forced to jump in at the deep end; there is too much jargon, too many assumptions, and several off-hand references without further explanation. It’s tough going, but persevere — the content improves and becomes easier to understand.

Chapters 3 and 4 describe how Analytics works with the standard JavaScript and mobile tracking code. I’m not convinced many people will need or care about those details, but I found it interesting background material.

For many, chapter 5 offers the first practical hand-ons advice and it describes how to configure your Analytics account, profiles and users. The author recommends creating multiple profiles for the same website. It’s an interesting thought, although potential benefits are not discussed in detail. Integration with ecommerce and search were facilities I’d not appreciated and the book encouraged me to adopt them immediately.

Chapter 6 provides details about a much misunderstood topic: filters. Filters allow you to alter incoming data, e.g. removing traffic generated by internal employees. Cutroni describes the options with an in-depth explanation of strings and regular expressions. I suspect many filters will be defined by programmers who already understand the concepts, but some will appreciate the detail.

Chapters 7, 8 and 9 explain campaigns, funnels and goals. For me, this was the core content and the author explains the concepts and usage clearly. However, further real-world examples would have been appreciated.

One quarter of the book is devoted to chapter 10 — advanced tracking techniques. It’s essential reading for web developers who need to monitor users across domains, sub-domains, frames, or when client-side events occur.

The final three chapters finish with enterprise considerations, CRM integration and tools. Few developers will be affected by these issues, but they are explained quickly and concisely. Two appendices follow which list Web Analytics Association (WAA) metric definitions and regular expression syntax.

I have a few complaints about “Google Analytics”. It can be incomprehensible to new readers, AdSense integration is glossed over, there’s little about comprehending statistics, and the chapters would benefit from real-world implementation examples. However, Cutroni is obviously passionate about the topic, the book is well-written, and it reveals facilities and best-practise techniques which few organisations have successfully adopted.

If you’ve only scratched the surface of Google Analytics, this book will change your understanding of visitor behaviour. Developers can master implementation and marketeers can leverage the data. Ultimately, it’s a low price to pay for the potential improvement in your conversion rates. Recommended.

High Performance JavaScript Build Faster Web Application Interfaces By Nicholas C Zakas

Amazon.com: $23.09

Amazon.co.uk: £14.58

“High Performance JavaScript” is a 202-page book aimed at intermediate to advanced JavaScript developers. However, everyone will learn something no matter what their programming background.

JavaScript has evolved. Ten years ago it was ridiculed as a kiddies scripting language (it’s name did much damage) but primarily because browsers were a hostile development environment. The rise of Ajax and Web2.0 applications led to a rediscovery of the language. Many developers, including myself, love JavaScript but it’s still widely misunderstood and misused.

Large web applications now contain many megabytes of JavaScript code. Engines are improving, but performance and optimisation remain a concern for every front-end developer. Could Nicholas C Zakas’s book be the answer?

Chapter 1 provides an important overview of how JavaScript is loaded and when scripts are run. Standard end-of-page techniques are discussed and Zakas recommends several non-blocking patterns to ensure the browser remains responsive while code loads in the background.

The second chapter throws the reader into complex areas such as scope, identifier resolution, closures, prototype chains and memory management. I suspect many will skip the section, but it provides useful background information.

Chapter 3 discusses DOM methods, innerHTML, node access, modification, HTML collections, events and animation. For me, this was the most interesting and useful section of the book. For example, did you realise that standard DOM methods are catching innerHTML and can be faster in webkit browsers?

The fourth chapter covers standard programming methodology including algorithms, loops, conditions, iteration and recursion. Efficient, good-practise techniques are discussed and many will apply to any programming language.

Chapter 5 delves the reader into string manipulation and includes an 18-page in-depth analysis of regular expressions. Like many, I often use the force when creating complex regexs and rarely consider operational efficiency. Zakas describes how they work and what you can do to improve their speed — again, essential reading for any web developer.

Chapter 6 is titled “Responsive Interfaces” and discusses browser UI threads, timers, split processing and the new HTML5 web workers API. Application developers will find the information increasingly useful as the processing burden moves from the server to the client.

The following chapter is dedicated to Ajax and compares the various techniques and data transport options such as HTML, XML, JSON, and custom formats. Although I found little new, there were still nuggets of information which will change the way I approach Ajax-powered applications.

The book finishes with Programming Practices — another informative section which could apply to any discipline. The final chapters (written by other contributers) cover building, deployment, pre-processing, minification, caching, CDNs, and profiling tools.

High Performance JavaScript is an excellent book. It’s concise, well-written, and matched — if not exceeded — my expectations. JavaScript optimisation is a hot-topic and I’d recommend it to all front-end developers.

TACS allows PHP code to be run either:

1. At render time when the page is cached. Typically, a menu could be generated once so all subsequent visits return static HTML.
2. At runtime when the user accesses the page, e.g. for a form submission.

The cached page therefore contains the minimum of PHP code. It allows you to use PHP as-is rather than forcing you to learn another API or syntax.

What’s more, TACS is free. Use it, abuse it, copy it, or do whatever you like with the code. We don’t mind — but a link to this website or PayPal donations are gratefully received!

TACS v1.3.1 fixes a path resolution bug and allows you to use permalink-like URLs.

Download TACS (12 Kb)

PHP The Good Parts Delivering the Best of PHP By Peter B MacIntyre

Amazon.com: $19.79

Amazon.co.uk: £14.94

O’Reilly set a high standard for their “Good Parts” series. Douglas Crockford’s outstanding JavaScript: The Good Parts (Amazon.com: $19.79, Amazon.co.uk: £11.82) changed the way developers thought about JavaScript. The language was misunderstood and misused, but Crockford’s book helped shift its perception as toy-town script to a professional programming discipline.

Peter B MacIntyre’s PHP work had a lot to live up to. Few PHP developers adhere to solid development practices, and many will purchase this book on the strength of the title. However, the first shock is that it contains fewer than 150 pages — are there so few good parts to PHP?

The forward makes it clear that the book is aimed at developers who are new to PHP programming. The first 5 chapters cover installation, basic syntax, functions, strings and arrays. It’s a good introduction to the language, although those with some PHP experience will find little new.

I was more hopeful for chapter 6 — Objects. Unfortunately, just 12 pages are devoted to the topic, and 4 of those are dry code with little explanation or real-world application. The author recommends Peter Lavin’s Object-Oriented PHP (Amazon.com: $22.76, Amazon.co.uk: £23.99), but it’s disappointing not to have deeper analysis of the subject.

Chapter 7 covers database interaction using MySQLi and PDO. Bizarrely, 11 pages are devoted to SQLite and flat-file alternatives. Although these might be appreciated in a larger book, I wouldn’t consider them to be the ‘best parts’ of PHP, especially for large web applications.

Chapter 8 introduces us to PHP’s friends: email, PDF and graphic generation. The author devotes another 12 pages to dynamically-generated PDFs — useful, but not something PHP developers encounter on a daily basis.

The following chapters provide some reasonable security information and new PHP 5.3 features such as namespaces and closures. The goto statement also receives 2 pages, even though the author denounces it as a bad part of PHP?

The final chapter wraps up with brief details of regular expressions, string functions, SimpleXML, IDEs and websites. It’s useful, but there’s little you couldn’t find in greater depth within the PHP manual.

An appendix, named “The Bad Parts”, finishes the book. Another couple of pages are wasted on the goto statement and the register_globals directive — which has been disabled by default since v4.2.

On the plus side, “PHP The Good Parts” is well-written and easy to understand. It’s a concise introduction to the language which may be appreciated by developers with experience of other languages.

Unfortunately, PHP developers expect more and will feel misled by the title. O’Reilly has diluted “The Good Parts” brand.

Visit the BetterInnerHTML download page…

What is BetterInnerHTML?

BetterInnerHTML is a replacement for innerHTML. Like innerHTML, you can pass a string of HTML and have it rendered somewhere on the page. However, it uses standard DOM node insertion methods that prevents the problems.

Did we mention that it’s free? However, we do accept donations toward future updates and support.

For more information and technical details, please refer to innerHTML: the ultimate alternative solution.

We hope you find it useful and appreciate your feedback.

SitePoint is one of the biggest and best resources on the Internet for web developers, graphic designers, technology evangelists and just about anyone who has an interest in the web. I’ve been visiting SitePoint for over 10 years and I’d suggest you take a look too.

View my latest SitePoint blog posts…