Blowing up markets

Banning sublets

Last week, the State of New York passed a bill that bans short-term rentals: specifically, no homeowner or renter may sublet their home for less than a month. The target is sites like AirBNB, an up and coming website that allows travelers to eschew pricey hotels – and their accompanying hotel room occupancy tax – in favor of private homes.

If the governor chooses to pass the legislation (as opposed to veto it), AirBNB will effectively be outlawed, and with it, a grassroots marketplace economy for short-term accommodation. New York State will have cemented hotels and bed & breakfasts as gatekeepers to the city for travelers who can’t stay with friends or relatives.

To me, this is an interesting reaction: it shows, once again, that established gatekeepers are terrified of the Internet. We’re used to that by now in the context of media content – we already know that newspapers, publishers, record companies and movie distributors aren’t as important as they were – but this is a scarcity-driven marketplace. It used to be that finding a safe, clean room in a strange city was a hard problem, so we turned to hotels as a trusted source. Running a hotel is in itself an expensive, tough business, and as a result there were a limited number in any given city, and the price went up according to demand. Although the hotel business is a ruthless game, it’s always been hotels competing with other hotels.

Now, though, we can visit websites like AirBNB and Couchsurfing, where private citizens can offer their homes to travelers, and the site will let us know who we can trust based on other peoples’ experiences. The marketplace has been blown wide open, and it turns out that a lot of us would rather go for a cheaper, friendlier option. I wouldn’t put money on New York blotting out short sublets for long.

Power to the people

We’re going to be seeing a lot more of this, in all kinds of market sectors. We’re already seeing ridesharing sites become popular, for example, blowing up the market previously owned by taxicabs and making it available to anyone who happens to be driving somewhere. Effectively this formalizes hitchhiking, making it both safer and more efficient.

It all comes down to one simple rule: People want to be free.

The Internet is opinionated: as a medium, it inherently works to empower people and eliminate hierarchies in society. It shouldn’t be a surprise that the most popular Internet companies hail from California; their philosophies are direct descendents of the civil rights activism that took place there in the sixties and seventies. In many cases, it’s even the same people. (Or – and here I put up my hand as the son of Berkeley “radicals” – their children.)

Gatekeepers – companies, structures or processes that act as exclusive barriers or filters – are not long for this world. Where gatekeepers exist, they do so because the alternative was inconvenient at the time when the gatekeeper became established – not because they’re inherently better than an empowered population. Those organizations, companies, and even governments, need to look at themselves very carefully and figure out what needs to be changed, before those things are changed for them.

Most Commented Posts

• Twitter DoS and single points of failure (31)
• Gender differences on the new frontier (16)
• Writing in Oxford? (15)

The product management cycle

Monday: “The plan is A! We’ll market it at A!”

Tuesday: “Actually, I was thinking B. A is stupid. Who would want to do that?”

Wednesday: “Goddamnit, we need to be working towards C. Why does no-one see that?”

Thursday: “Maybe A was right …”

Friday: “This team sucks.”

Hint: pick a direction and run. And make sure – just as your tech team does – that your management team has measurable metrics for success.

Most Commented Posts

• Twitter DoS and single points of failure (31)
• Gender differences on the new frontier (16)
• Writing in Oxford? (15)

Barcamp Oxford 2010

This weekend was Barcamp Oxford, and suffice it to say it was utterly fantastic. While a delicious ham cooks, I just thought I’d jot down some notes.

The day for me started early on Saturday with a walk into town (Oxford’s buses being resoundingly useless), still it was a lovely sunny day and I had some good tunes on my mp3 player so I didn’t mind working on my tan for a while.

Anywho, got to the Oxford Club in time for registration and for the slots to be put up… a dizzying array of very interesting possible discussions were soon put up.

Obviously I couldn’t go to all the sessions, but those I did go to were fantastically useful.

Highlights were a thought provoking, passionate and mature discussion of women in technology, and education in general. Additionally, met a bunch of very cool people who I hope to keep in touch with!

The beeper has just gone on my oven so I must tend to my food. Suffice it to say, I found the event both highly enjoyable, stimulating and very very useful.

Props to all the people who worked so hard to make this happen, you are all awesome!

Same time next year?

Image by Sylwia Presley

Write real-time web applications with XMPP, PHP, and JavaScript

I’ve written a tutorial for writing XMPP-based web applications over at IBM DeveloperWorks:

Real-time web applications are networked applications, with web-based user interfaces, that display Internet information as soon as it’s published. Examples include social news aggregators and monitoring tools that continually update themselves with data from an external source. In this tutorial, you will create Pingstream, a small notification tool that uses PHP and JavaScript to communicate over the Extensible Messaging and Presence Protocol (XMPP), a set of XML technologies designed to support presence and real-time-communications functionality.

You can read the whole tutorial here. IBM have made it a featured article, commenting, “bet you have it up and running before lunch.” I hope you find it useful. (And don’t forget to check out my introduction to Activity Streams, also written for IBM.)

Photo: IBM by antonfortunato, released under a Creative Commons license.

Related entries

• An introduction to Activity Streams (0)
• XMPP: powering the real-time, really live web (7)

An introduction to Activity Streams

I’ve written an introduction to the Activity Streams standard for IBM DeveloperWorks:

Enter Activity Streams, an evolving standard that extends Atom for expressing social objects. Although it is a young standard, Activity Streams is fast becoming the de facto method for syndicating activity between web applications. For example, MySpace, Facebook, and TypePad all now produce Activity Streams XML feeds. But this technology isn’t just for the consumer web environment. As corporate intranets and internal software become more social, solid business reasons support implementing Activity Streams as a feature. This article describes Activity Streams in detail, considers its potential uses in enterprise environments, and provides some examples for interpreting Activity Streams feeds using PHP.

The full article is over here.

Related entries

• Write real-time web applications with XMPP, PHP, and JavaScript (0)
• Direct messaging in a social web architecture (3)
• Activity Streams and OAuth: a social web architecture (3)

How to set up ProFTP, MySQL and Virtual Users

ProFTP is a configurable FTP server available on most *nix platforms.

I recently had the need to get this working and authenticating off a PHP maintained MySQL backend, and this post is primarily to aid my own memory should I ever have to do it again.

Installing ProFTP

In order to use MySQL as a back end you need to install some packages. If you’re using a Debian based distro like Ubuntu, this is easy:

apt-get install mysql-server proftpd proftpd-mod-mysql

The database schema

Next, you need to install the database schema to store your users and passwords.

CREATE TABLE IF NOT EXISTS users (
userid varchar(30) NOT NULL default '',
passwd varchar(128) NOT NULL default '',
uid int(11) default NULL,
gid int(11) default NULL,
homedir varchar(255) default NULL,
shell varchar(255) default NULL,
UNIQUE KEY uid (uid),
UNIQUE KEY userid (userid)
) TYPE=MyISAM;

CREATE TABLE IF NOT EXISTS groups (
groupname varchar(30) NOT NULL default '',
gid int(11) NOT NULL default '0',
members varchar(255) default NULL
) TYPE=MyISAM;

One important thing to note here – that caused me a fair amount of hair pulling when I tried to use encrypted passwords – is that the password field shown in many howtos on the internet is much too short. This causes the hashed password to be quietly truncated by MySQL when saved.

This results in a somewhat misleading “No such user found” error to appear in the logs when using encrypted passwords.

To end all argument I’ve allowed passwords up to 128 chars, but this field could probably be a good deal shorter.

The user table looks much like /etc/passwd and is largely self explanatory. The uid & gid fields correspond to a system user in most cases, but since we’re using virtual users they can largely be ignored. Homedir points to a location which will serve as the user’s default directory. Shell is largely unused and can be set to /bin/false or similar.

Configuring ProFTP

Next, you need to make some changes to the ProFTP configuration files stored in /etc/proftpd. While doing this it is handy to run proftp in debug mode from the console:

proftpd -nd6

proftpd.conf

1. Make sure the AuthOrder line looks like:
   

   AuthOrder mod_sql.c

2. Ensure that the following line is uncommented:
   

   Include /etc/proftpd/sql.conf

3. For belts and braces I’ve included the following at the end, although I’m not entirely sure it’s strictly required:
   

   <IfModule mod_auth_pam.c>
AuthPAM off
</IfModule>


4. Our users don’t need a valid shell, so:
   

   RequireValidShell off

modules.conf

1. Make sure the following lines are uncommented:
   

   LoadModule mod_sql.c
LoadModule mod_sql_mysql.c

sql.conf

1. Set your SQL backend and ensure that authentication is turned on:
   

   SQLBackend mysql
SQLEngine on
SQLAuthenticate on

2. Tell proftp how passwords are stored. You have a number of options here, but since I was using mysql’s PASSWORD function, I’ll defer to the backend.
   

   SQLAuthTypes backend

3. Tell proftp how to connect to your database by providing the required connection details, ensure that the user has full access to these tables.
   

   SQLConnectInfo database@host user password

4. Define your table structure in the format tablename fields….
   

   SQLUserInfo users userid passwd uid gid homedir shell
SQLGroupInfo groups groupname gid members

Adding users

I manage users from within a PHP web application that I’m developing, but in a nutshell adding FTP users from this point is a simple insert statement looking something like:

mysql_query("REPLACE INTO users
(userid, passwd, uid, gid, homedir, shell)
VALUES
('$userid', PASSWORD('$password'), $uid, $gid, '$homedir', '$shell')");

Have fun!

Feeds objects within feeds objects

We’ve been doing a lot of work with the Drupal Feeds module recently. The frontend is nice enough, although the sub-navigation was rendered almost illegible by our theme’s CSS. The online tutorials need work, and the admin navigation needs to be made a bit more robust to layout changes; but then it will be the de facto way for people to consume feeds on their Drupal sites.

The most recent work we’ve been doing involved custom integration with RSS feeds arriving effectively as PHP string variables containing all the XML. This is different from either a file on disk or a remote URL: in fact, we had a Python program creating the RSS file from us via a shell (which in turn, horribly, was hitting a remote Oracle database using cx_Oracle). Feeds was definitely up to the job in terms of power. In fact, it was quite a toolkit of useful functionality, which is Drupal code for “incredibly powerful but almost incomprehensible.

It’s not that the developer documentation for Feeds isn’t decent: it’s pretty good. But it’s limited in scope: it tells you roughly how to expose your own Feeds-like objects to the admin interface, but not really how all those objects interact. Most importantly, we wanted to know what happened on a cron run: this is the bedrock of how Feeds works on your site, after all.

I poked around a bit and this is what I discovered:

Here’s a summary of the above diagram to give you some idea of what’s going on.

1. Drupal’s cron creates a FeedsScheduler object and passes it a “job”, which is all the configuration for a feed call, including any configuration that was attached originally to the particular node which defines the Feed. The scheduler creates a FeedsImporter and passes it the job; the importer then creates a FeedsSource and embeds itself in it as a parent. In each case, the method ::work() is called to create the child/helper object.
2. The Source object is what now runs the three phases of feed consumption, via its parent Importer. The Source asks the Importer for the relevant Fetcher, Parser and Processor objects: for example, the HTTP Fetcher, the RSS Parser and the Node Processor objects are strung together to turn an RSS feed at a HTTP URL into a set of nodes, one per entry. Each of these have a relevant, verb-like named method: so ::fetch() for the Fetcher etc. The common currency is a FeedsBatch object, which gets passed around and needs to have methods that make it feel like a batch of feed objects.
3. After the three phases have run, the Source calls hook_feeds_after_import() to do any tidying, then quits to the Importer, which quits to the Scheduler, which then runs its ::finished() method on the job, and the cron run for this particular feed is done.

When you build a new plugin, you need to implement hook_feeds_plugins() in a module and reference a class file: this class will be selectable in the admin interface for one of the three consumption phases, depending on what class it’s ultimately based on. You should therefore extend existing classes rather than start from scratch: there are abstract PHP classes in the feeds module directories, which give you skeleton “interfaces” which you can then flesh out with relevant functions. But what’s better is to extend e.g. the HTTP fetcher to fetch from a command on disk (which is what we did) or, say, extend the CSV parser to interrogate JSON.

Class hierarchies mean you don’t have to spend a lot of time reinventing the wheel or hacking existing modules until they become unupgradeable; instead you can take existing classes and tweak them through inheritance, experimenting as you develop.