Nambu Gets Better and Shortener User Tracking is Undermined

When I last wrote about the tribulations of tr.im and the business of bit.ly, our heroes had just stepped away from their nose-to-nose struggle, with Nambu founder Eric Woodward having announced the shut-down of tr.im, his URL shortener, only to vow its revival a few days later. Bit.ly, with a cozy relationship with Twitter, seemed to have taken a dominant position in the URL shortening business, whatever that turned out to be. I speculated that Bit.ly would use its position in the ocean of usage data to build psychographic profiles of users to help target advertising.

Since then, Woodward decided not to sell the tr.im business and has instead released the tr.im software as free open source, making it that much easier for websites to do their own URL shortening. He's also focused his company's attention on its Nambu Twitter clients (Mac and iPhone), the development of which suffered a major setback when his Chinese developers left for richer opportunities as soon as their contract was up. Nambu for Mac OS X has been my preferred Twitter client; it has a much more Mac-like user interface than others I've tried. When I updated my system to Snow Leopard last week, I was disappointed to find that Nambu had not survived the system update.

After unhappily revisiting Tweetdeck, I decided to try the beta version of Nambu, even though it's described as being not quite done. So far, it looks pretty solid. One change in particular pleased me, and that's the way the new Nambu works with URL shorteners. It seems that by surrendering URL shortening to bit.ly, Nambu is now freer to innovate in the user experience. Nambu now pre-expands all the shortened links so that the user can see the hostname that the links are pointing to. This has a number of consequences:

1. The user can tell where a link will go. This will help avoid wasted clicks, and will help the user avoid spam and malware sites.
2. Because all of the links are dereferenced before use, the URL shortening sites will no longer be able to track the user's reading preference. The business model I previously suggested for bit.ly will be defeated, and the user's reading privacy will be protected.
3. The URL shortener will have to deal with an increased load. Nambu's going to make bit.ly work harder for the privilege of domination the URL shortening space.

Now I understand why bit.ly has been registering a bunch of instantaneous hits whenever I tweeted a link- it was robot agents, not people, that were clicking the links.

I was curious to see if Nambu was querying the URL shorteners directly or whether Nambu was trying to aggregate and cache the expanded links. I installed a nifty program called "Little Snitch" to see the outbound connections being made by programs on my laptop. It turns out that Nambu is doing a direct check for redirection on ALL of the links that it shows me, not just the shortened ones. Although this could break links that are routed as part of a redirect chain, I imagine that sort of link occurrs rarely in a Twitter stream.

The new behavior of Nambu and its effects on usage tracking points up a general problem faced by any system designed to measure and track internet usage. In my post on "bowerbird privacy", I mentioned that I use StatCounter to measure usage on this blog. StatCounter works quite well for now, but I imagine that its methods (based on javascript) might well stop working so well as web client technology evolves. That's one reason I expect that efforts to standardize measurements of usage in the publishing community, such as Projects "COUNTER" and "USAGE FACTOR" are doomed to rapid obsolescence.

Will bit.ly ever get a business model? Will Nambu find peace with the chilly kitty? Find out in next months installment of... As th URL Trns

Tr.im's Brief Demise and the Privacy Implications of the Bit.ly Monopoly

In case you missed it, last Sunday the URL shortener Tr.im announced that it was going to close. Then, on Wednesday, they announced that they were going to keep the service open. I spent yesterday morning listening to the TechZing interview with Tr.im and Nambu founder Eric Woodward, which was interesting in a lot of ways. His perspective on having worked with a Chinese development team was quite interesting, and though not relevant for this post, it explains why development on Nambu (an OS X and iPhone twitter client) has been slow in coming- the Chinese developers left to get rich making iPhone apps. What I was more interested in was the discussion of business models for URL shorteners in specific and the Twitter ecosystem in general. It was the lack of any plausible business model for Tr.im that led to the decision to close the service. According to Woodward, there are only three plausible business models for a legitimate URL shortener:

1. you can charge users
2. you can sell advertising
3. you can sell data that you generate

and given that Bit.ly has an inside track with Twitter and offers everything to users for free, all of the three business models were just not going to work for Tr.im.

Part of the problem that Tr.im has experienced is that the cost of running a URL shortener scales with the amount of usage. (In a good web business, you have a high fixed cost and very sublinear scaling of cost with usage) Woodward says that he has to spend an hour a day just dealing with spam, and this problem is getting worse. Why is spam a problem for URL shortening? It's because spammers (and presumably phishers, too) will use URL shorteners to hide links to porn, scams, malicious content, etc. Afflicted users will report the links to the URL shortener's ISP host (Tr.im uses Rackspace) and the URL shortener will be shut down unless the spamming links are turned off. Another problem that afflicts popular URL shorteners is the problem of popular twitterers. When Ashton Kutcher or Shaquille O'Neill tweets a link to their millions of followers, the URL shortener can suddenly be hit with 10,000 hits per minute. A read-only website can handle this traffic easily by spreading service over multiple servers, but a URL shortener such as Tr.im, architected to generate dynamic usage data by writing to a MySQL database, needs beefy hardware to avoid getting overloaded.

In addition to the problems highlighted by Woodward, I see three core difficulties for URL shortener businesses:

1. It's really easy to build a small-scale URL shortener. A good web developer could probably build one in a day. Because of this low barrier to entry, casual users are forever going to be able to find free URL shorteners.
2. There are plenty of illegitimate (or at least annoying) business models for URL shorteners. These involve stealing traffic, stealing "Google juice", putting interstitial advertising in links, framing links, etc. This attracts entrants who make it harder for someone wanting to run a non-annoying business to attract paying users.
3. Links need to be reliable, because if your shortener fails, the user doesn't get the content they are trying to access In a lot of applications, links are meant to keep working forever. Reliability and persistence are expensive.

So basically, URL shorteners are a high cost, tiny revenue business.

Nonetheless, URL shorteners are very useful in today's 140 character world. Woodward had concluded, and I agree, that there is no room for more than one URL shortener business in the Twittersphere, and that Bit.ly has won. Bit.ly thus finds itself with an odd sort of natural monopoly. Of the three plausible business models for Bit.ly, it seems to me that generating and selling data is the only one that would maintain the monopoly, and thus the business. What kind of data might Bit.ly sell? I'll place my bet on "psychographic data". With its URL shortening monopoly, Bit.ly has access to a huge number of clicks. Bit.ly knows who I am, because I signed up for an account. Whenever I click a bit.ly link, my browser sends a cookie to Bit.ly which it could be using to track my interests, what I read. Aggregated over all the people who click Bit.ly links, the dataset of who clicked what could be very interesting to advertisers. Just as Google has the ability to tailor advertising to me based on my search history, Bit.ly could use a psychographic profile of me to help advertiser do targetting. Bit.ly has an interesting advantage over Google, however. Because so many of its properties rely on the user's perception of Google as a company that can be trusted with sensitive information, Google is quite limited in how far it can go in tracking users. In contrast, Bit.ly is not inhibited in this way. In fact, Bit.ly's ability to profile users could make it even more attractive to people putting links into tweets. Bit.ly could even provide this sort of profiling without violating its privacy policy which promises that it

... discloses potentially personally-identifying and personally-identifying information only to those of its employees, contractors and affiliated organizations that (i) need to know that information in order to process it on Bitly, Inc.‘ behalf or to provide services available at Bitly, Inc. websites, and (ii) that have agreed not to disclose it to others. Some of those employees, contractors and affiliated organizations may be located outside of your home country; by using Bitly, Inc. websites, you consent to the transfer of such information to them. Bitly, Inc. will not rent or sell potentially personally-identifying and personally-identifying information to anyone. Other than to its employees, contractors and affiliated organizations, as described above, Bitly, Inc. discloses potentially personally-identifying and personally-identifying information only when required to do so by law, or when Bitly, Inc. believes in good faith that disclosure is reasonably necessary to protect the property or rights of Bitly, Inc., third parties or the public at large.

Ironically, URL shorteners could also be used in ways that enhance user privacy from a different direction. As I discussed in my post on the semantics of redirectors, most URL shorteners use HTTP redirects. Although it's browser-dependent, either 301 or 302 redirects will result in the originating page being sent in the referrer header. "META refresh" redirects, on the other hand, can be used to wipe (or replace) the value of the referrer header. (Unfortunately, these can be also used annoyingly to cause "referrer spam".)

Redirectors deployed for purposes other than URL shortening also have market-share related privacy implications. For example, the dx.doi.org redirector which handles most DOI traffic could be a very useful vantage point for industrial or technology espionage. Because this redirector serves so many scientific article links, a spy agency might be able to monitor everyone in the world doing research on nuclear fission or anthrax weaponization, to give two examples.

In preparing my post on privacy mechanisms for Google Book Search, I was struck at the many directions that someone intent on privacy intrusion could take to collect potentially sensitive information. Part of the skepticism I expressed about being able to "sell" privacy comes from a feeling that privacy as traditionally thought of is pretty much a lost cause on the internet, no matter what Google, Bit.ly, or anyone does. Somehow, traditional concepts of privacy need to be recast into something that people still value as they use the internet.