« October 2008 | Main | December 2008 »
I was on Last.FM today and noticed this ad from Epson prompting me for my user name.
Entering your user name, generates this:
Clicking through takes you to this - a cool data visualization of your musical taste over time:
This ad is clearly built from the Last.FM API, and is the first example that I've seen of a targeted ad campaign for a social media community built off that community's API.
Pretty cool. Nice work by Epson.
Posted at 07:13 PM | Permalink
Here is a presentation that I gave about API strategy last week at Pubcon.
Posted at 10:25 AM | Permalink
Here is a presentation that I gave at Pubcon last week about the rise of the social media river / newsfeed format.
Posted at 10:21 AM | Permalink
I sat in on an excellent Pubcon session last week that was called “Getting Rid of Duplicate Content Once and for All.”
For those of you not familiar with the issue, it’s a big one in the SEO community (a site search for “duplicate content” on SEO forum WebmasterWorld yields 63K+ results). Duplicate Content has long been one of those hazy issues that nobody could really give you a clear answer on.
The most common questions go like this: Can syndicating out your site’s content adversely affect your site’s ability to attract organic search traffic? Can scrapers outrank you for your own content? Does the so called “duplicate content penalty” exist for sites that syndicated too much of its content from elsewhere? And can duplicate content on one page of a site hurt the organic search rankings of unique content on that same domain?
The panel was moderated by Rand Fishkin, one of the better known SEOs, and the panelists were Ben D’Angelo, engineer and Director of Duplicate Content at Google, Priyank Garg of Yahoo!, and Derrick Wheeler, an SEO from Microsoft.
Even the rumor of something called a Google applied “duplicate content penalty” should be a frightening thing for the social media crowd. Frankly, the social web is very much about duplicate content nowadays: from RSS feeds, to reblogging comments, to exposing and consuming APIs, to mashing up services. If Google were to penalize sites in the organic SERPS for either exposing or consuming content to / from other sites, it could be a real damper on the innovation of remixed services.
This summary post is based on what I’ve seen with my own sites, what I’ve read on the subject, and based on the official positions of Google and Yahoo! as presented in last week’s sessions.
The Mechanics of Duplicate Content
Duplicate content, similar or identical content that appears on more than one url, is problematic for search engines. Search engine users don’t like to see the same content under different pages in the organic search results, and crawling and indexing duplicate content is an unnecessary drain on search engine resources.
Both Google and Yahoo! attempt to deal with duplicate content the same way:
Step One is to cluster similar pages (from either the same domain or different domains)
Step Two is to decide which of those pages is the “best” representative of that cluster
Stuff can go wrong during both steps of this process. The wrong pages might be clustered together (meaning the content is not duplicate). And Google / Yahoo! might choose the wrong page as the best (i.e. “original”) source of the content.
The guiding principle for both Google and Yahoo! is to display one url in the index for one piece of content. So if something goes wrong with either Step One or Step Two, the original producer of the content will be left out of the organic search results.
Types of Duplicate Content
There are two types of duplicate content:
Duplicate Content on a single domain
Duplicate Content on different domains
Duplicate content within your domain is a hassle for two primary reasons; 1) it unnecessarily taxes your servers as search engine crawlers hit these pages; and 2) it can use up your site’s quote of indexed pages, pushing good, unique pages out.
Duplicate content on a single domain can result from any number of factors. Perhaps the site owner has chosen to point multiple URLs at a single piece of content. Maybe the site owner has forgotten to redirect his non-www domains to www, or vice versa (it’s important to not have both www.mysite.com and http://mysite.com available to search engine crawlers, it can dilute your ability to rank). Pages with pagination, sorting options, or other parameters in the URL are notorious sources of duplicate content. No matter how you sort the page (title, size, color); the content on the page is still the same. You only want to show one page to the search engines.
More examples include print friendly pages and urls that change depending which link you hit (www.rateitall.com vs. www.rateitall.com/default.aspx).
Duplicate content on your domain and someone else’s matters primarily to the original publisher of the content. If you are using somebody else’s content, you should not expect to outrank them in the organic search results. But if someone else is using your content, either with or without your consent, you probably expect your page to outrank the other domain’s. When this doesn’t happen, it’s a problem for you and an opportunity for those who are using your content.
Duplicate content on multiple domains can also come from a variety of sources, many of which are familiar to readers of this blog. APIs. Widgets. RSS Feeds. Mash-ups. Reblogging. And then there are the scrapers – those that crawl your site, oftentimes without your consent, and pull content back.
What Can be Done to Avoid Unnecessary Duplicate Content?
There are a number of tools available to site owners to deal with unnecessary duplicate content – and by unnecessary, I mean the accidental kind.
In terms of your own domain, Google offers Google Webmaster Tools – a dashboard that alerts you to pages that have duplicate title tags or meta descriptions. This can be helpful in identifying duplicates due to things like sorting, or url parameters, which you can then (theoretically) stamp out using no index tags.
However, once a deep page gets indexed, it is very hard to get it unindexed. My site had about 100K duplicate pages indexed, due to a sorting option on our top ten lists. We have had no index no follow tags on all the dupe pages for months now, and we still have thousands of duplicate pages in the index.
Yahoo! offers a much better solution for this type of issue – a tool that dynamically rewrites urls related to “content neutral” duplicates (like parameters in the URL) and concentrates all of the urls’ link juice into the primary URL. This is a big deal, especially as the Yahoo! engineer who I heard describe it spoke specifically of a Yahoo! index page quota – meaning duplicate pages caused by URL parameters might be keeping your good pages out of the index.
If you are syndicating out content via feeds or APIs, Google suggests that you require a link back to your site to show the search engines where the original source of the content is. However, I know from personal experience that this does not always work.
For folks taking your content without your consent, you can block them from crawling your sites and issue a DMCA takedown request.
Google vs. Yahoo: a Subtle Difference in Mindset
One of the things that struck me in listening to the Google and Yahoo! reps discuss this issue side by side was a difference in tone. The Google rep was much less ideological – he spoke of how Google treats duplicate content with filters and not penalties (generally). He did, however, seem to speak of duplicate content as a problem that needs to be addressed, as opposed to a core characteristic of the mashup generation.
The Yahoo! rep was much more confrontational. He used terms like “dodgy” and “abusive” to describe duplicate content conditions, and spoke out against “weaving” content together to win more organic search traffic (mashups??).
Conclusion
I walked out of the session thinking to myself that both Google and Yahoo! need to adjust their attitudes about duplicate content, and Yahoo! more than Google. Duplicate content is not a scourge. It is not an indication of dodgy or abusive behavior. And convincing webmasters that they need to avoid proliferating duplicate content seems to be both an impossible task AND bad for the web.
However, the search engines need to a much better job at figuring out the original source of the content. One of the suggestions thrown out during a discussion after the Pubcon session was that the search engines agree on a “not original source” tag that API and feed providers could require as part of their TOS.
I am pretty sure that Google does not often penalize sites for duplicate content. They might filter the duplicate content out of the index, but I don’t think they penalize the associated domains.
I’m not so sure about Yahoo!. Listening to the Y! rep speak, it’s very clear that he views duplicate content as an intentional drain of Y! resources. I would not be surprised if Y! makes frequent use of a duplicate content penalty.
As a lover of the distributed web, I’m looking forward to the search engines A) learning that dupe content is not the enemy; and B) figuring out how to better identify the source.
Posted at 08:25 PM | Permalink
During the recent Mashery Business of APIs conference, I heard from a lot of different companies on how, why, and to what end they were exposing APIs. I was particularly interested in the business model piece of it.
Here is a brief summary of some of the business models that can be associated with an API.
As a BizDev Lead Generation Tool / Funnel
The BizDev Funnel approach towards APIs was something that I heard over and over again. This model goes something like this – you use your API to give away a taste of your content / service, but not so much that somebody can actually build a business with it. Typically, API calls under this approach are capped per day. If the API partner exceeds that amount, they are invited to contact the company’s business development team to explore a partnership.
CNET is one company pursuing this model. The public CNET API is limited to 1500 queries per day, and also offers only limited content (for example, you can pull a review synopsis for a given product, but not the review itself).
Hoovers is another company that is taking this approach. Unlike the CNET model, third parties need to be approved to interact with the Hoovers API (HAPI!) at all. Each applicant is screened by the Hoovers bizdev team, and custom pricing is worked out for each one. Because Hoovers is a data subscription business, they are being very careful about who gets access to their data – if they give away too much, they risk destroying their business. From what I understand, Hoover pricing can range from per listing to, in some cases, revenue sharing with third party API partners.
User Acquisition / Affiliate
Another popular business model related to APIs is the user acquisition / affiliate model. Two great examples of this are Netflix (api review) and Amazon. Both these companies expose massive amounts of commercially valuable data with the hopes of extending their business model (ecommerce and subscription respectively) to the edge.
What’s notable about these two companies is that neither one of them is advertising supported. Each company monetizes its data in a way that presumably, virtually no other third party provider could do (though Netflix apparently through very hard about the implications of Blockbuster grabbing their data via the API). Because they are not advertising supported businesses, they don’t care about eyeballs going to their API partners’ websites – all they care about is that the transaction – for Amazon a sale, and for Netflix a subscription – is initiated on the partner site, and then fed back to Netflix / Amazon for execution.
It sounds like another non-advertising supported business, Best Buy, is pursuing a similar API strategy (called Remix) as well – exposing its product data with the belief that they shouldn’t care where the transaction is initiated as long as they get to execute it.
Pay Per Drink
The Pay per Drink model is the online version of the classic data subscription model. Companies that have traditionally sold big data dumps to subscribers – like Hoovers, InfoUSA, etc – might explore the API as another way to deliver this data, and charge for each call or each record.
A company that is pursuing this model is Alexa. They charge $0.30 per thousand requests.
Advertising
If you suspect that your API has the potential to achieve massive reach, you may want to consider an advertising driven model in which you reserve the right to feed ads out to your API footprint as a condition for allowing others to use the data.
The best example of this model is Google Maps, which recently began serving ads on its API partner sites.
Content Acquisition
The API as content acquisition tool is a variation of the Hub & Spoke model that I’ve covered so often on this blog. You expose your UGC functionality and content via an API with the hopes that third party developers will build additional tools to collect even more content for your database.
Presumably, this is the Twitter approach. By distributing their API for free, they have achieved huge reach, and this reach is generating massive amounts of additional content for Twitter’s database.
While it is certainly possible that Twitter might start charging for its API (pay per drink), or some subset of its API (the firehose), I would expect them to try and monetize primarily at Twitter.com. Twitter’s aspirations seem big enough that I doubt that they will sacrifice reach for revenue at the edge.
Branding / Non-Commercial
This is the approach used by companies like the New York Times, NPR, and many of the Yahoo! APIs, and can be seen as a subset of the BizDev funnel approach.
The idea is this: expose your content and functionality to hobbyist developers and see what kind of stuff they come up with, in the process extending your brand. The non-commercial clause is designed to protect you from other folks monetizing your content.
This is my least favorite kind of API. I feel that if you’re not able to tap into a developer / publisher’s selfish interest, your API is not likely to gain distribution. It seems to me that there is also more policing required to go this route.
Summary
So there you have it – six different strategies you can take for using an API to forward your business goals.
Did I miss any?
For a great summary of API trends, check out Dion Hinchcliff's post.
Posted at 03:44 PM | Permalink
My notes from Business of APIs conference: New York Times API
New York Times
Marc Frons CTO
19.5M uniques
Strategy: Times Open
Syndication - times widgets, pushing content out
Aggregation - be a focal point for discussion and quality content
Community - news sharing app released called times people
Identity - attempt to understand users better
Syndication
Data Universe - taking content and thinking it of data, getting it tagged, exposing it via apis, tools for devs and users (apis and widgets)
Times Open APIs -
Community APIs
NYTimes Developer Network
Aggregation
TimesExtra - took BlogRunner, displays related stories to NYT articles, then expose an api of that feed, combination of algorithmic and editorial judgment
Community - TimesPeople, IBM Many Eyes that takes data and users can make visualizations with that data, user blogs (proposed), q&a (proposed), socail mapping (proposed)
Why release open apis?
Syndication drives traffic back to NYT.com
Innovation - when others build, we learn
Journalism - fulfills mission by pushing info out to the public
Branding
New Business Models - we expect new business models will emerge around apis, probably around advertising and licensing
Developer network was launched in October.
First api was campaign finance database.
Five active apis
Times Tags - matches query to NYT meta data.
Movie Reviews - returns critics picks and links to reviews.
Community API - can retrieve user generated content. Ratings and reviews will be available in a later version.
TimesPeople - returns profile and activities of a particular user
API ROadmap:
Recipes
Movie Reviews - two new releases
Congressional Votes
Article Search
Q1 2009
Event Listings
Weddings
NewsFeed (river of content from a source, near realtime feed)
Bestsellers
and more
Theme: Making public record data more available and accessible
Two types of apis
internal and external
Open api includes subset of information
what isn't included:
not giving away complete sets
trying to drive traffic back
What can be done?
Create mashups
Restrictions:
Canit' modify
can't change or distort meaning
can't imply nyt endorsement
can't display any other logo other than nyt that implies ownership
API Limits
qps, and rate limiting
Emails sent if user exceeds limit
Can't be used for commercial use, except for campaign finance api
may loosen this policy up.
TimesOpen Developer Day
Februay 20, coming out party for APIs.
Posted at 06:10 PM | Permalink
From business of apis conference, my notes:
Dan Jacobson - NPR
45M pageviews, 8M monthly uniques
Major Media lagging other verticals in API exposure, instead sticking with RSS.
Target Audiences:
end users
npR's digital media team
NPR member stations
NPR biz partners
Content aggregators like Yahoo and Google
Measures of Success:
Business
- get content out to stations
- improve connections with biz partners
- scalability for npr.org
- creat epossible revenue opportunities
Audience
- distribution of content to existing users
- reach new users
- drive traffic back to npr
- extend the npr brand
challenges:
legal
- rights for redistributing content (e.g. ap photos)
- terms of use
- identifying what needs to be withheld
Business challenges
- enable sponsorship functionality
- create levels of access for different targeted audiences
Metrics
- identify users through registration
- track api requests
No access restriction, no query caps.
beacons to determine where the content is residing.
all audio content is served from npr servers
More challenges:
stations - how to give stations more value
monitoring / policing
technical
-handle load
- create registration system
- extend application architecture
Everything on NPR.com is available through the api
250K stories
400K unique audio files
5700 unique lists
400 musical artists
twelve progams
90 topics
700 editorial columns
content from member stations
what's not in it:
some npr programs
Non npr public radio programs
some text, images, audio
video and blogs not offered
Stats
1000 registrants
4.3M requests
200K pvs on tech center documentation
Formats:
NPRML
RSS
MediaRSS
JSON
Atom
JavasScript widgets
HTML widget
Future:
station finder
posting to the api
full story html widget
Content and Format
other output formats
blogs
video
other npr programming
more content from member stations
geo information from stories
create your own podcast
Posted at 05:49 PM | Permalink
These are my live notes from Michelle Azar's presentation at the Business of APIs conference.
Best Buy, VP emerging channels
150K employees, most tech savvy, most under 25, many developers.
Growth will come not from making one big store. But by putting stores where the people are. APIs reflect this.
Best Buy Remix is the Best Buy API
Affiliates was the first effort
API now available for commercial use.
API Objectives
Brand Presence
Traffic, and conversion
Brand is the priority for now.
Clip from Best Buy CEO: it's hard for managers to accept that innovation can come from grassroots, but it's the reality.
Metrics:
net revenue
developer engagement
api usage
traffic to commerce sites
Things we can't measure yet
- emabling customer delight
Posted at 03:57 PM | Permalink
These are my live notes from Allen Hurff's of MySpace's presentation at the Business of APIs conference.
Allen Hurff from MySpace (svp myspace)
History of MySpace APIs
2006
Helio Project
SOAP
SnoCap
Didn't even have internal apis at thsi point.
2007
Built a mobile API team.
Core API Team
OpenSocial .5
REST
2008
OpenSocial .8
REST
Oauth
OpenID
Data Availability
Developer Platform Team
Web Identity
Content Generation
Social graph
reputation
authentication
profile
activity
What makes me me:
what i say about me
what others say about me
What i do
my social graph (xfn - appending urls to indicate relationship in a way that is crawlable, e.g. rel=friend, rel=me)
what I create
my id
THe Old World
walled garden, gap btw social sites and social networks
New world: open identity
Push out from the social network, and draw activities back in.
Identity standards:
activity: rss, atom, xmpp, opensocial
profile, hcard, hcal, opensocial, sreg, attribute exchange
authentication: oauth, openid
social graph: opensocial, xfn, foaf, portablecontacts
repution: microID
content: xpspf, exif, etc.
why this matters:
user acquisition - lowers barrier to entry
retention - become complementary and indispensable
traffic - make your service a fulcrum
monetization
everyone is building on the open stack
Social Windows: Google Friend Connect - make it so a mom and pop blog social
Closing THoughts:
web is open and social, and standards can power social functionality
Posted at 03:54 PM | Permalink
These are my live notes from the Netflix presentation at the Business of APIs conference.
Michael Hart - director of engineering at Netflix
Strategy:
API Goals
Enable Dvlpt of of Netflix experiences offsite
Leverage External developers, platforms, and users of platforms, ideas
Success Measured by:
- registered developers
- live applications
- subs engaging with netflix
new sub acquisitions
- subscriber delight
Partner Value Prop:
- Engagement with 8.7M subs
- association with a strong brand
- No ads. No pressure to drive traffic, as its is subscription business
- help partners make money - subscriber bounties
Design of API
Long tail of developers
- online support tools, developer registration, mashery helped
Enabling scale through design:
Simplicity - fastest ramp up, used REST, modeled around netflix catalogue, leveraged standards
Flixibility - wide range of functionality, technical javascript
Authority - about standards, avoding negotiation of terms, e.g Oauth
API Components
JavaScripti API
Rest API - developer controls ux
RSS and Atom Feeds
JavaScript API
Simplest API. embed netflix buttons
REST API
title search, ratings (read and write), subscriber activity, queue mgmt, previews (limited basis)
1612 registered users
197 active
mobile applications
one iphone app
amrtflicks for Blackberry
expect to get
Nice - gave rateitall props
LoveNetflix - dating site
Plaxo used javascript api and rest...
Lessons learned:
oauth is tough in the short term. It's a new standard.
good documentation isn't easy
APIs foster internal development too
What's next?
Application showcase
Opening up catalog access even more, may even lose authentication
More widgets (add and play today, add box art and stuff
Parting Thought
Put your data to work. (nice mention of a Sexy Widget post)
Posted at 03:52 PM | Permalink
