Flags and Lollipops - Bioinformatics Blog

Disappointed with Popfly

2008-05-24T22:46:00.002+01:00

Popfly is the mashup editor that Microsoft released last year. The idea is good. The 3D graphics are good. Silverlight is a bit buggy in Firefox (sidebars don't always redraw properly) but that's OK.

If you're going to create a web 2.0 mashups builder, though, don't you think it's be a good idea to provide some Atom support?

Meta-analysis

2008-05-19T17:08:00.006+01:00

The journal platform team here at NPG just rolled out machine readable metadata for the papers we publish in Dublin Core, PRISM (good PRISM, not to be confused with evil PRISM) and Google metadata formats.

No more scraping to automatically get the citation for a paper, it's all in the HEAD:


<meta name="citation_journal_title" content="Nature" />
<meta name="citation_publisher" content="Nature Publishing Group" />
<meta name="citation_authors" content="Paul Schenk, Isamu Matsuyama, Francis Nimmo" />
<meta name="citation_title" content="True polar wander on Europa from global-scale small-circle depressions" />
<meta name="citation_volume" content="453" />
<meta name="citation_issue" content="7193" />
<meta name="citation_firstpage" content="368" />
<meta name="citation_doi" content="doi:10.1038/nature06911" />

Useful for apps like Zotero and Connotea (which before now downloaded two files each time you bookmarked a Nature paper: the page itself and then the linked EndNote file to parse).

The metadata will be there for all papers going forward and back through some of the archives.

For fulltext indexing of papers behind the paywall you can use the linekd OTMI file (I only just saw Twease, which does just that) although there's only OTMI for Nature papers at the moment, I think.

Lastly at some point in the future we're aiming to put XMP metadata in our PDFs, which should make it much easier for scripts and applications (like Papers) to look at PDF files on your filesystem and work out what they represent.

Nice work Pedro!

2008-04-18T11:41:00.004+01:00

Noticed while leafing through today's Nature that Pedro has a paper out (Isalan et al., Evolvability and hierarchy in rewired bacterial gene networks).

There's more on this over at Public Rambling.

Ian owes me a pint

2008-04-17T16:34:00.001+01:00

(update: Gavin Bell at Nature gave up one of his app spots so that I could put this live, which I did: only to discover that Google App Engine is even more unforgiving of timeouts than Facebook. Currently trying to work out how to make the bookmarking process, for now it doesn't work very well. Also the search is broken, though that's Google's fault and not mine.)

I bet Ian earlier that I could rewrite Connotea on App Engine in six hours. I can't remember why. Probably ego (mine, I mean). He didn't actually bet me a pint but he should have done...

... because the original estimate was a tad optimistic (ahem). After twelve hours I've produced pycite, though, which is pretty good going I think. I'll admit it: Python is actually very cool.

pycite is three hundred lines of logic and a set of html templates that implements a (very simple) social bookmarking service. Sadly I don't actually have an App Engine account so it's not live on the web anywhere (I'll buy whoever does have an account and puts it up first a pint - let's spread the love), you'll have to download it and run it locally to see it in action.

What you can do with it:

run it without owning a server of your own

add new bookmarks (the citation will be collected automagically)

view everybody's bookmarks

filter bookmarks by user:
```
http://path.to.pycite/users/bob.smith
```

and by tag:
```
http://path.to.pycite/tags/diabetes
```

and by user and tag:

http://path.to.pycite/users/bob.smith/tags/diabetes

and by keyword (the full text of each bookmarked page is searchable):
```
http://path.to.pycite/users/bob.smith?q=t2d
```

get atom feeds for all of the above

What you can't do with it (yet):

edit or delete bookmarks

anything else

I've put it all up on Google Code. It's fairly straightforward stuff so if you've got any brilliant social bookmarking ideas then go for it. Send me an email and I'll give you write access to the subversion repository.

Gaggle

2008-04-07T14:26:00.003+01:00

I hadn't heard of Gaggle before but both Deepak and Sutee Dee (who needs a homepage.. ;)) from the ISB mentioned it last week so I figured it was worth a look. It's a system built by Paul Shannon at the ISB in Seattle to share data between different bioinformatics applications on the fly. It has been around for a while, I think - there was a BMC Bioinformatics paper describing the system in March 2006.

A small server program (the ´Gaggle Boss´) provides communication among analysis and display programs (the ´geese´) which are modest and minimal adaptations of existing (or novel) bioinformatics and computational biology programs, and web resources. The Boss and the geese all run as separate programs on the user´s desktop computer, communicating with each other, at the user´s behest, by passing simple messages.

(from the ISB's 'about Gaggle' page)

I ran through a tutorial showing data sharing between (modified versions of) Cytoscape (also developed by ISB), R and a data matrix viewer no problem. Quite cool.

You can't share data from an arbitrary application (I don't think?), they need to be modified to send messages to the Boss goose. Having said that there's a Firefox extension called Firegoose which lets you pass messages to and from web apps, Entrez etc. I couldn't get it working properly but suspect that's something to do with my install rather than the extension itself.

Anyway, it's good to see stuff like this. Truth be told it's not the slickest thing ever, but it's still pretty cool - and it works. I wonder if you could turn it into a simple lab notebook - could you write a brief description of what you're going to try and do for the Boss app every time you send data to another app or something?

Why you should try online dating

2008-04-04T00:22:00.006+01:00

(you can jump to the short answer here, if you're feeling impatient)

Onto the psychology of social media. Kristin Stecher of the University of Washington and Dave Evans of Psychster LLC both gave interesting talks about profile pages.

Psychster is a consulting company dedicated to "the social science of social networking". Recently they've been looking at interpersonal perception (how does person A perceive person B? How close is that to B's self perception?). Most research into this uses 'fake' people - i.e. A is given a detailed written description of B and works off of that, rather than meeting anybody face to face.

To try and get a large 'real people' dataset Psychster created a Facebook application (and later a website) where users could fill out a questionnaire that rated their personality on a variant of the big five personality inventory (the big five being openness, conscientiousness, extraversion, agreeableness, and neuroticism). They then had the option of rating the personalities of other people (not just their friends), the idea being to collect how users saw themselves, how others saw them and the correlation between the two.

On the standalone website users created profiles to reflect their personalities. Profiles could contain any number of elements (name, location, gender, favourite movie, most embarrassing moment...) chosen from a large list.

The results in general:

people do 'get' each other (where to 'get' a person means to guess a personality close to their actual, self-rated personality).
people on Facebook get each other better (this kind of figures - you'd want to go rate your real life friends).
women are better guessers than men - but only when guessing random strangers.
women are easier to get.

Psychster looked at different profile elements on the standalone website to see if the presence or any in particular were correlated with higher rates of accuracy.

Profile elements that make somebody easier to get:

A link to a funny video (the number one predictor of personality)
What makes me glad to be alive?
Most embarassing thing I ever did:
Proudest thing I ever did:
My spirituality:
A great person:
I believe this:

Profile elements that make you harder to get:

Profile picture (but only if it is of a non-person)
An awful website:
An awful person:
A great book:

That last one (naming a great book making it harder to guess your personality) is pretty interesting. Dave did say that he hadn't yet done any proper analysis of why it might be. I wonder if there's any research into how much (or little) reading habits have to do with your personality? Here's a tangent (why do some people get interested in science fiction?) if you're interested. Here's another (people who read lots of fiction aren't socially awkward, in fact the tendency to get absorbed in a story correlates with empathy scores).

OK, anyway...

Why were women easier to read? Because they tended to fill out the profile elements that were good predictors ("my most embarrassing moment").

At this point you might be wondering (well, I wondered) who cares how well an online profile reflects your true personality. One answer is the online dating industry who have a vested interest in not setting you up with anybody plainly unsuitable. If profiles were set up the right way then maybe you could tell in advance if the guy or girl messaging you is worth seeing in the real world.

Sticking with the online dating theme, it turns out that the levels of agreement (between actual and guessed personalities) you get by looking at Facebook profiles approach those you see in long term acquaintances. They're certainly better than what you get after a short face to face meeting (like a date). In fact, short f2f meetings are particularly bad at helping you gauge levels of agreeableness and neuroticism - not good. I think this means that stalking potential partners online actually makes good, practical sense and should be encouraged.

In case you needed any reassurance.

Do you use language differently when you're depressed?

2008-04-03T19:32:00.004+01:00

Can you tell if somebody is clinically depressed by analyzing their use of language? I'm not a psychologist, so take the background info below with a pinch of salt but the topic came up at ICWSM (more on how later) and I thought it was fascinating.

In 2001 Stirman et al compared the collected works of nine poets who eventually committed suicide and nine poets who didn't (as a control set). Their theory was that the depressed (and eventually suicidal) poets would use more first person singular (I, me, my) and words related to hopelessness and desperation (hate, worthless, death, grave) and that was supported by the data.

Rude et al later found something similar when they compared essays (on a common topic - "coming to college") written by college students. Depressed students used "I" and negative words significantly more often than controls.

Interestingly Oxman et al has found that spoken language patterns can be a good discriminator for classifying patients as depressed or not, so it's not just written language use that may be different.

Anyway, at ICWSM Nairán Ramírez-Esparza from the University of Texas presented a language analysis of some depression discussion boards on About.com. She ran a two part study: the first to confirm Stirman and Rude's findings and the second making use of the fact that the About.com boards are bilingual (there's a Spanish section too) to see how different cultures talk about depression.

Her approach was pretty simple - she collected ~ 400 posts from the depression forum and 400 posts from a breast cancer forum as a control, broke each post down into single words and then used off-the-shelf software to classify them (as verb, adjective, pronoun, positive emotion, negative emotion, etc.). She did this for both English and Spanish sections of the site.

Her results seemed to confirm the earlier studies: first person pronouns were found three times more frequently in the depression forum posts than in the controls and words relating to negative emotions occurred four times as frequently. This was true for both English and Spanish datasets.

The second part of her study was to see if English and Spanish speakers approach depression differently; what do they talk about? She studied this by using normalized word frequency counts then grouping different words into themes.

The top five themes discussed in the English dataset:


Treatment (medicine, doctor, therapist...)
Disclosure (tell, discuss, talk...)
Family (mom, dad, brother, sister...)
Symptoms ...
School

And the top five themes from the Spanish dataset:


Family
Relationship history  
Hopelessness
School
Treatment

I'm a bit suspicious of results that are so intuitively appealing (family and romance are more important to Spanish people?). One thing that I did wonder was how much the results are skewed by different community expectations: if you visit a discussion forum where people are sharing stories about their depression and everybody else mentions their family maybe you feel compelled to mention your family too. Maybe the English language forums are dominated by a younger age group and so older visitors shy away, or v.v.

Anyway, it was interesting stuff. Somebody in the audience wondered aloud if this means that you could build a system to identify people at risk of depression (or perhaps more to the point suicide) by analyzing their language online. Maybe this could be built into the next version of the anti-plagiarism software used in high schools and colleges (I'm not advocating that, just saying)...

Analyzing MySpace profiles

2008-04-02T02:26:00.011+01:00

This morning James Caverlee presented his study of almost two million (well, two sets of ~ one million - one set of profiles picked at random and one gathered by traversing the social graph) MySpace profiles. It was interesting stuff. Some bits and pieces below.

MySpace users live up to gender stereotypes, rather disappointingly:

Words most frequently appearing in MySpace profiles

Women	Men
love, people, dancing, life, shopping, can, girl, family, hearts, being, have, notebook, are, dance, favourite, things	dating, sport, networking, metal, serious, football, relationship, sht, single, wars, straight, band, video, fck, guitar, gay

And geographic ones (didn't manage to write all of these down in time):

users in Oregon	users in Alabama
camping, hiking, pixies, snowboarding, wine, vegans	football, jesus, gospel, nascar

Demographics wise ~ 50% of the profiles that they picked at random had one or no friends (i.e. weren't active). Age wise the peak is at 24, with smaller peaks at 69 and 100. The 69 peak is a secret MySpace code, apparently - it means that you're interested in, uh, one-handed typing (this wasn't made clear, but I'm guessing). By having a common age - 69 - you can use MySpace's advanced search to find others looking for the same thing. 69 year olds on MySpace are most similar (in their use of language) to people in their mid thirties.

Younger users are overwhelmingly female. There is a 2:1 ratio of girls to boys at age 14. This difference decreases as age increases. The flip over point is at 20 - after that you start seeing more men than women.

About 20% of the profiles in the connected dataset were marked as 'private'. Over time this percentage is rising. Having privacy preferences set is negatively correlated with age.

He had a fantastic slide showing top terms wrt to age... will post it and a link to the slideshow when it's online.

Tossed Salad and Scrambled Eggs

2008-04-01T06:43:00.003+01:00

I'm in Seattle for the ICWSM. The first day just finished and I'm going to blog about the more interesting talks tomorrow when I'm more awake. In the meantime:

Crowdvine for conferences is actually pretty useful
Lions for Lambs is terrible
St Trinians is actually quite good
The Century of the Self is brilliant
Seattle looks really nice from the air
Note to self: US Milky Way bars = UK Mars bars, tricksy bastards
Steak + beer + bay views = awesome (thanks Deepak!)
More Starbuckses than normal
Everybody is disconcertingly friendly. People keep offering to take me skiing. And to see waterfalls. People here big on waterfalls
MS Live Labs are hiring
Brad Fitzpatrick is a great speaker but I found his talk disappointing - too much hand waving about OpenID / OAuth / XMPP / XRDS. Dude, it's a room full of social network developers, you're preaching to the converted
Sadly Bernardo Huberman has cancelled. Marc "most unGoogleable name ever" Smith is talking instead. Marc is either founder of Poetry Slam (cool) a Happy Hardcore DJ (not cool) or a senior research sociologist at Microsoft Research (as yet undecided)

Dawkins officially bigger than Jesus - datamining Scienceblogs.com

2008-03-19T11:31:00.008Z

I've run all of the posts from Scienceblogs.com in 2007 through the ClearForest API. ClearForest extracts entities - people, places, organizations - from plain text.

I'm in the process of pulling things together for a visualization, but here's a quick answer to the 'who are Sciblings talking about?' question. The 'count' is the number of times that each entity was seen (could be multiple times in the same post) across 2007.


+-----------------------------------------------+-------+
| term                                          | count |
+-----------------------------------------------+-------+
| Michael Egnor                                 |  1855 | 
| Richard Dawkins                               |  1737 | 
| Bush                                          |  1669 | 
| Congress                                      |  1430 | 
| Charles Darwin                                |  1226 | 
| Michael Behe                                  |  1031 | 
| Chris Mooney                                  |   927 | 
| FDA                                           |   920 | 
| DCA                                           |   765 | 
| National Aeronautics and Space Administration |   745 | 
| National Institute of Health                  |   741 | 
| Bush administration                           |   721 | 
| Google                                        |   700 | 
| Guillermo Gonzalez                            |   691 | 
| White House                                   |   658 | 
| Supreme Court                                 |   655 | 
| Thomas Jefferson                              |   632 | 
| John Edwards                                  |   614 | 
| Casey Luskin                                  |   605 | 
| George W. Bush                                |   603 | 
| Jesus Christ                                  |   601 | 
| Discovery Institute                           |   596 | 
| the New York Times                            |   587 | 
| Larry Moran                                   |   576 | 
| World Health Organization                     |   543 | 
| Hillary Clinton                               |   517 | 
+-----------------------------------------------+-------+

Bear in mind that ClearForest extracts entities, not key terms. It can't tell us how often blog posts are talking about mammoth DNA, supernovae or dicyemid mesozoa. That's a different dataset entirely...

.... this one, in fact, generated using the Yahoo! term extraction API which pulls out important concepts (terms) from text. The dataset is about half the size of the above as I'm only including ScienceBlogs indexed in Postgenomic. Here 'count' is the number of distinct posts containing a term:


+---------------------+-------+
| term                | count |
+---------------------+-------+
| evolution           |   963 | 
| carnival            |   923 | 
| global warming      |   640 | 
| intelligent design  |   543 | 
| new york times      |   542 | 
| blogosphere         |   468 | 
| religion            |   460 | 
| brain               |   437 | 
| climate change      |   432 | 
| creationist         |   420 | 
| birds               |   415 | 
| creationism         |   409 | 
| creationists        |   398 | 
| pz                  |   378 | 
| darwin              |   367 | 
| discovery institute |   354 | 
| atheists            |   351 | 
| atheist             |   333 | 
| biology             |   314 | 
| richard dawkins     |   301 | 
| skeptics            |   290 | 
| love                |   289 | 
| genes               |   288 | 
| job                 |   286 | 
| money               |   283 | 
| orac                |   281 | 
| god                 |   276 | 
| atheism             |   266 | 
| animals             |   261 | 
| bush                |   258 | 
| google              |   258 | 
+---------------------+-------+

In light of this data it's tempting to revisit that Bayblab post suggesting that Sciblings spend too much time discussing ID. That'd be a mistake, though: the numbers above are absolutes. 963 posts had 'evolution' as a key term but that's only 2.4% of all posts that year (my 2c: I think that Sciblings do talk about Egnor, ID and creationism too much, but hey, it's their blogs - I just skip over those posts).

I also had a look at linking patterns - who do ScienceBloggers link to the most? Here 'count' is the number of unique posts that have a link to a particular domain.


+-------------------------+-------+
| domain                  | count |
+-------------------------+-------+
| www.scienceblogs.com    | 15966 | 
| en.wikipedia.org        |  2016 | 
| www.technorati.com      |  1797 | 
| www.nytimes.com         |  1388 | 
| www.amazon.com          |  1078 | 
| www.sciencedaily.com    |   661 | 
| www.washingtonpost.com  |   478 | 
| feeds.feedburner.com    |   467 | 
| www.nature.com          |   453 | 
| news.yahoo.com          |   401 | 
| news.bbc.co.uk          |   333 | 
| www.youtube.com         |   305 | 
| www.del.icio.us         |   297 | 
| www.cnn.com             |   260 | 
| www.eurekalert.org      |   260 | 
| farm3.static.flickr.com |   259 | 
| www.sciencemag.org      |   231 | 
| www.ncbi.nlm.nih.gov    |   225 | 
| www.pandasthumb.org     |   224 | 
| www.google.com          |   219 | 
| www.latimes.com         |   213 | 
| www.gnxp.com            |   208 | 
| sandwalk.blogspot.com   |   197 | 
| www.dailykos.com        |   196 | 
| www.donorschoose.org    |   194 | 
+-------------------------+-------+

Presumably the technorati links are from tags. Sciencebloggers link to scienceblogs.com far more than anywhere else - but I'd guess that this is simply because there are a lot of good science blogs on one domain there.

Wikipedia's reliability might be in question but it's interesting that almost everybody uses it to define terms.

Drilling down, where do ScienceBloggers link to papers?


+--------------------------------+-------+
| domain                         | count |
+--------------------------------+-------+
| www.nature.com                 |   241 | 
| www.sciencemag.org             |   194 | 
| www.dx.doi.org                 |   177 | 
| www.ncbi.nlm.nih.gov           |   111 | 
| www.pnas.org                   |   104 | 
| www.plosone.org                |    89 | 
| biology.plosjournals.org       |    76 | 
| content.nejm.org               |    67 | 
| medicine.plosjournals.org      |    65 | 
| www.sciencedirect.com          |    43 | 
| www.arxiv.org                  |    33 | 
| genetics.plosjournals.org      |    22 | 
| www.jneurosci.org              |    15 | 
| www.cell.com                   |    14 | 
| compbiol.plosjournals.org      |    10 | 
| pediatrics.aappublications.org |    10 | 
| www.jcb.org                    |    10 | 
| mbe.oxfordjournals.org         |     9 | 
| www.ajp.psychiatryonline.org   |     8 | 
| www.current-biology.com        |     8 | 
| www.journals.uchicago.edu      |     8 | 
| www.plosntds.org               |     8 | 
| www.blackwell-synergy.com      |     7 | 
+--------------------------------+-------+

Nature and Science are at the top, perhaps unsurprisingly - but if you add up the counts from the different PLoS journals it'd be up there too.

Science streaming

2008-03-19T00:39:00.003Z

Michael Barton has a nice post up:

I currently use Subversion to back up my project files, and I noticed Twitter status updates are very similar in length to subversion log messages. I created a short script so that every time I do a subversion repository check in, the message is also sent to Twitter.

I'd like to see activity aggregators accept arbitrary updates - sort of like Facebook's Beacon updating people's News Feed, but done properly.

Nature archive visualized - draft

2008-03-18T03:08:00.006Z

I'm using up my annual carry-over vacation days by taking some time off work this week. Normal people probably use this valuable breathing space to bond with their loved ones, play badminton and learn exciting new hobbies. So far I've sat alone in my flat for thirty six hours straight writing Processing sketches *.

So... here's a draft visualization (14mb MP4, should play in your browser with Quicktime) of the key words and phrases found in Nature journal over the past thirty years.

The video starts with the phrases from 1970 and continues until 2007.

Phrases appear on the right in the year that they were first seen, then travel leftwards, disappearing in the year they were last seen.

The size of each phrase is related to how often it was seen relative to all the other phrases.

The hue of each phrase is related to how many distinct journal issues it appeared in - green / yellow phrases are relatively transient while red / brown phrases are stable, appearing in many different contexts.

The data is incomplete (it's a bit sparse after '88) and I took lots of shortcuts to see how things might look, so don't read too much into which phrases appear and when for now... a better version will follow - this is just a release early, release often draft.

Eventually I'd like to have a sort of Pop-up Video timeline of science from the 50s till today, with major events (and relevant terms) flashing up on screen.

If you're particularly impatient here's a version from Vimeo. The quality is rubbish, mainly because I munged the file with iMovie (which is crap) to add some rockin' beats. I still suggest you get the mp4 instead, though.

Tommorrow I'm going to the park.

* I watched Rendition, too, it was quite good.

Seattle

2008-03-11T10:44:00.005Z

I'm going to be in Seattle the first week of April for the ICWSM.

There are a whole bunch of awesome looking talks and one or two wild cards. Wild cards like:

Spontaneous Inference of Personality Traits from Online Profiles
Kristin Stecher, Scott Counts

Which sounds interesting, anyway.

Let me know if you're in the area and fancy meeting up for lunch or a drink. I'm in town from the 29th of March to the 5th April.

New JoVE blog & commenting on papers

2008-03-10T23:11:00.003Z

Anna Kushnir's new blog for JoVE is up and running (actually it has been up and running for a while, I'm a bit behind with blogging. Those January Open Science posts are coming at some point, too). It's a nice mix of content.

Of particular interest are a couple of interesting entries talking about the online participation - or lack thereof - of scientists. See also Noah Gray's take on neuroscientists and web 2.0 and David Crotty's 'why web 2.0 is failing in biology' post.

Did you skip over all those links? You shouldn't, really. At least read David Crotty's.

So, yeah, anyway, why scientists don't comment on papers - my take is that being too busy and being afraid of the consequences don't come into it.

Sure, they're valid concerns - but everybody is busy at work and everybody realizes that what you say on the internet is recorded forever by Googlebot. People still write ranty forum posts and blog comments.

IMHO the main reasons scientists don't leave comments are:

There's no point - who's going to read it? Will you get any feedback? Will you get any credit for it?

and

It's too much work - writing a comment should be a one click operation. Well, two clicks, one to get the focus in the textbox and the other to press 'submit'.

Science publishers can address both of these issues, but we've been failing to do so.

Hot diggity

2008-02-21T01:09:00.002Z

The Collective Intelligence Foo Camp looks awesome, especially if you're into blogs and recommender systems (and who isn't, right? Right...? Everybody?).

Some particularly cool attendees:

Greg Linden (worked on Amazon's recommender system, great blog)
Matt Hurst (ex of Nielsen BlogPulse, now at Microsoft, great blog)
Niall Kennedy (ex of Technorati, Microsoft, bunch of other places, great blog)
Bernardo Huberman (also talking at ICWSM '08 next month, sad lack of blog)

(via Greg Linden)

Uh, lazyweb...?

2008-02-19T14:18:00.005Z

Update: Thanks, universally helpful lazyweb!

I know it was only a month ago that you were invoked here last. But... Freebase. It's killing me (I was inspried to finally build something using it by Pierre, whose examples I have sadly failed to adapt :( ).

What MQL can I use to get *all* of the information about *all* people with profession of 'author'?

I have



{
  "query" : [
  {
    "*" : [
       {}
    ],   
    "guid" : null,
    "limit" : 5,
    "name" : null,
    "profession" : "author",
    "type" : "/people/person"
  }
  ]
}

(thanks to Pierre for pointing out a syntax error here originally, oops)

Which should list all of the properties for 5 authors, right? But I only see properties from the /people/person schema.

How do I get all the author properties too?

I ended up using the MQL in Pierre's second comment to get a list of GUIDs for all authors (having changed the limit to 10k) then iterated over them getting all of the properties from /people/person (D.O.B, nationality...), /book/author (list of books) and /common/type (image, article) in three different calls (oy). It works, though - again, as Pierre suggests. ;)

There is a way to get everything in one MQL call, though, as Alf says:

{
"query" : [
{
"*" : null,
"/book/author/books" : [
{
"*" : null
}
],
"/common/topic/article" : [
{
"*" : null
}
],
"/common/topic/image" : [
{
"*" : null
}
],
"limit" : 5,
"profession" : "author",
"type" : "/people/person"
}
]
}

The disadvantage to this is that it's a lot of data to get in one go (considering there are thousands of authors each with lots of books)... I guess that's where paging through the results would come in (as Brendan correctly predicts).

And finally, for future reference.... skud points out that the mailing list is here.

Publishing house scale of web server evil

2008-02-17T15:22:00.008Z

Kevin Burton recently checked to see which operating system the websites of different US presidential candidates are built on. The executive summary: Democrats use lots of Linux while Republicans (Ron Paul excepted) mainly use Windows.

Some have suggested a correlation between Windows web server usage and being evil*. This makes sense as only somebody with no soul could love ASP.

Does the theory hold true in the publishing world?

PLoS run Linux
Nature run Linux
Science run Linux
Wiley run Solaris
Elsevier run Windows 2000
Springer run Windows 2003

So from a purely progressive science on the web point of view.... yeah, sort of.

Springer, Elsevier and Wiley are pretty big companies and have lots of different sites, so maybe it's doing them a disservice to assume that whatever serves their root domain is their primary choice of OS. For example, I couldn't tell what Elsevier's ScienceDirect site runs because NetCraft returns 'unknown', so maybe it runs Linux.... or maybe NetCraft just doesn't have an entry for CRUSHED UP PUPPIES AND THE SWEAT OF THE OPPRESSED.

Trade publishing, for completeness:

Canongate (arty Edinburgh based independent) run Linux
Penguin (ironically) run Solaris 8
Macmillan (publish Jeffrey Archer, employ me) run Windows 2000
Simon & Schuster (publishing Lindsey Lohan's autobiography) run Windows 2003
HarperCollins (owned by News International) run Windows 2003

* Don't read this the wrong way. Microsoft do lots of cool stuff nowadays too. But IIS is cold and heartless.

Nature on Facebook

2008-02-12T21:24:00.000Z

It's in my Nature.com is Nature's splendid new Facebook group (there's a fan page, too, with a selection of fresh Nature.com content on it in case you need a quick fix of science news without leaving the 'book).

Anyway, it's splendid because it's a chance to interact with NPGers on an informal level - or rather a chance for us to interact with students & scientists on an informal level. The group started out as a mini-project run by a couple of Facebook early adopters and while it has picked up a lot of advice and support from within the company since then it's still a friendly place where everybody knows your name*.

Some of that company support has come in the form of free Nature-header-red iPods, so sign up and keep an eye out for giveaways in the near future.

(if you haven't seen it already you should also check out the PLoS group. After you've joined the Nature one, of course).

* though it's not the right place for ask the editor type questions

Where I lazily recycle news from OpenHelix

2008-02-10T14:36:00.001Z

Geoff Bilder over at CrossRef has announced that their citation plugin for MT and WordPress is now available for download.

CrossRef is the shadowy cabal that runs the DOI system for journals. Shadowy because despite the fact that DOIs underpin academic publishing who outside of publishing tech circles has ever heard of them? Anyway, they've been doing a lot of cool science 2.0 stuff recently, probably because of Geoff, for whom I have a lot of respect. He does need to update his blog more often, though. ;)

(via OpenHelix)

Also at OpenHelix is a writeup of Gene Characterization Index: Assessing the Depth of Gene Annotation, published in PLoS One at the end of last year. It's a nice project.

One (very) minor niggle: the whole genome dataset they provide looks like this:

Gene ID GCI
2 10.0
19 10.0
24 10.0
25 10.0
...

WTF kind of gene ID?

It is blatantly obvious once you've read the actual paper (they're from Entrez), of course, but still. Dudes, some better column names or /* descriptive comments */ wouldn't go amiss.

Viral spread visualized, update

2008-01-15T01:38:00.000Z

Viral spread of Facebook app from Stew on Vimeo.

The movie above is a (blurry when streamed, try the 3.6mb MPEG download) visualization of the signup and invite data from Bookshare from July 18th of last year to now. As users sign up they are geolocated and plotted on the map. More users signing up in a particular location mean bigger, brighter spots. Invites from one user to another are drawn as snaking lines that travel from the city of the inviter to the city of the invitee.

It's written in Processing.

I'd love to hear from anybody who owns a large Facebook application and has kept id / install date / invitee data (bearing in mind Facebook's data privacy rules...). See also this.

First draft available here (1.6mb MPEG)

Lazyweb, I invoke thee

2008-01-13T03:14:00.000Z

Dear maths geeks,

On Bookshare each user owns a set of books. The same book can be owned by multiple users.

What algorithm can I use to find the smallest possible set of books that contains at least one book from each user?

LAMP performance for dummies (e.g. me)

2008-01-11T21:47:00.000Z

I've spent the last six months combing Google for 'slow mysql' results. It was enlightening. Now in the spirit of giving back to the community I give you:

Euan's Top Tips For Medium Sized LAMP Powered Web Apps

Where 'medium sized' means you have four or five concurrent users on ten tables with around half a million rows each and 'top' means that you've already done all the basic stuff - picking table types, adding indexes, designing the database properly in the first place etc.

Some are common sense. Some are not appropriate for all situations. Use at your own risk.

If most operations are reads turn the MySQL query cache on.

Don't use mysqldump for backups. Users don't like five minutes of being unable to write anything to the database because the tables are locked. Use mysqlhotcopy or an LVM based system if you're feeling adventurous.

If you're retrieving large amounts of data with PHP but you don't need the entire rowset in memory at once (or mysql_num_rows()) then use mysql_unbuffered_query instead of mysql_query. Stops you running out of memory so fast.

xdebug is a PHP profiler. With kcachegrind it's like hot coder porn. It's incredibly useful to see how long your code takes to run and why.

in_array()s are innocent looking... and very slow. If the values are unique then use associative arrays instead - i.e. instead of


$data = array("a", "b" .. large amount of data .. "zzz");

for ($i=0; $i < 1000; $i++) {
   print "is c in array? ".in_array("c", $data);
}

use


$data = array("a", "b" .. large amount of data .. "zzz");
$a_data = array();
foreach ($data as $point) {$a_data[$point] = true;}

for ($i=0; $i < 1000; $i++) {
   print "is c is array? ".isset($a_data["c"]);
}

key_buffer_size is the amount of memory that MySQL puts aside to keep table indexes in memory, which is a very good thing. As a rule of thumb it should be 25%-50% of your database server's total RAM (reduce this if you're also using it for other things, natch). The default is 8mb, so once you've got more than 8mb worth of indexes...

How can you tell if you need a bigger key buffer size?
```
mysql> SHOW STATUS LIKE '%key_read%';
+-------------------+---------+
| Variable_name     | Value   |
+-------------------+---------+
| Key_read_requests | 6375479 |
| Key_reads         | 130562  |
+-------------------+---------+
2 rows in set (0.00 sec)
```
There should be at least 100 key_read_requests (from memory) to every key_reads (from disk), as above.

Do a lot of sorting? Increase sort_buffer_size but bear in mind that unlike key_buffer_size it is allocated per connection - i.e. increase it to 32mb and MySQL will gobble 32mb x 4 = 128mb the next time you have four connections open at once (also remember that your code will - probably - sometimes fail or forget to shut down the connection).

Do a lot of ORDER BYs? Increase read_rnd_buffer_size. Allocated per connection, as above.

Fetching frequently used data from disk can sometimes be faster than using memcached or similar (no, really. Far less overhead + the file is cached by OS and so is in memory anyway).

serialize() and unserialize() are slow. Use JSON where possible instead (nowhere near as flexible, sadly)

To paraphrase the best optimization is avoiding doing the work in the first place: do you really need to return ALL of the rows, or just the first few hundred? Do you really need to sort them? Do they need to be distinct? Is it faster to get more data than you need and process it in code?

Avoid ORDER BY RAND(). Do something clever in code (pick some randomly selected ids?)

SQL_CALC_FOUND_ROWS can be faster than rerunning a query with COUNT() - but not always.

Facebook code on Bebo

2008-01-10T18:47:00.000Z

Yes, nothing to do with bioinformatics, but social networking:

Bebo announced that it'd support Facebook style apps as well as OpenSocial late last year. They've launched properly now.

The documentation has been up for a while already and there aren't any real surprises, though there's a couple of new Bebo specific tags (which seems like a bad idea to me; I mean, the whole point is code compatibility between multiple systems).

One major omission is Javascript support; there's no equivalent to FBJS (Facebook's sandboxed Javascript version). This means that a lot of the bigger apps will hold off moving over, I'd have thought - though it's not a problem for simpler quizzes and Flash based stuff like Scrabulous.

There are a couple of OpenSocial containers now but I haven't heard of any Facebook server components being made available (just a 'top level overview' document on their wiki). We've just finished building an API for Nature Network (private, for now), I wonder how much work it'd be to start supporting apps?

Diversions, excuses, etc.

2008-01-05T16:33:00.000Z

OK, so it's going to be a week of open science spread out over the month. Probably. I've been distracted by:

Silverlight: actually it's quite cool - writing simple multimedia apps is a cinch. Shame that it's probably not going to catch on... it's not that much better than Flash.
All that Zed Shaw ragging on the Rails community stuff - mainly cause I think Rails is overhyped (though definitely has its place). I also liked his quote about implementing the semantic web over existing web infrastructure: "Einsteins brain on a crack whores body isn’t going to happen". I disagree but made me laugh...
BioMoby (via Egon's delicious bookmarks)
Hangovers (there is no cure).
Mass Effect - incredibly cheesy sci-fi plot. Naked alien ladies. Voice acting from Seth Green and Lance Henriksen. Had to buy it. Is OK.
Annual appraisal time at work.

Open notebook - what's a disease again?

2007-12-31T16:41:00.001Z

Is there a super-semantic-web-enabled phenotype database out there*? I want to ask a question like 'give a list of monogenic disorders whose locus has been confirmed by at least two labs, broken down by type of causative mutation type' and get an answer.

(* on a tangent: is 23andMe's gene book thing freely available?)

OMIM falls quite a long way short of this... it never set out to be a resource for programmatic access so you can't really blame them. The morbid map is available for download and contains all of the gene -> disorder mappings in their database.

A couple of issues:

OMIM's weird entry categorization system (#*%+...) is very confusing. There are 2229 'phenotypes' (note: not 'Mendelian phenotypes') with a known molecular basis in the database, apparently, but only 386 genes with a phenotype associated with them? Some of those phenotypes are going to be caused by gross insertions / deletions / whatever and not small mutations in single genes, multiple phenotypes might arise from different mutations in the same genes but even so... what's with the disparity?
It contains polygenic disorders (diabetes, schizophrenia) as well as monogenic ones
You can't tell which is which - you could count the number of genes associated with the disorder but a 'monogenic' disorder might be a complex one whose OMIM entry hasn't been updated yet
It's not a disease database - it has other phenotypes in it too. Longevity? Wet or dry ear wax? Novelty seeking personality?

The last point is interesting, really. When is a phenotype a disease? If you have a novelty seeking personality and so are relatively impulsive and prone to climbing mountains, swimming with sharks, cycling without a helmet etc. then are you ill?

Well, no, is the obvious answer. But where do you draw the line? Is autism a disease?

Neh. Beyond our remit. For us a monogenic disease = a clinically recognized disorder with a single, genetic cause.