A Computer Scientist in a Business School

Tuesday, January 26, 2010

Universities and Intellectual Property: A Minefield?

One of the things that I never understood at NYU is what are the rights that the university has on the work produced by the faculty members and students.

Following the intellectual properly law, there are four basic types of intellectual properly:

Copyright
Patents
Trademarks
Trade Secrets

If someone works for a corporation, things are pretty clear. Any paper, before being published needs to get approval. Any developed algorithms and code written is the intellectual property of the company and the company owns the copyright for the code, and can treat the algorithms as a trade secret. The company may also patent useful inventions and register some valuable trademarks. But, in all these cases, everything that is being produced within the corporation is work made for hire and owned by the corporation. The employee has typically no ownership of the produced work and it is commonly prohibited for the employee to work for another company or provide any sort of consulting services..

For the work of faculty, I always felt that everything falls into a grey area. Most of the work is made public as soon as possible. Code is often released as open source, following some pretty liberal licensing scheme, or even released to the public domain. Papers are written and publicized without much, if any, vetting and the algorithms and methods described there are typically in the public domain. The only case where a university has some control over intellectual property is when a patent is filed and granted.

Now, the great confusion arises when the faculty wants to work with a corporation and the university allows faculty members to engage into consulting agreements. Who owns and controls the expertise and discoveries of the faculty member?

Let's say that myself, Panos, invented an algorithm in area X, wrote a paper, and published the code in an open source format. Corporation A, comes to me and asks me to consult them on area X. What is the control that my employer, NYU, has on my work? Yes, Corporation A wants to hire me because of the IP that I produced while at NYU. This IP though is publicly available, so I do not really transfer anything protected under copyright law.

I have asked this question to our own tech transfer office. Unfortunately, I did not get back a clear answer. They told me that I cannot transfer code and that any patent is owned by NYU. Correct, these are indeed intellectual property assets. (Although for the case of open source code, this is again confusing.) But what about the expertise that a faculty member develops? In corporations this is often protected using some no-compete clauses in the employment contract, effectively preventing employees from directly transferring know-how. In universities, there is no such provision.

I find this merging of academic and corporate worlds to be particularly confusing and I find this to be a potential minefield. Who owns what? Any ideas? Any experiences? How other universities treat the concept of tech transfer?

Monday, January 25, 2010

Did you find this helpful?

Last week, the New York Times Sunday magazine had an article titled The Reviewing Stand, starting with the following:

Here’s a challenge for students of expository writing: review a popular product on Amazon and aim to get your review chosen by readers as “most helpful.” It’s dead hard. The product review, as a literary form, is in its heyday. Polemical, evocative, witty, narrative, exhortative, furious, ironic, off the cuff....

What I found amusing was the fact that, after reading this article, I got a notification that the journal version of the paper Estimating the Helpfulness and Economic Impact of Product Reviews: Mining Text and Reviewer Characteristics, co-authored with my frequent co-author, Anindya Ghose, has been accepted for publication at the IEEE Transactions on Knowledge and Data Engineering (TKDE) journal.

As the title suggests, one of the problems that we attack in the paper is how to predict the usefulness of a product review. For example, if you go on Amazon, you will see, on top of many reviews, how many people considered a particular product review helpful:

So, the question is: Can we predict how helpful a particular review will be?

Our first attempts to address this problem appeared in the WITS 2006 and the ICEC 2007 papers. Following the scientific zeitgeist, a large number of other papers appeared these years, all tackling the question of predicting helpfulness of reviews. (See the actual paper for references.)

What I found rather surprising was the relative easiness of the task. A few relatively straightforward features can be used to predict with good accuracy whether a review will be deemed helpful or not.

Check the readability of the article, as measured by one of the many readability metrics, check the number of spelling errors, and measure basic statistics of the text, such as review length. Using just the readability and the fraction of spelling errors in the article we can estimate with 70%-80% accuracy whether a review will be deemed helpful or not.
Check the history of the reviewer. If the reviewer has been writing helpful reviews in the past, it is highly likely that reviews in the future will also be helpful. Also, if a reviewer has disclosed personal details (name, location, etc) the reviews are more likely to be helpful. Again, using just reviewer history and disclosure details, we get 70%-80% accuracy, as measured with the AUC metric.
Check the "subjectivity" of the review. We call a review objective if it contains mainly information that can be found in the product description and specs. A subjective review contains information that depends on the personal experiences of the reviewer. Helpful reviews tend to contain a mix of both.

Interestingly enough, all three feature sets seem to have equivalent predictive power. Even using them all together does not seem to increase substantially the predictive performance.

While preparing the final version of the paper, I also checked other papers that were attacking the same problem. While many papers were trying to predict helpfulness using textual features, I noticed that a few papers were using a set of alternative and interesting features:

Coverage of product features. Many products can be considered an aggregation of multiple product features. For example, a digital camera has resolution, size, battery life, sensor size, etd. How many product features are being discussed in the review? This feature tends to have predictive power, according to (Liu et al, EMNLP 2007).
Dynamics of reviews. Reviews that are posted early on get a higher fraction of helpful votes. In contrast, later reviews need to be more informative and comprehensive to attract the same fraction of helpful votes (Liu et al, EMNLP 2007).
Controversy. The helpfulness of a review depends not only on its own content but also on how controversial is the product under consideration (Danescu-Niculescu-Mizil, WWW 2009).
Social network of reviewers. If reviewer A trusts the reviews of reviewer B, then the reviews of B are likely to be more helpful than the reviews of A. ("Exploiting Social Context for Review Quality Prediction"; by Lu, Tsaparas, Ntoulas, and Polanyi; WWW 2010)

Although I have not seen a paper combining all the above features in order to predict the helpfulness of a review (or for ranking reviews by helpfulness), I guess that these set of features will bring predictive accuracy pretty close to its limit for this task.

What is next? I guess personalized recommendations are going to appear sooner or later, matching users with reviews that are more likely to benefit them. (Update: See Eugene's comment below for related papers.) For example, a beginner in photography will be interested in a different type of review when buying an SLR, compared to a seasoned professional. We already know that reviews from similar users can be used for recommending products (see Netflix) so it is not unlikely that different types of reviews will be deemed helpful by different types of users.

So, did you find this blog post useful?

Saturday, December 5, 2009

Prisoner's Dilemma and Mechanical Turk

I have been reading lately, about the differences between mathematical models of behavior and real human behavior. So, I decided to try on Mechanical Turk the classical game theory model of Prisoner's Dilemma. (See also Brendan's nice explanations and diagrams if you have never been exposed to game theory before.)

From Wikipedia:

In its classical form, the prisoner's dilemma ("PD") is presented as follows:

Two suspects are arrested by the police. The police have insufficient evidence for a conviction, and, having separated both prisoners, visit each of them to offer the same deal. If one testifies (defects from the other) for the prosecution against the other and the other remains silent (cooperates with the other), the betrayer goes free and the silent accomplice receives the full 10-year sentence. If both remain silent, both prisoners are sentenced to only six months in jail for a minor charge. If each betrays the other, each receives a five-year sentence. Each prisoner must choose to betray the other or to remain silent. Each one is assured that the other would not know about the betrayal before the end of the investigation. How should the prisoners act?

If we assume that each player cares only about minimizing his or her own time in jail, then the prisoner's dilemma forms a non-zero-sum game in which two players may each cooperate with or defect from (betray) the other player. In this game, as in all game theory, the only concern of each individual player (prisoner) is maximizing his or her own payoff, without any concern for the other player's payoff. The unique equilibrium for this game is a Pareto-suboptimal solution, that is, rational choice leads the two players to both play defect, even though each player's individual reward would be greater if they both played cooperatively.

My first attempt was to post to Mechanical Turk this dilemma in a setting of the following game:

You are playing a game together with a stranger. Each of you have two choices to play: "trust" or "cheat".

If both of you play "trust", you win $30,000 each.

If both of you play "cheat", you get $10,000 each.

If one player plays "trust" and the other plays "cheat", then the player that played "cheat" gets $50,000 and the player that played "trust" gets $0.

You cannot communicate during the game, and CANNOT see the final action of the other player. Both actions will be revealed simultaneously.

What would you play? "Cheat" or "Trust"?

Basic game theory predicts that the participants will choose "cheat" resulting in a suboptimal equilibrium. However, participants on Mechanical Turk did not behave like that. Instead, 48 out of the 100 participants decided to play "trust", which is above the 33% observed in the lab experiments of (Shafir and Tversky, 1992).

Next, I wanted to make the experiment more realistic. Would anything change if instead of playing an imaginary game, I promised actual monetary benefits to the participants? So, I modified the game, and asked the participants to play against each other. Here is the revised task description.

You are playing a game against another Turker. Your action here will be matched with an action of another Mechanical Turk worker.

Each of you have two choices to play: "trust" or "cheat".

If both of you play "trust", you both get a bonus of $0.30.

If both of you play "cheat", you both get a bonus of $0.10.

If one Turker plays "trust" and the other plays "cheat", then the Turker that played "cheat" gets a bonus of $0.50 and the Turker that played "trust" gets nothing.

What is your action? "Cheat" or "Trust"?

I asked 120 participants to play the game, paying just 1 cent for the participation. Interestingly enough, I had a perfect split in the results. 60 Turkers decided to cheat, and 60 Turkers decided to cheat. The final result was 20 pairs of trust-trust, 20 pairs of cheat-cheat, and 20 pairs of cheat-trust.

In other words, the theory prediction that people will be locked in a non-optimal equilibrium was not correct, neither in the "imaginary" game, nor in the case where the workers had to gain some actually monetary benefit.

Finally, I decided to change the payoff matrix, and replicate the structure of the TV game show "Friend or Foe". There, participants get $50K each if they cooperate, $0 if they do not, and if one chooses trust and the other cheat, the "cheat" gets $100K and the "trust" gets $0.

You are playing a game together with a stranger.

Each of you have two choices to play: "trust" or "cheat".

If both of you play "trust", you both win $50,000.

If both of you play "cheat", you both get $0.

If one player plays "trust" and the other plays "cheat", then the player that played "cheat" gets $100,000 and the player that played "trust" gets $0.

You cannot communicate during the game, and CANNOT see the final action of the other player. Both actions will be revealed simultaneously.

What would you play? "Cheat" or "Trust"?

Interestingly enough, in this setting ALL 100 players ended up playing "trust", which was quite different from the previous game and from the behavior of the players in the TV show, where, in almost 25% of the played games, both players chose "cheat" ending up with $0, and in 25% of the games the players collaborated and played "trust" getting $50K each.

So, in my final attempt, I asked Turkers to play this "Friend of Foe" game, having monetary incentives. Here is the task that I posted on Mechanical Turk.

You are playing a game against another Turker. Your action here will be matched with an action of another Mechanical Turk worker.

Each of you have two choices to play: "trust" or "cheat".

If both of you play "trust", you both get a bonus of $0.50.

If both of you play "cheat", you both get $0.

If one Turker plays "trust" and the other plays "cheat", then the Turker that played "cheat" gets a bonus of $1.00 and the Turker that played "trust" gets nothing.

What is your action? "Cheat" or "Trust"?

In this game, 33% of the users decided to cheat, resulting in 6/50 games where both players got nothing, 23/50 games where both players got a 50 cent bonus, and 21/50 games where one player got $1 and the other player got nothing.

I found the difference in behavior between the imaginary game and the actual one to be pretty interesting. Also, the deviation from the predictions of the game-theoretic model is striking.

Although I am not the first to actually observe that, this deviation got me wondering: Why do we use elaborate game theory models for modeling user behavior, when not even the simplest such models do not correspond to reality? How can someone take seriously the concept of an equilibrium when a game, introduced in the intro chapter of every game theory textbook, simply does not correspond to reality? Do we really understand the limitations of our tools, or mathematical and analytic elegance end up being more important than reality?

Monday, November 30, 2009

Anchoring and Mechanical Turk

Over the last few days, I have been reading the book Predictably Irrational by Dan Ariely, which (predictably) describes many biases that we exhibit when making decisions. These biases are not just effects of random chance but are rather expected and predictable. Such biases and the "irrationality" of human agents is one of the focuses of behavioral economics; these biases have been also extensively studied in the field of cognitive psychology, which examines the ways that human agents process information.

One of the classic biases is the bias of "anchoring". Dan Ariely in his book shows how he got students to bid higher or lower for a particular bottle of wine: He asked students to write down the last digit of their social security number before placing the bid. As the anchoring theory postulated, students that wrote down a lower number, ended up bidding lower than students with a higher last digit in their SSN.

Why? Definitely not because the last digit revealed anything about their character. It was because the students got "anchored" to the value of the last digit they wrote down. I am certain that the experiment could be repeated by using the middle two digits as anchor, and the results would be similar.

Interestingly enough, at the same time that I was reading the book, I got contacted by Gabriele Paolacci, a PhD student in Italy. In his blog, Experimental Turk, Gabriele has been replicating some of these "classic" cognitive psychology experiments that illustrate these biases. As you might have guessed already, Gabriele has been using Mechanical Turk for these experiments. Gabriele tested the theory of anchoring using Amazon Mechanical Turk, replicating a study from a classic paper. In his own words:

We submitted the “african countries problem” from Tversky and Kahneman (1974) to 152 workers (61.2% women, mean age = 35.4). Participants were paid $0.05 for a HIT that comprised other unrelated brief tasks. Approximately half of the participants was asked the following question:

Do you think there are more or less than 65 African countries in the United Nations?

The other half was asked the following question:

Do you think there are more or less than 12 African countries in the United Nations?

Both groups were then asked to estimate the number of African countries in the United Nations.

As expected, participants exposed to the large anchor (65) provided higher estimates than participants exposed to the small anchor (12), F(1,150) = 55.99, p<.001. Therefore, we were able to replicate a classic anchoring effect - our participants’ judgments are biased toward the implicitly suggested reference points. It should be noted that means in our data (42.6 and 18.5 respectively) are very similar to those recently published by Stanovich and West (2008; 42.6 and 14.9 respectively).

References

Stanovich, K. E., West. R. F. (2008). On the relative independence of thinking biases and cognitive ability. Journal of Personality and Social Psychology, 94, 672-695.

Tversky, A., Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124-1131.

Gabriele has more experiments posted in his blog, and I am looking forward for more experiments.

So, here is a question: Definitely we should take similar biases into consideration when collecting data from humans, and when conducting user studies. In a more general setting, can we use such biases more productively, in order to get users to complete tasks that are useful?

Ignore.. (Test)