up vote 7 down vote favorite
1

Could we please make a trial run of modified "hotness formula" for Programmers questions?

  • Modification details are described in this MSO post as follows:

    As far as I can tell, substantial part of Qanswers in current formula is fake.

    (log(Qviews)*4) + ((Qanswers * Qscore)/5) + sum(Ascores)

    About 1/3 of the answers studied here (83 of total 254) have score less than 1/100 of top voted post in respective question.

    Given that questions checked were ones with tens thousands views, insultingly low score indicates that assuming these answers to be popular wouldn't even be in the ballpark. Still, the formula pumps these into Qanswers value, as if it is something everyone would be happy to read (hint: it isn't).

    Consider tuning the formula to make it deviate less from voting results. Ignore answers with non-positive score. Or better yet, ignore answers scored less than some reasonable fraction (eg 1/10) of the top one.

Given that current formula appears to give an unjustified value to crappy answers in highly upvoted questions (Qscore/5, no matter how much is answer downvoted), I would like to perform test run to find out if suggested change would make an impact to issues outlined in Programmers meta posts:

  • What can we do to help users understand our site better?

  • Answers quality in hot questions
     
    Note: results of the trial run to be analyzed using study analogous to one performed in Answers quality in hot questions. Current evaluation shows about 101-118 low quality of 218 answers sampled.
     
    Upon completion of trial run, similar evaluation is to be done in order to estimate whether there was a substantial impact, positive or negative.

Complementary information to this request is provided in comments below, marked with "for the record..."

Note I expect modified formula to be competitive to current one at "moderately hot" questions, following the reasoning outlined here:

For questions that are not too hot (likely 2-3 clicks away from top of the list) it's natural to see things working exactly as intended. Answers and comments quality is mostly maintained by site / tag regulars (business as usual), collider brings moderate amount of interested newcomers from other sites with their views, votes and fresh perspective, everything is nice and cool...

Recent hot question about automated testing gives an example of why I would want to try a modified formula. In 17 hours, question collected 17 answers, merely 5-7 of which provide useful original content.

Overheat of fake hotness impacted question like a Black Saturday fire.

http://cache.boston.com/universal/site_graphics/blogs/bigpicture/ausfire_02_09/a01_17890753.jpg

After the fire settled, what is left for future visitors of the question looks like a wasteland of low quality garbage (note that due to high views, it will score high in web searches).

http://cache.boston.com/universal/site_graphics/blogs/bigpicture/ausfire_02_09/a04_17906927.jpg

In the light of above, community wiki status stamped on the question looks especially despicable. It kind of suggests that after damage has done, community members are invited to go over the crap brought in by braindead formula, carefully analyze it and try to tame the pain by downvotes. As if they don't have anything better to do!

share
3  
IMO downvotes on a hot question should count against a post for more than an upvote, so "controversial" posts don't rise up as far; those are usually the popular but fluffy stuff that's not actually as good of content. – Ben Brocka Jan 25 at 15:00
2  
+1: For proposing a new "hotness" formula. – Jim G. Jan 25 at 16:10
1  
@BenBrocka interesting that current system ignores downvotes in quite a brutal way. It kind of says we don't give a shit about your DVs; even if you put answer to -5, it will still be added to Qanswers and multiplied by Qscore - in highly popular questions this means almost any non-deleted answer will only add to "hotness", no matter how crappy it is – gnat Jan 26 at 9:21
1  
Yeah that's a problem too, one I hadn't realized until reading this post (even after seeing the formula). Often hot questions have lots of bad or meh answers that are bad enough to drag down quality of the content and general discourse (encouraging similarly bad new answers) but not quite bad enough to delete. It's an icky situation to be in – Ben Brocka Jan 26 at 17:13
@JimG. thanks, though I wouldn't say it's totally new. :) Proposed modification builds rather heavily on current formula; this is intentional because I think it works fairly well unless the question gets too close to top. The change (especially the second option) rather attempts to adjust the formula for particularly slippery use cases – gnat Jan 29 at 20:02
2  
@gnat: Understood. I just don't like naysayers who prefer the status quo just because it's the status quo. Thanks for being progressive and trying to improve things. – Jim G. Jan 29 at 20:18
for the record, idea to try SEDE to model potential impact has been discussed at Whiteboard: trial run of modified hotness formula and SEDE "Did you try it out on SEDE? etc..." – gnat May 16 at 21:12
for the record: detailed discussion on how modified formula is supposed to work, along with other change options is here. "And it's really frustrating when a good question with a single authoritative answer fails to rank high because it doesn't have enough crap answers to bump it up..." – gnat May 17 at 6:15
for the record: discussion on how to manually work around the bug in current formula ("guerilla") is here. "I think it's safe to say that we may need to take things into our own hands..." – gnat May 17 at 6:16
for the record: canned message (markdown source) for related authoritative reference bounties at MSO: Looking for an authoritative reference to address related feature request: **[Trial run of modified "hotness formula" for Programmers questions](http://meta.programmers.stackexchange.com/q/5482/31260)** – gnat May 17 at 6:30
for the record: though trial run is a feature-request, the suggested formula modification per se could be rather considered a fix for the bug that has been reported here (implementation doesn't match the spec) - 'tuning the formula to make it closer match observed voting evidence (when it becomes sufficient to learn from) with the initial assumption of "a lot more voting on the answers"...' – gnat May 17 at 11:20

1 Answer

up vote 2 down vote

As a suggestion to your proposed change in the algorithm:

((Qanswers * Qscore)/5) this section should ignore counting any answers that have a total negative value. (ie. -1 or less)

However, sum(Ascores) should still accrue the negative votes.

The net effect is that poor quality answers (aka negatively scored or with many down votes) will drag down the total hotness value.

I don't have a data set to run the proposed change against. Actually, I'm a bit of a n00b in that regard and don't even know where to get said data from...

share
I've been using data provided in this MSO question: Don't let questions stick to the top of the hot questions list forever. There are 9 examples of "sticky questions" there, having total 254 non-deleted answers. By the way none of these 254 answers have negative score, which makes me suspect that testing ignoring only negatives will show very little difference compared to current approach: in all 9 example questions Qanswers will remain the same as now – gnat Jan 25 at 17:51
Not a very good data set then, is it? very good == representative of P.SE hot questions – GlenH7 Jan 25 at 18:06
well, if we define P.SE hot by number of views (on open questions), 62K will currently cut same number of questions (9) as in original data: programmers.stackexchange.com/… - okay, let's see... About 750 non-deleted answers total, 10 with negative score. Or, if we drop out two "historically locked" questions that were apparently cleaned up, there will be 10 with negative score of total about 250 "votable" answers – gnat Jan 25 at 18:37
1  
...by the way my original idea was just like yours, to drop only neg-score answers (I thought community regulars DVs would suffice to make a difference). It's only after I studied data a little and re-checked my own observations that I changed my mind – gnat Jan 25 at 18:41
counting zero-score answers assumes that downvotes are needed to make a difference, right? It's like, you know, downvote if you want it to go lower -versus- upvote if you want it to go higher. I wouldn't mind trying either of these approaches but I'd rather prefer to first test one that feels less conflicting. Downvotes are painful, for both posters and voters. Does that make sense? – gnat Jan 25 at 22:19
1  
(upon further thinking) one benefit of having 0-score answers count is, it's more sensitive to potentially-hot questions at early stage, when there's not yet enough votes to judge better. Frankly, this is the part I really like in current formula. As far as I can tell, modification that suggests cut off at "1/10 of top-voted answer" behaves about like that in the beginning; it only drops after first upvotes received by answers - if this is too early, one tweak to consider is to cut at top/10 - 1, this will keep zero-voted answers in until one of the answers reaches score more than 10. – gnat Jan 29 at 12:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged