Could we please make a trial run of modified "hotness formula" for Programmers questions?

  • Modification details are described in this MSO post as follows:

    As far as I can tell, substantial part of Qanswers in current formula is fake.

    (log(Qviews)*4) + ((Qanswers * Qscore)/5) + sum(Ascores)

    About 1/3 of the answers studied here (83 of total 254) have score less than 1/100 of top voted post in respective question.

    Given that questions checked were ones with tens thousands views, insultingly low score indicates that assuming these answers to be popular wouldn't even be in the ballpark. Still, the formula pumps these into Qanswers value, as if it is something everyone would be happy to read (hint: it isn't).

    Consider tuning the formula to make it deviate less from voting results. Ignore answers with non-positive score. Or better yet, ignore answers scored less than some reasonable fraction (eg 1/10) of the top one.

Given that current formula appears to give an unjustified value to crappy answers in highly upvoted questions (Qscore/5, no matter how much is answer downvoted), I would like to perform test run to find out if suggested change would make an impact to issues outlined in Programmers meta posts:

  • What can we do to help users understand our site better?

  • Answers quality in hot questions
     
    Note: results of the trial run to be analyzed using study analogous to one performed in Answers quality in hot questions. Current evaluation shows about 101-118 low quality of 218 answers sampled.
     
    Upon completion of trial run, similar evaluation is to be done in order to estimate whether there was a substantial impact, positive or negative.

Complementary information to this request is provided in comments below, marked with "for the record..."


Note I expect modified formula to be competitive to current one at "moderately hot" questions, following the reasoning outlined here:

For questions that are not too hot (likely 2-3 clicks away from top of the list) it's natural to see things working exactly as intended. Answers and comments quality is mostly maintained by site / tag regulars (business as usual), collider brings moderate amount of interested newcomers from other sites with their views, votes and fresh perspective, everything is nice and cool...


Recent hot question about automated testing gives an example of why I would want to try a modified formula. In 17 hours, question collected 17 answers, merely 5-7 of which provide useful original content.

Overheat of fake hotness impacted question like a Black Saturday fire.

http://cache.boston.com/universal/site_graphics/blogs/bigpicture/ausfire_02_09/a01_17890753.jpg

After the fire settled, what is left for future visitors of the question looks like a wasteland of low quality garbage (note that due to high views, it will score high in web searches).

http://cache.boston.com/universal/site_graphics/blogs/bigpicture/ausfire_02_09/a04_17906927.jpg

In the light of above, community wiki status stamped on the question looks especially despicable. It kind of suggests that after damage has done, community members are invited to go over the crap brought in by braindead formula, carefully analyze it and try to tame the pain by downvotes. As if they don't have anything better to do!

Update

Related feature request has been posted at MSO:

In hotness formula, discard answers when voting evidence indicates that these are not good data points

Please stop counting proven low score answers in hotness formula. Please roll the dice fairly, let user voting and time decay contribute to hotness score as intended. Please promote to collider audience less brain-damaging content to learn from.

share
4  
IMO downvotes on a hot question should count against a post for more than an upvote, so "controversial" posts don't rise up as far; those are usually the popular but fluffy stuff that's not actually as good of content. –  Ben Brocka Jan 25 '13 at 15:00
2  
+1: For proposing a new "hotness" formula. –  Jim G. Jan 25 '13 at 16:10
1  
@BenBrocka interesting that current system ignores downvotes in quite a brutal way. It kind of says we don't give a shit about your DVs; even if you put answer to -5, it will still be added to Qanswers and multiplied by Qscore - in highly popular questions this means almost any non-deleted answer will only add to "hotness", no matter how crappy it is –  gnat Jan 26 '13 at 9:21
1  
Yeah that's a problem too, one I hadn't realized until reading this post (even after seeing the formula). Often hot questions have lots of bad or meh answers that are bad enough to drag down quality of the content and general discourse (encouraging similarly bad new answers) but not quite bad enough to delete. It's an icky situation to be in –  Ben Brocka Jan 26 '13 at 17:13
    
@JimG. thanks, though I wouldn't say it's totally new. :) Proposed modification builds rather heavily on current formula; this is intentional because I think it works fairly well unless the question gets too close to top. The change (especially the second option) rather attempts to adjust the formula for particularly slippery use cases –  gnat Jan 29 '13 at 20:02
2  
@gnat: Understood. I just don't like naysayers who prefer the status quo just because it's the status quo. Thanks for being progressive and trying to improve things. –  Jim G. Jan 29 '13 at 20:18
    
for the record, idea to try SEDE to model potential impact has been discussed at Whiteboard: trial run of modified hotness formula and SEDE "Did you try it out on SEDE? etc..." –  gnat May 16 '13 at 21:12
    
for the record: detailed discussion on how modified formula is supposed to work, along with other change options is here. "And it's really frustrating when a good question with a single authoritative answer fails to rank high because it doesn't have enough crap answers to bump it up..." –  gnat May 17 '13 at 6:15
    
for the record: discussion on how to manually work around the bug in current formula ("guerilla") is here. "I think it's safe to say that we may need to take things into our own hands..." –  gnat May 17 '13 at 6:16
    
for the record: canned message (markdown source) for related authoritative reference bounties at MSO: Looking for an authoritative reference to address related feature request: **[Trial run of modified "hotness formula" for Programmers questions](http://meta.programmers.stackexchange.com/q/5482/31260)** –  gnat May 17 '13 at 6:30
    
    
for the record: though trial run is a feature-request, the suggested formula modification per se could be rather considered a fix for the bug that has been reported here (implementation doesn't match the spec) - 'tuning the formula to make it closer match observed voting evidence (when it becomes sufficient to learn from) with the initial assumption of "a lot more voting on the answers"...' –  gnat May 17 '13 at 11:20
    
for the record: preliminary SE team review of feature request at MSO by Shog9 –  gnat May 23 '13 at 18:07
    
for the record: hotness formula damage case study - "+25, +14, -1, -1, -1" "I think this question is going to be very hot..." –  gnat May 29 '13 at 7:30
    
for the record: related proposal at MSO to tame the damage of drive-by votes in hot questions "the most straightforward way to prevent over-voting from newcomers who just arrived at the hot question would be to simply delay granting them upvote privilege for a day or two..." –  gnat Jun 26 '13 at 21:21
show 6 more comments

2 Answers

As a suggestion to your proposed change in the algorithm:

((Qanswers * Qscore)/5) this section should ignore counting any answers that have a total negative value. (ie. -1 or less)

However, sum(Ascores) should still accrue the negative votes.

The net effect is that poor quality answers (aka negatively scored or with many down votes) will drag down the total hotness value.

I don't have a data set to run the proposed change against. Actually, I'm a bit of a n00b in that regard and don't even know where to get said data from...

share
    
I've been using data provided in this MSO question: Don't let questions stick to the top of the hot questions list forever. There are 9 examples of "sticky questions" there, having total 254 non-deleted answers. By the way none of these 254 answers have negative score, which makes me suspect that testing ignoring only negatives will show very little difference compared to current approach: in all 9 example questions Qanswers will remain the same as now –  gnat Jan 25 '13 at 17:51
    
Not a very good data set then, is it? very good == representative of P.SE hot questions –  GlenH7 Jan 25 '13 at 18:06
    
well, if we define P.SE hot by number of views (on open questions), 62K will currently cut same number of questions (9) as in original data: programmers.stackexchange.com/… - okay, let's see... About 750 non-deleted answers total, 10 with negative score. Or, if we drop out two "historically locked" questions that were apparently cleaned up, there will be 10 with negative score of total about 250 "votable" answers –  gnat Jan 25 '13 at 18:37
1  
...by the way my original idea was just like yours, to drop only neg-score answers (I thought community regulars DVs would suffice to make a difference). It's only after I studied data a little and re-checked my own observations that I changed my mind –  gnat Jan 25 '13 at 18:41
    
counting zero-score answers assumes that downvotes are needed to make a difference, right? It's like, you know, downvote if you want it to go lower -versus- upvote if you want it to go higher. I wouldn't mind trying either of these approaches but I'd rather prefer to first test one that feels less conflicting. Downvotes are painful, for both posters and voters. Does that make sense? –  gnat Jan 25 '13 at 22:19
2  
(upon further thinking) one benefit of having 0-score answers count is, it's more sensitive to potentially-hot questions at early stage, when there's not yet enough votes to judge better. Frankly, this is the part I really like in current formula. As far as I can tell, modification that suggests cut off at "1/10 of top-voted answer" behaves about like that in the beginning; it only drops after first upvotes received by answers - if this is too early, one tweak to consider is to cut at top/10 - 1, this will keep zero-voted answers in until one of the answers reaches score more than 10. –  gnat Jan 29 '13 at 12:08
add comment
up vote 2 down vote accepted

This request has been warp driven to the following requests at MSO:

Side note - this very answer has been used as a "buffer" to draft the latter request: details in revisions history.

share
1  
Hey gnat, I am not going to directly edit because it would be quite significant, but here are my comments. (A) I think this should focus more on 'What questions we want in the hot list' followed by 'How we can accomplish that', (B) Worrying about technical limitations is a side-note at best and shouldn't even be in the same league as the goals of (A), (C) I recommend adding weight to close votes to allow questions that may not be on topic to have a tougher time getting on the list with a ton of interest prior to getting closed and giving a poor impression of on-topic to the SE network –  jmac Jan 14 at 0:49
    
@jmac thanks, I will try to edit to your points (A) and (B). Point (C) looks worth adding, too. By the way, please feel free to directly edit - as opposed to some, I am huge fan of your aggressive edits :) –  gnat Jan 14 at 3:28
    
@jmac I think I fully integrated points (A) and (C), would you mind taking a look (edits are welcome, aggressive or not - I trust you there:). As for (B), it's a bit trickier: I moved it visually into the "basement" by separator, but it sort of stays in top league, because it just addresses the main reason for decline of the prior request. It even has been stressed in rev 4 per discussion at Whiteboard: chat.stackexchange.com/transcript/message/13166653#13166653 –  gnat Jan 14 at 7:51
2  
looking great! Thanks for all the hard work, the entire community of small SE sites is grateful for your vigilance on this topic (it really is a bane that should be a boon in many cases). –  jmac Jan 14 at 23:53
    
@jmac Glen said that version based on your edit was much better than original one. :) To me, your points (A), (B) and (C) were even more valuable than that (Glen couldn't tell because these points were integrated in both versions I asked him to compare) –  gnat Jan 15 at 9:09
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .