In September 2012 registration requirement to ask questions has been introduced:

In the past 60 days, 76% of all questions from unregistered users have been either down-voted below 0, closed, deleted, or some combination of the three.

That's a total of 182 questions, compared to 877 similarly bad questions from registered users. It's a relatively small number, but as a percentage of the total it's high enough that implementing #1 holds little risk - at worst, you'll lose maybe a question a day from someone too lazy to register.

So, I've turn on the registration requirement in order to ask questions here. Let's check back in a month or so and see if it made any difference...

Now that more than 3 months passed, is it time to somehow see if it made a difference?

If I recall correctly, back then one of major concerns against this was that it may somehow inhibit asking good questions. With this in mind I think it would make sense to find out about the differences in the amount / rate of "good" questions (defined as say, open with non-negative score).


Given that registration presumably simplifies handling of question bans I think it would be particularly interesting to estimate if there was a difference in amount (rate) of such bans.

It would be also interesting to study the difference in "ban failure rate", which could be estimated as percentage of users caught at attempts to overcome it: those who created fake accounts that were later discovered and merged into original banned account.

  • It looks quite simple how to get upper bound of detected attempts to abuse question ban. First, let's count how many accounts were added in last 4 months, as of Jan 12 these are roughly between 62044 and 78338, 78338-62044=16294. Now if we find how many of these accounts are gone, as indicated by respective user profile being 404 Not Found, this will give us how many accounts were either self-deleted, or gone through legitimate merges, or... and this is exactly what I'd want to estimate... or were merged as a result of detected abuse of question ban. It is of course impossible to manually scan 16K+ profile pages, wonder if this could be done somehow automagically?
share
    
Some of these can probably be answered with some SEDE queries, now that there are regular updates (we should be getting another one sometime on the 22nd or 23rd as well). Addressing socks and account merging is something that we can't do, since that would be revealing mod-only information. Something to look at would be number of questions from newly registered users (perhaps asked within 7 days of their registration) and how many of those were closed. –  Thomas Owens Dec 21 '12 at 13:42
    
Thanks for asking this @gnat, I completely forgot about it. –  Yannis Rizos Dec 21 '12 at 21:30
    
@YannisRizos so did I; we should be thankful to the guy who recently bumped similar question at MSO which made me recall our own plans :) –  gnat Dec 22 '12 at 8:37
2  
@ThomasOwens For Data.SE to be accurate we'd need SE to start including deleted questions –  Rachel Dec 24 '12 at 3:57

2 Answers 2

up vote 6 down vote accepted

After digging into this a bit further, I can't find any conclusive evidence that this has really helped much.

In the 60 days prior to registration being enacted, Programmers got 877 bad* questions from registered users, and 182 from unregistered users, making up 48.05% of those asked by registered and 76.15% of those asked by unregistered users respectively.

In the past 60 days, Programmers got 1221 bad questions from registered users, making up 55.53% of the total asked. There are some holidays in there though, so that doesn't really tell the whole story. Here's a picture:

questions by source and quality over time

What I didn't see (but hoped for) was a significant drop in the number of questions asked. So far as I can tell, the folks asking crappy questions with unregistered accounts just registered their accounts and kept on a-goin' - the number dropped slightly right after the change went in, but then went right back up again.

It probably improved the effectiveness of the quality ban slightly, but you're not gonna see this reflected in account deletions, and I'm not gonna talk to much about that anyway. But, here's a picture for gnat:

User creation and deletion

The best reason for doing this is probably just the slight bit of extra resistance it provides, and the somewhat fewer headaches involved in having registered accounts (both for the asker, and the folks interacting with him). I wouldn't turn it off again, but I wouldn't encourage sites with less traffic to enable it either.

As an aside, we're working on changes to the registration system that should help to encourage folks to register their accounts without requiring it - this should ease the transition elsewhere.

*closed, deleted, or down-voted below 0.

share
    
thanks! no need to hurry; I think this is the case when we better be thorough than fast. Relax, enjoy your vacation and come back to this when you have enough time to do a good analysis :) –  gnat Dec 22 '12 at 8:42
    
would you mind diving into this a little bit? As far as I can tell current status can be summarized roughly like "so far so good, let's re-check things in 3-6 months from now to find out about longer-term change impact" - does that make sense? –  gnat Jan 10 '13 at 8:08
2  
@gnat: I've added some more detail. I'd call this at best a small positive change. That said, it might be worth checking back in a couple of months, since I suspect the holidays kinda skewed stuff. –  Shog9 Jan 15 '13 at 1:12

If I've done my math right based on Shog's answer, no it hasn't made a difference.

60 days prior to registration:

Registered Users:

877 bad questions
948 good questions
1825 total questions
48.5% bad questions

Unregistered Users:

182 bad questions
57 good questions
239 total questions
76.15% bad questions

Total:

1059 bad questions
1005 good questions
2064 total questions
51.3% bad questions

60 days prior to now:

1314 bad questions
1104 good questions
2419 total questions
54.32% bad questions

If anything, it looks like our percentage of closed, deleted, or downvoted below 0 question rate has gone slightly up since we started requiring registration.

The reason for it going up may be unrelated to requiring registration though, as 2 sets of data isn't really enough to draw any solid conclusions.

share
5  
Closed/deleted question count has probably gone up due to the new review queue... more moderation by non-mods. –  Dynamic Dec 23 '12 at 14:35
3  
I know that I've seen far fewer flags to close questions since the review queues went up, so that's definitely a factor. Questions that get at least one close vote are now visible to people who can vote to close. I can't speak to the quality of those closures, though, but there haven't been a significant increase in complaints of closures or flags for moderator review on closed questions that I've seen. –  Thomas Owens Dec 24 '12 at 7:13
1  
thanks Rachel! it looks like short-term, things weren't severely broken by the change (though to be on the safer side, I'd rather compare against year-ago data, to account for possible seasonal changes in activity). Your point about insufficient data also looks well taken; I believe we will need at least yet another checkpoint (maybe in 3-6 months) to find out about longer-term change impact –  gnat Dec 24 '12 at 9:39
1  
@Dynamic That's a good point to make. We might need to ask Shog to see if he can get us stats from a 60-day period from before the Review queue was implemented –  Rachel Dec 24 '12 at 17:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .