A different fundamental approach to the rating system

#1
I'm a teacher, and one frustration I have with the rating system is that any given piece of writing will, and should, receive different levels of assessment based on how it is being judged.

The exact same paper could get an A in an 8th grade class, and would get an F in university. To give one overall grade depends on the perspective that the reviewer has in judging it, on what "stage" they see the author as presenting it. Whether the author means it as a serious entry into the finest works of humanity, or as a fun romp, or as a learning experience makes a big difference.

So, I think it'd be useful to include a different sort of master category for stories - the stage.

Example stages could be:

World Stage: The author feels that this work ought to be judged according to the standard of the best humanity has to offer. It should be judged alongside Douglas Adams, Isaac Asimov, and modern major works, like Harry Potter.

Standard Publication: The author feels that this work is on par with normal published works, and feel it should be judged accordingly.

Just For Fun: The author's writing is whimsical, or at some level, not pursuing perfection, and as such, it should not be taken seriously. Therefore, it should be judged by how fun it is to read, rather than any serious standards.

Learner's Lounge: The author feels that this work ought to be judged with the understanding that the author is learning or otherwise presenting a notably imperfect work, but it nonetheless has value.


If an author marks his story as World Stage, he's inviting heavy criticism and harsh reviews. Getting 3 stars average is a major achievement. If marked as Standard Publication, he's inviting serious criticism, but 

Many stories that get a 5 star rating under the current system ought to get 3 stars or less with such a standard. 

Take Arcane Emperor, for instance - it's a fun story, to be sure, and a lot of people like it. By the standards of "just for fun," it definitely deserves 4-5 stars. By "Standard Publication," it should get 3-4 stars, tops, because of poor technical issues, like pervasive grammar issues. By "World Stage," it should get 2 stars, tops, and that'd be wildly generous. I've spoken to the author about technical writing skills, and the author spoke of being unconcerned about the technical errors, since they weren't bad enough to entirely wreck the story. That implies that the author feels the work should be judged by "Just For Fun" standards.

The basic idea of allowing works to be evaluated, not by a global standard, which invites excessive flattery (shown by the insanely high ratings that stories get, on average), but rather, by different standards according to what the author is going for, can be implemented in a variety of ways.

It could even be part of the review process exclusively, not letting the author decide. Reviewers could pick which "stage" they feel the story is best judged by, and judge it accordingly.

However it is implemented, it would be an enormous commitment and overhaul, to be sure. That said, I feel the idea would address the key, well established flaw with RRL, which is the near uselessness of the reliability of the ratings.

RE: A different fundamental approach to the rating system

#2
It's a nice thought, but the problem with your premise is the fact that you expect a different pool of readers for each of your stages when in reality every author in RRL is sharing a unified reader base.

It doesn't matter if as an author I want people to judge my work just for the fun quality of my story instead of hoping to be the next literary gem destined to win the Nobel prize one day, I will still be judged by the casual readers, the trolls, the grammar nazis, the serious reviewers and all the other tropes out there.

It is my experience that 90% of the people who review the work of others because they have a strong knee-jerk reaction to one story, and not to actually help the author.

Book 1, author pours heart and soul in the story because he is hoping to become a serious author in the future:
Casual reviewer: Meh, no harem.  take my 0.5-star review!

Book 2, an author is just bored and think it will be hilarious to make a parody of other works in RRL.
Casual reviewer: still no harem, 0.5 stars to you as well!

RE: A different fundamental approach to the rating system

#3
It's not that I expect a different pool of readers. Any one reader can judge the same work by different standards. I have students and colleagues do the same thing all the time.

It's that I'm hoping that reviews, etc, can be attached to a certain judgement standard.

Also worth noting that both of the example reviews you mentioned are invalid, according to current RRL rules, and ought to be reported for removal. Those are not proper ratings.

By the current system, by having practically all remotely half decent stories have nearly 5 star ratings, then literally anything other than 5 stars is hurting their stats. My story is 4.7 stars - it takes two five star ratings just to recover from a single 4 star rating, let alone 3s or lower.

Which means that authors feel like they're being punished by 4 star ratings, when, with a remotely sane review system, 4 stars should be honestly a good review.

With something like a "World Class Stage," inviting reviews to be actually serious, then it becomes acceptable to give a 3-star rating to a story you like, and have that be a good thing. Rather than needing to give 5 stars just to avoid hurting their score, you can give a more nuanced appraisal.

Regardless of whether my suggestion is the best solution, the fact remains that the current system is extremely flawed, far beyond simply RRL. RRL is known for this, in particular, but it's a problem at large.

RE: A different fundamental approach to the rating system

#4
I don't really get what problem you're trying to solve here. To get more accurate reviews, I guess? But why would dividing the reviews up into more categories address the underlying issue - the unreliability of the reviews in the first place. The numbers would be just as skewed in the new system.

In the first place I think that our current star system already has the most important - unspoken - qualifier. That is to say, if you see a story with five stars you have to qualify that statement with "it is a story with five stars... on RRL". That is the category we are all laboring within to begin with and probably the only category the userbase is qualified to weigh in on. I mean how many people come to this site with both the time, ability, and inclination to weigh the new fiction they started reading on a "world class stage"? I somehow doubt there are a bunch of Masters and PhDs with degrees in technical editing and literary theory running around.

The reviews are skewed because they come from the specific userbase of this website and are reflective of our niche communities' tastes and interests, which isn't really a bad thing I don't think - just a reality everyone should be aware of.

RE: A different fundamental approach to the rating system

#5
10/27/2017 10:08:08 PMrubycona Wrote: [ -> ]By the current system, by having practically all remotely half decent stories have nearly 5 star ratings, then literally anything other than 5 stars is hurting their stats. My story is 4.7 stars - it takes two five star ratings just to recover from a single 4 star rating, let alone 3s or lower.

Which means that authors feel like they're being punished by 4 star ratings, when, with a remotely sane review system, 4 stars should be honestly a good review.


Dude. One of my stories have 4 0.5 star ratings. For some reason it also has 4 5.0 star ratings (and 1 4.0 rating) to swing it to 2.94. And you know what? The exact number doesn't matter to me. What matters is whether more people like the story or not (5 versus 4). Is 2.94 a terrible rating? Sure. But it doesn't tell the whole story (I just have to use that word...). But rather than the value itself, I look at how many people rated it 0.5 as opposed to those who rate it highly. If there's more people in one side than the other, then I can tell whether it's worth continuing the story regardless of whether it's a 2.75 or 3. Recover? What do you mean, "recover"? It sounds like you want to maintain your story at a 5.0 rating to justify that it's a good story. I mean, I don't think that's the case, but honestly, let's not get obsessed with ratings.

I used to do that in the past, but after a while I realized it's better to measure myself by seeing how many readers like the story as opposed to being obsessed with the actual value.

Not to mention, your idea is hardly going to solve the problem. Almost everyone, including the authors of Arcane Emperor, etc. will more than likely put their work in Just for fun, and the guys whose English is their second or third language will just stay in Learners' Lounge. And most likely it'll just go back to the whole 5.0 ratings in the just for fun category (where the vast majority of the works will be). It changes nothing, except with a couple of World Stage categories that nobody will use. Because let's be honest - if they think their works are master class or publication standard, they wouldn't be posting here in a writing website. They would be submitting them to publication companies or self-publishing on Amazon.

Honestly, authors feel like they get punished by 4.0 ratings? Then what about those of us who have ratings of 0.5, 1.0, 1.5 or 2.0? Those of us with ratings of 3.0 and less? Should we pack up and quit writing for good?

RE: A different fundamental approach to the rating system

#6
I agree about the knee-jerk reaction reviews. Holy crap. I just changed the ending to my novel to better fulfill the needs of my next book. Two people gave me .5 stars cause they hated the change. The one thing I wish for, and I know its never going to happen. I'd like to see who gave me those reviews, so I can at least ask them why.

On the note about review pool. The system we have isn't perfect, far from it, but it does even out. What we do have is a slice of the real people. There are smart, consciences readers who can articulate their reasons for liking a novel. Then we have the morons and the lazy. Just like being Amazon, the number of stars you get averages out. I have two (.5 star) reviews, and two (1 star) reviews now. On the flipside, I have like 37 (five star) reviews. I know my novel isn't perfect.

On Newgrounds they used to have a weighted system. Users who gave good reviews, or were active were treated as being super reviewers. My buddy Chris used to do this. His vote would be worth like fifty or a hundred reviews. I remember this because sometimes he'd find some awful piece of drivel. He'd be like, "be banished the void." and slam negative review button.


I just went to look at their system. (Haven't been on Newgrounds in years! Wow, it's changed.) Copied their stuff here.



Voting Power -
Voting Power is based on a user's Level and Blam / Protection points. Take a look at the respective sections to see how voting power is affected by each.



Level
A level is a number (between 1 through 60, inclusive) along with an icon of a fist, glove, or weapon for a user based on that user's experience. Levels serve no purpose except to increase the voting power in the Flash Portal and to indicate status. For this reason a higher level often commands a certain level of respect.



Experience Points
Much like a traditional role playing game, registered users may acquire experience points and gain levels resulting in higher status. This is done by voting on five games and/or movies per day, which results in a deposit of ten experience points. You can only log ten points per day, there are no exceptions.

Experience points add weight to your votes. Users with an abundance of points will have the strength of multiple voters and can hurt the movies they hate or help the movies they love. Remember, you must be logged in for your voting power to be augmented!

Experience points also unlock secrets on your account page.

When the system was introduced in 1999, users could vote on each unique submission once daily. This meant you could return to Newgrounds on a daily basis and upvote the content you liked, or downvote the content you disliked. This resulted in a habit of "zero bombing" that was frowned upon and resulted in many discussions about the voting system.

In 2015 the voting system was changed to allow only one vote per unique submission.



Blams / Saves
Blam and Protection points are earned by voting on a movie or game that is "under judgment." If a user votes a 0-1 on a movie and it is subsequently deleted (see deletion qualifications below), the user receives one Blam point. Conversely, when a user votes 2-5 on a movie that successfully passes the judgment phase then that user receives a protection point.

Movies are deleted if they fall below the listed score at the listed number of votes:
50 votes: 1.0
70 votes: 1.5
90 votes: 1.75
Blam and Protection points augment a user's voting power (which otherwise would be strictly based upon experience). The increase is a percentage bonus based on the user's level. You also get a fun little badge to show how 1337 you are.

RE: A different fundamental approach to the rating system

#7
You know, while we're on this topic, and because we all know the rating is highly subjective, we can ditch the numerical value system altogether (keep it for the reviews). So instead of rating from a scale of 0.5 to 5.0, readers can dislike it like it in the same manner as Facebook or something. So you have Hate, Dislike, Neutral, Like, Love. Kind of equivalent to the old rating of 1, 2, 3, 4 and 5, but it more clearly emphasizes how subjective the rating is rather than make it look like a grade. Furthermore, it will not be as discouraging to new writers who aren't that good in the language because if you like the story even though the grammar or whatever deserves a low rating, you can still Like the story or something.

Of course, I don't know what's worse - to see half of your readership giving you 0.5 ratings, or tell you directly that they Hate the story, so this idea might not be practical at all.

RE: A different fundamental approach to the rating system

#8
The problem with this system is that the vast majority of reviewers either don't care about or don't understand the finer points of literary craftsmanship. A reader might say "This chapter needs to be fixed author." and then say something about showing instead of telling, whereas their actual issue isn't the issue they named but rather the fact that there wasn't significant plot progression in a chapter.

I encounter these kinds of reviews more often than not, and that means you need to analyze and reinterpret a reader review to make sense of it.

Another thing that I should mention is that this type of review stratification already exists. I, as somebody who has written multiple stories placed quite highly on various independent accounts (so no readers checking out my works strictly because they enjoyed a different story of mine, they're all independent), have noticed that once you hit about 100 reviews or around the 500-800 follower mark your reviews will start becoming much harsher and more critical. This is because your story is stacking up in a different crowd among different stories. It's no longer a matter of you simply being on the map, but rather a matter of what your rank is among other good fictions.

Ultimately however, the ranking system is highly subject to taste. Invariably, higher ranked fictions will be stories that people on RoyalRoadl simply like more. There's little that can be done to change than, since any new implemented system is bound to follow suit.

If you want a different kind of feedback, your most reliable option is to simply publish your story for review in different communities. If you want feedback in a 'learners lounge' category, post your story something like Reddit Writing prompts or the spacebattles creative writing forum. Or here on RoyalRoadl, though most high-visibility stories on this website fall more under your just-for-fun category. Fictionpress and Wattpad tend to cater to this and the semi-serious crowd.

If you want to see if your stories are competitive in a live market then your best bet is to start self-publishing. There's a mountain of quality difference between what I publish on sites like this one as practice work, and what I actually self-publish over at amazon.  And because they're published works they're held to a higher standard.

If you plan on moving to an even higher playing field than that so you can start contending with the big boys, your best option is to find a traditional publisher. If you can't get one, your story probably isn't good enough.

Finally 'World-class' literary works can't really be fully evaluated for years. What makes or breaks them is the readership. Harry Potter did not get very flattering reviews from other writers when it was first introduced to the masses. I personally find 'A Song of Ice and Fire' to be somewhat poorly planned. More exaggerated examples of this phenomenon are the Twilight and 50 Shades of Gray books. Universally mocked among literary communities but still quite successful. It's my opinion that this isn't really a category you can claim to aim for. You just write your best and maybe your fans will decide to put you here.

Ultimately that's my advice to any aspiring writer. Don't write to be the next Hemmingway or Shakespeare. Write the best story you can, focusing your efforts on what you consider to be the most important. Leave the rest up to the readers.

RE: A different fundamental approach to the rating system

#9
RRL 2.5 had an interesting alternative to the current system. A like button and a dislike button. Nothing complicated. Nothing which you could overthink. Just a simple system. If people don't like your content, then you don't have as many likes / more dislikes, if they do, then vice versa.

It would solve the problem with 4 stars being considered an insult and 2 being redundant, but it would lose a lot of data.

RE: A different fundamental approach to the rating system

#10
I'm sad to say I have to agree with a lot of the points here. My original idea does fall flat.

With that being the case, a like/dislike thing, with maybe something like a 1-5 star rating for detailed reviews, might work better.

The reason, btw, I felt like a 4-star rating was a "punishment" is that someone reviewed my story, thought it was excellent, and gave it a 4.5 star review. Despite receiving exceptionally high praise, my rating lowered.

The reason that mattered is that I'm hoping my story can eventually become reasonably found on the "Best Rated" category. High rating doesn't cut it - it needs high followers, or favorites, or total reviews, or something. I was hoping to get there.

Aiming for the top means that every single rating below 5 stars is a serious setback. If it were a ratio between "likes" and "dislikes," it'd be a different story.

But part of the problem is that, knowing the problems with the rating system, if I want to review something, I have to consider this problem. If I think a story "deserves" 4 stars, because it's good, but not amazing, what do I do? Do I give it 4 stars and contribute to its inability to rise to the top? Like "Arcane Emperor" - that story is rife with grammatical errors. If I give it a 4 star review, the rating will be harmed, even though I like the story.

It's a mess :/ I don't know what the best solution is, but the present standing just seems awful.

RE: A different fundamental approach to the rating system

#11
The ranking system doesn't actually work solely on your rating average. It ranks you up against all other fictions and how they have been scored. It also puts into account how many times has a certain review been applied to your fic, weighing frequent ratings higher than one-offs.
So even if your rating average falls if you get a 4 or a 4.5 review, your ranking will more than likely go up.

RE: A different fundamental approach to the rating system

#12
Quote:The reason, btw, I felt like a 4-star rating was a "punishment" is that someone reviewed my story, thought it was excellent, and gave it a 4.5 star review. Despite receiving exceptionally high praise, my rating lowered.


It's unfortunate, but star ratings are averaged on places like Amazon as well. For authors who are looking to eventually publish, places like RRL can help prepare them for what they will have to deal with on sites like Amazon.

That said, people who actually will publish/plan to publish are likely the minority on RRL.

Figment's reaction system or a like/dislike system would probably be better than ratings overall. A breakdown of ratings could remain on reviews, though since reviews are much less common we could see even harsher shifts in ranking placements.

The biggest problem is how to migrate to a new system.

If RRL were to nix ratings, they would have to convert them into something else or let the data slip away entirely. If it was a like/dislike system, 3 star and higher could be a like while anything under would count as a dislike (that's basically how users seem to use ratings right now anyway.) If it's a "reaction" system, then certain emoji could correlate to certain ratings.

That said, with reaction/like/dislike systems it may be prudent to have reactions available on individual chapters. Encouraging more interaction on the stories and possibly giving the site more data to pull from when it comes to ranking fictions. Or, the total reactions/likes shown on the fiction page could be the total across all chapters.

Seeing a rank drop by 130+ places over a .06 change in rating is really disheartening, even for someone like me who is preparing to publish on Amazon soon. Something a little more casual to replace ratings would probably be more encouraging to all the fresh/newbie authors on RRL.

I also question who should be allowed to review, or if reviews should be split into reader vs. literary criticism categories. Generally speaking, reviews are meant to help potential readers determine whether or not they want to give the particular book or series a chance. However, a lot of authors on RRL seem to be under the impression that reviews are meant to give feedback, tips, or criticism. Finding a way to sort and distinguish between review types may be beneficial for all involved.

I'm just spewing ideas/thoughts at this time. Think that means it's time for me to sleep. lol

RE: A different fundamental approach to the rating system

#13
A thumbs up/down system wouldn't actually change how much that .06 change would affect your rank. The underlying system would still use numbers, and basically the same numbers in a different value range at that. So it'd basically just re-map our scaling system from 1 to 10, to 0 and 1. Mid-range ratings would be completely impossible, and the maths behind the rest would be the exact same.

Our experiment for V2.5 showed that there was barely any relative movement in the final ranking after shifting to the up/down vote system compared to before (there really wasn't any fiction that shifted SIGNIFICANTLY).

The problem is not with the rating system; any sufficiently advanced rating system is only as good as the users using it. I believe our current, weighed (yes, it is) Wilson score interval based system does fit that part - you can't get into the toplist with a single 5 star rating, and it weighs both rating and popularity into your ranking.

RE: A different fundamental approach to the rating system

#14
I find the current rating/ranking system fairly robust.

I think it would be cool if Royal Road had a leveling system like Newgrounds did where reviewers could level up through leaving reviews. I've noticed that it makes people more likely to write detailed reviews, which is useful feedback as an author.

At the same time, I understand how implementing that manner of system takes a great deal of time and effort.

RE: A different fundamental approach to the rating system

#15
I was reading one of Luciferia's reviews, for her stories, and one of her longer reviewers said "Short and to-the-point reviews ARE FOR THE WEAK!"

I agreeish.

Longer, more in-depth reviews, should be given more weight, as those probably rate the story more accurately than those that write only to meet the word count.

Logic would justify weighing a review from a reader who has rated 15 other novels than someone who reviews for the first time. I doubt those type of reviews would sink into handing out 5s, just by their nature. The readers who take the time to review novels on regular basis probably care more to put more energy than those who don't. Of course, this isn't democratic, but though this might reek of elitism, democracy by the uninformed hurts everyone.

*Cough Cough* Trump *Cough Cough*

So in the end, instead of just reforming the entire system, improving it would be a lot better.  If the main issue of the rating system is that it's inflated by readers who give 5 stars to a good story, and still 5 stars for a great story, then the rating loses its value. All of the sudden, having a 4.5 just doesn't feel good. That's the main issue.

As an author, this current system just doesn't feel good. As a reader, I can't tell the difference between a story rated as 4 stars and a story that is rated as 4.8. Creating a system won't just serve the author and reader, but the website too. Having a reason for people to sign up (yes, I'm talking to you Guest ) and start getting engaged further in contributing to the website will benefit everyone. 

Besides, most people on this website love LitRPG elements and titles. I doubt anyone will complain about getting levels.

Giving more weight to experienced reviewers, even though Goodreads and Amazon don't do that, might add more value to the ratings. Maybe thumping up a review as useful might give more points, I don't know, but I do know we can do better. 

Just because others don't improve their systems doesn't mean we should settle for less, too.

Though personally, I think a poll, one for readers and the other for readers, and maybe reviewers, would clarify whether change is even warranted.

RE: A different fundamental approach to the rating system

#17
11/17/2017 10:14:38 AMNecamijat Wrote: [ -> ]The problem here is that you're trying to solve the community with the system, rather than improving the community itself.

But this is a multi-k anonymous community. How do you improve something like that?
You can't say a rating is "right" or "wrong" as it's subjective and therefore each 0.5 drive-by-rating can be justified with "I didn't like it". And besides that you are open to everyone, people don't lose anything even if they are punished and the anonymous character of the community makes finding and discussing stuff near impossible.
I mean, It's hard to improve small groups with well-known names and clear rules, but how do you improve something like RRL Oo

I think - no matter what one thinks about the status of the community itself - it's always better to prepare a system that limits the bad apples instead of hoping for only good ones...

RE: A different fundamental approach to the rating system

#19
11/17/2017 2:16:39 PMNecamijat Wrote: [ -> ]Which is why we have a weighted system that has the outlying scores count less than ones that are repeated more often.


That's not going to solve inflation.

11/17/2017 11:35:48 AMLostLibrarian Wrote: [ -> ][quote='Necamijat' pid='830027' dateline='1510913678']

I think - no matter what one thinks about the status of the community itself - it's always better to prepare a system that limits the bad apples instead of hoping for only good ones...


I agree with this. You can't punish people, but you can give them incentives to do "good"

And again, giving people a reason to sign up and get involved will benefit everyone.

RE: A different fundamental approach to the rating system

#20
11/17/2017 2:57:22 PM13900IPForThis Wrote: [ -> ]
11/17/2017 2:16:39 PMNecamijat Wrote: [ -> ]Which is why we have a weighted system that has the outlying scores count less than ones that are repeated more often.

That's not going to solve inflation.


Not the fault of the system, but how it is used.
kanadaj Wrote: The problem is not with the rating system; any sufficiently advanced rating system is only as good as the users using it.

New Reply