Reviewing Reviews

sTeve:

you hit this one straight on. I 100% agree with you... :)

Personally (as much as everyone disagrees with this), I think the "try before you buy" mentality is the best review. If you get a demo (or sadly, warez the game if it's not available in a demo) and like it, then go out and buy it. If you don't like it, throw the disc away and delete from your HD.

sTevem I think it would be cool if you rated games on a scale of 1-100, with a minimum of three decimal places. You know, just to be different. :p

-J

A German PC Game magazine has a curve for every game they review.. it shows how they feel when they play the game.. FX in the start it will typically be frustrating (learning new controlls etc) and then maybe after an hour they will be happy/thrilled with the game.. at 18 hours they might be tired/bored and so on.

In addition to this they have a Box with many computer configurations that displays how much detail and what res you can play the game in 20 different PC systems.

So do all these features help? A bit , but the single most important thing that decides wether to checkout a game or not is Word of mouth from people you know or at least you know their taste in games One Million Graphs and Bars wont do that.

The Shack is a place were you will find many gamming people. Maybe Steve should make a "Just Released" page with the new games and people could post about them..

My take on reviews: I find as many reviews as possible on games that I have played. When I find a reviewer that more or less has the same take as myself, I gravitate towards his reviews for purchase decisions. As has been stated here, no single reviewer/review system is perfect as this is all subjective. I've seen horrible titles (bad reviews across the boards) that friends have really liked. This proves the theory that there is a key for every door.

So I wouldn't get hung up on creating the best reviewing system. People will make up their minds as to whether or not they agree with your overall assessment of a title and then it won't matter what system you create. Though, having said that, here are some takes I have on what should be included in reviews:

I agree that the Value category is far too subjective and should be left out.

Interface: I would like to know if the interface is intuitive, possibly using other games as baselines. I avoid titles with clunky interfaces that are mouse-click-a-thons.

Gameplay: I like reviewers impressions. You know, after you've played a game for 50+ hours, whether you are enjoying yourself or are not. Don't tell me about the lack of an inch thick manual or the fact that the particular genre is growing old. Tell me the aspects of the game you would change or the parts that you enjoy or could do without. I like knowing intimate details about a title because I love depth. In a sports game I want stats up the ass. In a RPG I want to know that the events are or are not all scripted.

I could go on, but I have to go to lunch.

Enjoy,

Build it and, if they agree with you, they will come.

Matt

Undoubtedly the best game reviews, for my money, are those that average gamers write on message boards and on Usenet. These are often only biased by one thing -- the player's sense of return on investment and the enjoyment had. That's all I need. Did you have fun? What didn't you like? What did you really dislike?

Beyond that, you're wasting my time. Don't bore me with your summary of the plot; video game plots are moronic, anyway, and the screenshots will more than adequately reveal what's up. Mainly, though, if you're in business writing game reviews, spare me your thoughts entirely. I simply ignore reviewers whose site's business model obviously depends on pleasing game publishers -- the Avaults, the IGNs, and other big name shills. They rarely meet a game that doesn't make them feel like they've just been blown by Jennifer Lopez, but average gamers know that's a rarity; most games make you feel like you're stuck in a hot tub with Laura Bush.

I've always thought that IMDB's review process (Matrix example: http://us.imdb.com/Ratings?0133093) is fairly accurate. I mean, 10,000 people can't be wrong. So let the gamer have a say. Eventually, you'd be able to see a weighted average and that would be more realistic.

The best reviews ever were made my the UK PSX magazine "PLAY". They usually were right when judging games. They also had a scale of enjoyable a game was after a set of hours. In some of the larger reviews (especially) they dug deep into what the core of the gameplay was like and (if any) the problems with it.
Of course since it was a UK mag the words "bloody" and "hell" were all over the crap game reviews.
Reading the reviews of crap games were always enjoyable with the good sense of humor the reviewers had. Something I hardly ever see in online reviews.

http://psone.totalgames.net/reviews/reviewsfull.epml?reviews.REF=1184&import.REF=1&format.REF=2&gameType.REF=1

I tried to deal with this in the past before. Basically, it's ideal to have an essay-format from multiple reviewers of totally different tastes, preferably some who like whatever is being reviewed, and some who don't.

You want the review presentation to be somewhat like a discussion between two friends about a game. You DON'T want it to be some hogwash comparison of features rated on some totally irrelevant score such as 4/5 or 8/10 or 80%. Nor do you want it to be "10 for graphics, 5 for gameplay."

The idea is to present arguments for both the positive and negative elements of the game which are worth noting in the review. Instead of using silly scales to rate "things" in the game, you discuss the arguments using comparisons to OTHER games, or whatever is relevant, for that matter. This gives your reader a foothold in understand where you're coming from, so that they can decide how THEIR opinions for what you're reviewing fit in relative to YOUR opinions, you being the reviewer.

Large numbers of people reviewing one topic (a game) isn't practical. I ended up deciding that in order to represent the game most fairly, it'd be best to have one person review the game in a positive light, then another person review it in a negative light. It would be preferable if you could find one person for each side who genuinely HATED or LOVED the game, but in such a way that they could be objective in their review, and not spew "fanboy" garbage. The point of splitting it up is to give impressions of the game from the two possible demographic extremes; those who hate it, and those who love it. This is opposed to some random opinion from some random reviewer who you don't know anything about, which is what traditional reviews are.

Think of it as being similar to the legal system of a defense attorney and a prosecution attorney. But it's not a COPY off of that model, it's simply a situation where the same solution will work.

Or, if you believe yourself to be very objective (like I decided I was), you could proceed to review the game yourself, first presenting the NEGATIVE side of of your attitude towards the game, then presenting the POSITIVE side.

However, this only works if you're one of the few people around who can honestly put an objective argument into text. This requires the reviewer(s) to be impartial and objective to the topic/game being reviewed... ie someone with an Intel banner reviewing the P4 isn't a good idea. However, with the current state of website advertising and revenue-hunting (which, to be honest, is a crazy battle, one which I'm glad I don't have to fight), you're pretty much forced to be in positions of looking like you've been bought out.

This is where brand loyalty comes in, and you've gotta make sure your viewers understand that you AREN'T bought out. It's a constant battle, you've gotta be impartial all the time, so that you can build more respect, and keep the respect you have.

I think the standard overview-and-score review paradigm has become essentially worthless (not that it was ever any good). Thinking back across a dozen or so recent games I've acquired, I can't think of a one the experience of which could be acceptably communicated by a typical review. Lets just have it off with some examples.

Final Fantasy Tactics
Holy christ, I love this game, but fuck if aint subtle. I watched some friends playing this game at one point, and thought it looked absolutely boring. Then sometime later, on a whim, I borrowed it and was instantly hooked. I've sunk so many hours into this game it's not even funny. But there is ABSOLUTELY NO WAY I can convey the manner and extent of the fun in this game with words, or even a demonstration. To understand why this game contends strongly for Best Game Of All Time, you MUST play it. Thankfully, it was re-released so you don't have to drop $80 on eBay for it ;)

Trade Empires
I'm only just getting into this, I'm on the 4th episode. TE is a semi-god game, similar to SimCity, Caesar, Roller Coaster Tycoon, etc, except that you don't quite have godlike powers. Your primary goal is to make money, and your mechanism for doing so is building markets for goods, and moving goods from production to consumption. Being an economist, I find this game really intriguing, but there's no way a reviewer could know that. Nor could any survey-type review communicate the hundreds of minute flaws in this game that drive me crazy half the time. I think the gist of this game could be communicated in a review, but it would have to be very detailed, to the point of boring the casual reader.

Max Payne
Reviews and mini-reviews of this game made it sound like Just Another Shooter. But it's actually quite a gem, innovative and uniquely entertaining. I could tell you that it has an engrossing storyline, wonderful art, great atmosphere, and the super-sweetness of Bullet Time, but none of that even begins to communicate the contribution that each of those factors makes to experience of playing the game. To truly know why Max Payne is fun, you must play it.

I think we draw two salient points from this. One is that far and away the best way to evaluate a game is to play it. This is I prefer PC gaming to console gaming, because a PC demo costs a few minutes of bandwidth, and a console demo costs $5 for 3 days, assuming the video store even has it. After not a small number of $50 failures, I'm not willing to shill for a game sight-unseen (game-unplayed? whatever) anymore. Reviews try to provide a surrogate source of information to aid in buying decisions, but as we've seen, they pretty much fail across the board. The other point is that reviews probably are the way they are for journalistic reasons. There are games for which it is possible to write a proper review, but such a review would be much longer than your typically anemic GameSpot blurb, and would disinterest the casual reader. And since review sites are, at some level, more interested in click-thru's than disseminating useful information, the status quo is likely to persist.

Anyway, better clock out and go home ;)


MoNsTeR

blah blah blah, they are game reviews, who gives a shit, it's not like people are going to be making decisions that could alter the fate of mankind based on game reviews...

The thing I like about having the shack is the people resources. People come here who enjoy games and usually if the majority of the shack lkes it (GTA3, Max Payne) you can bet that it's an entertaining game. Reviews of any sort (games, or otherwise) will always be subjective.....no matter how you look at it. But if the reviewer can outline the parts that he likes/does not like, and make it easy for the reader to compare his interests....that would be the best system.

I would be so happy if the shack decided to do reviews. From the writing Steve has done in the past, I know it's short and concise, but informative at the same time. I'm sure Maaaaa'eeen is a good writer too. Also, I agree with others....on the review page it could have a 'Rate this game' kind of thing, so the shack can vote as a whole on the game and see what the general consensus is. Amazon's system of recommendations ("customers who bought this item also enjoyed: ....", listmania (people can post their top 15 albums in a genre or whatever), and average customer rating) is great for me to figure out which CDs I can consider. I can easily see a similar system being implemented here on the shack, with us rating games and posting our comments on a thread off the main review (and perhaps having Steve select a few choice comments covering a range of opinions to link from the review itself like he's been doing with XBOX/GC impressions).

I'll have to say that I agree with Steve. Most reviews do not give you enough information about the game from the player aspect. At the same time I like the idea for compensating user changes.

If you were going to review a rehash of a pack of old games on a newer system, 15 year old Joey will give you a completely diffrent review from 28 year old Gerald. I am tired of "Classical" reviewers who want to snob their way through a review. Too many good games and good things are turned over by these types, it is a shame that game makers work hard to put out a game. I wouldn't even mind knowing in a review if the groud that made the game were only 5 or 6 people or a team of 40. I at least want an element of fairness.

Yeah man, those review systems out there are ALMOST as annoying as when people editorialize on a news page.

I think you should keep your guidelines very loose, and let the perceived strengths and weaknesses of each game dictate what you address in a particular review. That way, if you're reviewing Chessmaster 7000, you don't feel obligated to give it a "control" rating. I think reviews that address the specific pros and cons of a game, and then summarize at the end with a bulleted pros & cons list are pretty good. Don't feel obligated to give a numeric rating. Maybe just have a special award you give to games you feel are especially great, and for the rest, just let the review speak for itself. Or, you could use a four-star rating, where everything above 2 stars is a recommendation, and everything 2 and below is a non-recommendation.

Hm. I always miss the factor "haptics" in reviews, the overall smoothness in control,
interface and graphics; not the sheer fps, but the general feeling of how fast the game reacts to your input and how smooth the resulting action is. GrandRulios in this discipline were the BitmapBrothers in good ol' AmigaR-Times (vs.Ensemble today).

The only way I see to share impressions like these and other, never-mentioned fact0rs
that might be important for one's personal enjoyment of a game (you know those guys with a mustache that are completely obsessed with intros!), is a review by maaa'en followed by shacker's babbling. Make it so. Roger and out.

Let's see if I understand some of these categoies.

Gameplay - I agree that gameplay needs to broken down a bit more, but feel that it is by far the most DESCRIBED part in a review. So why go overboard with the review and nitpick indvidual parts of gameplay even further? Example of a game with poor gameplay: Red Faction.

Innovation - Each game SHOULD be a little bit different in order to give the gamer a tougher challenge to win the game and offer a fresh perspecitve of this type of gameplay. It might be something unexpected that happens in the game that makes you go "Holy shit!", or the general design of the game that tries to mix a lot of different game elements and make them work together to really give the gamer a truely distinctive game. Now Red Faction DID have good innovation. It's just that they couldn't get the gameplay the way they really wanted to make it fun to play.

Graphic Interface: Believe it or not, with all the sequals of games comming out in the next few months this category becomes a little bit more important because the reviewer has to say whether or not the interface has improved at all or not since the first game. New game titles are less important for this category. A good example of this is Mechcommander 1 and 2.

One combined score - I don't think that this category is all that confusing. This is the one category that either will sell the idea that this is a great game to buy or not. It's simply how much does this reviewer like the game? So the question always asked is from a scale of 1 to 10(or 1 to 5), HOW MUCH DO YOU LIKE THIS GAME? I really can't think of an objective way to weigh the overall score of a game based on it's sub category scores. So why bother? I'll just post my two cents about this game and leave it as that. Generally speaking, after browsing through like 10 reviews of the game from ten different people, you should start to get a good idea whether or not this game is cool or not. An example of this is MechWarrior 4. The first few reviews I saw said that it was a pretty good game and the next six or seven said it wasn't much fun compared to Mech 3. So the general opinion was that wasn't all that great of a game.

What is cool about the Shack is that we have a lot of good opinions about games and our good comments are finally being recognized. Real gamers opinions should be valued a little bit more since we play these damn games for fun and aren't waiting for a paycheck in the mail after posting a very positive review of Unreal II. Yeah you got to look out for those kinds of people, but after a while you find your way to a comment board and start to listen to what the real gamers have to say to get a better idea if the game is cool or not. Steve, I don't think the categories bother you so much as to WHO is reviewing the game. Seems you like our reviews of the Xbox and Gamecube better than anyone else's reveiw and maybe that's a really good thingTM. Finding a way to cut out all the media hype of a game and just get peoples opinions of it just seems to have more value.

Discuss

Steve just go visit ign and look at their game reviews. Now do the exact opposite of what they do.

Having a 'review' team is a good idea. Say you, Maarten, and one other person review a game. Each person works from the same 'template' - i.e. What steve was saying about different sections on environment, plot, weapons, etc... Have a 'fact' and an opinionated section for each one. Have each reviewer do their own opinionated review while one person does the factual aspect (avoid repeating the same stuff about the game). The important thing is to have each review voice their own opinions without the other team member's influence, making it as objective as possible.
Maybe it would look something like this:
Super Donkley Ball Turbo Fighter Review:
-Plot
**Plot facts about the game** (minus spoilers of course) done by one person
**Short paragraph from each of the three reviewers on their opinion of it**
-Graphics
**Facts (engine use, some benchmarks, color scheme, etc..)
**Opinions from each reviewer
-Repeat as needed for each section

Yah? What do ya think?

I like how gamespot does it. The editorial "reviewer," and right next to it are some reader reviews (I believe they are moderatred, which is a *good* thing.

Take each with a grain of salt. The reviewer is going to be overly critical, and the readers are going to usually be extreme in either direction. Else, why would they bother writing unless they could be passionate about it.

Do you review games from the perspective of the newbie gamer? (as the population represents?) or the seasoned gamer?(MOst likely to read the review) Or the Gamer of that particular games genre?(Most likely to ignore the reviewer's slant and go by peer's judgement.) Or of a specific age?(Why are adult males playing barbie Vid games and trashing it when that's not the intended audience?) All of those items make a significant difference in the outcome, and you'll never be able to please everyone, all the time. EVen the reviewer adjusting his or her review to suit someone is by definition compromising.

Even the 9.x's that GTA3 and MGS2 got could be argued from the perspective of a mormon that it breaks all rules of common behaviour.

*Shrug*

I think each shackster should have a rating section associated with them. Then you can learn which shacksters tend to like the same games you like. When you want to check the review of a game you can look at your preferd list of shackster game reviews.

My opinion on reviews depends on the actual site its posted on.

e.g. Avault reviews alot of differnt types of games. So when they post a RTS game I either trust them to know what is good or I can go back and see what they said about past games and if I agree. Samething with hardware and books and resturants.

In the case of the shack, if you post a review once a month, I might read it. But for sure it wouldn't change my opinion/buying decision.

Yeah? Well, that's just like, your opinion man.

Ok I'm going to post (again) one of the problems I have with many reviews: an overly high level of precision. Why do so many people rate games out of 100? How do you quantify the difference between a game that is given 96 and one that is given 97, ESPECIALLY given differences between reviewers?

This is one area where movie reviews generally have it right - give a review out of 5 or even 10. The maximum score doesn't mean the game is perfect of course, it just means it's a damn good game that most people would enjoy.

Oh, and of course adding a decimal point to a review out of 10 is of course a cop-out and exactly the same as a percentage, should be whole numbers.

The idea of somehow measuring value is a good one though, as long as you can work out how to do it right. Problems with single vs multiplayer, play time and so on.

One thing is i like to know the biases of the reviewer... it lets me know how likely i am to agree with them...

so if they list their 3 favourite game types, and top 2 games in each, then people can get an idea of where their tastes lie in relation to the reviewer's.

Here's my system:
Talk about the various 'features' or 'aspects' of the game. These could vary greatly between games. Rate the game on what it's got and whether it sucks the poo poo or rocks your world or you could care less. The important thing is to justify your reasoning. Let's compare Half-Life to Quake3. Half-Life's puzzles rule +1 because ... Quake 3's plot I could care less about +0 because ... (you don't need to bother considering puzzles in Q3 'cause it has none)

This gives you the flexibility to talk about the game on the things it tries to do instead of worrying about how it relates to another game ... which is pretty subjective. the this is better than that is a pretty flawed view. instead i want to know: this game's item collection rocks but it's fighting is wicked slow and sucks. so if i like to collect things ill buy it ... if all i want is to blow shit up i wont.

you can talk about as much stuff as you need to ... are classes in Wolf good or bad? you dont need to talk about them in Urban Terror. one class system may be better or not .. who knows. does it work in the game? is it good?

when you get all done you could do a ratio type total score = good / all. this will let people go ... hmmm ... this game is a great game for running around and solving puzzles that's all it does (it doesn't do anything badly) and it does it very well and i like puzzles so im gonna buy it.

What about sound? For fuck sakes why does sound always get the shaft? Don't think it's important, then unhook your speakers and see how fun your games are.

Usually when I read the ratings it's when I don't feel like reading a whole long review and just want a short take on the game. Maybe a solid summary that touches concisely on most of the points of the review could replace that, for my needs at least.

Another idea is how USA Today does editorials with a pro and a con view. It's hard to have 2 reviewers who always disagree, but if a positive review could be thrown up next to a negative view, it allows someone to better gauge the game. Most reviews are very onesided where they'll pretty much just include a brief footnote at the end with things that aren't in line with the rest of the review. A dissenting view is used by even the Supreme Court to illustrate the issue better.

Instead of just having one long review and a rating, I propose you have one page with a summary of the positive and negative review, and links to those reviews in case someone wants to read them all. This would be a fairly intuitive system.

so is the shack gonna do reviews fo sho'?

My webrowser might never leave the shack now :D

Have you considered "communal reviews"... using the comments system and an editor?

have a structure for the review... (I think replayability is an important part) and have people give commentary on the components of the game that most appealed or disagreed with them...

an interface can also be easy to operate or not, apart from it being easy to understand or not. i'd say the latter has to do with the learning curve, not with actual use.
though often an interface that's hard to understand is also hard to use once you do understand it.., i think it's two seperate issues.

IMO, when you score something, you should explain your scores. Why does Widgets get a 84% when Hoozits gets 93%?

Or have a +/- system where you tell the positive aspects of the game as well as the negative...then a "who will this appeal to" section

the only reason why I personally like reviews is to get a 2nd opinion, though when i search for reviews on a certain game, I usually read 4 or 5 different ones to see what the general concensus is.
It has never affected my purchasing though, I'll still buy/listen/watch what I like, no matter what the critics have to say, but if you're curious about something, its an ok place to find feedback.
Alot of the times I catch myself reading reviews of games I already own, just for curiosity. :) I did that with Operation Flashpoint a hell of alot, hehe.

I tend to read many reviews of any particular game,just to see if the reviews say the same things. And thank GOD for demos.I remember having only a couple of screenshots and a short blurb to go by when buying Nintendo and Super Nintendo Games you never heard of.Stare at the 1" squares and determine whether or not to plunk down a day's pay ....

Um,speaking of demos,does the Pool of Radience deme wipe your hard drive like the game,or did they fix that :)

Regarding ratings: It's a good subject, and one we at strategy-gaming.com have thought a lot about. I wrote http://www.strategy-gaming.com/reviews/ratings.shtml some time ago (kind of a work in progress, since the examples haven't been filled in...sorry) discussing just that subject.
I'd be interested personally in people's comments on the rating system and on the levels indicated. We're using this document as a reviewer GUIDLEINE however, not a hard-and-fast rule system. There have been a few games where I've arbitrarily raised or lowered the final value of a review purely because of how I feel after having played.

As a reviewer, I know many people will disagree with my ratings (sometimes the flammage is ridiculous), but my opinion about reviews in general is this: read lots of reviews, and keep track of the authors. After a while, you'll start to find that a certain few seem to be consistent with your own preferences and are worthwhile to check regularly. But anyone who reads ONLY the cgonline.com reviews or ONLY the strategy-gaming.com reviews is invariably going to be disappointed eventually.

Any review is going to be subjective, that's the nature of reviews. If you tried to do a completely objective review, the only things you could write about would be performance issues (framerate, tearing, misaligned textures, sound fuzz, lockups/crashes, interface bugs, whatever). You have to allow the reviewer his/her opinions, and that blows away complete objectivity right out of the box.

There have been games that were rated really highly that I couldn't stand (like #26, I hated Twisted Metal Black; I played the Sims for about an hour before I wanted to puke, etc.). When a game comes out that I am interested in, I usually go out and dig up all the reviews I can find, including player reviews, and browse over them. There are reviewers at a few websites, like Desslock @ Gamespot, whose reviews I usually trust because they seem to enjoy the same things in games that I do.

I like the way Gamespot allows users to post their own reviews of games. Even if some of them look like they were written by JeffK, a lot of em are well-thought and provide a good counterpoint to the actual Gspot review. If Shack gets into doing occasional reviews, it'd be nice if you did the same thing, without making casual visitors to the site dig through the article comments.

Gameplay - Why the hell is this only one category when there are so many different elements/types of gameplay that appeal to different types of gamers?

Agreed. Should be broken down.

Value - Entirely subjective, $40 is worth a heck of a lot more to some people than it is to others.


This is very important for many people. Usually this is dependant of game length and replayability. Had I know Max Payne would have been 7 hours gameplay with no replayability (BAD developers) then I would have waited a few months before buying it. I like to know if a reviewer thinks it's worth the money.

Innovation - Who the hell cares except for game developers trying to one-up each other and reviewers trying to schmooze developers telling them how cool and original they are? If it's fun, then it's fun.

Pretty much everybody who hates playing clones cares very much about this. I'm not going to purchase wolfenstein because it's going to be same old same old. I'm looking forward to thief, deus ex2 and enjoyed Alien VS predator because they all have and will bring something new to FPS. Fun is not always fun if it's just a big game of Deja Vu, been there done that. I hate clones and a lot of people do. This is one thing I look for the most out of a review.

One combined score - [long winded explanation here] I don't agree with this system. Different people weight different aspects of games differently.

Well of course everybody rates aspects of games differently. Reviews are not there to give you the decisive yes or no on a purchase. Reviews are meant to be taken as the reviewers point of you and not your own. If you purchase a game based on Steve's review only or Gamespots review only then your a llama. Go out and read all the reviews you can and only then make a decision. It's like getting an opinion from your friend, he will judge the game from his point of view and you take that information and use it as only part of your decision. The overall impression of the game is really important if you want to properly gauge the reviewers opinion.

There is no perfect review, perfect guide or perfect whatever. I personally don't like reviews that talk about all the little details you can get from the box or the games website. I don't want a 234908 page reviews that give me more info then playing the game would. I want their point of view, the things that stuck out as impressive, what was not so good and a brief overview of the game.

I don't want to spend my days reading reviews. If a review takes longer then 10minutes to read. You should get your review published as a "full-out" hardcover book to be sold in stores.