A few words about reviews

By: aiko

Reviewing levels is a difficult topic. The critic has to judge other peoples’ works and thats not as easy as one might think at first.
Of course, every review only gives the subjective, personal opinion of the reviewer, which is caused by emotional impressions and feelings. Nonetheless, a review has to be objective in a certain manner. It has to be fair and unbiased.

In my humble opinion, the most important points when reviewing a level are the following: design/layout (how are the platforms placed?, how easy is it to navigate around the level? is the level circular or linear? how is the weapon selection and how are the weapons placed?), eyecandy/layer work (how many layers did the author use? whats on the layers? how “beautiful” is it?), music selection (does the background music fit to the atmosphere of the level? how large is it?), tileset choice (does the level use most of the tiles in the set? is the set overused?). All of those aspects go into each of my reviews. It’s not just the “level itself”, as some people mistakenly think.

Regarding tilesets, the size (it should have at least 50 tiles), the kind of tiles that are included (useful or not?), the colours, the transparencies and the masking play the important roles. Not to forget the filesize, because most people who play on the net will have to download the set, and sets that are large for no reason could cause problems like the annoying download lag. There are sets that are bigger than 250 KB but that should always be exceptions. Most people don’t know how smart color reducing could help decreasing the file size.

Every reviewer has a different opinion on all of the above mentioned factors, no doubt about that. For example, I prefer certain tilesets to others (I find Carrotus and Diamondus boring, to name two examples), I have a favourite style of tracked modules (demostyle!) and I definitely like a certain way of level building better than others (I prefer levels that use custom and/or new tilesets, I prefer custom music for custom tilesets [d’uh :~~] and I love open and circular maps). People are different, thats what makes the world an interesting place :~~) Hey, sorry… I had the feeling I should repeat all these stock phrases because many people sometimes forget about them. Be tolerant, respect other opinions. Of course, every reviewer should stick to the facts. If I criticize a level, it’s never meant as a personal attack. Believe me.
In my own reviews, and as you probably know, I’ve done many for J2O, I try to follow these rules. I’m pretty sure it’s not always fair and sometimes I am really completely undecided about my final rating. In the beginning, when (the “new”) J2O started, it seemed that I frightened some people with my “harsh” reviews. Later, about a month after the start, most other reviewers also got more into what they called “harsh” reviewing.
My theory on this matter is that reviews HAVE TO BE what they called “harsh”. Actually, I think “harsh” is only a different word for “fair”. If you have a level pack like, let’s say, “Another Story” by stripe, that contains about eight levels, loads of great custom music and an animated GIF, you can’t give it the same rating as a one-level submission, even if the latter is nearly perfect. I think you just have to compare the work and effort that was put into it. And if you need orientation, you should always remember that 5 is the average and a 10 (which I never gave) would mean a perfect level in every way, and there wouldn’t be anything better forever after. Isn’t that unlikely?

At the end of this article I would to draw your attention on the other major Jazz2 download sites that were big in the past (you will probably come up with lots of other sites I forgot to mention :-). In my opinion they were Jazz2City and, for a short time, from January until August 2000, UniverseJazz.

The Jazz2City download section was maintained by Steven Wakeman and in my opinion it really was (aside from the JMMB) the heart of the “community” at that time. Everyone submitted his or her work by e-mail and Wakeman actually wasn’t able to post and review them all quickly because it was so much work. But he did. And he really deserves our respect for all that hard work.
The J2C d/l section had about 2000 levels(!) when it closed in early 2000 and it still (at the time this article was written) is a great source for downloading levels and tilesets. Naturally, there were also some downsides on it: the authors weren’t able to submit their creations immediately, and there was only one reviewer who had all the load on his shoulders. Also, the rating system was, this is only my opinion, a bit underdeveloped. In the beginning there was a system using a 1-4/1-5 scale which was too limited, then it changed to a 3-star-system that is quite legendary now.
The reason for this fact is that Wakeman was a very good (others may say “harsh”) reviewer. His english was quite exacting (at least from a non-english speaking view :-) and only the elite submissions received the famous 3-star-download-recommendation.
Unfortunately, the download section got a bit week near the end of J2C, the reviews got very short and it took too long of a time before the levels were posted. Thats probably why Wakeman quit after over one year. But boy…he did an awesome job! I’m bowing down again.

However, a site called UniverseJazz came and tried to take J2C’s place. It pretty much copied most of the content of J2C, but it wasn’t able to copy the download section. The reason for this was that Alienators reviews were good, but most people found them too harmless and uncritical. He and the other staff members that posted the download also used the 3-star-rating-system that was made famous by J2C. I already described the disadvantages and this time they showed up better. There were simply too many bad levels that got a three star rating. There also was a download ratings “highscore” (there already was something similar at J2C) that counted every click on that download link. To be honest, I never liked that idea because it’s too easy to manipulate and cheat. But apart from that, and the way how UJ suddenly died, we also should pay respect for Alienator’s work as the download master of his site. It kept people making levels.

On a side note, i would like to say that I really think J2O has the best rating system ever. Be it the possibility for everyone to rate (everyone is able to rate and give his opinion, that is what we call DEMOCRACY) or be it the rating scale from 1-10 (including the less useful .X ratings), everything is just cool. The sort and statistics functions shows the user interesting facts and gives the abilities to compare authors, submissions and reviewers (for example, how “harsh” a reviewer is). Of course, there is always space for some improvements (hint, hint).

Okay guys, NOW you noticed it. I only wrote this useless article for one fair reason…here goes my plea: Do some reviews! It’s fun :-)

P.S.: Sorry for my bad english (vocabulary+grammar) (partially fixed by Violet CLM)

Comments

Wakeman on November 17, 2000 05:00

First thing, thank you for your kind words, Aiko. I truly appreciate it. My heart wrenches a little when I read stuff like that, as I still wish that Jazz 2 City could be alive, but as you can see, I don’t even have time to review levels here at J2O.

I definitely agree that reviewing levels is a difficult thing to do, much for the same reasons that you mentioned. I frequently received questions and complaints about ratings on J2C, but I realized that you just can’t please everybody. I have not read many of your reviews, Aiko, but it sounds like you must be doing an excellent job.

For Jazz 2 City, I found that simplicity was important for maintaining the Downloads section. The 3-star system was DethMan’s idea. We changed it because we found that it was too hard to remain consistent when you were rating to one-tenth of one point, and we felt that fractions of a rating were meaningless in the first place. However, I admit that just three stars was a little broad, which is why I later changed it to a kind of 10-star system with the addition of ratings like Low ** and High **. My reviews were short not only because of lack of time, but also because I felt that it would be more beneficial for downloaders to just have a concise summary of the level. I figured that people would rather play the level than read an opinionated review that they probably wouldn’t end up agreeing with on many points. Finally, the reason why I repeatedly denied requests from various people who asked to help me with the Downloads: I knew it would be far too complicated with our system, and two people would not be rating on the same scale.

In regard to how I rated: As you may have noticed, I was always mentioning Layout and Eye Candy as important factors. That was the general way of saying what Aiko was talking about above. You may also remember some of the “rules” I set down, but I mainly wrote those out of frustration. ;) Although I was frequently leniant, especially with the tilesets, I wanted to be harsh so that people would improve. Once you see 2000 levels, they all start looking the same if they don’t have a creative aspect to them. Some people made it their highest goal to get ***, and I was glad when I could help in that process.

After all that, I’ll give my comments on J2O’s system. Frankly, I’m very impressed. You managed to get around almost every problem J2C had. It’s instantaneous posting, the rating system actually does seem meaningful to me, and the fact that many people can review a level is very great. Being able to see the level from the point of view of many reviews is a definite plus. It’s nice that everybody gets to take part.. a democracy, as Aiko said. :)
 
Disadvantages? There’s less control over what is posted. That’s about all I can think of. J2O is not suffering from a lack of downloads, like most newer sites. That’s the only future problem I could see, but at least I don’t forsee this site going down like UJ, since it kinda runs itself.

Sorry to add a long comment to a long article. If you got anything out of this, it should be to USE MORE EYE CANDY! ;) I miss being able to say that.

-Steven

Ninja Dodo on November 19, 2000 05:00

There have been a lot of biased reviews lately. For example; one person submits something and doesn’t get a single review or just a few (uninterested) reviews while a well-known person with a similar submission get’s around 30 reviews that are between the 9 and 10. Just that you like a person doesn’t mean you should give their stuff high ratings…I know this sounds harsh but it DOES happen.
 
I also agree with the New spaz that people should stop giving tilesets low ratings because of a crappy example level. A good example level is a plus but it shouldn’t decrease the rating if it’s not good. The tileset should be rated on it’s own.

FQuist on November 20, 2000 05:00

Strangely, I agree to the people that lower their ratings a few points because of the lack of an example level.

It’s very hard to judge tilesets. Actually people that judge a tileset should try to make a level with it, since that’s the point. An example level demonstrates how userfriendly a tileset is. You can test masking much easier, you are impressed faster then when you look at a bunch of tiles.

Wakeman on November 20, 2000 05:00

I agree about the tilesets.. that’s why I was usually pretty leniant when it came to rating them. Any half-good tileset takes a lot of work. I also agree with Ninja. Even though I got my nickname “Wakeman is Biased” from J2C Downloads, it was for a whole different reason. :)

-Steven

Ice M A N on November 25, 2000 05:00

As you all probably know, I definately agree with the no-example-level-needed campaign. I also agree when Ninja Dodo says it does not show the user-friendliness at all. My tilesets are usually NEVER user-friendly, but I can still make a considerably good level with them.

Even if you guys still want your darn example level, you shouldn’t really rate it as a level (i.e. layout, music (which I don’t care about in levels, but am fine with having people adjust ratings because of it), etc…), just look for how well the tileset fits together, the environment, the various landforms, etc… The level itself should not effect ratings, therefore there should be no need to have it..

One more thing to you people who need an example level you should NOT base your tileset-rating on the example level and first impression alone. Make a little section of a level with it! I start of with just a (large) plain rectangular room and see how “beautiful” I can make it and how easy it is..

As for level ratings, I guess most people are doing them well enough…

Ninja Dodo on November 25, 2000 05:00

An example level shows what a tileset can do…no more, no less. If it does that there’s no reason why the rating should be lowered because it’s not a nice level.
 
Also an example level says nothing about the userfriendly-ness of a tileset. A good example level can be made with a tileset that is near impossible to work with for everyone except the author.

So, I stand by my statement that a crappy example level should not decrease tileset ratings.

KJAZZ on December 03, 2000 05:00

Few words? Ya gotta be kidding :-)

Ninja Dodo on January 06, 2001 05:00

YAJ! A new trend has emerged. People are now complaing about tileset-example levels not having music. ARGH!
 
I mean, demanding that there is an example level is one thing, but MUSIC?!

I think people should not expect that a ‘real’ level accompanies a tileset. An example level is a demonstration of a tileset’s capabilities, nothing more…

FQuist on July 14, 2001 04:00

I don’t care for crappy example levels. But I do care for example levels.

Disguise on October 16, 2001 04:00

NOTE: The below is taken from a reply I posted on Gargoyle’s article, which is about the same topic. I just hate writing it out again.
 
——====COMMENT====——
 
There are three things in thw world that REALLY annoy me, and I mean REALLY annoy me. I’m talking about the kind of stuff that will make you break your room down just to cool down. These are:

1) Someone calling me when I’m in the middle of a brainstorm.

2) Slow internet.

3) When people rate the example levels of TILESETS!!!! AAAAHHHHH!!!

Ohhh, I’m mad already. To think that someone would spend weeks (if not months) drawing a tileset, and then to just be nice to people, quickly makes an example level to show them how to use the set. Later on, only to find that people have ignored his weeks of work just to comment on a bad example level. That REALLY winds me up. As a matter of fact, so much, that I will make an example level for ANY tileset maker out there (if you want). If you’re being rated badly because of your example levels (which is VERY wrong), just ask me and I will make one for you.

Allow me to repeat myself:
 
AN EXAMPLE LEVEL IS NOT A FACTOR THAT SHOULD BE RATED WITH TILESETS!! LOOK AT THE TILESET, NOT THE LEVEL!!

Unknown Rabbit on November 06, 2001 05:00

Ninja Dodo, I believe I understand why people like music in example levels. If they ever use the tileset, the example level’s music gives them a clue on what music goes well with the tileset. Just my four (inflation!) cents.

Ninja Dodo on November 10, 2001 05:00

Oh well, you know what they say…“The customer is always right”.

I guess I’ll just have to adapt if I don’t want my reviews to drop.

Disguise on December 05, 2001 05:00

Music is nice for any level, but basically only for the niceness.

TILESETS ARE NOT PACKS! IF THEY WERE PACKS THEY WOULD BE MULTIPLE!! RATE THE TILESET, NOT THE OTHER STUFF!!!

Hmmm, perhaps I should post this 1000 times more to get my point across, you will see me whine about this until I die, so expect more.

(Duplicate post deletion edit. -Trafton)

Jazz Jackrabbit 2 Online