1. Quality matters: Its not the number of LEDs, but more to do with the quality of the LEDs, how they are positioned, the quality of the lens and reflectors used. I have seen some lights with 5 LEDS that doesn't have even half the visibility or brightness that some others with lesser number of LEDs .
2. Visibility: Is the light bright enough to be visible from a distance? At least from a couple of hundred metres? This is very important as you would want the car and truck drivers to notice you from a distance, and not when they are right behind you.
3. Side Visibility: Does the light have side visibility? This is useful when you are at a street intersection.
4. Waterproofing: Will the light be able to withstand rains and an occasional splash of water?
5. Mounting Options: Does it provide multiple options for mounting? Say on the seat post, on the seat stay and on your rear rack? Can it be clipped on to your saddle bag or on your shirt? You may not need all of them, but still worth considering based on your requirements.
6. Mounting mechanism: Is the mounting mechanism good enough to withstand the rigours of the toughest off road ride that you might be doing? There are numerous cases where people have lost their lights on a ride.
7 Battery life: This is generally not a constraint for LED rear light, but still worth checking .