Ok, I'll try to explain some of this (though you should read apples documentation on retain/release, its very good and very detailed.
1) Pablo is right, you should call [super dealloc] in the dealloc method.
2) You do not actually need your layer to be a property, your code will fully work without it. You only need it if you access the layer from outside of gameScene.
3) Assuming you keep your code as it is, here is what happens in your code:
layer = [GameLayer node]; // the node function creates a GameLayer for you which will be autoreleased when the end of init is reached. So current retainCount of layer is 1.
[self addChild:layer]; // You add the layer to the cocos2d scene, cocos claims the layer and retains it, retain count is now 2.
} // the end of the init function is reached, layer is autoreleased. The retain count is now 1 again (because cocos is still retaining)
You now no longer own layer! Only cocos owns it. If you call [gameScene removeObject:layer] somewhere the retain count will reach 0, and the layer will be dealloced. If you then call [layer release]; your retain count will reach -1 and your app will crash.
Heres whats wrong: if you want to own the layer using the property system, you need to not only declare layer as a property, but use it as a property. if you do self.layer = [GameLayer node]; (notice the self!) you use layer as a property and not as an objective C object. Now internally, because you said @property (nonatomic, [strong]retain[/strong]) GameLayer *layer;, cocoa will convert self.layer = [GameLayer node]; into
layer = [GameLayer node];
[layer retain];
As you now own the object correctly, you can dealloc it in your (void) dealloc. You could also do:
self.layer = nil;
because as you are using it as a property by calling it via self, it is actually converted to:
[layer release];
layer = nil;
I hope this helps, but again, i strongly recommend you read apples documentation on memory management and properties, else you'll have memory leaks all over the place.