[cocos2d 2.0] custom fullscreen/sprite effects

cocos2d for iPhone » Programming » cocos2d support (graphics engine)

[cocos2d 2.0] custom fullscreen/sprite effects

(18 posts) (5 voices)

Started 1 year ago by cndy
Latest reply from toglia3d

Tags:

cndy
Member

Hi,

i would like to perform "per pixel" operations either on the entire screen or on specific sprites. Is there an existing node/approach in cocos2d that allows me to do that? When i talk about per pixel operations, i have pixel-shaders in mind. As the iphone gpu does not support pixel shaders, i need to do the calculations on the cpu. I am aware of the fact that this is a problem in terms of performance (however, performance is not my primary focus for the time being).
Is anybody rendering fullscreen quads to a texture for further processing? Whats your experience with that?

thanks

Posted 1 year ago #
leocck
Member

Search for the term CCMutableTexture:
http://www.cocos2d-iphone.org/forum/search.php?q=CCMutableTexture

Posted 1 year ago #
cndy
Member

thanks leocck!

Posted 1 year ago #
toglia3d
Member

This post cries for a proper discussion now that the new cocos2d 2.0 is now supporting opengl shaders.

I'm very curious with what can really be achieved with this cause frankly all of the official shaders tests were not very exciting...

From what I saw the shader is expecting a Texture2d instance, so it is only a matter or getting screen grabs fast enough for it to be potentially usable. So what do you guys think? Is there a fast way of getting the screen buffer into a Texture2d instance?

I saw this code for doing screen captures, but its originally intended for the 1.0 api, so at first it didn't compile and after doing some minor changes it crashed the app.

Please comment!

Posted 6 months ago #
toglia3d
Member

I'm really surprised that not much has been talked about shaders throughout this forum. Is there something about shaders I'm missing? I read that they were somewhat slow... But nonetheless they are the core of modern graphics.

Posted 6 months ago #
araker
Moderator

@toglia3d, cocos 2.0 is still in early development. The shader tests are more of a proof of concept than to knock your socks off. Shaders give more control over the whole OpenGL pipeline, so performance is better if you know what you're doing and for effects there are much more possibilities, that would be impossible to implement (efficiently) without shaders.

If you want to know what you can accomplish with shaders, try a book. iPhone 3d programming is good place to start (it has a free online version).

Posted 6 months ago #
toglia3d
Member

@araker I know the 2.0 version is only a couple of months old, but still I was just curious why wasn't many people talking about this exciting topic. I actually have some shader programming experience but I'm quite new to mobile APIS...

Anyway, I'm also trying to get a fullscreen shader working but I don't know how to effectively get the screen buffer into a Texture2D thats what the shader pipeline is expecting.

Posted 6 months ago #
araker
Moderator

@toglia, since you have a specific problem, I think you're better off if you opened a new topic and use [cocos 2.0] as prefix for the subject.

Why there hasn't been much talk about it yet, I wouldn't know. Most of the talk about 2.0 is done in this topic.

Posted 6 months ago #

Ok, so I have done some progress that I would like to share. Maybe get a little feedback too.
This is the Blur.fsh applied in realtime to a CCRenderTexture of the entire screen buffer. The RenderTexture is rendered on top of the main layer with some opacity altered.
I took the liberty of using some code from other users of this forum.

Wel, first of all, I'm using cocos2d 2.0 so I can use shaders. As you can see the fps are pretty low, like 11 fps average. Nonetheless, I think my next step is finding out how to make a layer (CCLayer) draw into a virtual context so the image only renders once. If somebody sees a big mistake here please post! I'm starting to think maybe fullscreen shader are too expensive to be useable in realtime. I've read people writing about RenderTextures being slow, but I have to keep reading.

After grabbing the blur test code from the cocos2d shader tests you'll have something like:

@interface SpriteBlur : CCSprite
{
	CGPoint blur_;
	CGFloat	sub_[4];

	GLuint	blurLocation;
	GLuint	subLocation;
}
@end

@implementation SpriteBlur

-(id) initWithTexture:(CCTexture2D*)texture rect:(CGRect)rect
{
	if( (self=[super initWithTexture:texture rect:rect]) ) {

		blur_ = ccp(0.001, 0.001);
		sub_[0] = sub_[1] = sub_[2] = sub_[3] = 0;

		self.shaderProgram = [[GLProgram alloc] initWithVertexShaderFilename:@"PositionTextureColor.vsh"
                                                      fragmentShaderFilename:@"Blur.fsh"];

		[self.shaderProgram release];

		CHECK_GL_ERROR_DEBUG();

		[shaderProgram_ addAttribute:kCCAttributeNamePosition index:kCCVertexAttrib_Position];
		[shaderProgram_ addAttribute:kCCAttributeNameColor index:kCCVertexAttrib_Color];
		[shaderProgram_ addAttribute:kCCAttributeNameTexCoord index:kCCVertexAttrib_TexCoords];

		CHECK_GL_ERROR_DEBUG();

		[shaderProgram_ link];

		CHECK_GL_ERROR_DEBUG();

		[shaderProgram_ updateUniforms];

		CHECK_GL_ERROR_DEBUG();

		subLocation = glGetUniformLocation( shaderProgram_->program_, "substract");
		blurLocation = glGetUniformLocation( shaderProgram_->program_, "blurSize");

		CHECK_GL_ERROR_DEBUG();
	}

	return self;
}

-(void) draw
{
    ccGLEnableVertexAttribs(kCCVertexAttribFlag_PosColorTex );
    ccGLBlendFunc( blendFunc_.src, blendFunc_.dst );	

    ccGLUseProgram( shaderProgram_->program_ );
    ccGLUniformModelViewProjectionMatrix( shaderProgram_ );        

    glUniform2f( blurLocation, blur_.x, blur_.y );
    glUniform4f( subLocation, sub_[0], sub_[1], sub_[2], sub_[3] );

    ccGLBindTexture2D(  [texture_ name] );
    //ccGLBindTexture2D(  [newTexture name] );

    //
    // Attributes
    //
#define kQuadSize sizeof(quad_.bl)
    long offset = (long)&quad_;

    // vertex
    NSInteger diff = offsetof( ccV3F_C4B_T2F, vertices);
    glVertexAttribPointer(kCCVertexAttrib_Position, 3, GL_FLOAT, GL_FALSE, kQuadSize, (void*) (offset + diff));

    // texCoods
    diff = offsetof( ccV3F_C4B_T2F, texCoords);
    glVertexAttribPointer(kCCVertexAttrib_TexCoords, 2, GL_FLOAT, GL_FALSE, kQuadSize, (void*)(offset + diff));

    // color
    diff = offsetof( ccV3F_C4B_T2F, colors);
    glVertexAttribPointer(kCCVertexAttrib_Color, 4, GL_UNSIGNED_BYTE, GL_TRUE, kQuadSize, (void*)(offset + diff));

    glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);

}
@end

I still don't know how to render my CCLayer into a virtual CGContext, so I'm basically duplicating the screen inside a CCRenderTexture.

mRenderTexture = [[CCRenderTexture renderTextureWithWidth:s.width height:s.height] retain];
CCSprite *auxRenderTextureSprite = [mRenderTexture sprite];
[auxRenderTextureSprite setPosition:ccp(s.width/2, s.height/2)];
[auxRenderTextureSprite setOpacity:255*0.5];

We are creating a full screen CCRenderTexture and lowering the opacity.

Then we create the BlurShader and pass the CCTexture2d instance of the Render Texture and flipping the Y coordinate cause render textures invert themselves for some reason.

SpriteBlur *blurSprite = [SpriteBlur spriteWithTexture:[[mRenderTexture sprite] texture]];
[blurSprite setPosition:ccp(s.width/2, s.height/2)];
blurSprite.scaleY *= -1;
[self addChild:blurSprite];

Lastly we need to update the CCRenderTexture with the actual screen buffer. This goes in the update(tick) method.
The first line makes our CCRenderTexture have alpha opacity.

[[mRenderTexture sprite] setBlendFunc:(ccBlendFunc){GL_ONE, GL_ONE_MINUS_SRC_ALPHA}];
[mRenderTexture beginWithClear:0 g:0 b:0 a:0];
[self visit];
[mRenderTexture end];

Posted 6 months ago #

slembcke
Administrator

You can't render OpenGL into a CGContext, you can only render CoreGraphics into a CGContext. Rendering into a render texture and then blitting that to the screen is the correct thing to do.

Is the image you posted supposed to contain the source for the fragment shader? It's in a password protected directory apparently and it prompts me to log in to see it. Looking at the rest of your code though I suspect that you are doing a loop inside the fragment shader to look up adjacent texels. This is going to cause an indirect texture read and will be very slow. For performance reasons you want to read texture coordinates directly from varyings passed from your vertex shader without any computations as then the GPU can fetch the texels for you before the fragment program is even run.

The usual way of performing a fast blur is to perform an iterative blur. You use a second render texture and ping-pong the results back and forth between the two. Each time averaging 4 texture samples around each texel as passed in directly using 4 sets of slightly offset texture coordinates. This is how they did the effect in real time before shaders came around, and AFAIK it's still the fastest way to do it. You also probably don't want to use a full size render texture either. It's going to get blurred anyway and halving the texture resolution saves you 4x the fill rate.

Posted 6 months ago #

toglia3d
Member

I'm sorry, its supposed to be a picture of my results. Can you please check again (I'm having some problems with my server).
Anyway, the shaders I used come with cocos2d 2.0 and I think its a fast box blur:

fragment shader:

#ifdef GL_ES
precision highp float;
#endif

varying vec4 v_fragmentColor;
varying vec2 v_texCoord;

uniform sampler2D u_texture;

uniform vec2 blurSize;
uniform vec4 substract;

void main() {
	vec4 sum = vec4(0.0);
	sum += texture2D(u_texture, v_texCoord - 4.0 * blurSize) * 0.05;
	sum += texture2D(u_texture, v_texCoord - 3.0 * blurSize) * 0.09;
	sum += texture2D(u_texture, v_texCoord - 2.0 * blurSize) * 0.12;
	sum += texture2D(u_texture, v_texCoord - 1.0 * blurSize) * 0.15;
	sum += texture2D(u_texture, v_texCoord                 ) * 0.16;
	sum += texture2D(u_texture, v_texCoord + 1.0 * blurSize) * 0.15;
	sum += texture2D(u_texture, v_texCoord + 2.0 * blurSize) * 0.12;
	sum += texture2D(u_texture, v_texCoord + 3.0 * blurSize) * 0.09;
	sum += texture2D(u_texture, v_texCoord + 4.0 * blurSize) * 0.05;

	gl_FragColor = (sum - substract) * v_fragmentColor;
}

vertex shader:

attribute vec4 a_position;
attribute vec2 a_texCoord;
attribute vec4 a_color;

uniform		mat4 u_MVPMatrix;

varying vec4 v_fragmentColor;
varying vec2 v_texCoord;

void main()
{
    gl_Position = u_MVPMatrix * a_position;
	v_fragmentColor = a_color;
	v_texCoord = a_texCoord;
}

Thanks for your advice. I'll be checking how to reduce my RenderTexture size. But my main problem is that I don't want my main layer to be drawn on the screen but in a Texture2d that I can modify with the shader THEN rendering it to the screen. Right now I'm doing two screen size renders, the blurred one and the normal "hello world" layer. What if I wanted to render only the blurred one?

Posted 6 months ago #

slembcke
Administrator

Box blur yes. Fast box blur no.

The texture coordinates are being looked up like this:
sum += texture2D(u_texture, v_texCoord - 4.0 * blurSize) * 0.05;

That is an indirect read because it has to compute v_texCoord - 4.0 * blurSize before it can fetch the texel you want. Compute an array of texture coordinate offsets in the vertex shader and pass them into the fragment shader in a varying. That way the GPU can fetch the texels before it even starts running the fragment shader and it should be much faster. With a lower texture resolution, you can probably get away with only 5 texture reads further saving on VRAM bandwidth.

Posted 6 months ago #
toglia3d
Member

@slembcke Thanks again. I'll definitely need to invest some time on shader optimizations. For now I'm still trying to draw only the CCRenderTexture on the screen. My second attempt was to put the RenderTexture into another CCLayer. This is a slightly better solution cause the RenderTexture does not "stain" itself, also for some reason I got a much beter performance, almost double.

I had the hope that the "visit' method would still work even if I hid the HelloWorldLayer but no. So, I'm closer but still not there...

Posted 6 months ago #
slembcke
Administrator

Oh, also a couple other things to keep in mind to get the performance up. Blending is very expensive, disable it when blitting the render texture. Also, because you'll just be drawing over the entire screen anyway, disable clearing the screen as that also burns up a lot of fillrate.

Posted 6 months ago #
toglia3d
Member
@slembcke I'm have been trying to lower the resolution of my shader texture like you said.

So the idea is to apply the shader on a lower resolution grab of the screen, right? I see how this can improve the performance but I'm stuck implementing it.

Right now, I'm using the visit method onto the render texture to get the screen capture. In this case how would I shrink that texture down. If I try creating a small rendertexture like this:
```
renderTexture = [[CCRenderTexture renderTextureWithWidth:s.width/2 height:s.height/2] retain];
//notice the texture is half the size of the screen
```
When doing the "visit" I get a cropped image of the screen. On the contrary if I use the scale method lo lower the resolution, I'll still have the same buffer size although the content is small...

Could you please share some specifics implementation of creating a half size texture of the screen?
Posted 6 months ago #
toglia3d
Member

So this is what I did.

I create 2 render textures. One with half scale but full screen resolution, and the other with double scale but half screen resolution. The main layer visits the full screen texture (its grabs the whole screen and shrinks it), and this last one visits the small one. Cause the small one has double scale it fills the screen up again perfectly. The shader is applied to the small render texture of course.

I was hoping some performance boost, but I didn't see much. I'm in 23 fps now. The shader is now having half the work it had before, and none of the textures have any blend functions applied.

@slembcke Is this what you ment?

Posted 6 months ago #
slembcke
Administrator
Are you applying the shader to the small texture when drawing it back to the screen (the shader is per-pixel, not per texel) or when drawing the big texture into the small one (1/4 as many pixels to draw). Is blending disabled when drawing into the small texture?

Also, you could use a small texture buffer for the initial one too if you did this:
```
[rt begin];
node.scale *= 0.5;
[node visit]
node.scale *= 2.0;
[rt end];
```
Posted 6 months ago #

toglia3d
Member

Ok, I got it working on almost 60 fps, it lowers to 56 when adding a few boxes.

CGSize s = [[CCDirector sharedDirector] winSize];

renderTextureA = [[CCRenderTexture renderTextureWithWidth:s.width/2 height:s.height/2] retain];
renderTextureB = [[CCRenderTexture renderTextureWithWidth:s.width/2 height:s.height/2] retain];

blurSprite = [SpriteBlur spriteWithTexture:[[renderTextureA sprite] texture]];

[[renderTextureA sprite] setBlendFunc:(ccBlendFunc){GL_ONE, GL_ZERO}];
[[renderTextureB sprite] setBlendFunc:(ccBlendFunc){GL_ONE, GL_ZERO}];

renderTextureB.scaleX *=2;
renderTextureB.scaleY *=-2;

//On the lower left quarter of the screen.
[blurSpriteA setPosition:ccp(s.width/4,s.height/4)];
[renderTextureA setPosition:ccp(s.width/4,s.height/4)];

//On the center because scaled by 2 will fill the screen.
[renderTextureB setPosition:ccp(s.width/2,s.height/2)];

[blurLayer addChild:blurSprite];
[blurLayer addChild:renderTextureB];

//I scaled the "HelloWorldLayer" and moved it to the lower quarter for good. This is because the visit starts from the 0,0 of the screen to the texture's size.
[mainLayer setScale:[self scale]*0.5];
[mainLayer setPosition:ccp(-s.width/4,-s.height/4)];

on update:

//This captures the screen on a half scaled render texture.
[renderTextureA beginWithClear:0 g:0 b:0 a:1];
[mainLayer visit];
[renderTextureA end];

//blurSprite's texture instance points the renderTextureA. renderTextureB stretches the half shaded image to fill the screen again.
[renderTextureB beginWithClear:0 g:0 b:0 a:1];
[blurSprite visit];
[renderTextureB end];

@slembcke Now I have gathered some questions:

1) Is there a way of "shading" a texture without actually drawing on the screen? Like having the shader code run on a "visit"? I found out I had to add the blurred sprite to the layer hierarchy for it to work.

2) So if I wanted to add several shaders effects, I would have to draw and read the screen several times, right? Is that what you call the "ping pong"?

Posted 6 months ago #

RSS feed for this topic

Reply

You must log in to post.