iPad + external display

Forums Programming cocos2d support (graphics engine) iPad + external display

This topic contains 26 replies, has 18 voices, and was last updated by  roots 2 years, 5 months ago.

Viewing 25 posts - 1 through 25 (of 27 total)
Author Posts
Author Posts
June 1, 2010 at 5:35 pm #222105

wcrtr
@wcrtr

Hey,

Has anyone experimented using the UIScreen class to display a cocos ipad app on an external monitor / tv?

The code I’m using is throwing an error in the EAGLView swap buffers method, and the external display is not being set correctly.

OpenGL error 0x0506 in -[EAGLView swapBuffers]

here’s the code. any thoughts? I swear I got this to work once, but I’ve yet to duplicate it.

if([[UIScreen screens]count] > 1) //if there are more than 1 screens connected to the device
{
CGSize max;// = CGSizeMake(1024, 768);
UIScreenMode *maxScreenMode;
for(int i = 0; i < [[[[UIScreen screens] objectAtIndex:1] availableModes] count]; i++)
{
UIScreenMode *current = [[[[UIScreen screens] objectAtIndex:1] availableModes] objectAtIndex:i];
if(current.size.width > max.width);
{
max = current.size;
maxScreenMode = current;
}
}

//Now we have the highest mode. Turn the external display to use that mode.
UIScreen *external = [[UIScreen screens] objectAtIndex:1];
external.currentMode = maxScreenMode;
window.screen = external;

} else {
window.screen = [UIScreen mainScreen]; //otherwise we just use the normal screen
}

// cocos2d will inherit these values
[window setUserInteractionEnabled:YES];
[window setMultipleTouchEnabled:YES];

// Try to use CADisplayLink director
// if it fails (SDK < 3.1) use the default director
if( ! [CCDirector setDirectorType:CCDirectorTypeDisplayLink] )
[CCDirector setDirectorType:CCDirectorTypeDefault];

// Use RGBA_8888 buffers
// Default is: RGB_565 buffers
[[CCDirector sharedDirector] setPixelFormat:kPixelFormatRGBA8888];

// Create a depth buffer of 16 bits
// Enable it if you are going to use 3D transitions or 3d objects
// [[CCDirector sharedDirector] setDepthBufferFormat:kDepthBuffer16];

// Default texture format for PNG/BMP/TIFF/JPEG/GIF images
// It can be RGBA8888, RGBA4444, RGB5_A1, RGB565
// You can change anytime.
[CCTexture2D setDefaultAlphaPixelFormat:kTexture2DPixelFormat_RGBA8888];

// before creating any layer, set the landscape mode
[[CCDirector sharedDirector] setDeviceOrientation:CCDeviceOrientationLandscapeLeft];
[[CCDirector sharedDirector] setAnimationInterval:1.0/60];
[[CCDirector sharedDirector] setDisplayFPS:YES];
[[CCDirector sharedDirector] setDepthBufferFormat:kDepthBuffer16];

// create an openGL view inside a window
[[CCDirector sharedDirector] attachInView:window];

CCDirector sharedDirector] runWithScene: [HelloWorld scene;

[window makeKeyAndVisible];

June 1, 2010 at 6:24 pm #284850

wcrtr
@wcrtr

btw, this basic code for setting the screen mode works using a standard UIKit app… any thoughts on how the cocos director / openGL stuff might be throwing it off?

thanks!

June 3, 2010 at 5:28 am #284851

sea
@sea

Hey there,

I’ve been experimenting with it. I haven’t gone down your route though, but your code looks ok to me. What actually happens?

I want to mirror the display. However, the solutions that I’ve seen floating around (notably the google one), are pretty useless unless you just have a standard simple UI. Once you try doing anything that needs performance (ie cocos2d), these solutions kill both displays.

Perhaps someone with some deeper knowledge of the internals of cocos could help point the direction (hello riq??). I was thinking that there should be a way to dump a buffer to the external display per frame somehow, instead of having to grab a screen that is being displayed. Not sure if it’s possible though.

I think that having an easy way the mirror the cocos2d display and/or add multiple displays would be *very* useful to a great many people in the future…

Any suggestions would be greatly appreciated

June 3, 2010 at 12:58 pm #284852

rwenderlich
Participant
@rwenderlich

I am also very interested in this.

June 3, 2010 at 3:02 pm #284853

Dunc
Participant
@dunc

Me too; I have a little iPad project in mind that would need to do just this, so would be interested in any hints…

June 5, 2010 at 3:31 pm #284854

rwenderlich
Participant
@rwenderlich

Bump! Any hints on this? :]

June 5, 2010 at 4:21 pm #284855

riq
Keymaster
@admin

I haven’t played with multiple screens yet, so I don’t know how to do it.

What I can say is that cocos2d renders everything in a EAGL view, which is a subclass of UIView ( see Support/EAGLView.m)

This view (EAGLView) will be added as a child of the main window.

The method CCDirector#attachInView creates the EAGLView and it adds it as a child of the window.

So, my intuition tells me:

- in order to render to a different screen, the only thing that you should do is to obtain the window of that screen, and pass that window as argument to CCDirector#attachInView.

- if you want to render the same scene in 2 different UIScreens, then unless UIKit / SDK has an special API that connects 2 screens, you should capture 1 screen using RenderTexture or any other Frame Grabber technique, and re-render it in the other screen. This technique is somewhat expensive FPS-wise, but I think it is cheaper than rendering the scene twice.

June 5, 2010 at 7:44 pm #284856

rwenderlich
Participant
@rwenderlich

Thanks much riq!

June 6, 2010 at 6:09 am #284857

legend
@legend

Hmmmmm, has anybody here figured this one out yet?

June 6, 2010 at 6:14 am #284858

riq
Keymaster
@admin

quick update: I think that drawing the scene twice (once in one screen and another time in the other screen) might be faster than using a frame-grabber technique, and not slower as I mentioned earlier…

June 6, 2010 at 9:35 pm #284859

sea
@sea

Thanks riq for the info. Very useful pointers to the right direction.

You say that drawing the scene twice might by faster. But how can cocos2d talk to two different screens/views at the same time? Maybe I’m missing something fundamental here.

When you previously talked about grabbing the screen, are you referring to CCGrabber? I noticed this class, but not sure what it’s purpose is. I’m assuming it’s some sort of frame grabber (as the name may suggest).

Cheers.

June 6, 2010 at 9:45 pm #284860

riq
Keymaster
@admin

@sea: cocos2d, by default, doesn’t support multiples screens.

You need to modify the CCDirector in order to support it… probably by creating 2 different OpenGL contexts/views. It doesn’t seem to be difficult to do it, but it might require some OpenGL low level knowledge.

What you have to do is:

- update / draw scene using 1 openGL context/view, as it is today. See CCDirector#mainLoop

- draw (no need to call “update” again) everything again in the other context/view.

As I mentioned earlier, this is just an intuition… I think this is the “right” approach, but I might be missing something.

November 5, 2010 at 12:23 am #284861

asinesio
Participant
@asinesio

Sorry to bring up an old thread, but I just got a VGA adapter and am curious about trying this out.

@riq , your technique is assuming that we would draw the same scene twice (which might be worthwhile). However, I think most games would use the external display as an auxiliary screen with a different purpose… since it doesn’t have touch input. For example, it could show the current game’s statistics and other general information.

CCDirector is a singleton, so we can’t attach it to two different views today.

Would it make more sense to target external displays without cocos2d and use RenderTexture instead, as @riq initially suggested, in that case?

November 5, 2010 at 2:42 am #284862

abitofcode
Moderator
@abitofcode

@rwenderlich Which approach did you use for ‘Battle Map for iPad’

November 5, 2010 at 2:48 am #284863

riq
Keymaster
@admin

@asinesio:

Yes, there is an open feature regarding multiple directors:

http://code.google.com/p/cocos2d-iphone/issues/detail?id=974

I still don’t know how to do it without changing the whole director API.

Ideas ?

November 5, 2010 at 1:21 pm #284864

CJ
Moderator
@wiseganesha

@riq that issue addresses the issue of the singleton director but does anyone have an example of rendering to the external screen with the same or an aux OpenGL context?

November 5, 2010 at 4:24 pm #284865

rwenderlich
Participant
@rwenderlich

For Battle Map I simply used CCRenderTexture to get an image of the screen, and put it in the external display in an image view. Like riq said, this is quite expensive but was OK in my situation, since I had it only update the external screen when something significant changed (i.e. at the end of moving a token, rather than continuously animating the token being moved).

From what I understand, the iPad is just too slow to render to both an external screen and local screen continuously and get good FPS, however you could switch the rendering to only the external window (and have a more simple UIView on the device for a joystick or stats like asinesio suggested) and get decent performance. Mike Daley had an example where he did this with raw OpenGL at the VTM conference a while back.

November 5, 2010 at 9:47 pm #284866

Andrew
Participant
@andrew

This is what I use for CoachPad (but it’s just for mirroring the display): It’s cobbled together from all sorts of sources:

TVOutManager.h

//
// TVOutManager.h
// TVOutOS4Test
//
// Created by Rob Terrell (rob@touchcentric.com) on 8/16/10.
// Copyright 2010 TouchCentric LLC. All rights reserved.
//

#import <Foundation/Foundation.h>

@interface TVOutManager : NSObject {

UIWindow* deviceWindow;
UIWindow* tvoutWindow;
NSTimer *updateTimer;
UIImage *image;
UIImageView *mirrorView;
BOOL done;
BOOL tvSafeMode;
CGAffineTransform startingTransform;
}

@property(assign) BOOL tvSafeMode;

+ (TVOutManager *)sharedInstance;

- (void) startTVOut;
- (void) stopTVOut;
- (void) updateTVOut;
- (void) updateLoop;
- (void) screenDidConnectNotification: (NSNotification*) notification;
- (void) screenDidDisconnectNotification: (NSNotification*) notification;
- (void) screenModeDidChangeNotification: (NSNotification*) notification;
- (void) deviceOrientationDidChange: (NSNotification*) notification;
- (UIImage *) screenshotUIImage;
@end

TVOutManager.m

//
// TVOutManager.m
// TVOutOS4Test
//
// Created by Rob Terrell (rob@touchcentric.com) on 8/16/10.
// Copyright 2010 TouchCentric LLC. All rights reserved.
//
// http://www.touchcentric.com/blog/

#import <QuartzCore/QuartzCore.h>
#import "TVOutManager.h"
#import "cocos2d.h"

#define kFPS 15
#define kUseBackgroundThread NO

//
// Warning: once again, we can't use UIGetScreenImage for shipping apps (as of late July 2010)
// however, it gives a better result (shows the status bar, UIKit transitions, better fps) so
// you may want to use it for non-app-store builds (i.e. private demo, trade show build, etc.)
// Just uncomment both lines below.
//
#define USE_UIGETSCREENIMAGE
//CGImageRef UIGetScreenImage();
//

@implementation TVOutManager

@synthesize tvSafeMode;

+ (TVOutManager *)sharedInstance
{
static TVOutManager *sharedInstance;

@synchronized(self)
{
if (!sharedInstance)
sharedInstance = [[TVOutManager alloc] init];
return sharedInstance;
}
}

- (id) init
{
self = [super init];
if (self) {
// can't imagine why, but just in case
[[NSNotificationCenter defaultCenter] removeObserver: self];

// catch screen-related notifications
[[NSNotificationCenter defaultCenter] addObserver: self selector: @selector(screenDidConnectNotification:) name: UIScreenDidConnectNotification object: nil];
[[NSNotificationCenter defaultCenter] addObserver: self selector: @selector(screenDidDisconnectNotification:) name: UIScreenDidDisconnectNotification object: nil];
[[NSNotificationCenter defaultCenter] addObserver: self selector: @selector(screenModeDidChangeNotification:) name: UIScreenModeDidChangeNotification object: nil];

// catch orientation notifications
[[UIDevice currentDevice] beginGeneratingDeviceOrientationNotifications];
[[NSNotificationCenter defaultCenter] addObserver: self selector: @selector(deviceOrientationDidChange:) name: UIDeviceOrientationDidChangeNotification object: nil];
}
return self;
}

-(void) dealloc
{
[[NSNotificationCenter defaultCenter] removeObserver: self];
[super dealloc];
}

-(void) setTvSafeMode:(BOOL) val
{
if (tvoutWindow) {
if (tvSafeMode == YES && val == NO) {
[UIView beginAnimations:@"zoomIn" context: nil];
tvoutWindow.transform = CGAffineTransformScale(tvoutWindow.transform, 1.25, 1.25);
[UIView commitAnimations];
[tvoutWindow setNeedsDisplay];
}
else if (tvSafeMode == NO && val == YES) {
[UIView beginAnimations:@"zoomOut" context: nil];
tvoutWindow.transform = CGAffineTransformScale(tvoutWindow.transform, .8, .8);
[UIView commitAnimations];
[tvoutWindow setNeedsDisplay];
}
}
tvSafeMode = val;
}

- (void) startTVOut
{
// you need to have a main window already open when you call start
if ([[UIApplication sharedApplication] keyWindow] == nil) return;

NSArray* screens = [UIScreen screens];
if ([screens count] <= 1) {
NSLog(@"TVOutManager: startTVOut failed (no external screens detected)");
return;
}

if (tvoutWindow) {
// tvoutWindow already exists, so this is a re-connected cable, or a mode chane
[tvoutWindow release], tvoutWindow = nil;
}

if (!tvoutWindow) {
deviceWindow = [[UIApplication sharedApplication] keyWindow];

CGSize max;
max.width = 0;
max.height = 0;
UIScreenMode *maxScreenMode = nil;
UIScreen *external = [[UIScreen screens] objectAtIndex: 1];
for(int i = 0; i < [[external availableModes] count]; i++)
{
UIScreenMode *current = [[[[UIScreen screens] objectAtIndex:1] availableModes] objectAtIndex: i];
if (current.size.width > max.width)
{
max = current.size;
maxScreenMode = current;
}
}
external.currentMode = maxScreenMode;

tvoutWindow = [[UIWindow alloc] initWithFrame: CGRectMake(0,0, max.width, max.height)];
tvoutWindow.userInteractionEnabled = NO;
tvoutWindow.screen = external;

// size the mirrorView to expand to fit the external screen
CGRect mirrorRect = [[UIScreen mainScreen] bounds];
CGFloat horiz = max.width / CGRectGetWidth(mirrorRect);
CGFloat vert = max.height / CGRectGetHeight(mirrorRect);
CGFloat bigScale = horiz < vert ? horiz : vert;
mirrorRect = CGRectMake(mirrorRect.origin.x, mirrorRect.origin.y, mirrorRect.size.width * bigScale, mirrorRect.size.height * bigScale);

mirrorView = [[UIImageView alloc] initWithFrame: mirrorRect];
mirrorView.center = tvoutWindow.center;

// TV safe area -- scale the window by 20% -- for composite / component, not needed for VGA output
if (tvSafeMode) tvoutWindow.transform = CGAffineTransformScale(tvoutWindow.transform, .8, .8);
[tvoutWindow addSubview: mirrorView];
[mirrorView release];
[tvoutWindow makeKeyAndVisible];
tvoutWindow.hidden = NO;
tvoutWindow.backgroundColor = [UIColor darkGrayColor];

// orient the view properly
if ([UIDevice currentDevice].orientation == UIDeviceOrientationLandscapeLeft) {
mirrorView.transform = CGAffineTransformRotate(CGAffineTransformIdentity, M_PI * 1.5);
} else if ([UIDevice currentDevice].orientation == UIDeviceOrientationLandscapeRight) {
mirrorView.transform = CGAffineTransformRotate(CGAffineTransformIdentity, M_PI * -1.5);
}
startingTransform = mirrorView.transform;

[deviceWindow makeKeyAndVisible];

[self updateTVOut];

if (kUseBackgroundThread) [NSThread detachNewThreadSelector:@selector(updateLoop) toTarget:self withObject:nil];
else {
updateTimer = [NSTimer scheduledTimerWithTimeInterval: (1.0/kFPS) target: self selector: @selector(updateTVOut) userInfo: nil repeats: YES];
[updateTimer retain];
}

}
}

- (void) stopTVOut;
{
done = YES;
if (updateTimer) {
[updateTimer invalidate];
[updateTimer release], updateTimer = nil;
}
if (tvoutWindow) {
[tvoutWindow release], tvoutWindow = nil;
mirrorView = nil;
}
}

- (void) updateTVOut;
{
UIImage *img=[self screenshotUIImage];
mirrorView.image=img;
}

- (UIImage*) screenshotUIImage
{
CGSize displaySize = [[CCDirector sharedDirector] winSize];
CGSize winSize = [[CCDirector sharedDirector] winSize];
//Create buffer for pixels
GLuint bufferLength = displaySize.width * displaySize.height * 4;
GLubyte* buffer = (GLubyte*)malloc(bufferLength);

//Read Pixels from OpenGL
glReadPixels(0, 0, displaySize.width, displaySize.height, GL_RGBA, GL_UNSIGNED_BYTE, buffer);
//Make data provider with data.
CGDataProviderRef provider = CGDataProviderCreateWithData(NULL, buffer, bufferLength, NULL);

//Configure image
int bitsPerComponent = 8;
int bitsPerPixel = 32;
int bytesPerRow = 4 * displaySize.width;
CGColorSpaceRef colorSpaceRef = CGColorSpaceCreateDeviceRGB();
CGBitmapInfo bitmapInfo = kCGBitmapByteOrderDefault;
CGColorRenderingIntent renderingIntent = kCGRenderingIntentDefault;
CGImageRef iref = CGImageCreate(displaySize.width, displaySize.height, bitsPerComponent, bitsPerPixel, bytesPerRow, colorSpaceRef, bitmapInfo, provider, NULL, NO, renderingIntent);

uint32_t* pixels = (uint32_t*)malloc(bufferLength);
CGContextRef context = CGBitmapContextCreate(pixels, winSize.width, winSize.height, 8, winSize.width * 4, CGImageGetColorSpace(iref), kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big);

CGContextTranslateCTM(context, 0, displaySize.height);
CGContextScaleCTM(context, 1.0f, -1.0f);
int deviceOrientation_;
deviceOrientation_=[[CCDirector sharedDirector] deviceOrientation];
switch (deviceOrientation_)
{
case CCDeviceOrientationPortrait: break;
case CCDeviceOrientationPortraitUpsideDown:
CGContextRotateCTM(context, CC_DEGREES_TO_RADIANS(180));
CGContextTranslateCTM(context, -displaySize.width, -displaySize.height);
break;
case CCDeviceOrientationLandscapeLeft:
CGContextRotateCTM(context, CC_DEGREES_TO_RADIANS(-90));
CGContextTranslateCTM(context, -displaySize.height, 0);
break;
case CCDeviceOrientationLandscapeRight:
CGContextRotateCTM(context, CC_DEGREES_TO_RADIANS(90));
CGContextTranslateCTM(context, displaySize.height-displaySize.width, -displaySize.height);
break;
}

CGContextDrawImage(context, CGRectMake(0.0f, 0.0f, displaySize.width, displaySize.height), iref);
CGImageRef imageRef = CGBitmapContextCreateImage(context);
UIImage *outputImage = [[[UIImage alloc] initWithCGImage:imageRef] autorelease];

//Dealloc
CGImageRelease(imageRef);
CGDataProviderRelease(provider);
CGImageRelease(iref);
CGColorSpaceRelease(colorSpaceRef);
CGContextRelease(context);
free(buffer);
free(pixels);

return outputImage;
}

- (void)updateLoop;
{
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
done = NO;

while ( ! done )
{
[self performSelectorOnMainThread:@selector(updateTVOut) withObject:nil waitUntilDone:NO];
[NSThread sleepForTimeInterval: (1.0/kFPS) ];
}
[pool release];
}

-(void) screenDidConnectNotification: (NSNotification*) notification
{
NSLog(@"Screen connected: %@", [notification object]);
[self startTVOut];
}

-(void) screenDidDisconnectNotification: (NSNotification*) notification
{
NSLog(@"Screen disconnected: %@", [notification object]);
[self stopTVOut];
}

-(void) screenModeDidChangeNotification: (NSNotification*) notification
{
NSLog(@"Screen mode changed: %@", [notification object]);
[self startTVOut];
}

-(void) deviceOrientationDidChange: (NSNotification*) notification
{
if (mirrorView == nil || done == YES) return;
if ([UIDevice currentDevice].orientation == UIDeviceOrientationLandscapeLeft) {
[UIView beginAnimations:@"turnLeft" context:nil];
mirrorView.transform = CGAffineTransformRotate(CGAffineTransformIdentity, M_PI * 1.5);
[UIView commitAnimations];
} else if ([UIDevice currentDevice].orientation == UIDeviceOrientationLandscapeRight) {
[UIView beginAnimations:@"turnRight" context:nil];
mirrorView.transform = CGAffineTransformRotate(CGAffineTransformIdentity, M_PI * -1.5);
[UIView commitAnimations];
} else {
[UIView beginAnimations:@"turnUp" context:nil];
mirrorView.transform = CGAffineTransformIdentity;
[UIView commitAnimations];
}
}

@end

To start it up:

[[TVOutManager sharedInstance] startTVOut];

To stop it:

[[TVOutManager sharedInstance] stopTVOut];

Works for me, but I’m not too worried about frame rates.

Cheers,

Andrew.

March 26, 2011 at 8:16 am #284867

SharkDodge
@sharkdodge

Sorry to dig up a buried thread. The code above works, but only if I use

#define USE_UIGETSCREENIMAGE    YES

Which is not allowed for distribution in the app store. Is there another way of doing this?

April 22, 2011 at 7:48 pm #284868

davelassanske
Participant
@davelassanske

I am also very interested in this

May 19, 2011 at 1:00 am #284869

bryce
Participant
@bryce

Digging up an old thread again… I tried the method Andrew posted and I get video out, however it’s horizontally smushed. Maybe since my game it landscape it’s trying to squish it into portrait?

Either way, I’m not sure it will work because the performance isn’t very good with TV Out enabled and I’m trying to create a trailer. It’s very hard to get the controls to work properly so I’m hoping to get video off an actual device.

June 17, 2011 at 6:39 pm #284870

drewconner
Participant
@drewconner

Was wondering if there had been any progress on this. With the new features coming soon in iOS5, it would be really cool to have multiple display support. Especially with different content on each display.

June 17, 2011 at 11:35 pm #284871

Stepan Generalov
Moderator
@ipsi

Mmm CCTVOutManager…

@Andrew could you license it under MIT, please? This is needed to add it to the extensions repo.

Also if you have any recent changes of it – everyone will be happy to see them.

Thanks!

Moved to the extensions sub forum.

June 30, 2011 at 4:12 am #284872

nickthedude
@nickthedude

any movement on this? would be a great feature of ccdirector to support 2 screens, what if we created a second director class to handle 2 screens? then we wouldn’t screw with all the legacy stuff with ccdirector, how bout ccDualScreenDirector?

July 5, 2011 at 9:41 am #284873

seti
Participant
@seti

Hello, I’m trying to use external sceen too. But what I see is CCDirector has a limit of window size (1024×768)

Where should I look to solve this? I need to support full HD resolution on a TV.

Viewing 25 posts - 1 through 25 (of 27 total)

You must be logged in to reply to this topic.