What device do you use for performance testing?

cocos2d for iPhone » Cafe

What device do you use for performance testing?

(16 posts) (9 voices)

Started 1 year ago by Questor
Latest reply from Shogan

Tags:

Questor
Member

Greetings,

Right now I am using a 3g ipod touch (faster CPU version) for testing my app. However, in order to truly gauge the performance that most users would experience, I need to get a slower device.

So a few questions have been jumping around in my head (with the first one being the most important):
1) Would a 2g ipod Touch be sufficient for performance testing, or should I really be looking to get a 1g ipod (or even a 1g iphone) for testing?
2) How big is the performance difference is between 1g and 2g (or even ipod vs iphone)?
3) Assuming there is a large difference between 1g and 2g, does it make sense to limit an apps features to make it run well on a 1g device?
4) On the other hand, can/should a developer create an app for a minimum generation of device? And can that minimum be specified in the app-store listing? (or does Apple require that all apps be available for all device generations?)

Any help/insight would be appreciated.

Posted 1 year ago #
hactar
Member

I have an iPhone 1G, iPhone 3G, and iPod Touch 2G. In my experience the iPhone 1G and iPhone 3G have a similar performance, they are the slowest devices I have. The iPod Touch 2G is faster than my iPhone 3G. The performance difference being clearly noticeable, especially when loading large images. So loading my gameScene takes around 5 seconds on the iPod touch, while it takes 10 on the iPhone 3G. However this major difference is because of the iPhone 3G having to quit safari due to memory warnings, quit mail due to memory warnings, etc, the iPod touch is only a testing a device and as such does not have these other apps running.

In the end, decide what the lowest device is you want to support. For me its the iPhone 3G.

Posted 1 year ago #
Grizzly
Member

I haven't done any performance tests between the devices, but some time ago I read that the iPod Touch CPU is always clocked a little bit faster then the same gen iPhone. The reason for this being that the iPod doesn't have a cellular antennae that's constantly draining the battery, so they're able to provide up the CPU and still keep similar battery life. I don't remember where I read this, but it does make sense to me at least.

Posted 1 year ago #
Questor
Member

Thanks for the replies and info.

Hactar,

Beyond loading times, is the iphone 1g/3g similar in performance to the ipod touch 2g? I mean, while I realize that loading times are important, there is so only so much optimization that can be done in that regard, so I am wondering if once an app is loaded, is the performance of an ipod touch 2g indicative of what the performance would be on the iphone 1g/3g?

Posted 1 year ago #
robodude666
Member

The devices are pretty different, though 1st/2nd gen iPhone share a lot in common.

The original iPhone and iPhone 3G both run on a ~400MHz processor. The iPhone 3GS runs at 600MHz. The iPhone 4 will use the Apple A4, which is used in the iPad and is rumored to clock at about 1GHz (1000MHz).
The 1st gen iPod Touch shares the same processor as both the original iPhone and iPhone 3G at ~400MHz. The 2nd gen is slightly faster at 533MHz, and the 3rd gen iPod Touch shares the same processor as the iPhone 3GS running at 600MHz.

As for onboard RAM, both the original iPhone and 3G had 128MB of DRAM. The 3GS has 256MB, and iPhone 4 will have 512MB.
The iPod Touch 1st and 2nd gen have 128MB whilst the 3rd has 256MB.

For graphics power the original iPhone and 3G have the same graphics processor, while the 3GS' is quite a step up.
The iPod Touch 1st and 2nd gen have the same graphics processor as the original iPhone and 3G. The 3rd gen iPod Touch has the same graphics as the 3GS.

As you can see the original iPhone and 3G are on par with one another. The 3G is slightly faster, supposedly, but not much more. The 3GS is leaps and bounds better than either of the two. If you have a really sprite or post-processing heavy app you might want to limit it to newer devices.

The iPod Touch 1st gen is identical to the first two iPhones. The 2nd gen is slightly faster, though not much. The 3rd gen is on par with the 3rd gen iPhone.

If you want good coverage in testing, use your 3rd gen iPod Touch and possibly get a 1st/2nd gen iPod Touch. It will give you an idea of how the 1st and 2nd gen iPhones will respond to your app. Note that the 2nd gen iPod Touch is slightly faster, which will make your app run smoother than on the original iPhone/3G. So if you're getting like ~30FPS on the 2nd gen iPod Touch you might not want to let original iPhones be supported as they'd possibly run it slower.

-robodude666

Posted 1 year ago #
Questor
Member

robodude666,

Thanks for that detailed summary. That really cleared up the (slight) performance difference between the iPhone and the ipod Touch 2g.

Also, that is a great tip regarding using the framerate to guestimate iphone performance. I will definitely remember to check for that as well.

Posted 1 year ago #
robodude666
Member

As I don't have an actual iPhone Developer Program account I haven't actually tested anything on a device. I've been testing all of my stuff via the Simulator, which while is a fairly good representation of what the device would perform like it may lie sometimes.

So... any hardware testing is good.

If you're getting 60FPS on a 3rd gen iPod Touch and don't have anything that could possibly hurt the FPS (say, 100 particle effects going off at once or say.. 500 bodies in Box2D colliding) you should be good on an older device. If you are, I'm not sure how a 1st/2nd gen iPhone or iPod Touch would respond to it.

Clearly, if you're using a ton of resources and getting ~30FPS on a 3rd gen iPod you're most likely going to get unplayable results on a 1st/2nd gen iPhone/iPod Touch. In some cases I've noticed that it takes the simulator about ~7 particle effects to drop from 60FPS down to like 45.. Then at about 25 it drops to 30FPS and then it requires ~100 to drop to 15FPS.

I think it would be a VERY good idea if a performance test suite could be put together that stresses the hardware and simulates what you could find in a real game (lots of physics, lots of particles, lots of math, lots of animation, sprites, etc.) and have it distributed with Cocos2D so that people can run the tests themselves and report back what they get with their hardware. Then compile a wikipedia article with performance results and some statistical analysis to smooth out any variation between same generations.

Posted 1 year ago #
riq
Key Master

and have it distributed with Cocos2D so that people can run the tests themselves

cocos2d already includes a few perf-test. See Release Notes, see the test subdirectory. Feel free to contribute more perf-test.

Posted 1 year ago #
Shogan
Member

A bit of thread revival here, but I am currently looking at performance of my game (in development). I am testing primarily on my iPhone 3G. I have read up the specs, and as mentioned above, it has a 400mhz Samsung ARM 11 processor which appears to be the exact same as the original iPhone. Same memory for both of the above too (128MB RAM). So I am assuming that the performance I get on my 3G is pretty much the worst it can be. For now, I get 45-60 fps on my iPhone 3G, and this drops down to around 30fps at times when a lot of objects are on screen, especially when I have my full backdrop (512,512 texture) along with a couple of other sprites at around 64x64 and a larger 512x256 sprite that flies by every now and then...

Do you guys think this is acceptable performance if I would like to support iPod touch and iPhone 3G / original iPhones for my game? I have tried on a 2nd gen iPod touch, which from what I read has a 533MHz processor and it runs nice and smoothly (almost constant 60fps). The iPhone 4 doesn't even hiccup of course!

What are your thoughts? I think it seems to be playable enough not to complain on the 3G...

Posted 1 year ago #
itlgames
Moderator

I also test on a iPod Touch 1st generation and I found it slightly slow than the iPhone 3G

Posted 1 year ago #
abitofcode
Moderator

I've been working on some drawing code recently, a variation of this a cross between the ribbon code and http://www.flashandmath.com/advanced/smoothdraw/index.html .When its running at 60fps it feels very smooth but after adding some UIKit stuff it dropped to 45fps and felt painful in comparison. However when i let someone play on it at 45fps who hadn't experienced 60fps they thought it was great. So I guess a lot of the experience is subjective.

If you've got any doubt go with you're gut feeling cos once you're on the app store the reviewers won't pull any punches and you'll kick yourself if you end up agreeing with them.

Try and remove as many of the things that would annoy you as possible such as slow loading times, unskippable cut scenes and buttons in hard to reach places etc. Then drop the quality on the textures and the size/number of sprites showing and see if the game plays better. If the difference in gameplay isn't noticeable then don't worry. Listen to beta testers, they give you the feedback directly rather than splashing it across the app store. :)

Posted 1 year ago #
Shogan
Member

Thanks for your feedback on testing @itlgames, i'll keep that in mind.

@abitofcode, looks interesting! What you are saying also kind of makes sense. At the moment, my only two beta testers are my wife and one of her friends :D I will definitely pass it on to a few more friends when it is almost ready for second opinions and after they are done ask them if they noticed any performance issues. My loading times aren't bad at all so far - just using one 1024x512 spritesheet and the backdrops are separate. Guess thats why. But I have just noticed that after doing many scene changes my game starts slowing down to a crawl... :/ Going to have to try and find out what is wrong. Seems like the scenes don't unload themselves completely from memory (or perhaps something I am declaring inside them). d_d

Posted 1 year ago #
Dragyn
Member

@Shogan: Anything you don't release in a scene will keep the entire scene from unloading. I had some problems for a while with that, and the problem was traced to a single NSString I missed a release on. Using Instruments to check for leaked objects will be your friend... Good luck!

Posted 1 year ago #
Shogan
Member

Hey @Dragyn, awesome thanks for the advice :) I have actually just found a bunch of layers that aren't being dealloced in a few different scenes. Busy sorting those out. I am going to take your suggestion and get cracking on the instrument testing now :) I have also put CCLOGs in all dealloc methods now to check that they are getting called and it appears so thus far. Holding thumbs that I don't get that massive slow down again after testing every single level a few times over in the same app session.

Posted 1 year ago #
abitofcode
Moderator

There are some useful memory management bits in the following post;

http://www.cocos2d-iphone.org/forum/topic/8594

Posted 1 year ago #
Shogan
Member
Thanks @abitofcode :) The last post in that thread is of particular interest to me. I have set up logs in all my dealloc methods of each scene - also of each layer too, and they all get called. So I don't think there is an issue there. But, I will be adding in that hack to show the memory size next to the FPS counter and I'll see if the slow downs start appearing every time at the same amount of RAM usage (which I am guessing it does).
```
[[CCDirector sharedDirector] purgeCachedData];
```
That may be useful in purging memory between scenes. I'll have to do some more reading on exactly what cache that purges. As you can see I have a lot to learn. Especially regarding memory/resource management!
Posted 1 year ago #

RSS feed for this topic

Reply

You must log in to post.