Public Beta Testing Advice?
by Chris Jorgensen · in General Discussion · 09/17/2007 (6:39 pm) · 10 replies
I'm about finished with preparing a public beta for my current game. I've had a few alpha testers take a look at a pre-beta version and give me feedback. It worked great. But now I want to get public feedback and make sure that I don't shoot myself in the foot in the process. Specifically, I'm curious about:
1- How many features go into the beta? I want to give a good feel for the game, but obviously not release a complete enough version of the game that it virtually competes with the version I plan to sell.
Final: 27 ships, 8 levels
Beta (planned): 9 ships, 1 level
2- There's currently no expiration on the beta. Is this dumb?
3- What's the best way to get info out of people? I'd hate to have only 10% of my downloaders give me feedback. And if I don't get feedback, then what's the point of the beta?
Your thoughts and advice would be most appreciated. :)
1- How many features go into the beta? I want to give a good feel for the game, but obviously not release a complete enough version of the game that it virtually competes with the version I plan to sell.
Final: 27 ships, 8 levels
Beta (planned): 9 ships, 1 level
2- There's currently no expiration on the beta. Is this dumb?
3- What's the best way to get info out of people? I'd hate to have only 10% of my downloaders give me feedback. And if I don't get feedback, then what's the point of the beta?
Your thoughts and advice would be most appreciated. :)
About the author
Owner of Cascadia Games LLC
#2
09/17/2007 (7:19 pm)
I would suggest however that you keep in mind that you could have a support issue regarding people using your beta test release post-go live, and having issues categorizing what they are using. This is one of the reasons some beta programs (if not demo, but they are two completely different things) have a calendar date based expiration. 
#3
Beta test should be all the function of the product, it SHOULD only run a set number of times, and it SHOULD force the BETA user to register and leave comments, call them the "BETA USERS" club. A BETA is not the full product but a test of the products functions. The BETA should not in any way TRY to DEMO the products capabilitys.
09/17/2007 (7:35 pm)
Before the edit the questions were about DEMO using demo/beta interchangeably; after the edit its about BETA, and most my comments dont relate to a beta test. (thats what i get for taking so long to type a reply)Beta test should be all the function of the product, it SHOULD only run a set number of times, and it SHOULD force the BETA user to register and leave comments, call them the "BETA USERS" club. A BETA is not the full product but a test of the products functions. The BETA should not in any way TRY to DEMO the products capabilitys.
#4
What I have done for beta's, is to setup a form on the web allowing them to sign up for the beta that states what is required to qualify for the beta, what the rules are, and what they are expected to do. You can ask them for info such as their hardware setup, their expertise level, etc. to qualify them. The basic rules are: 1) they can't give the app to anyone outside of the beta, 2) they must complete the questionaire at the end of the beta to qualify for whatever they are going to get in return for testing the software. and 3) notify them that they agree that you can use their responses to the final questionaire for marketing purposes (including minimal personal info such as first name, last initial and city they live in). I generally give them a complete copy of the completed app as payment for participating in the beta.
I put code in the app to stop execution after a certain date (after the beta is ended). I also turn on debug code if appropriate and have the program generate a log file (to be sent in if needed). I have a web page or form in the app that they can fill out to report a bug or send a comment.
During the beta, I will fix bugs and release new versions of the software. You can extend the beta test as needed.
On the final questionaire, you can ask them them for their opinions on the game, the graphics, specific features, etc. and a testimonial or recommendation if they liked the game.
09/17/2007 (7:40 pm)
When I think of a beta test, I think of testing the complete app before release. I do the beta test at the point where I and other (generally) internal people have beat up the app and can't break it, deadlines could change this, of course. The point of the beta test is to let the customer's try it out and beat it up as best they can. I haven't beta tested any games, however. I would suggest that if you don't beta test the whole game (minus final art perhaps), you will probably ship a game with lots of bugs.What I have done for beta's, is to setup a form on the web allowing them to sign up for the beta that states what is required to qualify for the beta, what the rules are, and what they are expected to do. You can ask them for info such as their hardware setup, their expertise level, etc. to qualify them. The basic rules are: 1) they can't give the app to anyone outside of the beta, 2) they must complete the questionaire at the end of the beta to qualify for whatever they are going to get in return for testing the software. and 3) notify them that they agree that you can use their responses to the final questionaire for marketing purposes (including minimal personal info such as first name, last initial and city they live in). I generally give them a complete copy of the completed app as payment for participating in the beta.
I put code in the app to stop execution after a certain date (after the beta is ended). I also turn on debug code if appropriate and have the program generate a log file (to be sent in if needed). I have a web page or form in the app that they can fill out to report a bug or send a comment.
During the beta, I will fix bugs and release new versions of the software. You can extend the beta test as needed.
On the final questionaire, you can ask them them for their opinions on the game, the graphics, specific features, etc. and a testimonial or recommendation if they liked the game.
#5
I think you need to ask yourself the following:
I'm going to release a...
1. demo, or
2. beta of the final product.
These two have different requirements. A demo is used to drive sales so is often restricted in some manner (features, time, etc.). End user feedback here is infrequent.
A beta is used to discover bugs in both technology and gameplay. If you want to prove out your game in its final state then you'll also need to test it as such. There's nothing wrong with doing this in stages -- in fact it is often encouraged -- but keep in mind that your results are limited to the current stage. Adding content can have unanticipated results.
As for feedback, if you're not testing with a controlled group where communication is easier, then you'll want to include feedback as part of the game. You could have a web form open up upon exit or even have a questionnaire from within the game itself that is sent to you. Try to keep your questions focused as large text boxes often receive "Was good." and "Was bad." responses.
Oh, and I'd recommend a big YES on at least a time restriction for your beta releases. Stephen's correct on how these things can come back to haunt you.
The above is just my own opinion based on my observations. I'm sure you'll get a lot of these. :o)
- LightWave Dave
09/17/2007 (7:44 pm)
Greetings!I think you need to ask yourself the following:
I'm going to release a...
1. demo, or
2. beta of the final product.
These two have different requirements. A demo is used to drive sales so is often restricted in some manner (features, time, etc.). End user feedback here is infrequent.
A beta is used to discover bugs in both technology and gameplay. If you want to prove out your game in its final state then you'll also need to test it as such. There's nothing wrong with doing this in stages -- in fact it is often encouraged -- but keep in mind that your results are limited to the current stage. Adding content can have unanticipated results.
As for feedback, if you're not testing with a controlled group where communication is easier, then you'll want to include feedback as part of the game. You could have a web form open up upon exit or even have a questionnaire from within the game itself that is sent to you. Try to keep your questions focused as large text boxes often receive "Was good." and "Was bad." responses.
Oh, and I'd recommend a big YES on at least a time restriction for your beta releases. Stephen's correct on how these things can come back to haunt you.
The above is just my own opinion based on my observations. I'm sure you'll get a lot of these. :o)
- LightWave Dave
#6
> 1- How many features go into the beta?
As others have stated, you need to determine if you are releasing a beta or a demo. A demo is good to have limited functionality. A beta should be fairly close to a final version. That does not mean the beta is feature complete, but it should have the major features completed and feel like a final game. Unimplemented features should be noted and should not cause a large integration issue, making previous beta testing obsolete. For example, maybe certain menu options are not working yet but adding them won't effect the game. On the other hand, not having a resource implemented in an RTS will essentially mean a complete retest of everything once it is implemented.
> Final: 27 ships, 8 levels
> Beta (planned): 9 ships, 1 level
This is a good example. If the beta contains 9 ships, how confident will you be in the game balance for the final release? Game balancing is really tough. That said, you can still get some good feedback looking for obvious issues, but keep in mind, you are not getting a full representation of the final game.
> 3- What's the best way to get info out of people? I'd hate to have only 10% of my downloaders give me
> feedback. And if I don't get feedback, then what's the point of the beta?
You should determine how many people you feel will be best for feedback. Not everyone will respond. I don't know what typical percentage figures are. I like the suggestions from some in regards to throwing up a webpage when the user exits the game. Another option is to e-mail questionairres and get it in people's faces.
One thing to think about is beta testing a game is not the same as playing a game. Many people will want to try out the game but as soon as they find crashes or problems, they may move on to something else. Part of testing is about trying those weird and strange things, testing limits, doing the unusual. While other times, you want people to play as normal. You need to determine entrants level of desire to test.
I just want to point out I like how you are looking at doing a beta round and asking questions. Beta periods are extremely important. This is a good discussion.
09/17/2007 (8:11 pm)
I saw some screenshots of your game just the other day and it looks pretty neat.> 1- How many features go into the beta?
As others have stated, you need to determine if you are releasing a beta or a demo. A demo is good to have limited functionality. A beta should be fairly close to a final version. That does not mean the beta is feature complete, but it should have the major features completed and feel like a final game. Unimplemented features should be noted and should not cause a large integration issue, making previous beta testing obsolete. For example, maybe certain menu options are not working yet but adding them won't effect the game. On the other hand, not having a resource implemented in an RTS will essentially mean a complete retest of everything once it is implemented.
> Final: 27 ships, 8 levels
> Beta (planned): 9 ships, 1 level
This is a good example. If the beta contains 9 ships, how confident will you be in the game balance for the final release? Game balancing is really tough. That said, you can still get some good feedback looking for obvious issues, but keep in mind, you are not getting a full representation of the final game.
> 3- What's the best way to get info out of people? I'd hate to have only 10% of my downloaders give me
> feedback. And if I don't get feedback, then what's the point of the beta?
You should determine how many people you feel will be best for feedback. Not everyone will respond. I don't know what typical percentage figures are. I like the suggestions from some in regards to throwing up a webpage when the user exits the game. Another option is to e-mail questionairres and get it in people's faces.
One thing to think about is beta testing a game is not the same as playing a game. Many people will want to try out the game but as soon as they find crashes or problems, they may move on to something else. Part of testing is about trying those weird and strange things, testing limits, doing the unusual. While other times, you want people to play as normal. You need to determine entrants level of desire to test.
I just want to point out I like how you are looking at doing a beta round and asking questions. Beta periods are extremely important. This is a good discussion.
#7
1- Content: do a round of tests at each big addition of content
2- Expiration: yes
3- Feedback: try to force it
Which leads me to my new question: audience?
My intention was to get a beta out there to those on this site for help in getting the basics tested: interface, controls, responsiveness, strange behaviors, etc... I always figured "feature complete" and "content complete" were different. All the key features are in and ready for testing. All the content (ships/levels) is not, with exception to that which I've intentionally polished for the beta.
At the same time, however, I'd hoped to get the beta into a few hands from the "Star Control community." (Where I've been attracting attention, getting concept feedback, and so forth.) And in that sense it's a demo because I'd expect the comments would be less on testing, and more about game-play, dream features, how it compares to SC, and so forth. I had two alpha testers from that community; one gave awesome feedback, one never responded.
Is releasing a beta in more than one internet "venue" a good idea? Should I limit how many betas go out altogether?
Again my sincere thanks. I'm dedicated to getting this game done, and getting it done right.
09/17/2007 (10:29 pm)
This is good feedback. So: 1- Content: do a round of tests at each big addition of content
2- Expiration: yes
3- Feedback: try to force it
Which leads me to my new question: audience?
My intention was to get a beta out there to those on this site for help in getting the basics tested: interface, controls, responsiveness, strange behaviors, etc... I always figured "feature complete" and "content complete" were different. All the key features are in and ready for testing. All the content (ships/levels) is not, with exception to that which I've intentionally polished for the beta.
At the same time, however, I'd hoped to get the beta into a few hands from the "Star Control community." (Where I've been attracting attention, getting concept feedback, and so forth.) And in that sense it's a demo because I'd expect the comments would be less on testing, and more about game-play, dream features, how it compares to SC, and so forth. I had two alpha testers from that community; one gave awesome feedback, one never responded.
Is releasing a beta in more than one internet "venue" a good idea? Should I limit how many betas go out altogether?
Again my sincere thanks. I'm dedicated to getting this game done, and getting it done right.
#8
As for content, I would use beta art. (or programmers art as it's commonly called) and not the real art. That way, there is something new for your beta testers to see when they buy your final game.
As for your last question, It doesn't matter where your beta testers come from as long as they agree to post feedback. This is another plus with a forum. Your users would have to register on your forum to get the download link after you've added them to the beta testers forum. (keeping beta comments private is a good thing.)
09/18/2007 (12:15 am)
Chris, Do you have a forum on your website? Forums provide a great place for beta testers to post their comments and bug reports.As for content, I would use beta art. (or programmers art as it's commonly called) and not the real art. That way, there is something new for your beta testers to see when they buy your final game.
As for your last question, It doesn't matter where your beta testers come from as long as they agree to post feedback. This is another plus with a forum. Your users would have to register on your forum to get the download link after you've added them to the beta testers forum. (keeping beta comments private is a good thing.)
#9
I used to have a pretty complex site, including a forum. But spammers for sex sites, drugs, etc... quickly pirated the forum. Since then I've decided that it's not worth having one unless you have lots of traffic and a dedicated moderator.
There's not much "beta art" for my project. Some anime characters are missing. And obviously the levels aren't all there. But I started out with art pirated from a defunct other project of mine. So I never had temp ship art. Plus, I think there's value in seeing the "real" art. If I have a green ship and a green background and the players can't see well enough, I need to know that. If I use placeholder art, I lose that feedback.
What I've decided to do, based on ideas posted here, is to go forth with my beta as-is, with the addition of an expiration date (Oct. 7th at the moment). I've created a Yahoo! group dedicated to discussing the beta, and the beta will be uploaded to its file space. Thus, if you want to try out the beta, you have to at a minimum join the discussion group. Hopefully, people who sign up have a serious enough interest to receive some emails regarding it, and maybe be inspired to chat as well. Plus, this allows me some sort of minimal user tracking.
Hopefully I can get that announcement up soon. :)
09/18/2007 (7:39 pm)
Mike:I used to have a pretty complex site, including a forum. But spammers for sex sites, drugs, etc... quickly pirated the forum. Since then I've decided that it's not worth having one unless you have lots of traffic and a dedicated moderator.
There's not much "beta art" for my project. Some anime characters are missing. And obviously the levels aren't all there. But I started out with art pirated from a defunct other project of mine. So I never had temp ship art. Plus, I think there's value in seeing the "real" art. If I have a green ship and a green background and the players can't see well enough, I need to know that. If I use placeholder art, I lose that feedback.
What I've decided to do, based on ideas posted here, is to go forth with my beta as-is, with the addition of an expiration date (Oct. 7th at the moment). I've created a Yahoo! group dedicated to discussing the beta, and the beta will be uploaded to its file space. Thus, if you want to try out the beta, you have to at a minimum join the discussion group. Hopefully, people who sign up have a serious enough interest to receive some emails regarding it, and maybe be inspired to chat as well. Plus, this allows me some sort of minimal user tracking.
Hopefully I can get that announcement up soon. :)
#10
09/26/2007 (9:20 pm)
Well, just as an update on this topic, it appears people are pretty weary of joining a Yahoo group (even though you can set it up to not email you. It seems like a little bit of a catch 22; you want lots of testers, but only testers who are interested in giving feedback. Surprisingly, only about 2 out of the 8 signees left any feedback. Disappointing indeed! I think I might just switch to putting the link out there wherever and see what kind of response I get. :(
Torque 3D Owner Caylo Gypsyblood
The most minimal, but most exciting features. It helps to be able to show what other features your product have, even if they are not 100% in the demo. The best game demo to date (as per number of actual product sold) was the Half Life demo, it took near 1 hour to play the first time, it was a full game in of itself, and it was actually fun to replay.
Most games I actually purchase is because i replay the demo many times, and have FUN playing the demo.
2- There's currently no expiration on the beta. Is this dumb?
I feel having any type of already crippled software expire is kinda dumb. I often download something and fiddle with it for a few minutes, only to come back sometime later to actually try and evaluate the software only to find the demo time expired. My impression of that software is they must not truly want people exploring it very fully, or perhaps they are attempting to hide some of its weak spots.
3- ...I'd hate to have only 10% of my downloaders give me feedback....
Before the demo start, open a web page to your product page, and be sure FEEDBACK is easy to see. Perhaps let it be known that by leaving feedback the demo player may unlock some other demo game functions. Also when the game is over, once more open that product page. Its less 'nagging' to simply open the web page for more product information then make your demo player sit for another 60sec looking at game advertisements before the game will actually close (or click multiple times on different 'full game' screen shots to get out).