Introduction
Austin Roorda
That’s me! I work with the team as the Project Manager to keep things running smoothly!
Karl Lewis
Karl is our resident Systems & Audio designer. He works like a madman to get our game to work & sound amazing!
Emmett Friedrichs
Emmett is also multi-disciplined as a designer, working with both Systems and Level Design. He helps lay out our levels!
Josh Grazda
Josh is the Programmer for our team. He specializes in Gameplay & Systems programming and makes the game work!
Adam Streeter
Quoted as saying how he wanted to create “The most aesthetic capstone game ever created” Adam is our artist. His work makes even Picasso jealous!
You know the drill by now, these guys are all incredible at what they do within their respective disciplines and all work hard to get the game up and running. A lot has happened since the last blog post so I’m going to jump right into this! If you remember from my previous posts, the capstone structure has each team going through stages. This starts at the initial conception of the idea – all the way up to a more fleshed out, featured vertical slice.
Below I have crossed out the completed steps just to make it easier for you to remember where we’re at. The team has just challenged the Third Step this past class period!
Steps to Development:
Step 1: Choosing a Starting Point
Create multiple game concepts, and begin to choose what games you may want to develop in the future. Our team ended up coming up with 50 separate and fairly unique ideas, and boiled that list down to 10.
Step 2: Selecting a Concept
Choose one of those concepts to develop for the rest of the semester and give a presentation to the class explaining the rationale behind it as well as it’s feasibility and viability.Since the last post, the team has successfully challenged the second step and is now looking to challenge the third. We selected Cash Force and have been grinding for the past three weeks to get the game’s core loop implemented into the build.
Step 3: Defining the Core Experience
Shape and define the core experience of the gameplay. This includes refinement in the needed documentation from each discipline as well as iteration on the prototype with game testing results.
This is the current step that the team is on, and has challenged. The team wants to move forward to the next step (outlined below – Step 4).
Future Steps
Here are the future steps that the team is going to have to take in the future. I figured that if you don’t care too much about that you can skip on into how the decision making process went for the second step! Click on the titles of each Step if you’d like to read about the criteria for it.
Fully develop a proof of concept prior to the final presentation. Demonstrate the intent of the concept through a demonstrable prototype, iterated documentation as well as game testing results.
When teams reach this step, it demonstrates a fully realized vision of the initially designed idea. Not many teams make it to this stage, but those that do usually pursue release of some sort.
Now, this blog post is going to be all about QA Testing and the process the team went through in order to improve the game based on user feedback! If that’s not all that interesting to you, don’t worry! There is still good content here that shows how all that testing benefited the game.
Be sure to stick around and read through the post to see how all this testing paid off!
QA / Game Testing!
Over the course of the past three or so weeks it’s been since the last post, the team has completed three sprints (one week each) and added a lot of new features into the game. With these new features, we need feedback to test their usability in game and refine these systems. This is to better fit the intended player experience that we want players to have when playing our game. As the Producer for this project, I act as the QA Liaison where I organize, set up, and facilitate testing sessions. I act as a person for contacts for the testers to field their questions to and assist them with any issues they may come across while testing the game.
Also, if you want to skip this in-depth analysis and go straight to the key feature comparison(s) down at the bottom to see the drastic changes, feel free to do so! The following section below is going to be an in-depth analysis of each of the three sections, and will give great insight into the QA process we undertook to improve our game!
10/19/19 QA Test Session
The intent of this test was to was to gather data in regards to several key features within the game. These features are needed in order to complete our game loop and ensure that players can interact with objects within the game. The main points that we were testing were:
Player Interaction with game objects
Usability of interactions
Feeling of interactions
Usability / Feeling of Shooting + Reloading of weapons
Object Interaction Testing
We wanted to make sure that players were able to interact with objects (such as weapons, magazines, etc) with ease. On the scale shown on the graph (as well as those below), a 1 means that grabbing is difficult and unnatural, while a 6 means that it is easy and natural to preform the action.
Many of the testers had relative ease using the controllers to pick up objects. There were a few reports of bugs with the grabbing system (hands phasing through objects, not holding onto objects, etc) but overall not any major issues, just bugs with the grabbing that the team is aware of and worked on developing fixes for.
Anyone with firearms experience can tell you that using two hands to stabilize the weapons helps in a lot of ways. From reducing recoil, improving accuracy and being safe – we wanted players to have to engage with the weapons in game in a similar way.
By using two hands when grabbing a weapon, players receive reduced recoil and are able to aim down the sights to line up precise shots.
This question ties into the one above, as it requires the object interactions to be functioning as intended. Due to some of the issues with the grabbing system that were mentioned above (hands phasing through objects, not holding when the player is gripping the trigger, etc) aiming with two hands was more difficult that we would have liked. However, this feedback is extremely useful and the testimonials from players helped us to narrow down what issues are occurring when grabbing the weapon(s) with two hands.
We wanted to make sure that players could line up their shots by using the iron sights on the weapons when they aimed. Due to the mentioned issues with grabbing, some players experienced difficulty interacting with the weapons and aiming down the sights.
Shooting and Reloading Questions:
The players were easily able to understand how the shooting worked in the game due to the simplicity of the controls. In order to shoot, a player had to load a magazine into the weapon. Once a magazine is loaded into the weapon, all the player would need to do is pull the front trigger to fire. This worked with either controller so player could shoot ambidextrously (or even dual wield weapons).
All testers were able to understand how to shoot, so that’s good!
Reloading is a key mechanics that players must be comfortable with doing and understand in order to play Cash Force. When a weapon runs out of ammo, a player must manually reload their weapon by removing the empty magazine from the weapon and inserting a new magazine before continuing fire.
All players were able to understand this mechanic and had no trouble with preforming this action in game.
As mentioned above, reloading the weapons requires the players to physically interact with the magazines and load them into their weapon(s) before they can re-engage targets.
While most of the testers had no real issues performing this action, some testers experienced bugs (mags not inserting, not able to grab mags, etc) that impacted their experience. The team was aware of these issues and developed fixes for them.
During this test, the players were able to shoot at dummy enemies that would react to getting shot. We wanted to gauge how easily players could hit their targets from a medium distance and see how well they were hitting their shots.
In this build, the testers had relative ease shooting at the targets in the range that they were placed in. This laid a good foundation for future development, since we now know that players can hit targets if they aim.
For the last question, we wanted to get some direct feedback from testers on how it felt to shoot and hit targets when shooting them. Much of the feedback shows that it felt good to shoot and hit enemies, but that there was some player feedback that was desired in order to make it feel more satisfying.
Several players wanted a one-shot kill headshot mechanic to be added, to reward them for preforming a skill shot. Using this feedback, the team addressed these issues (as well as several bugs) before bringing the game to QA the following week.
10/19/19 QA Summary:
Overall, this QA Test Session helped the team identify key issues with core gameplay systems that need to be ironed out before future testing. The grabbing system needed a slight overhaul with key bug fixes related to how two-handing weapons works, as well as grabbing and interacting with magazines. Once those issues were patched, issues with these systems decreased significantly, which is evident in future QA results shown below. The test was very successful and gave the team a lot of good data to work with.
10/26/19 QA Test Session:
The intent of this test was to build off of the results of the previous week’s QA Test and gather feedback on those systems, as well as test some new features that were implemented. These include the following features:
Weapon Interactability (Handling, Loading, etc)
Van Motion (+ Effects on Player)
Enemy Spawning (Targets on Map Edges that players can shoot)
Object Interaction Questions:
The team wanted to test to see if there was any improvement with how the grabbing was working in relation to two handing the weapon. Unlike the previous test, there were no results in the 0 range, with significantly fewer negative responses.
This makes sense considering the team worked for a sprint to reduce the issues that players faced when grabbing weapons and items with two hands. It appeared that there were still issues that needed to be ironed out, but is overall a better experience for the player.
As was the case with the two handing interaction mentioned above, the team was testing to see if there was any improvement in players abilities to aim down sights with the weapons, and the ease of doing so.
Compared to the prior sprint, players had an easier time aiming down sights and engaging targets. The previous week had the curve towards ease of use being a lot more gradual, while this curve is a more of a bell curve with the majority of users feeling as though it was nearly natural or natural to interact with the weapon(s) in this way.
The last question in regards to object interaction that we asked our testers was in regards to any difficulties that they may have experienced while trying to grab and interact with items or weapons.
While there were a few cases of user error (forgetting to hold grip to grasp objects), there were still a few bugs related to the grabbing. Although it was fewer than the prior sprint – grabbing still needed an overhaul to work properly and without issues.
Shooting and Reloading Questions:
Compared to the prior sprint, it seems that shooting and hitting targets became more difficult. In a sense that is correct, since players are now moving inside the back of the van. The motion of the player inside the van, as well as the stationary nature of the targets made for some skill needed to land hits.
Many players were quickly able to adapt and lead some of their shots. Despite the increase in difficulty, this is where we wanted players to be – no too easy to get hits but not so hard it’s not achievable
For our next question, the team wanted to get direct written feedback as to how the shooting actually felt to engage with. Many of the testers believed that the shooting was in a good place and a lot of fun to use, although there were some issues. Most notably, some players believed that their shots were going above where they were aiming.
This is due to a bug with the recoil that the team is aware of and working on fixing. Other players were wondering about the use of attachments, such as improved grips for weapons and sights to improve their aim.
Despite the outliers, the results for this specific question indicated that most of the testers believed the reloading was fluid and easy to execute.
When reviewing the comments, the issues that were mentioned in regards to reloading were mostly in part to user error (not understanding how to reload) but can easily be rectified with non-intrusive UI and better describing the controls to the player when they start.
Weapon recoil was a new addition into the build that was tested. The team was looking towards making the recoil noticeable, but not obtrusive to deal with.
It seems like based off of these responses that the recoil was / is in a good place as of right now, even if the vertical recoil wasn’t at that point working as intended. Only the horizontal recoil is working currently, but has still yielded good results.
Scenario and Van Related Questions:
Motion sickness is something that all VR developers have to take into account when making games. With Cash Force, players are not only playing an FPS, they’re in the back of a moving van engaging multiple targets – which could easily cause motion sickness due to the motion of the van, the movement of the background elements and so on.
Luckily, the movement wasn’t a huge issue due in part to good level design and objects in the peripheral vision of players to help ground them to where they are standing. Most didn’t have any issues with how the motion looked and felt.
Of the players that did have an issue with the motion, they either moved around too much (Leaning / walking around shouldn’t be needed, so changes to the Van’s interior design is needed) or were just generally motion sick when playing VR regardless.
Overall that’s not bad at all considering the concerns that the team had initially for the concept. Considering that there was little to no motion sickness and that players were easily able to look out of the van, development can move forward on level design with no major concerns.
The speed that the van the player is moving was really important to determine. We wanted to see how players felt about our default movement speed (around 30mph). Using this scale, we wanted to have results land in the middle, making for a nice even bell-curve.
These results are nearly ideal, since players were able to recognize their targets and shoot them as they drove by.
The length of the level is important to gauge as well, as the final gameplay demo for Mid-Mortems is only 15 minutes. We wanted to see how players felt about the length of the current level (around 2 minutes). Many players felt as though the level could have been a bit longer.
We plan on extending the level and giving the player more things to shoot at come our next test. With more things to do and enemies spawning in, the events taking place should hold the players attention longer, as this build only had static targets to shoot at.
The final question was asked our testers was in regards to bugs when the Van was moving. Luckily, there were only two bugs, both of which the team already knew about and have fixes for incoming.
10/26/19 QA Summary:
Overall, this QA session gave the team a lot of extremely valuable insight into the development of our systems and mechanics. We’re going to use this feedback to work on some areas of the game, such as the recoil or the levels pacing by making changes that directly address these issues and concerns.
Compared to the prior week’s QA test session, the results from this session are more favorable and positive due to the increase in attention to higher profile issues such as the grabbing bug. Although it’s not fixed, it was still improved from the first week to this one. This sprint’s QA results laid a positive groundwork for the next sprint, where the team tested new features and Quality of Life changes to update (and hopefully finally kill) some of the bugs we’ve been dealing with for the past month of development.
11/2/19 QA Test Session:
This QA Session had the team testing some new systems and the fixes for some of the previously mentioned bugs. These include the following features:
Points / Scoring System(s)
New Weapon Reload Cues
Damage Models
Object Interaction Questions:
After three weeks of tweaking and slight changes and modifications to the grabbing systems – the results are finally more consistently curved / concentrated at the positive end. This came after lots of small and incremental changes to the grabbing code. However, there is still a few issues with it and some bugs won’t be rectified without a complete overhaul of the grabbing system.
Regardless, a bell curve that is heavily focused towards the positive end are results that we have been working towards, and are nearly ideal.
We also wanted to test the 2 handed grabbing again to see how easily players could interact with their weapons. This was due to the issues some players had had during previous tests.
Similar to the last QA session, players were easily able to grab their weapon with two hands with a single exception. Players responses proves that the work done to refine the grabbing system for the weapons has done its job.
Compared to the other QA Tests that were done in the past, the responses are leaning a lot more towards the positive end of the scale.
Even with the addition of recoil, players felt as though it was still easy to aim at their targets when aiming with two hands. This is vastly improved feedback compared to prior tests, more than likely due to the grabbing issues being mostly rectified.
Vocal feedback from testers let us know that the aiming felt smooth and easy to control, with several testimonials (show below) to back those claims up.
The last question that we asked was in regards to difficulty / bugs related to grabbing or using two hands at once. Outside of some minor glitches that occurred due to the sensors not reading the locations of the hands, the only major bug that appeared was a known grabbing glitch, where sometimes the player’s hand wouldn’t grab the grip-point on the gun if they tried several times in rapid succession. The team has been working on a fix for a few weeks, and should be able to fix this by the end of the semester.
Object Interaction Questions
Since the last QA Test, the team has implemented a rudimentary and iterable scoring system to count a players score as they play the game. We also added bones to the rigs that allow for one hit headshots that can instant-kill an enemy.
We wanted to test the functionality and ease of getting headshots. Luckily, every player was able to aim and shoot the targets in the head this test. Nice shooting from our testers!
When a player executed a headshot, text exclaiming ‘Headshot!’ would pop up over the target accompanied by the sound of an old-fashioned mechanical cash register opening.
The team wanted to test how satisfying the pop-ups were to players, with all players finding the feedback that the pop-ups gave to be satisfying, useful or informative.
As mentioned above, when a player shot an enemy – numbers displaying the damage would appear above the enemy. These numbers would be directly added into the players score count. Rifle Shots dealt 42 points of damage, SMG shots dealt 25 points and Headshots gave 200 points. It also displayed the ‘Headshot!’ text above the enemy indicating the precise shot.
Almost all of the players were able to see these damage numbers and interpret what they meant, with three cases of players not being able to. As is, the team believed that the damage numbers are in a good space and do not need to be modified further.
For players, points were earned by damaging the enemy targets that were in the range with them. As mentioned above, The SMG deal 25 damage per shot (4 shots to kill), while the AR dealt 42 damage per shot (3 shots to kill). Headshots were an instant kill, dealing 200 damage.
Damage directly correlated with points, with the damage value being banked into a players score when dealt damage to enemies.
We wanted to gauge how satisfying it felt to players to earn points as they shot enemies. Having the numbers fly off of enemies as they accumulated damage seemed to appeal to our play testers, as all responses were either somewhat or extremely in favor of the current system that we have in place.
The team plans to continue to iterate on this system in the future, but we believe that this is a good starting point for this system.
We wanted to see how receptive players were to potentially being able to purchase upgrades. These would include upgrades to their weapons with new modifications, body armor, van upgrades a power ups. All players surveyed were receptive to this idea and would like to see the team develop some of these ideas.
This is good to hear, since the team was already considering developing a system like this in order to keep gameplay fresh and to encourage players to earn more and spend more cash.
We ended up asking the testers for some ideas for upgrades that would fit the theme of the game (1970’s). Many of the concepts that were shared were already ideas that the team had come up with. This is good, as the team will be able to implement (if this feature is developed in the future) these new items and upgrades into the game.
This will make it so players are able to use their cash for upgrades and bonuses that reward them for playing well and help to enhance their gameplay experience.
Thankfully no testers encountered any bugs with the scoring system. This is good to hear, since the team won’t have to fix anything before they iterate upon it further. The team plans to expand on how players earn point and how those points affect gameplay later down the road (cash earned, other bonuses, etc).
Design is working on laying out how all of this will work for future sprints / future implementation.
Shooting and Reloading Questions:
Much like the last sprint, all testers were able to figure our and understand how to fire their weapon. Arguably, the shooting is the most important feature within the game since it’s the main way that players interact with the game world and engage enemies. Since all players were able to understand how the guns were fired, it shows the design is straightforward enough for players to be able to figure out.
Since reloading is such a critical function that players need to understand in order to play the game, it was important to gauge how well players understood this feature. Especially with the new bolt pulling mechanic, where players have to pull a charging handle on the weapon back to cock the weapon before it can fire.
With almost entirely positive results, we’re secure in players ability to reload the weapons. The intuitive UI on the weapons telling players what to do helped a lot with this.
Mentioned briefly in the section above, a new feature that was added since the last QA test was the addition of blue outlines near the bolt of the weapon. Below that outline, a blue arrow indicates the pull direction to the player. When a magazine is loaded (indicated by a blue outline), the bolt lights up with a blue arrow pointing backwards below it.
This indicates to the player which way to pull the bolt before they can shoot. All players understood this and had no trouble performing this action during gameplay.
Although the players were able to hit their targets with relative ease, there were still some smaller bugs and glitches that were recorded. Many of them related to the grabbing system (which if approved to move forward, will be rebuilt after Mid-Mortems). It’s also important to note that the Van Motion Scene wasn’t tested this sprint – meaning that all targets (and the player) were stationary. This could impact the difficulty of landing hits on the targets.
The team is aware of the issues surrounding grabbing and has a plan in place to fix these issues once the grabbing system is rebuilt. There is currently not enough time left in this semester to fix the grabbing system, but due to the low-impact of the bugs they are able to be mostly ignored until the next semester.
Making sure that the manually reloading feels good and smooth to players is important. With the recent updates to the grabbing system it seems like many of the issues that play testers had last sprint have been rectified. Once the grabbing system is overhauled next semester (if the game goes through) these issues that players are having will be all but eliminated. For now, these results are exactly where we expected them to be.
Lastly, we wanted to know if players experienced any bugs while shooting and reloading the guns. The only major issue that some players faced was that the charging handle (bolt) sometimes would not grab, or would take several tries. This issue is present with many of the handling based mechanics in the game, and the grabbing system needs to be overhauled in order to deal with these issues, as mentioned in prior QA Test analysis from previous weeks.
11/2/19 QA Summary:
Overall, this QA session gave the team a lot of extremely valuable insight into the development of our systems and mechanics. This feedback has helped the team narrow down what features that the team should work on more (scoring system, potential upgrades, grabbing fixes, etc) and the priority of each of these features.
3 Week Overview- High Level Comparison
Now that all of that is out of the way, or even if you skipped past all of that just to get the basic overview of it all without wading through the ocean of test results – I can go over the basic changes from sprint to sprint. This will be the main feature, important stuff that we tested each of the three weeks as well as how it changed over time. Sound good? Let’s get into this:
Two Handed Aiming Down Sights (2h ADS):
Week 1: 10/19/19
At first, many players were encountering bugs or glitches that prohibited or made it difficult to grab the front of the weapon. This made aiming down the sights harder than it needed to be, and let to negative reception / feedback.
Week 2: 10/26/19
With the second round of QA Testing, the team had worked on the grabbing and made significant progress on it making it a lot more user friendly. There were still reported issues, such as items not detaching or not always grabbing, but reports of this were a lot less frequent than the first test session.
Week 3: 11/2/19
After another week and even more tweaking, the team was able to bring it together and make the two-handed grabbing of the weapons function as intended. Although there are still occasional bugs that come up every once and a while, the only way that the team will be able to fix them is to remake the entire grabbing component, which is not within scope for the remainder of the semester.
Overall, between these three weeks there was exceptional progress in making the 2h ADS system function as intended, at least to the best of our abilities given the time and resources left in this semester. The progress shown in the graphs above between each sprint has been really helpful to me as a producer to gauge how much more work needs to be done on the system. Now that the results are all positive and receptive of how it feels, we can focus our attention to other key features needed in order to complete our game loop.
Manual Reloads:
Week 1: 10/19/19
The first sprint of QA had many players feeling somewhat comfortable with the weapon reloading mechanics. It was simple – insert mag and shoot once it’s in the gun. Even with the grabbing issues it was mostly well received, despite needing some attention to patch out the issues players faced. These issues with the grabbing were addressed in future sprints, as explained in the sections to the right.
Week 2: 10/26/19
With the Second Sprint, players were now situated in the back of a moving van. The physics of the van as well as some of the issues that grabbing still presented made for some players to have a lot of issues holding onto items and weapons. Despite this, the majority of players were able to interact with their weapons as intended, and indicated a better overall gameplay experience in regards to this feature.
Week 3: 11/2/19
This most recent Sprint had a new mechanic introduced to the reloads – a charging handle / bolt. This bolt had to be pulled after a magazine was inserted to cock the weapon, enabling it to fire. Some players had some trouble remembering to pull the bolt after inserting a mag, but after a few rounds they became fluid with their reloads. This Sprint took the players out of the van since the level was being worked on, which may also be a factor as to why players had an easier time reloading.
Overall, these three weeks saw a lot of good changes being introduced into the reloading mechanic. At first the changes were just related to the issues with grabbing, but with a new feature being added to the mechanic (bolt pulling to enable fire once mag is inserted) players were still able to interact with the weapons as intended.
Final Thoughts!
Wow… That’s a lot of content to go over! Three weeks worth of QA Tests and each of their analysis doesn’t sound like that much when you do it once week at a time, but when it’s laid out all at once it’s pretty overwhelming! I plan on posting a sprint recap each week from now on, going over QA and the road to Mid-Mortems now that it’s only 22 days away.
These posts should also be easier for me to write, since it’ll be a lot less content to go through, format, and post. Either way, these past few sprints have been extremely productive, with a lot of awesome content making it’s way into the game! Next post will have more pictures and gifs of the build in it, I promise! I’m sure looking at spreadsheets, bar graphs and pie charts wasn’t the most riveting read unless you’re a producer, so for that I apologize.
Here’s to the next sprint and more awesome content making its way into the game!
Talk to you then!
~ Austin