Apple rejects game for the blind – Part 2

I had a call with the Apple Resolution group two weeks ago to discuss why screenshots of an iPhone app are required in apps that are designed for visually impaired people. See my prior blog for the full story.

Their answer was “Those are the rules. People need to see what the app does through screen shots.” In other words, regardless of the fact that the intended audience can’t see the screen shots, Apple will not allow the app to be available on the app store unless the sighted community can see something.

We modified the app to conform to Apple’s requirements by including the main menu screen shot, pictured below. Not very exciting, but sufficient to meet their requirements. They accepted the app yesterday.

ss4-2-960x640

I’m surprised that a company that spends so much effort in making their iPhones and iPads be accessible would have such an inconsiderate rule. If you are visually impaired, please let me know how you feel about Apple’s policy, and ask other people how they feel.

Advertisements

Creating Frogger with Blind Gamers

Over the past few months, I’ve been collaborating with about 10 blind gamers to build a hopping game, slightly similar to the old video game called Frogger. Initially, I was going to build an audio version of Flappy Bird, but we needed to resolve some other audio game issues first before taking on that challenge.

My initial thoughts for Hopper was that you would hop the frog from lily pad to lily pad until you reach the other side of the river. Unlike most audio games, Hopper was going to not be first-person centric. Instead of actually being the frog, you would be listening to what the frog was doing.

Most video games are not first-person centric. In Angry Birds, for example, you launch a bird, and watch where it goes. This is considered the third-person game, since you are an observer of the entire game field. If Angry Birds was a first-person game, you would see things from the perspective of the bird that is flying into the building.

I wanted to make Hopper a third person audio game, and you would play by listening to the game field. The lily pad you are on would make one sound, and the target lily pad – the lily pad you must hop to – would make another sound. To play the game, you press the hop button when you hear the two lily pads – actually the lily pad sound – line up. For example, if the lily pad on which you are sitting is a guitar scale, and the target lily pad is a piano scale, you should press the hop button when the two musical instruments line up. One lily pad moves from left to right, the other from right to left, and they travel at different speeds. The two lily pads might line up in the center of your head, or slight to the right, or slightly to the left. If you press the hop button when they are not lined up, you splash into the river.

The first version compared piano notes, animals sounds, and piano scales – representing lily pads – moving towards each other, and most of the gamers said it was annoying. The second version compared 3 note scales from several musical instruments, and the gamers said they liked the guitar and the piano, but not the other instruments, and it was boring. The third version used music loops (it was about 5 seconds long, and we picked loops that were used in Blindfold Racer), and the gamers said that was much better.

With the music chosen, the next step was to test the gamers ability to line up two sounds at a position other than the center of their head. Most of the gamers could do it with two music loops – the lily pad you are on, and the lily pad you must jump to. When I added a third lily pad – the one you would have to jump to next – it was too confusing.

Staying with just two lily pads, the next test was to control your lily pad by manipulating the phone. We gave them several choices. One option was to tilt the phone left and right to change direction; and the more you tilt, the faster your lily pad moved. The second option was to move you arm (or your body) to the left or right to move the lily pad; the further you moved, the faster your lily pad moved.

Most of the gamers rejected the tilting – they liked moving their arm or rotating their body on a swivel chair instead. However, several of the gamers said that the movements were counter intuitive. When they moved their body to the left, that moved the lily pad further to the left. The gamers were used to first person games. In other words, they thought that if the lily pad sound is to the left, and they move to the left, they should get closer to the sound. What actually happened was that when they moved to the left, the lily pad moved further left (away from them). To get the lily pad to get closer to them, they had to move to the right. They said that just didn’t make sense.

That resulted in two options for the game: make a tutorial for the game, to teach gamers how to succeed at a third-person game, or just build it as a first person game. I asked the gamers which they preferred, and all but one said to switch to first person.

We’re now making those changes, and getting ready for the next set of tests.

Apple rejects game designed for blind people

About a week ago, I submitted a newer version of Blindfold Sudoku Mini to the iTunes App Store. The version contained some bug fixes, and enabled a gamer to purchase additional puzzles after they completed the first set of puzzles that comes with the free app.

Blindfold Sudoku Mini is a 4-by-4 Sudoku game, where you fill out puzzle with the numbers from 1 to 4, so that there are no duplicates in any row or column. Blindfold Sudoku Mini is an audio game- you must picture in your mind what is in each cell on the puzzle board, and the game will speak the contents of each square, or row, or column. It responds to gestures like flicking left and right, up and down, or two and three finger swipes. Unlike most Sudoku games in the App Store, its designed for rapid audio play. It does not use or need voice-over, and it was built in collaboration with several blind gamers. And to make the game even more fun, you can set Blindfold Sudoku Mini to use animal names or animal sounds, instead of the numbers from 1 to 4.

Apple rejected the app because they said “Your marketing screenshots do not sufficiently reflect the app in use, which does not give the user an accurate understanding of what the app does, as required by the App Store Review Guidelines. It would be appropriate to revise your screenshots to demonstrate the app functionality in use.”

Remember – this is an audio game, designed to be used primarily by visually impaired kids and adults. Here’s the screen shot (if you are visually impaired, the screen shot is a picture of a pair of headphones with the wording AUDIO GAME):

ss2-960x640

I’m quite confused what Apple would expect for screen shots if the game is designed for people who are visually impaired, and what’s displayed on the screen is irrelevant to the playing of the game. Are they saying that they will only accept games that are designed for sighted people?

Visually impaired gamers use the voice-over feature of the iPhone to hear the description of apps in the App Store. That’s how they decide if they want to download the app. I would have assumed the Apple reviewer would have read Blindfold Sudoku Mini’s App Store description – here’s what it says:

“Blindfold Sudoku Mini is a fully accessible Sudoku game for both sighted and visually impaired people, designed for rapid audio play. It’s a starter game for Sudoku – a 4-by-4 puzzle game that’s great for kids and adults just getting started with Sudoku.

The Sudoku puzzle board is not visible; instead, you play by listening. Tap on a cell to hear what’s in the cell, swipe left to hear the row, swipe down to hear the column and swipe up to hear the square. Blindfold Sudoku gives you audio cues to tell the difference between the initial puzzle cells and the cells you fill in, and it lets you set multiple candidates for a cell.”

I hope this is just a mistake by the reviewer and not a move by Apple to discriminate against visually impaired gamers.

A very, very, very big iPhone

When the iPhone 6S came out, people were talking about how big the screen was, and how much more you could do on the phone with such a big screen. Larger screens means that games can display a bigger environment, and they would be more fun.

Having designed audio games for the past year or so, these new screens are tiny compared to the image you can maintain in your brain. I stumbled onto this idea when I was building the cryptogram game.

When I design an audio game, I first design the screen, so that I can see how the game operates. For example, with the Sudoku game on a 9×9 board, I create a board on the iPhone screen, and the start programming the game. The position for row 1, column 1, is in the upper left corner of the phone, and the position for row 9, column 9 is in the lower right corner of the phone.

When the audio game is completed, and all the blind gamers that have tested the game say the game is ready for the App Store, I modify the app to make the screen dark. That way sighted gamers playing the game have no advantage over blind gamers.

When a blind gamer plays, she flicks her finger right or left, up or down, to move around within the puzzle. The screen could be one inch by one inch, and the flicking would still work. The screen could be 1 foot by 1 foot, and her flicking would still work. Screen size is completely irrelevant.

When a game designer builds an audio game, the playing field is your brain – not the screen. That gives the game designer an infinite space to layout the game. In a visual game, the screen is a tiny window into a virtual world, and the gamer must move the window to play the game. In an audio game, you are in the virtual reality of the game, and not restricted by a tiny window. Your ears tell you what the virtual world looks like, and you use your body – your hands, your arms, which way you are facing- to move within this world.

If you’ve never tried an audio game before, check out Blindfold Racer and keep your eyes closed. You’ll appreciate how immersed into an alternate reality you can be while playing a game.

Another podcast review of Blindfold Racer

logo

I was recently contacted by Out Of Step. Their organization help people with disabilities create better economic success. It is a free and innovative online marketplace and platform where people with disabilities can make money and promote a business by selling goods and services. And through Out of Step, consumers can now find all sorts of great things, from books to industrial tools and dating services to computer help… all by people who happen to have a disability.

Alvaro Alvarito does a podcast called “Syncopating: Apps for People With Disabilities”, and it is hosted at the TOOST Radio section of Out Of Step’s website. Alvaro did a podcast about Blindfold Racer, and he was nice enough to say “This is one of the most played games for people with disabilities in general all across this world.” I know the game is popular – it has entertained thousands of people – but I’m not sure it’s that popular. But with time, it will be.

I think Alvaro only played a few levels so far, since he said the “it’s very, very easy to use, very easy to play”. The game is easy to play and easy to use, but as you start playing the higher levels, it gets trickier and trickier. I was at a meeting a few weeks ago, and received a nice compliment about the game. The manager told me it was so addictive that they had to remove Blindfold Racer from the teacher’s iPads. The teachers were spending all their time playing the game, instead of doing their normal jobs.

The 5 minute podcast is here:

Podcast

The transcript for the podcast is here:

Podcast Transcript

Accessibility & app design blunders

Doug Wakefield, Judy Dixon and I have been collaborating on Blindfold Cryptogram. I started with the framework we built for Blindfold Sudoku, and then started making changes so the game would be great for rapid audio play.

You play the game by tapping on the screen, and it tells you the encrypted letter in a man’s voice, and your solution for that letter – if any – in a woman’s voice. As you swipe your finger from left to right, or right to left, it tells you the encrypted letters that you are touching. Swipe left or right with two fingers, it moves one word and reads it to you. Swipe up and it reads your solution for entire puzzle – words where you’ve solved it, and letters or beeps where you didn’t solve the word.

To enter your solution for a letter, you tap the screen with 3 fingers, and then select from the letters you have not used yet. That screen – called the keypad screen – also shows you the frequency of each letter and has a list of letters that you’ve already used, and some other buttons for hints and things.

Doug’s first issue was that due to his hearing, he couldn’t distinguish between the voice generation for different letters. He suggestion was to have a mode in the app to state the word – “apple” for the letter “a”, “baker” for the letter “b” and so on. That’s an easy improvement.

Doug’s next issue really shows the difference between how sighted people use an audio game and how blind people use one. When I play, I put my finger somewhere on the screen and listen to the puzzle letter, and then move around – looking at the blank screen and tapping at where other words should be. Doug, simply flicks left and right one letter, or uses the 2 finger swipe to the next or previous word. He never really taps the screen with one finger since he’s not using any visual cues.

From a programming perspective, distinguishing a tap from a flick is non-trivial. A tap starts and ends in more-or-less the same place , and lasts a short time – about a tenth of a second. A flick to the left starts somewhere, ends slightly to the left of the starting position, and lasts a short time – about a tenth of second. The problem is that its very hard to tell the difference between a tap and a flick. Based on the roundness of your finger, a tap doesn’t end where it starts – it’s usually a few millimeters away. So the question becomes – how to write a program that can tell the difference between a tap and a flick?

Another problem is distinguishing between a single tap and a double tap. If the app works with both, then it knows a double tap is when two single taps occur within a half of a second. The app must wait a half second when it gets a tap, to know the difference between a single tap and a double tap. That means that the app must wait a half second before it can respond to the single tap. A half second is a long time to wait when you are playing the game.

I spent a long time trying to solve this problem, and I researched alternatives on Google from what other programmers attempted. Blindfold Sudoku and Blindfold Cryptogram have reasonable, but not perfect abilities to tell the difference. Remember – when playing these games, the normal voice-over assist is shut off, so the app can be more responsive.

Had I thought about what Doug navigates the puzzle, I could have solved it much more simply. Doug, for the most part, never uses a single tap to navigate on the screen, since doing so would be pointless – he would randomly end up somewhere on the puzzle screen. Instead, he’s flicking and swiping. That frees up the tap gesture to be available for something else that’s more game appropriate, or eliminate it altogether.

Doug wants the alphabet to appear at the bottom of the screen and use the app as follows: First, he’ll navigate to the puzzle letter using flicks and swipes, then use a 2 finger tap to say that he wants to pick a solution letter. Then he flicks through the alphabet at the bottom of the screen, picks the solution letter (with a double tap), and moves onto the next puzzle letter to solve. Now that we’ve eliminated the importance of the single tap, that’s easy to do.