Evoke Home About Evoke How To Play Powers My Profile EVOKEblog Missions Quests Evidence Agents Leaders Discuss Reserve Your Spot

EVOKE was developed by the World Bank Institute, the learning and knowledge arm of the World Bank Group, and directed by alternate reality game master Jane McGonigal.













What Went Right, What Went Wrong: Lessons from Season 1 of EVOKE.

Posted by Jane McGonigal on 26 Jul under Behind the scenes

The EVOKE Post-Vita: What Went Right and What Went Wrong

For the next 3 days, we’re holding our Post-Vita meeting for the first season of EVOKE here in San Francisco. The lead team has gathered together to assess the first season and power up for future seasons and translations. We wanted to share with you our thoughts and ideas as we go, so we’re creating documents on the fly and live blogging them. We want to be as transparent as possible to help other social good games learn from our EVOKE experiences.

So first up: Here’s our top list of things that went right during EVOKE (and why – so we can do it again!) and things that went wrong (with at least one good idea for how we could improve next time.) In the great tradition of Game Developer Magazine, each lead team member made their own list of Top What Went Rights and Top What Went Wrongs. Please know: The solutions/recommendations aren’t necessarily locked in for future EVOKE seasons or experiences; we will continue to discuss and brainstorm, so please feel free to add your feedback and ideas in the comments.

Jane McGonigal (Creative Director/Game Designer)

Lessons on creative direction, game design, game running, community management

Top 10 What Went Right

1. We created an extremely active, productive community from scratch, virtually overnight. A significant period of pre-registration combined with a massive, targeted outreach in South Africa timed for launch helped us achieve a critical mass of players from the very beginning. The first 24 hours were extraordinary.

2. We focused on real, intrinsic motivation and real activity. We didn’t adopt a “sugar with the medicine” approach. The rewards weren’t artificial; the rewards were to learn world-changing ideas and to be creative and to master social innovation skills. And we didn’t do simulation or virtual worlds. We linked real-world stories and efforts with online interaction and feedback.

3. We defined and bounded the experience very clearly: “a crash course in changing the world”, brought to you by the World Bank Institute: 10 Weeks, 10 Missions. Unlike many alternate reality games or social good games, we defined very clearly what kind of engagement was expected; how long it would take; who was producing it; and how it would pay off.

4. We made it social. Early versions of EVOKE were decidedly unsocial. We amped up the engagement factor massively by employing social media (blogs, photos, videos, commenting) and social networking (friending, messaging, status updates). Players were never in it alone; they were in it together, and could be inspired and helped by others.

5. We designed multiple win levels. We correctly anticipated our pyramid of participation and calculated thresholds for engagement. We made it possible for casual players and lightweight players to meet a goal (e.g. founding member), and for active players to meet a goal (e.g. certification), alongside the most active players, who received heroic or legendary certification.

6. We invented the Leader Cloud. This is a significant innovation in social gaming. It emphasizes discovery and broad-based positive feedback as opposed to competition and hierarchical success. (We think Xbox Live should have more leaderclouds in the future!)

7. We created a highly addictive activity feed. The site never felt static; the community felt large, active and buzzing. The activity feed was the first “hook” for many players to stick around on the site. It added a level of transparency to how many people were actively  playing; throughout the entire 10 weeks, we averaged a 25-minute cycle in which the activity feed entirely replenished itself. There was live gameplay going on 24/7 for 10 weeks.

8. We created a super-satisfying feedback loop: runes automatically lighting up for completed quests and missions. I lost count of the number of times I saw comments, status updates, or received emails describing the runes as “surprisingly satisfying”. Putting control in the players’ hands to declare “Mission accomplished” or “quest complete” sped up the feedback loop in a way that not only helped game running, but also made players feel happy and rewarded.

9. We designed a great hero’s journey (the quests). Our real-play not role-play approach helped players identify their own personal strengths and imagine their personal best-case scenario futures. Although most players completed just one or two quests, the actual structure/content we created for this part of the game has real value and should be repurposed and integrated in future efforts.

10. We created a real “game-changer.” We took full advantage of media opportunities to create an extremely high-profile project – and to tell an urgently optimistic story – and as a result, EVOKE changed what people think is possible. We successfully engaged media in covering the innovation story of EVOKE, from CNN morning news and Wired magazine to New Scientist and The Stanford Social Innovation Review. We planned our launch strategically to take advantage of a major media opportunity – the TED conference – and ultimately inspired the world with the EVOKE story.

Top 10 What Went Wrong

1. We failed to start the gameplay soon enough – ideally, during registration. Although our overall participation rate was twice as high for registered members as most social media, I would like to aspire to 100% gameplay participation.  A quest built into registration – that would light up the player’s first rune and trigger the first positive feedback– would dramatically up our percentage of participation and hook players into further participation.

Solution: Build Quest #1 into registration.

2. The social world wasn’t bounded enough. The 20,000-person social environment was overwhelming, and “super-motivated” players drowned out other strong players who would have shone more in smaller social contexts. Furthermore, our completely open environment was too inviting for trolls and griefers.

Solutions: Some seasons of EVOKE could be playable by invitation only and require registration through a known EVOKE group based in the real world: a school or community program, e.g.

For this kind of experience, it also makes more sense to play in social networks that better reflect our real social participation instincts and capabilities: interacting regularly with a group 7- 15 people, up to 5 of whom are essential/strong ties (direct collaborators or allies); and in a larger social environment of up to 150 people with whom we have weak social interaction (commenting, viewing, messaging). Interaction should be limited to these social groups. More Rock Band/Farmville than World of Warcraft or YouTube., and some even more lightweight mentoring mode on top of that for global open interaction with anyone, anywhere.

3. Weekly missions and the final EVOKATION weren’t connected enough. Ideally, the missions would build to an evocation without requiring too much additional writing or content creation. This was the original intention of the early EVOKE platform, but our creative took a turn away from having the missions lead directly and cumulatively to a social enterprise pitch. As it was, the missions may have helped players come up with Evokation ideas, but the Evokation itself was a really big leap.

Solution: Make one stream of missions the EVOKATION stream. (Learn, Act, Imagine, EVOKE.) EVOKE could potentially replace ACT: Learn, Imagine, Evoke. The EVOKE missions would generate one part of a business plan/social enterprise pitch each week, or at least teach players about that part of the plan/pitch and how to make it.

4. We didn’t design meaningful SMS gameplay. We thought about SMS as a push medium only, and didn’t take good advantage of that either. SMS text is a unique medium requiring a radically different game design approach, and we did not truly push ourselves to go down this path.

Solution: Research SMS text based games and derive some key design principles for new seasons of EVOKE.

5. We missed the opportunity on real-time mentors. I have a strong suspicion that many of the people who registered but didn’t complete any missions or quests were actually mentors who never figured out how to mentor. Our mentor-app was great but not well-integrated, and we had no idea which players to push the mentor-app to, because we didn’t identify mentors at registration. Making it more difficult this season, the game runner tasked with the special project of matching mentors with specific players by email was ill and unable to complete the project; and others were too time-crunched to step in. Overall, we had an amazing resource (willing mentors) that we did not use effectively.

Solutions: Identify mentors at registration and separately their profiles out from players. Start the mentoring process immediately: Ask for a piece of wisdom or advice at registration, and publish it. Have a separate mentor weekly email/update. Integrate mentoring more clearly with the overall functionality. For example: Have a button on content submitted by players that says “Ask a mentor for feedback” which puts it in a cue/stream that mentors can resond to.  We should also ask mentors to publish ideas for how they would like to use the EVOKE network – many of our mentors were excited when players volunteered to work with their existing organizations and projects, and that is its own win.

6. The +1 voting wasn’t fun or meaningful enough. The EVOKE powers were too serious and too hard to differentiate/apply to the content; most people voted for “creativity” or ‘knowledge networking” all of the time. Also, the +1 leveling up system also became too competitive. A less quantitative way of providing feedback could be helpful.

Solution: Consider virtual gifts instead of +1s.

7. The quests would have been more valuable if they were more integrated with the missions – players using unique strengths and vision to respond to the URGENT EVOKES. I enjoyed reading players quests’ more than their missions!  They were, overall, more unique and revealing. We want players to bring that passion and individuality to missions – even learn missions.

Solution: When designing missions, link them explicitly to a quest. Require players to use one of their self-described superpowers. Or, require them to work with a self-described ally. This would be a creative constraint that would differentiate individual missions (making them less same-same and more enjoyable to review), make them more fun (it’s better game design to add a meaningful constraint), and spark creativity further.

8. Overall, the collective experience of the EVOKE network subsumed the individual journey toward social innovation. At the end of the day, we’re not trying to build a global communications network. We’re trying to create small hubs of innovation, individual enterprises that are strengthened by their global connectivity. This season, I felt players were more focused on “building the EVOKE network” than getting their actual personal goals accomplished or individual skills developed. I think want to find better balance between these two. It’s easier to contribute to an online network than to make an effort in the real world. We need to emphasize action over communication.

Solution: I think this is a copy and design solution. We need to sharpen our copy and re-configure our metrics to focus more on tracking the progress toward social innovation and less on social networking.

9. We needed more meaningful opportunities for strengths-based collaboration and teamwork. I still believe that we make our best efforts when we are playing to our individual strengths, in a context where those strengths are needed. We did not have collective missions that allowed players to differentiate and play to their strengths. This kind of mission is very complex to create, but it would be well-worth considering – particularly around the EVOKE stream of missions.

Solution: This might be a more open-ended solution: Ask players to collaborate and require them to identify roles/tasks and what makes each player the best person to tackle that task. (Rather than us designing roles to be filled and defining the skills required). We could make it more about teaching the skill of strengths-based contributions than having them fit into our boxes.

10. We definitely want to do a better job presenting the EVOKE Code of Ethics and getting player buy-in. We can also rethink what the community can manage on its own versus what we want game runners to manage. This evolved a lot during the game – and we saw that players wanted to take on more community management responsibilitys. We need to make clear that we have a commitment to a positive environment and we did lean towards being really proactive around direct outreach to players to ask them to “speak like heroes.” Even though our proactive approach led to some controversy – and some players feeling stifled and wishing for more unfiltered “no holds barred” social interaction – we feel this directly reflected the desires of the schools and community groups we were working with. Roughly 20 outspoken players (out of 20,000+), or .1% ,of players wanted less rules/more freedom to be harsh with, or critical with, each other and the game runners; by far people we heard from preferred a “positive social environment” and we need to do a better job of sharing this desire with the community so they realize that it is a bottom-up desire, and not top-down enforcement.

Solution: Have a clearer community code of ethics at the outstart. This is a crucial part of registration and perhaps even your first mission or quest – opting into the EVOKE Hero’s  Code of Ethics.

Robert J. Hawkins (Executive Producer, Education Specialist)

Lessons on education goals, outreach, and partnerships

The WINS

  1. Amount of quality posts and interaction – the quantity and more importantly the quality of dialogue and posts was phenomenal. The missions supported reflection and dialogue around global challenges.  The Quests helped refine personal aspirations.
  2. Out of game energy – a huge positive surprise was the amount of out of game activities and actions that the network embarked upon. I’m not sure if those actions were sufficiently rewarded; recognized and leveraged – but the ability of the game to inspire self-learning and internally motivated action was a huge win.
  3. Building of community – diversity of ideas – the number of people attracted to the site and the building of a community of learners was fantastic. The diversity and number of countries represented (over 150 countries!) which created  a truly global learning environment was tremendous.  The critical mass of active participants also fostered peer review; peer learning; and peer assessment.
  4. Narrative as a pedagogical tool. The creation of the EVOKE world; the characters; the story and the plot line helped drive, motivate and inspire learning.  The investigation files were a great way to build learning opportunities into the narrative.
  5. Focus on real world action – blending of education, fictional world and real world action. The move away from theoretical to real challenges and issues was a critical element of EVOKE success.  Creating opportunities and incentives to get players thinking about their own communities and taking the first steps toward action was a win.  This is something universities in Africa have been asking for a long time: action-oriented, real-world, practical curriculum that matches more closely to the real skills and knowledge needed to succeed and innovate. While the evaluation suggests that not all were able to move from the learning to action – having the learning set around real, global issues was great.
  6. Use of Social Networking – EVOKE I think created a niche that did not exist in other social networks for the players to talk about and debate serious issues with their friends. I think while many would like to engage in serious discussions in Facebook, the community of friends may not be conducive for this type of discussion.
  7. World Bank Institute – The World Bank, although a polarizing brand, helped attract and encourage quality dialogue and exchange. We were able to leverage the outreach abilities of the World Bank to bring in a huge community of participants, attract extremely successful mentors for our players, and get partnership with Global Giving. More however could be done to further integrate into Bank business.
  8. South Africa Marketing – the outreach and marketing in South Africa was a success based on the number of participants and their actions relative to the rest of the community. Not sure if the marketing materials, campaign by Student village or intense road show by Thiru and Titi had that most impact – but the combination was effective.
  9. Pyramid of participation – we received 10 times more EVOKATIONS than expected and overall the pyramid of participation numbers shattered estimates.
Target % Actual % Difference
Visitors 87,500 177,673 103%
Registered 6,875 8% 19,324 11% 181%
Active 700 10% 4,693 24% 570%
Certified 70 10% 223 5% 219%
EVOKATION 7 10% 74 33% 957%

10. Great mentors — I think we were able to attract some fantastic mentors for the EVOKATION winners. How sustainable this will be for the future and the impact of the mentorship will need to be reviewed.

The Improvements (and Solutions)

  1. We were not entirely successful in getting EVOKE fully integrated into the classes that were taught by the University professors who were our original target audience. (However, after-school clubs or weekend clubs seemed to be a very strong fit.) We also did not effectively define the professors’ role within EVOKE as clearly as we could have.   Part of the problem is that mentor play in general was not well defined and would need to be further refined for next season. More follow-up is required to improve our integration.

Solution:  Carve out defined activities and roles for professors/teachers.  Further define what it means to be a mentor.  Provide some incentives for teachers to use the game as a teaching tool – give them some type of special recognition.  More fully integrate them into marketing campaign.

1. Improvement in mapping of missions and powers. Despite having a core network of 13 professors in South Africa help drive and define the education mission of the game, we were not explicit enough with sharing this work and making explicit the types of skills that the missions and game activities were designed to develop vis a vis the EVOKE powers.  Many professors and partners have asked for this type of link to illustrate how the game is developing this set of 21st century skills that are represented in the powers.

Solution:  Work with team of educationalists and professors to review missions and map learning objectives and skill development objectives to each one.

2. Information management – too much to digest; poor tagging. I feel that we missed a lot of good ideas/submissions and that players also missed potentially interesting connections.  This is clearly a consequence of success.  Also, the lack of a uniform tagging scheme resulted in difficulty in searching relevant info.

Solution:  Implement automatic tagging system of mission evidence.  Also, is there a technical fix to be able to bring the most read/commented on/ voted for evidence to the top?

3. EVOKE is not a traditional Bank project and was something new for the Bank. Issues such as legal; contracting; ning contracts; office of information security; certificates, etc. needed an extra push.  Also being able to more effectively use EVOKE as a tool in Bank operations or to integrate into a country program would help.

Solution:  Future versions should be easier due to this year’s experience

4. While marketing and outreach in South Africa was successful, we would like to reach more participants in Africa outside of SA. Also, we did not reach enough female participants; it was roughly a 75%/25% split. Our graphic novel aesthetic may have been inadvertently skewed to aesthetics that resonate with men.

Solution:  Work with defined partners on a country regional level; put more resources into marketing in other countries; design the look/feel and marketing to attract more girls/women.

5. Access – Many of our target countries still have serious Internet access issues. While I think we did a good job of trying to develop a lightweight version, more thought needs to be given to mobile access.

Solution:  Design next version for mobile access with opportunity for Internet access as opposed to the other way around.  Look at ways to work with existing mobile platforms such as Mixit in SA.

6. Post-game support – not enough resources were put into post-game support and communication resulting in miscommunication and dissatisfaction among some players while waiting to hear about the status of their EVOKATIONS and mentorships.

Solution:  Clarify between game expectations and budget sufficient level of personnel to maintain communication and execution of post-game activities.

7. Time frame – length of season and time between missions: Is 10 weeks too long?  Is a week between missions too short? This is very difficult to design to different speed players – some complained that they ran out of things to do during the week. Others complained that they did not have enough time to reflect on missions and complete missions in a week.  I however really liked the flow of a weekly episode to keep people coming back.

Solution:  Have more activities – both longer term – requiring more thought and shorter term – easy to accomplish that would allow for both ambitious and less ambitious players to make their way through the game.  We could make it easier to play at your own pace – not feel that you have to “catch up” halfway through.

Kiyash Monsef (Story Director/Media Producer):

Lessons on world building and story

What went right:

1 – We developed a compelling world: dystopian enough to be intriguing, but ultimately optimistic, and most of all, inspiring. The concept of Africa as the source of the world’s solutions in the year 2020 was both surprising and easy for anyone to grasp – and the students in Africa loved this message.  Add to that the mystery and mystique of the EVOKE Network and “stealth innovators”, and I think we had a great concept to start from.

2 – We engaged a talented artist who gave us powerful images. Jacob Glaser, who invented the look of all of the EVOKE characters and drew every episode. deserves a ton of credit for what he did for EVOKE.  I think it was his art, even more than the story or characters, that inspired players to invest their time and attention in EVOKE.

3 – The trailer was extremely successful as a direct referrer to urgentevoke.com. Of the 39,000 people who viewed it, just under 10% clicked directly through to the site, which is impressive because the link is small and below the fold.  It was one of the top ten referring sites.  Also, people who clicked through from the trailer spent more time on the site than other referred viewers, viewed more pages, and had the lowest bounce rate of any of our major referrers.  It’s also possible that many people just typed in the URL after watching the video, so the direct referral rate may be higher.

The trailer was also a very effective promotion tool for indirect referrals.  CNN played most of the trailer (some shots multiple times) during their piece on the Early Show.  Jane (and others) embedded the trailer in blog posts about the game.  And it was an essential tool for Jane and I at TED.

From a production standpoint, I felt I had adequate resources to produce the trailer at the level that it needed to be, and we budgeted enough time to do it well.  I was also happy that it served a narrative purpose as well as a promotional one – it developed Alchemy’s character, gave him a voice, and showed how his avatar mask worked.

If we do a trailer for the next season, the (exciting!) challenge will be to leverage the existing recognition and connection that people have with these characters, while promising a new level of the experience and a new depth of story.

4 – We published on a weekly schedule. I think we were able to integrate the experience well into players’ lives by promising and delivering a regular product in such a way that they could plan for their interactions.

5 – The decision to build in an overarching story, rather than keep each episode totally modular, was definitely the right one. Players seemed to engage well with the story, and seemed invested enough in it to return each week for the new content.

Also, episodes didn’t feel repetitive.  The story model was very similar from week to week – new locale, team solves a different social issue – but I think we did a good job of giving the locations distinctive looks and feels, and the overarching story beats gave each episode something different narratively.

While this worked for us this season, in future seasons, I don’t know if I would rely on this model as much.  I think we’ve learned that people will keep coming back for the story, and I feel like that gives us license to do more in the future.  I think we have a strong world and strong characters, and if we can embed the learning into a more strongly threaded overarching thruline, rather than trying as hard to create so many standalone stories, we give ourselves better cliffhanger opportunities.  I’m imagining the next season might be three or four scenarios, each one spread out over 2-3 weeks of story.  We can get deeper into the issues, deeper into the characters, and deeper into the story, without having to rush through to a resolution and on to the next city.

What went wrong:

1 – Story arc could have been better defined earlier on. For Season 2, I’d like to take a more organized early approach to defining the story beats and the story structure.  I think we had some nice ideas that came out our open approach this season, but I think an deeper story map at an early stage would still allow for those kinds of emergent ideas, and would also help address some of my other “what went wrongs”.

2 – The rules and the size of the EVOKE network were never well-defined. I don’t know if they needed to be – it was a part of the mystery of the narrative and drove a lot of really interesting forum discussion.  But this was something that I struggled a bit with as a writer.  What resources could the network realistically call on?  In a way, I think the network sort of defined itself throughout the course of the game, so maybe this wasn’t such a problem, but it was a piece of the story that I thought was incomplete.  I’d love to spend part or all of a future season telling the story of what happened between 2010 and Tokyo, and try to flesh out some more details. We also would have benefited from a clearer EVOKE manifesto in the story – thankfully our players wrote quite a few for us by the end of the game that can inspire a future manifesto.

3 – Similar feelings about the Citizen X storyline. I really see these pieces as opportunities, though.  We can spend more time with Citizen X next season.  The emergence of WikiLeaks since EVOKE launched suggests a really interesting real-world parallel to Citizen X and could inspire future stories. Maybe we can do a whole episode told from the perspective of a Citizen X report.

Matthew Jensen (Art Director for the Website)

Lessons on art direction and UX

What went Right:

1. We weren’t an empty crowdsourcing site. Instead of just asking questions of the audience, providing empty shells for them to fill, or expecting user-generated stories, EVOKE actually adopted the role of storyteller, laying a strong creative springboard and context for players, and creating a beautiful visual assets to boot.

2. Willingness to be bold. Stylistically, the choices on the site (both the graphic novel and the art direction it inspired) made a commitment to aesthetics uncommon to the realm of serious games, and targeted to a unique group of users.

3. Responded to issues within the context of the narrative. While EVOKE straddled in-world and out-of-world throughout, the narrative was regularly reinforced through crafty in-world communications. When Alchemy contacted members and Citizen X took responsibility for hacking the leaderboard, players were ushered right back in to the emergent storyline. (Ask us sometime off-the-record about a very fun story about who was watching when the leaderboard was “hacked” by Citizen X, and how it enlived a very unusual meeting…)

4. Delivery tempo. Not only did weekly episodes maintain a tempo of player participation, but from a development perspective it created a predictable window for site fixes and improvements.

What Went Wrong:

1. Ning platform requires hacks and doesn’t render precisely. Nathan shares my insistence upon pixel-precise design / development, but Ning simply did not expose enough of the framework in order to code a consistent display of content. This was the greatest compromise of the platform, from a design perspective.

Potential solution: Partner more closely with Ning developers in order to better understand the platform (and unlock base code); explore out-of-the-box alternatives like Buddypress.

2. Graphic novel not optimized for screen viewing. While we reduced the size and compressed the file for reasonable consumption, it was still an extremely heavy graphic and bled off the screen. Additionally, page-turning functionality was not as usable as it could be.

Potential solution: Design assets from the get-go for 1024×768 screens, including clipping individual panels to best tell the story on screen. Explore Flash-based or more elegant JavaScript page-turning solutions. Create a novel that can be bookmarked and read in its entirety (rather than loading individual episodes).

3. Use cases and complexity only became apparent midway through dev. All things considered, EVOKE saw too little testing and early wireframing. As a result, usability suffered and new features were a bit cobbled into the experience.

Potential solution: Plan a more robust wireframe, prototyping, and testing phase, leveraging the best practices Nathan and I have learned in our UX experiences. Build player personas, use cases, and click paths. Tighten up architecture and Call To Actions accordingly.

4. Profile page was overwhelming. (I take 95% responsibility for this one, and Ning can take the other 5%.) Over time, the already crowded profile page became increasingly so with the addition of quest content and comments. A deluge of data made at-a-glance understanding difficult.

Potential solution: To retain the payoff, introduce animations or one-time rewards that then gracefully degrade (perhaps real-time notifications when someone votes up your content?). Use internal tabbing and collapsible element to minimize elements in the default view. Bubble up the most important profile details and greatly subjugate the rest. Simplify the visual appearance of runes, powers and such.

5. Limited navigation design. The restrictive top-level HTML was one of the culprits preventing useful lateral movements between sections, or “deep dives” into content. Additionally, naming conventions were not totally intuitive, and compounded the difficulty of browsing and finding content.

Potential solution: Integrate dynamically generated navigation with intuitive dropdown structures. Introduce consistent bread crumbs for backpedaling and an overt, ubiquitous sitewide smart search.

Nathan Verrill (Technology Lead, Director for Development)

Lessons from EVOKE on interaction design, usability, mobile development, online development

The Good

  1. The opportunity to work on an inspiring, world-changing project with real-world impact. Thank you.
  2. Delivered on time - for TED, launch, weekly release of content and shutdown.
  3. Stable, full-featured platform. Relatively speaking we had very few platform issues – 2 small outages are the only events I remember. Game runners were able to moderate content, remove items from the activity feed and award achievements to players very quickly. Additional invaluable features on the social network include: registration, blogs, comments, photos, videos, discussions, profiles, friending, activity feed, seo-friendly urls. Keith did an awesome job with the gamerunner backend. We were over-prepared with regard to hosting bandwidth, capacity and performance — but that’s a good thing! Given time and budget constraints Ning and Rackspace were good decisions.
  4. Responsive tech support. Ning consistently responded to help requests within their 24-hour service level agreement; Rackspace responded immediately; built-in Ning help request links and the tech support discussion forum notified us and allowed us to respond immediately. Google Wave was a good source of issues reported by game runners.
  5. Relatively speaking, despite the lack of pixel precision and some sections that remained un-styled, it looks sweet. Very distinctive and un-Ning.
  6. Staged release of content, fixes and features and use of a parallel testing environment. It was easy to view and approve changes on our test site.
  7. Analytics – good data from Google Analytics, Ning user data and game data.
  8. In-story ‘hack’ of the leaderboard and creation of the leader cloud. I would like a multi-faceted view of game content in the future, much like player performance in the leader cloud.

The Bad

  1. Usability. The ‘now what’ problem after registration. Clunky flow from objective acceptance to evidence submission. Clunky navigation between episodes for the graphic novel. Lack of good tagging for content; some content never viewed. Poor search. Clunky navigation on profile page for quests and evidence log. Many of these issues are due to Ning limitations, so even with user testing we likely wouldn’t have been able to do anything about it within our time constraints. Recommendation: Redesign the primary use cases using UX and user-centered design best practices. Perform user testing early and often, preferably on paper prototypes or low-fidelity html clickthroughs. Use ‘starbucks testing’ or (ideally) earlier testing with participants in Africa (they likely will have a different mental model of how the site should work). Test with realistic scenarios including high volumes of content and users. Use a social networking platform that gives us complete control over the navigation.
  2. Fragile integration of game features. The game features — power voting, evidence submission, quest completion, mission acceptance, leader cloud — had to be wedged into the Ning pages using jQuery. As a result they were not rendered until after all page assets were downloaded, meaning the ‘popped’ or ‘bounced’ into view; they were hackable since they used publicly viewable query strings; they were difficult to style once pulled into the Ning site; they added to the number of requests required to render the page (which can considerable time to the perceived load time due to connection latency). Recommendation: Only use Ning if we can integrate game features in a more seamless manner, such as with Open Social, although Open Social isn’t fully implemented on Ning and severely limited.
  3. Ning hacks. In addition to above, had to hack Ning to extract the user ID for the player and content owner; wrestle with an inconsistent HTML DOM that made styling very difficult; wrestle with Javascript and CSS limitations. All of which resulted in a very fragile website, imperfect visual design and very time-consuming dev cycles for tasks that should have been much simpler. Recommendation: Modify the design to be more suitable to Ning and integrate game features with an API; or use a platform on which we have complete creative and server-side game feature integration control.
  4. Undetected quest bug. The backend used Ning user names as the unique player identifier. As players changed their names quest data was over-written when a quest was updated by someone with the same player name as another. Initially we couldn’t extract the player ID, but after digging through the code we were able to find it. Recommendation: More experience with Ning (like we do now) or an implementation on another platform with tighter integration with error handling.
  5. No low-bandwidth option. We weren’t able to create a low-bandwidth version without totally recreating content without game features – which we did late in the game with the /m version. The site is heavy – the graphic novel, the graphics required of the visual design, the multiple css and javascript libraries automatically included by Ning, the multiple requests for game features. Recommendation: Use a platform that gives us complete control over rendered HTML, CSS and Javascript, allows us to merge files into one request, and serve a low-band version from the same platform some game features are automatically included. Test over a low bandwidth/high latency connection.
  6. Mobile version was not as fully anytime-anywhere as envisioned/promised. Same as above. We tested the look and feel on Opera and it was difficult to scale the graphic novel for a mobile device and maintain readability. Furthermore, when Ning auto-detects mobile device user-agents it serves a very limited set of the social network functionality and we therefore were not able to integrate game components into the mobile version. Recommendation: Use a platform that allows us to specify a mobile device stylesheet; Design a version of the graphic novel for a mobile device; Design a version of the quest/missions/evidence collection suitable for mobile device interactions – small screens, limited typing, etc. Work closely with mobile providers and test site performance in Africa.
  7. Content navigation. Content not linked to episodes or missions automatically and not tagged well. Recommendation: make automatic linking of content to mission objectives a requirement, design something akin to the leader cloud that provides multiple views of the content, use a more robust tagging system.
  8. Comment rewards. We were not able to reward people for commenting, either on content, forums or on user profiles. Recommendation: Reward people for participating in the game via comments. Better yet, give players ability to rewards comments they find useful, helpful, fun, or rewarding on their on content.
  9. Backend. The gamerunner backend is held together with duct tape. We were able to make it work, but it is less than optimal. We also have not spent any time on it from a usability perspective for the game runners. Recommendation: Consider re-writing the backend.

10.  Multiple instances. Currently there is no way easily create another EVOKE instance. It also requires additional site setup on Ning (although not Rackspace). Recommendation: Make creating multiple EVOKE instances a requirement, as close to a 5-minute install as possible. Centralize the game platform on Rackspace so there is only one hosting bill regardless of number of EVOKE instances.

9 Responses to “What Went Right, What Went Wrong: Lessons from Season 1 of EVOKE.”

  1. Lynette Young

    As a strategist, this information is like a dream! THIS is why it’s important to determine milestones and goals before getting started. THIS is why it’s important to take a regular step back and objectively look at how your strategy and tactics are doing. THIS is why it’s important to LISTEN TO FEEDBACK and be sure there is room in the equation to make changes on the fly. I could go on and on. ***Brilliantly done.***

  2. Grazie to Jane McGonigal! for posting the ARG Urgent Evoke Debrief » What Went Right, What Went Wrong « Transmedia Camp 101

    [...] via blog.urgentevoke.net [...]

  3. Turil

    I’d like to point out that while the expectation of respecting free-speech might have been less popular with the majority, the minority who were promoting the more realistic approach of mediation and tolerance, rather than authoritarian censorship and banning, were the older, more experienced, and more active, and more popular players. So perhaps the two goals of embracing diversity and conflict (free speech) as well as helping people feel taken care of (not being forced to read too much negative stuff) can both be attained at the same time. Allowing people (or reminding them, if this is already an option) to “black-list” agents who they simply aren’t able to feel comfortable around, so that those agent’s posts and comments are hidden to them, would give people the power to take care of themselves, rather than having some authority figure decide for everyone what is valuable and not valuable. And, of course, actually having mediators available upon request will help give people the feeling of protection, if they need it, while also teaching them better, healthier ways to solve problems other than violence (banning, censorship, insults, etc.). Mediation is something schools and even government have started to do in many areas, with great success. And it’s extremely valuable for helping people from such a diverse range of cultures together.

  4. Aseel Honein

    I would like to echo Lynette..
    It is very informative and rewarding to read this feedback/ evaluation report. Maybe a monitoring evaluative tool could be established for more elaborate results. I am so looking forward to season 2!

  5. Urgent Evoke – social good alternate reality game. - Applications For Good

    [...] lead team has posted the game Post Vita with each member offering the Top 10 What Went Right followed by Top 10 What Went Wrong (with [...]

  6. The democratization of aid. — GlobalGiving Blog – News and Updates

    [...] also admire their documentation of best practices and lessons learned, including what hasn’t worked. That’s brave and serves as powerful learning for the entire development [...]

  7. A look at Urgent Evoke: Reflections for Season 2 | TechChange | The Institute for Technology and Social Change

    [...] model designed to keep the internet and real-life closely related and intertwined.  The game’s shortcomings—namely, that a genuine sense of community did not develop amongst participants, that mentors with [...]

  8. Robert Hawkins: The Rise of Education Gaming

    [...] The development team also recently met to review the season — we have posted the lessons learn…. [...]

  9. edReformer: Robert Hawkins: The Rise of Education Gaming

    [...] The development team also recently met to review the season — we have posted the lessons learn…. [...]

Leave a Reply