Mystery Hunt 2017, Part 3: Procedural Issues and Gratitude

(Note: The 2017 Mystery Hunt was created by Setec Astronomy, a team of about fifty people. I did a small fraction of the work and was not in charge in any way. Any opinions or perspectives below are my own and don’t necessarily reflect that of my teammates.)

Those of you who follow me on Facebook know that I had a very stressful year; a lot of things contributed to that, ranging from personal to professional to political, but Hunt was a big one. There were a number of things I wish we had done differently (some of which I complained about at the time), and I want to document them here not to throw anybody under the bus (I’m certainly not going to name any names) but rather to provide a warning to future construction teams, especially a first time construction team like Death & Mayhem, about small things that can cause you headaches later on.

I also want to emphasize that some of the people I was most frustrated with over these issues really stepped up in the last few months, which I deeply appreciate. But in a perfect world, November and December could have been a tune-up period, rather than a period in which a third of the Hunt’s puzzles were still being drafted.

1) We didn’t do enough to keep the whole team aware of the process throughout the year.

We had two primary ways for the team to communicate: Slack, and the team e-mail list. Between the day the theme was chosen and December, we didn’t use either enough for my tastes. Early on, a committee of about twenty people interested in metapuzzle writing was split into two halves and worked on metapuzzle development, with the idea that the other half could be the first testers and develop the metas further, before they were tested by folks outside the committee. This took several months, and as it turned out, (a) we didn’t do much testing outside the committee after all, and (b) we got used to not talking about what was going on using the main list, since what was going on was a secret to most, and I don’t think we ever kicked that habit. Just to show that I’m not just assigning blame to others, I was one of the people principally in charge of story, and in December, when I told a constructor their flavortext was inconsistent with the story, they told me that they didn’t know what the story was at that point. That’s as much my fault as anyone else’s.

We also had a small executive committee that accomplished a lot of really important tasks (props to them) but didn’t always keep the rest of the team updated on what they were doing (and I think that made it look to people outside that committee like there wasn’t much to do, which probably contributed to #2). I was on that committee briefly but ended up leaving out of frustration, and sort of designated myself “person outside the committee who constantly annoys the committee about keeping the rest of us aware of what’s going on.” When Setec ran the Hunt, Ray Jones served as “Team Asshole” (his term, I believe, not mine) whose job it was to step in bluntly when people weren’t meeting deadlines. We didn’t officially have a team asshole this year, but I think I unofficially stepped into that role, and I suspect some of my teammates would consider that an apt name for me during this process. (They would not be wrong.)

In any case, I suspect there are casual members of our team that would have happily done more than they did if they had been in the loop earlier, and I found that very frustrating.

2) We didn’t have enough urgency in the first half of the year.

The beginning of Hunt construction is very bottlenecky. You can’t do anything until you have a theme and structure. Then you can do very little until you have metas. (I know some teams have actually split up round creation and have thus started building the puzzles in a round as soon as that meta is complete, though I’ve never done so. We did start accepting character puzzle proposals while the quest metas were being written.)

I already mentioned that our meta process was a bit isolated, but even within that process, there were sometimes stages where nothing happened for several days. Sometimes that was because it wasn’t clear that we had to move on, and sometimes it was because the person in charge of the next step vanished and didn’t delegate to anyone. In any case, there were definitely times when, from my perspective, the Hunt seemed like it was on pause, and we suffered for it later.

3) We spent too much time using a one-editor system.

In 2011, Metaphysical Plant created a Hunt construction platform called Puzzletron, which will almost certainly be passed on to you if you run a Hunt. It tracks each puzzle from proposal to draft to testing (which can be done by testers through Puzzletron without a moderator, which is AMAZING) to postproduction to final version. It’s an invaluable tool, and I would never write a Mystery Hunt without it at this point. (Thanks, MPP!)

One of the key elements of that process is assigning editors to puzzles. My experience in 2014 was that every puzzle got three editors; at each advancement stage (most importantly idea-to-start-writing, draft-to-testable, testing-to-ready) happened once 2 out of 3 editors clicked a button, and it was all a very smooth and automatic process. The last time Setec wrote a Hunt was pre-Puzzletron, and editing was a more ad hoc process… likely stemming from that, this time we started with one editor assigned to each puzzle. This did not work; having one editor and one author is not conducive to discussion, and if the editor got busy or didn’t log into Puzzletron for a while, everything ground to a halt. Eventually we made more people editors (establishing two tiers of editors, the top tier of which would end up with more assignments) and put at least three editors on each puzzle and a lot more started happening. Based on these two Hunts, I think it’s good practice to assign a lot of your team members to be editors; even if someone has no experience, if they’re part of a team of three where others do, they’ll likely be useful.

Again, the good news is that none of these missteps (or at the very least, they’re things I considered missteps) damaged the Hunt, thanks to a lot of work from everyone in the last two months. But those two months were much more frantic than they would have had to be if we’d have a smoother process. I think the moral of all this is that in April, the Hunt seems like it’s a long way away… it really isn’t. If a week goes by and not much happens, imagine it’s December 1, at which point losing a full week would be terrifying. Get everyone involved in the process early, so that everybody doesn’t have to drive themselves crazy by the end to get everything done. (And if it seems impossible to stay on a well-paced schedule, ask Erin Rhode for advice.)

= = =

I’m grateful to virtually everybody on our Hunt team for the work they put in, but I’d like to give particular thanks to a handful of people that made this experience better for me personally, and whose contributions might not be obvious from the outside. (I’ll also tip my hat again to the kickoff cast, whom I already thanked in Part 1.) So thanks:

To Philip Loh and Guy Jacobson, both of whom accepted a field promotion in terms of editing and ended up doing a lot more work than they initially signed up for. Their contributions were absolutely crucial to getting things going at a necessary pace, and I was grateful that they were willing to volunteer much more of their time than they probably intended.

To TK Focht and Jenn Braun for proposing a structure that I could really get excited about (see Part 1 of my recap for more on that), and for letting me get involved with the story development.

To TK Focht (again) and Matt Gruskin for building a smoothly functioning website (which, as we learned in 2016, is not a given) and showing us enough partial work throughout the process to keep me confident that the one thing I have no control over (I do a little programming, but a project on the level of the Hunt website might as well be magic to me) was going to be just fine. And they both wrote some great puzzles at the same time; each of them was fully or partially responsible for one of the puzzles in the top three I listed at the end of Part 2.

And to Jenn Braun (again) for keeping me sane for most of the year. Jenn and I started a Hunt team (not this one) together eleven years ago, and having run two Hunts together, we agree on a lot of things and generally understand each other when we disagree. I was figuratively on the ledge a lot writing this Hunt, and Jenn did a great job talking me down when I was overreacting, and helping to advocate for me when I wasn’t. She’s one of my best friends, and I wouldn’t have made it through this project without her.

So that’s it! Nothing left to talk about… Oh, right. Stay tuned for Part 4, where I’ll talk about the amount of time it took Death & Mayhem to find the coin, and maybe we can all have a little chat about what it means for the Mystery Hunt in the long term.

17 thoughts on “Mystery Hunt 2017, Part 3: Procedural Issues and Gratitude

  1. One thing Plant has done (which may not be as useful for teams with more experienced puzzle writers who have more fully-baked ideas), is opening up idea submission for puzzles before the metas are complete. You can’t actually write the puzzle since there are no answers, but you can still do a lot of the initial idea developing. This doesn’t come anywhere close to solving the bottleneck problem, but it does definitely help. (As does approving some rounds earlier, as you discussed.)

    In addition to the bottleneck issue, there’s a risk with meta writing running on too long that too many of the high-volume puzzle-writers on the team are spending too much time on meta writing and not enough on puzzle writing. In 2011 we got our first world set in early June and the last one in late August, and although I’m very proud of our 20 metas and 5 meta-metas, definitely our puzzle quality suffered from too many of us concentrating on metas in the summer. I think the document about how to write puzzle hunts that Setec sent us in 2006 said to have the metas done by the beginning of the summer, but looking back in 2006 and 2011 we finished in late September and late August, so we didn’t really come close to the goal. I wonder what’s the earliest a team has gotten all their metas done?

    Liked by 1 person

    • In 2014 we finalized metas and the answer list in late May. I’d have to scour my e-mail archives to figure out when metas were finished for other Hunts I helped write, but I’m almost certain it was before August.

      But 2011 is my favorite Hunt to date (keeping in mind there are seven that I didn’t get to solve) so that turned out okay in the end, even if the process might have driven me nuts as a co-constructor.

      Liked by 1 person

      • Thanks for the kind words, glad you liked it! My point wasn’t so much that I was unhappy with the end product, as that it’s really clear that an extra month of work on the puzzles would have made the 2011 hunt quite a lot better (in contrast to say SPIES where I think an extra month wouldn’t have made as big a difference). Even more specifically, the Zelda round puzzles (which had a bit in common with metas, and were the clear weak point of the hunt) certainly suffered from the meta-writers being burnt out.

        (And I should also say none of this is to blame my team, in fact much of the blame for each of having a structure with so many metas, having the Zelda round, and not having enough Zelda puzzles falls in large part on me, and maybe the other people heavily involved with metas were less burnt out than I was. I also had the strange situation of not teaching in the spring but teaching in the fall, so the end-of-summer transition hit me harder than most, and might color my opinion about the timing.)

        Like

  2. “And if it seems impossible to stay on a well-paced schedule, ask Erin Rhode for advice.”

    Well, hi.

    Harvey Jones should get equal credit on the schedule in 2013 (for the 2014 hunt). He and I got together in early February and mapped out checkpoints for the entire year (theme and structure decided by this time, metas by Memorial Day, x% of puzzles done by this date, y% by this other date, art completed by this date, etc) — maybe one checkpoint a month or so. Then we published them on the wiki for everyone to see, which, in our minds if not in practice, set up an agreement with the team that this is what we were all signing on to. And if it looked like we weren’t going to meet a specific checkpoint, we sprung into action *before* the checkpoint to crack the whip. Because we had that schedule, it allowed us to say things like “It’s not personal, but your puzzle needs to get done by Friday or we’re going to miss the deadline. If you don’t do it, I’m taking your answer word back so someone else can write.”

    In the end, we actually missed very few of our checkpoints because we had the deadlines in advance and worried about them early instead of having to react after it seemed like we didn’t have enough time left.

    Early on in the process, people wanted to extend the theme and structure discussion and I took a lot of heat (including from Dan, if I remember) for not allowing that to go on due to the schedule, but I do think that’s part of why our January was remarkably less stressful than January 2004 was. Dan’s point about losing a week in April feeling different than losing a week in December, even though a week is still a week, is key.

    If anyone from Death wants our schedule, I’m pretty sure you know how to contact me and I’m happy to give it to you.

    Liked by 1 person

  3. Wow, this brings back so many memories of 2015. I was in two leadership roles for Luck while we wrote the 2016 hunt. One editor job, which was that second tier you mentioned, and the testing administrator. I used Puzzletron to assign puzzles to test solving teams. I absolutely could not imagine writing another hunt without Puzzletron! It was easy to see who had seen what, and links to spreadsheets allowed me to see all the work a team or person had done on a puzzle.

    I don’t think I was Team Asshole in name, but I certainly felt a lot of the pressure of the hunt and regularly took to the team email list to send out calls for help. I’d usually get a couple guilty-sounding replies of “sorry, work’s/life’s busy,” but I remember a lot of “thanks for letting us know the situation. I’m free at this time, please assign me something” replies too.

    I can’t speak to why Luck broke up this year, but in my opinion, we could have done better at planning and cooperating. It is absolutely a huge undertaking for any sized team, but on an already small team, it’s especially tough.

    The theme we went with was not my first choice – in fact, we had a theme very similar to your theme fleshed out, but it split votes with another theme. In hindsight, I wonder if going with Inception took the wind out of some people’s sails.

    It’s a lot of hard work for one year, and then after a few short days, it’s over. This blog can allow your thoughts and experiences to aid Death & Mayhem and future writing teams for years to come! Well done on putting the blog together, and thank you for your part in writing a wonderful hunt.

    Liked by 1 person

  4. Just to make concrete the extent to which lack of communication was an issue, there was an email to our team mailing list (subject line: “Pause meta testing!”) on April 17, and then exactly nothing more until June 8 (except for one request of the “does anyone on the team have skill XYZ?” sort). I — neither on the executive nor the meta committees — was honestly wondering by then whether we were actually writing a hunt.

    As someone who had not intended to write any puzzles (though I ended up writing one), I was very happy when the editorial work was cranked up, since that’s how I had wanted to contribute from the start. I think I could/would have done more, but it wasn’t at all clear to me how to step up — things were sufficiently compartmentalized that there wasn’t really a way to request more editing work, at least not through Puzzletron.

    Liked by 1 person

    • The difficulty of keeping an entire writing team informed of the status is a bit of a hobby-horse for me. There are good reasons to compartmentalize information, and it’s easy for those who have access to all of it not to appreciate the alienation felt by those who have access to little of it. Even if they did, the people who know everything are typically the busiest, and so simply don’t have time to keep informing everyone else of what’s going on. This tends to exacerbate the problem, because you can’t help out with a task if you don’t know that the task exists, and so the same group of people end up doing everything.

      The first time Plant wrote (2006), I was mostly not in a leadership role, and having no idea what was going on was a big source of frustration for me. The next time (2011), I ended up being in charge of sending out weekly emails to the team to keep everyone updated. I hope it was better that year, but I still think we could have done much better. I think this is probably under-appreciated as a difficulty in writing a hunt, and I’d be curious if any teams felt they had done it really well.

      Like

  5. > I wonder what’s the earliest a team has gotten all their metas done?

    We released the final batch of answers to writers in November 2011, in case you wanted a sense of how bad it can get…

    (This was mostly the fault of the editing team, i.e. my team, lest it seem like I’m throwing stones.)

    Like

  6. Every year I consider it a minor miracle when I arrive at MIT and … lo, there actually is a Hunt! A team of 40-100 unpaid volunteers really did, somehow, spend an entire year quietly crafting a dazzling array of unique puzzles, for an event which lasts for 2-3 days and then is never repeated.

    Somehow, year after year, this miracle keeps happening…

    Liked by 3 people

    • The “never repeated” bit always gets to me. It feels like it shouldn’t take *that* much effort to convert some old Hunts into a Puzzle-Boat-like system where one can register a team and replay stuff. Especially since very few individuals end up seeing all the puzzles. Sure, you’d have to cut out or emulate events and other interactive features, but that should all be doable on a case-by-case basis.

      Like

      • I read this book called Ready Player One (maybe you guys have heard of it); this comment got me thinking that one day everyone will be able to play every mystery hunt again in VR…

        Like

      • If memory serves, Random Fish had intended to do this. But in practice, after a team finishes running Hunt, there is very little energy left for doing additional work…

        Even an automated answer-checker would be pretty nice, and probably doable within the bounds of MIT’s web hosting (some JS hooked up to a basic input field). Re-implementing the hunt structure and unlock order is probably a bit much, though.

        Like

  7. Two specific things about the website that I was particularly happy with and would encourage future teams to at least consider:

    1. The Activity Log. It made our team organizer’s life a lot easier, and for everyone else it cut out a few steps in the enter solution into website -> tell organizer what you submitted -> wait for call -> wait for organizer to tell you the result process.
    2. Ordering puzzles on round pages consistently: if the puzzles were not ordered or otherwise arranged in a way important to the meta, they were always given in alphabetical order by title. This eliminated some red herrings for our meta team. (and provided the clue that if they weren’t ordered alphabetically, the order was important)

    Like

    • Agreed about the log. I’ve written two Hunt collaboration apps from scratch and been on teams with at least two other ones. This year was my first time in years to have a team whose only online collaboration was chat (Slack), a puzzle progress spreadsheet, and spreadsheets per puzzle. It worked great for us, but only because of the high quality of the Hunt site itself — I am very happy that my team for last year’s Hunt (static sites only) had sophisticated custom software.

      Big obsessive teams with people who want to put lots of time outside of Hunt writing custom software got a much better user experience from 2016’s Hunt than casual teams, but on years like this one where the Hunt site is useful, that’s so much less the case. Honestly, I’d love this to be something that future Hunt writers commit to ahead of time — tell teams “Yes, you can count on the Hunt site to track whether or not you have an answer in the queue, previous wrong answers, etc — you really don’t need to build your own software just to do that”. (Of course then people would be angrier if a last minute problem requires changes like in 2016, I guess.) That’s one way to work on the issue of equalizing experiences between big old teams and more casual teams — give the small teams the good user experience of a site that tracks your team’s full history, etc.

      (Yeah, I know plenty of dedicated teams are good at tracking their puzzles even in years with less helpful Hunt sites without custom software — whether by just being fewer people, moving slow enough that it’s easy to keep track of all the state in person or on a simple spreadsheet, etc.)

      Like

      • Yeah, as a fellow contributor to collaborative solving software, I agree this year’s site was impressively full-featured. I’ve sometimes wondered about a keen constructing team even publishing an API in advance, so you could plumb your software directly into the hunt software. But people might not like the notion that a team’s competitive edge would then be partially about software, rather than just puzzling skillz.

        Like

Leave a comment