Yahoo’s Big Fantasy Football Outage
If you play in a Yahoo Fantasy Football League, and waited until the last minute to update your lineup last Sunday November 11th, you might have been left out in the cold. Around 12:15PM, Yahoo suffered a server outage that caused their Fantasy Football site to become very slow and in some cases even made it inaccessible. This outage effected the fantasy site until after the 1:00PM ET games began.
Sure it’s always good to set your lineup earlier in the week, but lineup changes right before kickoff are a necessity in Fantasy Football due to last minute injury updates. So, this was a major problem for many fantasy owners.
Here is an email the went out to fantasy players early Monday morning from Ken Fuchs, Head of Yahoo Sports.
Dear Yahoo! Fantasy Users,
I want to sincerely apologize to all of you about today’s Yahoo! Sports Fantasy outage. As the head of Yahoo! Sports and as a Yahoo! Sports fantasy player myself, I am disappointed that we failed all of our fans today. Our first priority is having the best experience for our users, and today we fell short.
The outage started around Noon ET (awful timing we know) and while our team was on it immediately we are still working on various pieces. Our team is continuing to work on identifying and resolving the root cause. We have restored full functionality on the website, and we’re working for a final fix for our mobile apps. Currently data and scores can be viewed but for now you cannot make transactions or change line-ups from the apps.
We will also use today as an opportunity to improve our set-up so that we hopefully never have an outage like this again. Our fantasy commissioners and players are our biggest priority – we pride ourselves in being able to offer our users with the best fantasy sports experience possible and we take our job to deliver that to you very seriously. Rest assured we will work hard to make sure we continue to deliver on that commitment.
Thanks for playing with us and your patience today,
Head of Yahoo! Sports
On Wednesday, Yahoo sent out another email that explained in somewhat technical detail, what went wrong last Sunday. The most heavily affected appear to have been mobile users. According to the second email, it sounds like Yahoo’s Fantasy Football mobile app was a necessary casualty in order to get the browser-based version back up and running.
Dear Yahoo! Fantasy Users,
As I mentioned in my earlier email, Yahoo! works hard to provide great experiences for our users every day. For you, we understand that Fantasy is a core part of your daily routine during this time of year. On Sunday we experienced technical issues which caused the Fantasy experience to be inaccessible. We’re sorry. Our entire team worked tirelessly to re-establish the service – however, we had a significant outage. We know this created inconvenience and frustration for millions of our fans, and we feel we owe you an explanation.
So what happened? At Yahoo!, we have giant machines called “filers” that process a lot of the real-time data and stats for us and for you. We do millions of calculations every hour for our games, and normally our machines can handle this with no problem. Recently, we discovered a hardware issue in one of the filers that caused the other one to overload. We replaced some hardware, re-configured the setup, and did some testing. However this Sunday – at approximately 12:15 p.m. Eastern – the new configuration failed. This created an overload on storage capacity and took the Fantasy part of our site down.
We had dozens of engineers from various teams working together to try to determine the cause and fix it. One option was to fall back on another data center, but that would have meant shutting the game down and losing scoring data. We wanted to avoid that at all costs. Ultimately, we were able to move our mobile apps to a back-up data center, free up storage to get the PC version of the game working, and get the mobile apps up in a “read-only” state – meaning you could see scores and data, but you still couldn’t set lineups and interact.
We spent Sunday night and most of Monday looking at dozens of potential causes. Monday afternoon, we stress-tested our system. Everything seemed to be in working order, so we turned on all our mobile app functions in time for Monday Night Football. Everything performed as expected and continues to do so. We’ll have all hands on deck this coming Sunday to closely monitor performance and ensure we can respond quickly in case of any abnormal activity.
We know many of you have questions about your leagues and games, so we’ve also set up a FAQ here that will explain how we are going to handle various scenarios.
Finally, we know we failed you this weekend, and there’s no way to fix that. As a token of our appreciation for your patience and continued support, we wanted to give something back. First, we’re offering a complimentary football weekly Scouting Report for the remainder of the year. Hopefully this will help you down the stretch. The report includes weekly expert analysis, rankings and news to help you make smart decisions. Second, we are offering all our users a 20% discount for the rest the year at the Yahoo! Sports Store. For any updates, we will keep you informed on @YahooSports and facebook.com/yahoosports.
We won’t stop working to provide you with new and improved features that help you enjoy your Sundays.
Thanks again for playing and for your patience.
Head of Yahoo! Sports
Unfortunately, I doubt any of the niceties that Yahoo is offering will soften the blow for any users in money league that ended up losing a game due to missing out on a last minute lineup change.by