Heroes of Newerth servers suffer 'catastrophic hardware failure'

Due to a "catastrophic hardware failure," Heroes of Newerth is offline. Developer S2 Games is working to restore the game.

11

It would seem that catastrophe has hit the DotA-esque world of Heroes of Newerth. "During our scheduled maintenance period this morning we had a catastrophic hardware failure across a number of systems including some of our backup systems," a post on the game's official site has revealed.

On the Heroes of Newerth forum, an S2 Games employee revealed that the failure occurred during scheduled maintenance to upgrade the processors on the database boxes. "Both of our servers were simultaneously taken offline while the new hardware was being put in place. During this period of time, the file systems on BOTH [emphasis by the employee] boxes were destroyed."

The post adds that it "seems almost statistically impossible" for a simultaneous failure to occur; however, it is in fact what happened. "If this boggles your mind to the point that you can't believe it could happen, you would be in the same exact boat we are!" the forum post notes.

According to the employee, developer S2 Games is left with no hardware to operate its databases and replacements are not readily available. "We are scrambling to bring our systems back online temporarily so you can play the game through other short term methods while we get new hardware online."

S2 Games has not offered an estimated time for the game's revival. Shacknews has contacted representatives for S2 Games to learn more details and to get an ETA, but has yet to hear back at the time of publishing.

[Thanks, Ecks]

Xav de Matos was previously a games journalist creating content at Shacknews.

From The Chatty
  • reply
    June 21, 2011 6:45 PM

    Xav de Matos posted a new article, Heroes of Newerth servers suffer 'catastrophic hardware failure'.

    Due to a "catastrophic hardware failure," Heroes of Newerth is offline. Developer S2 Games is working to restore the game.

    • reply
      June 21, 2011 6:58 PM

      I bet they were hacked and this is the cover up!

    • reply
      June 21, 2011 7:09 PM

      Shoot this sucks! I love HoN!

    • reply
      June 21, 2011 7:19 PM

      Poor guys :(

    • reply
      June 21, 2011 7:50 PM

      game been down more than amy winehouse these days

      • reply
        June 21, 2011 8:01 PM

        Bit like League Of Legends Europe then.

    • reply
      June 21, 2011 7:59 PM

      i bet they dropped them

      • reply
        June 22, 2011 9:44 AM

        I was thinking someone spilled their drink and landed on both production and backup systems.

    • reply
      June 21, 2011 8:04 PM

      so... they can't restore from a DB backup? or is that their plan and it just means some added downtime and a little data loss?

    • reply
      June 21, 2011 8:55 PM

      A "destroyed" or corrupt file system can be any number of things, and not just hardware failure. A bad configuration or write can lead to this, or some other unforeseen firmware bug.

      I'm confused though, in that he says that the file systems (software) are down, but that there is no replacement hardware. If they have an adequate backup (or any backup for that matter), then they can rebuild the box from bare metal.

      One theory: If there was any sort of data replication being done between the two servers, and a fault occurred on one, then it's entirely possibly for the fault to propagate to the other as well. I've seen this happen in a number of SAN environments where one was replicating to the other, the first one develops a fault in the controller, writes bad info to disk, which is then happy replicated to the non-faulting SAN.

      Short of it is, keep backup often, and keep it offline from your running systems.

      Hope they figure out the cause soon.

      • reply
        June 21, 2011 9:02 PM

        I hope they also tell us what happened. I'd love to know more about what exactly happened. What OS was this on? How was it configured? Could I run across this problem myself at work?

    • reply
      June 21, 2011 9:04 PM

      They dropped a server? Did they hire a new guy?

    • reply
      June 21, 2011 9:55 PM

      Sounds like some kind of ground fault.

      Brushed against the power units the wrong way perhaps? : (

    • reply
      June 21, 2011 10:32 PM

      * G I B S O N ' D *

    • reply
      June 21, 2011 10:37 PM

      [deleted]

    • reply
      June 22, 2011 12:06 AM

      Sucks. If you're upgrading the processor I bet they scraped something while working in the racks/pharm. I just changed my processor out today and when I pulled off my heatsink, I almost couldn't take it out cause the paste was sticking really well and I almost ripped the dang thing a new hole.

    • reply
      June 22, 2011 1:49 AM

      [deleted]

      • reply
        June 22, 2011 2:33 AM

        Ouch. That will not bode well.

      • reply
        June 22, 2011 4:32 AM

        How can their last backup be two weeks old?

        • reply
          June 22, 2011 4:54 AM

          They deleted their primary backups along with the live system.

          This, BTW, is why you NEVER deploy code on your live systems.

          • reply
            June 22, 2011 9:21 AM

            Sounds like a situation where a developer is also unofficially the lead sysadmin. Company is in need of some change management processes, stat!

          • reply
            June 22, 2011 2:00 PM

            LOLOLOLOLOLOLOLOLOL

    • reply
      June 22, 2011 9:03 AM

      I guess I now know why LoL has a login que going so early in the morning. All the HoN players are jumping on since they can't play that game right now.

    • reply
      June 22, 2011 5:43 PM

      S2 won't get back to you with anything relevant, Xav. Word is they're a real mess behind the scenes and don't have any competent technical staff.

Hello, Meet Lola