Update on Build Status for Xbox, PS4 & Patching Process

[Raphael van Lierop]

Hello community,

First of all -- I want to say how sorry I am for the state of the Xbox and PS4 versions at launch, and the struggles we've had since then, to fix issues and get you patches in a timely manner. It is totally unacceptable that there are issues with crashing, and progress blockers in the game at launch.

I thought it might be useful to provide a bit more insight into how the patching process works across the different platforms, to help explain why patches may come out more quickly on some platforms than others.

General Fix & Build Process: We make fixes in Unity and submit content and code changes through a version control system called Perforce. These changes can be checked in Unity (using the editor), but occasionally what you see in the editor is not the same as what is seen in the released game, so we have to "make a build", which is a process where we (using a "build machine") assemble all the relevant assets and code into a complete, playable, and standalone version of the game. We have various types of builds, but the one we base our releases on is a "Release Candidate" build, which means it has things like the debug menu turned off, etc.

Building the game for all the platforms we launch on takes about 4-5 hours (about 1 hour for Windows, an hour for Mac, etc.). We check each platform and then test internally, as well as with the support of an external testing lab in Montreal. If things look good, we release to the platforms, where (apart from on Steam) each will go through some kind of review process before it is released to the public. We go through this process for every patch we make.

STEAM: There is no certification process on Steam. Once we are confident about the builds, we upload them to Steam and then release them when we are ready. The whole process takes about an hour. Since there is no certification delay and the upload/release process is 100% controlled by us, we can update the game on our own schedule and release as many patches -- big or small -- as we like. This is how we updated The Long Dark throughout Early Access as well. This convenience and speed is why the Steam version of the game tends to be the most up to date version.

PLAYSTATION 4:  We have two separate PlayStation builds -- one for SCEA (Sony America), and one for SIEE (Sony Europe). Each is essentially a unique company so all the submissions, certifications, ratings, pricing, and patching are all managed through different channels depending on which PlayStation we're talking about.

The initial certification process for the game takes 5 days. There is also about a 5-day waiting period for the Sony Store to set up your game. Once that's all done, you can release your game to the PS Store. You have go through this process for both Sonys (and a third one if/when you launch in Japan). 

After you are certified, your next update is a Day 1 patch. This also takes about 5 days to approve. If it passes, you can then use the Rapid Patching system which allows us to update the  game within a few hours, provided we don't do anything that would require a new certification pass. 

For The Long Dark, the PS4 SCEA version is far enough along that we are able to use the Rapid Patching system to get updates to you, which is why our PS4 SCEA version is able to be as up to date as Steam.

Unfortunately, our PS4 Europe version is still getting through the Day 1 patch part of the process. Last night we received word that our Day 1 for PS4 Europe has been rejected due to a well-known crash that will be fixed as soon as we can Rapid Patch. Unfortunately, the way the process is set up, we may  have to re-submit a new Day 1 patch that does not have the crash in it, before we can unlock Rapid Patching. We are waiting to hear back from Sony Europe on this, and should know by tomorrow. This has delayed our updating of the PS4 Europe version. We're working with Sony Europe to get the Day 1 patch out ASAP so that we can rapid patch it to the most recent version of the game.

I should be clear that we are not blaming Sony for this -- they have a strict process that we have to work through, and it's designed to protect consumers from issues that could brick their PS4s. But the process takes a while and until Rapid Patching is unlocked, we have to go through these steps. So, we beg for your patience as we navigate this with Sony and work through it as best we can.

Our focus right now is to get through the PS4 Europe Day 1 patch so that we can unlock Rapid Patching, at which point we will be able to ensure both PS4 versions are as up to date as the Steam build.

XBOX ONE: For The Long Dark on Xbox One, every patch goes through a 5-day review by the certification team. If the patch fails the review for some reason, we have to restart the process over again. We try to uncover issues as the process unfolds so that we can cancel and restart early if needed, but sometimes we only find out about issues after the 5-day process is complete. We have released one CU (Content Update, i.e. Patch) for Xbox so far, and we have a second one in with the Cert team right now. We submitted that one on Tuesday so it's making its way through internal review, and hopefully we'll hear something about it tomorrow (at the earliest). 

Since Xbox patches take so much longer than Steam or PS4, we're bundling up fixes in them so that while you get less frequent patches, they should have more fixes in them. That said, this version of the game will be a week behind the others in terms of patches. Microsoft has been a great partner in this and they are doing everything they can to help us get these updates out to you, without compromising their internal verification process.

New Territories

Making the game available for purchase in new territories means different things on Xbox and PS4. In both cases, there is a certification process involved. Unfortunately, you cannot be simultaneously patching the game AND in certification -- the internal processes don't allow for it. So, until the worst stability and save loss issues are resolved on PS4 and Xbox, we won't be releasing in new territories such a Australia or New Zealand (which we now have a rating for). We'll release an updated schedule for AUS/NZ and any other territories that are currently in the pipeline (we are working on ratings approval for Japan, Brazil, etc.) as soon as we hear more from those ratings boards.

State of the Game

I'm not at all happy with how our launch has gone.

We didn't anticipate some of the issues that would arise at launch and we've been, frankly, scrambling to fix things as quickly as we can for you. Although it may not seem like it, we did extensive testing in-house, and also had 8 testers from a professional testing lab working on the game for the last 8 weeks of development, to help us identify issues. We caught and fixed a lot of issues before launch. Some things that have come up, were impossible to anticipate up front. Other things slipped through due to carelessness on our part. In hindsight, the amount of content we released in Episode one and Episode Two was too much for our team to test properly, even with external help. We took for granted that 3 yrs of Early Access testing and stability would translate to a very stable Story Mode launch, and we did not fully understand until too late, the scope of some of the technical issues that would be uncovered by our launch.

Our goal over the coming days and weeks will be to continue fixing all the issues we find and you report (thank you!), and get the game to the place it should have been when we launched it. After that, we will be spending some time in deeper investigation of what went wrong, to see how we can improve our processes, pipelines, testing tools, and the technology we're using, to ensure we never put you through this again.

I know reading this message doesn't give you much comfort if you are waiting for an Xbox patch because you can't get through Episode Two, or you are playing the European PS4 version that is missing fixes from the last 4-5 days, but I hope at least that you will understand that we are working very hard -- along with our partners at Microsoft, Sony America, and Sony Europe -- to remedy the issues, which we take very seriously.

We'll continue to be available to support you and answer questions, and in the mean time we're working to fix issues and get those fixes out to you as quickly as we're able, more quickly on some platforms than others.

Thanks for reading, and I'm very sorry for the instability, bugs, and patch delays.

- Raphael

[Thrasador]

Thanks for taking the time to reach out to the community @Raphael van Lierop, and I accept the apology for the bugs. In the future I don't mind helping play test the game, as I am disabled and only play video games anyways....

We can pretend this is "Open Beta" if you want.....

[Raphael van Lierop]

Just now, Thrasador said:

Thanks for taking the time to reach out to the community @Raphael van Lierop, and I accept the apology for the bugs. In the future I don't mind helping play test the game, as I am disabled and only play video games anyways....

We can pretend this is "Open Beta" if you want.....

Thanks for the offer. It was important to us that everyone be able to experience the episodes in their finished state, rather than part way through development, although it seems we  may have missed the mark on that goal as well. We'll take a different approach to playtesting story content in the future.

[The Gentleman Bastard]

2 hours ago, Raphael van Lierop said:

Thanks for the offer. It was important to us that everyone be able to experience the episodes in their finished state, rather than part way through development, although it seems we  may have missed the mark on that goal as well. We'll take a different approach to playtesting story content in the future.

Please do. As frustrating as this is for you and the team, it is making me want to pull out my friggin hair. Was soo pumped to play Story Mode, and now it looks like it'll be, what, another month or so before we can even play it properly

[jonque007]

I have bought AAA games that have a lot more bugs than the TLD lol. As an OSG " old school gamer" you kind of get use to it. I still love story mode bugs and all. Your team made a great game IMO. 

[The Gentleman Bastard]

1 hour ago, jonque007 said:

I have bought AAA games that have a lot more bugs than the TLD lol. As an OSG " old school gamer" you kind of get use to it. I still love story mode bugs and all. Your team made a great game IMO. 

No one is saying its not a great game... its soo frustrating because it IS a great game and we are unable to enjoy it properly

[The Gentleman Bastard]

2 hours ago, The Gentleman Bastard said:

No one is saying its not a great game... its soo frustrating because it IS a great game and we are unable to enjoy it properly

And if anyone did say it was not a good game, i would literally fight them over it. its my FAVORITE game. Only one i really play actually

[The Wanderer]

It's a first release from a first time developer. I'm not worried as they're keeping in touch with their community.

[Titusmamet]

I love your game and have played it for years on steam. I'm sure I've sent you bug reports. I have also taken many beautiful pics of the game.  So excited to play on PS4, but I've gotten far in the game both on Storymode and doing the Nomad challenge. Both times I've lost the saves. There there, in the PS4, but TLD won't give me them. If you fix anything first, for the love of winter, fix keeping the saves. I can handle all the other bugs.  Love you guys!  Keep up the good work!!

[Thrasador]

2 minutes ago, The Gentleman Bastard said:

And if anyone did say it was not a good game, i would literally fight them over it. its my FAVORITE game. Only one i really play actually

This is the only game I have been playing since I started playing it. I haven't been playing it very long though, lol. Less than two months....

That being said, I am already at like 274 hours or something like that. Typically at this point I am completely sick of a game, and even if I hadn't finished it....I would just lose interest and stop playing it.

For instance I have loved Fallout since it was GOTY in 1998...or was that Fallout 2? Regardless, I have loved Fallout for DECADES. I have played all of them, often more than once. Even Fallout 4, I was burnt out on it by now and just stopped playing it, and I never even finished the main story. I just got overwhelmed by all the places, and side quests, and my little settlements....I was just tired of managing it.

This game, I don't seem to be getting tired of it. Part of it is the game, and part of it is talking to the other people here who also play the game. I enjoy the sharing of information, and tips, and secrets about the game as much as I enjoy actually playing the game. I probably spend more time on the forums helping other people find things in the game, as I do looking for things in the game for myself....

It's a pretty unique situation actually, I've never invested this much time in a game's community, let alone a game itself....

[Raphael van Lierop]

7 minutes ago, Titusmamet said:

I love your game and have played it for years on steam. I'm sure I've sent you bug reports. I have also taken many beautiful pics of the game.  So excited to play on PS4, but I've gotten far in the game both on Storymode and doing the Nomad challenge. Both times I've lost the saves. There there, in the PS4, but TLD won't give me them. If you fix anything first, for the love of winter, fix keeping the saves. I can handle all the other bugs.  Love you guys!  Keep up the good work!!

Yes this is something we've been working on non-stop since launch. We are getting closer to a fix.

[Sashamorning]

Thanks for the update.  It's good to know where in the process things are.  Looking forward to the patches!

[LucidFugue]

So sorry to see the release go this way. I know you were very cautious about releasing too early and always conscious that you only get one chance to go live. 

I'll second Thrasador's offer. As a long time alpha player and bug submitter (and someone who works in software delivery teams as a job) I'm happy to be involved in test builds etc. if that's something you'd like to explore for future releases. 

Keep up the good work, and don't forget some of the many stoic passages that accompany the game as you work through this difficult time. 

[GeronimoTester]

I understand the process of the data base and the steps to ensure no breach of "European"  Sony Entertainment when porting a patch over.

Yeah it is head wreaking to know that one gaming console / machine can get updates without hesitation I do have The Long Dark for PC and for both Consoles, when you fixed the issue with the Bed roll not spawning in the first cave I was like sweat now I can play it on my PS4 Pro sweat and everything time I loaded story mode the game just crashed and went ok its not fixed at all, the thing with consoles and any company will say there absolutely rubbish they cannot run games like PC the games that run on it either crash, or freeze, etc.

" HinterLand Studio " Announced it when it was coming to PS4 most people where stunned bye it in amazed to think finally yet another PC game is ported to the PlayStation System. 

Don't get my wrong the game  is really enjoyable and I never had so more fun/time spend on a video game before.

I'm a Game Developer myself in the Modding Community and I know it take's time to fix bug's and find the code that sum lassie / lazy guy did or didn't mean to do nor add a fix in the game, So don't beat yourself for it. 

Your guys should be glad your have a good support community that gives you love and affection that's hard to get these day's,

So we'll done "Raphael Van Lierop" for coming out and explaining the situation with the Console fan's for both Xbox & PS4. Not many Game Directors come out and make a statement so good on yeah..

Thanks for talking the time to also read my long Statement.

@GeronimoTester

Edited August 11, 2017 by GeronimoTester
Needed it make it a few changes

[SnowWalker]

45 minutes ago, The Wanderer said:

48 minutes ago, The Wanderer said:

It's a first release from a first time developer. I'm not worried as they're keeping in touch with their community.

 

Definitely, and a lot more than we can say about some other developers and their guy at the top. Thanks so much Raphael, for keeping us in the know. It's all that we could ask. In turn, we will keep the faith.

[Thrasador]

5 minutes ago, GeronimoTester said:

So we'll done "Raphael Van Lierop" for coming out and explaining the situation with the Console fans for both Xbox & PS4.. Not many Game Directors come out and make a statement so good on yeah..

Those were my thoughts exactly, and it's why I accepted his statements and apology. I feel it was a good choice to say something, and the right thing to do at this juncture. Things were getting pretty heated in various threads, and I feel the moderators were beginning to get frayed nerves over the hostility in general in the forums....

"Hey boss....the natives are outside and they have pitchforks and torches.....can you address them please?!"

[SnowWalker]

47 minutes ago, Thrasador said:

For instance I have loved Fallout since it was GOTY in 1998...or was that Fallout 2? Regardless, I have loved Fallout for DECADES. I have played all of them, often more than once. Even Fallout 4, I was burnt out on it by now and just stopped playing it, and I never even finished the main story. I just got overwhelmed by all the places, and side quests, and my little settlements....I was just tired of managing it.

I wish I could put this in some other thread but I just wanted to say that 'Me Too!' I can't finish FO4. It's the micro managing that gets me after awhile. I get what I call FO Burnout. I have to give it a rest for two or three months and then get back into it when I miss it and play another couple weeks and then I get FO burnout once again I will finish it though someday, maybe by 2020, but then there will probably be a new one, oh god, lol.

[Thrasador]

22 minutes ago, SnowWalker said:

I wish I could put this in some other thread but I just wanted to say that 'Me Too!' I can't finish FO4. It's the micro managing that gets me after awhile. I get what I call FO Burnout. I have to give it a rest for two or three months and then get back into it when I miss it and play another couple weeks and then I get FO burnout once again I will finish it though someday, maybe by 2020, but then there will probably be a new one, oh god, lol.

I would say it is a trend with many Bethesda games....they just put "too much" in their epic games. The games are incredible....like FO3 and FO4, and Morrowind and Skyrim....but they are all just soooo damn big, I can't possibly finish them. I just drown in content. Part of it is my OCD gaming, I can't miss or skip any of it. So I have to make this giant flow chart in my head of how to cover everything and in what order, and then it just becomes so daunting and overwhelming I just give up, lol...

[HappyHarry]

Bravo to you for:

1) Not only apologizing for the problems, but taking the time to explain the process of fixing them.

2) Stating that:

3 hours ago, Raphael van Lierop said:

we are working very hard -- along with our partners at Microsoft, Sony America, and Sony Europe -- to remedy the issues, which we take very seriously.

People do expect a game to work perfectly when purchased, but we do realize that this is rarely the case, especially in large sandbox games. So your statements will quell user's disappointment. I, for one, can wait for necessary patches, especially considering the story is not finished.

[Tenasi Vol]

@Raphael van Lierop 

Thank you for the explanation of the current state of the game.

Is there a list of "Known Issues" for Xbox users so we can know what exact bugs are known before posting them?

At this point, I can no longer play the game and will await patching.  The constant crashes to dashboard on Xbox are just too frustrating to continue putting time into it...as you have stated that you and your team are aware of.

Good luck.

[Scyzara]

Thanks  for explaining the different certification requirements and processes, Raphael.  I bet most people haven't been aware of these things, especially not about how long it takes you to have a new patch certificated on consoles.

That being said I'd like to say my personal experience (Steam/Win8.1) with Wintermute wasn't bug-loaded or in any other way negative at all. I've hardly ever noticed a bug and none of those few I encountered was even remotely game-breaking. 

So please don't be too harsh on yourself - there are presumably also a lot of people out there like me which had a very smooth and enjoyable launch. When you'll release the next episodes you'll have more experience with all those different platforms and their required processes. And I for one am certain you'll then be able to provide a very enjoyable experience for everyone right from the start. 

Just analyze what can be improved and keep up the good work! 

[VinceCgto]

@Raphael van Lierop Thanks for this, I think a lot of people needed to hear it. The explanation of the entire patching system on different platforms is also quite enlightening, seems like both appear to have their own quirks.

As others have stated, I would also be willing to be more involved in the actual testing process. As of right now, I feel like a lot of critical issues with the game are getting lost when they are reported in your bug reporting tool. It's like searching for a needle in a haystack. As a software developer I understand how easy it is for bugs to slip through and how difficult it is to keep the overview.

I have both the Steam and PS4 (EU) version of the game and would be willing to test and report in a more direct way.

Good luck!

[Rick8rooke]

Thanks for the apology and update Raph, I understand these things can happen with a small indie studio and its very much appreciated when you keep us informed - please keep this communication going with your players in the future.

I just got the game for my PS4 Pro after hearing good things about it, and while I have had many bugs and crashes I will give it a chance to get patched as I can see there's a fantastic game underneath.

[Raphael van Lierop]

1 hour ago, Tenasi Vol said:

@Raphael van Lierop 

Thank you for the explanation of the current state of the game.

Is there a list of "Known Issues" for Xbox users so we can know what exact bugs are known before posting them?

At this point, I can no longer play the game and will await patching.  The constant crashes to dashboard on Xbox are just too frustrating to continue putting time into it...as you have stated that you and your team are aware of.

Good luck.

The Xbox crashing is fixed or significantly diminished by the CU2 patch which we hope will be out very soon. We'll update on progress ASAP, once we hear back from the MS Cert team.

[Raphael van Lierop]

10 minutes ago, VinceCgto said:

@Raphael van Lierop Thanks for this, I think a lot of people needed to hear it. The explanation of the entire patching system on different platforms is also quite enlightening, seems like both appear to have their own quirks.

As others have stated, I would also be willing to be more involved in the actual testing process. As of right now, I feel like a lot of critical issues with the game are getting lost when they are reported in your bug reporting tool. It's like searching for a needle in a haystack. As a software developer I understand how easy it is for bugs to slip through and how difficult it is to keep the overview.

I have both the Steam and PS4 (EU) version of the game and would be willing to test and report in a more direct way.

Good luck!

Thanks! We do constantly monitor the Public Bug Database and pull issues over, although if we already have an issue logged in our internal database we don't, so as to avoid having a bunch of duplicates. We'll try to find some time to clean up the Public DB so people see movement there, but I assure you we are looking at issues and triaging constantly.