Reply To: Star Citizen – Musings
“Unlike with a publisher, you can’t pull the wool over their eyes because it’s the real people who are going to be playing it”
-Chris Roberts, April 2013
IT’S OFF THE RAILS. COMPLETELY.
In last week’s situation report, I talked about the dismal state of the 3.0 build, as well as land sales. The 3.0 build is still FUBAR, and we have an entire forum dedicated to discussing just how messed up it is. As of right now, even the beefiest machines are still having trouble getting and sustaining even 15 fps on average. And that’s aside from the litany of bugs (in excess of 5K unique entries in the entire project), crashes, lockups etc. Seriously, they can’t even get something as simple as a cargo mission working. And it only involves going from point A, picking up a container, and delivering it to point B. It’s not working.
In the future, space men will be contending with actual manual labor
And it’s not just that – every single fundamental mechanic is either completely broken, or basically horrid. Flight dynamics, physics, navigation, Mobiglass UI, frigging doors (!), elevators, ships – the whole thing. And most of the streamers still trying to make money and subs off gullible backers, are inadvertently communicating to the world, just how broken everything is, and that – as I had written months ago – the 3.0 build is pure and utter rubbish.
To be clear – again – this has nothing to do with the fact that the “game” is still in development, and so this is to be expected. No. It’s to do with the fact that, going on six years now, with over $170 million (an unconfirmed and dubious claim) raised from backers, THIS is what they have and THIS is where they are. And it’s not even Alpha – and not by a long shot.
Yeah, so that happened.
By any and all accounts, 3.0 still does NOT represent even 25% of what was promised in the Kickstarter campaign, let alone 15% of what was promised over the years as Chris Roberts continued to expand the scope of the project once he figured out that doing so – while incessantly lying – is what kept bringing money in.
3.0 (w/ planet/moon access etc), ??/??/??
2.6.3, 04/27/2017
2.6.2, 03/31/2017
2.6.0 (w/ Star Marine fps module), 12/23/2016
2.4.0 (/w/ ArcCorp shopping), 06/08/2016
2.0 (w/ Persistent Universe, Multi-Crew Ships), 12/11/2015
1.2 (w/ ArcCorp social module), 08/28/2015
0.8 (w/ Arena Commander dogfighting module), 06/04/2014
0.x (w/ Hangar module), 08/29/2013
In contrast, they released (handy release timeline) 2.0 in Dec 2015, and it was a buggy mess. But it was the first big leap for the project in terms of functionality. It gave hope to the backers; though most of us just shrugged, and said that it was the beginning of the end because it was a clear, and present testament that, as I had stated in The July Blog, they simply could not build the game pitched in 2012, let alone at the scope that had been expanded in 2014.
I should also point out that 2.0 came right on the heels of when the 2015 controversy was in full swing. Hilariously, nobody envisioned that, two years later, the game would be no closer to completion.
When, after yet another year of screwing around with inconsequential things, even as the scope continued to expand, they released the Star Marine fps module in Dec 2016, it died almost immediately because it didn’t live up the hype and expectations. Nobody is even playing it anymore. And that was something they had all but killed – even as Chris Roberts was claiming it was already in Star Citizen. But then suddenly out of nowhere, they decided to resurrect it for that year end release because, well, they had nothing else. And at that time, Chris Roberts was already touting 3.0 as coming that December, knowing fully well that it was patently false as we later came to find out.
“So it’s our big end of the year release. We’re gonna get it out end of the year;
hopefully not on Dec 19th” – Chris Roberts, Aug 2016 @ 23:37
That’s not all…
FIX ALL THE THINGS. WITH SPINLOCKS
Though it’s highly technical, and beyond the scope of this article, the most hilarious thing happened last week. We’ve known for some time now that CIG tends to react to my writings in the most hilarious of ways. It’s even more hilarious when you consider that in between wanton acts of pillaging backer bank accounts, they don’t even listen to backers to any extent other than where it hits their wallet.
For sometime now, there are those who have hacked the game client to work in off-line mode. In this manner, certain game content have been discovered, shared etc. Others, like me, tend to dissassemble and decompile the game client in a bid to figure out various issues. The first time I did that, was back when I discovered that they had switched from Google Compute to Amazon S3. Yes, of course I wrote a blog about it. At the time, I claimed that they didn’t actually switch to the Lumberyard engine – a derivative of the CryEngine 3 that the Star Citizen game engine was built on. They had lied (shocking, yes, I know) about it. That aside from the fact that – for a whole year in which Chris claimed they were researching it – they never mentioned it once to backers. Not once. I discovered it in the 2.6.0 Star Marine patch, and wrote about it before it was even public.
As of this minute, those of us with access to CryEngine 3 and Lumberyard, know with certainty that they haven’t actually fully switched to Lumberyard as they had claimed. To the extent that most of the core and advanced functionality that Amazon has made to that engine, still do not appear in the Star Citizen game code. Even functionality that Amazon has completely ripped out of LY, still appear in Star Citizen game code.
With the dismal performance in 3.0, and wanting to see just how much new stuff was actually in it, what was causing the performance issues etc, some of us went tinkering and poking. So having discovered that Lumberyard had done away with most of the notoriously dangerous spinlock, we were shocked to find that Star Citizen had over 600 of those in the code. And in there, an eyebrow raising number of calls to Sleep(1).
I have never before written publicly about their coding because it’s considered poor form in dev circles. Even when I catch snippets of their Bug Smashers streams, and us devs get to discuss the “interesting” code among ourselves, we tend not to make it public. But then I went the other way and last week I broke protocol by writing this Twitter thread about spinlocks, and why they are affecting performance. I thought that was the end of it. So imagine my surprise when, in their patch released days later, this appeared in the notes:
In render mesh management, code lock contention has been optimized. Generally, frequent CPU spikes on server and client side due to spin locks have been removed. The relevant changes mention in last week’s report as in-progress have been submitted. People on the PTU have observed the effect of a degenerated “spin lock”. A spin lock used to control access to a shared resource when multiple threads are trying to work on it, such as a file or a memory space. It allows for very fast resource transfer between thread, but threads waiting for the resource are consuming a huge amount of CPU while waiting. It’s useful as long as each thread doesn’t wait long for the resource, otherwise it becomes a huge performance drain on all CPU cores.
I was floored. Before that, they had never mentioned spinlocks in any of their patch notes.
Mark Abent is the bravest dev on the project. Bonus @ 4:27. See the path?
But regardless, as I later wrote, as far as I can tell, they really didn’t do anything relevant or consequential. With access to several executable generations, and by using various dev tools (e.g. Hex-Rays disassembler and decompiler), it’s easy to track these things because calls like that are quite unique. Heck, aside from hacking the game client to run in offline mode, some have builds with these calls in the most critical areas, removed. Sure, it doesn’t do much – without adverse side effects – but the improvement in performance gains is the difference between +8 fps improvement on average, versus 7 fps in the current public build. Why? Because there is NO fixing bad architecture and design.
The running joke is that, with things like this, it’s plausible that the CIG/F42 devs are sending me a message. Hey guys, here’s a thought: quit. You’re never – ever – going to make a difference because you know that the project is FUBAR, and that Chris Roberts has FAILED – again. And he’s richer, and you’re not.
Oh, but it gets worse…
THE ELEPHANT IN THE CARGO CONTAINER
First, meet Clive Johnson. I know quite a few people who have worked with Clive. In fact, one of those people used to work for me. Clive is not your Jack-Of-All game dev tinker like most of us who started out back when we had to write all our own code – from scratch – regardless of discipline (graphics, physics, networking, UI, input etc). Clive, I heard, is a good guy. Clive is not a “networking guru”, but at the very least, he knows what he’s talking about. Which is why, what comes next, is not something that I really wanted to write. However, it plays directly into everything I’ve been saying since June 2015 about the state of the game’s development, and why they’re never getting an MMO from this.
Clive has been working on Star Citizen since Sept 2014, right in the middle of when Chris Roberts was increasing the project scope, while lying to backers, and making promises he simply stood no chance of delivering on. And that was before he shockingly decided that he was making an MMO after all.
Clive apparently has never worked on an MMO game before. And having been promoted from senior to lead network programmer, you would think that the promotion has to do with experience, instead of, you know, filling in the slot for a missing lead, or just moving into a promotion slot to keep you around with better pay. If you have been reading my blogs, then you should also know that I have written that there is simply nobody in the team leads who has ever worked on, let alone shipped, an MMO game. For these guys, Star Citizen is an on-the-job training gig, paid for with backer money. That’s why I had written an article saying that they were never – EVER – getting an MMO out of this train wreck. Ever.
Clive Johnson on ATV (@ 13:35), Jan 2016
For the longest time, backers have been fed a load of bullshit by Chris, his brother Erin, and even by some of the devs who were brave enough to get carted in front of a camera like show horses, in a bid by management to convey the impression that they were, you know, working on a game and not pissing away backer money. Due to the fact that, by it’s very nature, the networking engine in CryEngine is not designed for MMO games, the Star Citizen multiplayer experience was always shitty. The Arena Commander dogfighting module got by OK because, well, there’s really not much there. But once 2.0 arrived on the scene in Dec 2015, it became obvious that they were way in over their heads; and that networking was in fact, shit. Then Star Marine happened – and we’re still laughing at that one.
When they switched (a dubious claim I’ve written about before) to Lumberyard, the usual hype around networking started to pop up. This has been continuously fueled by even more meaningless (in the general scheme of things) bullshit such as serialized variables, network bind culling, server mesh network etc. All of which backers – despite our telling them it was all inconsequential nonsense – were thoroughly convinced would one day solve the networking issues with the game – and they were totally going to get an MMO. Heck, even though 3.0 is the worse ever build (it even tops 2.0, if you can imagine that) in terms of performance, stability, networking – and pretty much everything else – there are those who still want to believe that some day down the road, everything will be fine with networking because CIG said so. This despite the fact that, time and time again, those promises have either been flat out lies, or just simply didn’t materialize.
Clive Johnson on ATV (@ 12:25), June 2017
In the video above, Erin (yeah, he too was promoting that whole bullshit about server meshes, “hundreds of thousands” of players instances etc) mentions serialized variables, leading into Clive’s appearance at the 13:10 mark to talk about and explain…..serialized variables. Here’s the thing, those buzz words aren’t even noteworthy they are a fundamental part of any robust networking tech. But we get broadcasts like this because, like every successful con and confidence scheme, you have to keep your targets believing in what you’re selling and promising.
So it should come as no surprise that in the past 24 hrs, Clive has inadvertently started a massive shit storm with this post about the networking. On any other day, this would be par for the course with Star Citizen; but given how some backers have put so much hope and trust in CIG about the inbound networking improvements, this one is bit too close to the reality that they are about to face. This despite the fact that they know in their hearts that they’re never – ever – getting the game they were promised anyway.
Yeah, that doesn’t sound right – at all
I can’t imagine there being a single game developer – right now – who isn’t shaking their heads over this. While it’s common knowledge that the remaining Star Citizen backers know about game development about as much as they know about financial responsibility, this sort of response has raised the alarm bells of even those very same backers. Some of the comments are priceless, though my guess is that, as these things go, the mods are probably going to lock that thread soon enough.
“wtf….this is how you make change my mind within seconds…i always thought they handle the performance problems with this netcode thing, because i ALWAYS was sceptical about how they want to handle this amount of players. now you say you are not even close to handle 50 players without changing everything 10 times, generating such an amount of bugs that makes you holding back a major release for more than 1 year…
when do you want to release this game? in 20 years? you may should hire some more people to figure those problems out and let some artists go on the other hand. concept sales seem to be the only thing that works fine..“
Meanwhile, over at the Star Citizen Reddit water cooler and the no-cultists-allowed Reddit, the response is the usual hilarity that goes with things like this.
Here’s my comment on each of those statements.
1) The graphics pipeline does not wait for server updates
I am guessing that this one comes from the fact that some of those backers have been clamoring about the shitty networking code, impacting their game performance (which they tie to the frame rate counter).
In any game, there is a single game loop that runs the whole shebang. Everything (graphics, physics, networking, AI, input etc) happens within that loop. To better understand this, read this article which someone shared earlier today. It’s written such that any layman can understand it. That loop is timed depending on the type of game. A fast paced game would have a higher resolution timer, while another would not. For example, in some of my games, I have several timers within a single game loop; each slaved to a thread that runs a specific component (e.g. graphics, AI etc) at varying update (aka tick rate) resolution. It is not uncommon to detach, for example, the graphics subsystem and run it on its own tick rate within the loop.
So, the “graphics pipeline” does not wait for server updates because it can’t, and doesn’t need to. It’s in the game loop which is what determines the visual performance based on those updates. If the graphics component is so heavily burdened that it impacts the game loop, yes, all the other components within that loop, can and will suffer. Whether the graphics component waits for the server update or not, is irrelevant.
2) Server FPS does not affect client FPS
Despite the fact that we generally tend not to talk about the server in terms of Frames Per Second (tick rates are more like it) updates, in a multiplayer game, the client has to run a “simulation” of the game itself. If you have a “headless” (no graphics processing) dedicated server, even if it’s only displaying console messages, it most definitely needs to be running the game as the client would, otherwise things like position updates, weapons etc, simply would not be propagated to all the game clients connected to it. In fact, in peer to peer games, which is how most multiplayer games are designed, the client is acting as both a game client and game server. So if you have such a game, whatever your FPS is, that’s what both the client and server are running at.
With a client-server game, whereby the server isn’t doing any graphics rendering – hence no FPS – the server still has its own loop that’s oft measured in tick rate, which definitely will not match the client FPS, as that’s a completely different metric. On the other hand, if the dedicated server is in fact running in graphics mode, as some in fact do as I mentioned above, then the FPS could very well be higher or lower than the client, depending on the configuration (graphics card, memory, screen resolution) that affects frame rates.
So, in any case, whether not you are talking about frame or tick rate, it will tend to be different between the server and client; and one should never affect the other because networking updates (packets sent across the network) aren’t usually (and shouldn’t be in a game like this) tied to graphics updates.
64 players. Yeah, we’re totally making an MMO
3) Netcode does not make clients run slowly, and never has
Anyone who has done any work on multiplayer games, knows that this is an interesting statement to make. I don’t know if Clive was just oversimplifying this, or not, but in light of his prior statements, it presents a conundrum of sorts. I actually wrote about this back in May.
If you remember what I said about a game loop, and you can grasp the concept of how they work, then it should be clear why this statement is problematic. Everything running within the game loop is subject to the rate at which that game loop is running. Anything (e.g. graphics) within that game loop, and which causes performance issues, will affect everything else within that loop. That could be graphics, physics, AI and yes, even the networking code. Networking isn’t magic that makes it exempt from performance issues – at all. And any performance related component running within that game loop, can and will affect the client’s performance.
Knowing this, it’s possible that Clive is literally throwing his colleagues and the game tech under the bus by claiming that the networking component – his area of work – works fine, as expected etc; but it’s all this other stuff (namely graphics) that’s the primary cause of the client’s horrid performance. But then, you have to now reconcile the fact that if you accept that the networking component is just fine in 3.0, given the game’s shitty networking, then they’re never getting much beyond what they have now regardless of anything they do with the networking component down the road. I’ve been saying this for over a year now; but here we are.
Do you recall the Killer gaming network card? Probably not. Well, read this PC Gamer article about it. And for a more technical one, read the detailed Anandtech review.
4) Netcode does not make servers run slowly, anymore, even though we’ve added more clients
I am regarding this as him doubling down on the previous statement. It also seems to support my theory that he’s pointing the finger at other components which are the source of the on-going issues. This despite the fact that, on any given day, the networking (the primary component in a multiplayer game), is still shit. It’s the classic #notmyproblem type statement that heated arguments are about.
So, if the current 3.0 code base “does not make servers run slowly anymore”, that means it did before – as we know. But since the networking is clearly still shit – even on servers with as few as 8 clients, then they’re well and truly screwed (shocking) because it means they’ve reached the point that all programmers fear. That point where nothing you do in a code component will make a difference.
It also means that, the subliminal message to anyone who hasn’t been paying attention, is that they chose the wrong engine for their baseline, and that even with Lumberyard, they simply can’t go any further without having to start from scratch – or break everything in the process as he stated. Remember, this wasn’t even supposed to be an MMO.
Bonus points: read Valve’s 2001 paper.
Chris Roberts talking about networking, April 2016 (transcript)
5) You get better performance on newer servers because there are fewer players on them so your client has to do less work – like physics, animation, IFCS, and entity updates
6) Players hacking the game to play PU in “offline mode” get better performance than they do online because their clients don’t have to deal with all the load generated by 49 other players
Considering his previous statements, these two are the other conflicting ones. If you have a multiplayer game whereby a client’s performance is so heavily based on the server’s own performance, outside of it being an arbiter of critical network traffic, then you have a very serious problem. Unless the dedicated game server is designed specifically to run a “simulation” of the game, then it’s just a standard client pretending to be a server. And that’s a very serious problem for a game pretending to be an MMO.
The implications of this particular comment by Clive, which, given my own experiences with the game and code, I believe to be true, has far more serious implications than his 29 words could possibly convey.
Remember, this game was never supposed to be an MMO. It was neither pitched, nor designed to be that. And given the significant and insurmountable problems that 3.0 has now laid bare, it’s safe to say that they can’t even get away with a decent standard 16 player client-server game even if they wanted to.
Here’s the thing, you’re never – ever – going to turn what was developed as a standard multiplayer game into an MMO. Period. End of story. The networking component has to be designed for an MMO from the ground up. The yojimbo library which they sponsored, and which some claim they are using isn’t going to do it. The networking component in Lumberyard, isn’t going to do it. And according to Clive’s own statement from this past Oct, they haven’t started anything remotely related to a server mesh yet. So there’s that too.
What is so hard about fixing the performance problems is that the game is pushing the engine way beyond what it was designed to handle. Fixing that means fundamentally changing how systems work while simultaneously trying not to break everything in the game that uses them. Big performance gains that require making big changes take time. Sometimes we have to do a lot of restructuring before we can even start working on an optimisation. Making all these changes can introduce a lot of bugs, and fixing those takes even more time. Let’s also not forget that performance is not the only goal here – we’re also trying to achieve fidelity levels not seen before. Fidelity is often the enemy of performance, so we find ourselves having to optimize even further than we otherwise would have had to.
Yes Clive, we know. It’s almost as if you guys chose the wrong engine to make this game.
And that’s why the project is FUBAR and you guys are never – ever – going to get a “game”, let alone an MMO, out of this train-wreck.
Because it’s related, Clive’s follow-up comment (link)
Having reached zero barrier point, back when I wrote the April 2016 blog about the impending E.L.E, they all knew that the networking layer which needed to be rewritten from scratch, wouldn’t have even made that much difference to the game’s performance issues anyway. The decision about whether or not to scrap it all and build a new custom engine – which the execs didn’t want to entertain – but instead decided to go with Lumberyard – was the defining factor that has hastened the project’s demise. Which was probably the reason behind not disclosing to backers that they were in fact planning to switch from base CryEngine to Lumberyard.
By throwing out all these buzz words related to networking, then making statements indicating that networking was in fact going to not only improve, but would also solve the game’s performance issues, they created a scenario that’s now playing out. It was all lies designed to continue stringing backers along, because that’s how you raise money. You lie to backers, as you would to bankers, investors, business partners, publishers etc.
RIGHT. SO NOW WHAT?
CIG/F42, if you are reading this, as I’m sure that most of you are, as someone who has a lot of experience with this, and who has made a decent living out of complex and ambitious games, spanning over thirty years, my recommendation for 4.0 (3.0.x is screwed anyway, and there’s no saving that branch):
- Forget about making this into an MMO. It’s just not going to happen. You all know this.
- I know that you have a headless server. I also know that the code that runs on that server is already in the client in some form or another. Merge that back into the client. It’s easier to go from server -> client than from client -> server in such a merge. The latter is bad. Don’t do it because everything will break.
- After the above, create a module running on it’s own thread. All it should do is handle network traffic like a server. It’s basically the code you salvaged from the deprecated headless server. You now have a client and server within the same executable. You could, if you wanted to, split that into it’s own executable and spawn it on-demand if you want. More on this later.
- The backend database that handles the client sign-ons, cash shop etc can remain as-is. However, you should plan to be able to disable all of this because this project is going to crash; and when it does, you won’t want to leave backers with an unplayable game. NOTE: Please start using packet encryption for all such traffic. Not going to say more than that; but don’t be surprised to see hacks down the road with players having all kinds of shit they never paid for, nor should have in the first place. Whoever wrote this, did a sloppy job btw. Also, you could adopt something like PlayFab, or even move to SteamWorks. Like Amazon with Lumberyard, let a third-party handle all that crap because they will be around longer than this project will.
- The current matchmaking is sub-par. Get rid of it and implement Lumberyard matchmaking via GameLift. In fact, as luck would have it, Bruce Brown recently wrote a dev article about it. The goal is for Star Citizen Spergs to be able to invite, meet, and play with their friends in expensive chariots, all in the same session, without having to play guessing games. Yes, there’s a party system but like everything else in Star Citizen, it’s half-assed, restrictive, and needs work.
- Then, focus on fine tuning the 16 player experience in this new environment. It’s better to have a LOT of 16 player game sessions in which everyone is having fun, than to have none at all – which is precisely where this project is headed. As you are probably aware, 3.0 is as stable as 2.6.3; and the later is only stable because nobody is playing it.
- Having done all of the above, you can go back and release a headless console server to backers, allowing them to setup their own dedicated servers either on their own, or via third-party services like G-Portal, Ping Perfect etc. Note that this satisfies the private servers promise which, as you know, you’re never going to fulfill if you’re making an MMO. This also cuts down drastically on AWS costs which I know has to be killing you guys.
- With all of the above as the 4.0.x branch, assuming you survive (you won’t) the 6-8 months it will take to implement, go back and draw up a list of everything Chris promised (use ours). Then pick out the ones that at doable within the new non-MMO framework and present it to backers. Scrap everything else. Yes, backers are going to be mad as hell, but it’s better to deliver something, than to deliver nothing.
Furthermore, Chris’s upcoming plan to start pushing SQ42 as the way to bring in new money, aside from the fact that there is no game there either, is going to fail. It’s a non-starter, and it’s not going to bring in enough money to fund a project that’s burning over $3 million a month. So it’s better to spend the next few months building a stable 4.0.x branch so that backers will have something to play and own when the final lights go out.
Everything is fine, but let’s explain anyway. See what I mean? They do this shit – all the time
Yes, my guess is that some of you may have already considered some or all of the above, but was met with resistance because without Chris and Erin selling MMO, the money train will come to an end. But here’s the thing, I have to believe that there are some of you who are “leads” and “execs” who have some humanity and decency left in you, and which should guide you into seeing this whole thing for what it is now: a complete scam that you’re ALL a part of. The road to redemption and forgiveness, always starts with coming clean and admitting that you’ve done the wrong thing. Yes, it’s shameful, it’s distressing, and it’s career breaking, – but you ALL are going to suffer all of those anyway, so why not do the right thing sooner, rather than later?
You’re all so screwed, that I don’t even know how else to express it. And you have Chris Roberts, Erin Roberts, Tony Zurovek, and Sean Tracy to thank for that. When you have an entire company with four studios around the world, and there is nobody to stand up to the creators, you don’t have a leadership, you have a dictatorship. At some point, it has to stop being about a pay check. I know that work in the industry is hard to find, especially in the UK, but at some point, you have to do what is right, and just because there is no excuse for complicity.
Star Citizen is yet another example of why games – regardless of cost – get canceled. If this game were at any developer or publisher, it would have been canceled by now, as it would cost more to fix it because the alternative is that you’re in the territory of diminishing returns.
Jesus, these people were sold the gaming equivalent to a unicorn…
and CIG has brought out a goat with a dildo taped to its forehead – @dzunner