I wrote a thing about instancing over at FDEV
It remains the Holy Grail for online connectivity in terms of twitch games. There is a reason that companies with vast resources, still rely on instanced game sessions - even MMOs.
The Planetside games which are twitch based and tout the largest number of clients in a session, still lagged - badly - when > 32 clients were in the general vicinity. And when they went for the GBWR record for the most clients connected to a session, it was unplayable. The record was about connectivity - not playability.
Eve Online - which isn't twitch based - literally invented a mass of software to host their game. And even so, when it's heavily populated in an area, they use time-dilated updates to keep every one in sync.
The only time that "1000 client instances" makes sense, is if they somehow - automagically solve the n+1 connectivity problem. Considering the clown shoes involved in the project, that's highly unlikely. Again, we're in year 6 and they haven't progressed beyond standard networking in the original CryEngine. So there's that.
The thing with cloud servers like AWS & GCE is that you can do all kinds of nifty things. But they were never designed for the demands of twitch based games. That's why very few use them. Heck, even some of my friends working on games for Microsoft with Azure, are finding this out. See the upcoming Crackdown game.
Basically, you can't have "1000 client instances". What you can have are "1000 client sessions" via inter-instance communications. This - which is basically rocket science - means something like this:
i1(n+250) // instance + client count
i2(n+250)
i3(n+250)
i4(n+250)
Those are 4 are
Amazon EC2 Dedicated Hosts running in Intel Xeon hardware server clusters. Also
see the AMI requirement and
what an EC2 is. You can also use
the free tier to test your app before jumping off a cliff and actually doing it.
This is the part where panic mode sets in. See those instance types, bandwidth caps etc? Yeah.
Without getting technical, with my above example you have a situation whereby they have to create 4 (or more) instances (copies) of the game.
i1 goes live, then gradually fills up with clients. As it gets filled up, because AWS charges for BOTH in/out bandwidth, the more clients, the higher the costs. It's a lot scarier than that.
i2, i3, i4, all go live - same as above.
Nobody in i1 is going to see or interact with anyone in the other instances. Even if you imagine this being a walled off garden whereby i1-client1 is parked on the edge, he will never see i2-client1. They can't see, shoot, or interact with each other. For all intent and purposes they know nothing about each other.
In order to have "1000 client" instances, you need to have 1000 clients in an instance. Which would mean 1000 clients being able to connect and interact with each other in the above. It's
IMPOSSIBLE. Period. End of story. And there isn't a single Xeon hardware server on AWS which would somehow automagically spawn an instance configured for 1000 clients in a twitch based game.
If you "stitch" the instances using clever tricks, such that you have 4 instances each with 250 clients, it's no longer "1000 client" instance, but rather a "1000 client" cluster. And in order to give the illusion of 1000 clients in the world, you have to somehow come up with inter- and intra- instance communications such that, using the walled garden example above, all clients within range can somehow see, chat, engage each other.
Well guess what? Now you're in alchemy territory. You now have an instance whereby i1-client1 fires a missile at i2-client1 and that missile travels through the i1 instance, reaches an area where it is destroyed and appears (re-created) at in i2 at the location of i2-client1 <---- that fool has probably already buggered off, died etc by the time the server code figures out that i1 just fired off a missile at a target in a remote instance which may or may no longer exist.
It gets better. That missile, along with all the calculations for i1-client1 and i2-client1, need to be calculated (God help you if you aren't using server-side arbitration - which by SC isn't using) on-the-fly and in real-time. All the time. Think of the bandwidth.
Now multiply the horrendous notion above to n+1 for a set of clients.
Then plan to be on vacation when the AWS bill shows up for that month.
Here's the hilarious part. Instead of planning to build this from the start, much like Frontier did, they decided to just wing it. And now, six years later, they're still stuck with the basic CryEngine networking layer.
What is even more hilarious is that - right from the start - Chris (it's in the Kickstarter, interviews etc) claimed he wasn't making an MMO. Then, out of the Blue, he was. Suspiciously that was after it dawned on them that they would make more money by selling the entire Verse as an MMO through the sale of assets. They would never - ever - have been able to raise this much money for a single player or session based game. But the fact is, assuming they deliver (which imo they won't) both of these games, the multiplayer is going to remain as it is now. A session based instanced game which will need a witch doctor to get it to handle more than 16 (let alone 1000) clients in combat.
Further reading to see how experts who thought long and hard about this before designing it; but still ended up with a less-than stellar solution to a massive problem:
VERY basic guide for ED networkingAWS re:Invent 2015 | (GAM403) From 0 to 60 Million Player Hours in 400B Star SystemsThis is why most of who do this stuff for a living, and with decades under our belt, simply can't fathom how they could possibly be making these FALSE statements. Especially when you consider that when this whole thing collapses, and the lawsuits start flying, these are the sort of statements that are going to end up coming back to haunt them.
hairychris;5166795]I still have no idea how 1000+ will be technically possible, but I know sod-all about game development.
And that quoted statement doesn't make a huge amount of sense, unless they're having instances ("servers") within instances ("instances") in which case it's still instanced, just called something different.
It's all rubbish tbh.
An "instance" is just a copy of the entire game. It came to be when describing a single server (hardware) running multiple copies (instances). Even a single server running a single copy of the game, is a "dedicated server instance"
And cloud servers are no different, except a GCE|AWS instance is just a software copy running on hardware servers and with no access to physical machines.
e.g. LoD runs only on hardware servers (co-lo at a data center). And we run separate "scenes" (aka levels) each with the
ability to handle 1-256 clients. Each server is powerful enough to handle multiple scenes. So we can run either n+1 space scenes on a server or just 1. In short, the hw server is hosting the instances.
And the way it's all connected is
based on architecture we built specifically so that we could control the number of clients in each scene. So if a scene has a client cap (which is server-side configured), no more clients can connect to it until one client drops or leaves. And all scenes are connected in such a way that it all appears as one universe (though it's just 13 connected scenes
stitched together with magic). A player going from a space scene on one server to a planetary scene on another server, doesn't notice anything, as it's just an IP connection via a jump gate. And during the jump handshake, if the target server is full or off-line, the connection is rejected, you get a message - and you stay were you are and try again later.
Also, a single hw server runs a number of scene instances depending on their resource requirement e.g. space scenes don't have as much stuff as planets; so we can run 2-4 space scenes on 1 server, while running 1-2 planetary scenes on another server. Our scenes are of 3 types. space (x4), planet (x4), interiors station|capital ship (x5).
There is no gain to having 1000 clients in an instance if the performance is just going to suffer, thus yielding a horrid experience for gamers. And even if you did it, the bandwidth costs alone - especially on cloud instances - would be cost prohibitive.
When running based on an architecture like ours, not only do you get around the n+1 client issue, but player-player comms is a non-issue because it's all one universe. You can be in a scene instance (e.g. space) and communicate with someone in another instance (e.g. planet). Sure, you won't see them due to distance and location, but you can still communicate with them. And if by some fluke a 256 scene instance ends up being full, unless all of them are within a certain range bubble, the packets are priorititized based on proximity.
And we don't have the problem of "grouping with friends" because it's all one cohesive universe. No matter where or when you connect, you will always find your friends; and can join them as long as the scene they are on isn't pop-locked.
A small team of renegade indies, led by a semi-retired mad man, built this. In a span of under two years. And it just works. To the extent that, if you look at our
changelog, we haven't messed with networking in over three years. And never underestimate the power of AI bot clients to use for load balancing and testing.
