J HOW
Forum Replies Created
-
Posts
-
Can vouch for the technical issues, some of the things I’ve heard have been terrible from the programming / engineering side. Some I’ve heard has been over the last month or so from people looking for jobs from CIG they applied at my place of work. We are already seeing the negativity at being associated with a failed project as we did not hire any who had applied.
Trouble is because they failed to fix the underlying platform wide issues they now have a severe technical debt. Their engine is practically kaput as no one knows exactly what does what as the experienced staff have already left and no one has full documentation. I was skeptical of various things I had been told about the UK office (more so some than others), I did not think some were physically possible to break Cryengine in so many “wonderful” ways to cause some of the issues they are seeing. The engine is I understand it is layers and layers of tentacles stacked on top of each other as part of “features” added into the engine but no one has looked at how they all work together – assume it’s a complete mess similar to their server and architecture components. To make matters confusing they refer to certain features as modules of the engine and refer to features-of-features also as modules code side.
Some things I couldn’t verify or refuse to believe were:
- Refactor is a keyword within CIG to mean looking at the code, sorry that’s not the word means and having difficulty believing it
- Parts of the modified engine have no documentation as to how it interacts with the base of the engine or the API to how a client interacts with a server and vice versa
- Some branches only compile successfully on certain dev machines and they don’t know why
- Their programmers or engineers usually have to trawl through code to find specific modules which have no standard naming convention
- There is no internal standard naming convention for features-of-features for modules
- Some underlying base code has changed so much some previously working assets and previously working code no longer works (had heard this is specifically to do with the FPS module), pretty much the two options were to either maintain two code bases or start again
- Some days some engineers are disheartened they fail to write any new code and “refactor” old code to maintain a CVS quota (which is apparently an arbitrary number based on commits per-line)
Taking the above with massive pinches of salt it still looks bad, again they are applying for new jobs for a reason so some of it may likely be true but can’t validate it. I’m wondering to see if it matches what you or someone else has.
Never backed Star Citizen, applied for a job there once, the more I think about what they did and how they did it, the more annoyed I am at the whole thing. Whilst my issue happened before Derek had his account refunded I can assure you my experiences with their hiring practices, Derek does have valid points. Think my experiences were just scratching on the surface too.
It’s not just the consumer getting ripped off although that if very bad in of itself. This risks the gaming industry as a whole and when Star Citizen fails it will likely lead to less funding for other games and in the case of game developers – it leaves them without a job and a black smear on the record to whomever worked for them. Worse still depending on how it fails it does risk large wide reaching repercussions.
Even now they are completely changing their server side environment and making breaking changes without an end goal or documentation on what they want to achieve. Depending on who you speak to that is 3 – 4 – 5 years of them going without a plan, without a schedule and without a road map.
Can appreciate the sheer WTFery of the situation and of everyone who has been ripped off. One reason I am now dead set on helping in any way possible is I’ve now experienced first hand (not signed an NDA nor do I want to) on the sheer ineptitude of management towards system operations, devops, and backend. “Make it great, make it support up to 64 players, make it support 64 bit in a clustered environment, make it multi role and multi instance, make it HA capable, make it work without spending too much time on it” The staffers are trying to implement what they are told but what they are told changes on a regular basis and there is no documentation to go with it – apparently there IS a spec sheet but thus far no one has seen sight of the damn thing. They are fighting a losing battle with management, it seems thus far that management do not know what a “cluster” is in terms of server side to even begin to understand why that is impossible in their current setup.
I work within the industry and can say hand on heart that it is NOT going to work within their current architecture or backend to how it has been presented. Suspect this is a reason they have dropped co-op as the attempts to make this work failed drastically to the point absolute failure. Also for the PU testing of adding more players per instances also failed spectacularly, everything was so badly out of sync the instances crashed shortly afterwards.
Twisted irony is the project management team for a games company don’t understand game development, whilst the community spout that those who raise questions and concerns are told by the moderators and community that they themselves don’t understand game development. The madness will have to end eventually and where ever the pieces land, it’s unfortunate that ultimately it will result in the loss of confidence in the gaming sector as a whole.
If they deliver anything of note this year I’ll personally be amazed.
Publicly it appears to be a new article on their netcode here, apart from being a sentence or two it doesn’t make much sense.
“We’re using the Dead Reckoning system which means not sending data that isn’t needed, along with other systems and refining them to ultimately have scalability in the future. The impact on bandwidth won’t be major, but performance on servers and clients is our big focus right now and later on we’ll continue to optimize the network code with a more careful eye when things are looking good on performance.”
Interesting to note they have been experimenting with running Cryengine as a server, headless… Don’t have to explain to you why that’s a bad idea. Initially I thought they were joking. This is an attempt to manage clients in an attempt to fix physics issues, location issues, mapping, 64 bit on the server side. Based on what I can extrapolate they are trying to reverse engineer a new type of server without the graphical head to handle server functions to run from console and to develop it from there. I wish I was joking.
Such a broad subject there is no actual subject matter of specifics. Scalability without changing the fundamental architecture will not achieve results. If anything it’s a waste of time, effort and resources IMHO.
At a loss for words. For this to happen and to continue to happen then it means that they have lost sight of their objectives. To think they have my personal details on record and on file is more than troublesome with that type of cloak and dagger stuff going on. Wonder if I get them removed.
Further to this; the roles in the UK also come with an NDA which seems rather fitting. Not seen this NDA yet but I will share if I get eyes on, don’t remember this being a requirement before for technical roles in the UK. What irks me is that I have NOT reapplied for ANY position yet they must of kept me on file for >6 months to then contact references and then try to ‘vet’ me.
Still struggling to understand whys behind what they are currently doing with recruitment. If the roles are mandated per cost or resource then paying below average will not get great people into the roles. Contacting those that failed recruitment earlier also strikes me as very odd, although I do have theories – I don’t have enough to offer an adequate explanation. Technically they SHOULD be offering a lot more money in the UK, can’t speak for the US roles which I heard they are paying more than average for – can you confirm Derek or anyone else?
Not sure if it’s just me but do other people get the impression they are clutching at straws? Changing architecture, changing staff at this late stage, changing structure, adapting their roles based upon criticism seems REALLY off. As a wise man once said don’t attribute malice which can be explained through incompetence.
Not entirely sure what to think about it. This means that they have very few people on the server side which is particularly bad, also if the open jobs are anything to go by; looking at the job roles they are half way through a re-architecture without staff in place and at a guess – don’t have documentation to go by to continue with new staff, hence getting an architect in at such as a late stage. This paints a much grimmer picture of the current state of affairs than what has been released. This means they have to potentially restart the architecture process all over again.
Normally you would have so many candidates for those specially skilled job roles but no one wants them. Not just because of salary (£15k / year for that role is a complete joke – around $21k) but I’m guessing because of other reasons. I can understand that role at £25k / year minimum with no prior experience – around $35k minimum but certainly not anything less than £22k / year or $31k.
If this is anything to go by you’re more than likely correct, more than 1-2 people have left the company. My gut feeling is a group of staff – possibly on the same team have walked out and left suddenly, they have researched similar roles online and most likely read this thread to repopulate their current staff R&Rs with what they want.
The fact HR are looking through a list of candidates who previously applied for positions ages ago smells of desperation, but without the funding to go with those roles they will prove very difficult to fill. That is not directed at CIG either, a very common saying ‘pay peanuts, get monkeys’.
Today I discovered why they were chasing my references, after many many months. It appears they are now chasing former candidates who had previously applied for positions and got turned down, and are now trying to make offers to those for completely different positions than they applied for. Aware of a couple of people have also been contacted for customer service positions in Manchester UK.
Unofficially they are looking for customer service personnel (who can double as moderators) in the UK. In the UK they are also looking for a backend engineer to keep the servers in order.
Without making it obvious who I am they are also now recruiting for;Cloud Solutions Architect to re-architect their server side infrastructure
They are now trying to recruit a Build Engineer for the server side software, to allow for rapid deployments of their new code on the new infrastructure setup
Senior Server Engineer to listen and implement what the Cloud Solutions Architect says
DevOps Engineer to implement the server / code side between the server engineer and the build engineer.In the UK this is the job role for backend engineer with the new Job Description:
Requirements:
• Strong C++ and multi-threaded programming skills
• Experienced in <span class=”caps”>TCP</span> and <span class=”caps”>UDP</span> networking and creating highly reliable, scalable distributed server systems (This is new)
• Development experience in Windows and Linux
• A passion for making and playing games
• Work well in a cooperative team environmentPluses:
• Online game and/or <span class=”caps”>MMO</span> development experience
• Experience profiling and optimizing server systems (network, <span class=”caps”>CPU</span>, memory, IO) (Also new)
• Experience with client/server development including security and network traffic management (It seems like they are reading this thread)
• Java and/or <span class=”caps”>LAMP</span> (Linux, Apache, MySQL, <span class=”caps”>PHP</span>) development experience
• Experience with Cassandra, Riak, MongoDB, or other NoSQL databases (100% Reading this thread – this is not how their current infrastructure operates)
• Development experience with zeromq, ActiveMQ, and/or RabbitMQ or other messaging systems (Really seems like they are reading this thread – not what their current server side looks like – similar to service bus) – Hello Guys! *Waves*
• Use of Hadoop, Thrift, MapReduce, and/or <span class=”caps”>REST</span>ful <span class=”caps”>API</span>s
• CryEngine development experience
• Space combat sim development experience
• One or more shipped products, especially PC productsJobs are now listed here and they seem to have immense trouble hiring people, maybe why they are looking through past candidates. https://cloudimperiumgames.com/jobs
Trouble is, for the reasons mentioned in this thread, it’s not as easy as hiring new people.
Been thinking of the 64 bit implementation on Star Citizen and how it would impact Star Citizen as an MMO and to be honest struggling to see how it would be implemented. Let’s look at how CIG want to implement their Persistent Universe and how large it is expected to be. Looking on their website I’ve come up with some calculations. I’ve simplified the figures below.
Based on how I knew things worked at the time:
Based upon their 32 bit figures and their universe explorer using 64 bit implementation at minimum the universe or grid in 3d space would be 15.2GB (assuming nothing is moving, and assuming all data is static, this is a very conservative estimate here). The maths behind this is the basic grid of data under 32 bit (this was calculated by CIG for their previous plans as 500MB maximum per instance (to hold the world only when it was literally just the 3d matrix “the world you fly around in”) that was based upon the projection of 50MB per grid, that’s 10 grids at 5000×5000. Now under 64 bit it would be twice the size (100MB per grid – assuming the grid is the same size at 5000×5000), now the universe in the PU map is a grid size of 152, that would mean 100MB * 152 = 15.2GB. This is VERY big.
Now the above is just the world size in 3d space without any objects loaded into memory, assuming nothing is being streamed (unlikely on the server side) this means one instance at minimum would be occupying 15.2GB of memory alone just for the 3d world in co-ordinates, the trouble would come in the form of adding objects into the world (space stations, planets, NPCs), assuming conservative estimates again, let’s say each object has a 32x 32 bit array (under 64 bit) of properties (i.e. IsPlanet, IsNPC, Team, Description, NPCName etc) that would be 8MB per object unstreamed, based upon the size of the world 10,000 entries (again very conservative), 8MB * 10000 = 80GB! Now you can under IsPlanet, IsNPC you cut this down to a boolean after speaking to some devs I know – BUT you are still left with 32 bit arrays taking 64 bit address space. I base the 8MB on the 32 properties set in the cryengine under the modifications for Star Citizen – I would say 4-6MB could be shaved off but I am going by their figures.
Now let’s say for arguments sake you can get past all this (or the above calculations are wrong, or way off), you still have to account for the rest of the server software (connections, overhead, protocols, main API calls, logging, persistent server data held in memory, is AI handled by the server?, overhead), you would add at least 50% to any figure you come up with (if you have any sense so you don’t run out of virtual memory and end up using swap), so going with the above figures 15.2GB + 80GB = 95.2GB + 50% = 142.8GB personally I don’t see a single instance of this being possible on one node.
We have the problem and thinking about the solution, how about we have a number of the grids loaded on a specific instance. i.e. 10 grids per node. Now it becomes possible to host, BUT you have to redo your architecture and your platform. Let’s assume for arguments sake we have done this already. 100MB * 10 = 1GB for the world alone, 80GB / 16 = 5GB for objects + 50% = 9GB per instance BUT over 16 instances, this becomes very expensive, plus it only hosts a number of players per instance with a cap – let’s assume we add 16 players per instance, how many instances would we need to allow 100 players to play in the same area at any one time? It would be 6.25 duplicate instance of the same grids, say 6 to round that number down (conservative)
16 instances for the entire world * 6 instances for the amount of players in each area = 96 instances (very very expensive) considering the memory requirements for such a setting would be 96 * 142.8GB = 13.71TB and this is for less than 100 players in each instance, limited to 96 instances.
I cannot see how this would be possible without massively scaling down the persistent universe size or using nasty workarounds like streaming data from disk on the server (very bad (may lose data)), or storing areas where the players are not in to hard disk (slow), or running from SSD storage to stream to memory where needed (sensible but would be prohibitively expensive). It would also be very costly to maintain.
