24 replies to this topic

[Learner]

Currently the Test server & Forums are on low end systems with bandwidth & resource (mostly RAM & CPU) limits where it's been convenient for me. Recently we've had more conflicts with the code development work, IM testing/training, detail map testing, & experimentation with settings that I'm considering if the location & performance of the system used needs to be evaluated:

Factors/considerations:
* Increasing the CPU & RAM can improve performance & allow it to do more
* Should NOT be located new the game server to reduce the chance of issues when BHS connectivity is disrupted
* Cost, we need to keep the costs down to avoid undue burden on players. The current support costs are $200/mth and this would add to the minimum required support (approx $50/mth $40/mth).
* This would NOT affect most of WebAccess. Due to the nature of WebAccess that is handled by the main game server.

Pros:
* A low cost server in France avoids being too close to the same facility are available at low cost
* Can also be used as an alternative proxy to the OL game server for EU players when direct routes are having problems.for the server.
* Code changes will be able to be on it's on 'server' so that people on Test (players/IM's etc) so my teting won't affect them and I don't have to delay until nobody is on to run code updates.
* Once it's setup and running we can evaluate if OL wants to have an official Wiki installed on it (it would have spare capacity still)
* One more possible place away from BHS ro critical backups, like to server source, and the main game setup & players.
* The current server being used is older so a risk of h/w issues plus older software
* The server would have two mirrored drives for reliability.recovery (as does the current system), but the hard disks would be newer.
* Being a leased machine, they will repair it at no additional cost, just give me a headache for my side of restoring operation (from experience if a HD dies down time is required to rebuild).
* Would be at the same version of the OS & tools as is being used for the map/docs/audit server that is located in BHS (the main game server is one major version of the OS older, test & forums is MUCH older currently).
*

Cons:
* Would require additional minimum monthly support of $50.month for a total of $250.month to keep everything running
* Being a leased server if there is not enough support, there is a risk they would wipe the server.
* Higher lag to Test & forums for players in North American and AU (about 80ms - 100ms higher ping times)
* Setup time & downtime while everything is gets transferred.
* The hardware isn't brand new, but much newer then the current server, but it would have a totally fresh OS/compiler install.
* If we want a 'brand new system' so it will be longer before it has issues, we'd have to pay about 50% more which I think we need to avoid.
* Would be tempted to also have it be an EL proxy server since it would have spare capacity and in a very different location. (but EL a 'free' EL proxy can help cover the costs, though not much)
*

This isn't a show stopping issue, just things have come to light over the last year and some recent price cuts on these systems as they brought new ones online makes it more attractive.

After upgrading the OL setup would be:
Game: main game server in Canada to play on
Proxy1: map/docs server in Canada secondary proxy connection to Game, only useful in strange emergencies
Proxy2: test server secondary proxy connection to Game intended for EU player that are experiencing issues
TestProxy: test server in France, unstable when proxy code is being enhanced/debug.
Docs: map/docs server in Canada for developer perm storage of files
Maps: map/docs server in Canada for developer checking of .def files & MapAuditing, also includes amps game server that restarts on a whim with map audits
Dev: test server in France, for code changes and anything that can require a lot of restarts, very unstable. Most players would want to avoid this server
Test: test server in France, for testing changes and maps, effort will be made to reduce restarts
Stage: test server in France, to prepare updates for what will be on Game, won't change often and config will be for what the next update to Game will be
TestUpdates: test server in France, will include downloads for what's in Test to make testing easier (use with caution, can mess up your main maps)
StageUpdates: test server in France, will include downloads for what's in Stage to make testing easier (use with caution, can mess up your main maps when major changes are being prepared)
WWW: test server in France, the Forums, possibly a Wiki in the future.
WebAccess: main server in Canada. Most WebAccess features related to the main game MUST be on the server that is running that at this time. There will be Dev/Test/Stage WebAccess setups for testing though less useful to players.

Updates: will be provided by all three servers, full updates & customs for what is used in Game
backups: All three servers will have critical backups from the others done at various intervals. automatically (some is in place currently, weakest link at the moment is the forums DB isn't backed up often)

Donations just for this upgrade received to date: $85 of the $300 $240 needed.

[AlddrA]

Well darn, not going to enjoy the potential lag, but if it is better, then go for it.....

[Learner]

AlddrA, on 12 July 2017 - 03:02 PM, said:

Well darn, not going to enjoy the potential lag, but if it is better, then go for it.....

That sort of lag is what everyone in North American had before EL moved to where OL is now. it's very livable and will affect Test but not Main. Probably won't notice a difference in the forums since this server is more powerful and lag is less noticeable there.

[butler]

AlddrA, on 12 July 2017 - 03:02 PM, said:

Well darn, not going to enjoy the potential lag, but if it is better, then go for it.....

Just pointing out that EU players will be getting something like 100-130 ping regular on main, which is perfectly survivable, and since your's is normally 23 ald, you will be running at exact rate I tend to on main, so it'll be fine.

Curious, why will it impact aussies that much, and what's there current pings?

My main concern would be wipes. Not as big an issue as the main, but considering development, that could be a shitstorm.

Can i ask how invasions are stored, whilst being done by IM's? If there was a secure way of keeping their progress that would be beneficial.
I'd also assume you'd keep backups of whatever you're working on L?

(these are just questions i'd like answered before I vote)

I would say we need to be told openly what the test servers funds are at, and that we should ensure it be done in blocks.

[Learner]

butler, on 12 July 2017 - 04:08 PM, said:

AlddrA, on 12 July 2017 - 03:02 PM, said:

Well darn, not going to enjoy the potential lag, but if it is better, then go for it.....

Just pointing out that EU players will be getting something like 100-130 ping regular on main, which is perfectly survivable, and since your's is normally 23 ald, you will be running at exact rate I tend to on main, so it'll be fine.

Curious, why will it impact aussies that much, and what's there current pings?

My main concern would be wipes. Not as big an issue as the main, but considering development, that could be a shitstorm.

Can i ask how invasions are stored, whilst being done by IM's? If there was a secure way of keeping their progress that would be beneficial.
I'd also assume you'd keep backups of whatever you're working on L?

(these are just questions i'd like answered before I vote)

I would say we need to be told openly what the test servers funds are at, and that we should ensure it be done in blocks.

AU players are always over 300ms often > 400ms. But on Test issues with lag at worst just help simulate worst cases. I was just trying to present all issues I was aware of at the time I posted this. Remember that EL originally was hosted in France, and on slower servers as well.

Wipes? If we move it shouldn't cause wipes but I may take Forums down for a while so that people can't post to the old forums server and wou.d stay down until the new server was totally in sync and available. Remember from the player point of view this should only affect Test & Forums. For devs we will have additional options due to more resources. For example we currently do have a server on Maps that is auto updated & restarted anytime I run a map audit. Then I coordinate when I move then to Test and restart Test. The extra resources means we'll have an option to have a server that is more stable then Maps, but less stable then Test. It also means I can create a Stage server (intended mostly for me, but others may pop in to verify things) to make sure everything is ready to be published to Main reducing the risk of Main breakage Many places I've done development at  do have 4 different layers of servers, in our case 5 or more may make more sense so the my development is less likely to impact map & .def work.

Invasions are in two different storage's currently. The canned ones are in .def's that have been uploaded, or in RAM via #commands. Any time there is a restart the canned ones are reloaded and the ones in RAM are wiped. If an IM has issued a command what was created in RAM is dumped to a file in a .def format, but it wipes out anything previously so I hesitate to make it automatic. With the size machine I'm looking at, I might just be able to have it save different versions under the name of the IM or automatic that requested it, allowing less conflict (but more URL's for them to remember )

There are backups made regularly as well automatically that are copied to other servers, as well as point in time snapshots made at critical times. I was trying to make the comment about that as lower priority since I do already have some things in place. The server I'm looking at will have 2x 2T or 3T drives so there will be lots of spare space to be willing to allow more backups then I have been.

I don't keep the Test vs Main/Maps funds separate, and so far I've found it tedious to keep updating where we are at and I'm concerned on putting extra pressure on people if it seems we are low. for example when a month does have more then I need, I do carry that over as a reserve to help full for when it's low. The payments for the servers I actually make quarterly to help balance things out in the long run (it was yearly originally, but they removed the 10% discount for yearly, so why bother?)

[Learner]

To clarify some questions people have asked:

1) The proposed server would be with the same ISP as the current Game & Map/Docs server is, just at a different facility in France instead of in Canada like those servers are. The current Test & Forums is being handled near me in MN

2) The proposed server is almost as powerful as the Game server is and would have high speed links to the servers in Canada that don't have to go over the public Internet (public IP address, but OVH has separate internal routing between their major facilities).

3) We do already have lots of backing up going on from Game -> Maps/Docs & Dev -> Maps/Docs as well as sometimes making additional copies to a non-OL machine, and all machines a using RAID 1 disks so one disk failure doesn't mean anything is lost, just hassles and having to rebuild the new driver after they replace. The proposed server just means we can do automatic backups from Game to two servers.

4) Checking stats in detail of available machines in Canada, it would be possible to to replace Game & Map/Docs with less expensive system that weren't available previously and remove about $25/mth in monthly costs. That needs to be a separate discussion on Priorities, but I'm biased to make sure Perks is being tested before touching that. See also http://www.other-lif...assle-vs-costs/

5) Once the proposed server is up as Test, that will have immediate benefits where I won't have to interrupt map/IM testing as often with restarts as I'm upgrading & testing the code. Then I can get Forums installed and copied across to there and work on getting the additional new server environments (Dev/Stage) copied in and setup. That would put Dev/Test/Stage/Forums & Game/Docs backups in France, and leave Game & Map/Docs in Canada all on machines with similar specifications, fast connections & DDOS protection by the ISP

Feel free to ask more questions to full understand anything. We aren't in any sort of rush to make any decisions.

[butler]

Voted yes, but with 6 months assurance, after reading the additional information you provided

[AlddrA]

After a short discussion with Learner about test and development, I can see the advantages of moving the server to France despite the extra cost.  I fully support the move and so should all those that voted to wait 6 months....

The reasons for the move are very advantageous to the progress of the game.

[Learner]

AlddrA, on 14 July 2017 - 10:13 AM, said:

After a short discussion with Learner about test and development, I can see the advantages of moving the server to France despite the extra cost.  I fully support the move and so should all those that voted to wait 6 months....

The reasons for the move are very advantageous to the progress of the game.

To clarify, she was puzzled why the Test server up-time was only 2 hours, It's common for me to have to restart that server for code, settings, or map changes and some days it might have to get restarted as much at 10 times. The proposed server would allow a separate Dev server that most of the restarts would be moved to so that the number of restarts of Test can be greatly reduced. That means my work will impact others less, and their work won't slow down when I need to restart for testing changes.

[CoduX]

I get an average ~340 on a good day, can spike as bad at ~500 (which is playable but annoying).
~340 you can still interact quite quickly with things.

I would love to be able to play at even 200. 23ms is like a dream lol

[Learner]

A big reason reason the Main server is in Canada is to reduce AU lag, and EU lag is still very acceptable

[Learner]

I just added to the first post some details of how OL will be functionally split up between the three servers after doing this change along with a status of Donations received specifically for this project.

[Learner]

I'm currently watching what servers are available at what locations in France since they do change. So far I haven't been the 'best one' show up there. The second best one which this thread is based on seems available in large numbers.

For those into numbers/specs:

Location: Roubaix (RBX) Processor Intel  Xeon E3-1245v2 Cores/Threads 4c/8t Frequency 3.4GHz /3.8GHz RAM 32GB DDR3 1333 MHz Disks SoftRaid  2x2TB   SATA Bandwidth 250 Mbps Traffic Unlimited

All that RAM is the reason it'll be able to handle multiple game servers and the forums. The CPU/Cores combined with the light load on the games means it won't bog down with the forums even with 3 games running. the 2x 2T disks adds reliability and backup space for docs & games while barely making a dent on disk space used. Other smaller options  don't save enough money to be worth the bother unless the 'best one' becomes available.

The 'best one1 is actually faster and costs less, which is why I'm looking to see if that one shows up (might only be available in BHS Canada). There are other faster ones available, but more expensive, so no need to consider those.

[Learner]

Scratch that ... change of plans ... the 'best server' just became available in France Again!!!

Grabbing one right now and paying for one month so they don't run out on me!!! In RBX still
Processor Intel  Xeon E5-1620v2
Cores/Threads 4c/8t
Frequency 3.7GHz /3.9GHz
RAM 32GB DDR3 ECC 1333 MHz
Disks SoftRaid  2x2TB   SATA
Bandwidth 250 Mbps
Traffic Unlimited


Notice the higher clock speed, and that is an E5 chip instead of E3, so more CPU cache ... and will cost less!! This drops OL costs to $240/mth

Update: server software install and data transfer has started, and they already sold out of available servers of this type within 4 hours, so glad I jumped on it when I did.

[Learner]

As part of testing to make sure some things are setup correctly, eu.other-life.com port 443 is now in France and connects players to the game server. The official entry for that server will be added to the server.lst file that goes out via auto-updates later. This will allow EU players to setup a second shortcut to connect to the game via that server as an alternative to the default route for when they are having problems directly connection.

Now back to getting everything needed for Test on to that server

[Learner]

The game server accessed by test.other-life.com has just been switched to the France server. Working on issues with that server being able to build new working versions of the game currently. The ones compiled on the old test are working, so the functionality has been moved to start the process of improving the Test experience while isolating my debugging from Test.

Once the compile issue has been resolved, I'll be working on the setup up needed to move the forums as well as develop & test WebAccess.

[Learner]

Any dev.other-life.com URL's that have been passed out in the past (specially for IM's or WebAccess for Test) are now test.other-life.com . This is part of improving how things are organized under the new server and creating a dev environment that is separate from test. In the near future the dev.other-life.com URL's will be for the dev instance that I will mangle regularly.

Please report any issues with test.other-life.com URL's so I can look into any oddities. There is dev instance up and running currently on the new server to help in testing the setup, but the web portion of it hasn't been brought on line yet since WebAccess needs tuning to better handle this multiple server setup.

Test server is running OL server compiled on the old dev box, While Dev is running OL code compiled on this box.

[Learner]

The full Dev setup including WebAccess is now in place. Next steps are Stage (including putting the next map updates there) and then the forums. Once Stage is up, I'll push out the updated servers.last that people can copy into their install directory so the full configuration is defined.

[Learner]

Automated backups of the Main game server are in place and it had finished full synchronizing. From now on on a daily basis it will synchronize the game server and any backup files that have been created (Maps/Docs also gets these files). The initial copy took a while since old backups aren't normally deleted, so it copied 181G of files. Eventually old archived backups will be deleted.

Automated backups from Maps/Docs for the docs & maps files & scripts is on place and will happen daily. Other files from that server aren't backed up since we have the important ones from other sources.

[Learner]

All proxy's on the France/EU server now automatically restart if they crash and on server hardware reboot as does the Test game server. Stage will be enabled to auto restart as well. Test is now running the version of the server that was being tested on Dev compiled on that machine.

The France/EU server is now running win Xen virtualization. Setup for the Forums has been started.