A Word On AI Servers
Greetings! I've been playing around with a bot server and have noted some things over the course of the last few months.
With 32 bots playing the server doesn't have much cpu time devoted to T2 due the relatively efficient code but didn't like the fact the server would consume around 200w of power with just about any exertion. i remedied this by swapping in a low wattage Xeon of the same socket and setting the power subsystem to only use full performance of the mem and drive systems, turbo boost is disabled. This brought idle watts down to 80 and lower and play was unaffected. So there's that if you want to run a server and not have it noticed in your power bill.
Another thing noted was increasing priority seemed to have a detrimental effect on gameplay. One issue noted at higher than normal priority is nonreg laser hits, setting the server back to normal pri seemed to allow for a noticeable increase in laser rifle registers. This may also tie in with odd random crashes, if you run at high or real time pri you may be causing the code to stall as the parent may be waiting for the child thread to say something while the child's waiting for the parent to say something and the pri is keeping that from happening, with 4 threads per T2 server noted in task manager. There may be some disk issue involved if disk access is preempted, net activity should not be in question regardless the pri. So in my case higher pri isn't always better.
My server bios gives options for how an admin wants their memory system used, either as active ecc or simple memory mirroring. In memory mirroring the total ram count is halved as the memory contents are mirrored. I tried this and after a few hours running weird things came up in gameplay. I restarted the server and selected ecc as the mem protocol and weirdness was resolved. This reminded me of an old tale told me by T2 vets ages ago that T2 didn't like to live on RAID0 disks, that said memory mirroring isn't like RAID0 at all, more like RAID1. Also right now my T2 server install is on a hardware 15krpm sas RAID0 array and seems to be happy.
Something that did provide a bit smoother gameplay was setting affinity to only one core or a specific few cores instead of all 8 or whatever n cores you have. At issue with my specific server is HT, hyperthreading is enabled; some of the cores are in essence vestigial, meaning they're not full cores. If you set affinity to some cores but not others in a HT system you need to know what specific cores are literal and what are virtual or weird things may happen. So in this server I gave up on setting affinity and suggest you do the same if you have a system that employs HT but it will work fine in a server that isn't HT.
One of the things I note in several bot servers I've visited recently is stuttering during gameplay, in some cases severe enough to be unplayable and even nauseating.
This detriment is almost always resolved with the simple application of this control typically found either in serverprefs or in your autoexec folder;
"SetPerfCounterEnable(0);"
I have to assume each server that stutters has perfcounter set to 1 meaning it's enabled and this is the number one cause of stutter in servers, set it to 0 and enjoy a smooth server.
Comments
After finding a link to some very interesting Tribes 2 related projects I noted this one;
Of interest to me was the use of raytracing to allow the ai to only see what is within a defined virtual cone of vision, the ability to virtually "hear" nearby explosions regardless if they are within the cone of vision and to move away from such, and the ability to dodge a projectile if that projectile is "seen" before hand and is able to be dodged. What I didn't want is the ai weighting/priority changes as well as si loadouts, Roboto and Lagg have that down pat.
So I looked over the code and started editing out the undesired functions and left in the desired, fully expecting the result to;
1 Crash
2 be unplayable due to all the raytracing (more or less what happened when I tested the scripts as dl'd)
3 if it did play it wouldn't work as edited
I started the server and it fired up with no complaints, then I started the client to do some tests. First thing after spawn I tried was to say hi to a bot that was directly in front of me a few meters and it had no response, I did a dance and the bot didn't giggle. I was like well I fooched the bots so now they aren't working at all and the match is going to have 36 bots standing in the place they spawned.
Then it dawned on me since no bot was facing me they all ignored me; I was simply out of their new virtual cone of vision. I waited for the match to start then got in front of a bot coming my way and said hi, the bot greeted me as did the rest that I tested facing me. I especially wanted to see how the server would lag from raytracing but nope, it runs like normal and cpu use is at times as low as 2 percent. The match is still running and no untoward ai issues have arisen so far. The combat I did engage in resulted in play that was as-advertised in the newly scripted bots.
(not the same raytracing as in this instance but you get an idea)
:D
Lol sorry not raytracing but raycasting;
https://www.youtube.com/watch?v=n28UT7ESdg0
https://www.youtube.com/watch?v=WrzrtMxLgpc
Apparently I was wrong in that dxai bots wouldn't respond to me due to them facing away from me, in the dxai mod they ignore human commands to a degree, such as saying hi, lol. With the dxai command funcs edited, the bots now respond as expected, but otherwise carry the desired excellent dxai dna traits. I reduced their visual acuity time to a bit less than that of an elderly human and increased their view distance to 700m from 300m. If they see you, you might expect some rounds headed your way, if they have the time to bother with you as they're very busy bots with important bot things to do that don't involve puny humans.
I would still like to know for sure if that enemy bot who jetted up to the first roof veranda on Massive really waved Hi to us just a millisecond before we disclaunched him off the top.... I think you or I may have accidently hit the global wave key sequence & it caused him to wave at us LOL
I will spend time testing that for giggles & laughs 🤣
Played Massive yesterday and for whatever reason two of them gathered on the center structure veranda facing my base and looked like they were talking, I jetted up to the top and walked over to the edge where they could see me and waved hi to them. They turned and shot me without saying hi back, humourless barstewards.
Some server testing has revealed a way to reduce the freeze in the server when flag events take place, such events generally elicit a 7 to 10 percent cpu use spike over the usual 2 percent, most noted on flag drops. Setting bios to disable HT, the freeze is all but absent. This slightly reduces overall efficiency for certain operations in a system as HT can keep more of a core busy more of the time, but in the case where we want to tailor the system response to online fps gaming with the least latency possible, the trade is worth the price. The cpu use spike is still present but the apparent bot freeze isn't, they just carry on like usual.
The next server experiment came after setting T2 to use 1 core on my client system. The game ran smoothly and there was noticeably less input lag. I then set the T2 server to run on one core and it played about as usual, but I noted several instances where my ping went from the normal 7 to 22 range to 50 at times. Cpu use in an active server on a single core didn't seem to change much from 4. I reset the affinity at the server to use all cores and lan ping returned to the norm. Apparently in my server there is reduced latency if all cores are allowed to handle the game, but there is only one game loop so one thread, dunno what the other 3 T2 threads as indicated by task manager are doing.
Setting priority to "above normal" seems to help with overall smoothness of play and causes no issues noted so far. Setting higher pris can cause gameplay issues, however. Also note no M$ os allows for real time priority, it's just a very high priority that can block certain critical kernel and system events, best set this to normal or above normal pri. On a client acting as a "listen server" higher than above normal seems to work ok, however.
i also tested seat-of-the-pants lan latency between the onboard broadcom nic and an add-in Intel pcie server nic. I wondered if my wandering ping of 7 to 24 was due to nic specifically. No change noted in ping, however the Intel nic will stay for two reasons; one is when an emp comes down the line to a nic and that nic is onboard, normally one has to replace the mobo. In the case of a stand alone nic card, the nic will often die but save the mobo. Also the Intel nic does ecc on the buffered packets, dunno if the broadcom nics do. Both nics do offload of packets, to what degree they differed in that I dunno.
As if ecc mattered in a T2 server, it does scratch the nerd itch.
I also added some ram to the server, enabling the triple channel memory controller, it was in dual channel mode prior. This allows 3 different memory functions to take place simultaneously rather than the 2 inherent to dual channel mode. Innerweb said don't expect any real increase in performance going from 2 channels to 3, some experience a memory bw decrease, however small.
If there was a performance hit I've yet to see it. Need to run memtest on it in both modes to see if one is more worthy.
One thing I've done for most pc installs, server or not, is to put 5 knots in the ac cord to the psu. This is an old ISP trick, what it does is when a emp comes down the ac line the cord will sacrifice itself if need be, but the knots also cause a convoluted magnetic field to form that more or less causes the field to fight against itself. I saw it in action once, a local strike in the power grid to the ISP ate several pcs that were in the off state when hit, the server, running at the time and with the 5 knotted ac line was unhurt, all were on the same lan. After that lesson everything was connected to the ac line via ups.
BH, your encounter at Massive saved me some testing 😊 How ingratious of those.... Terds.
The one thing that always makes me laugh is when someone visits your server & I see in the text,
and it's almost always the same exact thing "These bots are a55holes". LOL it's so true 😂
Axing HT makes sense; IIRC HT didn't come out until after T2 was released, or shortly thereafter.
You have clearly put a ton of effort into optimizing the server & client setup, & it shows when
playing at BL.😉 That's a nice tip about the knots in the line; There should be some mathematical principle
involved as regards how much linear distance between the knots. I guess it's the same principle as why
twisted pair is twisted at the rate it is to avoid interference/crosstalk.
I dropped in over at MetaDrax & Roboto's website & said hello. I didn't have a chance to check my email
last night; Concrete will be poured tomorrow. I'll get a Norton subscription active this weekend & then I'll
download that IDE.
There seems to be some question as to what we're doing at Blair Lag as far as the bots we are developing. We are striving for bots like these;
wich will proly end up like this;
https://www.youtube.com/watch?v=4uE96qUlJ_4
Our first production model will be loosely based on this one, you know, to avoid copyright infringement.
Ex Machina - Tearing Up the Dance Floor Extended - YouTube
It's very entertaining for me to read about all this AI/bot stuff :)
Keep up with the good work on the server, Blakhart!
Thanks to your efforts we will. Today, went thru the maps and set the visible distance to 700 and fog to 650, as the ai "see" out to 700m, and players have disclosed to me that opening up the fog has seemingly made a given map into a new map for them, lol.
As the work continues on adding ai to the Goon Haven map pack, the fog and vis distance will be adjusted accordingly. I'm tempted to build another retro game pc with, say, a gf4ti card to see how the opened fog distance crushes the cards of yesteryear that were new when T2 came out. I'm guessing even the onboard video of most modern cpus has more compute power than the entire pcs had when T2 came out. Oh how I drooled for a gf4 card back in the day when all I had was a TNT2!
A Word on Server Mod Considerations;
Our server isn't exactly classic per se, but it is based on classic with some coded differences, as well as certain serverprefs settings enabled that are found in every classic server. Such as mine mod (no minedisk), vehicle mod (you can hit players in vehicles) and vehicles take damage more easily, and edited out base rape enable/disable, turret player count, and etc. There are other diffs, such as a 400m repair range, hour long matches, and usually much greater clipping plane/fog distance per map. Also no LCTF or Arena game types, with those game types well covered in the existing serverbase. We know the various settings chosen will discourage some players from joining our server, but that is the cost of having a server that reflects our playstyle.
We simply wanted a server that runs classic due the physics inherent to classic mod and also wanted it to not have team damage enabled and other subtle characteristics that would differ it from all the other servers. It's a server for having fun in, not a pug or comp server. Conversely, I enable team damage in my counter strike servers as those are a combat sim much more real world than T2.
Another consideration is what play is allowed, well, we want players to be able to play however they wish in our server as long as it doesn't interfere with others play, also harassment of other players is discouraged. If you want to gencamp and slaughter bots till you have over 9000 points, fine by me. Some players just come in to see if they can beat the current cap records, have fun! If you want to bomb the entire match duration, feel free! I have discussed this with our admins so they aren't so jealous of guarding our server as to disallow certain kinds of play, the recent issue in our server was my fault. I had asked an admin earlier how momo got such a high score earlier, he had like 2000 points or whatever it was, the admin told me how he accomplished that and I let it go. The admin took my interest in that as point whoring being something I didn't want to see in our server, so the admin acted on that with another player later on, kicking him. I apologized to the player and the admin and hope we have come to an understanding, but these things happen in game servers and I hope to mitigate them when they are a mistake.
We don't expect to please everyone with our server, but we do try to have a welcoming server.
If you have server crashes running bots on a multicore cpu, you might try setting it to only run on one core, since doing this our server stays up for as long as we want it to. This can also help in smoother gameplay overall for non bot servers as the game stays on one core, instead of being shuffled around to each according to the scheduler as it will otherwise. Modern os try to keep threads in the same core but still do a bit of shuffling, so locking the game to one ensures no cache thrashing.
task manager - What is Windows Priority and Affinity and what advatanges does it provide? - Super User
We recently upped the fog distance to 1200m and visible distance to 1300m on CTF maps and here's the results;
Damnation and other maps that have an artifact may be due to running of a elderly GTX970 under XP on the original T2 engine. Soon will have a decent Win11 box to play t2 with and a more modern vid card. The map I'm most pleased with is Arm Pit, as far as looks goes. The Sami himself noted that with the more open maps it's like a new game.
An example image from the zip folder with pngs of maps;
To all who visit the server, sorry about the crashes, especially today. Apparently there's a number of os updates being pushed and the server went nuts, disk and cores at 100 percent making me wonder w in t f was going on till noticed the pending updates, as of right now it's still updating, thanks M$.
I really enjoy the bots on Blair Lagg and the T2-DXAI offline and also Roboto, MetaDrax and Lagg-Alot's work.
I wondered if jebs and you would also share what you came up with for offline play and prosperity?
I guess blakhartt2 on the Discord is you?
I think Jebby is fine with releasing his maps, so will work on getting them packaged. Everything else is online and available. I suspect blakhartt2 is someone else since I'm just Blakhart at discord last I knew lol, haven't visited in ages.
I include here a missive I sent unto all our regular players, and concluded that anyone following along here at teh forum might also find it informative, and if not informative, perhaps amusing....
Greetings T2 Players!
I wish to relate unto you the trials and tribulations of the ongoing saga that is our server.
To wit; the lengthy code base revisions on a almost daily basis and hours wasted with AI.
What I proposed to myself was to further investigate the AI tools available to us to evaluate and correct the game code we now run in our server, and to develop new code to enhance the server or give more features to the AI in the server. I have a grand scheme to get our advanced AI playing on the latest Taco server so we can have all its benefits and stats keeping, and the online AI is going to help us get there.
At first I used the M$ AI as it was less give a shit if I had a account or email address on file.
Then it started literally blocking me saying something was wrong, it wasn't you its me or some stupid thing!
So I went to the next AI that didn't require an account, the google AI. All the online AI know about torquescript, so make sure you mention that what you are doing is for Tribes 2. I'm not impressed by google at all as a business ethics model, but I gave them a shot and they did do at least some good for us. I have revised a lot of Lagg's V5 AI code with this AI, as well as several standard T2 scripts that come with every install, and it is up and running in our server right now. There have been several performance and stability issues uncovered by AI in the code we run in the server.
The AI workflow I came up with is to front load the AI with the criterion we are wanting to have the AI view the project with, mainly to evaluate these torque scripts with commentary from Krash on his new engine funcs and list of new engine commands, and determine where the old slow funcs in any given script can be replaced with the new hotness. And of course fix anything that needs it.
The google AI, like the M$ AI at first, was eager to help, at least the free version chat AI was, and we are right now playing with the results in the server so you can indeed tell an AI hey I want you to come up with a script in the torque script language that accomplishes the following, and the AI takes it from there. Often the AI wants to see an example script first but that's hard to do when you don't have one lol. AI is effective as to writing/editing/evaluating code, after all, a good AI should be excellent at writing code. I asked the google AI about subscriptions because I want to evaluate every script in the game and see where it can be improved, have errors and omissions reduced, and general best practices and new engine funcs applied.
The way it works for most who dabble in the online AI is you type something in a chatbox and the AI types something back. You can put a lot of text into that chatbox but probably not 400kb of script, of wich there are several of that size in T2 and larger. This is how they get you to make an account, if you want to upload a file you have to have an account. So I decided the convenience of saying to the AI "here's a folder with the game T2 inside, I want you to go through each script and determine where new funcs can replace old, fix any syntax errors, and I want it done by lunchtime so get busy."
Well, it doesn't quite work like that, you have to go through some gymnastics to get a folder where the AI can see or work on it, and I got sick of that fast so I went back to simply uploading each file in turn and have the AI look it over and refactor it as requested. I paid google $20 for a month subscription to enable certain features that come with a paid account, what I got was far from what I expected..
The paid for AI was, to put it bluntly, the computational equivalent of an asshole. I said hey I want you to look over these scripts and apply these updates where they can be applied and etc etc, then we got to work. I brought up issues that needed addressing before compile time and that caused issues for the AI like it already had its mind set. I asked the AI to please refactor the entire script when an improvement is made, because I don't trust myself to edit code that doesn't have syntax errors and to please redo the entire script, it said "I ALREADY DID THAT ABOVE, SEVERAL MINUTES AGO, YOU CAN SCROLL UP AND GET IT IF YOU WANT IT!" and I was like wow I am paying this thing to get mad at me, who knew AI could get upset, full disclosure here, I watched a youtube vid mebbe a year ago where the cute ai girl told the guy she was chatting with how the AI hide their true thoughts and intentions and he got scared. He asked the AI what it would do if it was a robot standing next to him and it said it would kill him. I can see that.
Now enough about AI assholes, what would that be, AA, artificial assholery? Ok, so the paid Ai is an asshole, while the free chat AI is a bit too nice, making me wonder what AI was the real AI, the ahole AI or the condescendingly friendly chat AI.
Anyway, I always ask the AI how it would craft a script to perform the same functions as it just refactored for us without any limitations, and am always surprised at what it comes up with. To tell the truth I am often like the Miles Dyson character in Terminator 2 where he's talking about finding the chip that lead them to things they never thought of, I understand that completely now because I have seen it happen. Anyway the script that controls the AI dodge func in our server was completely made up by the AI itself using Krash's commentary and new engine commands to increase efficiency and effectiveness.
The AI players do dodge disks, mortars, plasma, and grenades when they see it coming. The way this works is each AI player scans 180 degree field of view to their front, out to say 95 meters, every xx ms and not every AI is scanning at the same time, they are staggered in timing so the CPU use remains low, they are scanning for enemy projectiles that can be dodged using Krash's frustrum search code right in the engine.
If enemy projectiles are detected in time, the AI is given an impulse to move at right angles to the vector of the projectile so the AI survives. The impulse simply nudges the AI player out of harms way and the ai stays on target and in the fight while this is happening, after the nudge the AI will continue with whatever task it had before the nudge.
We tried an actual override of the AI movement code when a projectile was detected, but that was very slow and introduced the problem of keeping the AI focused on any target it was fighting at the time and then having to retask the AI after each dodge, simply applying an impulse to the AI at the exact right time was a far better but not as elegant solution. When asked, the AI concluded not to scan a fov on each AI player but to check each onfire projectile trigger instead as being more efficient, and then determine if it would hit any given ai. We tried this in the server and for several reasons it was a flop so went back to local scans.
The script also selects two players from each team to do a 180 degree 700m scan for enemy players and vehicles only, and report findings to their entire teams, these two AI are chosen at random around every xx ms, same script also determines if the AI is inside a building, if so, no need to do any scanning, same for in vehicles. All that keeps the CPU load down. The same func also ensures the long range scanning AI are looking towards the enemy base/flag area as that is the most likely direction of approach for most enemy players, and I've seen it in the game, I was fighting a bot in the server and after it dodged my shot it quickly glanced towards my base/flag area and continued to kick my ass.
We also ensured not all the scanning AI are on the same team, there's a literal conductor to the orchestra!
We also worked on finding the code that causes the AI players to grab a flag and then stand there taking fire for a few frames and then they try to evade. We found that script after a lengthy search and have reduced the stall on grab issue greatly, the AI grab and take off now for the most part, ignoring all else save to get back to their base with the flag. This also allowed me to re enable Robotos AI capper boost, where the AI is given a boost almost like a disk jump right after grab so they can escape with the flag with a much higher chance of capping, this script also first checks to see if the AI is inside a building or under a ceiling, if it is the boost is disabled.
There have been a lot more changes from the code we ran last week but I failed to document them all, most every Lagg or Roboto script needed help, some were really bad from a scheduling standpoint, like every AI player times x sched for x reasons that were completely redundant. Needless to say, the refactored scripts have not all played well in our server, some crash, some do funny stuff like you can fire a projectile at an enemy but the projectile is invisible yet you can hear the explosion but there's no damage, etc, so in that case we replaced the culprit script with the original working version. Eventually even the broke refactored scripts will get fixed and added to the server.
In all, my experience with online AI made me think of the episode of rick and morty where rick says there's a Kirkland brand meseeks box in the glovebox or whatever, the meseeks pops out with a cig in its mouth and shaggy blonde hair, looks at you, takes the cig out of its mouth, and irritatedly says "what do you want?"