Shoveling Shit Algorithm

-=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- (c) WidthPadding Industries 1987 0\|445\|0 -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=- -=+=-
Socoder -> Concept/Design -> Shoveling Shit Algorithm
Mon, 12 Dec 2022, 06:50
Afr0	Shoveling Shit Algorithm (Redesigning MMO networking architecture) Not sure I’m looking for comments on this. Please respond if you have some answers to the questions I’ll inevitably be asking, I guess. I had an idea today for redesigning my low level networking architecture to make it idiot proof. The following is me thinking out loud about technical design. And stuff. Hmmm…. So step one here is the simplest part. 1. When you’ve asynchronously received some data, keep shoveling shit into a processing buffer. 2. This requires me to make a ProcessingBuffer class, which is the novel bit, as it allows me to separate the logic for receiving the data from the packet processing/construction logic. 3. When shit is added to the buffer, make that function check to make sure the buffer contains a maximum packet size’s worth of data. I think 1024 bytes should do it, for now. IF the buffer already contains data, then the new data needs to be added to the end of the buffer. 4. If a maximum packet size’s worth of data has been received, read the header (first three bytes), then construct a PacketStream instance with a header’s worth of bytes and call the handler for the packet. 5. This all seems too simple. Surely I must have glossed over something??? 6. Is this thread safe???? Hmmmmm, in order for this to be thread safe, I need to make sure that the internal buffer is always locked by the AddShit() function of the the ProcessingBuffer. Hmmm, what data structure would be best to use for a buffer that takes data from the front and enqueues from the back??? -=-=- Afr0 Games Project Dollhouse on Github - Please fork!
Mon, 12 Dec 2022, 08:03
Jayenkai	Depending on the language, different methods would be used. If you can get the ID of the thread, then make a ProcessingBuffer[ThreadID] , remembering to clear it out afterwards. If you can trigger events once the element is ready, you can automate a lot of this stuff. And, yes, that would be quite simple. If not, then make a ProcessingBuffer[] array with, say 100 elements or so.. (I would imaging 100 is a large enough number, but .. experiment!) Have a NextProcessingBuffer=0 variable, and when it starts getting some data.. NextProcessingBuffer=(NextProcessingBuffer+1) % MaxBuffers; Then place data into ProcessingBuffer[NextProcessingBuffer] Every nth loop of the game or so, scan through these, and if ProcessingBuffer[n] is ready, do whatever it is you're doing with it, then clear it out again. But like I say, this pretty much depends on what language you're using, and what kind of threading abilities it has. -=-=- Additional, if you NEED the data in the right order, then have a LastProcessedBuffer variable, and only ever check if the one after that is ready. Then bump it up. -=-=- ''Load, Next List!''
Mon, 12 Dec 2022, 09:07
Afr0	Thanks! TCP guarantees in-order delivery so that’s not a concern. -=-=- Afr0 Games Project Dollhouse on Github - Please fork!
Mon, 12 Dec 2022, 17:33
shroom_monk	This all seems too simple. Surely I must have glossed over something??? Grabbing the data as it arrives and putting it into a buffer, as your algorithm does, works fine and should be pretty straightforward, as you've found - I don't think you've missed anything here. But my understanding is that the hard part of networking isn't getting the data, but rather how you handle all the weird ways that data might differ between the various clients of your game - how you compensate for lag, dropped packets, etc. In particular... TCP guarantees in-order delivery so that’s not a concern. I suspect this is the bit you might want to revisit. TCP does indeed handle reliability, in-order delivery, etc all for you, but that doesn't come for free. If your MMO is remotely real time, you'll probably hit problems sooner or later where dropped packets cause your entire network stream to stall, because the later parts cannot be delivered until the dropped packet is re-sent. UDP, on the other hand, doesn't give you any of those guarantees - you have to handle them yourself - but can be much more responsive in the case of dropped packets. In a real time game, you usually only care about the latest data (e.g. latest position of characters), so you can just ignore dropped packets and only concern yourself with the latest values. This does require you to maintain your own sequence order of course but that should be fairly easy. TLDR: TCP works, but you might hit performance problems down the line. If you're taking the opportunity to redesign your low-level networking architecture now, you might wish to consider UDP! -=-=- A mushroom a day keeps the doctor away... Keep It Simple, Shroom!
Tue, 13 Dec 2022, 05:14
Afr0	Thanks for your input! I never learned UDP. And there's a reason for that: World of Warcraft and Guild Wars use TCP. If you want a fast-paced game (even with fewer clients), UDP should be used. But TCP is more than good enough for most RPGs (even with real-time combat), especially if you interpolate. Given that this networking architecture is first and foremost to be used for The Sims Online (at least for now), there's absolutely no need for the added complexity of UDP, as TSO is a very slow-paced game. That said, I will be looking to learn UDP in the future! -=-=- Afr0 Games Project Dollhouse on Github - Please fork!
Tue, 13 Dec 2022, 05:53
Afr0	Ok. So I ended up having to make the algorithm slightly more complex, but now it just seems to work. using System; using System.Collections.Generic; using System.Linq; using System.Runtime.CompilerServices; using System.Text; using System.Threading.Tasks; namespace GonzoNet { delegate void ProcessedPacketDelegate(PacketStream Packet); /// <summary> /// A buffer for processing received data, turning it into individual PacketStream instances. /// </summary> internal class ProcessingBuffer { public static int MAX_PACKET_SIZE = 1024; private static object m_BufferLock = new object(); private Queue<byte> m_InternalBuffer = new Queue<byte>(); private object m_HeaderLock = new object(); private bool m_HasReadHeader = false; private byte m_CurrentID; //ID of current packet. private ushort m_CurrentLength; //Length of current packet. public event ProcessedPacketDelegate OnProcessedPacket; Task ProcessingTask; public ProcessingBuffer() { ProcessingTask = Task.Run(() => { while (true) { if (m_InternalBuffer.Count >= (int)PacketHeaders.UNENCRYPTED) { if (!m_HasReadHeader) { lock (m_BufferLock) { m_CurrentID = m_InternalBuffer.Dequeue(); byte[] LengthBuf = new byte[2]; for (int i = 0; i < LengthBuf.Length; i++) LengthBuf[i] = m_InternalBuffer.Dequeue(); m_CurrentLength = BitConverter.ToUInt16(LengthBuf, 0); } if (m_CurrentLength == 0) //TODO: Fail gracefully if no handler is registered... m_CurrentLength = PacketHandlers.Get(m_CurrentID).Length; lock (m_HeaderLock) m_HasReadHeader = true; } } if (m_HasReadHeader == true) { //Hurray, enough shit was shoveled into the buffer that we have a new packet! if (m_InternalBuffer.Count >= (m_CurrentLength - 3)) { byte[] PacketData = new byte[m_CurrentLength - 3]; //3 bytes is the size of the header. lock (m_BufferLock) { for (int i = 0; i < PacketData.Length; i++) PacketData[i] = m_InternalBuffer.Dequeue(); } lock (m_HeaderLock) m_HasReadHeader = false; PacketStream Packet = new PacketStream(m_CurrentID, m_CurrentLength, PacketData); OnProcessedPacket(Packet); } } } }); } /// <summary> /// Shovels shit (data) into the buffer. /// </summary> /// <param name="Data">The data to add. Needs to be no bigger than MAX_PACKET_SIZE!</param> public void AddData(byte[] Data) { //This protects us from attacks with malicious packets that specify a header size of several gigs... if (Data.Length > MAX_PACKET_SIZE) throw new Exception("Tried adding too much data to ProcessingBuffer!"); lock (m_BufferLock) { for (int i = 0; i < Data.Length; i++) m_InternalBuffer.Enqueue(Data[i]); } } } } --v Here's the rundown if you want to implement it in a different language: 1. When the ProcessingBuffer is instantiated, launch a thread (or a task, or an asynchronous operation, or what have you) that checks if a header has been read. If it hasn't, and we've received a header's worth of data, then read the header and set the flag that indicates that the header's been read. Make sure to lock on this flag and the buffer when reading from it! 2. In the same thread, make sure to check if we've received a packet's length of data according to the header! If we have, dequeue that number of bytes (minus the header) and tell your client (or server) that you've received a new packet! 3. When your socket receives data, shovel that shit into the buffer as fast as fuck! Lock the buffer while doing that. 4. That's literally it! -=-=- Afr0 Games Project Dollhouse on Github - Please fork!
Tue, 13 Dec 2022, 06:26
Jayenkai	\|edit\| Trimmed the title on the sidebar. It was annoying me being so big \|edit\| -=-=- ''Load, Next List!''

Shoveling Shit Algorithm (Redesigning MMO networking architecture)