Generic Architecture

[daeken_bb]

Ok, here are some thoughts I've been having. We have enough experience in the development team to pull all of this off, even if it is a bit more difficult. All of these things will make the project easier to maintain without it, well, turning into EQEmu lol. Let me know what you think.

1) We should make the architecture layered and generic, that way even if sony completely breaks the netcode, we don't have to rewrite from scratch. E.g. we split the code into 3 layers, the netcode, the abstraction, and then the modules that do the actual work... perhaps even a 4th layer that adds additional functionality, depending on the module.

2) Everything should be 100% modular. If this is done right, it means that we can use modules from other languages as well, which could mean greater efficiency (e.g. using C++ for one) or greater control while testing (e.g. using Python). I don't think I have to speak on the importance of modularizing code.

3) Database code should be 100% abstracted and modular, ADODB-style. Even though we are all using much the same data, certain RDBMS's are more well suited for certain loads or even architectures.

4) WindCatcher and I have been discussing how network code can be completely high-level (e.g. a networking metalanguage) without losing speed. This would give us the advantage of being able to change structs without recompiling (if in dynamic mode) or make changing from one structure to a new one a trivial task. Endianness would also be a non-issue with this system. I will post more on this concept in a seperate thread.

5) Modules should be capable of processing in parallel. We should have a thread pool, and when we get a request, we should start the appropriate module's handler in an open thread from the pool. This will allow us to process unrelated requests in parallel, and speed things up both on uni-processor and SMP machines. (Thanks to Image for bringing this up)

If you have any ideas, I'd love to hear them.

Thanks.
Lord Daeken M. BlackBlade
(Cody Brocious)


Edit #1: Added #5.

[RangerDown]

The cynic in me will remind you that he's heard these very mantras repeated, multiple times, throughout his tenure at EQEmu -- and while some strides have been made to achieve that goal, it still comes nowhere close to your ideal level of modularization. Database code is not modularized in any way that could even come close to supporting an alternate database provider. Point: there's ideals, then there's the real world.

That said, you're starting fresh with a new code base so you don't have the level of baggage EQEmu has. You have a better chance of making that happen here.

Here's the real problem you're going to run into:

• During EQEmu's prime, it was blessed with literally a dozen or more knowledgeable developers who were able to write to the CVS. Hence what a boon to have them find and fix errors in others' code -- and contribute new features of their own -- quickly and easily.
• During that same time period, EQEmu was cursed by having literally a dozen or more know-it-all developers who had direct access to clobber any code in the CVS. Hence, what a headache to find that some other dev -- who only knows enough about that part of server code to be dangerous -- totally rewrote the module you'd introduced only a week before. Not to mention every time a dev announced they were about to put in a new feature, you literally stayed up that night with anxiety -- just KNOWING their implementation of the feature would be completely short sighted and make absolutely no effort to work properly with the rest of the server code.

This isn't a technical problem at all, it's a people problem. Coders like yourself who are terrific at single-handed tasks lots of times tend to bump heads when they have to watch out for others' code. I'm not saying you specifically can or can't work with a team, just saying that's something to watch out for.

Your best bet is to actually work a design in a similar fashion to how you would at a big job. And make every dev adhere to it. If you're lucky enough to get a team of developers whose coding styles, personalities, etc. just mesh together, then you're pretty much set. If one of you takes a more "senior developer" position where he's not only coding himself but keeping an eye on quality control as well, all the better.

[Doodman]

In rebuttal:

1) The actual netcode is encapsulated pretty well at this point. Most of zone/world did not have to change with the new netcode (except that I esstenially renamed APPLAYER to EQApplicationPacket). I basically rewrote the lower level (and tried to preserve the API structure as made sense) and the plugged it in and tweaked it.

2) I completely agree. But this is a -major- undertaking. The new eqcollector (replaced packet collector) is very modular and supports the use of plugins for processing the packets. The major issue here is the way windows uses DLL's (at least in vs.*). In unix a shared object is brought in to the memory space of the running executable and can directly call the functions in there. Not so in windows. Doing this would involve putting -everything- that might need to be called into a series of DLL's, which is of course doable.

3) I agree, in principle. But, you'll get a very small return on the time investment. Mysql is the most popular opensource database followed, probably, by PostgreSQL (which I prefer). I don't forsee anyone using Oracle or Sybase or SqlServer for a eqemu backend, it's either outright piracy or it's cost prohibitive.

4) First of all, to me this is not "network code". This is application transport level (i.e. spawn struct, player profile struct, etc). If you are talking about a preprocessed ML for generating C/C++ structs/functions, yeah that'd be great. I started on this as well, at least in theory. Having each object know how to build the packets that it's interested in. That way it's completely encapsulated and can be changed without affecting external code. Having a ML to generate those methods would be great. I have concerns about having it purely dynamic during runtime. It would be very hard to do effeciently on the server (clients have it easy). Remember, multiple the work the client has to do in these cases by 10, 20, 80, 100, however many people are on the server. More work = more lag = more instability.

5) I kind of agree with this. More threads isn't always a good thing. More to manage, more to join back in, more to coordinate. And as far a taking advantage of a SMP machine, if you have multiple people in multiple zones, you already are doing so. Adding more threads may not help, plus I'd wager a lot of things done per client can't reallt be done in parallel outside of slower database type things (like saving).

I'll add #6. There are a lot bigger issues with the eqemu code (bugs, stability issues, live compat, etc) that I feel a lot of this time and energy could be focused on.

I'm not trying to be a nay-sayer, but more of a realist. These are great ideas and things that I agree with on principle. Some of them may not be worth the effort when you look at the bang you'll get for your time/effort -buck.

[Trumpcard]

From an architectural design perspective, heres my comment..

I'd just stick with mysql.. its completely portable and lightweight, and it'll keep your support channels down. Go out to too many database systems and you'll end up with endless installation, configuration and support questions for a myrid of database systems. Just do yourself a favor and lock the database requirements.

I'd also consider trying to gear towards mysql 5 and writing as much as possible as stored procedures. The queries for the most part will be really simple, but doing them as stored procedures will make the queries alot easier to support (they'll be in a master mysql stored proc definition file versus scattered all throughout the code in inlined queries ) , and from a performance engineering perscpestive, it'll be ALOT easier to track down poorly performing database calls by using the server side performance timings and call frequencies that the database itself will generate. Needed optimizations will be alot easier to perform, as you'll have better metric data to decide what needs to be optimized.

Also, doing the calls as stored procedures, it'll make it easy for you to develop add on tools in other languages such as PHP (web interfaces/info sites) that exploit the exact same stored procedures to retrieve data without having to reinvent the data calls inside a different language. I know from experience that 20 different people developed 20 independent web tools and what not so there was a ton of code duplication in terms of the queries. Just make a set of stored procedures that handle all those requests and when new folks need new types of queries, just merge them into the SP base.

[BlueHi]

I agree 100% with Trumpcard. Although I use coldfusion instead of php I could do the exact same thing. I would love to see stored procedures. Maybe I am talking about something a little differant but here is how I would use them. I work with SQLserver and don't know to much about mysql. With my experience of working with stored procedures and using web based code like php and coldfusion is that you don't need to know how the program works you only need to know what value's the stored procedures returns which is great. This makes development a lot faster and also it can be used over and over again in other programs. Like I say I don't know much about php but in coldfusion I would simply just include a program that uses the stored procedure so there would be no duplication for that query. I am not so sure how much that would benifit the development of EQEmu but I do know that it would help those of us who link a website to there database for eqemu.