EQEmulator Forums - Lua Scripting+Content Databasing system

EQEmulator Forums (https://www.eqemulator.org/forums/index.php)

- Development::Development (https://www.eqemulator.org/forums/forumdisplay.php?f=590)

- - Lua Scripting+Content Databasing system (https://www.eqemulator.org/forums/showthread.php?t=36380)

Lua Scripting+Content Databasing system

Long story short: a few weeks ago I started making a server, but I had a problem: I didn't know any Perl, and the Perl wrapping code and some other things were incomprehensible to me. I had recently gotten a good handle on Lua, though, so I decided to embed my own Lua-based scripting system. At the same time, I couldn't get some of the SQL editors working out of the box, so I decided to just use Lua to database content (items, spells, npcs, spawn points, path grids, etc).

My own server was very custom and I didn't put any care into maintaining compatibility with standard EQEmu, but I've now gone back and started making a clean version, mostly separated by preprocessor definitions, in case anyone else might want a look at it. This post is just to gauge interest to see if I should keep working on this and make it available.

Forewarning: as I said, I don't know much of anything about your Perl system in practice, and I'm not sure whether you're still stuck with static loading for items and spells, so maybe a lot of the things I think are improvements will seem quaint. Also, this is going to be a long post full of example Lua code. Oh, and one more thing: probably none of this will be of much interest to established servers since the cost of switching would be high. But some tinkerers and start-ups might be interested -- if that's you, read on! Oh, and to make it clear, all the following stuff isn't just pie-in-the-sky theory crafting; my server runs using all these systems in place of the Perl/MySQL versions.

Standard spiel: Lua is a flexible, easy-to-learn language with good documentation. It's often used in video games and it has the benefit of being designed to embed in C, making it relatively easy to work with from inside C/C++ without the need for complicated third party wrappers. Deep in the manual it's also described (at least historically) as a "data-description language," which we're taking advantage of in using it for databasing.

As far as being a scripting language, I'm assuming it probably wouldn't be any better than your Perl system, and would probably take some time to catch up with Perl in terms of functionality (I'm basically just writing functions as I find a need for them on my own server -- I'll need to start working on documentation soon, too). The syntax is certainly different, and some of the standards will be different too (arguments to functions are explicit rather than implicit, and there's no strict distinction between clients and npcs, instead always being handled as mobs).

In any case, I intend to keep the scripting system proper and the databases under different preprocessor defs, just in case anyone comes out interested in the databasing but less interested in switching from Perl for scripts. Quick rundown of those:

EMBLUA - compile the Lua-handler class. Required to use anything below.

LUA_ITEMDB - use Lua for items.
LUA_SPELLDB - use Lua for spells.
LUA_SPAWNS - use Lua for spawn points and path grids, and effectively for NPCs (as we'll see below).
LUA_SCRIPTS - use Lua in place of Perl for script EVENT calls. The functions will still be loaded if you don't use this, in case you want to use them for LUA_SPAWNS.

To begin looking at Lua databasing, let's start with the most basic difference: rather than sitting in an SQL table, data will start off sitting in Lua scripts. This allows us to make use of the organization of the /quests/ folder; by an large, developers will not need to think about global id numbers for anything except spells and occasionally items. IDs for spawn points, path grids, and items (technically) will always start at 1 within the zone in which they are being loaded.

Also, while we lose SQL's arbitrary searching abilities, we generally won't have to go far to find what we're looking for. Spawn points for gfaydark will always be located in the "spawn_list.lua" file in the /quests/gfaydark/ folder. Items will still have global ID numbers, but they will be standardized so that we can easily work backward from their global ID to see which zone they are defined in.

SPAWNPOINTS
Now, to get into some real, practical examples, let's start with a look at spawnpoints. Spawnpoint scripts, as noted above, are called "spawn_list.lua" and located in the folder for the zone they are spawning things in. When a zone loads, it looks for this file and compiles it into Lua's memory to refer to from then on. A simple spawnpoint script with a single entry might look like this (some basic familiarity with Lua's syntax will help from here on):

Code:

--spawn list for gfaydark

spawn_list = {}



spawn_list[1] = {

        {name="a guard",race=4,gender=3,size=5,texture=3,aggro_range=25,loc={5,120,-2,220}}, 60

}

To parse it out: first we create a table called "spawn_list" (it must be called this) and then we put an entry into it. The entry is indexed by an ID number (starting at 1, as here) and consists of two values: a table full of NPC data, and a respawn time in seconds. To help visualize it:

Code:

spawn_list[id] = { 

        npc_table, respawn_time

}

When it is time to respawn, the code looks for the first value, the NPC data table. If it finds a table there, it ships it off to a spawn function, using any values it finds (for example, if name="X" in the data table, the NPC's name is set to X) and providing sensible defaults for anything that was left out.

Similarly, when the NPC associated with this spawnpoint depops, the code looks for the second value, the respawn time. If it finds an integer there, it uses that value as the respawn time in seconds.

That's all well and good, but where are all the added functionalities Spawn2 gives us? This is where Lua's flexibility comes in. Though we are free to use constants for NPC data and respawn time as we did above, we can also provide functions in their place. If the code finds a function instead of constant data, it will call that function and use its return value as the data it was looking for.

The spawn point and path grid databasing systems coexist with the scripting system, giving us access to its functions -- and meaning that we can perform arbitrary checks inside the functions that will use for our spawnpoints. As a silly example, let's make it so our NPC will only spawn if there is a client with the letter "v" in their name somewhere in the zone at the moment of spawn:

Code:

--spawn list for gfaydark

spawn_list = {}



spawn_list[1] = {

        function()

                local clients = GetClientList()

                for key,cli in pairs(clients) do

                        if string.match( string.lower( cli:GetName() ), "v") then

                                return {name="a guard",race=4,gender=3,size=5,texture=3,aggro_range=25,loc={5,120,-2,220}}

                        end

                end

        end,

        60

}

If no NPC data table is returned, the code will consider the spawnpoint to have depoped again, restarting the respawn timer.

The respawn timer can have similarly arbitrary functions, though in its case it must return a number value or the spawnpoint will be completely disabled. Here's a quick example to make our NPC respawn faster when there are more clients in the zone:

Code:

--spawn list for gfaydark

spawn_list = {}



spawn_list[1] = {

        {name="a guard",race=4,gender=3,size=5,texture=3,aggro_range=25,loc={5,120,-2,220}},

        function()

                local clients = GetClientList()

                return 60 - ((#clients < 30) and #clients or 30)

        end

}

If there are under 30 clients in the zone, the respawn timer will be 1 second quicker for each client; otherwise it will cap at 30 seconds quicker. (The (x and y or z) is equivalent to C's ternary operator, x ? y : z)

These examples are a bit silly, but I think it shouldn't be too hard to see how these two functions (well, basically just the first) effectively replace spawn groups, spawn conditions, spawn disabling, spawn limits, and so on while offering a huge range of potential uses besides -- the developer is given full control. Also note that spawn points are not explicitly associated with particular locs; when it comes down to it, a spawnpoint is really just a process -- the only time the loc matters is at the very moment the NPC is spawning. As such the loc is instead part of the NPC data, and we are free to change it, to give our spawnpoint variable locations. Or more likely, to reuse NPC data at various spawnpoints, and just change the loc each time. For example:

Code:

--spawn list for gfaydark

spawn_list = {}



spawn_types = {

        guard = {name="a guard",race=4,gender=3,size=5,texture=3,aggro_range=25},

}



spawn_list[1] = {

        function()

                local guard = spawn_types.guard

                guard.loc = {5,120,-2,220}

                return guard

        end,

        60

}

spawn_list[2] = {

        function()

                local guard = spawn_types.guard

                guard.loc = {20,130,-2,250}

                return guard

        end,

        60

}

I could go on all day about all the permutations possible here, but for one last example, let's say our spawnpoint can spawn either a named NPC or a trash NPC. We want there to be a 20% chance of spawning the named NPC on any given spawn, but we also want it to be guaranteed to spawn within 5 consecutive kills. For this we can use Lua's closure logic:

Code:

--spawn list for gfaydark

spawn_list = {}



function NamedSpawn()

        local spawn_count = 0

        return function()

                spawn_count = spawn_count + 1

                if spawn_count >= 5 or math.random(5) == 1 then

                        spawn_count = 0

                        return {name="Named NPC", ...}

                end

                return {name="a trash NPC", ...}

        end

end



named_spawnpoint = NamedSpawn()



spawn_list[1] = {named_spawnpoint, 600}

The outer function (NamedSpawn) acts like a constructor for the inner function. The local variable in the outer function (spawn_count) keeps its state between calls to the inner function, letting us conveniently keep track of data from spawn to spawn. If spawn_count hits 5, the named spawn is guaranteed. Whenever the named spawn does happen, spawn_count is reset to 0. The value of spawn_count will be lost if the zone goes down, but other than that Lua will quietly handle it on its own.

Of course, a constructor that only works for one spawnpoint is not very useful, so let's go ahead and generalize it:

Code:

--spawn list for gfaydark

spawn_list = {}



function NamedSpawn(named,trash,chance,max)

        local count = 0

        return function()

                count = count + 1

                if count >= max or math.random(100) <= chance then

                        count = 0

                        return named

                end

                return trash

        end

end



named_spawns = {}

named_spawns.example = NamedSpawn({name="Named NPC", ...}, {name="a trash NPC", ...}, 20, 5)

named_spawns.evilington = NamedSpawn({name="Professor Evilington", ...}, {name="a stupid wizard", ...}, 10, 10)



spawn_list[1] = {named_spawns.example, 600}

spawn_list[2] = {named_spawns.evilington, 1200}

Now all of the relevant information is kept in state variables. And again, NamedSpawn() acts as a constructor, so the state variables for "named_spawns.example" are kept separately from those of "named_spawns.evilington".

Even after all of this, our spawnpoints still need to be recorded so that their spawn times will persist between zone boots, so we aren't escaping SQL entirely. At the very least, however, we can replace six (?) interconnected tables (spawn2, respawn_times, spawngroups + spawnentry, spawn_conditions, spawn_condition_values, any more?) with one nice, compact table:

Code:

CREATE TABLE IF NOT EXISTS lua_spawn (

        id INT UNSIGNED NOT NULL DEFAULT 0,

        zone VARCHAR(32) NOT NULL,

        instance INT UNSIGNED NOT NULL DEFAULT 0,

        spawntime INT UNSIGNED NOT NULL DEFAULT 0,

        PRIMARY KEY (id, zone, instance)

);

Not too shabby.

It's worth making this explicit: by default, there is no NPC database. NPCTypes are constructed anew every time an NPC is spawned, whether from a spawnpoint or a mob script. In the above examples, NPC data has always been given as constants, but we are free to keep a running list in Lua's memory or to load a list from a secondary file or whatnot. But assuming you aren't doing anything fancy, the spawn_list.lua file also acts as the zone's NPC database.

One last note: in order to update or add spawn points, after putting the updated spawn_list.lua file on our server, we will need to enter the zone in question and use the #reloadspawns command in order to A) inform the zone that the spawn data in Lua's memory is out of date and B) create SQL entries for any new spawn points (this part could probably be automated, but doing it on command seemed safer/easier/more efficient).

PATH GRIDS
Anyway, enough about spawn points! Let's take a quick look at path grids. Path grids are loaded from "quests/<zone shortname>/path_list.lua". A lot of this is going to look familiar.

A pathing grid consists of two values: a table full of locs, and a function. In this case, the function is not optional. Whenever an NPC on a grid reaches one of the nodes, it calls this function and expects two results: the index of the next loc, and the time to wait before proceeding to it. A quick example of a circular path, in maximum-readability form:

Code:

--path list for gfaydark

path_list = {}



path_list[1] = {

        {

                [1] = {10,-20,-2},

                [2] = {20,-20,-2},

                [3] = {20,-30,-2},

                [4] = {10,-30,-2}

        },

        function(index,grid_id,npc)

                if index == 4 then

                        return 1, 2000

                end

                return index+1, 1000

        end

}

Fairly simple: our grid has four points. When an NPC on that grid reaches a point, it calls the function, passing the index of the loc we just reached, the id of the grid it's on, and the npc itself. If we are at the final loc in the grid (index 4), we tell it to go back to the first loc after waiting 2000 milliseconds. Otherwise, we queue up the next loc in the list and proceed to it after 1000 milliseconds. (For reference, we would put an NPC on this path by putting "path=1" in its NPC data in spawn_list.lua; it can be varied by spawnpoint just as loc is in one of the above examples.)

The benefits here should be fairly obvious. We no longer need to remember magic numbers for wander type, and we are free to customize pathing behavior without needing to go into specific NPC scripts. For example, let's say we have a 20 point circular grid, but we want points 16-20 to be off limits unless "Named NPC" is dead:

Code:

--path list for gfaydark

path_list = {}



path_list[1] = {

        {

                [1] = {10,-20,-2},

                [2] = {10,-50,-2},

                ...

                [20] = {100,200,-2}

        },

        function(index,grid_id,npc)

                if index == 15 then

                        local npcs = GetNPCList()

                        for key,NPC in pairs(npcs) do

                                if NPC:GetName() == "Named NPC" then

                                        return 1, 1000

                                end

                        end

                elseif index == 20 then

                        return 1, 1000

                end

                return index+1, 1000

        end

}

The additional arguments (grid_id and npc) can be used to make generalized functions (we don't want to have to rewrite code for the basic circular path every time, naturally) or to add NPC behaviors upon reaching a point, much like EVENT_WAYPOINT_ARRIVE, but independent of which NPC is on the grid at the time.

Note that the list of grid locs need not be constant. If we assign the table full of locs to a variable, anything that makes calls to Lua -- like spawn points, or mob scripts -- can add, remove, and/or replace locs in the grid in real time. In fact, with an appropriate algorithm we could have the grid's own function alter its nodes, for example creating a geometric pattern without ever having more than 1 node in the list at any one time, like so:

Code:

--path list for gfaydark

path_list = {}



active_geo_path = {[1] = {0,0,0,0}}



path_list[1] = {active_geo_path,

        function(index,grid_id,npc)

                local cur_loc = path_list[1][1][1] --i.e. path_list[grid][loc_table][first_loc_entry]

                local next_loc = {}



                --put code to generate next loc here based on heading etc



                path_list[grid_id][1][1] = next_loc

                return 1, 1000

        end

}

As with spawnpoints, there aren't a whole lot of limits to what you could do within the path function, but I don't want to bog this post down with overly complicated examples. In any case, once more we get to drop a couple of SQL tables (grid, grid_entry), so that's a plus.

ITEMS
I don't know if you guys are still held captive by Sharemem/static item loading, but I was on my old server, and it sucked. In this Lua system, items are instead loaded into Lua's dynamic memory. All GetItem() requests are thus simply rerouted to Lua's dynamic item list and const-casted on the way to C++. Cross-zone consistency is maintained in a fairly hacky way: when the #reloaditems command is used, the user sends a special private message to themself through the world server with an impossible channel id, an admin check, and finally the special trigger string (this means an extra check on every /tell, but most will stop at the channel id check, so it's minimal extra processing. Someone tell me if there's a better server packet to co-opt for this!). Every running zone will see this and tell their instance of the Lua interpreter to reload any loaded items. Any zones that aren't currently running will simply load the up-to-date items when they boot. This has the price that clients won't see changes to items they already have until they zone, but that's fairly minor (and could probably be fixed, if you don't mind flooding your players with item packets occasionally). Furthermore, the item database gets its own Lua interpreter, making it impossible to directly access the loaded items from within NPC scripts or what have you, for safety.

Anyway, that's the theory, now for the nitty-gritty:

Most items, like spawn points, are associated with a particular zone, and are loaded from "item_list.lua" in a zone's folder. A file with a single, simple item might look like this:

Code:

require "item"

item_zone = "gfaydark"



item_list[1] = {

        name="Ragged Fur Jerkin",hp=10,mana=10,endur=10,ac=5,slot=item.slot.chest,

        icon=662,itemtype=item.type.worn,material=1

}

require "item" just loads a module (item.lua) with predefined values, like item.slot.chest and item.type.worn. Otherwise, items data is handled a lot like NPC data. Provided values are sent off to an item-creating function at the appropriate time, and any values not provided are given sensible defaults. Some shortcomings are apparent: you'll need to use a third-party tool to find icon and color values and whatnot.

But unlike spawnpoints and path grids, there is a fair bit of work being done under the covers. Notice that we don't declare item_list (no "item_list = {}") and we have to set a variable with the shortname of the zone. item_list is already defined, with special behaviors, in master_item.lua which will be located in /quests/items/. You won't need to worry about exactly what it does, but it's good to know it's there.

Also, notice that the item ids for the zone start at 1. Under this system, items effectively have two id values: intra-zone ids, and global ids. The system gives every zone 1000 spaces for items by default (which I think is fairly generous) and the global id for an item is simply set to zoneid*1000 + intra-zone_id. This is mostly to simplify things for developers (no need to check if an item range is completely free); if you use the Lua scripting system, functions like AddItem() will take intra-zone ids by default, with an added option to use global ids, as well. And, again, we can easily find out what zone an item is from by doing a quick floor((global_id-1)/1000) = zone_id. Of course, not all items are associated with particular zones; for them, there will be overflow spaces (at least 100,000 of them) as well as spots where there are gaps in zone ids; these items will be defined in "global_items.lua" in the /quests/items/ folder by default, though they can be separated into further files and simply loaded by global_items.lua.

These restrictions might seem off-putting, but they come with another benefit. Since items are kept in relatively small files, and since the file an item comes from can easily be deduced by its global id, we can implement a dynamic loading regiment.

Say your server has 50,000 items. It seems silly to have all those items loaded in every running zone, especially if only, say, 1000 ever appear in a particular zone between boots.

By default, when a zone boots up, it will only load the items in its own item_list.lua. After that, item loading is dynamic. Whenever a GetItem() call happens, Lua will check to see if the item being asked for has already been loaded. If not, it'll check the requested id number and determine if it comes from a particular zone, or from the global item file. After that it will load the correct file into memory (as Lua tables full of item data) and check to see if the intra-zone id appears in that file's item_list. If so, it'll pluck out that one item data table and load it (as an Item_Struct pointer) and then return it to C++.

So basically, items will be loaded as needed, when someone enters a zone with foreign items, or when an item link is sent from another zone, or when someone summons something.

It may add a little loading time, but your RAM will thank you. Furthermore, this takes some of the load off of the worldwide #reloaditems command -- zones will only reload items that they have already loaded up to that point. Anything not loaded will be up-to-date when and if they are dynamically loaded, naturally.

If there is one other downside to all this, it's that a particularly bad typo in an item-defining file can potentially wipe out all the items for a zone. In case of this eventuality, Lua will create a placeholder this-is-clearly-an-error item in place of any item in can't find to load. If the item is in an equip slot, Lua will also set the placeholder item's slot to the value matching the slot the item is in, so hopefully inventory will be kept fully consistent (still need to test that bit, though).

As a last little note, since items are defined in Lua scripts, if you had some really consistent items you could easily have Lua generate their item data algorithmically, but that is a pretty tiny benefit.

SPELLS
Spells are more or less like items, although I haven't bothered to make a dynamic loading system for them yet (it would have to load on scribe/mem/click to be worth bothering).

I think I'm running out of steam here, there isn't a whole lot to say about spells specifically. They're loaded from "spell_list.lua" in the /quests/spells/ folder. You will still need to generate a spells_us.txt file (I have a script to do this) to give to players whenever you make any changes the client needs to know about, like name, icon, spell gem color, cast time, grey out/recast time, and whether the spell exists in the first place. But other than that, changes can be made dynamically using the #reloadspells function -- things like effects an effect values, reagents, can be changed without needing to get a new spellfile out right away (the reloading is globalized in the same manner as items). And even when you do put out a new spellfile, it'll be enough that the players download it, log out, and then log back in; you won't need to actually bring your server down to add new spells and whatnot.

The spell template currently looks like this:

Code:

spell_list[3] = {

        name = "",

        teleport_zone = "",

        you_cast = "",

        other_cast = "",

        cast_on_you = "",

        cast_on_other = "",

        fade_msg = "",

        range = 0,

        aeradius = 0,

        knockback = 0,

        knockup = 0,

        cast_time = 5000,

        recover_time = 1000,

        recast_time = 0,

        duration = 0,

        duration_formula = 0,

        aeduration = 0,

        mana = 0,

        effects = {

                {0,0,0,0,100},

                {0,0,0,0,100},

                {0,0,0,0,100},

                {0,0,0,0,100},

                {0,0,0,0,100},

                {0,0,0,0,100},

                {0,0,0,0,100},

                {0,0,0,0,100},

                {0,0,0,0,100},

                {0,0,0,0,100},

                {0,0,0,0,100},

                {0,0,0,0,100}

        },

        icon = 0,

        memicon = 0,

        components = {

                {-1,1,-1},

                {-1,1,-1},

                {-1,1,-1},

                {-1,1,-1}

        },

        light_type = 0,

        beneficial = 0,

        activated = 0,

        resist_type = 0,

        target_type = ST.target,

        basediff = 0,

        skill = skill.evocation,

        zone_type = -1,

        environment_type = 0,

        time_of_day = 0,

        level = 1,

        cast_anim = 44,

        target_anim = 13,

        travel_type = 0,

        affect_index = -1,

        no_sit = 0,

        newicon = 161,

        spell_anim = 0,

        uninterruptable = 0,

        resist_diff = 0,

        dot_stacking_exempt = 0,

        deletable = 0,

        recourse = 0,

        short_buff_box = -1,

        desc = 0,

        typedesc = 0,

        effectdesc = 0,

        bonus_hate = 0,

        endur_cost = 0,

        endur_timer = 0,

        endur_upkeep = 0,

        is_disc_buff = 0,

        hate_added = 0,

        num_hits = 0,

        pvp_base = 0,

        pvp_calc = 100,

        pvp_cap = 0,

        category = -99,

        can_mgb = 0,

        dispel_flag = -1,

        min_resist = 0,

        max_resist = 0,

        viral_targets = 0,

        viral_timer = 0,

        nimbus_effect = 0,

        directional_start = 0,

        directional_end = 0,

        spell_group = 0,

        restriction = 0,

        allow_rest = true,

        max_targets = 0

}

As with NPCs and items, any values not provided are given sensible defaults. You'll also want to start id values at 3 rather than 1, just because. There is a module to provide certain familiar pre-defined values, like ST.target, skill.evocation, and SE.current_hp (may change the capitalization to be more like spdat.h #defines in the public version)

Annnnd that's it, I guess. I also use Lua in place of SQL for pets, titles, and saylinks (as an EVENT and a standardized function rather than a database file). I don't use factions on my server, but a system similar to items and spells could surely be made for them, and I'll probably work on that when the time comes. I also just use EVENT_SPAWN for loot drops to get away from loot table SQL, but that is surely nothing new.

Anyway, any interest/questions/pointing-and-laughing at how backwards my system is, is welcome!

i knew there was more to what you were doing than had been done elsewhere, i just didn't know what. i'd have rather you chose python, but i've been looking for a reason to become familiar with lua. i'm going to have to read this more instead of skimming over it when i wake up and can actually focus. :)

Lua support is something I have been very much interested in. But, it isn't plausible to convert PEQ's quest scripts and DB tables over at the moment. If it could be made an optional supplementary component that would be ideal for now, so conversion could be done over time. Meaning, Perl and the DB are checked first, and then a Lua script is used if found. Conflicts would cause Lua to not load.

As you mentioned there isn't a whole lot Lua can do in the quest department that Perl can't. But, I certainly would love the option. You're right on the money when you say that Lua integrates better with C++ than Perl does (Let's face it, our Perl wrapper while it works well enough is ugly as sin.) So over time, I could very much see Lua depreciating Perl much like Perl depreciated our old .qst system simply due to the fact that new features have to be added to Perl, with Lua up to speed with the current EQEmu code they would pretty much just work. Though, in those days the quest scripts were far less complex so conversion was much, much easier. That's my biggest concern. Conversion would be a major undertaking and while many minor scripts could be automated, most of the raid and other major events would have to be done by hand. I've already looked into this in the past ;)

I'm a DB guy, so I don't have too many problems with our current DB system. But I tend to like flatfiles as well, provided a suitable editor is created. I also like the idea of having spawns and especially grids script-able. Yes, Perl can be used but it isn't as elegant as Lua. It's clunky and slow. I think Lua is the key to recreating the truly dynamic world that exists on Sony's servers and that I have been chasing after forever. I think we've done a decent job over at PEQ and many of the zones do appear very dynamic to many folks. But truth be told in those same zones that I have practically memorized all the spawnpoints and the NPCs that could potentiality pop from them. In my ideal situation, I shouldn't be able to do that. (I'm talking about yard type zones, not dungeons which are static by nature.) I think I should be able to say well, there is a chance these guys could spawn in this general area, but I am not exactly sure where. But again, conversion is the big stepping stone here.

I think the long and short of it is I'd love Lua support, but it would need to be implemented in a way that conversion can be done over time. I doubt anybody in EQEmu has the time or man power to do a quick overnight conversion for their servers.

i would devote countless hours to conversion efforts in order for this to see the light of day. :)

Yes, but not everybody has the ability to convert their server over. A good number of servers are completely custom or at the least altered versions of PEQ. A dual system would be needed to accommodate everybody and not force anybody to keep up or get out. We wouldn't want to make Lua an on or off type situation either, because then we'd have Lua folks on one fence, and Perl/DB folks on the other and we'd essentially be forking the project... Something we do not want to do.

i only meant i'd be more than happy to help with conversions for those who are interested if it would help drive zaela (and/or others who are capable) to continue development efforts, not that everyone should be forced to choose one over the other and that i would do everything in my power to make it so.

the possibilities that would open as hinted at by the examples of using anonymous functions instead of fixed data had my head spinning for a few moments and i got a little excited is all.

additionally, my old ass xp laptop would very much appreciate any memory savings i can come up with. :)

Quote:

Originally Posted by cavedude (Post 217362)

I hadn't even really considered the possibility of not-fully-custom servers being interested, just because it would be so much work converting everything over, and also considering the dip in productivity that comes with having to learn a new system as well, though that's tiny in comparison.

I'll definitely think about having the Lua systems coexisting-with-but-secondary-to the Perl/SQL equivalents, but I think I'll take me a few days just to process the mental gymnastics it'll take to work out how to manage that (at least for items, spawns and paths probably wouldn't be too bad). I still need to get the clean version up to par with my server and make sure all the bits and pieces work properly before making any major changes into how it all operates.

Adding a lua parser was actually what I wanted to truly do when I added the ability to even have extra parsers besides Perl but have been doing other things.

And Concrete I do believe if you want a python parser you will need to fight off Cavedude trying to murder you first.

i use ESF Database Convert - Professional Edition here is the website http://www.easyfrom.net/ following database formats: Oracle, MySQL, SQL Server, PostgreSQL, IBM DB2, IBM Informix, InterSystems Caché, Teradata, Visual Foxpro, SQLite, FireBird, InterBase, Microsoft Access, Microsoft Excel, Paradox, Lotus, dBase, CSV/Text and transfer any ODBC DSN data source to them.

Quote:

Originally Posted by KLS (Post 217369)

And Concrete I do believe if you want a python parser you will need to fight off Cavedude trying to murder you first.

I'll hold him, and Cavedude can stab him, repeatedly. :)

Code:

Nothing

  wrong

    with

      python

Oh wait.

i'm getting the idea that nobody likes python but me around here.

nevertheless, my posts were more about wanting to see this lua thing work out than having someone embed python. from what i understand, lua will work just as well as perl, with a much smaller footprint than python, and easier integration than either of them.

A++ would use anyday.

Seriously; this is the kind of scripting system I would like to see used in lieu of all of our database calls. While I am not saying replace the database for normal EQEmulator servers, I would love to see if we could incorporate this in some way shape or form, even if it's just chunks or bits of it..

You have put a great deal of effort into coding this and I am glad you did.

PS: Mitch Lawrence.

Quote:

Originally Posted by cavedude (Post 217364)

Yes, but not everybody has the ability to convert their server over. A good number of servers are completely custom or at the least altered versions of PEQ. A dual system would be needed to accommodate everybody and not force anybody to keep up or get out. We wouldn't want to make Lua an on or off type situation either, because then we'd have Lua folks on one fence, and Perl/DB folks on the other and we'd essentially be forking the project... Something we do not want to do.

While I see no problem with adding LUA as a scripting language option, as far as integration with C++ goes I've been told that the integration is beautiful in what you can do, it is not something this far into this project to fully replace the Perl system. That is said with complete fairness in the purpose of a scripting language meant for driving events flexibly within the world by not needing to export stuff constantly in the source is a great benefit.

Though if you go beyond the flexibility of a scripting language and integrating it within the source code and start talking into more practicality and use, Perl has thousands of modules and other abilities that make it also have incredible potential that is not nearly seen here on the forums. Perl also has the capability of doing shared memory functions within one of its modules and has thousands of other modules made to bring functionality to a higher level beyond basic script parsing.

If the quest scripting option is the only thing that is wanted/justifiable, I'm fine just working on that and keeping the rest to myself. It would be a lot less work, and I don't really have anything to gain in working on it anyway, just figured I might as well offer. Integrating it into the Parser/QuestManager handler classes shouldn't be too hard, and I think that'll get it just where it's wanted.

Lua does seem to have a very minimalist, "if you want something done, write the module/library yourself" mindset behind it, which is fine for me but I can see where that would be problematic for a works-out-of-the-box project like EQEmu.

Quote:

Originally Posted by c0ncrete (Post 217375)

I use about 30 tools written in python to create and manage the content on my server. <3 4 python.

Quote:

Originally Posted by Zaela_S (Post 217391)

I'm actually more interested in the spawning stuff, to be honest. The quest scripting is more a it'd be nice to have it, but ultimately there is nothing wrong with Perl for that. But, don't kill yourself. If something can't be done easily, forget about it.

Quote:

Originally Posted by cavedude (Post 217401)

Spawns would probably be the single easiest thing to implement without disturbing any of the existing systems -- just need to add a lua_spawn_id variable to NPCs, and then do all the LuaSpawn checks/processes after the Spawn2 versions. Wouldn't be any problem to have Lua-spawned NPCs use Perl quest scripts, of course.

Could probably have just the spawns and maybe path grids done in an evening or two. Then I'd just need to figure out where and how to commit stuff ;p

Quote:

Originally Posted by Zaela_S (Post 217410)

Could probably have just the spawns and maybe path grids done in an evening or two. Then I'd just need to figure out where and how to commit stuff ;p

Just post a diff here for the server code changes

It'll be more than a diff. Depending on the lua interpreter used it'll also be a static library + headers, or a bunch of c/cpp files and headers.

If it is correctly optional a diff against the trunk should be viable.

Just my 2cents,

converting the perl eqemu quest to lua isnt that big of a deal - just remember - perl only can parse perl =)

Example from convert for my own engine:

http://pastebin.com/fBQnYXv3

(havent implemented all methods so some are commented)

Writing the initial converter might have taken about 10h.

Quote:

Originally Posted by cavedude (Post 217362)

As you mentioned there isn't a whole lot Lua can do in the quest department that Perl can't.

Au contraire, it is a million times better than perl. Why would basically every game in the last 15+ years use lua as scripting backend?

Its small, no depencies, can be embedded easily, is 10times faster than perl.

Oh and the biggest aspects - coroutines!

npc:Say("Something...")
Wait(1second)
npc:Say("Something more..")
SpawnSomething()
Wait(10seconds)

you are basically writing a movie scripting without going forth and back with throwing timed events and if(timer==1) etc...

This is also not a real thread, just everything on the stack. No multithreading issues or objects going out of scope.

The most important thing, if you really want to redo the scripting engine - going back to the drawing board before coding anything really - this is the hardest part - designing a good API itself that is extend-able, takes more time then actual coding..

Good luck!

Quote:

Originally Posted by Harakiri23 (Post 217458)

The most important thing, if you really want to redo the scripting engine - going back to the drawing board before coding anything really - this is the hardest part - designing a good API itself that is extend-able, takes more time then actual coding..

That's the key right there, most of the problems we do have with Perl are mainly due to the way it was implemented into EQEmu, and not inherent to the language itself.

Quote:

Au contraire, it is a million times better than perl. Why would basically every game in the last 15+ years use lua as scripting backend?

As far as games are concerned I fully agree with you. Lua is light years ahead of Perl. However, to be fair I think Perl has the edge otherwise. I use it to do so much of my bidding over at PEQ, at work, and I even have cron running Perl scripts on my personal boxes that do various things for me. Perl is clunky and slow no doubt, but it's robust and external libraries make it even more so.

What I meant is that there is no immediate need for Lua in regards to scripting, Perl handles our needs just fine. We haven't hit a wall yet forcing us to say well, Perl has to go. Though, if Lua is added to our code base, I suspect that will quickly change.

Quote:

Its small, no depencies, can be embedded easily, is 10times faster than perl.

Oh and the biggest aspects - coroutines!

Preaching to the choir there, sir. Perl's implementing coroutines in version 6, but that's too little too late I think. Still, it'll come in handy for my utility usage of the language.

Quote:

Originally Posted by cavedude (Post 217485)

However, to be fair I think Perl has the edge otherwise. I use it to do so much of my bidding over at PEQ, at work, and I even have cron running Perl scripts on my personal boxes that do various things for me. Perl is clunky and slow no doubt, but it's robust and external libraries make it even more so.

sorry, i was referring to lua as more powerful as in "for scripting engines in games" due small footprint, easy to read and coroutines, also a selection of binding wrappers

obviously perl takes the cake - it can do anything - but its a heavy weight

i considered python and lua actually, while i would have preferred python - its also bloated already

I'm not very familiar with coroutines, would probably be up to someone else to figure out a good way to implement them into the event-based system if it gets that far ;p

Quote:

Originally Posted by cavedude (Post 217485)

That's the key right there, most of the problems we do have with Perl are mainly due to the way it was implemented into EQEmu, and not inherent to the language itself.

Again, I'm not very familiar with the Perl system, but I do see some bits and pieces of the Perl code that seem eerily similar to qst (which I'm more familiar with), so I think I might understand what you mean, at least to some extent.

I'm probably not the best person to try to write a fully fleshed-out quest parser; I don't have that much experience with Lua and mostly just make things up as they occur to me. Just for the sake of it though, a random grab bag of features in my Lua quest parser that seem significant from my point of view:

*Mobs, items, spells and timers are all treated as objects with their own methods and can be stored in variables etc. Adding new methods in either C++ or Lua is fairly straightforward.

*Two scripts are run as a zone is booting, before NPCs spawn: /quests/global/zone_init.lua and /quests/<zone shortname>/zone_init.lua; these give the opportunity to initialize variables or, more likely, to set user defined functions and/or hook extra code onto "core" functions from C++, either globally or for a particular zone (the zone-specific one runs second in case you want to redefine something from the global one). Can also use the opportunity to rename some functions if you don't like the default names. Some standard functions are defined in the global zone_init.lua by default, e.g. mob:say() is just a call to mob:text() with some pre-defined parameters, and GetDist(), IsBehind(), and InCone() are all mostly just math that Lua can handle itself, only making calls to C++ to find coords/heading.

*EVENT_SIGNAL's trigger does not go through C++ at all, letting us pass any kind of data in the signal, including tables, references to mobs/etc, and functions.

*Every EVENT call attempts to trigger two scripted functions: EVENT_<WHATEVER> and GLOBAL_<WHATEVER>. For clients the use is clear enough: GLOBAL versions of events should be defined in playerglobal.lua, while zone-specific EVENTs should be defined in a zone's player.lua. For NPCs, it's up to the user if/how to use the GLOBAL events -- they don't need to be truly global, though they easily can be. They could be used within a zone as a sort of inherited/shared behavior without clashing with normal npc-specific EVENTs, or similar. Lua is fast and can check whether an EVENT/GLOBAL exists without doing much processing, so the double check is not noticable, even if it's rarely used.

*General Lua features: every Mob is given its own environment to store its variables and functions, but these environments are ultimately just tables sitting in specific variables within the global table; we can easily peek into MobA's environment from within the environment/script of MobB. This can be questionable and messy, but there are some good uses, like defining the EVENTs for an add's "script" from within the script of the mob that spawns it (also making it easy to vary behaviors from add to add without needing to juggle multiple script files -- at the extreme end, you could redirect the adds' environments to tables within spawner's environment, completely encapsulating them). The global table can also be accessed easily, making it a convenient place to put data where multiple mobs/scripts can refer to it (and without being tied to the lifespan of any one mob). Mob environments also automatically inherit from the global table, offering another way to define default EVENTs (for example, if you wanted most of the NPCs in a city to share the same EVENT_SAY, you could just plop it straight into the global table; any NPCs given an EVENT_SAY in their own script will "overwrite" the one inherited from the global table) or take it a bit further and have some EVENT inheritance hierarchies (though that can get messy quickly).

...Some of that might not fit too well into the existing framework, though. Not sure how important keeping things familiar and hopefully easily to convert over from Perl is compared to letting it go its own way without having to bend to some possibly-questionable decisions made however many years ago.

Quote:

Originally Posted by Harakiri23 (Post 217486)

also a selection of binding wrappers

This thread has kind of died down and maybe there wasn't that much interest in the first place, but: I just re-read this and want to take a moment to make a case against using any auto-generated bindings/wrappers if a Lua quest scripting system does become a thing.

I realize that manually writing functions to expose C++ will take a lot of time and that in the process some bugs will inevitably creep in, requiring even more time to notice and fix them. But I want to argue that going that route will ultimately leave you with a more sensible, user-friendly system oriented towards its actual purpose -- i.e., writing quest and combat scripts for mobs -- rather than the as-close-to-the-C++-as-possible result a wrapper will likely give you (as far as I know, anyway). Main points:

Simplification

There are some things the script writer should just never have to worry about. CastToNPC/Client is probably the most obvious example. Handing a script function a Mob reference should always be enough -- any needed checks for NPC or Client should happen on the C++ side and split the function's behavior as necessary. (Maybe the Perl system has already eliminated the need for these, but I see them used in a few old scripts up on svn so I dunno.)

For another example, consider Mob::CastSpell(). Specifically, the fact that it takes an entity id for the target rather than a Mob reference. We know that it always expects an entity id. But we also know that, in our script system, whenever we want to cast a spell on something we will always have a Mob reference to the target before we have the target's entity id. Therefore, any function exposing CastSpell() to our script system should get the entity id implicitly. There's no reason every script-writer should need to write this

Code:

self:castspell(437,target:GetID())

every time when they could just write this

Code:

self:castspell(437,target)

instead.

Furthermore, CastSpell() does not have any default targetting. It seems reasonable to expect that if a script-writer writes this

Code:

self:castspell(437)

they should expect that the spell will self-target the given caster (or even better, target the caster's target if they have one and the spell is detrimental, and target the caster otherwise). We're given a fair amount of leeway to write pseudo-overloaded functions like this, so it seem sensible to use this to provide defaults that the C++ function in question does not.

Limiting Redundancy

Going back to automating NPC/Client casts and checks, consider the case of instantaneous intra-zone movement. When we want to teleport an NPC, we use GMMove(). When we want to teleport a client, we generally use MovePC while specifying the zone they're already in, since the movement is cleaner (unless we want to put them in mid-air, anyway).

But again, the script-writer should not need to know about this distinction. From the point of view of the user, all we care about is that the function instantly moves the given Mob. As such the script system should provide a single function that does this for both NPCs and Clients, with the GMMove/MovePC distinction being handled automatically on the C++ side. For Clients, clean movement is the most sensible default, but we can easily provide the mid-air variant with a single extra boolean parameter, like so:

Code:

target:move(x,y,z,heading) --clean

target:move(x,y,z,heading,true) --mid-air

As a simpler example, consider these two Client functions: IsInAGuild() and IsInGuild(id). It's a little thing, but we can save a function here too. If we see a call like this

Code:

if client:InGuild() then

...

with no argument, it's reasonable to assume that we're checking if the client is in any guild at all. On the other hand, if we see something like this

Code:

local guild_id = other:GetGuildID()

if client:InGuild(guild_id) then

...

it's reasonable to assume that we're checking if the client is in the guild of the given id. The less the user has to remember little distinctions between functions, and the less functions there are in total, the less they're going to be tripped up. There are lots of little examples like that -- anywhere the script system can combine similar functions without making things more obscure, it should.

Opportunistic Improvements

The other two are kind of about this as well: when you take the time to look at functions one by one and think about how they're going to be used, it gives you a good chance to customize how the function is handled, to make it more sensible without sacrificing any functionality. At the same time, it also gives the chance for you to say "you know, this function should really do x, in case anyone ever wants to do y."

As an example, on my server I made the Damage functions ultimately return the final damage value caused to the target, so that this value could be returned to Lua when used in scripts. This allows for easy scripted lifetaps at any percentage of the dealt damage, or simple tracking of dealt damage for other purposes. Or, if you feel like disabling the standard AI melee and making an NPC's melee damage cycle purely script-based, you can detect when the NPC's attacks are dodged, parried, riposted, etc. Just a general functionality that should really be available to anyone who writes custom raid-level encounters.

Maybe we could go back and do most these things after auto-generating functions with a wrapper, but I think much of the impetus would be lost if the functions are already there and, in the worst case, by the time someone gets to working on it they'll run into that dreaded situation where they have to choose between making an improvement or maintaining compatibility with the scripts that have already been made. Maybe that could be avoided by just using modules to provide simplified function calls on the Lua side of things, but modules would be optional and, in the end, we'd still have a needlessly bloated, needlessly unfriendly base system. No good!

Maybe this is all dumb and I just don't know enough about how wrappers work. In any case I think it's probably clear that opening up content development is kind of my main area of interest ;p

At least my opinion on this as someone who does a lot of heavy duty event scripting with PERL for my servers is that I just can't contemplate the need to change the scripting language when I can literally do anything my heart desires with PERL and easily at that. It does take some time to learn the functions related to eqemu but honestly it is so powerful once you have it down. Is every function super optimal probably not, but most of the examples your citing, at least to me just elicit a visceral reaction of 'who cares that isn't a big deal' at least not enough to warrant changing languages. Please don't take that the wrong way, I am not trying to be negative towards your work I respect what your doing..

I think it would ultimately negatively impact the project to fork quest scripting languages. Just because something can be done doesn't necessarily mean it should.

Anyways just my opinion.

Kayen
GM Storm Haven

there are at least a few of us that are VERY interested. some of us just spend loads of time reading and poking around with things while we're learning about them instead of commenting or asking questions. :)

as far as how the perl interfaces to c++ are implemented, i couldn't agree more. there are a number of functions that could be consolidated, expanded upon, moved, or just corrected.

for example... the functionality of Mob::EntityVariableExists, Mob::GetEntityVariable, and Mob::SetEntityVariable could be combined into Mob::EntityVariable(evName, [evValue]), and it would be assumed that we wanted to return the value (evValue) of the variable name (evName) if no value was passed. there are tons of other things like that and and the ones you covered that i've been puttering around with to see what i can come up with on my personal server.

i also realize that once things are implemented and used a bit, it's difficult to want to change them for everyone because of migration issues. do you leave the old code in there for a while until people get a chance to move it, or you just yank it out unceremoniously?

Quote:

Originally Posted by c0ncrete (Post 217804)

Quoted for truth.

Do you have this lua variant on github? I would love to pull this from you.

there's another post containing diffs (including one for cmake) to build what has been released to the public thus far.

http://www.eqemulator.org/forums/showthread.php?t=36401

I'm in the process of re-writing everything to be cleaner, better organized, more compatible (the itemdb in particular I need to redo so that it'll be easier to convert directly to and from SQL) and have better error checking. I still need to learn the basics of Git too. But I do hope to get a test branch up somewhere while I work on it, hopefully not too long from now.

Lua vs Perl / MySql

Pros of Lua vs Perl:

Smaller footprint, memory and cpu wise.
Can be distributed with the code base.
Removes an outside dependency.
Popular among game enthusiasts.
If it's used to replace Perl & MySQL then we also have the option of creating 'eq server in a box' linux distribution with a great deal less headache since it's only the server to deal with and not the server and sql issues.
Item look up side can become far more dynamic for the custom users. Easy set bonuses, for example.
Lua to c interface is far simpler and cleaner. Means that whole interfacing point could become much cleaner for scripter's. Examples where shown in the thread. Could lead to a standard library for EQ Lua scripts since it's far simpler to reuse.
Some tools might be easier to write since we can reuse much of the lua script code directly into the tools themselves. This is difficult to do with Perl as it stands.

Cons of Lua vs Perl:

Switching systems means we have people who need to learn Lua who all ready know Perl.
Perl has a huge number of packages all ready made (may not be relevant considering our current uses).
need to rewrite many of the current quests to use Lua.
Some will not switch since they have a large number of custom scripts.
Under some conditions the Lua GC can become finicky. (rare but can happen, usually just a set gc level change at the global scope)

Pro's of Lua vs MySQL:

Imperative as well as Declarative for those who want more dynamic content behavior or are used to imperative but not declarative systems.
A chance to clean up the item table and other components since it's getting a little....crazy? yeah. crazy. :grin:
Lua embedding is just as supported as SQL in almost all cases.
Easy starting out for modders / content creators. All that is required is a text editor.

Con's of Lua vs MySQL:

SQL is a very common language. Many developers know it. Lua? not as much.
Thousands of SQL tools out there.
very very very easy to do global modifications in SQL, may not (but doesn't have to be) easy to do this in Lua, depends on implementation.

What else am I missing? I'll edit as I people add.

Quote:

I'll edit as I people add.

Welcome to the forums, let us know how that works out for you.

I know nothing at all about Lua but it sounds pretty cool.

Lua and MySQL aren't even in the same problem domain.

Quote:

Lua and MySQL aren't even in the same problem domain.

I understand that, but it can be used in the same way, if we want. I'm just saying it's an option, not that we *should* :)

I think it's somewhat impractical is all.

Also a few of the developers do have plans to somewhere down the road change up the quest system and the database system but we're not close enough to be ready to talk about it yet.

I like Lua for content databasing for a few reasons:

1) Content data is essentially static -- it only changes when new stuff is put in there, and doesn't really change at runtime. The only thing you really get out of having it be SQL is searchability and the ability to invalidate a whole table in a single brainfart. (Oh, and making sweeping changes, which I guess is nice for this project; I hope to make at least items and spells convertible back and forth from SQL, which would still allow that.)

2) For some things (like the spawnpoint and path grid stuff I put up) open-ended scriptability is a lot simpler than having to add new fields (sometimes new tables) and new code for every little new abstract functionality or feature you want to add.

3) Having everything organized into the folder of the zone it belongs to is nice. Not that e.g. the NPC database could not be changed to include zone and reduce the need for global NPCID numbers, but that's probably one of those "too late to consider changing" things.

Definitely would not want to force anyone to switch. But there are reasons both for and against.

Quote:

Originally Posted by Zaela_S (Post 217799)

You want to simplify things but still dont want to use a wrapper ? I skimmed over your code - and the problem is - the same as with the perl system - only 1% will understand it and maintaining and extending will be done by copy&paste of existing code without really understanding it (e.g. people currently just paste code in the embperl classes for new functions - but thats not how its done, there should be a clean header file to perl exported functions and then the perlxs/convert is called which generates all the func bodies).

Whats worse - lua is stack based - thats a whole new level of understanding for most people. For perl, you were unable to expose complex objects (func args) to the quest system - now with direct lua calls you have the same issue - every complex object has to be manually coded - where in bindings its done in one line.

Look how WoW emus, EQ2, Ryzoom and other implemented their event system - and compare all the bad things and improve upon this - (BTW - they all use bindings). A quick example how a binding could look like for getters or setters

Code:

          .def("GetName", &NPC::GetName)          

          .def("GetCleanName", &NPC::GetCleanName)          

          .def("GetID", &NPC::GetID)          

          .def("GetRace", &NPC::GetRace)          

          .def("GetClass", &NPC::GetClass)          

          .def("Say", &NPC::Say)

Thats everything you need - number of args, arg types, etc all automatically handled - everybody understands this. No need for

Code:

static int Lua_GetName(lua_State* L) {

        int num_args = lua_gettop(L);

        if (!num_args || !lua_isuserdata(L,1))

                return 0;



        Mob* check = *(Mob**)lua_touserdata(L,1);

        if (check) {

                if (check->IsNPC()) {

                        if (num_args > 1 && lua_isboolean(L,2) && lua_toboolean(L,2)) {

                                lua_pushstring(L,check->GetName());

                        }

                        else {

                                lua_pushstring(L,check->GetCleanName());

                        }

                }

                else {

                        lua_pushstring(L,check->GetName());

                }

                return 1;

        }

        return 0;

}

You still need at least one or two who understand lower level lua api - e.g. ever thought about garbage collection ? If you dont have the right approach here your embedded lua will be very slow and the memory will grow very quickly.

More things you need to take care of : script scoping - want to make sure that nobody can overwrite a var of another npc ? or do you want that ?

How can entities interact with each other directly?