Anyone interested in client development?

[PixelEngineer]

Before you all go running let me say a few things. This client, or at least as far as I will take it, will likely never have full network support for interaction with the EQEmu code. I am doing this to simply learn about coding, file formats, OpenGL, rendering, and my passion, engine development.

I am a few days into the project and things are looking good. The main development will focus on classic zones and work its way forward to support newer zones. If you are interested in helping and know C++, even if you are a beginner and want to help, please PM me.

The eventual goal, as I am very interested in engine development, is when I have EQ zones perfectly rendered, to add new stuff EQ never supported. Per pixel lighting, shadows, dynamic weather, realistic water. And let me state right here that no zone files are modified in any way and will never be. This is purely an educational and learning project and I figured a few of you might want to help.

The project will be open source as soon as I have the code base in a relatively stable manner. Also, if you are interested in helping you must obtain your own zonefiles as I will never distribute any.

Also, on a related note, is Windcatcher around at all? Have a few questions for him about the stubborn WLD format.

Thanks for your time!

Preview:

http://i.imgur.com/EmaS3.png
http://i.imgur.com/zaTek.png
http://i.imgur.com/0xHC8.png

Note: Texture masking/fog and light sources are not implemented yet. I have only been developing for about a week in my spare time. Also, the frame rate is a representation of the rendering of every polygon in the zone without culling.

[KLS]

I actually had a similar thing that worked for eqg v2 and v3 zones but I stopped working on it, I probably had about a 3rd of the zones rendering flawlessly before I got bored with it.

WC isn't really around much that I know of and actually EQ does support some of those things now but meh (not very well imo). One of the best resources for WLD is azone if you haven't already looked at it.

[PixelEngineer]

I have taken a peek at an old version of azone. From what I see, it is badly if at all commented. Might be helpful for some of the confusing fragments though. I will keep this thread updated with progress. In the process of loading these polygons into a BSP tree for super fast rendering.

Edit: Does anyone have a link to the last released version of the azone source? Thanks!

[PixelEngineer]

Found it! Thank you. This is fantastic.

[Grebbin]

As long as you are developing clients, develop one that works on a droid x! I think I'd cry from happiness. I've seen people can stream the video from their computer over wifi, but I'd love a true client running on my droid. :)

[PixelEngineer]

A phone based client is entirely possible although I am only able to develop for phones I have access too. My first order of business after I get the rendering engine polished up is porting to Mac and Linux. I would definitely consider phones as a viable platform after that.

[Damilis]

What techs are you using? Build systems? Sounds neat, but I'd like to hear a bit more of the specifics =D

[PixelEngineer]

Writing this in Visual Studio in C++ but using as few platform dependent things as possible. This thread wasn't meant to grab attention for the project release, just to see if anyone was interested in coding with me. But I am sure I will port the client to other systems. It is just a matter of free time and drive although I have plenty of the second.

[Damilis]

Well, I am trying to see how you are approaching the project. It sounds interesting to me, provided I know a bit more about it :)

I understand you're using MSVC++, so if you're planning cross platform, I suggest you switch to standard C/C++ asap :) (I do cross platform development for a living, and MAN is MSVC++ a PITA)

Are you rolling your own GFX engine or using a pre-existing one?

[PixelEngineer]

Well, I am trying to see how you are approaching the project. It sounds interesting to me, provided I know a bit more about it :)

I understand you're using MSVC++, so if you're planning cross platform, I suggest you switch to standard C/C++ asap :) (I do cross platform development for a living, and MAN is MSVC++ a PITA)

Are you rolling your own GFX engine or using a pre-existing one?

Making my own engine currently. That's the whole reason I am doing this. To learn and to have fun at the same time. Going to do a little bit of everything. A graphics engine, sound engine and some networking too. Trying to be well rounded before I start applying for internships/jobs.

[Fridgecritter]

I would love to see a Mac version. I actually played on EQmac server for a while, but it just wasn't the same. It would be nice to have a Mac client for eqemu for sure.

[PixelEngineer]

Just a quick update. I am continuing to work on this and iron out issues. If there is anyone who knows where Windcatcher is, please let me know or get in touch with me. I am trying to finalize everything on the rendering front and it has proved to be quite the challenge. Obtaining the (RLE encoded) PVS has proved extremely difficult so I have to go about doing this another way.

Here is a screenshot from today: http://i.imgur.com/JfLU6.png

Again if you would like to help PM me. And if you are hesitant, wait a few weeks. Development will get much more fun when it comes time to add features rather than fight with the cryptic WLD format.

Edit: Forgot to mention what was different. From the screenshot, it might not look like it but I fixed a number of texture bugs (that were offscreen in the first few screenshots), completely redid the way I render zone geometry and documented and did huge cleanups on nearly every WLD fragment. As soon as I get my line of sight issues fixed, I am taking on lighting (vertex and per-pixel) and am going to add things like shadows, something that was never in the original EverQuest.

[pfyon]

I'd love to get into client development as i think a customizable gui client would be invaluable to the project, but I'm afraid I don't have any experience with gui stuff.

[PixelEngineer]

I'd love to get into client development as i think a customizable gui client would be invaluable to the project, but I'm afraid I don't have any experience with gui stuff.

That's the point ;) . Learn as I go along. I mean, animation seems to me like it will be the most difficult but I'll figure it out when I get there.

[PixelEngineer]

A quick update: I have been able to extract the PVS and have implemented frustum culling for the zones. It's been difficult but I have learned a ton and it was a major hurdle trying to get all of the standard knowhow of graphics engines to work with the EQ format.

EQEmulator user Leere has been beyond helpful with providing help and other sources for the PVS and I thank him greatly.

Next up is polishing off the render side of the graphics engine with animated and masked textures. After that is lighting and some other crazy stuff I have in the works which will add a crazy new element to old zones.

I have gotten quite a few PM's from people which is why I continue to update this thread. Let me know if you're interested in the project or want to help develop. Also, as soon as the codebase is stable, it will be open source so anyone who wants to make a client won't have to do as much guesswork as I have had to do.

Cheers!

[steve]

Thanks for keeping us updated! It is interesting to read about progress being made on this project. Though I am not able to help myself, I look forward to whatever is accomplished by your efforts!

[erde]

Hi,

maybe this helps ;)

wlddoc.pdf (http://ignum.dl.sourceforge.net/project/eqemu/EQEMu%20Admin/WLD%20Reference%201.1/wlddoc.pdf)

greetings Stefan

[PixelEngineer]

Hi,

maybe this helps ;)

wlddoc.pdf (http://ignum.dl.sourceforge.net/project/eqemu/EQEMu%20Admin/WLD%20Reference%201.1/wlddoc.pdf)

greetings Stefan

It definitely has. So has azone and any old source I could find. My three main goals for this project:

1. Learn about engine development.
2. Add to the knowledge base of the EQEmulator (and yes, there are areas where Windcatcher wasn't always clear)
3. Supply a cross platform client which people can use/modify/learn from.

Cheers!

[Packet]

I have worked with WindCatcher's zone builder tool which gave me a lot of options. Unfortunately s3d's format lacking so many things, it would be very nice to work with eqg considering it supports shading, raytraced materials, bump mapping, etc. Since you're digging so far, it probably wouldn't be so difficult to implement added support to other formats. It would be a lot of fun to re-create classic zones the way they should look rather than the bastard child that sony created. Ever see the most recent freeport? It's not original in the slightest.

None the less, I imported a few of WoW objects into a couple of zones for fun a long time ago... couldn't go too far with s3d format considering the severe limitations.

http://xonos.net/Haz/castlesiege2.png

http://xonos.net/Haz/hazg1.png

http://xonos.net/Haz/hazg2.png

http://xonos.net/Haz/hazg4.png

[PixelEngineer]

Wow, that's some awesome stuff there! I'd love to hear more about that stuff. Windcatcher did some truly amazing stuff and pushed the boundaries of what you can do with the EQ engine. My only sadness is that he coded in languages I have never used like Pascal. But again, his contributions have been invaluable.

I've seen that the whole development of client side stuff has slumped a bit here so I am hoping to pick it up again so that everyone can contribute.

Send me a PM. i'd like to hear more.

[Packet]

My availability varies so it's best to email me.. I pm'd you my info. :)

[PixelEngineer]

Got a question for the EQEmu community if anyone knows. In the 0x30 reference, there is bit that is set if the texture is masked. The only thing missing is information is about what color the texture is masked with. I know for items it's pink and for trees in Greater Faydark, it's green but there doesn't seem to be a field in the file.

Thanks!

[Leere]

I think I've always just used the alpha channel of the texture file itself, which seemed to work reasonably well.

[PixelEngineer]

Thank you again Leere.

[Mitchell]

Would be awesome to see a native Linux client. Keep us updated.

[Tyen05]

The work you guys are doing is awesome, sent all of you PMs so I can locate animations, lol~

Trying to get this browser based in Unity Engine.

[PixelEngineer]

The nature of the EverQuest format makes it a bad candidate for most of the engines out there, sadly. That's not to say it cannot be done but the way the zones are formatted is not quite ideal for the way Unity and Unreal handle level data. Now I may piss some people off but to be honest, the EverQuest zone format is a hack job of the Quake 3 file format and they are very similar.

I will have this engine's main rendering stuff squared away in a couple of days and you all may have a look and tinker around as you wish. The main hold ups in development have been the fact that I have learned as I went which is obviously much slower than simply writing code. If I could start over with what I know, it would take me a day or two to do most of the rendering stuff. The other hold up is that while Windcatcher and the azone developers have done a tremendous job of documenting, there are still major gaps in what we know and I intend to flush them out.

Cheers,
Pixel

Edit: Noticed your screenshots. That's was about as far as I got in Unreal. I must ask, how many frames per second are you getting?

[PixelEngineer]

Not really an update, just a question for anyone who might have some insight into the EQ file formats.

http://www.eqemulator.org/forums/showthread.php?p=205162#post205162

Post in that thread if you can lend any knowledge to the matter.

Cheers!

[PixelEngineer]

Here goes nothing. Here is the link to the code for my project I titled Lantern. Lantern is an open source cross platform (not yet) EverQuest client.

This is not a release, it's just a link to the project source. If you are looking to explore zones and such, I would recommend holding off. There are a ton of bugs I am fixing and a few zones that simply won't yet load. In addition, there are a number of problems with my code but it's a start. I am happy I was able to get this far and I will continue and the codebase will continue to get more solid and fine tuned. If you are interested in developing, I have included everything you need.

The license is very straightforward:

Let's keep this simple. You are allowed to use, copy, modify, merge, publish, and distribute this source and software. You are however NOT allowed to create closed source variants under any circumstances. Knowledge should always be free and open source.

If you are interested in developing, the client is written in C++ on Visual Studio 2010 although I am always open to adding support for more IDE's. I write code on this client every single day and look forward to the possibility of more people getting involved. Again, this client is going to be a huge learning experience for me and whoever would like to partake.

Also, you must own your own copy of the zone files. I am not going to distribute anything of the sort. Furthermore, if you see an error in my code or a way I should have done something differently, please correct me. I am not an arrogant developer. I am here to learn and will help anyone who wants to learn as well. Please not also that there are areas that I just have not fixed up yet.

If you have any questions, please let me know and I look forward to making some serious progress in the coming months. I will continue to add more documentation in the next few days to make it easier for people who want to help.

Cheers!

http://www.assembla.com/code/lanterneq/

[PixelEngineer]

My apologies. Here is the link: http://www.assembla.com/code/lanterneq/subversion/nodes

[Tabasco]

My first step will probably be to set up a cmake build option. You chose a solid set of cross-platform libraries and it looks good so far.

[PixelEngineer]

Thank you. I have reduced most of the Windows specific functionality and will eliminate the rest when I get around to it. A bit of code is taken from various other projects and the way they handled fragments was a bit all over the place. Consider my work on textures in 0x03.cpp, 0x04.cpp, 0x05.cpp, 0x30.cpp and 0x31.cpp for a sample of what most of fragment handling will look like as I rewrite.

If you see any glaring errors, submit some changes and I will of course get around to reviewing them. Also, if I understand correctly you're looking to make it Linux compatible? Let me know how I go about doing this/helping to do this.

Cheers!

[Secrets]

Thank you. I have reduced most of the Windows specific functionality and will eliminate the rest when I get around to it. A bit of code is taken from various other projects and the way they handled fragments was a bit all over the place. Consider my work on textures in 0x03.cpp, 0x04.cpp, 0x05.cpp, 0x30.cpp and 0x31.cpp for a sample of what most of fragment handling will look like as I rewrite.

If you see any glaring errors, submit some changes and I will of course get around to reviewing them. Also, if I understand correctly you're looking to make it Linux compatible? Let me know how I go about doing this/helping to do this.

Cheers!

This is very good! I've got a great idea for creating new content: Why not a 3DS Max exporter plugin? You could have it build WLD files from scenes, and I believe 3DS Max already supports animations. Not sure how it handles BMP files but it should be trivial.

SOE already uses 3ds max for their animation files afaik.

[PixelEngineer]

As of now, the next time I have time (Friday), I will finish my rewrite of the PFS (S3D) extractor. It works fine for the most part but there are some weird issues and I think it might be related to this. Then, I will begin work on models and light and get them all working properly.

Those are just my 2 main objectives at this point but I don't have time for the next two days due to finals.

Cheers.

[Vampire1212]

I'd love to help out with this client. I'd personally like to do my own revamp of the classic zones by simply change textures, lighting not the graphics themselves.

[PixelEngineer]

I'd love to help out with this client. I'd personally like to do my own revamp of the classic zones by simply change textures, lighting not the graphics themselves.

Explain further. I am going to be implementing per pixel lighting which utilizes normal maps. Although I can generate them automatically via software, the higher resolution the texture, the better. If you are good at designing high resolution textures, give it a try. This was actually one of my eventual ideas.

Search for textures in the .s3d archive but if high resolution textures exist in an external directory, use them instead.

I am interested to know exactly what you mean.

[Vampire1212]

Explain further. I am going to be implementing per pixel lighting which utilizes normal maps. Although I can generate them automatically via software, the higher resolution the texture, the better. If you are good at designing high resolution textures, give it a try. This was actually one of my eventual ideas.

Search for textures in the .s3d archive but if high resolution textures exist in an external directory, use them instead.

I am interested to know exactly what you mean.

Hey PixelEngineer, what I mean is exactly what you said, all high-res textures that make it seem more graphical when in reality we aren't changing the graphics themselves.

[PixelEngineer]

Hey PixelEngineer, what I mean is exactly what you said, all high-res textures that make it seem more graphical when in reality we aren't changing the graphics themselves.

Go ahead and PM me. I have a few ideas you might be able to help with.

[PixelEngineer]

Texture animation has been implemented:

http://www.youtube.com/watch?v=4XSFx2Td4y8&context=C3912520ADOEgsToPDskKvMSv9_VvDZfhrSHuAh39k

More progress to follow...

Cheers!

[Akkadius]

Texture animation has been implemented:

http://www.youtube.com/watch?v=4XSFx2Td4y8&context=C3912520ADOEgsToPDskKvMSv9_VvDZfhrSHuAh39k

More progress to follow...

Cheers!

That's pretty badass man. I give you props for all of that hard work.

[PixelEngineer]

That's pretty badass man. I give you props for all of that hard work.

Thank you! The real hardcore development is going to start as soon as I get everything stable. The original EverQuest developers made some very odd design choices and I have struggled to both understand and implement some of the more challenging ones. Every day I get closer to a much more solid code base and then the real fun development can begin.


Cheers!

[Speedz]

This rocks, I have wanted this type of thing for years. But I never had the code know how to even grasp a start. Glad to see this taking off and having community backing.

[PixelEngineer]

Don't sell yourself short. I see you have written some programs in C#. Syntactically, it is not that different from C++. When I started writing the client, I couldn't write it either but i slowly learned and have learned an enormous amount. The link to the source is on the last page so take a look if you're interested.

Cheers!

[Speedz]

Thanks for the vote of confidence :-) I grabbed the source and now to get the time to look at it. My knowledge is pretty basic but getting better. Not too bad for no schooling or work environment in coding.

[KLS]

Do you guys mind if I move this to Dev and perhaps sticky it? (I think it's cool and has actually shown some promise)

[Speedz]

I just joined in on this thread but I like that idea.

[PixelEngineer]

Do you guys mind if I move this to Dev and perhaps sticky it? (I think it's cool and has actually shown some promise)

Don't mind at all, as long as you let me pick your brain here and there. :)

Also, Speedz, pick up the commit I push tomorrow. It solved 98% of zone loading crashes. Might be a better starting point.

[Speedz]

K will do, thanks

[PixelEngineer]

Revision 27

Changelog:
- Fixed a name length bug in the 0x03 fragment.
- Texture animation is now supported - still have to work on global timing
- Fixed up the 0x04 fragment issues. Figured out an unknown value in that fragment as well. Turns out it is animation delay, the number of milliseconds before the animated textures switches to the next texture.
- Finally fixed a weird texturing issue involving a NULL texture. For things like invisible wall, the polygons are still in the BSP tree (for collision). They do still have a texture reference but it points to a null value. This one was a pain to debug.
- Fixed a number of specific zone crash issues - some still remain, i.e. 'qeynos'

Known issues:
- Crash on texture load with some zones - have to look into this after the weekend.
- Animated textures do not always update at the same time.
- Some animated textures (waterfalls) flow the wrong way
- Oasis fails to load - stops on loading the file "canwall1.bmp" as it does not exist in the archive. canwall1a.bmp does however. I assume this is a fault on the
developer's end.
- Code base is still a huge mess. Bear with me while I slowly eliminate old functions that are not needed and rewrite certain fragment functions.
- Rendering seems to "freeze" when you go out of a leaf (travel back into a leaf and the frame will refresh). It hasn't actually froze but I disabled the whole zone rendering function until I finish the new rendering.

Goals for next commit:
- Texture crash fix
- Global texture animation timing
- Transparency and semi transparency with rendering (have to rewrite rendering function...again)
- Loading zone variables (sky/fog color/clip plane)
- Ambient light

[PixelEngineer]

One more thing before I disappear for the weekend, here is a concept UI I made. It needs to be something that's usable on different screen sizes. Again, just an idea but I am going to try and model the UI after this when the time comes.

http://i.imgur.com/c780P.png

Suggestions? Criticisms?

[Speedz]

Grabbed it but I I'm not sure whats the deal with my setup.

'Lantern.exe': Loaded 'D:\EQEmu\Lantern\Lantern\Debug\Lantern.exe', Symbols loaded.
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\ntdll.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\kernel32.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\KernelBase.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'D:\EQEmu\Lantern\Lantern\SDL.dll', Binary was not built with debug information.
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\advapi32.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\msvcrt.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\sechost.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\rpcrt4.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\sspicli.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\cryptbase.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\gdi32.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\user32.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\lpk.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\usp10.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\winmm.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\msvcp100d.dll', Symbols loaded.
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\msvcr100d.dll', Symbols loaded.
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\opengl32.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\glu32.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\ddraw.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\dciman32.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\setupapi.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\cfgmgr32.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\oleaut32.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\ole32.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\devobj.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\dwmapi.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'D:\EQEmu\Lantern\Lantern\glew32.dll', Binary was not built with debug information.
'Lantern.exe': Loaded 'D:\EQEmu\Lantern\Lantern\zlib1.dll', Binary was not built with debug information.
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\imm32.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\msctf.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\KBDUS.DLL', Cannot find or open the PDB file
'Lantern.exe': Unloaded 'C:\Windows\SysWOW64\KBDUS.DLL'
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\KBDUS.DLL', Cannot find or open the PDB file
'Lantern.exe': Unloaded 'C:\Windows\SysWOW64\KBDUS.DLL'
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\dsound.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\powrprof.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\dinput.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\hid.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\wintrust.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\crypt32.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\msasn1.dll', Cannot find or open the PDB file
'Lantern.exe': Loaded 'C:\Windows\SysWOW64\nvoglv32.dll', Cannot find or open the PDB file
The thread 'Win32 Thread' (0x1880) has exited with code 0 (0x0).
The thread 'Win32 Thread' (0x1854) has exited with code 0 (0x0).
The thread 'Win32 Thread' (0x1850) has exited with code 0 (0x0).
The thread 'Win32 Thread' (0x1b8c) has exited with code 1 (0x1).
The thread 'Win32 Thread' (0x14d8) has exited with code 1 (0x1).
The thread 'Win32 Thread' (0x280) has exited with code 1 (0x1).
The thread 'Win32 Thread' (0x1a14) has exited with code 1 (0x1).
The program '[8108] Lantern.exe: Native' has exited with code 1 (0x1).

[PixelEngineer]

log.txt contains all of the debug and error logging information you need.

However, I assume you don't have the .ini set up correctly. Open it. The top line is the zone shortname. The last one I used was qrg, Surefall Glade. Change it to arena or gfaydark and then drop the zone's .s3d in the same folder as the .exe

Then run as you normally would through Visual Studio.

When it doubt, check the error log. It's far from complete but definitely helps me to debug.

Edit: Also, you'll see a lot of texture related crap in the debug. I did a lot of that to solve most of the texture errors. Still have one more error so I am keeping them enabled.

[Speedz]

Thanks for the quick reply and tips. Will try again in a few as I am reinstalling VS due to other issues not related to this build.

Didn't know or look deep enough to know about the zone and s3d files.
Will get this going and dive in soon.

[PixelEngineer]

No worries. Pace yourself and don't get burnt out. It took me months to get things working and I took many breaks so just take your time and have fun. Modify variables, mess around and see what makes things work. I am going to move a lot of the OpenGL stuff into a separate init.cpp so the main.cpp can just be client setup related.

Let me know if you ever have any questions or problems here, in a PM or at red.pixel.engineer@gmail.com

Cheers mate!

[Akkadius]

One more thing before I disappear for the weekend, here is a concept UI I made. It needs to be something that's usable on different screen sizes. Again, just an idea but I am going to try and model the UI after this when the time comes.

http://i.imgur.com/c780P.png

Suggestions? Criticisms?

You are crazy talking to yourself. No really, I'm interested to see where you get with this

[Speedz]

Got the client to fire up. For all interested peoples make sure you don't use s3d files from the SoF or SoD client :-)

[PixelEngineer]

Got the client to fire up. For all interested peoples make sure you don't use s3d files from the SoF or SoD client :-)

This is something I will fix. The issue here isn't lack of support for the new WLD, it is lack of support for the texture formats (.dds, 24 bit bitmaps)

Told myself I wouldn't but I am going to try and fix that last pesky texture bug tonight.

[Speedz]

Sent you a few emails with some minor things I played with and a launcher tool that I made.

Btw, the launcher tool will delete .s3d files in the root of where it is then load the s3d file you select from the list from a Maps folder. Made this to keep things simple for me, hope it is handy for ya.

[Tabasco]

I downloaded this right after you published svn and then I got hit hard at work. Sorry to just disappear.

I fired this back up today and got it building under debian testing 64-bit.
The most notable issues were case-sensitivity in filenames and includes, '\' instead of '/' in include paths, and a handful of windows API functions that I moved to SQL or just cobbled together windows.h equivalents.

The one remaining issue appears to be related to 64 bit builds. I looked back at one of my old projects that borrowd PFS code from azone and it looks like we did things similarly. After quite a bit of debugging I started testing my old code again (as it was working in linux32) and it failed as well.

I won't be able to validate this in a 32-bit environment until later, but here's what I have so far.

http://midian.homelinux.net/lanterndiff.txt

[Tabasco]

uint32 was typed to unsigned long and long types are used for crc's etc, which explains why there's not 64-bit love.
Now I'm having a problem with bitmaps, but I suspect it's related.

It seems like the solution here would be to use SDL_Image since it sounds like better general image support is needed anyway.

[Vexyl]

I've compiled it on Fedora 13, 32-bit using Tabasco's patch for Linux. I didn't want to mess with code::blocks settings, so I just did it on command-line with g++.

I can't seem to get it to work properly though, it opens and crashes trying to load gfaydark. I have gfaydark.s3d in the same folder as the Lantern binary file.

lantern.ini:


gfaydark
800
600


Partial log.ini:


..........

<-!-> Error: Could not link mesh: 0
<-!-> Error: Could not link mesh: 1
<-!-> Error: Could not link mesh: 2
<-!-> Error: Could not link mesh: 3
<-!-> Error: Could not link mesh: 4
<-!-> Error: Could not link mesh: 5
<-!-> Error: Could not link mesh: 6
<---> Total polygons: 74981
<---> Total render vertices: 97028
<---> Total indices: 0
<---> Filename here: SGRASS.BMP
<-!-> SGRASS.BMP has an unknown bpp
<-!-> Unable to load texture SGRASS.BMP!
<-!-> Unable to load zone textures!
<-!-> Fatal Error: Unable to load zone data!


Have I forgotten something?

Edit: This is using the .s3d from a fresh Titanium installation. I've also tried qrg and unrest, they both give similar errors.

[Tabasco]

This is what I was getting in the 64-bit build after changing all the types, but I thought I just hadn't found everything yet.
My 32-bit build does the same thing so it's more likely that I messed something up when trying simulate the windows.h types for bitmaps.

I just disabled texturing and enabled GL lighting and shading. It's not very pretty, but you can at least wander around the zone and it demonstrates that everything generally works.
It should be simpler in general to use SDL_Image for textures and I can probably get that in there at some point if pixelengineer is agreeable to that. Also after looking at how the scene is managed, I wonder if it would be wise to first make a tool for processing maps into a different format.

[Speedz]

What release of gfaydark.s3d did you use?
I got exactly that till I used the titanium s3d files.

[Speedz]

I have thrown together a couple of simple tools that I was going to just use for myself. But after getting into them I figured more people might find use for them. Heck maybe even improve my amateur code.

I am posting this on this thread as one of the tools directly relates to this project. Its called Zone-Selector.

The other is a offline spell search tool. It is far from complete, just hardly started even.

https://www.assembla.com/code/emustuff/subversion/nodes

If you do mess with it and improve it, please send me patch files so I can update the svn.

[PixelEngineer]

Hey guys, sorry I have been absent for a while.

Linux guys, great work and I am very happy to have some people on different operating systems.

The texture loader is an offshoot of a texture loader from another engine and it looks like it's giving problems so we can definitely kick it and use SDL_image.

Currently, this only has been tested on classic zones. Therefore, the only image type that is currently supported is an 8 bit bitmap. The reason there is a bit more code there in there is because I had to add an instance where the 8 bit bitmap would compute an alpha layer for masked textures.

What kind of format does the new client use? I see that the log said bitmap with an unrecognized BPP which make me assume 24 for which support can easily be added. In addition to bitmap, do the newer zones use .dds or any other format?

Since my last commit, I have done a ton of cleanup to the WLD class, changed a lot of the uchar* to char* to avoid mixed character type warning, and fixed a few other things but have unfortunately been at the whim of a six week intensive calculus class. After an exam on Monday, I will be able to push that latest version and have a couple of hours each night to code.

I'll add support for the texture types and try to get the linux diffs sorted. I see that as the new number one priority besides stability; getting the client working on linux as well. We can slowly eliminate Windows specifics and have an SVN that is cross platform friendly.

Thank you again for showing interest and I will definitely put some time in next week to get this stable on linux and adding you guys to the SVN if you so desire.

Cheers!

[Speedz]

Life happens :-)

Good to see such interest in this. Add me if you like but with my limited experience in svn commits and coding anything past GUI utilities won't lend myself to heavy traffic at first. But gotta learn and get experience somewhere right?

[Taurinus2]

Hey PixelEngineer,

Have you checked out SOIL?

http://www.lonesock.net/soil.html

[PixelEngineer]

Hey PixelEngineer,

Have you checked out SOIL?

http://www.lonesock.net/soil.html

I have not. Have you used this? Looks like a pretty full featured and lightweight library. Definitely will consider using it.

Thanks for the suggestion. I will have time today and tonight to work on the client again.

[Taurinus2]

Yes, I've used it in the past. Small, efficient, one less dependency (can integrate directly into your source tree).

Depending on your decision to limit the amount of C you have in your project or not, you can easily slap a facade on it and be good to go.

[PixelEngineer]

Okay. I got SOIL implemented. Two potential problems. We are reading from memory in the loading of textures. I have the file interface as a workaround I would like to get rid of. I might have to add the functionality to SOIL to load from memory and not a file. Second issue is that when the textures do load, they look a bit weird. May just be an error in my code but I am getting the feeling the library might not support 8 bit bitmaps.

Do you have any idea if it does?

PixelEngineer

Edit: Found the option to load from a buffer. Wish there was a bit more documentation for this library but I really do like what I see.

Edit2: Went through the flags and found SOIL_FLAG_TEXTURE_REPEATS. This ended up denoting the textures as tiles. Not sure why this isn't enabled by default. All working. This will be part of the next commit. Thanks again Taurinus for the suggestion.

[PixelEngineer]

Commit 28

- Added SOIL library for image loading. Thank you to Taurinus2 for the suggestion. New zones now load. (textures are flipped)
- Integrated Tobasco's diff for Linux. Thank you Tabasco - huge help.
- Deprecated the use of the file interface library and the image loading library.
- Rewrote some of the texture documentation, debug output and 0x30 fragment handling.
- Fixed the texture crash error that prevented some zones from loading.
- Moved non Lantern related initialization into init.cpp and init.h.
- Added support for ambient light (0x2A), light source reference (0x1B) and light source (0x1C) fragments. Still not yet activated.
- Added specific categories of debug output for fragments (textures, lights, geometry...). Fragment errors will still output regardless.
- Huge cleanup to the WLD class
- Reimplemented the rending of the whole zone.
- Changelog.txt is now in the project directory.
- Added the SDL_ttf library. It is not used yet but I will commit my text on screen work in the next commit.

Known issues:
- Support for masked textures has been removed due to the new image loading library. Will have to re-add.
- A weird issue with mip masking makes far away textures have a tint to them. Still trying to figure this one out. Mip mapped images are disabled for now.
- Zone crash related to texture UV coordinates in some zones.
- Textures in Titanium zones are flipped

Priorities:
- Support for bitmap font rasterization
- Global texture animation timing
- Transparency and semi transparency with rendering (have to rewrite rendering function...again)
- Loading zone variables (sky/fog color/clip plane)
- Ambient and zone lighting
- Linux makefile
- Deprecate the current math library and introduce a new one.

[Speedz]

Great work so far, I noticed something odd tho. Zones are flipped left <> right.

I looked at unrest, Crushbone, and cshome. all were flipped/reversed. Not just textures, but the entire zones.

[PixelEngineer]

Great work so far, I noticed something odd tho. Zones are flipped left <> right.

I looked at unrest, Crushbone, and cshome. all were flipped/reversed. Not just textures, but the entire zones.

Thanks Speedz. Can you be more specific? Are we talking just textures or something like Mario Kart mirror mode? New zone files or old zone files?

Cheers.

[Vexyl]

Great work so far, I noticed something odd tho. Zones are flipped left <> right.

I looked at unrest, Crushbone, and cshome. all were flipped/reversed. Not just textures, but the entire zones.

I noticed that as well in the last build. If you want an example, just load gfaydark and go to the crushbone entrance, it's completely reversed.

[Speedz]

Sorry for a lack of description. Yes mirrored. Also it does this on all versions I have. Titanium to SoD s3d files.

[Speedz]

I'm also going to update my zone loader tool to read a user selected Game directory for the s3d files and copy from there as needed per launching. Instead of the original way I did it where you had to copy all s3d files to a zone directory.

Would make for faster maybe more handy zone comparison/work.

*
You select the EQ dir and it will load a list of the s3d files. You then select the s3d you wish to load from the list it generates and it will delete all s3d files in the lantern dir and copy the one you select. Basically its an as use/need basis so you don't have to either copy all s3d files to the lantern dir or constantly copy paste and change ini file every time.

If this makes any sense at all I will be surprised lol. I will try better next time.

[PixelEngineer]

Wow. I cannot believe I had not noticed that. I wonder if this is new in one of the latest versions. Thanks for mentioning that. I will fix it.

[pfyon]

I've never actually written a makefile or anything beyond compiling a single cpp file with g++, anyone have a working one for linux I could use for this?

[Tabasco]

This is based off of the latest svn. The archive includes the complete source tree, a diff, and the makefile. Also 32 bit binaries.

http://midian.homelinux.net/lanterneq_revised28.tar.gz

I renamed the FPS Counter directory with an _ instead of a space rather than trying to escape it in the makefile. There are also some header changes that probably aren't necessary since I'm not building in codeblocks anymore.

[Tabasco]

Updated download link:
http://216.49.224.132/eq/lanterneq_revised28.tar.gz

[pfyon]

Thanks, Tabasco. I was running into difficulties getting it to compile until I read the error messages and noticed it was complaining about x64 (this is a 32 bit machine). Just an FYI to anyone else who downloads the tarball, make clean first.

[PixelEngineer]

Did you get it up and running Pyfon? Got a screenshot?

[pfyon]

I haven't been able to get it working, probably due to the same issue that Tabasco et al ran into. I tried an s3d file from titanium, sof, and sod without luck.


pfyon@family-kubuntu:~/lanterneq/Lantern$ ./Lantern
24

pfyon@family-kubuntu:~/lanterneq/Lantern$



pfyon@family-kubuntu:~/lanterneq/Lantern$ cat log.txt
<---> Logfile Initialized on 2-5-2012 at 19:56:27
<---> Preferences from lantern.ini have been read.
<---> Loading zone: crushbone
<---> Screen width: 800 - Screen height: 600
<-!-> This is not a valid PFS archive
<-!-> Could not initialize the S3D file!
<-!-> Fatal Error: Unable to load zone data!
<---> Fatal Error: Unable to initialize Lantern!



pfyon@family-kubuntu:~/lanterneq/Lantern$ cat lantern.ini
crushbone
800
600

[Tabasco]

I had to do a lot of fixes for 64 bit that I have yet to publish. I'll get those committed tomorrow along with some of the header fixes so we can hopefully have a single source tree that builds without any modification for platform or architecture.

You might be able to quickly fix your copy by looking at the global datatypes include.

typedef unsigned int uint;

typedef unsigned char uchar;
typedef int8_t int8;
typedef uint8_t uint8;
typedef int16_t int16;
typedef uint16_t uint16;
typedef int32_t int32, dword;
typedef uint32_t uint32;


http://midian.homelinux.net/ltscreen1.png

[KLS]

The mirrored zones sounds very similar to an issue I ran into with my .eqg renderer because sony inverts x and y axis.

[pfyon]

Thanks again tabasco, that did fix that issue. I also had to change line 96 of S3D.cpp from


memcpy(&file_count, meta_block, sizeof(long));


to


memcpy(&file_count, meta_block, sizeof(int32));


Now I'm sitting at a black window from Lantern without any useful output.


pfyon@family-kubuntu:~/lanterneq/Lantern$ tail log.txt
<---> Logfile Initialized on 2-5-2012 at 21:16:32
<---> Preferences from lantern.ini have been read.
<---> Loading zone: gfaydark
<---> Screen width: 800 - Screen height: 600

[Tabasco]

With SOIL I haven't had any texture issues, but I haven't tested in 64-bit yet.
What do you get if you try to move around (sometimes 0,0,0 isn't ideal) or what do you get when you run something from mesa-utils like glxgears, or glxinfo, or try to load a different s3d?

KLS, I noticed some strange stuff when translating 2d map files to locs inside a web-based editor. OpenGL tends to use a Y-Up model, but every engine wants to apply its own translation anyway. In addition to that, they use one system for coords and another entirely for labels. I can only imagine what other idiosyncrasies we'll find inside their archives.

[lerxst2112]

It may not help that by default Direct3D is left-handed and OpenGL is right-handed with regards to coordinate systems. Not hard to account for if you know the format the data is loaded in and the destination system you want it to fit, but certainly a common pitfall.

[Taurinus2]

Any thoughts toward a sound lib? FMOD? irrKlang?

What threading model do you plan to adhere to?

Also do you have a design doc laying around?

I applaud your drive and progress on this project.

[PixelEngineer]

Any thoughts toward a sound lib? FMOD? irrKlang?

What threading model do you plan to adhere to?

Also do you have a design doc laying around?

I applaud your drive and progress on this project.

I think when the time comes, I will try FMOD. Looks like what a lot of people use today in the industry.

As for threading, I honestly have no idea yet. There is a design doc (readme.txt) on the SVN and it just lays out some goals I want to accomplish.

I probably won't be committing anything for a week or so while I add support for shaders and lighting. More than anything I am just reading about the OpenGL pipeline and trying to understand everything before I move on. After all, this is a project for learning.

Great to see it running on Linux and go ahead and commit that whenever you get the chance.

Cheers!

[Tabasco]

Revision 29 should build and function on linux 32 and 64 bit.

[PixelEngineer]

Revision 29 should build and function on linux 32 and 64 bit.

Fantastic. Thank you Tabasco. Also, for those who don't know, Tabasco is now a contributor to the project.

[pfyon]

Excellent, works out of the box. Performance shot up once I installed the nvidia proprietary driver too ;).

[Vexyl]

Revision 29 compiles/runs with no errors on Fedora 13.


[user@host ~]$ uname -a
Linux host 2.6.34.9-69.fc13.i686.PAE #1 SMP Tue May 3 09:13:56 UTC 2011 i686 i686 i386 GNU/Linux


`g++ --version`

g++ (GCC) 4.4.5 20101112 (Red Hat 4.4.5-2)
...


`make --version`

GNU Make 3.81
...



I did take a few screenshots while I was testing it, but didn't want to put all 5 in one post so here are the links:

Unrest: http://i.imgur.com/6wgCs.jpg

Mistmoore 1 (Overview): http://i.imgur.com/EH4QD.jpg

Mistmoore 2: http://i.imgur.com/eUgH5.jpg

Highkeep 1 (Front entrance): http://i.imgur.com/UGJ7o.jpg

Highkeep 2: http://i.imgur.com/V6XJB.jpg

That last one is interesting because the establishment plaque is upside down, and the left and right textures adjacent to the plaque are blurry.

Edit: Going to put up some neriak screenshots just for fun.

[pfyon]

I also noticed in gfay that all the trees' leaves were upside down.

[Tabasco]

This is indeed the case, and as you said, pretty easy to fix.

It may not help that by default Direct3D is left-handed and OpenGL is right-handed with regards to coordinate systems. Not hard to account for if you know the format the data is loaded in and the destination system you want it to fit, but certainly a common pitfall.

The resolution should now load from the ini file and you can change camera speed with +/- now as well.

[PixelEngineer]

As for the flipped textures, there is a flag in the WLD header that denotes if the zone is old or new. I assumed that would allow us to flip or not flip the texture across the y axis. Unfortunately, the classic and the titanium Greater Faydark both say they are old zones so it won't be that simple. My next idea is just to check the bit depth on the textures when they load. If the depth is 8 bit, we know it's a classic zone and 24 bits, a newer version zone. Again as I mentioned, I will be learning shaders for the next week or so and then I am going to tackle both per vertex and per pixel lighting in GLSL.

Awesome work Tabasco.

Cheers!

[PixelEngineer]

As I continue to progress through my study of shaders, I had a fun idea last night. For me, seeing Lantern grow and accumulate a number of DLL's in the directory has been a little depressing. I downloaded a Game Boy Advance emulator yesterday and it had only 1 file; the executable.

I am going to try as I progress to actively reduce the amounts of external dependencies and DLL's so that quite possibly Lantern can simply be an executable. It could as a first run, create the .ini.

Thoughts?

[Taurinus2]

My thoughts:

In order to achieve this on Windows, you would need:

1) Static linkage
1a) Source for every lib you use to achieve this.

Sure, lib developers/vendors make pre-built libs available, but depending on them means that the show is over once development ends (or you move on to another environment that they do not support) and no source is supplied. You also have to keep crt linkage in mind (cannot mix them).

Even being feasible it is simply not worth the effort IMO.

Secondly, why?

Your data dependencies number in the thousands already so it would seem pointless to me.

[PixelEngineer]

Just an idea I was kicking around in my head. Obviously it would not be possible with more advanced libraries like FMOD but in terms of smaller libraries where I only use 1-2 functions from them, it might be useful.

Again, not something I am set in doing, just an idea. :D

And thank you for the great reply!

Cheers!

[Speedz]

Made a small mod for a live client like feel to movement.

camera.cpp line 53

if(ks[SDLK_UP] || ks[SDLK_w] || SDL_GetMouseState(NULL, NULL)&SDL_BUTTON(1))

[Speedz]

I hope you guys don't mind my type of submissions. I too am just learning. At this point I can do simple adds, and whatnot. But to do some grand feature addon is a bit beyond me. I like to make simple apps or make aesthetic type changes or addons. I have a local working copy with a bunch of changes that haven't made it to the svn. But thats ok if they are not needed.

As I do this I will get a grasp of C+. But my main experience is C# and I have just a basic grasp of that with the help of google and microsoft help links.

[PixelEngineer]

I hope you guys don't mind my type of submissions. I too am just learning. At this point I can do simple adds, and whatnot. But to do some grand feature addon is a bit beyond me. I like to make simple apps or make aesthetic type changes or addons. I have a local working copy with a bunch of changes that haven't made it to the svn. But thats ok if they are not needed.

As I do this I will get a grasp of C+. But my main experience is C# and I have just a basic grasp of that with the help of google and microsoft help links.

Don't be discouraged at all. What I would recommend is looking at a simple fragment. How about 0x21, the BSP tree. Look in the source for my handling of that fragment and compare it to the WLD documentation: ftp://ftp.mirrorservice.org/pub/sourceforge/e/project/eq/eqemu/EQEMu%20Admin/WLD%20Reference%201.1/wlddoc.pdf

Then, pick a fragment that is not implemented, any one. Try to create a struct out of it. Try to implement a loading function mimicking the style that have already been created. And yes, I appreciate your contributions. Don't get down on yourself.

Cheers!

[Taurinus2]

PixelEngineer,

I know you aren't there yet but I would like to point you toward boost::asio for your networking. Boost as a whole is excellent and highly recommended.

[pfyon]

I was thinking about getting into the network side of things (as I have little interest in GUIs, but lots of interest in network code) and was thinking about using boost for cross-platform threading.

[Taurinus2]

Go for it.

When it came time to learn the asio library I decided to do a fun project to bang out some code with. I wrote a CLI client/server remote tool for my private server that features a handful of opcodes and a single port being utilized on either end. A modest project, but one that was finished in one sitting with all planned features.

Also, PixelEngineer, now that I am thinking of it, you will need a tighter timing mechanism because you are going to need to track your frame deltas. I say this because I foresee you needing to preempt certain operations if you do not have enough time to complete for that frame. Let's say for instance you arrive at some processing logic and you see that your delta is 15ms, you know that you must defer and return immediately (60hz = 16.6ms/frame). This assumes single-threaded execution with regard to that particular logic and your rendering but I believe that the principle will still be applicable in any case.

[PixelEngineer]

Great suggestions guys. You are welcome to start on the server/network component. :D

Also, a small update:

- Added support for fragment 29, which deals with regions with water, lava, PVP, etc.
- Added rendering support (temp) for these different regions.
- Plus/Minus keys now do not effect look sensitivity.
- F1 now takes screenshots
- Split, and moved the various text files (CHANGELOG, BUGS, LICENSE, README)

The underwater effect needs some tweaking and the specifics might be found in some of the unknowns in this fragment. I mainly added the fog effect so I could test the accuracy of the implementation.

http://i.imgur.com/CMFH7.png

It also implemented the PVP zones. As someone who has never been in Qeynos, can someone verify that the red X's were not areas that you would be in the PVP area? Only the green highlighted area is denoted in the zone files as the PVP area.

http://i.imgur.com/mOrOk.jpg

Thanks!

[Derision]

If you go in game, to the locations you have marked with an X, and then move back toward the centre of the arena, the client automatically generates the mesage 'You have entered an Arena (PvP) area', so I would say you have it right.

[PixelEngineer]

If you go in game, to the locations you have marked with an X, and then move back toward the centre of the arena, the client automatically generates the mesage 'You have entered an Arena (PvP) area', so I would say you have it right.

Fantastic, thank you.

[ableard]

So I have picked his up out of ccuriosity. I have VC 2010, typical versions running of eqemu, no opengl.
We I run in VS the window pops a nd closes. I get no signficant error or messages. Do I need to obtain any opengl distributables?
I have added .s3d file into debug folder, changed ini.
thanks

[Taurinus2]

OpenGL is a standard lib but you do need drivers that provide opengl acceleration.

With VS, you will find that the working directory for targets launched from within VS is the project root and not the target directories.

Putting the .s3d and configuration file into the debug/release folder should work if you invoke lantern from shell or console.

[PixelEngineer]

Can you copy/paste your log.txt file here or if it's huge, PM me with it. I will post the fix here.

[PixelEngineer]

Is anyone here experienced in reverse engineering files? I have been going through and documenting every type of file the client uses. There is one file however, which seems to stump me. The file is denoted by:

<shortname>_sounds.eff

This file is a counterpart to the other file <shortname>_sndbnk.eff which lists files that are either emitted or looped. This file is in plain text.

I am not that experienced in reverse engineering and wanted to know what you guys recommend. I assume the file contains things like x, y, z positions, sound file name, volume, radius and that sort of thing.

It's possibly ZLIB compressed or just a bizarre file format. It has no discernable header so I doubt it's a specific file format. Searches for .eff bring up EverQuest effect format but there are no specifics.

Any help/advice would be greatly appreciate. (hint hint Windcatcher)

:D

Cheers!

[Taurinus2]

Installing an api hook against the client and logging seek/read against files of that extension would be a start. That's half the picture, anyway. The other half is a memory scanner that locates what was just read in the client's memory. You can then trace execution from that point.

[ableard]

thanks for the reply.

Taurinus2 the .s3d is indeed in /debug

Log file

<---> Logfile Initialized on 3-5-2012 at 10:57:24
<---> Preferences from lantern.ini have been read.
<---> Loading zone: gfaydark
<---> Screen width: 640 - Screen height: 480
<-!-> Could not load the PFS archive!
<-!-> Could not initialize the S3D file!
<-!-> Fatal Error: Unable to load zone data!
<---> Fatal Error: Unable to initialize Lantern!

[PixelEngineer]

thanks for the reply.

Taurinus2 the .s3d is indeed in /debug

Log file

<---> Logfile Initialized on 3-5-2012 at 10:57:24
<---> Preferences from lantern.ini have been read.
<---> Loading zone: gfaydark
<---> Screen width: 640 - Screen height: 480
<-!-> Could not load the PFS archive!
<-!-> Could not initialize the S3D file!
<-!-> Fatal Error: Unable to load zone data!
<---> Fatal Error: Unable to initialize Lantern!

If you're in Windows, the .s3d should be in the same folder as the SLN. Try moving it there. The error there (so glad I did error logging) is from the .s3d not being found.

[PixelEngineer]

Installing an api hook against the client and logging seek/read against files of that extension would be a start. That's half the picture, anyway. The other half is a memory scanner that locates what was just read in the client's memory. You can then trace execution from that point.

Thank you.

Do you have a guide on how to do this?

[Derision]

Is anyone here experienced in reverse engineering files? I have been going through and documenting every type of file the client uses. There is one file however, which seems to stump me.

The file is denoted by:

<shortname>_sounds.eff



I had a look a this, since it piqued my interest.

First of all, since you suggested it may be ZLIB compressed, I wrote a quick program to read one of the files and then move through it byte by byte and attempt a ZLIB decompress.
That failed, so it wasn't zlib compressed.

The next thing I did was open up eqgame.exe in IDA and found where the string %s_sounds.eff was referenced and looked at the dissasembly.

I could see that it reads the file 84 bytes at a time, and indeed, the few files that I looked at were all multiples of 84 bytes.

Next thing I did was write a quick program to read one of the files, 84 bytes at a time, and print out each group of 4 bytes as both an int and a float.

I could see from my output some floats that looked very like groups of co-ordinates to me, so I went in game and did a #goto to some of them to verify
there appeared to be a sound emitted in that location (some were obvious, like camp fires or steam geysers).

Anyway, this is what I figured out:


Offset
00 int8 Unknown[12]
12 int32 Sequence Number ? // Increments by one for each entry in the file
16 float Y
20 float X
24 float Z
28 float Radius? // Radius from position that sound can be heard ?
32 int8 Unknown[52]

[PixelEngineer]

I had a look a this, since it piqued my interest.

First of all, since you suggested it may be ZLIB compressed, I wrote a quick program to read one of the files and then move through it byte by byte and attempt a ZLIB decompress.
That failed, so it wasn't zlib compressed.

The next thing I did was open up eqgame.exe in IDA and found where the string %s_sounds.eff was referenced and looked at the dissasembly.

I could see that it reads the file 84 bytes at a time, and indeed, the few files that I looked at were all multiples of 84 bytes.

Next thing I did was write a quick program to read one of the files, 84 bytes at a time, and print out each group of 4 bytes as both an int and a float.

I could see from my output some floats that looked very like groups of co-ordinates to me, so I went in game and did a #goto to some of them to verify
there appeared to be a sound emitted in that location (some were obvious, like camp fires or steam geysers).

Anyway, this is what I figured out:


Offset
00 int8 Unknown[12]
12 int32 Sequence Number ? // Increments by one for each entry in the file
16 float Y
20 float X
24 float Z
28 float Radius? // Radius from position that sound can be heard ?
32 int8 Unknown[52]


Absolutely gorgeous. Thank you so much. I have IDA in my toolbox but I don't really have a great understanding of how to use it. Did you read a guide on it or something?

That leaves virtually no files in the original EverQuest that are unknown now. I am in the process of interviewing for a job so that can explain my absence. I am still however, working on this project and am redoing the way the zones load with WLD classes with several levels of inheritance for their specific purpose.

Thank you again Derision. Very much appreciated.

[Taurinus2]

You payed 1000USD for a license to software you don't know how to use? How did you get your developer confirmation for it?:???:

If you got it through an unauthorized channel (would seem the case) then I can say with high confidence that you are root-kit'd (by someone other than yourself).

[Derision]

You payed 1000USD for a license to software you don't know how to use? How did you get your developer confirmation for it?:???:

If you got it through an unauthorized channel (would seem the case) then I can say with high confidence that you are root-kit'd (by someone other than yourself).

I think you are jumping to conclusions. I was referring to the IDA freeware 5.0 version in my post, and it is quite possible PixelEngineer has the free version too.

[Taurinus2]

True. I've forgotten that IDA Pro was once freeware. The warning about root kits still stands tho (for all who might read this).

PixelEngineer,

Have you ever considered switching to Qt Creator as an IDE? This would give you a nice IDE on multiple platforms and relieve you of having to maintain a makefile for linux. qmake really is quite capable and Qt Creator can be used even without using Qt libs in your project.

You would still need a compiler on Windows, but you can always use the latest Platform SDK to keep in sync with the compiler technology. You can also install Visual Studio and just use Qt Creator although that would seem a bit overboard when all you need is the compiler (which is what I do, because I use both and I also need the assembler).

mingw is another option (Qt Creator can use this toolchain also).

Just tossing ideas out there.

[PixelEngineer]

The IDA I was referring to was indeed the freeware version but for others that may have obtained it in other ways, thank you for the warning.

I like the idea of Qt although I am curious as to how much work it would do in translation for this project. I don't mind not using Visual Studio and if there would be something that would compile well on Mac OS (the other OS I want this client on), that would be fantastic. It is funny you mentioned it because two days ago actually, I downloaded the Qt framework and started messing around with it.

Can you list more advantages for making the switch?

[Taurinus2]

Bare in mind that I am only suggesting the Qt Creator IDE and not developing with the libs themselves (although Qt is quite solid - not really suited to your project).

The most profound thing in my mind that stands out is the ability to plug a new toolchain in without much hassle. You just invoke qmake and it can generate a makefile for the new toolchain. This simply cannot be said about Visual Studio.

Having the exact same IDE with the exact same interface on all of your target platforms is a really nice thing to have for productivity. Also, Qt is a standard lib in Debian/KDE distros (and probably more) so you would not face a dependency problem in getting up and running there.

Qt Creator can also use cmake if you wish.

In short, it just jives with cross-platform well.

Browse through the docs for qmake and see what you think.

[PixelEngineer]

From what I see, I really like this. A cross platform IDE is kind of a dream. I have learned a bit about Xcode but the lack of information about the newest Xcode (4.1) and its working with SDL is a little disheartening. If I choose to go with Qt as my IDE, what are we looking at with additional libraries/dependencies. I don't want the client to have a lot more external DLLs.

Cheers

[Taurinus2]

None. It uses your existing toolchain so your app will have whatever dependencies you had already and nothing more.

[PixelEngineer]

Small project update:

Hey guys. I think my next commit will actually be a switch to github. I could still push to assembla but I really like what I see from the way github works. I have ironed out the next steps and I am pretty sure the next version will support the loading of all the data needed to load a complete zone. This include objects, zone NPC models, skies, sounds and music. Most of the usage of these resources will not be implemented but the framework for their loading will all be in place.

It might be a while before I push to github but the next version will also include support for Mac OSX. I have decided against using Qt simply because I feel that learning Xcode is something that will come in handy down the road. That was a fantastic suggestion nevertheless and I will definitely consider it for future projects.

Any suggestions for the next release?

[PixelEngineer]

It's a start:

http://i.imgur.com/3tIui.png

Cheers!

[Akkadius]

It's a start:

http://i.imgur.com/3tIui.png

Cheers!

That's awesome guys :)

[ableard]

Yes, the s3d goes in the with the SLN.

this work as advertised (hehe).

Quick question concerning back clip plane. There appears to be a fixed geometry for the back clip plane, do I have this right. Would much prefer that we can adjust this. I thought I could set this in the ini file but I can not get his working. Also in the code it looked as if we can adjust it some what but once we reach 2000 (I think this was the number) we gain nothing else.

I can increase the back plane distance also by increaseing the display dimensions, but the frustrum looks to have a geometric constraint built in.

Do I have this right or have I missed something.
thanks

[PixelEngineer]

I am in the process of rewriting all of the render code but as far as I know, the frustum pulls the clip distance from this line in init.cpp:

gluPerspective(75.0f,(GLfloat)width/(GLfloat)height, 0.1f , 10000.0f);

The last value in that command represents the furthest view able region.Again, this will be changed when shader rendering is in place. Hope this helps though.

[ableard]

Update!!!
I have modified the code to display the entire zone!

The dislpayable regions is hard wired by the wld decryption by way of veiwable regions from each region.
The above gluperspective is true, you can move out as far as you want but the displayable world reaches a maximum, my system it was 2000.

Except!

I modified the code inside 0x22 to compile a new list of viewable "regions" for each region to see ALL regions. Shazam the entire zone gets loaded and viewed. I have written the code to read in all veiwable regions as the WLD is loaded in, so if there is a zone with funny regions they are no displayed. The orignial code also checks for compatability with region and mesh not sure if this is required.

I have used eastkarana/westkarana as my test and they work great. My video card has tons of memory so frame rates are unchanged. I actually suspect with some additional work we can load all four Karana files as one (ie corrdinate transformations required, maybe scaling).

Since your reworking your client, I will add my modifications after you are complete. Load times become longer as expected. I suspect with some work I can improve or improvise a faster way to load.
Cheers!

[jayfromtholuxe]

Just found this thread. Awesome work!

Huge +1 for moving to Github and using Xcode! I can't speak highly enough of Github. I've seen many, many projects move their code to Github and afterwards find the community and collaboration tools there a tremendous boon.

By the way, what are your plans for the server component of this? Do you want to write something from scratch or try to use EQ Emu eventually somehow? If you want to do something from scratch I'd love to help you out. I have a lot of experience doing this for web and telephony infrastructures.

I ended up here doing some research for a side project I'm considering starting, very similar to this but using WebGL / HTML5 for rendering. I don't intend to reuse the existing EverQuest data files, however. For an example of what this technology can do, check out this game:

http://plsw.net/webgl/wloom6/shooter.htm

My first step is a build the backend game logic server with game mechanics as close to EverQuest as I can get. After that, I'll work on the graphics. It sounds like our needs for the backend are very similar.

By the way, I tried to build the code on Lion and got this error:

https://gist.github.com/f1f69fa98a15c54e6f30

It looks like the Makefile isn't wiring up the system install of OpenGL well for GCC.

[PixelEngineer]

Update!!!
I have modified the code to display the entire zone!

The dislpayable regions is hard wired by the wld decryption by way of veiwable regions from each region.
The above gluperspective is true, you can move out as far as you want but the displayable world reaches a maximum, my system it was 2000.

Except!

I modified the code inside 0x22 to compile a new list of viewable "regions" for each region to see ALL regions. Shazam the entire zone gets loaded and viewed. I have written the code to read in all veiwable regions as the WLD is loaded in, so if there is a zone with funny regions they are no displayed. The orignial code also checks for compatability with region and mesh not sure if this is required.

I have used eastkarana/westkarana as my test and they work great. My video card has tons of memory so frame rates are unchanged. I actually suspect with some additional work we can load all four Karana files as one (ie corrdinate transformations required, maybe scaling).

Since your reworking your client, I will add my modifications after you are complete. Load times become longer as expected. I suspect with some work I can improve or improvise a faster way to load.
Cheers!

I am very confused. Lantern will only display the regions that are in the current zone's PVS. The PVS is a list of region's that you can see from your current region. This is to cut down on processing time and thus, framerate. The fact that the whole zone does not render at once is not a bug, it is how the original client worked.

The render function (although rewritten since) does work in the same way.

Here's the process.

1. Where is the camera? Iterate through the BSP tree until the current BSP region is found.

2. What can we see from this region? The vector visibility has the list of all of the regions you can see from the camera's region.

3. What regions can the camera see? The visibility vector is used again and the regions min/max values are checked against the camera's frustum. So only regions that are in the PVS and the camera's frustum will actually be rendered.

Now there is a special case where the camera is outside of any region. This would mean there is no PVS and thus no visible regions but then you couldn't see where you are. Therefore, there is a second function renderWholeZone in the WLD class. This will render the entire zone regardless of where you are.

In the Greater Faydark for example, what you are seeing might be regions "popping" up in the distance but you would never see those based on the clip plane of the zone and the fog. Lantern renders the zones using the same method.

If you want to force it to render the whole zone regardless, just change the render function in main.cpp:

From

// Render if we in a region -- else render the whole zone
if(!outside)
{
// If we are in a region, render every region in its PVS and frustum
zone.renderNew(cameraRegion, cfrustum);
}
else
{
// We are not in a region, render the whole zone
zone.renderWholeZone(cfrustum);
}

to just:

zone.renderWholeZone(cfrustum);


Hope this helps alleviate confusion.

Cheers!

[PixelEngineer]

Just found this thread. Awesome work!

Huge +1 for moving to Github and using Xcode! I can't speak highly enough of Github. I've seen many, many projects move their code to Github and afterwards find the community and collaboration tools there a tremendous boon.

By the way, what are your plans for the server component of this? Do you want to write something from scratch or try to use EQ Emu eventually somehow? If you want to do something from scratch I'd love to help you out. I have a lot of experience doing this for web and telephony infrastructures.

I ended up here doing some research for a side project I'm considering starting, very similar to this but using WebGL / HTML5 for rendering. I don't intend to reuse the existing EverQuest data files, however. For an example of what this technology can do, check out this game:

http://plsw.net/webgl/wloom6/shooter.htm

My first step is a build the backend game logic server with game mechanics as close to EverQuest as I can get. After that, I'll work on the graphics. It sounds like our needs for the backend are very similar.

By the way, I tried to build the code on Lion and got this error:

https://gist.github.com/f1f69fa98a15c54e6f30

It looks like the Makefile isn't wiring up the system install of OpenGL well for GCC.

Thanks for the kind words. On my next commit, I will be moving this project to github and including an Xcode project which does build on Lion. As for the server software, I am not sure yet. The thing about EQEmu, although it is very well written is that is was built to work with inflexible code, the client. With an open source client, we don't just have to send/respond to packets, we can design them ourselves.

I am not very versed in network programming but when I started this, I wasn't well versed in graphics programming.

Let me know what kind of ideas you have.

Cheers.

[ableard]

PuxelEngineer, I did see the zone.renderWholeZone. But when I used that not everything was displayed. I thouhgt it ws unfinished.

If you pull up qey2hh1, run up and look east. You do not see the entire zone, down by the river is not complete, no river, no river bank, there are patches out in the distance that are open. Some of the edge hills are not displayed.

[PixelEngineer]

PuxelEngineer, I did see the zone.renderWholeZone. But when I used that not everything was displayed. I thouhgt it ws unfinished.

If you pull up qey2hh1, run up and look east. You do not see the entire zone, down by the river is not complete, no river, no river bank, there are patches out in the distance that are open. Some of the edge hills are not displayed.

I will check that zone when I am home. In the meantime, I'd like to see your changes to the 0x22 fragmentfunction. Again, the idea isn't to add all of the regions to the PVS. The PVS is super useful as a first pass method of reducing the number of regions to consider for rendering. But if you are just rendering with renderWholeZone, and still see missing regions, chances are there is a bug and I seem to remember a region missing when exploring Sleeper's Tomb and just assuming it was the location for an object or something silimilar. Thank you Ableard.

Post the changes on pastebin or something and send me a link if you would. :)

Cheers!

[PixelEngineer]

Thank you for noticing this bug. Turns out you were right. There is a problem with the renderWholeZone function. I will go ahead and fix that before the next release. As for the normal rendering function, everything looks fine. The zone regions pop into view when they are needed. Remember there will be fog and clip in this zone as well.

Thanks for pointing this out to me though. I will go ahead and get it fixed up for the next release.

Cheers!

[ableard]

Very Good. So question about reading a zone of your own creation? Has this been revisited yet, besides azone. What is the limit in reading zones that you have run across, ie what era of
eq zones can you read?

cheers

[PixelEngineer]

Very Good. So question about reading a zone of your own creation? Has this been revisited yet, besides azone. What is the limit in reading zones that you have run across, ie what era of
eq zones can you read?

cheers

So far, the client has been tested with old zones and newer Luclin era versions of that same zone. I did away with SOIL (although a good library) because it was inflexible and I didn't really need it. All I need is the ability to load 8 bit bitmaps, to add an alpha channel to those and to load DDS format textures for the post Velious clients. I have no idea how it would react to the newest zones but I doubt I would get far seeing as how most of them are in a new format.

However, I may eventually add support for them. As far as our own zones, try it. I am sure they would load properly.

[PixelEngineer]

A proposal.

Hey everyone. I am still quite busy with school and such and I had a rather interesting idea. As I am pretty much a one man team working on an engine, I will likely never get this client up and running as solid as I have wanted. Sure, I will eventually implement full rendering, animation, UI and possibly client support but in a lot of those cases, I am reinventing the wheel.

I have had an idea that combines the work I have already done with a preexisting engine. I'd like to finish rendering with shaders, finish up some bugs and then switch to an open source engine. The problem with many EQ projects is that people try to convert the EverQuest format to a more common format and import it into an engine (Unity, Unreal, etc...) which is neither suited or optimized for the way the zones were originally created. What I propose is we pick an open source engine and load the EverQuest data into that.

I have had my eye on irrlicht, which is written in C++, is cross platform and has quite a few examples of loading Quake style BSP maps. The EverQuest format is similar and with the already extensive documentation of the fragments, I don't think porting my work would be that difficult.

The downside would be that I would no longer be working directly with the core engine code (although as it is open source, I could still see/modify it). The upside would be that a lot of the features that will take quite a while to implement (collision detection, per pixel light, parallax mapping) would be supported natively by the engine and I could focus on making sure the data from the zones are translated into this engine accurately.

I am very excited about the prospects of a quicker development cycle. I feel my learning of shaders thus far has been fantastic and I also realize that I will still learn a ton while working with an established engine especially because it is open source.

My plan is to finish up this next release of Lantern and then after that, begin the process of porting my work to the irrlicht engine.

Any comments/suggestion/criticism are welcome!

Cheers!

[Tabasco]

If you're going to use an engine I would suggest Ogre3d. I've used both and Ogre is much more capable and polished. It also has a better asset creation pipeline.
I'm sorry that I'm not able to dedicate more time to this, there just aren't enough hours in the day. When I do get a little time I'm not likely to get very far since the current SVN appears to be way behind and I don't know where the git repository is.
I would be more likely and able to work on it if commits were more frequent, maybe you could establish a release branch for feature complete updates that are sure to compile and keep the trunk always up to date.

[KLS]

I wrote my eqg loader in irr first then ogre3d (though the latter was a hacky mess as I was in the middle of getting advanced materials working). I'm not sure if they've changed anything recently with irr but it was a lot harder to work with the low level primitives that I needed to work with than it was in ogre.

The downside was ogre was a bigger set of dependencies to get up and running initially.

I also agree you shouldn't convert the assets, whatever you pick should just load them.

[diriel]

I read recently that Valve is porting Steam over to linux. This got me to thinking of this UI project. I have been following this with some interest from an end user standpoint.

Pretty cool work so far!

[Tyen05]

Have you guys made any headway in porting animations over to a new engine?

[PixelEngineer]

Hey guys. A quick update. I have been finishing up another semester in school and have been pressed for time. I am going to have a pretty open summer and will likely finish the client as I can spend upwards of 40 hours a week on it. I have still been looking at different graphical engines. I really like what I see from Ogre3D and the ability to still do a majority of the logic work and have control over how the data is handled and yet, utilize the work the talented Ogre3D programmers have done on their graphics engine.

I will keep you posted. Expect big things.

[PixelEngineer]

Have you guys made any headway in porting animations over to a new engine?

Tyen05, as I mentioned before, porting the EQ zones to another engine is very hard. Importing them natively via code is a much better route. If I remember correctly, you are using Unity. You may in fact have to write a converter which takes EQ animation fragments and outputs them in a format Unity understand. I am not saying it is impossible but the exact issues you are having are the reason I have not decided to use a closed source engine.

Cheers!

[diriel]

Hi PixelEngineer,

I know when you started this thread you said you would never enable networking. I thought I would touch bases with you and see if you had perhaps had a change of heart at all. I for one would love an open source linux client :) That would just rule.

That being said, once you feel satisfied you have got a solid grasp on the engine aspect, will you be developing an MMO? :)

Have a good one,
Diriel

[Tyen05]

outputs them in a format Unity understand.
Cheers!

A format in which any 3d modelling software understands would be just as good. Such as Maya/3d max.

[Tabasco]

The hardest part of converting EQ zones is wrestling with the various file formats. It looks like WLD is a well understood format, but none of the existing work on EQG files gives me any texture information (or I haven't found it yet, OpenEQ source might be handy here but all the links are broken.)

This loads the .map files the servers read so some of the geometry is omitted, but it would be easy to tell azone to output mapfiles with full geometry if one wanted to.
http://dungeoncrawl.us.to/eq/mapvis/?zone=crushbone

For WLD zones those json map files could easily include textures and UV's. The only thing you lose is the EQ spatial database of choice. Most modern engines use an octree scene graph with frustum culling by default, so just breaking the zone geometry into reasonably sized chunks should be serviceable.
I have some progress on a zone converter that borrows heavily from azone2. It can output the zone model and all the placeable objects as individual model files, but textures are hit and miss.

[KLS]

Sent you a PM, I have a partially working EQG zone viewer somewhere (up to v3) it's a mess as it was just a prototype but it's got better material support than OpenEQ and should be able to figure out how it is all parsed at least.

[Tabasco]

The zone viewer code has been really enlightening. Between the nice weather and other obligations I haven't has as much time to work on it as I would like, but I'm making progress here and there.

Rather than try to load all the various archives based on zone name and resource text files I've just decompressed everything into a resource directory. Since most of the compressed resources are binary the uncompressed directory is only about 10% larger.

[PixelEngineer]

Back at it again. More exciting stuff to come.

[rhyotte]

Can't wait to hear more!

[iso]

This is really awesome, amazing work! I'm glad to see how active you are.

I've been looking for a good client for zone viewing for a while now and this looks fantastic. Is there anyway I can get windows binaries of the latest release?

[PixelEngineer]

http://i.imgur.com/kBL5y.jpg
http://i.imgur.com/ib0XW.png
http://i.imgur.com/89wIt.jpg
http://i.imgur.com/RJIrX.png

[Amer]

Pretty awesome! Nice work Pixel.

[rhyotte]

Looking good! Hurry up with a full blow Linux Client! :)

[PixelEngineer]

Looking good! Hurry up with a full blow Linux Client! :)

I will definitely make this client run on Linux. This was pretty much a complete rewrite of the Lantern engine. I decided to work on OSX in the beginning because:

1) I wanted to learn Xcode and the OSX filesystem
2) OSX annoyingly only supports up to GLSL 1.2

Rather that work backwards in compatibility, I just wrote shaders that ran on the least compatible system. Get your stuff together OSX devs.

The next step is models and animation (probably will be done in 1-2 weeks). I will also figure out a compatible way to translate the model animations to a standard format to help those devs that have the Unity project going. Who would have though it would be easier in some respects to write an engine rather than one as capable as Unity :grin:

Cheers!

[KLS]

Both Lion and Mountain Lion both support up to OpenGL 3.2 with GLSL 1.5.

It's honestly pretty shitty support as 3.3 has been out for over two years and they don't support any of the 4.x profiles (derp) but not quite as bad as being stuck on 2.1.

[PixelEngineer]

Both Lion and Mountain Lion both support up to OpenGL 3.2 with GLSL 1.5.

It's honestly pretty shitty support as 3.3 has been out for over two years and they don't support any of the 4.x profiles (derp) but not quite as bad as being stuck on 2.1.

http://i.imgur.com/Gx3Ot.png

http://i.imgur.com/gW54Y.png

[KLS]

https://developer.apple.com/graphicsimaging/opengl/capabilities/GLInfo_1074_Core.html

Something is messed up with your setup.

[PixelEngineer]

https://developer.apple.com/graphicsimaging/opengl/capabilities/GLInfo_1074_Core.html

Something is messed up with your setup.

Good find. Not sure why that's happening.

[KLS]

I think i read somewhere the earlier drivers with the 10.7 update didn't support HD3000 very well or something. Perhaps it has something to do with that? Don't know really but in theory you should have GL3.2 support (which is roughly DX10 level capabilities).

[cavedude]

Found you on Reddit ;) http://www.reddit.com/r/everquest/comments/z1m59/by_request_informationscreenshots_for_my_original/

[PixelEngineer]

Found you on Reddit ;) http://www.reddit.com/r/everquest/comments/z1m59/by_request_informationscreenshots_for_my_original/

Yes you did!

Transparency:

http://s13.postimage.org/5b4izmqgl/Screen_Shot_2012_08_31_at_4_10_17_PM.png

[Caryatis]

That link is hilarious.

One guy been working a year on some screenshots and thinks being able to fly around will be awesome(maybe eqemu should support that? o wait)

Everybody else thinking that just because you can support new features, that it won't take a skilled artist to produce the things in their heads. Nothing against you Pixel, the engine is definitely an awesome project.

[Tabasco]

It would probably be more valuable to make a client that can authenticate and enter a world first. From there you can just use placeholders as you develop your modelling pipeline. At that point you have the real beginnings of a functional client instead of a model viewer.

The work on the various file formats is excellent, but considering that eqemu is probably the most successful open source MMO, a client that doesn't have a bunch of non-free attachment or lawsuit risk would be incredible.
Asset creation is daunting but at the point you have a cube running around inside a box and can see and interact with other cubes, community involvement would probably take over. You could even borrow quite a bit from a place like opengameart.org. Asset creation is hard, but it's a process, and if I can model and rig a character in blender, anyone can.

[PixelEngineer]

It would probably be more valuable to make a client that can authenticate and enter a world first. From there you can just use placeholders as you develop your modelling pipeline. At that point you have the real beginnings of a functional client instead of a model viewer.

The work on the various file formats is excellent, but considering that eqemu is probably the most successful open source MMO, a client that doesn't have a bunch of non-free attachment or lawsuit risk would be incredible.
Asset creation is daunting but at the point you have a cube running around inside a box and can see and interact with other cubes, community involvement would probably take over. You could even borrow quite a bit from a place like opengameart.org. Asset creation is hard, but it's a process, and if I can model and rig a character in blender, anyone can.

If I am not mistaken, this was the basis for Windcatcher's Simple Client. That project was fantastic but I would really love to support the actual zones of EverQuest and I don't think it would be legally problematic unless copyrighted materials were modified.

Take a look at all of these open source engine recreations: http://en.wikipedia.org/wiki/Game_engine_recreation

I doubt very many of them have run into legal trouble. That being said, I am really not that far away from the transition between zone viewer to client. I realize how much work will need to get done for it to be a full blown client but it is still a goal will work towards. I can't wait for the day when I can stop working on the graphics side and focus on the actual client game programming.

My posting the link to reddit wasn't the greatest idea (especially on my actual account) but I was proud of what I had and wanted to share. I think people there are hopeful for what this project can really be, as am I. Regardless, I want my contribution to this community to be an open source client that people can use for whatever they want.

Cheers

[Tabasco]

You've got plenty to be proud of, the engine is good work. It's not precisely open source at this point, but that's your call.

We have no way of knowing whether or not SOE will pull a page from Blizzard's playbook, but that's not the point. Eqemu is a server framework that can be devoid of non-free content but it's bound to a non-free, evolving client.
Windcatcher's client was in delphi and never worked without a custom login server as far as I know, so it's a pretty poor example.
I just hate to see a promising project consumed by the hassles of backward compatibility with a client that already does a pretty good job of representing the nostalgia of classic EverQuest.

[rhyotte]

Cannot wait for the open source linux client!

[PixelEngineer]

You've got plenty to be proud of, the engine is good work. It's not precisely open source at this point, but that's your call.

We have no way of knowing whether or not SOE will pull a page from Blizzard's playbook, but that's not the point. Eqemu is a server framework that can be devoid of non-free content but it's bound to a non-free, evolving client.
Windcatcher's client was in delphi and never worked without a custom login server as far as I know, so it's a pretty poor example.
I just hate to see a promising project consumed by the hassles of backward compatibility with a client that already does a pretty good job of representing the nostalgia of classic EverQuest.

The client isn't open source because I am ironing out the ugly details in my code. Any programmer of an open source program knows that it is often a daunting process of sending your code out for public scrutiny.

I understand the concerns of backwards compatibility but here's my thoughts:
- I want this project to be compatible with old hardware.
- I want to have this client run as fast as possible.
- I want this client to run on all platforms (including Android and iOS)

and most importantly,

- I want this project to teach me about graphics and game programming.

That's about it. I do appreciate the feedback and criticisms.

Cheers

[PiB]

First poster here, also working on re-creating a EQ client using OpenGL. Seems to be a popular pastime here :)

PixelEngineer, are you using backface culling? I have tried to turn it on but this removes a lot of faces that should be visible:

http://i.imgur.com/OXNLa.jpg

When backface culling is disabled:

http://i.imgur.com/FkXpd.jpg

Looks like the normals and winding order of many faces are wrong in the s3d files. Or maybe there's some flag to invert them that I overlooked? In Blender, the normals of the Pedestal are pointing down instead of up:

http://i.imgur.com/b42rw.jpg

I could just leave it disabled but this sometimes causes z-fighting (look at the water in the second image).

[KLS]

If you know the order of the vertices you can just calculate your own normals (hint: you do!) which is what we do in azone for .map files.

[PiB]

I know how to calculate face normals, but I think the order of the vertices in the file is wrong. Half of the faces use clockwise order (first picture) while the other half use anti-clockwise order (second picture).

The reason I mentioned the normals is that, if all the normals were pointing in the right direction I could detect which faces have the wrong order and change it. Unfortunately it seems faces specified clockwise have wrong normals and vice versa.

http://i.imgur.com/Joy3H.jpg

http://i.imgur.com/JtH69.jpg

[PiB]

My bad, I was looking at the faces from the wrong direction. Looks like they are all specified in clockwise order. Asking OpenGL to cull front faces did the trick. D'oh!

[PixelEngineer]

The zones were originally written for rendering with DirectX and it differs from OpenGL on the direction of the Z axis. Scaling your model view projection matrix by -1 in the Z direction. Then you need to make the front face wind clockwise and cull the back face. That should do it.

Are you using Blender for viewing the models or are you using it as your graphics engine?

The more projects the merrier. If you are interested in helping, I will hustle and get this github. Regardless, the more information that is out there, the better.

Cheers

[PiB]

I was flipping Z before but not setting the winding order so that's why I was surprised when half of the scene disappeared when I tried to turn on face culling! It seems to work fine now.

I started this project by writing a Python script to load WLD data like zones, zone objects, characters, skeletons etc and import it to Blender as a kind of prototype. When this worked well enough I rewrote the code in C++ and used OpenGL. But I kept the scripts around which can be handy for debugging sometimes.

The more the merrier, I agree! I will upload this code to GitHub too so we can share the code and information.

There are some features you mentioned I haven't got around to do yet (properly supporting transparency, animated textures, minor things like sky box...). How is your work on animations/skeletons going? I am currently trying out different ideas for implementing lighting, it's not as straightforward as I thought it would be.

[rhyotte]

This is getting better and better! Yaay! Can not wait till I can drop windows and run just linux and still be able to play EQEMU natively :)

[PixelEngineer]

I was flipping Z before but not setting the winding order so that's why I was surprised when half of the scene disappeared when I tried to turn on face culling! It seems to work fine now.

I started this project by writing a Python script to load WLD data like zones, zone objects, characters, skeletons etc and import it to Blender as a kind of prototype. When this worked well enough I rewrote the code in C++ and used OpenGL. But I kept the scripts around which can be handy for debugging sometimes.

The more the merrier, I agree! I will upload this code to GitHub too so we can share the code and information.

There are some features you mentioned I haven't got around to do yet (properly supporting transparency, animated textures, minor things like sky box...). How is your work on animations/skeletons going? I am currently trying out different ideas for implementing lighting, it's not as straightforward as I thought it would be.

I am currently working on animations/skeletons. I have not run into any problems. It's just a matter of getting all of the fragments loaded in a way that they can be used quickly when rendering.

For transparency, you really need to use the BSP tree while rendering. I assume you could get away without it but it would be much more work. I render every visible surface that is in the PVS and frustum recursively going front to back to prevent overdraw. Every time I come across a batch of polygons that are transparent, I add the offset and information to my "transparency stack". I chose the stack because you need to render back to front with transparency and a stack is an ideal data structure given the order of entry while rendering front to back.

For animated textures, make sure you have created a texture data structure that supports numerous bitmaps. One of the unknowns in the 0x04 fragments is the millisecond delay between texture switching. Keep track of the time and if it goes over the amount in the delay, switch index of the bitmap you will use for that texture.

The skybox was a bit more tricky. Just picture someone walking around with a dome around their head. Clear the depth buffer and render as usual. I can elaborate on this if needed.

As for lighting, I have implemented just the zone lighting that was in the original zones. Instead of dynamic lighting or lightmaps like Quake 3, they simply shaded the polygons with the color of nearby lightsources at each vertex. They essentially "faked" the lighting.

I will work towards getting my code on github this week as well. Probably getting static player/NPC models loaded would be a good place for me to take a small break. Let me know if you have any other questions in the meantime.

Cheers!

[PiB]

I am currently working on animations/skeletons. I have not run into any problems. It's just a matter of getting all of the fragments loaded in a way that they can be used quickly when rendering.

I think one issue you will probably run into is that many characters have few or no animations you can load through 0x14 fragments. I am pretty sure the reason was to save space instead of duplicating animations. Many characters seem to share animations. For examples, barbarians, dark/high/half elves, erudites and humans all use wood elf animations with some additions (see video (http://www.dailymotion.com/video/xtl9i8_equilibre-human-animations_videogames)). Same for dragons and quite a lot of mobs. I have made a list of the most common vanilla/Kunark/Velious characters I could find and which animations they use.

For transparency, you really need to use the BSP tree while rendering. I assume you could get away without it but it would be much more work. I render every visible surface that is in the PVS and frustum recursively going front to back to prevent overdraw. Every time I come across a batch of polygons that are transparent, I add the offset and information to my "transparency stack". I chose the stack because you need to render back to front with transparency and a stack is an ideal data structure given the order of entry while rendering front to back.

How do you sort from front to back, do you do it per object/region using its AABB or per face? Or can you traverse the BSP front-to-back somehow? I thought the division between planes were arbitrary. Anyway that's one more thing I have to implement, right now I'm using an octree and frustum culling for this. I guess this is not the most efficient. But it will probably come in handy for keeping track of characters. One thing I was wondering, isn't the usefulness of the PVS limited in outdoor zones like the Karanas where you can see from very far away? Obviously I'm sure this works pretty well in dungeons.

For animated textures, make sure you have created a texture data structure that supports numerous bitmaps. One of the unknowns in the 0x04 fragments is the millisecond delay between texture switching. Keep track of the time and if it goes over the amount in the delay, switch index of the bitmap you will use for that texture.

I am using 2D texture arrays for textures so that I can draw static objects (and eventually characters) in one draw call. This should make it straightforward to have animated textures. Each vertex has a third texture coordinate for the layer in the array. I plan to add a kind of offset table in the vertex shader so that you can specify alternate textures (if animated, or to have different kind of equipments) without changing texture bindings.

The skybox was a bit more tricky. Just picture someone walking around with a dome around their head. Clear the depth buffer and render as usual. I can elaborate on this if needed.

How did you determine the scale of the dome? Do you use some kind of scaling factor that you multiply with the width/length of the zone?

As for lighting, I have implemented just the zone lighting that was in the original zones. Instead of dynamic lighting or lightmaps like Quake 3, they simply shaded the polygons with the color of nearby lightsources at each vertex. They essentially "faked" the lighting.

I think I have tried something similar. I started with determining which lights affect an object and send the array of lights used in the vertex shader for the object. This approach didn't scale very well with a lot of lights. I tried to compute the per-vertex lighting once, when loading the zone files. Then I didn't have to do any lighting in shaders but the result was quite ugly (since it's done per-vertex and old zones have very large polygons). I will try deferred shading next (so I can do per-fragment lighting in one pass) but I think this will be quite a lot of work.

I will work towards getting my code on github this week as well. Probably getting static player/NPC models loaded would be a good place for me to take a small break. Let me know if you have any other questions in the meantime

Sound good, keep up the good work!

[PixelEngineer]

I think one issue you will probably run into is that many characters have few or no animations you can load through 0x14 fragments. I am pretty sure the reason was to save space instead of duplicating animations. Many characters seem to share animations. For examples, barbarians, dark/high/half elves, erudites and humans all use wood elf animations with some additions (see video (http://www.dailymotion.com/video/xtl9i8_equilibre-human-animations_videogames)). Same for dragons and quite a lot of mobs. I have made a list of the most common vanilla/Kunark/Velious characters I could find and which animations they use.



Is that your program? Nice work. Do you have a copy of the source?


How do you sort from front to back, do you do it per object/region using its AABB or per face? Or can you traverse the BSP front-to-back somehow? I thought the division between planes were arbitrary. Anyway that's one more thing I have to implement, right now I'm using an octree and frustum culling for this. I guess this is not the most efficient. But it will probably come in handy for keeping track of characters. One thing I was wondering, isn't the usefulness of the PVS limited in outdoor zones like the Karanas where you can see from very far away? Obviously I'm sure this works pretty well in dungeons.


The BSP tree has done pretty much all of the work for you. A BSP tree is made up of arbitrarily sized regions divided by split planes. Things in front of each split plane are found on the left side of each tree node and things behind it will be found on the right. The way to correctly render front to back is to recursively iterate the tree visiting the child nodes the camera is in front of first.

In terms of rendering transparency back to front, as I mentioned, I use a stack. It holds the offset in my VBO as well as the number of polygons. Because a stack is a last in first out data structure when I render front to back the polygon batches that go in come out last.

Here is some tree traversal code demonstrating what happens:



// Calculate the distance to the split plane
float distance = (camera.getX() * tree[node].normal[0])
+ (camera.getY() * tree[node].normal[1]) + (camera.getZ() * tree[node].normal[2]) + tree[node].splitdistance;

// We are not at a leaf
if (distance > 0)
{
renderGeometry(cameraMat, tree[node].left, curRegion);
renderGeometry(cameraMat, tree[node].right, curRegion);
}
else
{
renderGeometry(cameraMat, tree[node].right, curRegion);
renderGeometry(cameraMat, tree[node].left, curRegion);
}

I suppose you can use an octree but you will still have to take the BSP tree into consideration for information about region types (water, lava, PvP) and if you want to use the PVS. It is true that the PVS is pretty inefficient at reducing a lot of the regions you can't see but it's a very inexpensive check to do.


How did you determine the scale of the dome? Do you use some kind of scaling factor that you multiply with the width/length of the zone?


I think you are misunderstanding what skydomes really are. EverQuest's skydomes are small half spheres that translate and rotate with the camera. They are drawn first and give the impression that the sky is very large despite it being exactly the opposite. Picture someone walking around with a sky textured bowl on their head. This is essentially the idea and because it moves with them, it gives the illusion that the sky is vastly infinite. If you were to stretch the skybox over the entire zone and walk the distance, you could notice approaching the edge of the diameter and it would look a bit weird.

The first thing you should render is the skydome. Then, clear the depth buffer. Because the skydome is inches away from the camera if you didn't clear the depth buffer, none of the zone would render because it's all further away than the skydome actually is. After clearing the depth buffer, render as you usually do. It will give the illusions that behind everything rendered is a vast sky.


I think I have tried something similar. I started with determining which lights affect an object and send the array of lights used in the vertex shader for the object. This approach didn't scale very well with a lot of lights. I tried to compute the per-vertex lighting once, when loading the zone files. Then I didn't have to do any lighting in shaders but the result was quite ugly (since it's done per-vertex and old zones have very large polygons). I will try deferred shading next (so I can do per-fragment lighting in one pass) but I think this will be quite a lot of work.


First, determine what your goal is. My goal is have the zones rendering as close to classic EverQuest as possible. The original lighting was simply precomputed vertex colors which are blended with the textures to give the appearance of lighting. Objects also have vertex colors as the EverQuest client did not dynamically shade objects. I assume the lights.wld contains lighting details just for shading player and mob models.

After I have everything rendering, I will move on to per pixel lighting. You are correct that per pixel lighting with the provided surface normals will not look good at all. You really need to be utilizing normal maps for any surface that is rendered with phong shading.

[PiB]

Is that your program? Nice work. Do you have a copy of the source?

It is. Thanks! I pushed my local Git repo to Github (https://github.com/pixelbound/equilibre) if you want to take a look. This should build on both Windows and Linux (haven't tested MacOS X) using CMake. I really should add a README with build instructions.

The BSP tree has done pretty much all of the work for you. A BSP tree is made up of arbitrarily sized regions divided by split planes. Things in front of each split plane are found on the left side of each tree node and things behind it will be found on the right. The way to correctly render front to back is to recursively iterate the tree visiting the child nodes the camera is in front of first.

I didn't know that the planes in the BSP tree always split the space into front / back half-spaces. Thanks for the clarification.

In terms of rendering transparency back to front, as I mentioned, I use a stack. It holds the offset in my VBO as well as the number of polygons. Because a stack is a last in first out data structure when I render front to back the polygon batches that go in come out last.

Here is some tree traversal code demonstrating what happens:

...

I suppose you can use an octree but you will still have to take the BSP tree into consideration for information about region types (water, lava, PvP) and if you want to use the PVS. It is true that the PVS is pretty inefficient at reducing a lot of the regions you can't see but it's a very inexpensive check to do.

Yeah, I'm working on using this BSP tree now. This should be faster to traverse than my octree and I need it to determine the ambient light of the region anyway (even though many zones seem to have the same ambient light in all regions). The traversal code looks really straightforward, thanks.

I think you are misunderstanding what skydomes really are. EverQuest's skydomes are small half spheres that translate and rotate with the camera. They are drawn first and give the impression that the sky is very large despite it being exactly the opposite. Picture someone walking around with a sky textured bowl on their head. This is essentially the idea and because it moves with them, it gives the illusion that the sky is vastly infinite. If you were to stretch the skybox over the entire zone and walk the distance, you could notice approaching the edge of the diameter and it would look a bit weird.

The first thing you should render is the skydome. Then, clear the depth buffer. Because the skydome is inches away from the camera if you didn't clear the depth buffer, none of the zone would render because it's all further away than the skydome actually is. After clearing the depth buffer, render as you usually do. It will give the illusions that behind everything rendered is a vast sky.

Ah, that makes sense. I think I understand now. Does this mean you render the sky box in camera space?

[ankhamunn]

Hey guys. I was able to get VS2010 and AnkhSVN working and was able to even get the code compiled and running. It's a pretty big thrill when you're able to open gfay and have a look around in a completely separate client, compiled on your own machine.

Is there still being work done on this? PE - I sent you a PM and an email but haven't heard from you yet.

Also - I'm not very familiar with building/branching public SVN projects but I'm reading a book that highly recommends organizing any game development code into several folders: Docs, Media, Source, Obj, Bin, Test. Any thoughts on doing this early in the process (before the client gets more elaborate than it is now)?

Nice work though - I'm really excited to have this resource. Also, huge kudos for commenting and organizing your code such that someone can jump into it and understand, basically, what's going on! I have little/no experience with C++ or graphics in C but you've put together a project that makes sense to me - VERY Exciting!

[ankhamunn]

So I've spent more time tinkering with the Assembla repo and VS settings and realized that I can do all of the custom folder structure stuff on my local computer by myself - Silly me! Apologies - I've never used VS before so still getting used to all of it.

I also took the time tonight to read through some of the earlier posts in depth (rather than skimming through). Any word on when you'd feel comfortable releasing the code on github? I figure the assembla code would be good to get a leg up on what's going on but would it be worth working on this code at all?

[PixelEngineer]

I am still working on this client, albeit slowly. I have something super interesting I am working on. Still classic but will definitely change the way EverQuest is viewed. I will let everyone know when I am done.

Thanks for the interest in developing. I know there is another developer around working on some things as well. It's pretty cool as he's pretty much got animations figured out and I was able to help with other stuff I had already finished.

I will update soon.

[rhyotte]

Friggen TEASE !!

[ankhamunn]

No kidding ;) Looking forward to seeing the progress!

[KLS]

Un-sticking this for now as it includes more closed source development than current open source development. Which is fine but not deserving of a sticky post in a FOSS development forum any longer.

[Tyen05]

Animations/models/textures all extracted.

http://bit.ly/X5VUln

[Tyen05]

Example - animations in unity

http://bit.ly/VAqjuG

[Amer]

Awesome Tyen! What's your next steps?

[bristle]

Animations/models/textures all extracted.

http://bit.ly/X5VUln

took one of the models and convert fbx to dae and imported to opensim. then i converted it to obj and import to zbrush. i am sure i could do it for blender too and it has fbx export.

so in theory, i could convert fbx to collada. import to blender. add more animations and then export back to fbx.

i dont know what to do after that for EQ.

[bristle]

well good luck. i am going back to opensim.

[bristle]

i was going to write about 3rd party sl client and opensim and EQEmu but i am not ready. right now, i can do everything i need in unity and a fake server. then i can transfer in a 3rd party sl client in c++.