EQ-Zip S3D/EQG Manager
 

So, I made a thing.

I took a break for a couple of weeks from other projects to recharge and try something fresh. I had some fun replacing textures in the client's zone files with higher resolution ones with the same look and feel.

It was taking too many steps, though, to extract a texture, convert it to a format I and Windows Explorer could work with, find a replacement, make any edits I wanted to, convert the replacement back to .dds, and import the .dds over the existing file in the .s3d package. With the same extension, if the extension was different.

I looked up the latest S3DSpy source code to see if I could patch in some convenient shortcuts, but while I played with Pascal back in high school, I don't have a Delphi compiler or up-to-date knowledge on it. So that was a bust.

I decided to see if I could make my own to satisfy my needs. As I researched APIs and added features, and saw how well things we coming along, I felt a seductive motivation to see just how good I could make my application.

You're going to laugh at how much effort I ended up putting into it.

You can read about it and download a compiled executable or the source code from my website, which also has several screenshots:

http://www.shendare.com/EQ/Emu/EQZip/

Or you can download the source from the Github link:

https://github.com/Shendare/EQZip

Here's one screenshot for the forum thread, though:

http://www.shendare.com/EQ/Emu/EQZip/screenshot3.png

Hope others find it useful, too!

Definitely cool

Just want to chime in to say that this step is unnecessary. The client's image loader will accept PNG/JPEG/etc images just fine, as long as the filename is what it expects.

I also have an S3D/EQG extractor thing with an image viewer an export-as-PNG option I made about a year ago... probably not much use to you at this point, though. Plus it's written in Lua mostly.

Thanks, Akka!

Zaela, I noticed that I could put non-dds images in as long as they were named the same, but the client doesn't appear to create mipmaps for them, so they get really nastily pixellated at angles. Floors were especially bad, because they're -always- seen at angles, never head-on.

I made it an option in this app to turn off auto-dds conversion with mipmaps, but I don't see why anyone would turn it off once they see it.

The larger the texture, the less noticeable it will be, but even at 1024x1024, it's still bad, especially when moving.

(Watch at 1080p, of course, to avoid mpeg artifacting.)

1024x1024 ground as a PNG (no mipmaps, flicker flicker flicker):

https://www.youtube.com/watch?v=b9E7zcLiGbU

1024x1024 ground as a DDS with mipmaps (less sharp, but not flickering):

https://www.youtube.com/watch?v=5lIzF2LooMQ

Ah! I didn't know that it didn't generate mipmaps. I've only really looked at PNG images on weapons and such, much harder to see at a distance I guess. Makes sense if they're always pre-generated, though.

On the other hand, makes me wonder how they handle images that really are BMP -- almost all of the images in the Titanium client .s3d files named ".bmp" really do have BMP headers, and there are still some from recent free-to-play downloads that are like that -- mostly old Mob/weapon textures, but some zone ones as well. I guess there could be some BMP extension to support mipmaps, or they could be some old transitional semi-dds bmp format, although google isn't helping me there. Or I guess the images could just be so low quality that running them through a filtering algorithm doesn't make a noticeable difference. Might be worth looking into, in any case.

I wonder whether it's an optimization they did to speed up zoning times. Instead of building mipmaps at zone time for every zone texture, they decided sometime after Titanium to bake them into the zone files. Makes sense, in a way, for computers with slower CPU/GPUs.

Oh my god, I took a look.

Not only are they actual Bitmaps with no mipmaps, but they're 64x64 8-bit palette indexed bitmaps. The ones I was looking at in RoF2 were 256x256 with 16-bit DXT compression, and even they looked blocky and low-def in-game.

I had no idea that when I was playing with the Titanium client, things were looking -that- bad. Walking around, the banding/flickering appeared to be present, meaning no mipmaps were in use, but the sheer low resolution of the textures alleviated the problem a bit, since the lack of definition made the textures so blurry.

Graphically speaking... Titanium looks like a last-resort type of client to use if you don't have anything newer.

Tagging in.

Tonight I discovered Github's Releases feature, and that it can be used for uploading not only compiled binaries, but screenshots as well. I don't need to host on a separate website for those!

I've created the Release for EQ-Zip 1.0, and will be working to migrate my other projects to Github releases.

Classic zones were revamped with Luclin expansion and can be found on those CD's with higher 256x256 resolution textures, but Titanium install didn't include those higher resolution s3d revamps because Sony was just too cheap to fit them onto another couple CD's.

LOD Bias also plays a significant part in flickering/shimmering or how sharp/blurry a texture is in game the sharper it is the more flicker/shimmering you get with it so it's a trade off really personally I like it sharp and can deal with the shimmer compromise myself. You need something like RivaTuner/Nvidia Inspector/ect to adjust LOD Bias though. The lower resolutions actually wasn't such a huge deal with CRT's, but they really bad today at 1080p or greater especially LCD's are much sharper as well.

I was scratching my head over the higher res textures thing. I remembered them announcing it, and couldn't for the life of me picture it happening after DoD instead of long before. Cost cutting in the CDs totally explains it. Thanks for solving the mystery!

I've been toying with the idea of looking up sharpening algorithms to make the mipmaps less washed out after rescaling. I'm sure there's got to be someting out there I could implement.

Unsharp Mask is a good image filter when used in moderation.

BIMP plugin for GIMP would be useful if you were trying to do a batch of unsharp mask filtering to group of textures.
http://registry.gimp.org/node/26259

DDS plugin is also useful for GIMP if you want to export to DDS/DXT file types which can also create mip maps as well for them.
http://registry.gimp.org/node/70

The Luclin upgrade textures weren't too bad and Psycher on p1999 did a nice character texture upgrade at one point too that was quite a decent improvement.

7/28/2015 - Version 1.1

https://github.com/Shendare/EQZip/releases

Release Notes:

DXT1 Decompression - Improved Slightly
DXT2 Decompression - Enabled
DXT3 Decompression - Added
DXT4 Decompression - Enabled
DXT5 Decompression - Improved Significantly
V8U8 Decompression - Improved Significantly

Known Issues and Planned Updates:

• DDS conversion does not support compression. This is intended. DXT compression is very lossy ( see http://www.fsdeveloper.com/wiki/inde...sion_explained ), and graphics cards have plenty of video memory for uncompressed textures these days.
  
• Support for reading and writing 16-bit uncompressed DDS textures is planned, as it would cut image sizes in half with much lower loss of definition than DXT compression.
  
• A feature will be added to vertically flip a texture, as some in-game geometry expects a texture to be bottom-up, and it can be hard to tell until you see it in-game.

A 2048x2048 texture with compression artifacts you will only really see if you are looking for them is usually much nicer than a 1024x1024 downscaled and blurrier texture where the higher quality details have been lost completely. Both take the same amount of ram and memory bandwidth to read, but the perceived quality increase of having 4 times as many pixels is significant. The average user probably won't be able to tell if you are compressing their textures or not, but most will be able to see a halving of texel resolution quite easily. DXT3 and 5 are similar if you use them in the correct context (5 for smooth alpha gradients, 3 for sharp alpha transitions).
http://www.gamedev.net/topic/611111-...aled-textures/


For Grobb retexture project I used 2048x2048 textures with DXT3 for smooth alpha transitions though id probably use DXT5 instead now and also use that GIMP plugin BIMP with unsharp mask set to like 1 or 2 instead. DXT2-5 are 4:1 compression ratio so the size of the textures used offset most or all lossy behavior to be on par with Luclin or Luclin revamps to classic zone 256x256 and 512x512 texture sizes. Titanium came with embarrassingly poor worse classic zone textures some were 32x32 I think and others 64x64 compared to the revamped textures Luclin provided for older zones.

Just use higher resolution base textures to compensate for the 4:1 compression if your worried about the DXT 3/5 lossy aspect.

Something worth mentioning that is quite interesting is hq4x or xBRZ filters http://sourceforge.net/projects/xbrz/
https://code.google.com/p/hqx/

Character models and objects in particular would be handy do use it with in fact someone at p1999 did experiment with hqx3 awhile back I'm assuming hq4x wasn't released at that point in time. Even better perhaps would be to duplicate the base texture once or twice each for a overlay layer of XBRZ/HQx4 filter versions of original textures then merge it down to blend the two together. You could also use some transparency between the filter layers and a unsharp mask filter on the top layer prior to merge down or after it just as way to sharper the image a bit wouldn't hurt either.

I'd probably have a ZBRZ layer with unsharp mask 25-75% transparency overlayed on top of a HQx4 layer with 25-75% transparency overlayed onto the original texture merge it down and export it however you want DDS/DXT3/5.

The two filters work a bit differently and look a bit different, but combining the base texture with both the other two filters and overlay plus optional transparency should provide a good mixture that looks a bit more natural with a fairly gradual cell shaded look that's not so harsh in terms of aliasing, but also not over smooth and cartoon looking either.
http://a.fsdn.com/con/app/proj/xbrz/screenshots/ga.png

Cool.

/10char

I love the quality of information, knowom!

I totally hear you on 2048x2048xBCn textures being preferable over 1024x1024xRGBA. You've given me plenty to think about.

Biggest impact on the EQ texture-swapping project is the availability of the textures. Do you have good sources of 2k textures? Most of the places I've found just googling around have been 1k, or even lower.

I made some from scratch with GIMP I have 2 links to a variety of them they were all 2k resolution. They weren't overly hard to make either. Some are more usable than others, but they don't look too bad I feel personally. http://www.eqemulator.org/forums/sho...light=textures

Had a few demonstration of textures in this thread I was curious roughly how they'd look roughly in practice.
http://www.eqemulator.org/forums/sho...light=textures

I experimented a bit on the 4xBRZ and HQ4x filters to see how blending them would turn out with GIMP I settled on using 4xBRZ as a top layer with Lighten Only mode and HQ4x as second layer with Darken Only mode applied.

http://i127.photobucket.com/albums/p...pslxjscnqj.png

I also did that same procedure, but then duplicate of default texture layer used for top layer with a overlay though you could probably get close to the same results by adjust transparency with the other two filter layers or similar at least. What it shows though is that the two filters along with original can be conjoined to get a more subtle and natural transition between them. The filters are mostly used for old emulators for sprite based games though they've been used in some old FPS games too like quake and doom to make them look more modern to decent effect.

http://i127.photobucket.com/albums/p...ps6frewv94.png

Ah, I see.

Making new textures from scratch is fantastic, though outside the scope of simply making it easy for anyone to drag and drop new big textures over the small ones in older EQ zone files.

I still have plenty to ponder between 32bpp, 16bpp, and block compression, at various texture sizes. Gonna have to do some experimenting in-game and see where tradeoffs for file size actually get noticeable.

I feel like I'm spamming, so I think I may hold off on thread updates from now on until I feel
the program is complete and every planned feature is fully functional.

https://github.com/Shendare/EQZip/releases

Release Notes:

7/29/2015 - Version 1.2

• Noticeable improvement in quality of texture mipmaps with high quality bicubic sampling
• Added full support for 16-bit DDS textures (auto-sensing A1R5G5B5 or R5G6B5)
• Set default import format to RGB16
• Code cleanup (Settings -> Util.cs)

Planned Updates and Known Issues:

• A feature will be added to vertically flip a texture, as some in-game geometry expects
  a texture to be bottom-up, and it can be hard to tell until you see it in-game.
  
• An "Auto" import format will be implemented to auto-sense the best DDS format to use
  for an incoming texture based on how it uses an alpha channel (A1R5G5B5, R5G6B5, or
  A8R8G8B8). It will be the new default.
  
• DDS imports do not support compression at this time.

Well... that actually didn't take long... I think I've got all the features in it that I wanted.

Here's the full Readme for EQ-Zip v1.3:

EQ-Zip EverQuest Archive Manager
-----------

Current Version: 1.3

Last Updated: 7/30/2015

Github Link: https://github.com/Shendare/EQZip

To Download: https://github.com/Shendare/EQZip/releases

Features:

* Creates, Loads and Saves .S3D, .EQG, .PFS, and .PAK EverQuest package files

* Thumbnails of all supported texture types (RGB16/24/32, DXT1/2/3/4/5, V8U8)

* Automatically converts textures to .dds with mipmaps when importing (toggle)

* Automatically converts textures to .png, .gif, .bmp, or jpg when exporting (toggle)

* Drag-and-drop files into or out of an archive and Windows Explorer, or between EQ-Zip windows!

* Cut, Copy, and Paste files between archives and Windows Explorer, or each other

* Select a texture and choose Replace... to easily browse for a new texture to swap it out with

* Or just drag and drop a new texture from Windows onto an existing one in the archive!

* Import/Export/Replace textures via: Application Menu, Toolbar Button, or Right-click context menu

* Export all files in an archive to a destination folder

* Recent menu feature to remember the last 9 archives worked with (toggle)

* .Net 3.5 compatible, but archives will compress about 30% smaller when compiled to .Net 4.5

Screenshots:

https://raw.githubusercontent.com/Sh...creenshot1.png

https://raw.githubusercontent.com/Sh...creenshot2.png

https://raw.githubusercontent.com/Sh...creenshot3.png

https://raw.githubusercontent.com/Sh...creenshot4.png

https://raw.githubusercontent.com/Sh...creenshot5.png

Disclaimer:

EQ-Zip is not affiliated with, endorsed by, approved by, or in any way associated with Daybreak Games, the EverQuest franchise, or any of the other compression/archive based applications out there with the word "Zip" in them, who reserve all copyrights and trademarks to their properties.

License:

Portions of this software's code not covered by another author's or entity's copyright are released under the Creative Commons Zero (CC0) public domain license.

To the extent possible under law, Shendare (Jon D. Jackson) has waived all copyright and related or neighboring rights to this EQ-Zip application. This work is published from: The United States.

You may copy, modify, and distribute the work, even for commercial purposes, without asking permission.

For more information, read the CC0 summary and full legal text here:

https://creativecommons.org/publicdomain/zero/1.0/

Credits:

EverQuest game package file format determined from examination of the Delphi code in S3DSpy, by Windcatcher, without whose work this would be impossible.

http://sourceforge.net/projects/eqem.../S3DSpy%201.3/

DDS texture file parsing rebuilt from code by Lorenzo Consolaro. (MIT License)

https://code.google.com/p/kprojects/

TGA image loading class from David Polomis. (CPOL 1.02)

http://www.codeproject.com/Articles/...a-Image-Reader

Virtual Windows Shell File handling functionality thanks to David Anson (MIT License)

https://dlaa.me/blog/post/9923072

Icons made from the freeware non-commercial "Aqua Neue (Graphite)" pack.

Release Notes:

7/30/2015 - Version 1.3

* Added feature to easily horizontally or vertically flip an image to get it to show up properly in-game if the geometry
expects the texture to be stored in some way besides left-to-right, top-to-bottom.

* Added help descriptions to the Preferences window for choosing Import and Export auto-conversion options.

* Corrected 24/32-bit to 16-bit color scaling. NB: Integers truncate; they don't round.

* DDS Import Format - Auto

By default, importing graphics files to an EQ-Zip archive will automatically convert them to a .dds texture with mipmaps.
The default .dds format is now "Auto", which tells EQ-Zip to pick the best format for the new texture based on the graphics
file's use or absence of an alpha channel.

* All pixels fully opaque -> RGB16 - (R5G6B5)
* All pixels fully transparent or opaque -> ARGB16 (A1R5G5B5)
* Some pixels partially transparent -> RGB32 (A8R8G8B8)

This yields the most compact format (without lossy compression) for any texture, while preserving good color and alpha channels.

If you do not like Auto, you can specify how many bits-per-pixel to import .dds files at (16, 24, or 32), or deactivate
auto-conversion altogether.

This completes my planned features. If you think of a new feature you want to see, or come across a bug, let me know
at Shendare at Shendare DotNet.

Enjoy customizing your EverQuest experience with EQ-Zip!

- Shendare (Jon D. Jackson)

-----------

Former Release Notes:

7/29/2015 - Version 1.2

* Noticeable improvement in quality of texture mipmaps with high quality bicubic sampling
* Added full support for 16-bit DDS textures (auto-sensing A1R5G5B5 or R5G6B5)
* Set default import format to RGB16
* Code cleanup (Settings -> Util.cs)

7/28/2015 - Version 1.1

* DXT1 Decompression - Improved Slightly
* DXT2 Decompression - Enabled
* DXT3 Decompression - Added
* DXT4 Decompression - Enabled
* DXT5 Decompression - Improved Significantly
* V8U8 Decompression - Improved Significantly

7/25/2015 - Version 1.0

* Initial Release

Got a bug report.

When extracting files (copy-pasting from EQ-Zip to a folder, using the "Extract All" or "Extract File" options, click-dragging files from EQ-Zip to a folder) from "erudnext_chr.s3d", the same error is thrown for a large number of the selected files: "Generic GDI+ Error"

'Eres a lil' picture to demonstrate.

http://i.imgur.com/o7yHBsv.png

And here's the copy-paste of the error details.

Edit: I should mention my specs.

Windows 7 64 bit, 16GB DDR3 @ 1333MHz, AMD Phenom II x4 965 BE, AMD Radeon HD 7850 2GB GDDR5

Cool. I'll take a look. There are problems with a couple other files, too. I think it's global_chr.s3d that crashes completely. Probably some wackiness in the way some of the oldest s3d files were put together.

I'm not too sure if it's to do with the more venerable files. Whilst erudnint_chr.s3d and fearplance_chr.s3d get the error as well (file creation date: 1999), fgh_chr.s3d (file creation date: 2006) gets the error, as does cub_chr (file creation date: 2012).

Did a quick google, and it seems this error occurs when a stream (whatever that is) is closed when the program has created an image file. 'Eres the Stack Overflow link.

Hmm... all _chr.s3d files. Time to dig in. Thanks for the link.

Oi'm rootin' fer yer, sah.

This program of yours is a lifesaver. Only thing it needs aside from the remedying of the aforementioned issue, is the possibility of extracting resources from multiple .s3d files at once. Then it'll be perfect.

Heh. Like 7-zip. Select a bunch of s3d/eqg files, right-click in Explorer, and choose "Extract Files to *\".

That's the stuff dreams are made out of, sarge. I'd readily sacrifice one of my virgin goats for such a possibility.

The StackOverflow link led to a post with the cause and fix. Got it cleared up now.

https://github.com/Shendare/EQZip/releases/

Release Notes:

9/8/2015 - Version 1.4

* Fixed: Extracting from _chr.s3d archives with lots of files inside should no longer yield a "Generic GDI+ Error".

* Known Issue: Some files may still have too many files inside for C# to easily parse. global_chr.s3d is one that is known, with 1,617 files stored in it. A stack overflow occurs even when not extracting the images from the stored files for display. I'll have to put together a way to split up the processing of obscenely large numbers of files into batches.

Glory be to Crom!

Much obliged for the fix, milordship. Time to get to extractin'.

Just to make sure, when you talked about the right-click -> Extract to *\ functionality, that was to mean what would the best method of implementing multi-extraction, rather than that feature already being present, aye?

Right. Shell integration is not part of the program right now.

Aye aye cap'n. Was wanting to make sure, 'cause if that was the case and I did it all manually, I'd be kicking myself. With a sharpish stick.

Release Notes:

9/11/2015 - Version 1.5

* Fixed: No more crashing from opening archives with large numbers of files. Let me know if you run into one that does!

* Fixed: Hopefully the last of the "Generic GDI+ Error" situations has been eradicated.

* Improved: Vastly improved the speed of opening archives with large numbers of files.

* Improved: Vastly improved the speed of importing and exporting large numbers of files. If more than three files are exported at once, you will get a confirmation of how many files were exported, to let you know when it's done.

* Improved: Application responsiveness has been vastly improved when handling archives with large numbers of files. There should no longer be any conflicts between archive decompressing, interface updating, and thumbnail loading. Scroll through the list of files all you like while the thumbnails continue to load in the background.

https://github.com/Shendare/EQZip/releases/

Release Notes:

9/13/2015 - Version 1.5.1 Patch

* Fixed: Opening files was resulting in an exception being thrown in a number of intermittent situations.

* Fixed: Item selection changes weren't always being detected and responded to.

* Improved: CRC errors in names of files inside archives will now be ignored. This allows gequip.s3d to be opened.

https://github.com/Shendare/EQZip/releases/

https://github.com/Shendare/EQZip/releases/

Release Notes:

9/19/2015 - Version 1.5.2

* Fixed: Saving global_chr.s3d will no longer result in an error. Its 1,617 files were taking up more filename space than I ever expected (29k bytes, where the largest other archive I'd seen needed around 8k). Now allocating 64kb of memory for the filenames directory section, instead of 16kb.

Seriously thank you for this. Made a Project I'm working on way way way easier. We should link up on discord if you're willing to talk.

Thanks again!