Joined: 24 May 2004 Location: Flanders (Be) Posts:300000001
Posted: Wed Aug 15, 2018 2:23 pm Post subject:
Engie File converter + Game formats documentation
Subject description: New tool and up-to-date format specs
Hello, PPM!
For the past year or so, I've been working on adding new game file formats into a tool I originally created to convert files extracted from the Nintendo 64 ROM of Command & Conquer.
In the past few months I finally added the classic C&C formats into the tool, and with the help of the people of the Chronoshift (formerly RedAlert++) project, I managed to make my tool create versions of these files that are virtually indistinguishable from the original Westwood-created ones, and, with some optimisations of my own, usually even better compressed.
At a certain point in the development I stumbled on the Shikadi.net DOS Games Modding Wiki, and decided to document a bunch of the formats I had implemented. All existing documents on these formats are ancient, and many of them are full of inaccuracies, so since we gained so much knowledge from disassembling the exe files in recent years, this was long overdue.
Well, this documenting process is finally finished, so, with some pride, I present to you:
(image can differ slightly from current version; this is my WIP of the next version)
The font formats are probably better edited with the Westwood Font Editor, though. They are included because this tool has the ability to convert them to sprite sheets.
These formats are not supported in Engie (only in the Westwood Font Editor), but since this is a listing of all WW stuff I added on the modding wiki, I'll go ahead and add them for completeness' sake:
Joined: 24 May 2004 Location: Flanders (Be) Posts:300000001
Posted: Wed Aug 15, 2018 2:56 pm Post subject:
You just put the palette in the tool's folder and it'll add it to the list.
There's a feature to save a palette from a file into the list directly, which also puts it in the tool's folder, but there seem to be some bugs in the refresh logic at the moment, and palettes added that way don't seem to show up unless the list is somehow reloaded
That said, any image can be saved as palette, so if you got some converted image of a TS SHP you can just open it, go to File-> Save, select "6-bit palette", and save it in the tool's folder that way. But you'll likely need a program restart to load it anyway.
There are also still some bugs in the transparency handling of the palettes... at the moment it's forced by file type, rather than set to an initial state per file type, meaning it often can't be changed despite the palette editor having features to edit it. I still need to dig into that and clean it all up. _________________ QUICK_EDIT
The details from the SHP format is shown framewise, that is nice.
Some thoughts:
- Could scan for palettes in any subfolder (or use a Palettes subfolder to start with)
- Could show the first frame of SHP instead of -1th frame, when opening
- Open dialog could use last used folder
- frame counter could goto first frame after the last frame in a loop
- background color for TS SHP could be automatically be the first palette color QUICK_EDIT
Joined: 24 May 2004 Location: Flanders (Be) Posts:300000001
Posted: Wed Aug 15, 2018 4:07 pm Post subject:
A palettes sub-folder... yea, that could work. I will need to give this thing an options dialog eventually I guess.
The reason the view is set to -1 by default is because the tool is a converter, more than it is a viewer, and switching to a frame will change all processing to be relevant to that frame rather than to the whole loaded sequence.
So, if you select frame 0, and then go to File -> Save, you will save that one frame only, and will not get the options to save as frame-containing types.
Changing it to frame 0 automatically would get especially tedious if you load a frames sequence into the tool to convert it (Open a file from a sequence and, like RAD Video Tools, it'll detect that it's part of a sequence and will ask to load it all), since you'd need to go manually back to index -1 to save it as the intended frames-containing file type.
Uh, the Open dialog does use the last-used folder. In what situation didn't it do that for you? Do note that if you press "Cancel" on an "Open file" dialog, there is no way to retrieve what folder you were last in.
Frame counter looping is impossible; it's is a standard Windows control. Also, not sure if it should go to frame 0 or frame -1 then anyway.
Background for TS SHP is automatically the first palette colour. What I said before was that the rule gets enforced on every palette edit so you can't change it even if you want to.
Do note that that any transformation you do on the frames, like combining the shadows, will make it stop being a "Tiberian Sun SHP file" and will turn it into a general "listing of frames". These take over some properties of the original, but will no longer truly be treated as TS SHP files.
_________________ Last edited by Nyerguds on Wed Aug 15, 2018 4:19 pm; edited 2 times in total QUICK_EDIT
When closing and restarting the tool, the open dialog is on the tool folder, not last used.
When I open a SHP file, then select the palette, the Background: for the image shown is still pink, was expecting the blue color from unittem.pal. QUICK_EDIT
Joined: 24 May 2004 Location: Flanders (Be) Posts:300000001
Posted: Wed Aug 15, 2018 4:22 pm Post subject:
Nothing gets saved if you restart it; the tool doesn't have any kind of settings file. If that bothers you, use drag-and-drop to load your files; it's faster anyway.
The #FF00FF fuchsia is the default transparency colour; it can be modified by clicking the little coloured square at the bottom next to the word "Background:". The default 'background' colour on the TS units palette is black (as you can see on the palette viewer in the image I posted), which is not a very good colour for indicating transparency.
Obviously, if I add a settings file, the selected background colour would be one of the things saved in it. _________________ Last edited by Nyerguds on Wed Aug 15, 2018 4:50 pm; edited 1 time in total QUICK_EDIT
Joined: 24 May 2004 Location: Flanders (Be) Posts:300000001
Posted: Wed Aug 15, 2018 4:49 pm Post subject:
Yeah, most palettes I work with from Dune II, C&C1 and RA1 have black as well, hence why I went with "nope, automatic override" _________________ QUICK_EDIT
Joined: 24 May 2004 Location: Flanders (Be) Posts:300000001
Posted: Sun Aug 19, 2018 3:59 pm Post subject:
I released v1.3.1, which fully supports sprite sheets even if the frames in them have differing sizes, like they do in Dune II.
It also has a palette matching feature for high colour content.
And I fixed the forced transparency bugs; file type based transparency is now applied to all palettes once at the moment a file is loaded, but can be freely adapted afterwards. It'll be reset when a new file is loaded, of course. _________________ QUICK_EDIT
Joined: 24 May 2004 Location: Flanders (Be) Posts:300000001
Posted: Mon Aug 20, 2018 7:51 pm Post subject:
The Windows GDI+ system that's used by the .Net framework for graphics operations has certain upper limits, yea... it starts bugging out when you exceed a maximum possible image buffer size. And if you start zooming in on 3kx3k images I guess you'll get there pretty fast, yes.
I've never seen it crash on that, though, usually it just bugs out a bit. Problem is, this is pure UI code, so nothing I can catch in a function. The closest I can do is check what these buffer limits are and restrict the zoom to stay below it.
I've mostly been working with graphics from 320x200 games on this, so uh, a zoom out feature honestly never crossed my mind. I might look into that though, yea
The second issue is an odd one, though. Could you give me the sprite sheet? Or at least tell me its bit depth and the exact settings you used? Note that in the latest versions the preview you see in the "image to frames" window is generated by exactly the same function as the one doing the final conversion, so you should be able to spot this by just going through the frames there... _________________ QUICK_EDIT
Yeah just limiting the zoom if it would bug out would be nice. I guess honestly I'm probably trying to do things with this program that it wasn't really designed for.
I attached the spritesheet and some images showing the issue in a zip.
Joined: 24 May 2004 Location: Flanders (Be) Posts:300000001
Posted: Tue Aug 21, 2018 7:17 am Post subject:
Thanks for the feedback!
On the first one... did you get that problem at 3x zoom, or when exceeding 3x zoom? I've been implementing a solution that maxes the control on an average of 10.000 * 10.000 pixels, but I'm not sure if this suffices. I may need to max out both dimensions at 10.000 instead.
Sadly though, the control (a standard .Net PictureBox) does seem to mess up consistently when zooming too far, though exactly how far seems rather system-dependent. It's a side effect of it actually reserving the full amount of memory for the zoomed image. I've looked for alternatives online, but while there are optimised solutions of the "cut out the actual shown part" kind, those would come without scrollbars, making it quite hard to select the piece to look at. And implementing my own scrollbars... ehhh, not really something I'm too keen on.
On the second thing... seems the dpi set in the image is 72, whereas the default in the .Net framework is 96. This indeed leads to oddities when copying pieces out, since the new frames I make are of course set to that default of 96. I thought the method I used avoided that by setting its measurement unit to "pixels" specifically, but it turned out that was only the measurement unit for the selected rectangle, not for the target... so the DPI difference problem remained
The reason this didn't show up in the preview was because the whole image-to-frames window works with a cloned version of the image, something I did for the sake of having it in the most optimal format for repeated calls to to the cutting-up logic.
Anyway, that one's an easy fix.
On the negative zoom one, I do have a bit of a problem... I can set it to convert negative values to fractions, but... default zoom is "1", meaning 1x zoom. I can't really make it jump from 1 to -2...
Unless I go for percentages, that is, but that'd mean a complete overhaul of the zoom logic, since there are currently some logics in there that count on exact-multiples zoom.
[edit]
Released v1.3.3
Solved the zoom issue by... indeed making it jump from 1 to -2 _________________ Last edited by Nyerguds on Wed Jul 03, 2019 10:03 pm; edited 2 times in total QUICK_EDIT
Pretty sure it did crash at 3x zoom, but now it seems to be working in the latest version. Didn't get any exceptions by zooming. The zoom out is nice too. Splitting that sheet into frames appears to work properly now too. Thanks! QUICK_EDIT
Pity this is of no help for centering frames of different sizes when separated given many sheets out there are sometimes hand made and thus are jumpy :/
and yeah it causes the odd vertical lines when give it 24bit image using the split but 32bit source is fine...go figure. QUICK_EDIT
Joined: 24 May 2004 Location: Flanders (Be) Posts:300000001
Posted: Fri Aug 24, 2018 5:01 am Post subject:
Oh wow, that's a silly bug... especially since I specifically designed the whole system to retain the original colour depth. And then right at the end I forget to pass that original format to a function at one crucial step in the conversion _________________ QUICK_EDIT
I released a new version; the problem should be fixed now.
ApolloTD wrote:
Pity this is of no help for centering frames of different sizes when separated given many sheets out there are sometimes hand made and thus are jumpy :/
Well, I can't really help with that. I can't just magically make this into a full editor with image shifting controls... that'd take loads of extra development. As it stands, this tool is only a converter. You'll need to use something like good ol' Gimp to solve that. I mean, you can always use this to convert it to frames, then dump those in gimp as layers to line them up, and re-save the frames you realign.
The only somewhat related thing to that here would be the trimming code, but that was really just meant for handling file types with unequal frame sizes, like Dune II, and it works in tandem with the equivalent "fill around frames" option in the frames-to-image conversion. The TS SHP saving would reject frames of unequal sizes anyway.
By the way... dunno if anyone cares, but I rewrote the Westwood CPS Format article on the shikadi modding wiki... which also means I dug into all that, and the tool now supports all types of CPS, including the Amiga types. Was a bit worried that they'd be hard to distinguish, but given the fact Amiga uses 5-bit image instead of 8-bit, and CPS images are always 320x200, one check on the uncompressed size value in the header was all it needed _________________ QUICK_EDIT
Joined: 24 May 2004 Location: Flanders (Be) Posts:300000001
Posted: Thu Dec 06, 2018 1:59 pm Post subject:
ApolloTD wrote:
Pity this is of no help for centering frames of different sizes when separated given many sheets out there are sometimes hand made and thus are jumpy :/
Mind you, I could put a simple "enlarge frames to same size" option... with the side options to select a fill colour, and to center instead of just filling the lower right. That's actually not hard at all.
But that'd only work with cropped frames anyway, and if they jump around they'd still need extra work... _________________ QUICK_EDIT
Joined: 19 Aug 2009 Location: Moscow State University
Posted: Fri Jan 18, 2019 3:48 am Post subject:
How can I convert a SHP(ts) file to SHP(td)? Or convert a frame sequence to a SHP(td) with a certain palette? When I try to save, the allowed format list does not include any SHP types.
/Edit: OK, I can only save as SHP when displaying frame -1, otherwise the converter thinks I want to convert it to images......
Joined: 24 May 2004 Location: Flanders (Be) Posts:300000001
Posted: Sat Feb 02, 2019 2:56 pm Post subject:
The tool does not do palette conversions, though... you have to do that yourself.
Though I guess a quick fix for that is to use the sprite sheet function so you have it as one image, open that in Gimp or something and convert it to RGB there, and then import it matching it to a TD palette.
[edit]
What the heck is up with that UI, though? The font is different, and the sizes are all messed up. It doesn't even show the zoom controls... _________________ QUICK_EDIT
Joined: 22 Nov 2010 Location: Iszkaszentgyorgy, Hungary
Posted: Sat Feb 02, 2019 4:42 pm Post subject:
Nyerguds wrote:
What the heck is up with that UI, though? The font is different, and the sizes are all messed up. It doesn't even show the zoom controls...
Haven't looked at your sources yet, but I suspect you use fixed window sizes, or relate them to text sizes. Chinese Windows installations have different font defaults, so .NET fonts are remapped with ones which includes both English and Chinese glyphs, resulting with the different font. _________________ "If you didn't get angry and mad and frustrated, that means you don't care about the end result, and are doing something wrong." - Greg Kroah-Hartman
=======================
Past C&C projects: Attacque Supérior (2010-2019); Valiant Shades (2019-2021)
=======================
WeiDU mods: Random Graion Tweaks | Graion's Soundsets
Maintainance: Extra Expanded Enhanced Encounters! | BGEESpawn
Contributions: EE Fixpack | Enhanced Edition Trilogy | DSotSC (Trilogy) | UB_IWD | SotSC & a lot more... QUICK_EDIT
Joined: 24 May 2004 Location: Flanders (Be) Posts:300000001
Posted: Tue Feb 12, 2019 5:33 pm Post subject:
Well I can understand the fonts being odd, then.
But the sizing of the image viewer is all messed up too, and that's simply set to align to the top/bottom/left/right of the form. Same for the palette viewer; the buttons are not supposed to be on top of it. And that's all fixed in size. _________________ QUICK_EDIT
Posted: Tue Apr 09, 2019 6:38 pm Post subject:
Elvira, Mistress of the dark
Hello.
I'm experiencing problems when passing bitmaps, pngs, etc to vga files in Elvira, mistress of the dark. For example, in the file 332.vga there's a picture of a castle with a poster that says "All visitors report to gatehouse". With this tool, I get all the pictures and, in this case, edit this picture, respecting all colors. Newly, I return all pictures to the vga file, but when testing to see the results, they are very different.
Joined: 24 May 2004 Location: Flanders (Be) Posts:300000001
Posted: Sat May 11, 2019 4:42 pm Post subject:
Huh, interesting. I haven't actually tested Elvira support so far; the project it was added for (at OldGamesItalia) kind of got put on hold.
I never got around to making anything to actually extract the palettes from Elvira, though, so I'm not sure where you got that palette. the best way to get a palette is just to make a screenshot in dosbox and extract it that way.
Could you tell me which file and frame this is, and maybe even give me the entire series of bitmaps? It's the easiest way to test this kind of thing.
There seems to be something odd going on with that Engie screenshot, by the way... it's zoomed to x2, but the zoomed version seems dithered. That shouldn't happen... _________________ QUICK_EDIT
The file is 332.vga and the frame is 2. I tried with a screenshot from dosbox, but no success. I also tried with the UI (frame 4), but no success. I tried the web colorcodepicker. colorcodepicker is a program that gives me the palette from an image. It gives me the below palette for frame 2.
Joined: 24 May 2004 Location: Flanders (Be) Posts:300000001
Posted: Sun Jun 23, 2019 1:27 pm Post subject:
The frames in these .vga files have no palette of their own.
I think the issue is simply that the editor you used to edit the image changed the palette. More specifically, it apparently reordered the used colours from dark to bright, and left off all the rest. If you take the first 16 colours of the palette in the DOSBox screenshot you attached, you'll see that the original data in 332.vga opens perfectly with that. I advise using Gimp for your editing; it preserves the original palette.
This image should work, meaning, you can also save the palette from it to view the rest of the data in 332.vga.
Joined: 24 May 2004 Location: Flanders (Be) Posts:300000001
Posted: Sun Jun 23, 2019 4:54 pm Post subject:
This is the palette you should have, by the way. I included the entire edited file so you can test it. If that works, the tool works fine, and it's really just the fault of your image editor. It changed the order of the colours on the palette.
It is hard to find good image editors that support indexed images, these days... Gimp is pretty okay, but it's horribly slow with them.
Side note: the palette management system inside the tool is currently a bit of a mess. I'm currently in the process of cleaning it up and rewriting broken pieces of it.
I edit for example the ui menu (vga 332 file 004) with your palette and gimp is giving me the same result. In fact, I take your palette, use it in gimp or paint.net and these programs transform it to a 13 colors palette. A very strange thing.
I'm going to try with photoshop. I'll comment the results.
Joined: 24 May 2004 Location: Flanders (Be) Posts:300000001
Posted: Thu Jun 27, 2019 1:23 pm Post subject:
Odd. Gimp should not mess with the palette. Though if you do any kind of palette loading in Gimp, make sure to disable the "optimise palette" option.
As for fixing your image... there is a "match palette" function kind of hidden in my tool, in the "Edit" -> "Image to frames" option. You can perfectly use this for palette matching if you just convert a single image to a single frame and then save the resulting frame as png file.
The last few new versions have added some new features and fixed some old bugs:
Fixed the flicker when zooming on an image.
Apparently, the N64 C&C1 map reading was broken, and always detected maps as the PC type. This is code from the Old Days of the CnC64 File Converter, before the autodetect was properly implemented, though, so I'm not surprised.
Added a new feature to convert an image or a set of frames to a specified palette. That was already possibly by combining frames to a single image and then re-splitting it, but this way is obviously much handier.
Adjusted the "Paste on frames" function to also work on a single image.
The "Palette to image" and "4-bit palette from 8-bit palette" functions now work whenever a palette is shown on the UI. Before, it didn't work on the -1 frame of frame container types with a common palette, despite the UI showing the palette there.
The "Palette to image" function's default filename is normally the name of the loaded file from which the palette is taken. However, if the loaded file is a paletteless format (like SHP) and the palette was instead selected from the dropdown, then the filename for the new image will now also be taken from that selected palette.
I'm currently working on a rather major overhaul of some obsolete internals. The game's supported file formats used to expose the "number of colours" inside the file, which I needed to correctly support palettes with less than the full amount of colours (16 for 4-bit, 256 for 8-bit) on the Nintendo 64 image format, to limit the shown palette despite the images generated from the data having a full-sized palette. This was an issue caused by the fact the .Net framework does not allow you to create custom-sized palette objects; in fact the only allowed way to even create palette objects is by requesting a copy of the palette object from an existing image object. And newly created image objects always have a full palette...
However, I have since developed a sneaky way around that; png images supports incomplete palettes, and the .Net framework loads those correctly, so I can simply construct a png file as bytes in memory, with the desired amount of colours, load those bytes as image, extract its palette, and then use that palette for my other image
Anyway, since that issue was solved quite a while ago, the only real use for the "number of colours" was that it was used to differentiate between images that contain a colour palette, and images that need to be loaded with an external colour palette (like SHP files). So I'm overhauling that to simply be an indicator of whether it needs an external palette, now. Which, for the now 58 different supported types in the tool, is a bit more work than it seems _________________ QUICK_EDIT
You can post new topics in this forum You can reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum You cannot attach files in this forum You can download files in this forum