76 replies to this topic

[butler]

Best current workaround i can think of (w/o editing) is to make a mountatin base (w/ everything at sea level) and when you get to 4.00 you jump to another map that is the next 6.00 meters. These 6.00 meters could be done by (since you don't need water/2D tiles since it's mountain side) deleting 2D tiles and working from the lowest walkable tiles, -2.10 iirc. Depending on how tall the mountain is, this might need to be done several times. On the overworld, i'm not sure, but building mountains rather high is a pain to make them convincing as well as missing wholes thanks to a terrible camera in the editor. However lag wise, I'm not sure, as i think it would be roughly like my flying palace, which isn't too bad atm.

Seperate map making rant, but loosely connected to mountains.
What i would REALLY love, is if the edges of maps weren't straight lines & WEREN'T framed by mountain ranges around it's entire limits of only ONE height. Looks so unnatural in every single way. Means there was no point in varying anything else as the effect is still lost.

[themuntdregger]

themuntdregger, on 13 August 2014 - 07:16 PM, said:

Still pretty much at the research stage of designing the 'alpha searcher'. However, have at least managed to work out how to unpick the dds data sufficiently to isolate the alpha element.

The data is held in a 4x4 blocks of 128 bits with the first 64bits of each block being used to carry the alpha data. Alas its all in 4bit values which require some mangling to make them useable. Once that's done, it should then be a matter of scanning the data for any files that have all the alpha set to white. Interestingly, there's no need to scan the whole file. DDS files are comprised of a series of copies of the same image (known as mipmaps), each at a lower resolution than the last. Hence, scanning the first mipmap tells you everything you want to know.

Atm, i've got as far as locating the various mipmaps but, have yet to start testing the individual blocks. Just to test out my mipmap calcs, I coded a  small dds image viewer using the glut library which, could be a useful additional option for e3d_conv.

Have now got as far as decoding the mipmaps for the DXT3 format which, as per the last post, is made up of blocks of 128bits or 14bytes. The first 7bytes of that deal with alpha data, with the first 2 bytes providing reference alpha values. By comparing these values, we can tell if alpha values are being used, ie where the first value (known as alpha_0) is less than the second (known as alpha_1). We then have to repeat this comparison for each block in the first mipmap, after which, we'll know if alpha is being used in the image. We don't have to bother looking at the subsequent mipmaps as they are just progressively smaller copies of the first.

As a preliminary, i'm testing the new code as part of the diagnostic report in e3d_conv and will be included in the next release. However, as the latter can't specify individual dds files (only the e3d through which the dds is referenced), i'm creating a separate stand-alone utility.

In researching texture compression, it seems possible that we could possibly convert some DXT3 to smaller and faster sizes by using the DXT1a format which, is essentially DXT1 with alpha. Unlike DXT3, which can handle multiple gradients of alpha, DXT1a only handles black or white. However, in many cases, that might be all that's necessary. In the meantime, the increase in client performance might be useful.

Whilst the client (as currently coded) can't handle DXT1a, it shouldn't be that difficult to implement.  What is more challenging is that DXT formats are lossy, ie data is lost through the compression process when the file is created and, each time it gets edited or changed. Hence, there's a possibility that the conversion process may effect image quality. Not sure if it would be possible to programatically mangle the data without loss of image which, might be a way around this.

EDIT version 1.8 of e3d_conv now released. Source and 32bit binary download from https://sourceforge....er/version 1.8/

[themuntdregger]

Ok peeps

I remember making this thread wayyyy back ages ago. It started off being about a new client and then kind of ended up focusing on tools to convert EL's e3d graphics format. Not much to say on the latter other than the separate tools that were used to convert from e3d to obj and vice versa were combined into a single tool called ul_conv, which you can find at http://unofflandz.co...lay.php?fid=20or https://github.com/U...-landz/releases

Learner - if you'd like to create a specific OL version you're very welcome to use the codebase, or even kick me and say 'Munt - will you do them for me'.  

Anyhowz, the orginal subject of the thread was about developing a new client, so I thought I might as well reuse the thread to update peepz on where i'm at, rather than creating a brand new thread. So yeah, it's got lots of developer related stoofz which some peeps may find more interesting than others. I guess the point of making the post, apart from the joy of making long self-indulgent monologues, is to involve the wider community and let them know where i'm heading and why.

So some background. The official OL project intends to build the new client from an engine called Unity. Nothing much wrong with that except it's a closed source engine which has to be scripted in object orientated C#. Whilst i'm not entirely unfamiliar with C# or the whole object orientation paradigm, I don't want to have to bee bothered with that kind of thing at the same time as grappling with the monster that is working out how a 3d engine actually works. So, I opted for an engine (Irrlicht) which is coded in shitty C++ (which I hate), but which is sufficiently C-like for a mere mortal to understand. Actually, i'd have gone with Horde3d which is significantly C-Like and properly fast or; Raydium which properly C99 (yay), alas both suffer from lack of documentation needed for a complete nublet to get a clue. Irrlicht, on the other hand, has masses of documentation and even several books.

So I started on my experiments with Irrlicht which, unsurprisingly involved trying to import an OL map and render it somewhat like it looks in the current OL client. That ofc is easier said than done. The elm format used to store OL maps is a EL proprietary format that only the OL/EL client's understand. Extracting binary data from an elm file isn't massively hard, but wrangling it into something that can be understood by another 3d engine turned ut to be more challenging than I imagined. Essentially, what you have in an elm file is three structures, ie:

a ) a tile map
b ) a height map
c ) a 2d object list
d ) a 3d object list

The tile map is the easiest of all to deal with. OL maps consist of a grid of tiles. in the case of Isla Prima, the grid is 32x32. The tile map consists of a linear array of values corresponding to the texture for each tile. You can find these textures in the 3dobjects directory of your OL client installation in the dds files  named tile1.dds to tile242.dds. As the data type used to store each entry in the tile map is an 8bit unsigned integer, the maximum possible tile values are 255, so it would seem that we're already pretty much at the limit of what the current client can support.

Rendering the tile map is fairly straight-forward, after which you get something that looks (in the case of Isla Prima) something like this :



Yup, simple enough. However, next is the 3d object list and it's here where things start to get gnarly. For starters, the structure is a mix of text and a variety of binary values, some floating point, some integer, some 32bit with a few 8bit thrown in for good measure. However, C coders laugh in the face of this kind of challenge and, with the magic of structs, we are able to wrangle this into a series of values specifying the coordinates of each object, it's rotation and the name of the mesh file (e3d) which draws the object. Yup, look in the 3dobjects directory of your OL client installation and you'll find a whole load of files with the e3d sufffice, each of which specify how  a particular 3d object should be rendered.

Warning LIES ALERT: actually no. E3d is another of those shitty proprietary EL formats that only the EL/OL client understands. Try asking your favourite 3d modelling app to read an e3d and it will produce a message from Barclays Sperm Bank reminding you why your account is well and truly overdrawn. Hence, our guilty secret is that all the e3d's have been converted to wavefront obj. Worse still, all the accompanying texture files are converted to png as the compression formats used in the dds files are patented, not by radu, but by the kind of peeps who can and do collect revenues.  

So once we've wrangled our object coordinates, first thing we discover is that the values look too small. The second thing we discover is that the y and z axis values are misplaced. Hence, we swap the y and z values and then multiply then double the x and z values. Next we deduct 25 from x and 26 from z gets us (near as damnit) to the right place. Alas, when we render, everything looks too small, hence, we scale up the x and z axis by 17. Also, because everything looks kind of flat, we scale up the y axis by 13. So now our render looks like...




Ok, all that was comparatively easy. Now for the uber gnarly 2d object list. Just like the last list, it's a combination of text specifying a 2do file and floating point values specifying position and rotation values. However, first you have read the 2do file which is text based and looks something like this

texture: dirt.dds
file_x_len: 255
file_y_len: 255
x_size: 2.0
y_size: 2.0
u_start: 0
u_end: 128
v_start: 0
v_end: 128
type: ground

So what the above specifies is a particular part of a kind of image file known as a texture atlas, ie a single image containing lots of different textures. Here's an example...



As can be seen, there's four different dirt patches in the image, one of which is referenced through the parameters given in the 2do file. Extracting the image is easy enough, simply use u_start and v_start to determine where the texture starts and then u_end-u_start  and v_end-v_start to determine the height and width. In fact I wrote a little application that does precisely that...




N.B The little sliders are so that you can edit existing  or create your 2do's.

So far so good, but rendering your 2do's in the right place is problematic. Once again, you have to swap the y and z axis, multiply by 2 and add on some for the Queen, otherwise you end up with this ....



However, even once you've corrected the positioning, there's still weird artefacts with the rotation. A good example is grass borders. whilst some are properly orientated, others look like ...



Clearly there's something going on that I don't yet understand, particularly when you look at the gang plank in this shot...



and there's a trio of weird doors....



Yup, so clearly things are still a 'work in progress', but still progress is being made.

[Kiwi06]

Lmao! I'm sorry munt. You worked so hard, but the ending is truly hilarious. Excellent work so far, and a good sum-up.

[Learner]

Part of the reason for being willing to allow a closed source multi-platform engine is so they worry about the enhancements & optimizations across all platform, so we can focus on the game client, not the rendering across multiple platforms, even Web, Android, & IOS.

[themuntdregger]

Learner, on 20 April 2016 - 04:26 PM, said:

Part of the reason for being willing to allow a closed source multi-platform engine is so they worry about the enhancements & optimizations across all platform, so we can focus on the game client, not the rendering across multiple platforms, even Web, Android, & IOS.

I remember the pain involved in working out how to do a windows build of the current client. I shan't miss that at all. Anything that makes Xplatform builds easier has to be a good thing.

And you'll be pleased to know that i've started experimenting with the dubious delights of C#/Mono under Unity.

[themuntdregger]

So a little more progress on the client front.

Firstly, I managed to work out the reason for the 'trio of weird doors'. Seems that anything on the 3d object list which has a blended value of 20 is in fact  redundant data that should be ignored. Hence, excluding this redundant data not only removes the weird doors, but a lot of other artifacts. Actually, thanks for this needs to go to BlackKat, as it was only as a result of reviewing the code for the elm2pov converter mentioned in his post that I got an insight into what was going on.

I also managed to sort out the misplaced 2d objects so as the various grass and dirt borders now all line up neatly, eg:







What i've done is to deduct 180 degrees from the z axis rotation where where that rotation is between 45 and 135 degrees. This seems to cure around 95% of all the previous artifacts, although a few still remain, eg :

The end of this path is a mess




There's a strange piece of grass jutting out into the garden



Not quite sure what the reason is for these few strange artifacts. I suspect that it maybe has to do with some subtle difference between the way OL/EL and Irrlicht handle mesh rotations. Anyhow, some more experimentation is necessary.

So a new issue that's becoming apparent are the rather distinct borders to the 2d objects. A possible cure for this may be to increase the alpha in the object so that the object border becomes slightly transparent.

I've also started messing around again with the water plane. I'd post some of the results but I think i'm up to my maximum images on this post. In any case, the results are appallingly shit at this stage. Part of the shittiness arises because creating waves on the ocean also results in similar waves on the ponds, and which looks ridiculous. I could just have one boring water texture for both, but it would be far nicer to have waves on the ocean and ripples n the pond.

So now i'm getting towards (what is hopefully) the end of the map rendering, next step will be to try and add some map editing functionality.

[Kiwi06]

Awesome work munt! As far as the grass 2d artifacts, it looks like they are rotated 90? degrees. That being said, I have no idea how the other 2d grass seem to line up flawlessly. My theory would be that certain degrees (180 and 360) are fine when they get mixed up this way, but things that are at different angles (45 and 90) provide the opposite effect. The oars on the boat seem to have been affected by this rotation as well.

Also, I would think the outlines would be brought about by increases in height or tilt angle, but I could be wrong.

[Grimm]

looks to me the height adjustments to prevent flickering in the map editor/client rendering load of maps are creating a lot of tile outlines.

[themuntdregger]

Firstly, thanks for all your ideas and encouragement guys.

Tried height adjustments and slightly changing rotations but it didn't help. Eventually, I did solve the issue.

The problem was in fact two things. Firstly the level of alpha and secondly the textures sizes were not 'power of two'. The first part of the problem i've solved by slightly increasing the alpha content to make the texture partially transparent. This makes the texture borders less well defined and therefore appear seamless (see below)




However, when you get really up close, you can still see vague artifacts that arise from the texture being slightly stretched. That's because some of the 2do parameters are not quite 'power of two', ie divisible exactly by 2. This results in the GPU slightly stretching the texture. I'm currently finding the best way to fix that. Modern GPU's also work best if the width/height of a texture is in the ratio 2:1 or 1:1, hence i've also to see if that can be achieved.

[BlakKat]

woot, congrats Munt!

[Grimm]

\o/ pr0

[themuntdregger]

So i've started to work on coding the basic functionality for a map editor. What i'm going to do is to have separate editor modes for tiles, 2d objects and 3d objects. In all modes it will be necessary to select a specific mesh for editing, hence, i've been working on the mechanism to do that.

So i've completed the code that detects what mesh the cursor is over (it can even detect the specific triangle in the mesh). Atm, the detected mesh is rendered in wireframe (as per Stardarks original Unity code). Only problem is that this isn't a good way to deal with tile editing as the required functionality requires the user to see changes in texture. I've therefore got to find a better way to highlight the selected tile, either by outlining or, by increasing specular luminosity.

At the same time, i've also got to start work on creating the basic gui allowing the user to select from a range of different textures and which will require the screen to be split into separate sections (plus a menu bar at the top).

Finally, i've still got to deal with the rotational issues that are causing the few remaining visual artifacts (like the reversed gangplank and weird grass strips).

[Kiwi06]

themuntdregger, on 02 May 2016 - 03:47 PM, said:

Finally, i've still got to deal with the rotational issues that are causing the few remaining visual artifacts (like the reversed gangplank and weird grass strips).

Was your original solution for the backwards objects to add 180 degrees to get them to flip? It seems like that is what's wrong with things that were already rotated: now those objects are backwards :s

[themuntdregger]

I really tried that, plus every other combination I could think off. The problem is that, if I apply a corrective rotation to those few items, everything else is then distorted. I've also tried euler-quaternion conversions plus no amount of other embarassing shit.

This isn't an excuse, actually i'm totally appalled and embarrassed. The problem arises simply because I am a ridiculous arse who is failing to understand something at a basic level. However, before I wallow in self-pity, i'm going to shatter the existing OL client code into a million tiny shards until I get an answer.

This is the way of the troll. We smash our stupid heads against the brick wall until it simply breaks.

[Kiwi06]

I guess it would take some work, but is there a way to isolate certain numbers to not get changed during the conversion? Like ignore those that are half rotated so you can get yer backwards ones rotated? It could possibly take two swipes through conversion to change your backwards things only. Not sure how it would work though. You're the genius here.

[Learner]

Kiwi06, on 05 May 2016 - 01:23 PM, said:

I guess it would take some work, but is there a way to isolate certain numbers to not get changed during the conversion? Like ignore those that are half rotated so you can get yer backwards ones rotated? It could possibly take two swipes through conversion to change your backwards things only. Not sure how it would work though. You're the genius here.

Before trying that, I think he would need to identify if there is a pattern as to which are messed up.

[Kiwi06]

The pattern looks like he originally had some that were backwards, so he flipped them opposite. Now everything is opposite, hence the grass angled inward in places and the oars and the gangplank flipped the wrong way. It's basically that the conversion doesn't translate 180 angles right in the first place. So he corrected that and now other things are flipped.

[Learner]

I was referring to finding patterns to the one that are still messed up vs the ones that are good in the 'best' he's gotten so far.

[Kiwi06]

The pattern is them being backwards. You can see them correctly in OL's IP map. In the pictures they are turned 180.