Jump to content

  •  

Map Auditing Tool


  • Please log in to reply
20 replies to this topic

#1 Learner

Learner

    God

  • Administrators
  • 2638 posts

Posted 02 August 2016 - 06:59 AM

I've enhanced a map auditing tool I've used to check for some issues with maps and dump statistics to print additional information and search for walkable spots it thinks a player would be unable to walk to (unreachable). Some of the additional information it dumps includes the location of lights, the distance and location of the nearest light to each (since excess lights can cause lag) as well as the location and type of every harvestable item (which will be disabled after initial testing by the mapmakers so secrets can be added) using the .def harvesting restrictions.

How it handles unreachable spots:
  • Reads the .def & .elm files using the normal server routines
  • Marks any use_area as being walkable (even if not reachable)
  • Marks any use_area destinations on the same map as walkable
  • Marks both ends of each telepad as walkable
  • Marks all spots as walkable that are reachable from a walkable tile using the height difference rules the server uses ... repeat until done
  • Prints the coordinates of any unreachable spots
From now on, any time I update a Test map in c0 on Test, I'll be running the script that does that and the other statistics & dumping of information into a .txt file. Those files will be available named as
http://dev.other-life.com/files/c0/map_name.txt 
.
So a test copy I have of RI can be found at http://dev.other-lif...ravens_isle.txt since the maps .def file is ravens_isle.def .

If you look at Ravens Isle, you'll see it has a lot of lights often close enough together that slow systems might be affected. I'd be interested in hearing from people that lag in IP if they lag more at night, specially in well lit areas to see if that is a factor with RI lag!

More changes may be added to it in the future! Also, the version of the file with harvestables will be available to the original map maker on request, even when it's hidden from normal access via the links.

#2 BattleWizard

BattleWizard

    Newbie

  • Members
  • Pip
  • 60 posts
  • Location-

Posted 02 August 2016 - 01:29 PM

Very nice implementation! hope this helps us greatly in the future.

Edit: http://dev.other-lif...c0/map_name.txt page not found.
and also, many small things can cause a certain area to lag. perhaps the script could note if lights are above 3.0 brightness, too many 3d objects in one place that reduce fps (see Magic School main hall)  and other small features to help future map makers.

#3 Kiwi06

Kiwi06

    Advanced Member

  • Full Member
  • PipPipPip
  • 544 posts
  • LocationMissouri, USA

Posted 03 August 2016 - 01:10 AM

Map_name.txt was just an example. Forum created that link on accident methinks.

Nice work L :)

#4 Learner

Learner

    God

  • Administrators
  • 2638 posts

Posted 03 August 2016 - 03:17 AM

There are lots of things that can cause lag which this tool isn't attempting to address. So far the focus has been more related to statistics and the hard to notice or find things. Currently nothing has been done related to objects other then the Harvestable information.

Keep in mind also that this is based on the server data & code which does not automatically load and store information about every object in a map. This will need to be addressed moving forward in a way that won't impact the server itself.

#5 Learner

Learner

    God

  • Administrators
  • 2638 posts

Posted 03 August 2016 - 08:36 AM

Currently experiment with using the data the maps gather for ranging for tracking map density in sections of maps. Initial tests show that PL & WS both have very few sections with >= 20 objects, with the highest being 43 ... but RI goes as high as 113 and lots of areas >= 20. Along with the lighting, this may be a factor affecting RI lag.

Will be working with Zian on this to determine what qty in an area is worth reporting on plus I need to translate that information into coordinates still. Those numbers by themselves don't show lag, but they can at a minimum help determine where lag testing needs to take place. There are other factors beyond that wich are not being accounted for which include the type, complexity, & type of textures involved.

The RI example data has been updated ... and you'll notice the unreachable list has been shortened already thx to Zian using the previous sample. don't try to use the coordinates yet since they have not been adjusted for the map itself yet.

#6 Learner

Learner

    God

  • Administrators
  • 2638 posts

Posted 03 August 2016 - 09:58 AM

The map data dumps have been updated so that "center x,y count" information now should get players close to the crowded areas. Anyone thinking to investigate need to remember there are other factors including exactly where you stand, the direction you are looking, the object characteristics, what else is near by, video settings, time of day etc that all can affect lag.

The idea of the new information is to help identify places where lag could happen from high 3D object density, or if an area is determined to cause lag, help quantify possible causes (lights, objects).

#7 themuntdregger

themuntdregger

    Official Troll

  • Full Member
  • PipPipPip
  • 1001 posts
  • LocationBehind you

Posted 08 August 2016 - 12:07 PM

Nice,

Are you able to open-source the code as a reference example for style, but also so the community can help develop and bug test ?

#8 Learner

Learner

    God

  • Administrators
  • 2638 posts

Posted 08 August 2016 - 12:12 PM

The code uses actual server for certain things so it's view is as close to possible as to how the server uses the information on purpose.

On the other hand I'm also going to be getting one tool from Burn which will help visually show some issues and mark up a map with information from the .def's as well Will have to see if I can share that code or not.

#9 themuntdregger

themuntdregger

    Official Troll

  • Full Member
  • PipPipPip
  • 1001 posts
  • LocationBehind you

Posted 09 August 2016 - 10:49 AM

Thanks Learner,

If you're using the server then I understand and agree why you can't release that code.

Impressed that Burn is sharing with OL (well done to him). However, if the code isn't open-source, we should look to reverse engineer an open-source OL version rather than build dependency on code over which we have no control. Might be worth inviting him to come talk to us on the developers forum, if only because we probably have just as much of value to share with him.

#10 Learner

Learner

    God

  • Administrators
  • 2638 posts

Posted 09 August 2016 - 10:55 AM

I have Burns tool now on an as-basis and while not officially open source it can be shared with others. Currently there are a bunch of other changes I'm putting into it for my needs, so letting others have it yet could risk a fork.

I'm currently setup to run my MapAuditTool and Burns one one or more maps easily and and preparing to add a way that OL map makers will be able to submit .def & .elm's to be processed. The processing will be in a batch fashion since some maps can take 2-3 minutes to run and that's a simple way to avoid timeouts.

There are lots of reasons NOT to try to build a clean open source version. Including the fact that you basically shut everything down, turn everything off, and start the design process from scratch which means you can expect nothing will be usable at all for a VERY long time. How long have you been working on your server?

#11 themuntdregger

themuntdregger

    Official Troll

  • Full Member
  • PipPipPip
  • 1001 posts
  • LocationBehind you

Posted 09 August 2016 - 06:49 PM

As you know, I love you to death and support you in all things. That's partly because you is cleverer than a barrel full of bears, but also because it's mainly your money and effort going into this gig. Basically, i'm just a freeloading idiotic troll who just really enjoys playing your game and who really enjoys the fruits of your efforts.

With regard to my server, i'm fairly realistic. Someone more experienced in C, or better connected to EL, could prolly have produced a much better effort lmao. Tbh, i'm amazed I managed to produce something that actually works, let alone works pretty well. For this reason, my own project isn't a fair example of the virtues of open sourcing,  A better example is the work I did in developing the e3d/obj converter which, i've no doubt you could have done but, as a result of my releasing that code as open source, saved you the time and effort in doing so.

Tbh, I really enjoyed doing so and was kind of honored that you even asked me to help on that project. However, that's the power and potential of open-sourcing.

#12 Learner

Learner

    God

  • Administrators
  • 2638 posts

Posted 13 August 2016 - 02:45 PM

To summarize the current versions of the Audit Tool that uses the server code:

1. Uses the exact server parsing and reports errors (to be improved still)
2. Verifies that the .def points to the proper.elm and prints basic information from the .def
3. Read the .elm and reports on general information such as dimensions etc
4. Prints out all Lights and the distance & location of the closest light (to help avoid too many lights close together(
5. Prints how man use_areas is find in the .def
6. Audits use_areas for min/max coord confusion (min > max for example) - ERROR when fails
7. Audits use_areas so at least one walkable in the area - WALKERR when fails
8. Checks the size of the area for being smaller (can have false alarms) - SIZE when > 100
9. Audits attribute_areas, location_infos, & infos for min/man coord errors - ERROR when fails
10. Checks the object location & size vs the the use_area for overlap or being very close. Failure means either the wrong object_id supplied or the wrong location for the area reported as OBJERR
11. Checks the entire map as explained in the first post and prints and unreachable plaves that are walkable
12. Checks how many objects are in a 10x10 area and reports on any areas > 25. This by itself isn't an issue, but with really high number or multiple medium qty's close together that can be a source of lag (RI is was improved using this & lighting info)
13. Prints out all location_info & info tags with coordinate, name, and text
14. Optionally prints all harvestable locations & type (off by default to protect secrets).
15. ... more to be added

The text based AuditTool does all this in a second or less even for complex maps!

The MappingTool I got initially from Burn and have been working to improve includes:
1. Generating a basic tab map with 512x512 resolution that is too detailed but useful until real tab maps are available
2. Generates a 2048x2048 map with all talkable tiles marked on top of the objects (can help find walk thru objects)
3. Can generate some text warnings including unreachable tile using different methods then AuditTool
4. Marks various areas for review (defaults include use_areas, use_object, location_info, info, and a small set of attribute blocks), optional public info can include more attribute areas, particles, teleports, and light locations.
5. Marks optional areas on demand, these images are not normally released including harvesting objects, objects that could be harvested, objects that are too far from walkable tiles to be harvested, with text reporting of unreachable harvest objects
6. ... more being added

The MappingTool takes 1/2 - 3 minutes to run with the normal default settings and part of the work being done includes making it easier to change the settings.

I've also created a password protected site that can be used to upload .def, .elm, & .zip files and I can fire off the processing with a single command. Working towards the proper way to trigger the processing safely from the webserver as well. All the outputs from both tools are available in the same site.

I'm also currently preparing to make the tab maps available via the docs server to authorized users so they can be used to help with making the real tab maps.

#13 Learner

Learner

    God

  • Administrators
  • 2638 posts

Posted 25 August 2016 - 04:15 AM

A quick update ...

Additional checks related to use areas have been added for missing objects (such as deleted from the map) have been added to the AuditTool. The MappingTool has gained some features in preparation for marking additional types of items.

We've also added new information to the maps .def file for use with a new ContinentMaker tool that can automatically combined the tabmaps into a continent. The first draft version ran in under 3 seconds and produced the output is available from http://www.other-lif...iles/Alroth.png

The ContinentMaker tool will also help when we start adding more maps by allowing the mapmakers to make sure their maps align with others. Future enhancements are planned.

Edit: The clipped sample has been updated with one that has smoothed out some of the larger textures to be less bothersome (grass, sand, roofs, etc)

#14 Learner

Learner

    God

  • Administrators
  • 2638 posts

Posted 26 August 2016 - 06:53 PM

New feature being worked ... letting the mapmaker see the most common paths that could be used between the different use_areas using map walking to get to each other

http://www.other-lif...drakensbane.png

Posted Image

#15 Kiwi06

Kiwi06

    Advanced Member

  • Full Member
  • PipPipPip
  • 544 posts
  • LocationMissouri, USA

Posted 26 August 2016 - 10:16 PM

This is awesome! Finally we can create roads to walk on, since we know where people are going to walk :o

#16 Learner

Learner

    God

  • Administrators
  • 2638 posts

Posted 03 September 2016 - 07:51 AM

Will ass a picture in the future once a few things stabilize, but the new mapmaker that includes the continent maker has gained a lot more data it can show:

* Lights
* Teleports
* high object density areas
* names map locations
* NPC locations
* Common paths to NPC's
* Paths from NPC's to Storage
* no harvest areas
* pk areas

Classified information it can show now includes:
* spawn locations
* invasion location & names

All this information isn't intended for general distribution, rather tools to be used in improving the map designs and preventing errors.

Everything can be on wither the map map image, or the entire continent for an overview, and I can control what all is displayed.

#17 Learner

Learner

    God

  • Administrators
  • 2638 posts

Posted 06 September 2016 - 08:19 PM

Here's what the continent looks like with some of our extra markup layers to help visualize how things can work.

Posted Image

http://www.other-lif...roth-marked.png

There are other options usable by map makers I left off (like darkening all the unwalkable spots)

#18 Learner

Learner

    God

  • Administrators
  • 2638 posts

Posted 22 September 2016 - 07:02 AM

I have now take the text only version of the map auditing and created an expanded version that only accepts a .def file, but it can be ANY .def file, not just a map.def. The tools is accessible via a password protected page or I can run it on a game server directly and gives immediate results for any .def file!

For all .def's it does basic .def parsing:
* Comment handling
* Matching [node] ... [/node] pairs
* Checks for tag: value
* Start/stop quote checking
* Reports the fir [node] found and how many nodes at the top level there are

It then checks for what sort of .def file it is, and if it knows how to handle that type of .def file it does additional sever level processing including but not limited to:
* Map specific syntax, load & audit using the .elm on file
* NPC syntax, valid placement, and optional shop definition existence
* Shop syntax, item list verification, and full definition loading
* Shop item list syntax, markup/mark down audit, and valid item names
* Spawn syntax and full server load of definition
* Boss definition syntax & loading
* Object name/look handling
* Readable book list syntax and loading

This list is continuing to be expanded and more checks are being added to these as well while will also be checking that data against the normal server settings. Expansion will include additional .def files in the future even ones that other players aren't helping with.

The webpage version saves the file first without affecting normal server files and then does the major syntax error checking and reports those errors and when possible the location in the file and aborts processing. When it passes the major syntax checks and identifies detailed checking is possible it processes that the same way the server does with verbose warnings enabled and outputted to the webpage. If the .def is badly mangled that it crashes the audit tool, you get incomplete or no response back and the file is already available for me to look into in more detail as to the cause so that the file and/or audit tool can be changed.

In the past before the map text audit tool was created I had to load the .def changes on Test and restart test, and then fix or revert the changes if the server crashed or complained about the files. While this won't guarantee no errors it will allow all devs touch a .def file a chance to greatly reduce the simple errors with no assistance needed from me.

Several people working with .def's have already been given the info for where the URL is and feedback has been coming in as I continue to improve this tool.

#19 AlddrA

AlddrA

    Advanced Member

  • Full Member
  • PipPipPip
  • 671 posts
  • LocationCanada

Posted 22 September 2016 - 07:10 AM

These tools are amazing.   Thanks so much for all the help you are giving the developers.   They are very appreciated!

#20 Learner

Learner

    God

  • Administrators
  • 2638 posts

Posted 22 September 2016 - 09:06 AM

FYI, some of the code changes being added for additional error checking actually is going into the server also, so for some of the odd non-fatal errors the server will also complain in it's error logs in the future.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users