Sunday, May 21, 2017

2017 OSHG: Physics Engine Benchmarking for virtual worlds. Example for Testing WG, IMA

Physics Engine Benchmarking for virtual worlds. Example for Testing WG, IMA
Example of an engineering test to provide data for comparing the handling of physical objects in the common physics engines of virtual worlds. Presented here as an example of test plans that the IMA testing WG might develop.  

The study

  • Compared the load limits of physics engines: ODE, Bullet, and PhysX.
  • Quantifiable method to compare physics capabilities among virtual world systems.
  • Results suggested that PhysX could handle more physical objects in that test.

Educators and professionals note 

  • Such work can produce publishable results.
*******************************************************************
  • (More after the break, scroll down!)
********************************************************************** 

Metaverse events, current and upcoming

***************

Benchmark (computing)

In computing, a benchmark is the act of running a computer program, a set of programs, or other operations, in order to assess the relative performance of an object, normally by running a number of standard tests and trials against it. The term 'benchmark' is also mostly utilized for the purposes of elaborately designed benchmarking programs themselves.
-
Benchmarking is usually associated with assessing performance characteristics of computer hardware, for example, the floating point operation performance of a CPU, but there are circumstances when the technique is also applicable to software. Software benchmarks are, for example, run against compilers or database management systems.
-
Benchmarks provide a method of comparing the performance of various subsystems across different chip/system architectures. -- Wikipedia

 *****************

Benchmarking tests for evaluating virtual-world software

Benchmarking tests offer an example of test methods needed for software development in virtual worlds.   Benchmarks are regularly used to compare one software item with another as in this case.  The value of benchmarking tests is that they are highly replicable, can be done with little human intervention, and provide a measure of performance that can be recorded and compared with previous and subsequent results. 

For most such tests, the IMA test bed computer would probably be appropriate and available.

How do I know a benchmark measures what I am interested in?

You don't.  You start with a test that is plausible as a possible measure of what you want to measure.  If it produces useful discrimination among the test conditions, it remains a credible candidate.  If you have no other basis for choosing a condition to use, you might choose as indicated by the test results.  But you have no evidence that this benchmark is telling you what you need to know.  

 To demonstrate that the benchmark really tells you what you want to know, you would have to measure that and the benchmark at the same time, then demonstrate that the benchmark would correctly predict the results that you got with the actual measure you are interested in.  This kind of study is called a validation study.

 You probably would not go to the trouble of a validation study with just one benchmark test.  Instead you would develop several benchmark tests.  If they all gave the same results, the benchmarks would gain credibility and you might be ready for that validation test.

 This all sounds very elaborate.  And it is.  That is why engineering tests are taken seriously.  And that is why we need to develop standardized test procedures, archive them, and archive the results.  
*********** 

Are there other kinds of tests for the IMA Test WG to work on?

Immediately obvious are:

  • Quallity assessment suite
  • Validation of a bug fix
  • Investigation of a bug to determine conditions of occurrence.
  • Others (TBD)

Each kind of test would require different procedures.  That is why it is useful to recognize kinds of tests, where all tests of the same kind would require the same standardized procedure.

**********************************************************************************

News and Notes

The Hypergrid WIP Show

  • The Hypergrid WIP is a one hour "show & tell" of works in progress or recently completed. 
  • Everyone is invited.  Building, scripting, promotion, entertainment-- whatever you are working on.
  • Selby will be doing test video capture of presentations in voice.  These may be posted on YouTube.
-
  • Presentations are in voice and text.  
  • For text presentations, best bring the text in a notecard and paste it into chat.  
  • Voice presentations may be captured in video.  
  • Stills and videos from the show may appear in this blog and elsewhere.

Next WIP show

  • Sunday May 28, noon SLT (California) time
  • Cookie II location (fourth Sunday of the month)
  • HG address below: paste into the World Map next to Find. Click Find, TP
  • grid.kitely.com:8002:Cookie II 
  • in Kitely: paste into Nav (top) bar of Firestorm, Enter.
  • hop://grid.kitely.com:8002/Cookie II/68/369/22
  • Pandora Location: (second Sunday of the month)
  • Pandora allows presenters to run high threat OSSL functions.
  • world.narasnook.com:8900
  • Put the line above in your World Map next to Find.  Click Find. TP
  • At Narasnook, use World Map to search for Pandora

          Previous Articles from the WIP show 

          HG links-- depending on your interests 

          Communities in the virtual worlds

          Radio in the virtual worlds

          Metaverse beginner help

                                Schools in virtual worlds


                                Posted by at
                                Labels: , ,

                                6 comments:

                                1. UnknownJune 4, 2017 at 11:40 AM

                                  Hi Selby!

                                  You have forgotten one important physics engine. One that is the main one on OpenSim 0.9.X series. The one built by Ubit. That is ubODE. From my experience working with it I find it to work quite well with vanilla OpenSim's latest releases. That should be a candidate for testing as well. It supersedes the old ODE physics engine. If we are are going to test we need to test 'em all-at least the ones still used these days!

                                  Steve LaVigne

                                  ReplyDelete
                                2. UnknownJune 4, 2017 at 11:40 AM

                                  Hi Selby!

                                  You have forgotten one important physics engine. One that is the main one on OpenSim 0.9.X series. The one built by Ubit. That is ubODE. From my experience working with it I find it to work quite well with vanilla OpenSim's latest releases. That should be a candidate for testing as well. It supersedes the old ODE physics engine. If we are are going to test we need to test 'em all-at least the ones still used these days!

                                  Steve LaVigne

                                  ReplyDelete
                                  Replies
                                  1. Selby EvansJune 4, 2017 at 12:21 PM

                                    I did not forget ubODE. I was referring to a published study that did not include ubODE. Future studies would include it I expect.

                                    Delete
                                  2. Bill BlightJune 4, 2017 at 6:51 PM

                                    I see you have a link to the MOSES test you are referring to. With out this reads, without the link being prominent in the conclusion section, it kind of reads like this test was done by IMA recently. Just don't want anyone getting confused.

                                    Delete
                                  3. Selby EvansJune 4, 2017 at 7:10 PM

                                    I would not want anyone to be confused either. But there is a link to the source article in right at the beginning, when the article is mentioned. I do depend on readers to follow links, just as I used to when I used to cite publications on paper.

                                    Delete
                                  4. Bill BlightJune 4, 2017 at 7:45 PM

                                    Yes the link is there, but it does not say it is a/the study, I had to go back and look for it, because I did not see a link to a study .. We'll just have to agree to disagree on how easy it is to find the info ..

                                    "Physics Engine Benchmarking in Three-Dimensional Virtual WorldSimulation" Does not say it is a study to me ..

                                    Delete

                                Note: Only a member of this blog may post a comment.

                                Subscribe to: Post Comments (Atom)