Update README for regression test suite.

test/regression/README

@@ -3,83 +3,81 @@ Assimp Regression Test Suite
 
 1) How does it work?
 ---------------------------------------------------------------------------------
-run.py checks all model in the <root>/test/models folder and compares the result
+run.py checks all model in the <root>/test/models* folders and compares the result
-against a regression database provided by us (db.zip). If the test passes
+against a regression database provided with assimp (db.zip). A few failures
-successfully, Assimp definitely WORKS perfectly on your system. A few failures
+are totally fine (see sections 7+). You need to worry if a huge
-are totally fine as well (see sections 7+). You need to worry if a huge
+majority of all files in a particular format (or post-processing configuration)
-majority of all files in a particular format or post-processing configuration 
+fails as this might be a sign of a recent regression in assimp's codebase or
-fails - this might be a sign of a recent regression in assimp's codebase. 
+gross incompatibility with your system or compiler. 
 
 2) What do I need?
 ---------------------------------------------------------------------------------
- - you need Python installed - 3.x !!
+ - You need Python installed (2.7+, 3.x). On Windows, run the scripts using "py".
- - you need to build tools/assimp_cmd as described in the INSTALL file (
+ - You need to build the assimp command line tool (ASSIMP_BUILD_ASSIMP_TOOLS
-	make && make install on unixes,release-dll target with msvc).
+   CMake build flag). Both run.py and gen_db.py take the full path to the binary
+   as first command line parameter.
 
 3) How to add more test files?
 ---------------------------------------------------------------------------------
 Use the following procedure:
- - verify the correctness of your assimp build - run the regression suite.
+ - Verify the correctness of your assimp build - run the regression suite.
-   DO NOT continue if one or more tests fail.
+   DO NOT continue if more tests fail than usual.
- - add your additional test files to <root>/test/models/<fileformat>, where
+ - Add your additional test files to <root>/test/models/<fileformat>, where
-   <fileformat> is the file type (typically the file extension)
+   <fileformat> is the file type (typically the file extension).
- - rebuild the regression database using gen_db.py
+ - If you test file does not meet the BSD license requirements, add it to
- - run the regression suite again - all tests should pass, including
+   <root>/test/models-nonbsd/<fileformat> so people know to be careful with it.
-   those for the new files.
+ - Rebuild the regression database:
+   "gen_db.py <binary> -ixyz" where .xyz is the file extension of the new file.
+ - Run the regression suite again. There should be no new failures and the new
+   file should not be among the failures.
+ - Include the db.zip file with your Pull Request. Travis CI enforces a passing
+   regression suite (with offenders whitelisted as a last resort).
 
- - contributors: commit the db.zip plus your additional test files to
+4) I made a change/fix/patch to a loader, how to update the database?
-   the SVN repository. 
-
-4) I made a change/fix/.. to a loader, how to update the database?
 ---------------------------------------------------------------------------------
- - rebuild the regression database using gen_db.py
+ - Rebuild the regression database using "gen_db.py <binary> -ixyz"
- - run the suite - all tests should pass now. If they do not, don't continue
+   where .xyz is the file extension for which the loader was patched.
+ - Run the regression suite again. There should be no new failures and the new
+   file should not be among the failures.
+ - Include the db.zip file with your Pull Request. Travis CI enforces a passing
+   regression suite (with offenders whitelisted as a last resort).
 
- - contributors: commit the db.zip to the SVN repository 
+5) How to add my whole model repository to the database?
-
-5) How to add my whole model repository to the suite?
 ---------------------------------------------------------------------------------
 
 Edit the reg_settings.py file and add the path to your repository to
-<<model_directories>>. Then, rebuild the suite.
+<<model_directories>>. Then, rebuild the database.
 
-6) So what is actually verified?
+6) So what is actually tested?
 ---------------------------------------------------------------------------------
-The regression database includes mini dumps of the aiScene data structure -
+The regression database includes mini dumps of the aiScene data structure, i.e.
 the scene hierarchy plus the sizes of all data arrays MUST match. Floating-point
-data buffers, such as vertex positions, are handled less strictly: min,max and
+data buffers, such as vertex positions are handled less strictly: min, max and
-average values are stored with lower precision. This takes hardware- or 
+average values are stored with low precision. This takes hardware- or 
 compiler-specific differences in floating-point computations into account. 
 Generally, almost all significant regressions will be detected while the
 number of false positives is relatively low.
 
 7) The test suite fails, what do do?
 ---------------------------------------------------------------------------------
-Get back to ../results and check out regression_suite_failures.txt 
+Get back to <root>/test/results and look at regression_suite_failures.txt.
-It contains a list of all files which failed the test ... they're copied to
+It contains a list of all files which failed the test. Failing dumps are copied to
-../results/tmp. Both an EXPECTED and an ACTUAL file is produced per test.
+root>/test/results/tmp. Both an EXPECTED and an ACTUAL file is produced per test.
-The output of `assimp cmpdump` is written to regressions_suite_output.txt. 
+The output of "assimp cmpdump" is written to regressions_suite_output.txt. Grep
-To quickly find all all reports pertaining to tests which failed, I'd
+for the file name in question and locate the log for the failed comparison. It
-recommend grepping for 'but' because its a constituent of all error messages
+contains a full trace of which scene elements have been compared before, which
-produced by assimp_cmd :) Error reports contain detailed information
+makes it reasonably easy to locate the offending field.
-regarding the point of failure inside the data structure, the values of
-the two corresponding fields that were found to be different from EXPECTED
-and ACTUAL, respectively, ... this should offer you enough information to start
-debugging.
 
 8) fp:fast vs fp:precise fails the test suite (same for gcc equivalents)
 ---------------------------------------------------------------------------------
 As mentioned above, floating-point inaccuracies between differently optimized
 builds are not considered regressions and all float comparisons done by the test
-suite involve an epsilon. Changing floating-point optimizations can, however,
+suite involve an epsilon to accomodate. However compiler settings that allow
- lead to *real* changes in the output data structure, such as different number 
+compilers to perform non-IEEE754 compliant optimizations can cause arbitrary
-of vertices or faces, ... this violates one of our primary targets, that is
+failures in the test suite. Even if the compiler is configured to be IEE754
-produce reliable and portable output. We're working hard on removing these
+comformant, there is lots of code in assimp that leaves the compiler a choice
-issues, but at the moment you have to live with few of them.
+and different compilers make different choices (for example the precision of
-
+float intermediaries is implementation-specified).
-Currently, the regression database is build on Windows using MSVC8 with 
-fp:precise. This leads to a small number of failures with fp:fast and
-virtally every build with gcc. Be patient, it will be fixed.