Seek the Truth

Validation & Calibration Tool Test Results for Perlustro ILookIX v.2.2.7 +
November 16, 2012 © Perlustro LP

In 2009 the National Institute of Standards and Technology (NIST) announced a standards program for digital tool testing for specific topical forensics issues that it defined as being of governing importance to both Law Enforcement and commercial interests in forensics examinations and in their outcomes : http://www.cftt.nist.gov/DFR-req-1.1-pd-01.pdfThis standards-based approach provided a quantum leap in both the valuation of standards in digital forensics, and in providing admissible solutions for legal issues that may arise from expert witness requirements. In sum, this was a watershed event of high public value and it exists beyond reproach as an authoritative corpus of judicially admissible test data along with judicially admissible test result data.

The US Federal Rules of Criminal Procedure provide that NIST published data is admissible on its face. The promulgation of the CFTT test paradigm now allows a unique, rare, and first time opportunity for digital investigators, under various federal statutes such as 28 USCS § 1732, to obtain an array of test data which can be actively used for tool validation by users themselves. This method does not diminish the value of educational processes or “independent studies”, but it is complementary to those optional processes and can stand alone to meet certain evidence requirements. This is accomplished by provisioning a base standard test on which to find clarity for any of the test processes defined in the NIST test plans.  By having both the base test data, AND the NIST expected minimal results, a sufficiently experienced user can objectively test any tool or software for comparative and RELIABLE results of the digital corpus under examination and offer into evidence those findings relevant to a particular case. This is carried out through a showing of repeatable, objective and reliable results obtained from the Test data sets.

Where a tool test reveals no anomalies under the NIST standards, the tool has then been tested according to NIST published guidelines. No longer is it required that “independent” groups perform the testing for the specific areas under test.  As such, any capable operator can perform the test task as the fact witness, exactly like a test strip establishes a baseline for a glucose meter.  There is NO evidentiary Federal Law requirement that any claimed digital expert witness must find trustworthiness beyond the scope of this test process. On the contrary, it is only required that the relied upon findings, if externally generated, be published and have historical content that can be corroborated by peer groups or others in the “field”. In this context, NIST has become the objective third party as the inventor of the tests and by default now takes on the role as the “standard”. NIST are presumed in Federal evidence to be objective and clearly reasonable and subject to peer review. Here, for the first time in digital forensics, the test conditions themselves and their expected outcome are BOTH provided by NIST. Perlustro now voluntarily elects to comport completely with the NIST test result set through releasing here the “Expected findings” in total for each test set, in easy to execute and validate form. Details of the findings as to any particular set can then be directed to those particular areas per the NIST documentation itself. For DFR tests, the following URL is self explanatory http://www.cfreds.nist.gov/dfr-images/setup-july-10-2012.pdf {the URL will possibly change since NIST may make adjustments to the documents publication name}

Each week until the test scope is completed, Perlustro will publish its own Tool findings of a different CFTT group for the singular purpose of helping customers of Perlustro to quantify for admissible evidence purposes, the tools that Perlustro produces. Each version of any Perlustro tool, including IXimager in some Perlustro tests soon to be released, can now be corroborated for evidentiary purposes–at any time–by a short regimen of repeating the image tests as provided herein. Example: The first test phase concerning 4 filesystems, containing DFR tests (Deleted files), will be published in this order with others added by Perlustro for customer use:

(A) Extended Linux 2, 3, 4.  (additional linux systems will be provided by Perlustro beyond the scope of the DFR tests.  [TEST 1, below]
(B) Fat file systems : Fat 12, Fat 16, Fat 32, Extended Fat
(C) NTFS systems
(D) HFS systems (additional HFS HFS+ HFSX including DMG variant systems will be provide by Perlustro beyond the scope of the NIST tests.)
(E) Unix file systems – All test data will be provided by Perlustro unless provided by NIST
(F) Novell File systems – All test data will be provided by Perlustro unless provided by NIST
(G) UDF File systems – All test data will be provided by Perlustro unless provided by NIST  {NIST test data where it exists, will always take precedence over Perlustro Test data for the purposes of self validation}

The published criteria of the DFR test analysis functions are clearly articulated by NIST in the URL referenced documents, and as such, relevance to the current FRCP concerning expert witness testimony, is germane to the issues noted, and to very specific parts of the criminal procedure requirements.

The NIST criteria for the CFTT line of tests are critically important to the reliability factors required of evidence submissions by Experts, and include the following :

1.   A capability is required to ensure that forensic software tools consistently produce accurate and objective results.
2.  The objective of the Computer Forensics Tool Testing program is to provide measurable assurance to practitioners, researchers, and other applicable users that the tools used in computer forensics investigations provide accurate results.
3.  Each assertion generates one or more test cases consisting of a test protocol and the expected test results.
4.  The approach for testing computer forensic tools is based on well-recognized international methodologies for conformance testing and quality testing.
5.  Identify active files, deleted files and attempt to reconstruct or recover deleted files.
6.  [The] individuals using these tools [must] adhere to forensic principles, and have control over the environment in which the tools are used.
7.  Other types of latent data recovery such as file carving tools are not part of this specification {and are excluded as solutions in whole or part by Perlustro}

More than just tool validation itself,  NIST’s own findings allow authentication of a tool’s findings for any case, criminal or civil, and for specific file systems that comprise over 92% of the existing System market. Such user validation testing, for the first time, now allows a tool user to provide his or her own NIST certified data responses to tests which substantiate and corroborate, and fully in some cases,  F.R.C.P 702 {as amended 2011}. http://www.law.cornell.edu/rules/fre/rule_702) and as to admissibility, under Federal Rule of Evidence 803 (18).

The methods used to quantify digital forensics findings for judicial purposes; rely on the presumption the tool user is qualified as an expert witness, or is attempting to be so qualified. The methods also provide that sections (C) and (D) of 702, relating to reliable principles and methods and their application to the facts of a case, can now be addressed by the administration of “self testing and reporting” using these same data tests to qualify evidence findings.  The NIST test data therefore, fully addresses the required Daubert issues for Federal Court proceedings, as well as similar proceedings under like statues in most States and in many foreign countries.

Perlustro’s drive for The Truth, believes this methodology should be expanded to all digital forensics areas which we manage.  Perlustro is therefore committed to furnishing test base standards and published findings for the use of its tools in a methodology which measures the accuracy and efficacy of its tools in comparison to the expected results as so defined by NIST.  Perlustro knows of no other test method which can achieve the same evidentiary qualifications.

The submission of all test data sets by Perlustro are in the form of ILookIX image files, native to the Perlustro ILookIX tools.  The files (a single file form of the NIST data sets by file system) can be downloaded free from Perlustro using enclosed links.  HOWEVER, for authentication purposes, it is mandatory that the “source” data sets used by any KIX user, should first be the repository of NIST itself, and only in the second KIX layer corroboration tests, should those sets be used to verify and validate the Perlustro sets.  http://www.cfreds.nist.gov/dfr-test-images.html .  Once this corroboration has occurred, the Perlustro image sets can be used from that point on.  The Perlustro version is offered only for the purpose of speed to result and convenience to the user.  It is reliant 100% on the NIST set as the bedrock test basis data.

Errors and omissions or anomalies in the ILookIX tool should be noted in comparison to the findings produced by NIST and documented by the customer tester since future release versions of the Perlustro tools will change over time to adapt to other forensics venues. The Perlustro tool chain, while striving to continually improve, may through this testing identify a failure point as easily as an improved result set. One must be vigilant to either possibility.

To encourage this NIST methodology of self testing and validation, the following table lists the expected solutions you should achieve following an examination of the test data using the ILookIX feature “XFR” (Xtreme file recovery) for the DFR testing. To follow the documentation of NIST, no other processes, either automatic or inclusive of file salvage or magic string identification of any artifacts, can be used to achieve the designed independent feature result set or solutions for the presented problems of DFR. For the DFR tests, in ILookIX, this is accomplished by selecting MINIMAL and XFR options in the autoload menu sections known to all users, but NO other options. Map the image set or the original images of NIST in both conditions, with and without XFR set to true, and document the file counts, byte counts recovered and maintain that document for permanent future reference. It may also be used as an evidentiary log for any Daubert hearing and would be admissible through the person who did the testing. For this reason the person executing the tests should authenticate the tests results to provide a chain of custody to the findings.

Test 1

Published by Perlustro, November 16, 2012, is an EXT Linux 2,3,4 DFR KIX test image © Perlustro :

The ILookIX DFR-EXT234 KIX image base file (11 megs) is located here.  The file itself is a concatenated single ILookIX created compressed format file containing all of the Linux test images for the Deleted File Result test regimen (DFR) Ext 234 series. This file, once verified by using the NIST original rawbit files, can then be used in the future to validate the Perlustro Tools themselves including IXimager IV. The expected run time of the test is less than 5 minutes for any system in test within KIX, and in the single image final summary test, less than 20 minutes for all 4 systems in one 40 gig image. Prior to any judicial proceeding, it is suggested that it be run against the then current version of ILooKIX to validate previous findings of your investigation. The user generated KIX result report can also be contemporaneously used as an authentication report of your case findings if needed.

KIX Test Setup: Load the image with KIX, first with only [Minimal + Probe], then add the image a second time with [Minimal + Probe + XFR].  The expected summary results are in this table :

 

ERRATA & NOTES: Any failure to achieve these results is a notable anomaly in the Tool in test and it should be brought to Perlustro’s attention as soon as possible IF the differential count is less than stated above for the DFR test.

** KIX Image Media Hash Value Result File [SHA 3- 512]

Quick review comparison screenshots are available in this PDF for both the file system Explorer and the Filelist view of the “list all” commands :   PDF

Test 2

 Published by Perlustro, November 23, 2012, is a Fat 12, 16, 32 ExFat  DFR KIX test image © Perlustro :

The ILookIX DFR-FAT KIX image base file (7 megs) is located here.  The file itself is a concatenated single ILookIX created compressed format file containing all of the FAT test images for the Deleted File Result test regimen (DFR) FAT series. This file, once verified by using the NIST original rawbit files, can then be used in the future to validate the Perlustro Tools themselves including IXimager IV. The expected run time of the test is less than 20 minutes for any system in test within KIX, and in the single image final summary test, less than 30 minutes for all 4 systems in one 40 gig image. Prior to any judicial proceeding, it is suggested that it be run against the then current version of ILookIX to validate previous findings of your investigation. The user generated KIX result report can also be contemporaneously used as an authentication report of your case findings if needed.

KIX Test Setup:  Load the image with KIX, first with only [Minimal + Probe], then add the image a second time with [Minimal + Probe + XFR].  The expected summary results are in this table :

 

ERRATA & NOTES: Any failure to achieve these results is a notable anomaly in the Tool in test and it should be brought to Perlustro’s attention as soon as possible IF the differential count is less than stated above for the DFR test.

**  KIX Image Media Hash Value Result File [SHA 3- 512]

Quick review comparison screenshots are available in this PDF for both the file system Explorer and the Filelist view of the “list all” commands :   PDF

 

Test 3

 

 Published by Perlustro, November 30, 2012, is a NTFS DFR KIX test image © Perlustro :

The ILookIX DFR-NTFS KIX image base file (7 megs) is located here.  The file itself is a concatenated single ILookIX created compressed format file containing all of the NTFS test images for the Deleted File Result test regimen (DFR) NTFS series. This file, once verified by using the NIST original rawbit files, can then be used in the future to validate the Perlustro Tools themselves including IXimager IV. The expected run time of the test is less than 20 minutes for any system in test within KIX, and in the single image final summary test, less than 30 minutes for all 4 systems in one 40 gig image. Prior to any judicial proceeding, it is suggested that it be run against the then current version of ILookIX to validate previous findings of your investigation. The user generated KIX result report can also be contemporaneously used as an authentication report of your case findings if needed.

KIX Test Setup:  Load the image with KIX, first with only [Minimal + Probe], then add the image a second time with [Minimal + Probe + XFR].  The expected summary results are in this table :

ERRATA & NOTES: Any failure to achieve these results is a notable anomaly in the Tool in test and it should be brought to Perlustro’s attention as soon as possible IF the differential count is less than stated above for the DFR test.

**  KIX Image Media Hash Value Result File [SHA 3- 512]

Quick review comparison screenshots are available in this PDF for both the file system Explorer and the Filelist view of the “list all” commands:   PDF