Fire/Arson Investigations : Fire/Arson Investigations
A place to ask questions and add to probative and informative discussions associated with the various aspects of the field of fire investigation. -- FORUM RULES---BE CIVIL AND NO NAME CALLING, NO BELITTLING, NO BERATING, NO DENIGRATING others. Postings in violation of these rules can be removed or editted to remove the offending remarks at the discretion of the moderators and/or site administrator.
Re: Significant error rate
Posted by: John J. Lentini, CFEI (IP Logged)
Date: February 12, 2007 09:30PM
Jim:
When I report the "probable" cause of a fire, I do not mean to say that there is a 49% chance that something else caused the fire. I mean to say this is what I think caused it.
My testimony is, "To a reasonable degree of scientific certainty, I think this is the probable cause of the fire." If there is another hypothesis that even comes close to being probable, I have an obligation to explain that the other hypothesis exists, and that I have considered it, and then explain why my hypothesis better fits the facts on the ground.
You have conflated error rate with probability. They are quite different, and it is difficult to apply the term appropriately to what we do. When Justice Blackmun wrote about error rates in Daubert, he was referring to measurements, not determinations of cause. The case was about a statistical analysis of birth defects possibly associated with a drug. He was really talking about uncertainty or margin of error. The legal community, which has historically had problems understanding the Daubert decision, has misapplied “error rate” in many ways that Harry Blackmun could not have imagined. Look, for example at the IAAI amicus brief in the Kumho case. (A case that would have been unnecessary if lawyers or the 11th Circuit Court had understood Daubert).
As an example of what Blackmun meant when he discussed error rate, I have before me a bottle containing an acetic acid solution. I am asked to determine the concentration of the acid. I conduct a titration and determine the concentration to be 70%.
The question is not whether I am right or wrong when I say the acid concentration is 70%, the question is plus or minus what percent? That “error rate” is also known as “uncertainty.”
When pollsters report their numbers, they always talk about the “margin of error.” Usually it's 3 or 4 %.
It is not possible to determine the "error rate" for fire investigators, but if you believe it's under 5%, I have a bridge to sell you.
When I speak before groups of fire investigators, I often conduct a poll by asking everyone to stand. Then I say, "When your colleagues declare a fire to be incendiary, if you believe they are right 100% of the time, please sit down."
Of course, nobody sits. Then I ask those who believe such determinations are correct 95% of the time to sit. Nobody sits. A few folks sit when I get to 90, and a few more when I get to 85, then 80. But no matter where I am speaking, at 80%, at least half of the crowd is still on their feet.
Granted, these are folks who attend seminars and are therefore more likely to be conscientious, but on a good day, half of these folks think the error rate is 20%.
That does not mean that there have been a million bad calls. Many fires are no-brainers. An investigator is not even called to look at it because the cause is so obvious. What we need to be concerned about are the fires where the origin and/or the cause are difficult to determine.
Just looking at the fires called incendiary or suspicious, there are about 30,000 to 50,000 of those each year. And many of those are obviously arsons. Let’s say the number of fires that are hard to determine is 20,000 a year. I call that 20,000 chances to screw up. If the error rate is 10%, then 2,000 are miscalled. If things were even, there would be 1,000 arsons miscalled as accidents, and 1,000 accidents miscalled as arsons. That's only 20 per state. If Louisiana is average, ther are only two bad arson calls a month!
Unfortunately, due to the myths promulgated for years by the National Fire Academy, with a little help from NBS (and still not officially disavowed) the bias is to call fires arson when they burn "too fast" or "too hot," so I think more of the errors are accidents miscalled as arsons. The believers in myths, however, are retiring or dying, but new myths are coming along, disguised as “fire dynamics.” They will be harder to dispel.
Lately, I have seen miscalls based on a fire not living up to an investigator’s "expectations" of normal fire behavior. Even in fully involved rooms, some guys are so good a "predicting" how a fire will behave that they will say there "must have been" multiple origins, even though they cannot even show one origin, much less two or more. It’s all done with a little math, or maybe CFAST, or FDS if the client has the money or the juice.
In 1998, I wrote about a guy convicted of murder because his fire exceeded the “Standard Time-Temperature Curve.” These days, more sophisticated calculations are applied to fires, to do the same trick. If you can’t prove the cause with physical evidence, just “calculate” that the fire did not live up to your expectations and report multiple origins OR accelerants (take your pick).
John Lentini, CFI, D-ABC
Fire Investigation Consultant
Florida Keys
[www.firescientist.com]
When I report the "probable" cause of a fire, I do not mean to say that there is a 49% chance that something else caused the fire. I mean to say this is what I think caused it.
My testimony is, "To a reasonable degree of scientific certainty, I think this is the probable cause of the fire." If there is another hypothesis that even comes close to being probable, I have an obligation to explain that the other hypothesis exists, and that I have considered it, and then explain why my hypothesis better fits the facts on the ground.
You have conflated error rate with probability. They are quite different, and it is difficult to apply the term appropriately to what we do. When Justice Blackmun wrote about error rates in Daubert, he was referring to measurements, not determinations of cause. The case was about a statistical analysis of birth defects possibly associated with a drug. He was really talking about uncertainty or margin of error. The legal community, which has historically had problems understanding the Daubert decision, has misapplied “error rate” in many ways that Harry Blackmun could not have imagined. Look, for example at the IAAI amicus brief in the Kumho case. (A case that would have been unnecessary if lawyers or the 11th Circuit Court had understood Daubert).
As an example of what Blackmun meant when he discussed error rate, I have before me a bottle containing an acetic acid solution. I am asked to determine the concentration of the acid. I conduct a titration and determine the concentration to be 70%.
The question is not whether I am right or wrong when I say the acid concentration is 70%, the question is plus or minus what percent? That “error rate” is also known as “uncertainty.”
When pollsters report their numbers, they always talk about the “margin of error.” Usually it's 3 or 4 %.
It is not possible to determine the "error rate" for fire investigators, but if you believe it's under 5%, I have a bridge to sell you.
When I speak before groups of fire investigators, I often conduct a poll by asking everyone to stand. Then I say, "When your colleagues declare a fire to be incendiary, if you believe they are right 100% of the time, please sit down."
Of course, nobody sits. Then I ask those who believe such determinations are correct 95% of the time to sit. Nobody sits. A few folks sit when I get to 90, and a few more when I get to 85, then 80. But no matter where I am speaking, at 80%, at least half of the crowd is still on their feet.
Granted, these are folks who attend seminars and are therefore more likely to be conscientious, but on a good day, half of these folks think the error rate is 20%.
That does not mean that there have been a million bad calls. Many fires are no-brainers. An investigator is not even called to look at it because the cause is so obvious. What we need to be concerned about are the fires where the origin and/or the cause are difficult to determine.
Just looking at the fires called incendiary or suspicious, there are about 30,000 to 50,000 of those each year. And many of those are obviously arsons. Let’s say the number of fires that are hard to determine is 20,000 a year. I call that 20,000 chances to screw up. If the error rate is 10%, then 2,000 are miscalled. If things were even, there would be 1,000 arsons miscalled as accidents, and 1,000 accidents miscalled as arsons. That's only 20 per state. If Louisiana is average, ther are only two bad arson calls a month!
Unfortunately, due to the myths promulgated for years by the National Fire Academy, with a little help from NBS (and still not officially disavowed) the bias is to call fires arson when they burn "too fast" or "too hot," so I think more of the errors are accidents miscalled as arsons. The believers in myths, however, are retiring or dying, but new myths are coming along, disguised as “fire dynamics.” They will be harder to dispel.
Lately, I have seen miscalls based on a fire not living up to an investigator’s "expectations" of normal fire behavior. Even in fully involved rooms, some guys are so good a "predicting" how a fire will behave that they will say there "must have been" multiple origins, even though they cannot even show one origin, much less two or more. It’s all done with a little math, or maybe CFAST, or FDS if the client has the money or the juice.
In 1998, I wrote about a guy convicted of murder because his fire exceeded the “Standard Time-Temperature Curve.” These days, more sophisticated calculations are applied to fires, to do the same trick. If you can’t prove the cause with physical evidence, just “calculate” that the fire did not live up to your expectations and report multiple origins OR accelerants (take your pick).
John Lentini, CFI, D-ABC
Fire Investigation Consultant
Florida Keys
[www.firescientist.com]
Sorry, only registered users may post in this forum.