- Details
- Published on Thursday, 26 May 2016 22:58
VSP STATISTICS 101*
Every targeted audit by VSP has three purposes:
- Determine if you were or were not violating VSP rules in some manner.
- If you were violating their rules, should your network provider agreement be terminated?
- If you were violating their rules, how much have you been overpaid by VSP?
In this article I will focus on the third purpose above and will attempt to explain the statistical process VSP uses to determine how much restitution you owe back to them if you do not pass an audit. I will then explain exactly why I believe the VSP statistical process is mathematically and statistically incorrect.
I believe VSP's audit results can (and should) be legally challenged, and that ALL VSP audits that have determined money is owed back are incorrect and could be thrown out by a legal court challenge.
A. THE VSP PROCESS FOR DETERMINING RESTITUTION DUE
If you are audited by VSP, they typically follow this general process to determine how much you must repay them:
- The auditor will copy and audit 40 records randomly selected from your last three years. All records will be of the type they are auditing (e.g., if they are auditing MNCL, they will all be MNCL patients).
-
From those 40 records the auditor determine your non-compliance rate and your "error rate", or what they now call your "dollars at risk." The auditor does this by stating first what you were paid, then what the auditor thinks you should have been paid based on the audit, then finally, what the auditor thinks you were overpaid. The total paid minus what the auditor says you should have been paid equals the amount you were overpaid. These will be columns on a spreadsheet. The total overpaid divided by the total paid, times 100, is your error rate, or "dollars at risk." Here is an example for three patients:
Patient Total Amount Paid Correct Amount Paid Amount Overpaid Smith $150.00 $90.00 $60.00 Jones $275.00 $0.00 $275.00 Johnson $350.00 $350.00 $0.00 TOTALS $775.00 $440.00 $335.00
In this example, Johnson passed the audit. But Smith and Jones did not. The error rate/dollars at risk is determined to be $335/$775 * 100, or 43.2%.
[NOTE 1: The error rate/dollars at risk is not the non-compliance rate. Non-compliance rate in this example is 67% because 2/3 of the audited records were found to be non-compliant. But, because Johnson was a larger dollar patient, the dollars at risk is lower than the non-compliance rate. This, of course, can be reversed, especially in MNCL cases, where the non-compliance rate is actually considerably lower than the calculated error rate.]
[NOTE 2: While the non-compliance rate in this sample was 67%, one cannot necessarily conclude that the non-compliance rate for the entire universe of claims is 67%. In other words, you cannot necessarily extrapolate the non-compliance rate out to the universe of claims and maintain the 6.1% margin of error. To extrapolate out to all claims, as will be explained below, you must ensure that you have a sufficiently large sample so that your margin of error remains sufficiently narrow to provide useful information. This requires proper application of the "binomial equation."] - The auditor determines the total amount of money you've been paid for the last three years for the patient type/line of business that they audited (e.g., MNCL, elective CL, frame only claims, etc.).
-
Finally, the auditor multiplies your error/dollars at risk rate by the total money and that's the amount VSP will claim as restitution.
If you've been paid $265,000 for the past three years, your restitution is $265,000 * 0.432 = $114,480.
Based on documents available in public records, VSP asserts that, statistically, they base their process on a 95% confidence level, with a margin of error of +/- 6.1%.
The goal of the sampling process (any sampling process) is to draw reliable conclusions about the entire population of claims (the "universe of claims") from the sample. The question raised is, is the process above a statistically/mathematically valid statistcal method to calculate restitution owed?
I believe it is not, because the methods to determine restitution are not correct.
The determination of "error rate" or "dollars at risk" lacks relationship to the representation of statistical validity. The sample size is statistically too small to determine anything more than if the doctor is or is not compliant. It is too small to determine the non-compliance rate of the entire population of claims within a reasonable margin of error. And it is entirely unknown what the relationship is between compliance rate and dollars at risk, the determination of which is the ultimate goal.
The first step in an audit process is to determine how many randomly selected records to examine in order to have a reliable result that can be extrapolated out to the universe of claims. It is essential that the sample fairly represent the universe of claims. That requires the sample to be random, without any bias being introduced, and that it be large enough. For purposes of this discussion the focus is on the size of the sample. No conclusions are reached with respect to whether or not the sample is truly random and without bias.
1. The Binomial Equation for Determining Sample Size
There is an equation (the binomial equation) for determining the sample size necessary where there are two possible choices (a binomial), such as compliant or non-compliant. The equation has two variables, "p" and "e". There is one constant whose value is based on the confidence level you want. For 95%, that constant is 1.96. Thus, the formula for determining how many records are needed in the sample (n) is:
Where:
"p" is the "expected error rate." (The best advance estimate of what VSP thinks the non-compliant rate will be.)
"e" is the "precision", aka the margin of error, you want to have.
2. Applying the Binomial Equation to Determine the Sample Size
VSP's values for p and for e result in a sample size of about 40 records.
3. VSP's "p" Value Only Allows VSP To Determine If You Are or Are Not Compliant
The first problem is "p." The doctor has been selected to be audited, not randomly, but due to a belief by VSP that the doctor may be non-compliant. The number of records to be audited must take into account that this doctor was selected for an audit due to a belief he/she is not compliant, and "p" must account for this.
Because at the outset the error (non-compliance) rate is unknown, it is common in the statistics industry to use a value of 50% (0.5) for p. This results in the largest sample size and ensures a large enough sample for results that can be extrapolated to the universe of claims while maintaining an appropriate margin of error. If a value of "p" less than 0.5 is used, once doing the initial audit, the results of that sample can be used to adjust "p" and determine how many additional records are needed in the sample to proceed to step 2 of the process.
If we give VSP the benefit of the doubt and assume the sample size is statistically large enough, the ONLY thing VSP is actually testing in this first step, whether this doctor is compliant or is not compliant. But, what does this tell us about the non-compliance rate or dollars at risk in the larger universe of claims?
4. The "p" Value and Resulting Sample Size Is Too Small
Based on the audit of 40 records, only two things can be determined with statistical validity:
(1) whether or not the doctor is compliant and, (2) the rate of non-compliance among the audited records.
But, what about the non-compliance rate in the universe of claims?
a. Using the Corrected Value of "p" Results in a Margin of Error that Violates Due Process
First, if the non-compliance rate for the audited records audited is extrapolated out to the universe of claims, the margin of error increases to about +/-15% if a corrected or actual value of "p" is considered.
This is determined by using basic algebra to solve for "e" in the binomial equation. By way of example, using 40 for the sample size (n), and setting "p" equal to 35%, we find that "e", the margin of error, is +/-15%. This is a very large margin of error. Indeed, in Bell v. Farmers Ins. Exchange (2004) 115 Cal.App.4th 715, 756-757, as discussed by the California Supreme Court in Duran v. U.S. National Bank (2014) 59 Cal.4th 1, 46, the California courts rejected a damages calculation where the margin of error was +/-16%, stating that the "margin of error was so large that the resulting damages award violated due process." Id.
Having determined from the initial audit that the doctor is non-compliant with VSP's rules, VSP has actually determined the correct value of "p" for determining the compliance rate! But VSP does not modify the size of their sample before calculating non-compliance rates. That often results in a margin of error that is very large due to the small sample.
Again, if you do not know the actual non-compliance rate, it is common to use 50%. This results in a sample size of 249 records and at a 95% confidence level with a +/- 6.1% margin of error. If you do know something about the non-compliance rate, as is the case after the initial audit, one can adjust "p" to that known value for the purposes of ensuring a large enough audit sample for determining non-compliance rates. Assuming p = 50%, if the margin of error is +/- 4%, FAR more records would be needed to achieve a 95% confidence that the non-compliance rate is correct.
However, with a non-compliance RATE margin of error of +/-15% (it varies a bit, +/- 1%, depending on the actual or expected error rate) it means, if they determine you were non-compliant in 30% of the 40 records in the sample, they can be 95% sure you are non-compliant somewhere between 15% and 45% of the time. This is a huge range, and really doesn't provide much useful information. Several Courts have found that margins of error of this magnitude constitute a denial of due process.
b. The Goal of the Audit is to Determine Restitution, Not the Non-compliance Rate
Second, the goal and purpose of the audit is to determine restitution dollar amount owed. So, no matter what the non-compliance rate may be, or its margin of error, the real issue is the amount of money owed back due to the non-compliance.
Let's assume as an example the non-compliance rate found in a hypothetical audit. A total of 40 records are audited and non-compliance is found in 24 of the 40 records, a 60% rate of non-compliance. $3,153 in over-paid in those 24 records, out of a total of $5,219 paid. That was a "dollars at risk" rate of also 60%. This resulted in a restitution demand of $820,000. Now, here is the problem. The actual non-compliance rate was not 60%, it was 60% +/- 15%. Thus, the number of non-compliant records in the population as a whole was just as likely to have been 45%, or 18 records, as it was 60% (24 records). Both are 95% likely to be correct. This dramatically changes the "dollars at risk" because there is no way to know which of the 6 "extra" records would not be in the larger population. And, because the dollars at risk are not yes/no values, and are not a normally distributed (e.g., are not a "Bell Curve"), we know nothing about just how much having only 18 records (or 30 records for that matter) being non-compliant would change the restitution amount. If we assume the dollars are, in fact, normally distributed, then we may be able to say that instead of 60% dollars at risk, it would drop to as little as 45%. That would reduce the restitution by more than $200,000! But, scientifically, we cannot even say that, because a single outlier could dramatically change the dollars at risk values and resulting restitution.
5. Determining Restitution Due
The goal of the statistical process, its ultimate objective, is to determine RESTITUTION, not non-compliance or non-compliance rates. Here, there are no statistics applied to the non-binomial, non-normally distributed, dollars paid or dollars overpaid amounts. The binomial equation does not apply and cannot be used for calculating dollars owed because those are not a binomial (yes or no) values or answers, nor are they necessarily normally distributed (that is, the distribution of dollars paid, or amounts overpaid, may be skewed).
We simply cannot say that, for instance, that the restitution of $114,480 calculated in the sample above is 95% likely to be a correct representation of the universe of claims (within the margin of error) because nothing is known about the dollar data. In particular, the upper and lower limits, the mean, and the standard deviation are all unknown. For instance, assume there are 1000 claims. There may be only one claim with a value above $100, and that one might be $1200. The 40-record sample may have selected that $1200 claim. This will substantially skew the extrapolation and result in having to repay FAR more than you actually owe back. A proper determination of the dollars owed back requires applying statistical methods that normalize the data, then considers the standard of deviation of the non-binomial data.
VSP uses the binomial compliance vs. non-compliance to calculate non-compliance RATES, but then uses that to extrapolate to non-binomial non-normally distributed restitution. There are no margins of error or confidence interval values for the restitution calculation. When testing a data set with a non-normally distribution like reimbursement, the binomial equation isn't accurate.
C. CONCLUSION
Because doctors are not randomly selected for audit, VSP actually needs to audit more records to determine, at the 95% confidence interval and 4% margin of error, if the doctor is or is not compliant. We have no idea how many records are needed to determine how much a non-compliant doctor owes back in restitution, as no valid statistical methods are used to make that calculation.
So, when VSP says you owe restitution, they cannot say what the margin of error is on that determination, or what level of confidence they have that this is the correct amount. This is not to say if you've billed VSP incorrectly you don't owe them money back. You may, and it could be a lot more or less than VSP's conclusion. But determining the amount should to be done properly. If VSP is going to rely on sampling, the statistical methods should be correct to avoid the risk of substantial error.
* The foregoing is my opinion based on my analysis of the VSP process. Note that I am not an expert in statistics, and this only represents my beliefs based on information in public files and what has been explained to me by statistics experts. VSP contends it's methods, developed by a statistics expert, are statistically valid.