A software engineer read thru some of the github fix notes and code on the imperial model. This isnt Fergusons actual code, because that code was written in a way that it would be publically blasted as outdated nonsense by anyone seeing it. It would literally be like an astrophycist trying for a nobel peace prize by submitting a model of the solar system using styrofoam balls. It was really 4th grade stuff he did. So they hired a group of MS developers to try to clean it up and make the software appear as though it had been written sometime in the last decade, instead of using programming techniques that were abanonded in the 90's.

Being a software engineer for 35 years, I checked out what she was saying in her review of the imperial model, and she is correct.

One of the main issues with Fergusons code is that it does not generate the same result for the same "conditions". The code is so buggy that running the exact simulation multiple times will yield different results, sometimes wildly different, each time. This is a huge issue in software development, because it means you can not determine if you have a bug, or if you have injected a bug with a code change. When software developers make a change, they perform many different tests. One type of test is known as "regression testing." Regression testing tests different logical units of the code before and after the change, to ensure you get the same results. (What you intended to change did not impact other parts of the code you didnt intend to change) The microsoft developers who redid his code was so flummoxed by this that they had no idea if the changes they were making helped or hurt.

In the end, the team defended this model by saying that they simply ran the sim a bunch of times and averaged the model. So, they took a bunch of garbage outputs, and came up with an average of crap.

The reviewer also makes a very good point that universities should not be the place where these types of models are funded, because there is no "manager" who oversees the project to ensure it is sound and vetted well enough for public policy and decision making. Ferguson was the lead on this, and it appears that his model was coded using programming techniques he learned 40 years ago as a student. That is ridiculous, and there was noone over him who looked in on the work and told him it was crap like in the real world.

(05-06-2020 03:57 PM)UofMstateU Wrote:  The reviewer also makes a very good point that universities should not be the place where these types of models are funded, because there is no "manager" who oversees the project to ensure it is sound and vetted well enough for public policy and decision making. Ferguson was the lead on this, and it appears that his model was coded using programming techniques he learned 40 years ago as a student. That is ridiculous, and there was noone over him who looked in on the work and told him it was crap like in the real world.

I don't think that's the right conclusion.

In my field, the "experts" working at the Fed, the SEC, and think tanks are generally PhD graduates who weren't smart enough or well-rounded enough to get jobs in academia. So they become career bureaucrats. Occasionally they'll do some good policy work, but their "predictions" always support the bureaucratic goals of their boss.

And their predictions NEVER are allowed to criticize their own agency. I knew one guy who got fired from his government job because his co-authors used his data to prove that one policy enacted by his agency had some unintended bad consequences. (his agency actually tried to pull his co-authors' right to publish using the data, and they would have succeeded except their university's lawyers actually showed some cajones and threatened to sue the government agency).

I don't think running a simulation many times and "averaging" the results is necessarily a bad approach.

As a software engineer you might not like it, but as a statistician I don't see that as bad. There are other things that should be looked at including the distribution of the results, outliers, and nature of the randomization.

Having actually written commercial simulators, I don't see a problem with it as long as it was designed to be used that way.

(05-06-2020 06:11 PM)Claw Wrote:  I don't think running a simulation many times and "averaging" the results is necessarily a bad approach.

As a software engineer you might not like it, but as a statistician I don't see that as bad. There are other things that should be looked at including the distribution of the results, outliers, and nature of the randomization.

Having actually written commercial simulators, I don't see a problem with it as long as it was designed to be used that way.

Disagree, it would be more analogous to a math equation. The same simulation should result in the same outcome every time. If you get different results from the math equations it means you screwed up. Averaging it is nearly just averaging crap. It doesn’t do any good especially if this model is going to influence public policy

(05-06-2020 06:11 PM)Claw Wrote:  I don't think running a simulation many times and "averaging" the results is necessarily a bad approach.

As a software engineer you might not like it, but as a statistician I don't see that as bad. There are other things that should be looked at including the distribution of the results, outliers, and nature of the randomization.

Having actually written commercial simulators, I don't see a problem with it as long as it was designed to be used that way.

Its biological. That means it doesn't give the same answer every time. There is some randomness in transmission. Expecting it to get the same answer every time simply means that person didn't understand what they were looking at.

Now if you were using averages and got wildly different answers with each set of averages, well that just means the model is useless.

(05-06-2020 06:11 PM)Claw Wrote:  I don't think running a simulation many times and "averaging" the results is necessarily a bad approach.

Its not bad, if the sim itself is good. But Fergusons simulation is bad.

The best way to describe the flaw programmtically I guess is this:

Say you want want to simulate flipping a coin in a computer program. You have 10 numbers, 1- 10. Half of the numbers are assigned heads, and the other half tails, which represents the 50/50 odds of getting heads or tails on the flip of the coin.
We then run the program, and have it randomly pick a number 1 thru 10. We do that 10,000 times, and record which value, heads or tails, was selected. Over the 10,000 iterations, we would have seen the computer randomly select "heads" about 50% of the time. Thats how running a simulation over and over and averaging should work.

But Fergusons code doesnt do that. Instead of assigning 5 numbers heads and 5 numbers tails, it occassionally decides that heads get 7 numbers, or all 10 numbers, and the reasoning behind it doing that is not known. You can run the sim 10,000 times, but heads is going to be selected at a higher rate than the 50% expected, by a factor of many standard deviations.

So its like the part of Fergusons code that should be stable isnt. Its like if we ask the program to tell us how many of the 10 numbers are assigned heads, it should ALWAYS report back 5, to ensure the odds of getting heads is 50%. But Ferguson's routine doesnt, it deviates from a stable reproducable number. Even running his code on a different computer was changing the results. So his model had biases, completely unrelated to the data of the model, that affects the outcome.

(05-06-2020 07:33 PM)UofMstateU Wrote:  

(05-06-2020 06:11 PM)Claw Wrote:  I don't think running a simulation many times and "averaging" the results is necessarily a bad approach.

Its not bad, if the sim itself is good. But Fergusons simulation is bad.

The best way to describe the flaw programmtically I guess is this:

Say you want want to simulate flipping a coin in a computer program. You have 10 numbers, 1- 10. Half of the numbers are assigned heads, and the other half tails, which represents the 50/50 odds of getting heads or tails on the flip of the coin.
We then run the program, and have it randomly pick a number 1 thru 10. We do that 10,000 times, and record which value, heads or tails, was selected. Over the 10,000 iterations, we would have seen the computer randomly select "heads" about 50% of the time. Thats how running a simulation over and over and averaging should work.

But Fergusons code doesnt do that. Instead of assigning 5 numbers heads and 5 numbers tails, it occassionally decides that heads get 7 numbers, or all 10 numbers, and the reasoning behind it doing that is not known. You can run the sim 10,000 times, but heads is going to be selected at a higher rate than the 50% expected, by a factor of many standard deviations.

So its like the part of Fergusons code that should be stable isnt. Its like if we ask the program to tell us how many of the 10 numbers are assigned heads, it should ALWAYS report back 5, to ensure the odds of getting heads is 50%. But Ferguson's routine doesnt, it deviates from a stable reproducable number. Even running his code on a different computer was changing the results. So his model had biases, completely unrelated to the data of the model, that affects the outcome.

there's a reason Six Sigma drives upward/onward >>> 99.99%....

Not everything converges. It is entirely possible that a model that produces divergent results is a realistic model of reality. In those cases the best you may do is give a relative probability of the different outcomes.

That doesn't mean the code isn't junk, but it doesn't mean it isn't.

It sounds like it is behaving like a Random Walk, and that may be decent model. A quick Google shows random walks are used to model disease spread. A simple random walk would be to start at 0,0 on an X-Y graph and then move randomly up-down-left-right over and over. When modeling something like a disease, the probability of moving toward more cases would increase as more cases occurred. The randomization of the movement would begin to be weighted differently as the walk continues. It sounds like this was the base of this program. There are papers all over about using this method, but I'm too old and tired to dig into them.

Six Sigma :-) That's a process improvement method. This problem is a behavioral prediction question. Just different things. The virus may be using Six Sigma to improve its own infection rates?

(05-06-2020 07:33 PM)UofMstateU Wrote:  

(05-06-2020 06:11 PM)Claw Wrote:  I don't think running a simulation many times and "averaging" the results is necessarily a bad approach.

Its not bad, if the sim itself is good. But Fergusons simulation is bad.

The best way to describe the flaw programmtically I guess is this:

Say you want want to simulate flipping a coin in a computer program. You have 10 numbers, 1- 10. Half of the numbers are assigned heads, and the other half tails, which represents the 50/50 odds of getting heads or tails on the flip of the coin.
We then run the program, and have it randomly pick a number 1 thru 10. We do that 10,000 times, and record which value, heads or tails, was selected. Over the 10,000 iterations, we would have seen the computer randomly select "heads" about 50% of the time. Thats how running a simulation over and over and averaging should work.

But Fergusons code doesnt do that. Instead of assigning 5 numbers heads and 5 numbers tails, it occassionally decides that heads get 7 numbers, or all 10 numbers, and the reasoning behind it doing that is not known. You can run the sim 10,000 times, but heads is going to be selected at a higher rate than the 50% expected, by a factor of many standard deviations.

So its like the part of Fergusons code that should be stable isnt. Its like if we ask the program to tell us how many of the 10 numbers are assigned heads, it should ALWAYS report back 5, to ensure the odds of getting heads is 50%. But Ferguson's routine doesnt, it deviates from a stable reproducable number. Even running his code on a different computer was changing the results. So his model had biases, completely unrelated to the data of the model, that affects the outcome.

This should be expected. A virus is going to act this way.

The first person to bring it to the USA might have been an introverted researcher. He'll stay at his house or at his lonely lab, and never pass the virus to another soul.

Or, the first person to bring it to the USA might be a party animal who loves sports. He'll pass it to 20 people at a sports bar one night, and 200 people at a Knicks game the next night.

If the epidemic is widespread & tens of thousands of people unwittingly have it, then these personality types average out & can be modelled predictably. But when you're dealing with small numbers of infections at the beginning of a potential epidemic, the behavior of individuals is not predictable.

(05-07-2020 12:17 PM)Captain Bearcat Wrote:  

(05-06-2020 07:33 PM)UofMstateU Wrote:  

(05-06-2020 06:11 PM)Claw Wrote:  I don't think running a simulation many times and "averaging" the results is necessarily a bad approach.

Its not bad, if the sim itself is good. But Fergusons simulation is bad.

The best way to describe the flaw programmtically I guess is this:

Say you want want to simulate flipping a coin in a computer program. You have 10 numbers, 1- 10. Half of the numbers are assigned heads, and the other half tails, which represents the 50/50 odds of getting heads or tails on the flip of the coin.
We then run the program, and have it randomly pick a number 1 thru 10. We do that 10,000 times, and record which value, heads or tails, was selected. Over the 10,000 iterations, we would have seen the computer randomly select "heads" about 50% of the time. Thats how running a simulation over and over and averaging should work.

But Fergusons code doesnt do that. Instead of assigning 5 numbers heads and 5 numbers tails, it occassionally decides that heads get 7 numbers, or all 10 numbers, and the reasoning behind it doing that is not known. You can run the sim 10,000 times, but heads is going to be selected at a higher rate than the 50% expected, by a factor of many standard deviations.

So its like the part of Fergusons code that should be stable isnt. Its like if we ask the program to tell us how many of the 10 numbers are assigned heads, it should ALWAYS report back 5, to ensure the odds of getting heads is 50%. But Ferguson's routine doesnt, it deviates from a stable reproducable number. Even running his code on a different computer was changing the results. So his model had biases, completely unrelated to the data of the model, that affects the outcome.

This should be expected. A virus is going to act this way.

The first person to bring it to the USA might have been an introverted researcher. He'll stay at his house or at his lonely lab, and never pass the virus to another soul.

Or, the first person to bring it to the USA might be a party animal who loves sports. He'll pass it to 20 people at a sports bar one night, and 200 people at a Knicks game the next night.

If the epidemic is widespread & tens of thousands of people unwittingly have it, then these personality types average out & can be modelled predictably. But when you're dealing with small numbers of infections at the beginning of a potential epidemic, the behavior of individuals is not predictable.

No, not because what you're saying isnt correct, but because the PROGRAM that does this isnt doing what you think its doing. In the program 2+2 is occasionally equal to 7. And if you run it on a different computer system, its occasionally 3. Without any reason for there to be a difference. In these cases, 2+2 is always 4, and should never be anything different, but his functions simply throw a different result for no reason. The problem isnt the randomization and averaging out the outcomes, its that the base functions are not stable in results they produce.

WHat the team is doing, after patching a slew of bug fixes that they knew were wrong with the model, is simply trying to use multiple simulations to average out the errors, not average out the real probabilities.

(05-06-2020 06:11 PM)Claw Wrote:  I don't think running a simulation many times and "averaging" the results is necessarily a bad approach.

As a software engineer you might not like it, but as a statistician I don't see that as bad. There are other things that should be looked at including the distribution of the results, outliers, and nature of the randomization.

Having actually written commercial simulators, I don't see a problem with it as long as it was designed to be used that way.

Did you read the full article in the Twitter link? The model is complete junk. Here are the noted issues:

1. The code. It isn’t the code Ferguson ran to produce his famous Report 9. What’s been released on GitHub is a heavily modified derivative of it, after having been upgraded for over a month by a team from Microsoft and others. This revised codebase is split into multiple files for legibility and written in C++, whereas the original program was “a single 15,000 line file that had been worked on for a decade” (this is considered extremely poor practice).

2. The model. What it’s doing is best described as “SimCity without the graphics”. It attempts to simulate households, schools, offices, people and their movements, etc.

3. Non-deterministic outputs. Due to bugs, the code can produce very different results given identical inputs. They routinely act as if this is unimportant.

This problem makes the code unusable for scientific purposes, given that a key part of the scientific method is the ability to replicate results.

(Author goes on to list a number of specific bugs in the software)

4. No tests. In the discussion of the fix for the first bug, Imperial state the code used to be deterministic in that place but they broke it without noticing when changing the code.

Regressions like that are common when working on a complex piece of software, which is why industrial software-engineering teams write automated regression tests. These are programs that run the program with varying inputs and then check the outputs are what’s expected. Every proposed change is run against every test and if any tests fail, the change may not be made.

The Imperial code doesn’t seem to have working regression tests.

5. Undocumented equations. Much of the code consists of formulas for which no purpose is given. John Carmack (a legendary video-game programmer) surmised that some of the code might have been automatically translated from FORTRAN some years ago.

6. Continuing development. Despite being aware of the severe problems in their code that they “haven’t had time” to fix, the Imperial team continue to add new features...

Conclusions. All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one.

**********

Other than that, how was the play, Mrs. Lincoln?