"QuizID"	"AppID"	"AppTitle"	"TaskID"	"TaskText"	"RecordID"	"Record"	"Type"	"Note"
"DSAIRM_usanalysis"	"usanalysis"	"Uncertainty and Sensitivity Analysis"	1	"The creation of parameter samples involves some element of uncertainty, and we need to make use of random numbers. We still want results to be reproducible. That's where the random number seed comes in. As long as the seed is the same, the code should produce the same pseudo-random numbers each time, thus ensuring reproducibility. 


Let's explore this. Leave all settings at their default values. Run the simulation. You should find that the median for __B~steady~__ is around 10736 (as printed below the plot). Run the simulation again with the same settings, you should get exactly the same results. 


Now change the random number seed (rngseed) to 101. Run the simulation again. The median value should be lower. Note that the only thing that is different is that the computer drew different random samples for the parameter values. Keep exploring other rngseed values. You should notice that sometimes results change a bit, sometimes a lot, and that it doesn't matter if you change the seed by just a bit or a lot."	"T1R1"	"Median for __B~steady~__ with _rngseed_ = 101."	"Rounded_Integer"	"Report the rounded integer"
"DSAIRM_usanalysis"	"usanalysis"	"Uncertainty and Sensitivity Analysis"	2	"The more samples you have, the more robust the results are to changes in the underlying sample generation (determined by the random number seed). Try checking this by running 100 samples, instead of 10, with the two random number seeds from above. Be patient, this might take a good while. Since each sample means a run of the underlying simulation model, things start to go slow for increased sample size. 


For _rngseed_ = 100, the median of __B~steady~__ is around 9306, for the second _rngseed_ value it is somewhat higher. You might have noticed that the variability shrinks in the central quantities (mean, median) for the larger sample size, not much in the extremes (min/max). You might have also noticed that some __B~peak~__ is inherently more variable than __B~steady~__ and __I~steady~__. Also note the 'system might not have reached steady state' message. If for too many of the samples steady state has not been reached, the results for __B~steady~__ and __I~steady~__ are not correct. In that case you need to increase the simulation time to allow the system to settle into steady state. For some parameter combinations, that can take very long."	"T2R1"	"Median for __B~steady~__ with rngseed = 101, 100 samples."	"Rounded_Integer"	"Report the rounded integer"
"DSAIRM_usanalysis"	"usanalysis"	"Uncertainty and Sensitivity Analysis"	3	"Recall the underlying model and its behavior. If you can't, revisit the ___Basic Bacteria Model___ app. Use your understanding of the model to predict what happens if you increase both lower and upper bound for the immune response activation rate, _r_. Increase both lower and upper bounds (_rmin_ and _rmax_) by a factor of 10. Set seed to 100, with 50 samples. Leave everything else at the default values. Run the simulations and see how results change. 


Now go the opposite direction, lower the initial lower/upper bounds by a factor of 10 from their original value. Run simulations, see how results change. Especially for the steady states, you know from previous apps how things should change as you change _r_. Compare your simulation results with your expectations."	"T3R1"	"Median for __B~steady~__, _rmin_ = 0.001 and _rmax_ = 0.002."	"Rounded_Integer"	"Report the rounded integer"
"DSAIRM_usanalysis"	"usanalysis"	"Uncertainty and Sensitivity Analysis"	4	"Continue this exploration as you change ranges for different parameters. Note that while you change ranges for some parameters, every time you get samples for all parameters in those ranges. This is different from the ___Basic Bacteria Exploration___ app where only the parameter we scanned over changed and every other parameter remainded fixed. Usually, you have a parameter you are most interested in, and you'll explore that parameter in detail, while allowing for variability in the others. As you play around, it is likely that for some settings you'll see warning or error messages on the `R` console. That generally means that the parameters for a given simulation are such that the differential equation solver can't properly run the model. That usually corresponds to biologically unrealistic parameter settings. We'll ignore them, but if you did a research project and you got such warning or error messages, you'd have to figure out why you get them and only once you fully understand why is it maybe ok to ignore them."	"T4R1"	"Nothing"	"None"	""
"DSAIRM_usanalysis"	"usanalysis"	"Uncertainty and Sensitivity Analysis"	5	"So far, we only looked at the total variability in outcomes as parameters are being varied/sampled (i.e. the boxplots and summary measures printed underneath the plots). The above approach of exploring the impact of a parameter on results by varying bounds and repeatedly looking at boxplots or summary measures is somewhat tedious. This is where sensitivity analysis comes in. By running the same simulations, but plotting outcomes as scatterplots, we can explore how variation in a paramter leads to variation in outcomes. Let's explore this. 


Set all input values to their defaults. Then set sample size to 50 and switch the plot type from boxplot to scatterplot. Run the simulation. You'll see how the 3 outcomes of interest vary with one of the varied inputs. Here this is the initial starting value for the bacteria. Which in some sense is not a model parameter, but an initial condition. However, for the purpose of U/S analysis, either is considered a varied input/parameter. Also look at the text below the plots. For each parameter-output pair, the code computes a rank correlation coefficient. Numbers close to 0 mean there is essentially no correlation, close to 1 or -1 means a large positive or negative correlation. (One could compute p-values for these correlations, but they are somewhat meaningless since the values will get smaller the more samples you use, so you can basically produce any p-value you want.). 


Let's explore this correlation for a parameter we are familiar with from previous apps, the bacteria growth rate. Set all inputs to their defaults, then set samples to 100. Also switch the plot type to _Scatterplot_ and _Parameter for scatterplot_ to _g_. Run the simulation. We found earlier that __B__ at steady state does not depend on _g_, while __I__ does. The correlation coefficients show this as well. Note that they are not exactly 0 and 1 since there is additional variability due to the other parameters being sampled as well."	"T5R1"	"Rank Correlation Coefficient between _g_ and __I~steady~__ (2 decimal places)."	"Rounded_Numeric"	"Report the value rounded to two significant digits"
"DSAIRM_usanalysis"	"usanalysis"	"Uncertainty and Sensitivity Analysis"	6	"Let's explore in more detail how variability in different parameters impacts results. We'll pretend that we happen to know several model inputs with certainty. To do so, we'll impose no variability for some parameters. For the following parameters, set *both* their lower and upper bound to the specified value: initial value for bacteria and immune response = 1,  _B~max~_ = 1e5,  _d~B~_ = 1, _k_ = _r_ = 10^-4^. Let _d~I~_ vary between 1 and 2 and give _g_ a mean of 5 and variance of 1. 100 samples, seed at 100, scatterplot for _g_. Run the simulation. You'll see a very clear pattern relating the outcomes to _g_. Switch to plotting _d~I~_, you'll again see nice patterns. 


Confirm that, as expected from the steady state equations, __I~steady~__ depends positively on _g_ and negatively on _d~I~_. Both the scatterplots and the correlation coefficient should give you that information. Also note the distribution for _g_ and _d~I~_. The former has more points around its mean and less for lower/higher values, while _d~I~_ values are uniformly distributed along the x-axis. This comes from the underlying assumption about how the parameters are distributed, gamma distribution versus uniform distribution."	"T6R1"	"Rank Correlation Coefficient between _g_ and __I~steady~__ (2 decimal places)."	"Rounded_Numeric"	"Report the value rounded to two significant digits"
"DSAIRM_usanalysis"	"usanalysis"	"Uncertainty and Sensitivity Analysis"	7	"In the ___Bacteria Model Exploration___ app, we investigated the fact that for some value of _d~I~_, we get a switch from a steady state to no steady state (if you don't remember, I suggest you revisit that app first). This behavior can also (accidentally) happen when you sample over parameters. That's not ideal since it means you combine to qualitatively different results in a single distribution. Let's explore this. 


Reset all values to their defaults. Then, set ranges for _d~I~_ to be from 1 to 5, 100 samples, a scatterplot with _d~I~_ on the x-axis. Also change to plotly as the plot engine. Run the simulation. You should see mostly non-zero values for __I~steady~__ (you might need to click the auto-scale button on the plot). Now change the range for _d~I~_ to be from 10 to 14. You'll now see a large number of zeros for __I~steady~__, especially for high _d~I~_ values. 


The important take-home message from this is that the influence of a parameter on some outcome can be different over different ranges. For instance in range A-B, the parameter might have a major influence, but once the parameter value goes above B, the parameter does not further influence the result. If you have large uncertainty in your parameters, it might be worth considering both the full range, and dividing the range into smaller areas to see how the parameter behaves."	"T7R1"	"Rank Correlation Coefficient between _d~I~_ and __I~steady~_ for range 10-14 (2 decimal places)."	"Rounded_Numeric"	"Report the value rounded to two significant digits"
