Answers to tasks and questions in long exercise

Part 1: A one-bin counting experiment

A: Computing limits using the asymptotic approximation

Tasks and questions:

Show answer *Larger uncertainties make the limits worse (ie, higher values of the limit); smaller uncertainties improve the limit (lower values of the limit).*
Show answer *This is because the expected limit relies on a background-only Asimov dataset that is created* ***after*** *a background-only fit to the data. By changing the observed the pulls on the NPs in this fit also change, and therefore so does the expected sensitivity.*

Advanced section: B: Computing limits with toys

Tasks and questions:

Show answer *The uncertainty is statistical, because the values of CLs+b and CLb come from counting the number of toys in the tails of the test statistic distributions.*
Show answer *The agreement should be pretty good in this example, but will generally break down once we get to the level of 0-5 events.*
Show answer *For this we need the definition of CLs = CLs+b / CLb. The 0.025 expected quantile is by definition where CLb = 0.025, so for a 95% CL limit we have CLs = 0.05, implying we are looking for the value of r where CLs+b = 0.00125. With 1000 s+b toys we would then only expect 1000 * 0.00125 = 1.25 toys in the tail region we have to integrate over. Contrast this to the median limit where 25 toys would be in this region. This means we have to generate a much larger numbers of toys to get the same statistical power.*

Advanced exercises

Tasks and questions:

Show answer *A "good" approximation is not well defined, but the difference is clearly larger here.*
Show answer *This bump structure comes from the discrete-ness of the Poisson sampling of the toy datasets. Systematic uncertainties then smear these bumps out, but without systematics we would see delta functions corresponding to the possible integer number of events that could be observed. Once we go to more typical multi-bin analyses with more events and systematic uncertainties these discrete-ness washes out very quickly.*

Part 2: A shape-based analysis

A: Setting up the datacard

Only tasks, no questions in this section

B: Running combine for a blind analysis

Tasks and questions:

Show answer *When using --run blind combine will create a background-only Asimov dataset without performing a fit to data first. With --run expected, the observed limit isn't shown, but the background-only Asimov dataset used for the limit calculation is still created after a background-only fit to the data*
Show answer *You should see that with a signal injected the observed limit is worse (has a higher value) than the expected limit: for the expected limit the b-only Asimov dataset is still used, but the observed limit is now calculated on the signal + background Asimov dataset, with a signal at the specified cross section [X].*

C: Using FitDiagnostics

Tasks and questions:

Show answer `CMS_eff_t_highpt` *should have the largest pull (around 0.47)*, `norm_jetFakes` *has the tightest constraint (to 25% of the input uncertainty).*
Show answer *This is still a hot topic in CMS analyses today, and there isn't a right or wrong answer. Essentially we have to judge if our analysis should really be able to provide more information about this parameter than the external measurement that gave us the input uncertainty. So we would not expect to be able to constrain the luminosity uncertainty for example, but uncertainties specific to the analysis might legitimately be constrained.*

D: MC statistical uncertainties

Tasks and questions:

Show answer *Without autoMCStats we find:* `Best fit r: -2.73273 -2.13428/+3.38185`*, with autoMCStats:* `Best fit r: -3.07825 -3.17742/+3.7087`
Show answer *Without autoMCStats we find:* `Best fit r: 9.99978 -4.85341/+6.56233`*, with autoMCStats:*`Best fit r: 9.99985 -5.24634/+6.98266`
Show answer *At first the uncertainties increase, as the threshold increases, and at some point they stabilise. A Poisson threshold at 10 is probably reasonable for this analysis.*

Part 3: Adding control regions

A: Use of rateParams

Tasks and questions:

Show answer *As expected uncertainty you should get -0.42542/+0.458748*
Show answer *They are constrained to around 4-5%*
Show answer *The expected uncertainty is larger with only the SR: -0.463273/+0.499161 compared with -0.42542/+0.458748 in the SR+CR approach.*

B: Nuisance parameter impacts

Tasks and questions:

Show answer *The most important uncertainty is *`norm_jetFakes`*, followed by two MC statistical uncerainties* (`prop_binsignal_region_bin8` *and* `prop_binsignal_region_bin9`).
Show answer *These are freely floating parameters (rate_ttbar and rate_Zll). They have no prior constraint (and so no pull) - we show the best-fit value + uncertainty directly.*

C: Post-fit distributions

Tasks and questions:

The bin errors on the TH1s in the fitdiagnostics file are determined from the systematic uncertainties. In the post-fit these take into account the additional constraints on the nuisance parameters as well as any correlations.

Show answer *There are two effects at play here: the nuisance parameters get constrained, and there are anti-correlations between the parameters which also have the effect of reducing the total uncertainty. Note: the post-fit uncertainty could become larger when rateParams are present as they are not taken into account in the pre-fit uncertainty but do enter in the post-fit uncertainty.*

D: Calculating the significance

Tasks and questions:

Show answer A significance of $5\sigma$ corresponds to a p-value of around $3\cdot 10^{-7}$ - so we need to populate the very tail of the test statistic distribution and this requires generating a large number of toys.

E: Signal strength measurement and uncertainty breakdown

Tasks and questions:

Show datacard line You should add this line to the end of the datacard:
tauID group = CMS_eff_t CMS_eff_t_highpt CMS_scale_t_1prong0pi0_13TeV CMS_scale_t_1prong1pi0_13TeV CMS_scale_t_3prong0pi0_13TeV
Show code This can be done as:
python higgsCombine.part3E.MultiDimFit.mH200.root --others 'higgsCombine.part3E.freezeTauID.MultiDimFit.mH200.root:FreezeTauID:4' 'higgsCombine.part3E.freezeAll.MultiDimFit.mH200.root:FreezeAll:2' -o freeze_third_attempt --breakdown TauID,OtherSyst,Stat
Show answer *They are smaller than both the statistical uncertainty and the remaining systematic uncertainties*

F: Use of channel masking

No specific questions, just tasks