Often, the results of a performance analysis depend on the computers used and specific features of software or libraries used. Therefore, it is essential to provide a comprehensive description of the experimental/simulation setup. This not only allows others to replicate the experiments but also facilitates the scrutiny of results for rigor, statistical validity, and impartiality. Ironically, the “experimental setup” section often ends up being the briefest in many research papers and theses, as authors attempt to economize space by minimizing details.
Here are some insights and tips on what to incorporate, drawn from my research and supervisory experience, in addition to elaborating on the experimental or simulation setup:
Type of Simulation
- Specify whether the results stem from experimentation, emulation, or simulation. In the case of simulation, delve into additional particulars such as whether it’s a Discrete Event, Monte Carlo, Stochastic, or Deterministic simulation.
- Indicate the timing of result measurements, whether during steady-state, dynamic, starting/ramp-up, or terminating/ramp-down state(s).
Number of Experiments/Simulations
- Discuss the sampling methods employed—whether it’s random, systematic/deterministic, stratified, snowballing, etc.
- Number of samples taken, or experiments conducted. While some times it is ok to take 10 or 20 samples, ideally number of samples should be determined by the desired confidence interval.
- Include information about confidence intervals or accuracy levels. The commonly used 95% confidence interval implies a 95% probability that the population mean value falls within ∓1.96 standard deviations from the sample mean.
- Provide insights into random number generation, including whether a distinct seed is utilized for each run and specifying the seeds employed, especially when using widely available simulators/emulators.
Computer(s) Used
- CPU details — Cover clock speed, number of cores, cache sizes, specific CPU model (mention the model number instead of general terms like Intel i7, Xeon, AMD Opteron), and any special instructions utilized (such as MMX, SSE, AVX) or compiler options set.
- Memory specifications — Include capacity and speed details.
- Mention the use of accelerators, such as GPUs, TPUs, or Intel Xeon Phi, but only when their impact is significant.
- Provide information about the sensors and actuators attached, focusing on their relevance and potential impact.
- Operating system overview — Detail the type, version, any special features, additional services running, the node’s load, and any performance enhancements or optimizations implemented.
The choice of the subset mentioned above to include depends on various factors, including the context, type of publication, and the venue. It’s advisable to explore how the related work describes their experimental setups. As a rule of thumb, a thesis typically provides more extensive details compared to a paper, making the author’s thesis a valuable resource for readers seeking in-depth information.