Value vs Reference
Posted: June 19th, 2018, 3:52 am
For millennia, noobs all over the world have debated with themselves whether or not they should prefer to pass by value or pass by reference. The common concensus is to pass by value for basic types like ints and floats but also to pass by reference for structs. Today, we're going to test how true this is. More importantly, we'll test if it still rings true for the bigger basic types like doubles.
The test is very basic. You have your random number generator to generate random values(not seeded so it'll generate the same results every time we run the program for consistency) and you have a struct that represents a math vector. This vector has an x and y and will represent position, velocity and acceleration data. We have an array of cars with position, velocity and constant acceleration. The acceleration and velocity will be set once at the start and then the car will update every frame according to a fixed delta time. This roughly represents the kind of thing games would be doing.
Cutting right to the chase, the concensus is in fact true. Structs that contain multiple members are better off referenced than copied. Basic types on the other hand are better off copied. This is true even for doubles which are one of the biggest basic type in C++. The difference gets less noticable in x64 but it's still there. In both cases, the difference is in the order of milliseconds.
Honestly, you might want to use references for everything for the sake of consistency. The loss of performance for basic types isn't that big of a deal and you're better off optimising other things.
I made a repository here so that you can test it out yourself. When running it, be sure that all unnecessary background processes are closed(including Visual Studio!) to get accurate measurements. If you want to, you can even extend it to test out structs that only have one basic type as its sole member!
num tests: 1000
num objects: 10'000
Programmed with:
Average: 18455ns
Highest: 111589ns
Lowest: 14951ns
Median: 15681ns
Standard Deviation: 8554ns
Programmed with: [REFERENCE_SEMANTICS_FOR_BASIC_TYPES]
Average: 18911ns
Highest: 159726ns
Lowest: 14951ns
Median: 16045ns
Standard Deviation: 9611ns
Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS]
Average: 27631ns
Highest: 121800ns
Lowest: 22245ns
Median: 22610ns
Standard Deviation: 12228ns
Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][REFERENCE_SEMANTICS_FOR_BASIC_TYPES]
Average: 41978ns
Highest: 166656ns
Lowest: 24433ns
Median: 26256ns
Standard Deviation: 22999ns
Programmed with: [USE_DOUBLES]
Average: 29597ns
Highest: 141493ns
Lowest: 24797ns
Median: 25892ns
Standard Deviation: 11195ns
Programmed with: [REFERENCE_SEMANTICS_FOR_BASIC_TYPES][USE_DOUBLES]
Average: 33022ns
Highest: 228650ns
Lowest: 28444ns
Median: 29538ns
Standard Deviation: 13505ns
Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][USE_DOUBLES]
Average: 31469ns
Highest: 118153ns
Lowest: 23704ns
Median: 25527ns
Standard Deviation: 12047ns
Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][REFERENCE_SEMANTICS_FOR_BASIC_TYPES][USE_DOUBLES]
Average: 34496ns
Highest: 212604ns
Lowest: 29538ns
Median: 30632ns
Standard Deviation: 13105ns
Programmed with: [_WIN64]
Average: 17894ns
Highest: 178689ns
Lowest: 14586ns
Median: 15681ns
Standard Deviation: 10551ns
Programmed with: [REFERENCE_SEMANTICS_FOR_BASIC_TYPES][_WIN64]
Average: 19817ns
Highest: 106849ns
Lowest: 14951ns
Median: 17140ns
Standard Deviation: 8596ns
Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][_WIN64]
Average: 21191ns
Highest: 80228ns
Lowest: 14586ns
Median: 15681ns
Standard Deviation: 8803ns
Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][REFERENCE_SEMANTICS_FOR_BASIC_TYPES][_WIN64]
Average: 22053ns
Highest: 107213ns
Lowest: 14951ns
Median: 16411ns
Standard Deviation: 10465ns
Programmed with: [USE_DOUBLES][_WIN64]
Average: 26162ns
Highest: 83874ns
Lowest: 17139ns
Median: 28444ns
Standard Deviation: 9313ns
Programmed with: [REFERENCE_SEMANTICS_FOR_BASIC_TYPES][USE_DOUBLES][_WIN64]
Average: 30444ns
Highest: 121800ns
Lowest: 18233ns
Median: 29539ns
Standard Deviation: 10731ns
Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][USE_DOUBLES][_WIN64]
Average: 25892ns
Highest: 79134ns
Lowest: 17139ns
Median: 27715ns
Standard Deviation: 9265ns
Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][REFERENCE_SEMANTICS_FOR_BASIC_TYPES][USE_DOUBLES][_WIN64]
Average: 21076ns
Highest: 169208ns
Lowest: 17139ns
Median: 17505ns
Standard Deviation: 10713ns
The test is very basic. You have your random number generator to generate random values(not seeded so it'll generate the same results every time we run the program for consistency) and you have a struct that represents a math vector. This vector has an x and y and will represent position, velocity and acceleration data. We have an array of cars with position, velocity and constant acceleration. The acceleration and velocity will be set once at the start and then the car will update every frame according to a fixed delta time. This roughly represents the kind of thing games would be doing.
Cutting right to the chase, the concensus is in fact true. Structs that contain multiple members are better off referenced than copied. Basic types on the other hand are better off copied. This is true even for doubles which are one of the biggest basic type in C++. The difference gets less noticable in x64 but it's still there. In both cases, the difference is in the order of milliseconds.
Honestly, you might want to use references for everything for the sake of consistency. The loss of performance for basic types isn't that big of a deal and you're better off optimising other things.
I made a repository here so that you can test it out yourself. When running it, be sure that all unnecessary background processes are closed(including Visual Studio!) to get accurate measurements. If you want to, you can even extend it to test out structs that only have one basic type as its sole member!
num tests: 1000
num objects: 10'000
Programmed with:
Average: 18455ns
Highest: 111589ns
Lowest: 14951ns
Median: 15681ns
Standard Deviation: 8554ns
Programmed with: [REFERENCE_SEMANTICS_FOR_BASIC_TYPES]
Average: 18911ns
Highest: 159726ns
Lowest: 14951ns
Median: 16045ns
Standard Deviation: 9611ns
Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS]
Average: 27631ns
Highest: 121800ns
Lowest: 22245ns
Median: 22610ns
Standard Deviation: 12228ns
Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][REFERENCE_SEMANTICS_FOR_BASIC_TYPES]
Average: 41978ns
Highest: 166656ns
Lowest: 24433ns
Median: 26256ns
Standard Deviation: 22999ns
Programmed with: [USE_DOUBLES]
Average: 29597ns
Highest: 141493ns
Lowest: 24797ns
Median: 25892ns
Standard Deviation: 11195ns
Programmed with: [REFERENCE_SEMANTICS_FOR_BASIC_TYPES][USE_DOUBLES]
Average: 33022ns
Highest: 228650ns
Lowest: 28444ns
Median: 29538ns
Standard Deviation: 13505ns
Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][USE_DOUBLES]
Average: 31469ns
Highest: 118153ns
Lowest: 23704ns
Median: 25527ns
Standard Deviation: 12047ns
Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][REFERENCE_SEMANTICS_FOR_BASIC_TYPES][USE_DOUBLES]
Average: 34496ns
Highest: 212604ns
Lowest: 29538ns
Median: 30632ns
Standard Deviation: 13105ns
Programmed with: [_WIN64]
Average: 17894ns
Highest: 178689ns
Lowest: 14586ns
Median: 15681ns
Standard Deviation: 10551ns
Programmed with: [REFERENCE_SEMANTICS_FOR_BASIC_TYPES][_WIN64]
Average: 19817ns
Highest: 106849ns
Lowest: 14951ns
Median: 17140ns
Standard Deviation: 8596ns
Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][_WIN64]
Average: 21191ns
Highest: 80228ns
Lowest: 14586ns
Median: 15681ns
Standard Deviation: 8803ns
Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][REFERENCE_SEMANTICS_FOR_BASIC_TYPES][_WIN64]
Average: 22053ns
Highest: 107213ns
Lowest: 14951ns
Median: 16411ns
Standard Deviation: 10465ns
Programmed with: [USE_DOUBLES][_WIN64]
Average: 26162ns
Highest: 83874ns
Lowest: 17139ns
Median: 28444ns
Standard Deviation: 9313ns
Programmed with: [REFERENCE_SEMANTICS_FOR_BASIC_TYPES][USE_DOUBLES][_WIN64]
Average: 30444ns
Highest: 121800ns
Lowest: 18233ns
Median: 29539ns
Standard Deviation: 10731ns
Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][USE_DOUBLES][_WIN64]
Average: 25892ns
Highest: 79134ns
Lowest: 17139ns
Median: 27715ns
Standard Deviation: 9265ns
Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][REFERENCE_SEMANTICS_FOR_BASIC_TYPES][USE_DOUBLES][_WIN64]
Average: 21076ns
Highest: 169208ns
Lowest: 17139ns
Median: 17505ns
Standard Deviation: 10713ns