Value vs Reference

[cyboryxmen]

For millennia, noobs all over the world have debated with themselves whether or not they should prefer to pass by value or pass by reference. The common concensus is to pass by value for basic types like ints and floats but also to pass by reference for structs. Today, we're going to test how true this is. More importantly, we'll test if it still rings true for the bigger basic types like doubles.

The test is very basic. You have your random number generator to generate random values(not seeded so it'll generate the same results every time we run the program for consistency) and you have a struct that represents a math vector. This vector has an x and y and will represent position, velocity and acceleration data. We have an array of cars with position, velocity and constant acceleration. The acceleration and velocity will be set once at the start and then the car will update every frame according to a fixed delta time. This roughly represents the kind of thing games would be doing.

Cutting right to the chase, the concensus is in fact true. Structs that contain multiple members are better off referenced than copied. Basic types on the other hand are better off copied. This is true even for doubles which are one of the biggest basic type in C++. The difference gets less noticable in x64 but it's still there. In both cases, the difference is in the order of milliseconds.

Honestly, you might want to use references for everything for the sake of consistency. The loss of performance for basic types isn't that big of a deal and you're better off optimising other things.

I made a repository here so that you can test it out yourself. When running it, be sure that all unnecessary background processes are closed(including Visual Studio!) to get accurate measurements. If you want to, you can even extend it to test out structs that only have one basic type as its sole member!

num tests: 1000
num objects: 10'000

Programmed with:
Average: 18455ns
Highest: 111589ns
Lowest: 14951ns
Median: 15681ns
Standard Deviation: 8554ns

Programmed with: [REFERENCE_SEMANTICS_FOR_BASIC_TYPES]
Average: 18911ns
Highest: 159726ns
Lowest: 14951ns
Median: 16045ns
Standard Deviation: 9611ns

Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS]
Average: 27631ns
Highest: 121800ns
Lowest: 22245ns
Median: 22610ns
Standard Deviation: 12228ns

Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][REFERENCE_SEMANTICS_FOR_BASIC_TYPES]
Average: 41978ns
Highest: 166656ns
Lowest: 24433ns
Median: 26256ns
Standard Deviation: 22999ns

Programmed with: [USE_DOUBLES]
Average: 29597ns
Highest: 141493ns
Lowest: 24797ns
Median: 25892ns
Standard Deviation: 11195ns

Programmed with: [REFERENCE_SEMANTICS_FOR_BASIC_TYPES][USE_DOUBLES]
Average: 33022ns
Highest: 228650ns
Lowest: 28444ns
Median: 29538ns
Standard Deviation: 13505ns

Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][USE_DOUBLES]
Average: 31469ns
Highest: 118153ns
Lowest: 23704ns
Median: 25527ns
Standard Deviation: 12047ns

Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][REFERENCE_SEMANTICS_FOR_BASIC_TYPES][USE_DOUBLES]
Average: 34496ns
Highest: 212604ns
Lowest: 29538ns
Median: 30632ns
Standard Deviation: 13105ns

Programmed with: [_WIN64]
Average: 17894ns
Highest: 178689ns
Lowest: 14586ns
Median: 15681ns
Standard Deviation: 10551ns

Programmed with: [REFERENCE_SEMANTICS_FOR_BASIC_TYPES][_WIN64]
Average: 19817ns
Highest: 106849ns
Lowest: 14951ns
Median: 17140ns
Standard Deviation: 8596ns

Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][_WIN64]
Average: 21191ns
Highest: 80228ns
Lowest: 14586ns
Median: 15681ns
Standard Deviation: 8803ns

Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][REFERENCE_SEMANTICS_FOR_BASIC_TYPES][_WIN64]
Average: 22053ns
Highest: 107213ns
Lowest: 14951ns
Median: 16411ns
Standard Deviation: 10465ns

Programmed with: [USE_DOUBLES][_WIN64]
Average: 26162ns
Highest: 83874ns
Lowest: 17139ns
Median: 28444ns
Standard Deviation: 9313ns

Programmed with: [REFERENCE_SEMANTICS_FOR_BASIC_TYPES][USE_DOUBLES][_WIN64]
Average: 30444ns
Highest: 121800ns
Lowest: 18233ns
Median: 29539ns
Standard Deviation: 10731ns

Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][USE_DOUBLES][_WIN64]
Average: 25892ns
Highest: 79134ns
Lowest: 17139ns
Median: 27715ns
Standard Deviation: 9265ns

Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][REFERENCE_SEMANTICS_FOR_BASIC_TYPES][USE_DOUBLES][_WIN64]
Average: 21076ns
Highest: 169208ns
Lowest: 17139ns
Median: 17505ns
Standard Deviation: 10713ns

[albinopapa]

That's actually kind of interesting. I have wondered about that as well since a Vec2 is two 32 bit floats and a double would be the same size. It would also be the same size as the reference or pointer passed in to the function in x64 mode so I would have thought there would be no difference.

I'm sure the bigger the struct, the worse the performance is going to be passing by value.

[chili]

I wouldn't mind seeing what ref vs. value looks like for basic types with __declspec(noinline), since when the compiler inlines a ref im pretty sure it just turns into an alias for the original object.

[albinopapa]

I wouldn't mind seeing what ref vs. value looks like for basic types with __declspec(noinline), since when the compiler inlines a ref im pretty sure it just turns into an alias for the original object.

I knew there was something more I wanted to know.

Also, I've read that Visual Studio passes by reference anything over 8 bytes, but I'd have to find that info again. This could be dependent on which optimization level you use.

[cyboryxmen]

I've updated the project so that it uses CMake. Not only does it compile all 7 variations of the executables with different switches, it can also be compiled on Linux now! Looking at the results again, the average is lower on pass by value but the standard deviation is higher making the results inconclusive. I'd like to know how this will run on other people's machines.

[albinopapa]

You know, the whole point in having an IDE is so that it handles the "Build" portion or the project.

After a few minutes of floundering, I finally found the damned exe files. They got built to:
C:\Users\(Profile)\CMakeBuilds\92d03ae3-9226-fd38-a363-8dd89c927446\build\x64-Release\value-vs-reference-app
If there is a way to specify where to build these files ( like there is if you just used the regular VS build system ) that would be helpful.
The GUID folder name isn't going to be the same for everyone and was only able to determine which to check is by looking in the Output window of Visual Studio.

I know I could have used CMake GUI or something besides the built in CMake in VS, and honestly I should have since I've had horrible success with it in the past.

Anyway, results as follows:
AMD Ryzen 5 2400G APU
8 GB DDR4 2666

Programmed with: [x64]
Average: 15289ns
Highest: 29922ns
Lowest: 13963ns
Median: 14249ns
Standard Deviation: 1871ns

Programmed with: [REFERENCE_SEMANTICS_FOR_BASIC_TYPES][x64]
Average: 21011ns
Highest: 35622ns
Lowest: 18238ns
Median: 18524ns
Standard Deviation: 3165ns

Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][x64]
Average: 18549ns
Highest: 33627ns
Lowest: 15103ns
Median: 17954ns
Standard Deviation: 1951ns

Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][REFERENCE_SEMANTICS_FOR_BASIC_TYPES][x64]
Average: 18333ns
Highest: 92618ns
Lowest: 17953ns
Median: 18239ns
Standard Deviation: 2399ns

Programmed with: [USE_DOUBLES][x64]
Average: 15408ns
Highest: 137929ns
Lowest: 13394ns
Median: 13964ns
Standard Deviation: 4307ns

Programmed with: [REFERENCE_SEMANTICS_FOR_BASIC_TYPES][USE_DOUBLES][x64]
Average: 14071ns
Highest: 35622ns
Lowest: 13394ns
Median: 13679ns
Standard Deviation: 1466ns

Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][USE_DOUBLES][x64]
Average: 15232ns
Highest: 34768ns
Lowest: 13109ns
Median: 16528ns
Standard Deviation: 2066ns

Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][REFERENCE_SEMANTICS_FOR_BASIC_TYPES][USE_DOUBLES][x64]
Average: 17489ns
Highest: 45027ns
Lowest: 16244ns
Median: 17383ns
Standard Deviation: 1272ns

If you want x86 builds, you'll have to explain how to change it.

[albinopapa]

After some tinkering inside VS, I found a way to change settings and results are as follows:

Programmed with:
Average: 15102ns
Highest: 26503ns
Lowest: 13963ns
Median: 14249ns
Standard Deviation: 1693ns

Programmed with: [REFERENCE_SEMANTICS_FOR_BASIC_TYPES]
Average: 14241ns
Highest: 24223ns
Lowest: 13963ns
Median: 14249ns
Standard Deviation: 587ns

Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS]
Average: 17749ns
Highest: 40182ns
Lowest: 15103ns
Median: 18809ns
Standard Deviation: 2441ns

Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][REFERENCE_SEMANTICS_FOR_BASIC_TYPES]
Average: 15068ns
Highest: 35052ns
Lowest: 14533ns
Median: 14819ns
Standard Deviation: 1319ns

Programmed with: [USE_DOUBLES]
Average: 19144ns
Highest: 57280ns
Lowest: 15103ns
Median: 19094ns
Standard Deviation: 2768ns

Programmed with: [REFERENCE_SEMANTICS_FOR_BASIC_TYPES][USE_DOUBLES]
Average: 16012ns
Highest: 298942ns
Lowest: 14818ns
Median: 15104ns
Standard Deviation: 9174ns

Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][USE_DOUBLES]
Average: 16163ns
Highest: 27073ns
Lowest: 14818ns
Median: 15389ns
Standard Deviation: 1683ns

Programmed with: [VALUE_SEMANTICS_FOR_STRUCTS][REFERENCE_SEMANTICS_FOR_BASIC_TYPES][USE_DOUBLES]
Average: 17692ns
Highest: 38472ns
Lowest: 14818ns
Median: 18808ns
Standard Deviation: 2430ns

[krautersuppe]

You made a whole program to answer my doubts voiced out on discord chat some 10 days ago.

discord_help6.png

(187.75 KiB) Not downloaded yet

Amazing perserverance.

[albinopapa]

As far as references, I would imagine it would depend on what else is going on. I have tried something similar ( value vs reference ) in a ray tracing program and there wasn't much difference as the time spent making copies was very little of the time compared to the actual work being done to render the scene.

I've read in x64 mode in at least VS, anything over 8 bytes the compiler will mark those parameters as either pass by value through SIMD registers or by reference (address) depending on size and number of parameters, but I would imagine that it is best practice to use const reference and other const correct paradigms.