writing std::vector data to file..

The Partridge Family were neither partridges nor a family. Discuss.
MrGodin
Posts: 721
Joined: November 30th, 2013, 7:40 pm
Location: Merville, British Columbia Canada

writing std::vector data to file..

Post by MrGodin » January 21st, 2018, 6:50 pm

Hey folks I am trying to write mesh data to file so i don't have to load .obj file all the time. I am getting an output file with 0 bytes in it.
I am using the C style file write

Code: Select all

void SaveToFile(const std::string& filename)
	{
		Geometry::LOD_VertexMin* v = new Geometry::LOD_VertexMin[verts.size()];
		for (int i = 0 ;i < verts.size(); i++)
			v[i] = verts[i];
		FILE* file;
		fopen_s(&file, filename.c_str(), "wb");
		fwrite(v, sizeof(Geometry::LOD_VertexMin), sizeof(Geometry::LOD_VertexMin)*verts.size(), file);
		fclose(file);
		SafeDelete(v);
	}

any suggestions ?
If i go sizeof(v), this is just giving me the size of a pointer right?. the file outputs 256 bytes of data.
Geometry::LOD_VertexMin is a vector3 of position, vector2 texCoords and vector3 normals which should be 3x4 + 2x4 + 3*4 = 32 and what i'm guessing is a pointer is 8 bytes? so hence i get 256 bytes written
Curiosity killed the cat, satisfaction brought him back

albinopapa
Posts: 4373
Joined: February 28th, 2013, 3:23 am
Location: Oklahoma, United States

Re: writing std::vector data to file..

Post by albinopapa » January 21st, 2018, 9:05 pm

What you have should have worked, but you could try dereferencing the Geometry::LOD_VertexMin pointer.

size_t fwrite ( const void * ptr, size_t size, size_t count, FILE * stream );
elements_written = fwrite( source, sizeof(element), number_of_elements, destination );

You could also write it as:

Code: Select all

void SaveToFile(const std::string& filename)
{
      FILE* file;
      fopen_s(&file, filename.c_str(), "wb");
      const auto elements_written = fwrite( verts.data(), sizeof( verts[0] ), verts.size(), file);
      assert( elements_written == verts.size() );
      fclose(file);
}
or

Code: Select all

void SaveToFile(const std::string& filename)
{
      FILE* file;
      fopen_s(&file, filename.c_str(), "wb");
      const auto elements_written = fwrite( verts.data(), sizeof( char ), sizeof( verts[0] ) * verts.size(), file);
      assert( elements_written == sizeof( verts[0] ) * verts.size() );
      fclose(file);
}
Check the return value of fwrite to see how many elements it actually wrote to file.
If you use the first verion, then it should be equal to the number of bytes in the buffer.
If you use the second version, then it should be equal to the number of elements in the verts vector.
If you think paging some data from disk into RAM is slow, try paging it into a simian cerebrum over a pair of optical nerves. - gameprogrammingpatterns.com

albinopapa
Posts: 4373
Joined: February 28th, 2013, 3:23 am
Location: Oklahoma, United States

Re: writing std::vector data to file..

Post by albinopapa » January 21st, 2018, 9:16 pm

It's a little lengthy, but I prefer to store my vertex data as separate elements, grouping all the positions together, then all the texcoords together then all the normals together. I store their counts at the top of the file, in that order. That way, when I have models that don't have one of them ( either texcoords or normals ) I can still use the same data layout and skip whatever the file doesn't have. Normals can be calculated if there aren't any, the only thing that would be difficult or near impossible to generate would be texcoords.
If you think paging some data from disk into RAM is slow, try paging it into a simian cerebrum over a pair of optical nerves. - gameprogrammingpatterns.com

MrGodin
Posts: 721
Joined: November 30th, 2013, 7:40 pm
Location: Merville, British Columbia Canada

Re: writing std::vector data to file..

Post by MrGodin » January 21st, 2018, 9:21 pm

Thanks albino, i did this .. std::size_t result = fwrite(v, sizeof(*v), verts.size(), file); where result returned the size of verts (6144) in this case and the resulting filesize is 192kb (192,000), 6144 multiplied by 32 = 196,608 bytes.
Curiosity killed the cat, satisfaction brought him back

MrGodin
Posts: 721
Joined: November 30th, 2013, 7:40 pm
Location: Merville, British Columbia Canada

Re: writing std::vector data to file..

Post by MrGodin » January 21st, 2018, 9:28 pm

But yeah, i was thinking of creating a struct to use for file writing, something like..
struct
{
size_t bytesize = sizeof(Geometry::LOD_VertexMin);
size_t vert_container_size = 6144;(verts.size())
Geometry::LOD_VertexMin* vert_data;
}
basically just what you said there ^^.
Cheers
Curiosity killed the cat, satisfaction brought him back

MrGodin
Posts: 721
Joined: November 30th, 2013, 7:40 pm
Location: Merville, British Columbia Canada

Re: writing std::vector data to file..

Post by MrGodin » January 21st, 2018, 9:30 pm

I read somewhere that verts.data() contains the iterator pointers as well, is that true ?
Curiosity killed the cat, satisfaction brought him back

MrGodin
Posts: 721
Joined: November 30th, 2013, 7:40 pm
Location: Merville, British Columbia Canada

Re: writing std::vector data to file..

Post by MrGodin » January 21st, 2018, 9:52 pm

well OK, it works.. this is temp code just to see if it worked

Code: Select all

template<typename T>
struct modelData
{
	std::size_t byteSize;
	std::size_t container_size;
	T* data;
};

Code: Select all

void SaveToFile(const std::string& filename)
	{
		modelData<Geometry::LOD_VertexMin> mData;
		mData.byteSize = sizeof(Geometry::LOD_VertexMin);
		mData.container_size = verts.size();
		mData.data = verts.data();
		
		Geometry::LOD_VertexMin* v = new Geometry::LOD_VertexMin[verts.size()];
		for (int i = 0 ;i < verts.size(); i++)
			v[i] = verts[i];
		FILE* file;
		fopen_s(&file, filename.c_str(), "wb");
		std::size_t result = fwrite(&mData, sizeof(mData), 1, file);
		modelData<Geometry::LOD_VertexMin> d2;
		
		fclose(file);
		fopen_s(&file, filename.c_str(), "rb");
		fread(&d2, sizeof(d2), 1, file);
		fclose(file);

		std::vector<Geometry::LOD_VertexMin> v2;
		for (int i = 0; i < d2.container_size; i++)
			v2.push_back(d2.data[i]);
		SafeDelete(v);
	}
I changed the struct to a template and it worked fine
Curiosity killed the cat, satisfaction brought him back

MrGodin
Posts: 721
Joined: November 30th, 2013, 7:40 pm
Location: Merville, British Columbia Canada

Re: writing std::vector data to file..

Post by MrGodin » January 21st, 2018, 10:01 pm

I am doing the same thing to image files by loading the image and saving just the pixel data then loading it up and creating a ID3D11ShaderResourceView.. :)
Curiosity killed the cat, satisfaction brought him back

albinopapa
Posts: 4373
Joined: February 28th, 2013, 3:23 am
Location: Oklahoma, United States

Re: writing std::vector data to file..

Post by albinopapa » January 22nd, 2018, 3:44 am

Code: Select all

template<typename T>
struct modelData
{
   size_t num_positions, num_texcoords, num_normals;
};

void SaveToFile(const std::string& filename)
{
      modelData<Geometry::LOD_VertexMin> mData;
      mData.num_positions = mData.num_texcoords = num_normals = verts.size();
      
      std::vector<Vec3> positions( mData.num_positions );
      std::vector<Vec2> texcoords( mData.num_texcoords );
      std::vector<Vec3> normals( mData.num_normals );

      size_t i = 0;
      for(const auto& vert : verts )
      {
            positions[i] = vert.position;
            texcoords[i] = vert.texcoord;
            normals[i] = vert.normal;
      }

      std::ofstream file( filename, std::ios::binary );
      file.write( reinterpret_cast<char*>( &mData ), sizeof( mData ) );
      file.write( reinterpret_cast<char*>( positions.data() ), sizeof( Vec3 ) * positions.size() );
      file.write( reinterpret_cast<char*>( texcoords.data() ), sizeof( Vec2 ) * texcoords.size() );
      file.write( reinterpret_cast<char*>( normals.data() ), sizeof( Vec3 ) * normals.size() );
      file.close();
}
std::vector<Geometry::LOD_VertexMin> LoadFromFile( std::string filename )
{
      std::ifstream file( filename, std::ios::binary );
      assert( file.is_open() );
      
      modelData<Geometry::LOD_VertexMin> mData;
      file.read( reinterpret_cast<char*>( &mData ), sizeof( mData ) );
      
      std::vector<Vec3> positions( mData.num_positions ), normals( m_data.num_normals );
      std::vector<Vec2> texcoords( mData.num_texcoords );
      file.read( reinterpret_cast<char*>( &positions[0] ), sizeof(Vec3) * positions.size() );
      file.read( reinterpret_cast<char*>( &texcoords[0] ), sizeof(Vec2) * texcoords.size() );
      file.read( reinterpret_cast<char*>( &normals[0] ), sizeof(Vec3) * normals.size() );
      file.close();

      std::vector<Geometry::LOD_VertexMin> verts( mData.num_positions );
      size_t i = 0;
      for( auto& vert : verts )
      {
            vert.position = positions[i];
            if( !texcoords.empty() )
            {
                  vert.texcoord = texcoords[i];
            }
            if( !normals.empty() )
            {
                  vert.normal = normals[i];
            }
      }

      return verts;
}
This is what I was getting at.

No, I've never heard of std::vector::data() returning iterators, it should be a T* or const T* directly to the underlying buffer. cplusplus.com - std::vector::data()
If you think paging some data from disk into RAM is slow, try paging it into a simian cerebrum over a pair of optical nerves. - gameprogrammingpatterns.com

MrGodin
Posts: 721
Joined: November 30th, 2013, 7:40 pm
Location: Merville, British Columbia Canada

Re: writing std::vector data to file..

Post by MrGodin » January 22nd, 2018, 3:50 am

That's a better approach albinopapa. Works real well and i like the arbitrary size of the data. Certainly speed up the process of loading meshes.
Curiosity killed the cat, satisfaction brought him back

Post Reply