当加载清除大量的数据时，std :: vector会变得越来越慢

概述当加载/清除大量的数据时，std :: vector会变得越来越慢

问题

我有一个相当复杂的image processing应用程序，其中一个子模块需要将巨大的二进制位图加载到内存中。 实际上高达96 GB（意思是888 888 x 888 888像素的图像）。 磁盘是2xSSD raID0，读写速度约为1 GB / s。 它将图像加载到智能指针向量（每个元素表示8个像素）的向量中（每个元素表示一个位图中的一行）。 这里的一个奇怪的问题是，在重复加载和清除向量之后（我发现内存实际上是没有内存泄漏的填充和清除），每次迭代似乎花费的时间越来越长。 特别清理记忆需要很长时间。

testing

我做了一些简单的testing应用程序来testing这个孤立的，从不同的angular度。 用原始指针代替智能指针给了同样奇怪的行为。 然后，我试图使用本地数组而不是vector，并做了诀窍。 使用向量后，100次迭代的载入/清除24 GB时间急剧增加，而arrays实现时间稳定。 下面是testing应用程序填充内存与24 GB的垃圾，而不是加载一个实际的图像，具有相同的结果。 在windows 10 Pro上使用128 GB RAM进行testing，并使用Visual Studio 2013 Update 5进行构build。

为什么我的结构元素带有填充字节？

基于共享内存的聊天应用程序出现问题

在linux中计算内存区域的大小

Java进程内存比指定的限制大得多

在windows映射我的程序的DLL之前，如何保留内存区域？

这个函数使用vector来加载/清除：

voID SimpleLoadAndClear_Vector(int wIDth,int height) { time_t start_time,end_time; // Load memory time(&start_time); cout << "Loading image into memory..."; auto wIDth_bytes = wIDth / 8; auto image = new vector<vector<unsigned char>*>(height); for (auto y = 0; y < height; y++) { (*image)[y] = new vector<unsigned char>(wIDth_bytes); auto row_ptr = (*image)[y]; for (auto b = 0; b < wIDth_bytes; b++) { (*row_ptr) = 0xFF; } } cout << "DONE: "; time(&end_time); auto mem_load = (int)difftime(end_time,start_time); cout << to_string(mem_load) << " sec" << endl; // Clear memory time(&start_time); cout << "Clearing memory..."; for (auto y = 0; y < height; y++) { delete (*image)[y]; } delete image; cout << "DONE: "; time(&end_time); auto mem_clear = (int)difftime(end_time,start_time); cout << to_string(mem_clear) + " sec" << endl; }

该函数使用数组进行加载清除：

voID SimpleLoadAndClear_Array(int wIDth,end_time; // Load memory time(&start_time); cout << "Loading image into memory..."; auto wIDth_bytes = wIDth / 8; auto image = new unsigned char*[height]; for (auto y = 0; y < height; y++) { image[y] = new unsigned char[wIDth_bytes]; auto row_ptr = image[y]; for (auto b = 0; b < wIDth_bytes; b++) { row_ptr[b] = 0xFF; } } cout << "DONE: "; time(&end_time); auto mem_load = (int)difftime(end_time,start_time); cout << to_string(mem_load) << " sec" << endl; // Clear memory time(&start_time); cout << "Clearing memory..."; for (auto y = 0; y < height; y++) { delete[] image[y]; } delete[] image; cout << "DONE: "; time(&end_time); auto mem_clear = (int)difftime(end_time,start_time); cout << to_string(mem_clear) + " sec" << endl; }

这是调用上述加载/清除function的主要function：

voID main() { auto wIDth = 455960; auto height = 453994; auto i_max = 50; for (auto i = 0; i < i_max; i++){ SimpleLoadAndClear_Vector(wIDth,height); } }

在50次迭代之后，vector版本的testing输出如下所示（显然，加载/清除时间越来越多）：

Loading image into memory...DONE: 19 sec Clearing memory...DONE: 24 sec Loading image into memory...DONE: 40 sec Clearing memory...DONE: 20 sec Loading image into memory...DONE: 27 sec Clearing memory...DONE: 39 sec Loading image into memory...DONE: 35 sec Clearing memory...DONE: 24 sec Loading image into memory...DONE: 27 sec Clearing memory...DONE: 34 sec Loading image into memory...DONE: 33 sec Clearing memory...DONE: 29 sec Loading image into memory...DONE: 27 sec Clearing memory...DONE: 35 sec Loading image into memory...DONE: 32 sec Clearing memory...DONE: 33 sec Loading image into memory...DONE: 28 sec Clearing memory...DONE: 37 sec Loading image into memory...DONE: 31 sec Clearing memory...DONE: 35 sec Loading image into memory...DONE: 30 sec Clearing memory...DONE: 38 sec Loading image into memory...DONE: 31 sec Clearing memory...DONE: 38 sec Loading image into memory...DONE: 31 sec Clearing memory...DONE: 41 sec Loading image into memory...DONE: 32 sec Clearing memory...DONE: 40 sec Loading image into memory...DONE: 33 sec Clearing memory...DONE: 42 sec Loading image into memory...DONE: 35 sec Clearing memory...DONE: 43 sec Loading image into memory...DONE: 34 sec Clearing memory...DONE: 46 sec Loading image into memory...DONE: 36 sec Clearing memory...DONE: 47 sec Loading image into memory...DONE: 35 sec Clearing memory...DONE: 49 sec Loading image into memory...DONE: 37 sec Clearing memory...DONE: 50 sec Loading image into memory...DONE: 37 sec Clearing memory...DONE: 51 sec Loading image into memory...DONE: 39 sec Clearing memory...DONE: 51 sec Loading image into memory...DONE: 39 sec Clearing memory...DONE: 53 sec Loading image into memory...DONE: 40 sec Clearing memory...DONE: 52 sec Loading image into memory...DONE: 40 sec Clearing memory...DONE: 55 sec Loading image into memory...DONE: 41 sec Clearing memory...DONE: 56 sec Loading image into memory...DONE: 41 sec Clearing memory...DONE: 59 sec Loading image into memory...DONE: 42 sec Clearing memory...DONE: 59 sec Loading image into memory...DONE: 42 sec Clearing memory...DONE: 60 sec Loading image into memory...DONE: 44 sec Clearing memory...DONE: 60 sec Loading image into memory...DONE: 44 sec Clearing memory...DONE: 63 sec Loading image into memory...DONE: 44 sec Clearing memory...DONE: 63 sec Loading image into memory...DONE: 45 sec Clearing memory...DONE: 64 sec Loading image into memory...DONE: 46 sec Clearing memory...DONE: 65 sec Loading image into memory...DONE: 45 sec Clearing memory...DONE: 67 sec Loading image into memory...DONE: 47 sec Clearing memory...DONE: 69 sec Loading image into memory...DONE: 47 sec Clearing memory...DONE: 70 sec Loading image into memory...DONE: 48 sec Clearing memory...DONE: 72 sec Loading image into memory...DONE: 48 sec Clearing memory...DONE: 74 sec Loading image into memory...DONE: 49 sec Clearing memory...DONE: 74 sec Loading image into memory...DONE: 50 sec Clearing memory...DONE: 74 sec Loading image into memory...DONE: 50 sec Clearing memory...DONE: 76 sec Loading image into memory...DONE: 51 sec Clearing memory...DONE: 78 sec Loading image into memory...DONE: 53 sec Clearing memory...DONE: 78 sec Loading image into memory...DONE: 53 sec Clearing memory...DONE: 80 sec Loading image into memory...DONE: 54 sec Clearing memory...DONE: 80 sec Loading image into memory...DONE: 54 sec Clearing memory...DONE: 82 sec Loading image into memory...DONE: 55 sec Clearing memory...DONE: 91 sec Loading image into memory...DONE: 56 sec Clearing memory...DONE: 84 sec Loading image into memory...DONE: 56 sec Clearing memory...DONE: 88 sec

arrays版本的testing输出在50次迭代后显示如下（显然，加载/清除时间是稳定的，并且不会越来越多地增加）：

Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 17 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 17 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 17 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 27 sec Clearing memory...DONE: 17 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 17 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 17 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 17 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 19 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 17 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 19 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec

[b]问题

当处理巨大的std :: vectors时，这个windows处理内存 *** 作的方式不好吗？

它是性病::vector，只是执行蹩脚的庞大的数据，通过devise？

我完全错过了什么吗？

是否有任何其他明显的std容器，我应该用（而不是从不同的线程在X和Y索引访问图像数据）？

任何其他好的解释和build议解决scheme？

非常大的10G缓冲区的c ++ FIFO实现

你如何处理使用36位映射的IO？

当linux中的硬盘已满时，fwrite不会返回0为什么？

编译代码使用的内存

在linux中如何在同一进程中为线程分配堆栈或内存

我做错了，我正在为图像中的每一行（数千次）调用矢量分配器。 首先将整个事物作为一个向量进行分配，然后将不同的行映射到大向量中的正确位置，即可解决问题。

感谢@PaulMcKenzIE的回答，指出我正确的方向。

总结

以上是内存溢出为你收集整理的当加载/清除大量的数据时，std :: vector会变得越来越慢全部内容，希望文章能够帮你解决当加载/清除大量的数据时，std :: vector会变得越来越慢所遇到的程序开发问题。

如果觉得内存溢出网站内容还不错，欢迎将内存溢出网站推荐给程序员好友。