std::thread, some quick, some "long"

[hansaaa]

Hi,

I have some code, that is multithreaded, and, it has to do with SQL, and, file parsing.

I have a method, filestodb()

that parses, some files, and, then, writes their data, to a db.

I use this method "multi-threaded-ly", in the "main" function: here is _tmain:

	int _tmain(int argc, _TCHAR* argv[])
{

	struct tm newtime;
	__time32_t aclock;

	char buffer[32];
         errno_t errNum;
         _time32( &aclock );   // Get time in seconds.
         _localtime32_s( &newtime, &aclock );   // Convert time to struct tm form.

         // Print local time as a string.

         errNum = asctime_s(buffer, 32, &newtime);
         if (errNum)
         {
              printf("Error code: %d", (int)errNum);
         }
         printf( "Current date and time: %s", buffer );

         _TCHAR* tcthrea = argv[1];
         std::wstring a = std::wstring(tcthrea);
         std::string aaa = std::string(a.begin(), a.end());
         int its = atoi(aaa.c_str());

         SQLHANDLESTR *h = SqlConnectDBP();



	HANDLE hFind = INVALID_HANDLE_VALUE;
	WIN32_FIND_DATA *ffd = new WIN32_FIND_DATA;
	std::wstring szwDir = L"E:\\rental-pc-c\\out-xm\\*.*";

	LPCWSTR CszwDir = szwDir.c_str();
	std::vector<std::wstring> *filesP = new std::vector<std::wstring>;

	int fC = 0;
	if(hFind = FindFirstFile(CszwDir, ffd))
	{

		do{

			if (ffd->dwFileAttributes != FILE_ATTRIBUTE_DIRECTORY)
			{
				
				fC++;
				std::wstring szwDir = std::wstring(L"E:\\rental-pc-c\\out-xm\\").c_str();
				const WCHAR *wcDir = szwDir.c_str();
				WCHAR *wcFileName = ffd->cFileName;

				std::wstring szwFile(wcDir);
				szwFile.append(wcFileName);
				std::wstring szFile = std::wstring(szwFile.begin(), szwFile.end());
				filesP->push_back(szFile);

			}
			

		}while(FindNextFile(hFind, ffd));
	}


	for(int i = 0; i < filesP->size(); /*i++*/)
	{
		
		std::vector<std::thread*> ts;
		
		int j = 0;
		for (j = 0; j <its && i+j < filesP->size(); j++)
		{
			std::wcout << "parsing file " << i+j << std::endl;
			std::wcout << "starting thread " << i+j << " file: " << filesP->at(i+j) << std::endl;
			std::thread *thread = new std::thread(filetodb, filesP->at(i+j), h,j);
			ts.push_back(thread);
		}
		
		for(j = 0; j < its && i+j < filesP->size(); j++)
		{
			
			ts.at(j)->join();
			delete ts.at(j);
		}
		i = i+j;
	}
	delete filesP;

	SqlDisconnectDBP(h);

	return 0;
}

if the exe is "application.exe", and, I call, this app, with:

application.exe 10,

it does, create ten threads, and iterates, through these threads, until it has done all 88 files, which I have, in this folder.

This is a high level question, without, regarding the code, in filetodb.

The code in filetodb is, basically, the same, for all files.
The files are also, very similar: i.e., they are "xml-files", which are all very similar, (also, in size), except, for the contents, in their brackets.

So, theoretically, each thread (i.e., each call, to filetodb), should take, ca., the same amount o time, until it finishes.

So, I have 88 files, in the folder. I run this, with, application.exe 10.

So, it will, do this, for files 1-10, 11, 20, etc, up to, 71-80, and, then, if, the code is good, 80-88.

Each file, writes about 8000 entries, to my sql data base.

So, executing the first ten threads, takes about 1:30 min:sec.

after this, ca. 15,000 rows, are, in the db.

Then, executing thread 11 through 20, on, files, 11, through 20 takes A LOT longer (like 1:30 hours:mins, so, ca. 90 times, as long), to execute. My question is.

What gives?!?.

Assuming, my code is ok, and, each, individual file, should be treated, fairly, similarly (i.e., same conditions, so, same "if-else" statements are executed/entered), why, do the "second" "ten" threads (i.e., threads 11 through 20), take, at least, 90 times longer, to complete? Is this, that, the fuller, the data base gets, the longer it takes, for queries, to complete?

Or, is it, absolutely, something completely else?

I was hoping to experience "a gain" in speed, with multithreading, but, after the first "x" threads are completed, the next "x" threads take z times longer (e.g., z = 90).


How can this be?


Thanks, and, cheers!

C!

[helios]

Don't, do, this. It reads as, if, you, were running, out, of breath.

Sounds like your problem is I/O-bound. Your hard drive has only one arm, so adding more threads won't really improve performance. It's more likely to degrade it, since you're forcing the HDD to go back and forth between different regions as the different threads take control of it.

[hansaaa]

aha! Do you see, any way, of increasing performance? I hope, I am not, running out of breath ;)

If worst comes to worst, I will do it, iteratively :s takes a while

[helios]

Do you see, any way, of increasing performance?

Besides adding more disks, you could try loading big chunks of files to memory. You should do it in a function that can only be entered by one thread at a time, to avoid the performance penalties I mentioned earlier.
You've mentioned that you run 10 threads? That's probably too many, unless this is running on a server. Query the OS for the CPU count and start only one thread per CPU.

[hansaaa]

By the way, Helios, you do, seem, to answer, all my threads. For this, I thank thee! Even if, I, run, out, of, breath! ;)

I do hope, at some point, in this life, or, maybe, the next, I may, be able, to, buy, you, a beer!

Thanks!

This forum is actually, very, awesome!

Cheers!

[hansaaa]

Besides adding more disks, you could try loading big chunks of files to memory. You should do it in a function that can only be entered by one thread at a time, to avoid the performance penalties I mentioned earlier.
You've mentioned that you run 10 threads? That's probably too many, unless this is running on a server. Query the OS for the CPU count and start only one thread per CPU.

OK. Increasing HDs, is, unfortunately, at the moment, not an option.

I do agree that 10 threads, is probably, too many, although, apparently, the CPU load (according to windows, task monitor, is still, only, ca. %7). This was more, like, a, test run.

I have 2 CPUS. So, 2 threads would optimize the performance, as opposed to, possibly, 1, or 3?

Loading "big chuncks of files" into memory. you mean the xml files I am parsing?

Thanks!

C

[helios]

Yes, the XML files. The library will probably ask you for a stream-like object that implements read()ing, seek()ing, tell()ing. You may need to write such a stream to be able to parse directly from memory.