Get Images From a Web, VB13 (Best of how to include LibcURL, POCO, Google Image Search API)

[kkhalaf]

Hi,

I apologize if this might sound noob. I am finishing a windows form application using c++. This application creates one big image out of small images. I am adding the feature were my application will go online and download random images, save them, process all to generate the big image. I am stuck at the part were a function should go online and download many images. I am trying to implement it in a way that it will go online and get many random images and save all in a local folder. I don't know from where to start and if there already exist libraries/function that does this.

I appreciate any idea/guidance/discussion and thank you.

[coder777]

See POCO:

http://pocoproject.org/

Once you understand how it works it offers an amazing wealth of functionality.

[kkhalaf]

Hello coder777,

Thank you. I downloaded POCO and went through its documentation. The README files don't mention the functionality or capabilities of POCO. It provides the installation guidelines, its platforms (Windows/Unix..etc) and what/how to include.

What part of my code this POCO shall help me with? Would it go online and retrieve any 20 random pictures for example? Do you know if there is other guide for all POCO's functions and their behavior/job that I should look at?

I am done with saving images in a file, processing the images and generating one big images. I need a way to go online and get images. I am thinking that I will be fine by typing a word in some search engine to retrieve related images and save the first 20 image results, but getting random images will be the perfect solution and a great shortcut for my case. Thanks

[coder777]

Would it go online and retrieve any 20 random pictures for example?

It will help you to accomplish the job, but doesn't do it for you.

You find the documentation here:

http://pocoproject.org/docs/

Escpecially take a look at the net library:

http://pocoproject.org/docs/Poco.Net.html

If you want to download a file you need either a HTTP request:

http://pocoproject.org/docs/Poco.Net.HTTPRequest.html

or FTP:

http://pocoproject.org/docs/Poco.Net.FTPClientSession.html

Also take a look here:

http://pocoproject.org/documentation/index.html

for examples.

[kkhalaf]

Hello coder777,

Thank you for this. I will definitely look at all these and figure it out. By the time being, I would love to share this and see if I am on the right track or/and if you or anyone is familiar with it:

With some searches I found a library called libcURL that will also help me accomplish the job. (Source: http://www.cplusplus.com/forum/general/46477/)

This guy seems like he was trying to approach similar functionality like mine but discarded the attempt. And using libcURL my sequence design shall be:
Query Google -> Get results as Images embedded with HTML -> use XML parsing using libxml to get the image URLs -> get/save

I have no idea how to implement this but if it is correct I can do figure it out. Problem with this (even if the design is correct) is that it has big amount of codes which will slow down the execution time way down (just main.c is 6200 lines!).

So what do you think about that? Sweet thanks coder777

[naraku9333]

You might want to look into the google image search API, it's deprecated as of 2011 but still works AFAIK. https://developers.google.com/image-search/v1/devguide?hl=en
The response is in JSON format so just grab a JSON lib or parse the response yourself.

The updated API is google custom search https://developers.google.com/custom-search/
and there is also a c++ client library http://google.github.io/google-api-cpp-client/latest/

[kkhalaf]

Hello naraku9333,

This sounds really helpful. However I looked at it and it seems I can't use it on Windows OP. See the installation guide and system requirements here:
http://google.github.io/google-api-cpp-client/latest/start/installation.html#requirements

Am I still on the right track?

Note for readers:
I had plenty of problems in installing and including the libCurl library for C++. numerous online searches and videos seemed to be that I am not the only one yet there is no absolute generic guide for all user. I have had and tried more than 10 guides but none worked for me, however here are the best sources:
https://www.youtube.com/watch?v=ItbpJ51VvS0 >>> this doesn't show how to include. Just how to start an application and it's very good.
http://theetrain.ca/tech/?p=151 >> this is the best guide on how to include and install libCurl. However it didn't work for me. My problem was, not finding cURL>lib>Debug that I need to include in the libraries dictionaries . I extracted from the .zip file version 7.44.0 from http://curl.haxx.se/download.html and in the lib folder there is no Debug (folder nor file). I tried to install other and older versions from same and other sources but I only get an .exe file which literally does nothing once you launch it (just pop up and close in a blink of an eye).
I reached a wall in libCurl but watching videos shows how strong this library is. I wish if I can make it work to start playing with it.

(Posted on Sep 22, but edited on Sep 25. Scroll down and read other posts for getting the solution on how to include libcURL)

------------------------------------------------------------
// include libcURL or Visual Studio
// include libcURL or Visual Studio
// include libcURL or Visual Studio
// include libcURL or Visual Studio
------------------------------------------------------------

[naraku9333]

You can use the deprecated API which should just consists of making a request and parsing the JSON response.

Here is an example request:
https://ajax.googleapis.com/ajax/services/search/images?v=1.0&q=naraku

[E0p9LyTq]

@coder777,

Thank you for the link to the POCO library!

[Wyboth]

If you are using Visual C++, cURL has a solution already in the source code you downloaded. Go to projects -> Windows -> whichever version of Visual Studio you are using -> lib. Open libcurl.sln in Visual Studio. Choose whichever configuration you want (debug or release, static or dynamic, etc) and whether you want 32-bit or 64-bit, and build it. Once it it finished building, read the compiler output in Visual Studio to see where it created your .lib file. Cut that lib file, then create a new folder somewhere outside of the source code named "curl", and create a folder inside of that one called "lib", and paste the library there. Then, go back to the source code folder, and copy the entire include folder over to the curl folder.

Now, it's time to test the library. Create a new Visual Studio project, and copy this source code into the main cpp file:

#include <stdio.h>
#include <curl/curl.h>

int main(void)
{
	CURL *curl;
	CURLcode res;

	curl = curl_easy_init();
	if (!curl) { return -1; }

	curl_easy_setopt(curl, CURLOPT_URL, "http://example.com");
	/* example.com is redirected, so we tell libcurl to follow redirection */
	curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);

	/* Perform the request, res will get the return code */
	res = curl_easy_perform(curl);
	/* Check for errors */
	if (res != CURLE_OK)
	{
		fprintf(stderr, "curl_easy_perform() failed: %s\n", curl_easy_strerror(res));
	}

	/* always cleanup */
	curl_easy_cleanup(curl);
	return 0;
}

Then, make sure the program is being built for either debug or release; whichever one you compiled cURL for. Same goes for 32 or 64 bit. Then, go to Debug -> (project name) Properties -> Configuration Properties -> C/C++ -> General. Copy the path to the include directory inside of your cURL folder to the "Additional Include Directories" area. Then hop over to Linker -> General, and do the same for the lib directory and the "Additional Library Directories" area (scroll down if you can't find it). Then go to Linker -> Input, and paste "wldap32.lib;ws2_32.lib;libcurl.lib;" into the Additional Dependencies area. Finally, if you chose to statically link libcURL, go back to C/C++ -> Preprocessor and paste "CURL_STATICLIB;" in front of everything else.

I hope this helps. I know how bad compiling some libraries from source can be, so let me know if you have any other problems with cURL. I will help you out until you get it working.

[kkhalaf]

Wyboth,

This looks perfect. Great thanks as this is going to be very helpful and your post must be published for being complete, precise and straight forward on how to include libcURL.

Here is how my attempt went:
Yes I successfully created the library using the downloaded solution within curl 7.44.0. I cut/paste the .lib to my "Desktop\NewCurlAttempt\curl\lib" as you mentioned, then you wanted me to "go back to the source code folder, and copy the entire include folder over to the curl folder" which in my case is "Desktop\NewCurlAttempt\curl". But there is no "include folder" in the path where the library got created. There are only some .pdb, .exe, .lib files and only two folders: a "lib" folder which contain many .obj files, and another "src" folder which also contain .obj files. However, I copy/paste the entire "src" folder to my "Desktop\NewCurlAttempt\curl" and completed all the rest steps, but your example didn't work and my error said "Cannot open include file: 'curl/curl.h': No such file or directory"

Another issue was, you mentioned Go to projects -> Windows -> whichever version of Visual Studio you are using -> lib. I am using VC13 but there was no VC13 in that Windows folder, however there exist VC12 and VC14 (what a sorcery!). I chose VC12 for no reason.

Final issue I noticed is that, I didn't face a step for choosing the "static" thingy. I only had chose debug/release and win32/win64 at the beginning of the library creation, and then at the building phase for approaching your test example.

Thank you naraku9333, I had no time to check it out because it is too new for me but your post helped me and this page is my richest resource now. So sweet from you as well.

[Wyboth]

kkhalaf,

I am glad that I helped you so much. I had to struggle hard building this (and other) libraries from source, so I'm glad I spared someone that same frustration. I'm thinking of making my own video tutorial for all of this, just to make it crystal clear to everyone what has to be done to compile the library. But I can answer your other questions.

First one, I didn't mean to look in the folder where the library got generated, with lib and src. I meant go back up to the top directory, curl-7.44.0 or whatever you named it; the one that you unzipped from the curl website. There should be a ton of folders there (CMake, docs, include). Copy the include folder over to NewCurlAttempt\curl, and get rid of the src folder you copied over earlier - you don't need it. The include folder contains all of the header files for the library (and header (.h) files need to be included with #include, hence why the folder is named include). Once you copy it over, make sure you put "path-to-Desktop\NewCurlAttempt\curl\include" (if that's the path) into "Additional Include Directories" under Debug -> (project name) Properties -> Configuration Properties -> C/C++ -> General in Visual C++. Try recompiling now, and see if you get any further.

Second one, you chose correctly. For some reason, they didn't put in a VC13 folder, so there should be no issue with choosing VC12 instead. I use VC12, and some libraries only put in VC10 solutions. I can still open them, because Visual Studio will automatically convert them to VC12 solutions. I imagine the same holds with VC12 to 13.

Also, it doesn't say static and dynamic when you are trying to actually build the library; instead it has about 20 options that are something like "DLL Release", "Release", "DLL OpenSSL Release", "OpenSSL Release", etc. I assume you are just using libcURL at this point, and not trying to combine it with OpenSSL or other libraries, so you want to ignore all of the options that have OpenSSL or some other library names in them. That should leave you with 4 options (I don't have Visual Studio in front of me now, so I'm doing this from memory). Your options should be something like Debug, DLL Debug, Release, and DLL Release. The debug versions will let you work with the Visual Studio debug tools (and really any debug tools) like breakpoints, so that can be helpful, but you don't want to distribute a debug version of an application, since it will be many times larger than it needs to be. You can build one debug and one release version of the library if you wish - I only built the release, for simplicity.

Next are DLL and LIB, which are the dynamic and static modes I was talking about earlier. If you choose DLL, msvc will compile a DLL containing the libcURL code. Your finished application will need to have this DLL with it in order to run. If you're like me, you don't want to send a finished application with several DLLs, because it looks messy, and you don't want users to accidentally delete it, then panic when the application stops working. So, I use static compilation, which will make a .lib file for the cURL code, which will later be fused with the C++ code you write into one clean standalone executable. All together, this should be the "Release" or the "LIB Release" option with no extra SSL libraries in the name. Once you've got the right one, and the right 32 or 64 bit architecture, build it and do what you did before. If you chose the right one from the start, great; you don't need to change anything.

Hope this helps again.

[kkhalaf]

Wyboth,

I am very sorry to say that It didn't work, but no syntax/red-underlines anymore though. The error just says "LNK1104: cannot open file 'libcurl.lib' ".


Here is my approach step by step all over again, including my personal local folder paths:

Step one: I created the Library as DLL Debug, x64 and Succeeded to this path "build\Win64\VC12\DLL Debug\libcurld.lib"

Step two: I created a New Folder called "curl" on my Desktop, it contains:
- A folder called lib contains the libcurld.lib
- All files inside "cURL-unzipped\curl-7.44.0\include\curl" (which are curl.h, curlrules.h, curlbuild.h .. etc)
Therefore, the path of this Folder is going to be "Desktop\curl\"

Step three:
- I created a new project
- pasted your example
- reconfigured it as Debug and x64
- I included this path "cURL-unzipped\curl-7.44.0\include" in the C/C++>General>Additional Include Directory
- I included this path "Desktop\curl\lib" to the Linker>General>Additional Library Directory.
- I added this sentence "wldap32.lib;ws2_32.lib;libcurl.lib;" to this part "kernel32.lib;user32.lib;gdi32.lib;winspool.lib;comdlg32.lib;advapi32.lib;shell32.lib;ole32.lib;oleaut32.lib;uuid.lib;odbc32.lib;odbccp32.lib;wldap32.lib;ws2_32.lib;libcurl.lib;%(AdditionalDependencies)", within the Linker>Input>Additional Dependencies. Now it looks exactly the same as written here in this post.



Still gets the same error: LNK1104: cannot open file 'libcurl.lib'. No syntax/Red-underlines. Just getting the error once pressing on the Build or StartWithoutDebugging or StartDebugging.

Your assumption is correct. I am not trying to combine it with OpenSSL or other libraries, at the mean time.

[Wyboth]

That linker error means exactly what it says: you told it to look for 'libcurl.lib', and it can't find it in the place you told it to look. Since you chose to compile the debug version of libcurl, it made a file called libcurld.lib (d for debug). Since I only ever use the release version, named 'libcurl.lib' (with no d), the instructions I gave are for the release version. So, to fix it, go back to Linker>Input>Additional Dependencies, and change "wldap32.lib;ws2_32.lib;libcurl.lib;" to "wldap32.lib;ws2_32.lib;libcurld.lib;". Let me know if you have other issues.

[kkhalaf]

Hi,

How come I didn't notice that.. However, I changed it, and it succeeded building! Finally. Yet there is a progress but the struggle didn't end yet, when the black-console page launches I get this pop-up system error that "libcurld.dll" is missing form my computer.

"The program can't start because libcurld.dll is missing from your computer. Try reinstalling the program to fix this problem"

I found here "http://www.sevenforums.com/performance-maintenance/328523-keep-getting-libcurl-dll-missing-message.html" that I should repair my windows?! and in order to fix I must download this Microsoft-partner software !!! "http://www.tweakbit.com/land/fix-my-pc/support?build=7fFMPd&kw=Windows10&content=win10&utm_source=sevenforums.com&utm_medium=link&utm_campaign=Sevenforums.1"

Very thank full Wyboth. I hope this will be the last and the least.

[kkhalaf]

Btw. I see it, I see the file and it's in this path "cURL-unzipped\curl-7.44.0\build\Win64\VC12\DLL Debug" which is the library we created in Step one within my previous post on Sep 25, 2015 2:08 a.m

[Wyboth]

Copy the DLL over to where the executable is located, and run it again. Since CURL's code is in that DLL, your program needs to look at that DLL in order to run the CURL code. It's saying "help, I can't find the DLL!"

[kkhalaf]

Sweet beautiful adorable wonderful thanks. I appreciate your help Wyboth, truly. It's working and going to be awesome. Hurah! Now will get back to business and develop the feature.

I will edit the title of this post to add "how to include libcURL". Below is the output of your example which I will analyze its content later and see what information/data it is providing (I need to revise some videos and articles about both html and libcURL). But just to check if it's correct or if you can give me a brief description of what is happening, this is the page source of what exactly? because http://www.iana.org/domains/example has much bigger source code when I browse for it manually.
Below:

<!doctype html>
<html>
<head>
<title>Example Domain</title>

<meta charset="utf-8" />
<meta http-equiv="Content-type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<style type="text/css">
body {
background-color: #f0f0f2;
margin: 0;
padding: 0;
font-family: "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;

}
div {
width: 600px;
margin: 5em auto;
padding: 50px;
background-color: #fff;
border-radius: 1em;
}
a:link, a:visited {
color: #38488f;
text-decoration: none;
}
@media (max-width: 700px) {
body {
background-color: #fff;
}
div {
width: auto;
margin: 0 auto;
border-radius: 0;
padding: 1em;
}
}
</style>
<script type="text/javascript" src="http://gc.kis.scr.kaspersky-labs.com/1B74BD89-2A22-4B93-B451-1C9E1052A0EC/main.js" charset="UTF-8"></script><script type="text/javascript" src="http://gc.kis.scr.kaspersky-labs.com/1B74BD89-2A22-4B93-B451-1C9E1052A0EC/main.js" charset="UTF-8"></script></head>

<body>
<div>
<h1>Example Domain</h1>
<p>This domain is established to be used for illustrative examples in documents. You may use this
domain in examples without prior coordination or asking for permission.</p>
<p><a href="http://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>
Press any key to continue . . .

[modoran]

You should set browser user agent, some websites detects that you are using libcurl and gets you different response in return. Also some websites contains javascript content that libcurl can't display (javascript is executed by the web browser engine which libcurl obviously doesn't have)

[kkhalaf]

Hello modoran, excuse my late reply. What do you mean by set browser user agent? and how can I do that?

Thanks