Problem with adding Symbols

Hey guys , i would like to ask a question regarding some ways on how i can include certain characters into a string.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
using namespace std;
 
int main()

{


string line;
string total;
  ifstream myfile ("tr.txt");
  if (myfile.is_open())
  {
    while ( myfile.good() )
    {
      getline (myfile,line);
      total = total + line;
    }
    myfile.close();
  }




typedef boost::tokenizer< boost::char_separator<char> > tokenizer_type ;

const char* const dropped_delimiters = "',/!@#$%^&*()-_=+|]}[{:;'?/>.<,\\" ; // nothing

const char* const kept_delimiters = " \t" ; // space, tab

boost::char_separator<char> separator( dropped_delimiters, kept_delimiters ) ;

tokenizer_type toker( total, separator ) ;

int cnt = 0 ;

for( tokenizer_type::iterator beg = toker.begin() ; beg != toker.end(); ++beg )

std::cout << "token " << ++cnt << ": '" << *beg << "'\n";

}


Basically , what the code above does is to read a text file containing a text. and then the tokenizer will tokenize it based on the dropped delimeters which i specify. So for example, if my text contains a sentence such as "Hey how , is your ! day?" , the tokenizer will seperate each word from each other thus individual tokens such as "Hey" , "how" will be formed.
Those caracters should need a \ before them : http://msdn.microsoft.com/en-us/library/h21280bw%28v=vs.80%29.aspx
For the others, if they are in the first 128 ASCII characters, you should not have any problems. If not, you might have encoding related problems to solve too
Topic archived. No new replies allowed.