utf8mb4 encode/decode in c++

[aj3423]

A third-part server echoes string to my client program, the string contains both utf8 data and unicode emoji. for example:

http://i.stack.imgur.com/eQHaU.png

I googled some time and found this is called utf8mb4 encoding, which is used in SQL application.

I find some article about utf8mb4 in mysql/python/ruby/etc... but no c++. Is there any c++ library can do encoding/decoding utf8mb4?

[helios]

The reference you've read saying that UTF-8 sequences are 3 bytes or shorter is incorrect. UTF-8 sequences may be up to 6 bytes long. utf8mb4 is simply a 4 byte UTF-8 sequence:
(MSB on the left)
UTF-8: 11110aaa 10bbbbbb 10cccccc 10dddddd
UCS: 000aaabb bbbbcccc ccdddddd

Here's a blazing-fast UTF-8 decoder for sequences up to 6 bytes long:

template <typename T>
void utf8_to_string(std::basic_string<T> &dst, const unsigned char *buffer, size_t n){
	static const unsigned char utf8_lengths[] = {
		0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
		0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
		0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
		0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
		0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
		0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
		0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
		0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
		0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
		0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
		0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
		0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
		1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
		1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
		2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
		3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 5, 5, 0, 0,
		0xFF, 0x1F, 0x0F, 0x07, 0x03, 0x01
	};
	static const unsigned char bom[] = { 0xEF, 0xBB, 0xBF };
	static const unsigned char *masks = utf8_lengths + 0x100;
	if (n >= 3 && !memcmp(buffer, bom, 3)){
		buffer += 3;
		n -= 3;
	}
	boost::shared_array<T> temp(new T[n]);
	T *temp_pointer = temp.get();
	size_t writer = 0;
	for (size_t i = 0; i != n;){
		unsigned char byte = buffer[i++];
		unsigned char length = utf8_lengths[byte];
		if (length > n - i)
			break;
		unsigned wc = byte & masks[length];
		for (;length; length--){
			wc <<= 6;
			wc |= (T)(buffer[i++] & 0x3F);
		}
		temp_pointer[writer++] = wc;
	}
	dst.assign(temp_pointer, temp_pointer + writer);
}