Monday, September 14, 2015

Curing Qt UTF-8 console pain on Windows

The situation

It happens that sometimes I have to write console application in C for Windows, and having only ASCII output is not always an option: basically, I need to have a possibility to perform text output in Ukrainian which is okay when you use UTF-8. Unlike others, Windows use legacy code pages system to make console work with natural languages text in different countries with different localisations, and that's a weird thing for Linux/Mac user.

I work in Qt Creator when coding in C, and it brings another tricky thing: qt_process_stub.exe - a nasty utility which is run from the IDE, which actually runs your app. My task was to find a non-tricky solution to have a possibility to work seamlessly when debugging my C programs from Qt Creator.

Step 1: Changing the font

Most of Windows developers possibly know that there's a default font in Windows cmd.exe (which is a terminal itself) that does not support UTF-8 encoding (yeah...), so the first thing you have to do is to run cmd.exe (via Ctrl+R), click Command Prompt icon and choose Properties context menu item (pic 1.1)

Step 1.1: Enter properties

There, you have to choose Lucida Console font as it supports UTF-8 symbols (at least Cyrillic ones ;), and press OK button at the bottom of the dialog.

Step 1.2: Choosing the need font.

Now, we are capable of using UTF-8 text in console and even see something meaningful there.

Step 2: Changing the code page

By default, the code page used by cmd.exe on my system is 437 which is a DOS Latin US character set. You can easily check it by running chcp command in the terminal:


The same command is used to change current code page. Everything you need is to pass codepage number as an argument:

chcp 65001

Cool! Now we use UTF-8 code page. But the problem is that when you close the terminal and start it again, the codepage will default to 437 again.

Here we actually don't need to play with codepages, but rather use the information that Qt runs console application from its IDE via qt_process_stub.exe. And it uses default system encoding information from MS Windows System registry. This setting can be easily tweaked from command-line:

REG ADD HKCU\Console /v CodePage /t REG_DWORD /d 0xfde9

This command will fix the Qt Creator problem, but cmd.exe will still run 437 by default. If you want to change this setting globally, start regedir (if you have administrator rights, of course), and add a new setting to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Command Processor folder. Name it Autorun with value chcp 65001 >nul.

Step 3: Checking the Result

It is very easy now to check the result. Let's start minimal project in Qt Creator (Non-Qt Projects section -> Plain C Application), which main.c file will look like this:

#include <stdio.h>
#include <conio.h>

int main(void)
    printf("Привіт!\n"); // Means "Hello!" in Ukrainian
    return 0;

Warning! Calling getch() serves here not for stopping console from disappearing as it usually happens (Qt Creator stops it by default), but it helps to avoid errors of qt_process_stub.exe. The absence of some sleeping/waiting mechanism will force errors because the utility has some bugs in reading from a pipe :). And the result of running this small code will look like this:

As you can see, no weird character, only plain text.

Thanks for reading. Hope this post helps to fix your troubles.

Tuesday, September 8, 2015

Nimongo - pure Nimlang MongoDB driver


Nim logo
Lately I've spent much time coding in Nim. If you're still not acquainted with this fascinating, one of the fastest growing programming languages, please visit its official site, and get your grips on it. To be short, it's a statically typed compiled programming language that translates its source code ANSI C, which is then compiled into native executables. Such flow makes it easy to employ Nimlang on a broad range of software and hardware platforms.

Other features like, possibly, the most progressive compile-time programming features, smart type inference, multi-methods, and many others makes coding in Nim much faster than in its counterparts.

Meet nimongo

We have been adopting Nim as an experiment for quite a time at one of our commercial projects here, at ZeoAlliance. Lack of powerful ecosystem and wide community made me to start working on some tools that are crucial for the project. One of such tools (located at Github), which is fully open source, is a nimongo - pure MongoDB driver for Nimlang. Although there's standard mongo client, it is still in an unusable state, and has one minus - it uses libbson.c, and libmongo.c, which makes setting it up  a bit more painful, than if the package was native to Nim language.

Note: Nimongo implements thread-safe synchronous client, AsyncMongo is still in an active development.

Installing nimongo

In order to install nimongo, you can either, clone it from Github, or use Nim native package manager called nimble:

$ nimble install

For now, you have to specify repository URL directly. Later, when some stable state of the project is achieved, I will add it to nimble repo, so you could just type nimble install nimongo. 

Importing prerequisities

In order to start using nimongo, you have to import two of its modules - nimongo.mongo and nimongo.bson which contains BSON serialization/deserialization routines, helper templates, and converters that help integrate Bson objects into Nim source code as seamlessly as possible. Also you have to import oids which is a Nim standard library module for working with MongoDB Object IDs. Other modules like times will also be helpful if you plan to use date and time data in your application.

import oids
import times

import nimongo.bson
import nimongo.mongo

nimongo.mongo module contains objects needed to work with MongoDB: Mongo, Database, Collection, Find, and other supportive types and constants.

Establishing connection with MongoDB server

The main object to start with is a Mongo, which is a reference type that manages communication between your application and MongoDB server. You can use newMongo(...) constructor by passing host and port connection parameters, or leaving those empty in order to connect to localhost:27017, - default MongoDB server parameters:

var m: Mongo = newMongo()
let connected = m.connect()

Also you can see procedure connect() call here, which actually establishes connection to server, and returns bool result that equals to true if connection was established successfully.

Performing commands on Mongo server

If you're familiar with how MongoDB operates, you are aware that there are different groups of operations that can be also split into different levels: server-, database-, collection-level. In nimongo those are distributed into procedures bound to Mongo, Database, and Collection types respectively. For now, there is only one operation available for server level - isMaster():

let ismaster = m.isMaster()

This procedure returns true if connected MongoDB instance is master in a replica set, or false if not. Other commands will also be added, but the work is still in progress. Also, beware using such kind of commands, having in mind that such commands perform actual network communication with server over network.

Accessing Databases and Collections

As Nim has nice operator overloading implemented, accessing to databases, and collections are made via [] (indexing operator) the same way as you take values by keys from hash tables. Also the syntax becomes quite friendly to Python developers ;) It looks like that:

let db = m["db"]
let collection = m["collection"]

Here we have db and collection objects. Applying these operator does not carry any real communication with server. Database object procedures perform database-level operations, and Collection object performs actual data insertion, updates, and queries.

Dealing with BSON

MongoDB operates documents encoded in BSON format, and that's why nimongo.bson module exists. In order to perform data manipulations with nimongo you have to create such objects. They are able to be serialized into a byte stream that MongoDB server understands.

BSON document can be created using initBsonDocument() constructor:

let empty = initBsonDocument()

This constructor creates empty BSON document, and then have to be filled with fields. You can also use B template, which simplifies the way you create BSON document objects, by squashing empty document creation, and offering ability to add fields right in place. Documents are filled with fields using procedure call operator ():

let doc = B("name", "John")("surname", "Smith")("salary", 100)
let doc2 = B(
    "name", "Sam")(
    "surname", "Uncle")(
    "salary", 1000)(
    "skills", @["power", "government", "army"]
As you can see from the example, two sample documents are created. Documents are created as "field-value" pairs, and value must be of Bson type. Fortunately, Nim offers such kind of procedures as converters, so all values like "Sam", "Uncle", and 1000 are implicitly converted to Bson type objects. Also mention the last field which is a sequence. This field is converted to BSON array. Also, () operator, as you could have mentioned, returns Bson type object, so you can append fields to objects any time in future.

Whether you are doing inserts, updates, or find queries, Bson objects is a strong foundation on which queries to MongoDB are made.

Inserting data into MonoDB using nimongo

When we have some documents created, we are ready to insert() those into database collection:

if collection.insert(doc):
    echo("Document was successfully inserted.")

Here we insert previously created BSON document into collection. insert() procedure returns bool value indicating success of the operation. This example shows how to insert single document, though it is also possible to insert serveral documents at a time with a single insert query by using overloaded insert() procedure that accepts sequence of Bson objects:

collection.insert(@[doc, doc, doc])

Updating documents using nimongo

Almost the same way as you do insertion, it is possible to perform document updates. Here how you can easily update single document using update() procedure:

    selector = B("integer", 100'i32)
    updater  = B("$set", B("integer", 200))

if c.update(selector, updater) == true:
    echo("Document updated succesfully.")

In this example we set new value 200 for a field named integer, where previous field's value was 100.

Removing document using nimongo

Document removal can be performed with the remove() procedure:

if c.remove(B("string", "hello")):
    echo("Document removed successfully.")

This procedure also return bool value, which indicates success of the operation.

Querying data from MongoDB

Querying data from MongoDB is a bit more complex than inserting, updating, or removal. Finding data in MongoDB usually requires some kind of configuration depending on the performed query. That's why find() procedure does not perfrom immediate querying, but creates Find object that is configurable, and the actual query is run with one(), all(), and items() procedures of the Find object. Let's fetch one document from database:

    let fetched = collection.find(B("name", "John")).one()
except NotFound:
    echo("Document was not found in database.")

Another way to fetch objects from database is to fetch a sequence of objects:

let documents = collection.find(B("name", "John")).all()
for item in document:
    echo item

And the last way to use query results is to use iterator in order to work with fetched documents:

for item in collection.find(B("name", "John")).items():
    echo item

Also you can tweak querying process by configuring Find object with such procedures as tailableCursor(), slaveOk(), noCursorTimeout(), and others, which you can find in MongoDB documentation.


All this functionality, though does not fully implements MongoDB wire protocol, and does not expose whole power of MongoDB database, allows us to use it in small projects, or on early stages of large long-term projects.

I encourage all of those who plans writing Nim applications, and likes nimongo aproach, play with the driver, and support the project by contributing via pull requests, or filing issues, and wishlists.

In the near future, I plan to finish all BSON types support (now only widely used ones supported), add more service commands, and implement asynchronous driver based on asyncnet, and asyncdispatch Nim standard library modules.