As I wrote a few weeks ago in my post about Gnocchi 3.1 being released, one of the new feature available in this version it the S3 driver. Today I would like to show you how easy it is to use it and store millions of metrics into the simple, durable and massively scalable object storage provided
by Amazon Web Services.

Installation

The installation of Gnocchi for this use case is not different than
the standard installation procedure described in the documentation. Simply install Gnocchi from PyPI using the following command:

$ pip install gnocchi[s3,postgresql] gnocchiclient

This will install Gnocchi with the dependencies for the S3 and PostgreSQL drivers and the command-line interface to talk with Gnocchi.

Configuring Amazon RDS

Since you need a SQL database for the indexer, the easiest way to get started is to create a database on Amazon RDS. You can create a managed PostgreSQL database instance in just a few clicks.

Once you're on the homepage of Amazon RDS, pick PostgreSQL as a
database:

You can then configure your PostgreSQL instance: I've picked a dev/test instance with the basic options available within the RDS Free Tier, but you can pick whatever you think is needed for your production use. Set a username and a password and note them for later: we'll need them to configure Gnocchi.

The next step is to configure the database in details. Just set the database name to "gnocchi" and leave the other options to their default values (I'm lazy).

After a few minutes, your instance should be created and running. Note down the endpoint. In this case, my instance is gnocchi.cywagbaxpert.us-east-1.rds.amazonaws.com.

Configuring Gnocchi for S3 access

In order to give Gnocchi an access to S3, you need to create access keys. The easiest way to create them is to go to IAM in your AWS console, pick a user with S3 access and click on the big gray button named "Create access key".

Once you do that, you'll get the access key id and secret access key. Note them down, we will need these later.

Creating gnocchi.conf

Now is time to create the gnocchi.conf file. You can place it in /etc/gnocchi if you want to deploy it system-wide, or in any other directory and add the --config-file option to each Gnocchi command..

Here are the values that you should retrieve and write in the configuration file:

• indexer.url: the PostgreSQL RDS instance endpoint and credentials (see above) to set into
• storage.s3_endpoint_url: the S3 endpoint URL – that depends on the region you want to use and they are listed here.
• storage.s3_region_name: the S3 region name matching the endpoint you picked.
• storage.s3_access_key_id and storage.s3_secret_acess_key: your AWS access key id and secret access key.

Your gnocchi.conf file should then look like that:

[indexer]
url = postgresql://gnocchi:gn0cch1rul3z@gnocchi.cywagbaxpert.us-east-1.rds.amazonaws.com:5432/gnocchi

[storage]
driver = s3
s3_endpoint_url = https://s3-eu-west-1.amazonaws.com
s3_region_name = eu-west-1
s3_access_key_id = <you access key id>
s3_secret_access_key = <your secret access key>

Once that's done, you can run gnocchi-upgrade in order to initialize Gnocchi indexer (PostgreSQL) and storage (S3):

$ gnocchi-upgrade --config-file gnocchi.conf
2017-02-07 15:35:52.491 3660 INFO gnocchi.cli [-] Upgrading indexer <gnocchi.indexer.sqlalchemy.SQLAlchemyIndexer object at 0x108221950>
2017-02-07 15:36:04.127 3660 INFO gnocchi.cli [-] Upgrading storage <gnocchi.storage.s3.S3Storage object at 0x10ca943d0>

Then you can run the API endpoint using the test endpoint gnocchi-api and specifying its default port 8041:

$ gnocchi-api --port 8041 -- --config-file gnocchi.conf
2017-02-07 15:53:06.823 6290 INFO gnocchi.rest.app [-] WSGI config used: /Users/jd/Source/gnocchi/gnocchi/rest/api-paste.ini
********************************************************************************
STARTING test server gnocchi.rest.app.build_wsgi_app
Available at http://127.0.0.1:8041/
DANGER! For testing only, do not use in production
********************************************************************************

The best way to run Gnocchi API is to use uwsgi as documented, but in this case, using the testing daemon gnocchi-api is good enough.

Finally, in another terminal, you can start the gnocchi-metricd daemon that will process metrics in background:

$ gnocchi-metricd --config-file gnocchi.conf
2017-02-07 15:52:41.416 6262 INFO gnocchi.cli [-] 0 measurements bundles across 0 metrics wait to be processed.

Once everything is running, you can use Gnocchi's client to query it and check that everything is OK. The backlog should be empty at this stage, obviously.

$ gnocchi status
+-----------------------------------------------------+-------+
| Field                                               | Value |
+-----------------------------------------------------+-------+
| storage/number of metric having measures to process | 0     |
| storage/total number of measures to process         | 0     |
+-----------------------------------------------------+-------+

Gnocchi is ready to be used!

$ # Create a generic resource "foobar" with a metric named "visitor"
$ gnocchi resource create foobar -n visitor
+-----------------------+-----------------------------------------------+
| Field                 | Value                                         |
+-----------------------+-----------------------------------------------+
| created_by_project_id |                                               |
| created_by_user_id    | admin                                         |
| creator               | admin                                         |
| ended_at              | None                                          |
| id                    | b4d568e4-7af1-5aec-ac3f-9c09fa3685a9          |
| metrics               | visitor: 05f45876-1a69-4a64-8575-03eea5b79407 |
| original_resource_id  | foobar                                        |
| project_id            | None                                          |
| revision_end          | None                                          |
| revision_start        | 2017-02-07T14:54:54.417447+00:00              |
| started_at            | 2017-02-07T14:54:54.417414+00:00              |
| type                  | generic                                       |
| user_id               | None                                          |
+-----------------------+-----------------------------------------------+

# Send the number of visitor at 2 different timestamps
$ gnocchi measures add --resource-id foobar -m 2017-02-07T15:56@23 visitor
$ gnocchi measures add --resource-id foobar -m 2017-02-07T15:57@42 visitor

# Check the average number of visitor
# (the --refresh option is given to be sure the measure are processed)
$ gnocchi measures show --resource-id foobar visitor --refresh
+---------------------------+-------------+-------+
| timestamp                 | granularity | value |
+---------------------------+-------------+-------+
| 2017-02-07T15:55:00+00:00 |       300.0 |  32.5 |
+---------------------------+-------------+-------+

# Now shows the minimum number of visitor
$ gnocchi measures show --aggregation min --resource-id foobar visitor
+---------------------------+-------------+-------+
| timestamp                 | granularity | value |
+---------------------------+-------------+-------+
| 2017-02-07T15:55:00+00:00 |       300.0 |  23.0 |
+---------------------------+-------------+-------+

And voilà! You're ready to store millions of metrics and measures on your Amazon Web Services cloud platform. I hope you'll enjoy it and feel free to ask any question in the comment section or by reaching me directly!