<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>gnocchi — jd:/dev/blog</title><description>Posts tagged &quot;gnocchi&quot; on jd:/dev/blog.</description><link>https://julien.danjou.info/</link><item><title>Podcast.__init__: Gnocchi, a Time Series Database for your Metrics</title><link>https://julien.danjou.info/blog/podcast-init-gnocchi/</link><guid isPermaLink="true">https://julien.danjou.info/blog/podcast-init-gnocchi/</guid><description>A few weeks ago, Tobias Macey contacted me as he wanted to talk about Gnocchi, the time series database I&apos;ve been working on for the last few years.  It was a great opportunity to talk about the proje</description><pubDate>Tue, 11 Dec 2018 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;A few weeks ago, Tobias Macey contacted me as he wanted to talk about Gnocchi, the time series database I&apos;ve been working on for the last few years.&lt;/p&gt;
&lt;p&gt;It was a great opportunity to talk about the project, so I jumped on it! We talk about how Gnocchi came to life, how we built its architecture, the challenges we met, what kind of trade-off we made, etc.&lt;/p&gt;
&lt;p&gt;You can list to this episode &lt;a href=&quot;https://www.podcastinit.com/gnocchi-with-julien-danjou-episode-189/&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;
</content:encoded><category>gnocchi</category><category>talks</category></item><item><title>Gnocchi engine optimization</title><link>https://julien.danjou.info/blog/gnocchi-engine-optimization/</link><guid isPermaLink="true">https://julien.danjou.info/blog/gnocchi-engine-optimization/</guid><description>Software speed is relative.  After all, it is the result of a set of trade-offs made between the ease of programming and the speed of hardware. The comfort of the developer and its use of multiple abs</description><pubDate>Tue, 27 Mar 2018 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;Software speed is relative.&lt;/p&gt;
&lt;p&gt;After all, it is the result of a set of trade-offs made between the ease of programming and the speed of hardware. The comfort of the developer and its use of multiple abstraction layers has a direct impact on decreasing the cost in time (and therefore in money), while it on the other hand increases the hardware expenditure as the software is less performant. In the end, whichever between optimization and hardware that is the cheapest gets privileged.&lt;/p&gt;
&lt;p&gt;Of course, there are terrible exceptions, such as picking the wrong algorithm or including &lt;code&gt;sleep()&lt;/code&gt; calls, but the essence of it is here. Pick C to be fast, saving money on hardware and spending it on development, or pick Java to save money on development, while making rich hardware manufacturers.&lt;/p&gt;
&lt;p&gt;Last month, a co-worker at &lt;a href=&quot;https://redhat.com&quot;&gt;Red Hat&lt;/a&gt; picked Gnocchi for a test run and was disappointed by the performance he saw for his particular usage. After correctly understanding the use case scenario, I wrote a small test case that implemented this scheme and popped out my favorite code profiler. You know how I roll.&lt;/p&gt;
&lt;p&gt;The profiling result made the performance issue obvious. Along with its releases, Gnocchi evolved from a one metric at a time processing approach to a bunch of metric at a time approach – especially since Gnocchi 4 and the introduction of the &lt;em&gt;sacks&lt;/em&gt;. However, that batched approach is not yet complete in Gnocchi 4.2, and the processing engine still manipulates metrics one by one in parallel. The parallelization using processes and threads makes sure that the CPU usage is high and that the I/O latency does not impact the processing too much.&lt;/p&gt;
&lt;p&gt;Processing incoming measures can be therefore schematized as this:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-engine-4-1.png&quot; alt=&quot;gnocchi-engine-4&quot; /&gt;&lt;/p&gt;
&lt;p&gt;In the schema above, each operation in red is an I/O operation. The three branches I drew represents three metrics being processed. Obviously, if there were ten metrics, there would be ten branches, creating even more I/O operations. With the current Gnocchi 4.2 code, the number of I/O operations for processing a sack of metric can be roughly computed to &lt;code&gt;2 + (5 × M × D × G)&lt;/code&gt; actions, with &lt;code&gt;M&lt;/code&gt; the number of metrics and &lt;code&gt;D&lt;/code&gt; the number of definitions in the archive policy and &lt;code&gt;G&lt;/code&gt; the number of aggregation methods. For my test scenario, I used &lt;code&gt;D=1&lt;/code&gt; and &lt;code&gt;G=1&lt;/code&gt;, which is what can be seen on the diagram above.&lt;/p&gt;
&lt;p&gt;The obvious solution is to merge those I/O operations for each metric in a single I/O operations for a bunch of metrics. This allows for storage backends to batch the reading and writing operations, reducing latency and improving throughput.&lt;/p&gt;
&lt;p&gt;It took me a few tens of patches and a few code reviews from my peers to rework the internal storage engine of Gnocchi. The new engine is now ready to be used and merged into the &lt;em&gt;master&lt;/em&gt; branch.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-engine-5.png&quot; alt=&quot;gnocchi-engine-5&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The new engine reduces the number of I/O operations to process a bunch of metric to &lt;code&gt;5 + M&lt;/code&gt; – a (at least) five times reduction in the amount of operations. In my case, for 1000 metrics being processed in a batch, with only one aggregation, that decreases the quantity of transactions from 5002 to 1005.&lt;/p&gt;
&lt;p&gt;A typical metric has 6 aggregation methods defined usually, so that could reduce the number of I/O operations from 40,002 to only 1005 for 1000 metrics – a fourty times reduction of I/O operations. The benchmark code that I wrote, which implements the desired use case with a single aggregation, is now performing more than four times faster.&lt;/p&gt;
&lt;p&gt;Not all the drivers will benefit from this improvement, as some of them are better at doing batched operations than others; Redis is great at it, while Swift is not. And even if the number I/O operations has been largely reduced, they still need to be fully executed, which can take time depending on the backend performance. It&apos;s a really great improvement, not a silver bullet.&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;http://sileht.net&quot;&gt;Mehdi&lt;/a&gt; started to use that new internal driver API to implement a &lt;a href=&quot;http://rocksdb.org/&quot;&gt;RocksDB&lt;/a&gt; driver. While it has its own limitation (has to be single-threaded) that we will need to circumvent, it could improve performance for the non-distributed use-case by a large magnitude.&lt;/p&gt;
&lt;p&gt;This code will be included in the next Gnocchi major release in a few weeks, so stay tuned for further update. And benchmark, I hope!&lt;/p&gt;
</content:encoded><category>gnocchi</category></item><item><title>Gnocchi 4.2 release</title><link>https://julien.danjou.info/blog/gnocchi-4-2-release/</link><guid isPermaLink="true">https://julien.danjou.info/blog/gnocchi-4-2-release/</guid><description>The time of the release arrived. A little more than three months have passed since the latest minor version, 4.1, has been released. There are tons of improvement and a few nice significant features i</description><pubDate>Tue, 06 Feb 2018 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;The time of the release arrived. A little more than three months have passed since the latest minor version, 4.1, has been released. There are tons of improvement and a few nice significant features in this release!&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-logo.png&quot; alt=&quot;Gnocchi logo&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Most of the principal changes are recorded in &lt;a href=&quot;http://gnocchi.xyz/releasenotes/4.2.html&quot;&gt;the 4.2 release notes&lt;/a&gt;, but here are as a few that I find particularly interesting. There were 141 commits since 4.1.0 that we merged. As a comparison, it is a lot less than the 228 we had between 4.0.0 and 4.1.0 or the 375 we had between 3.1.0 and 4.0.0!&lt;/p&gt;
&lt;p&gt;We added two compatibility endpoints on the REST API for &lt;a href=&quot;http://influxdb.org&quot;&gt;InfluxDB&lt;/a&gt; and &lt;a href=&quot;http://prometheus.io&quot;&gt;Prometheus&lt;/a&gt;. We want users coming from those other database systems or using tools that are compatible with them to be able also to use Gnocchi. This is now possible as Gnocchi offers endpoint to write data using the InfluxDB line protocol and Prometheus HTTP API. Reading data using their API is not supported yet though. For example, this has been tested with &lt;a href=&quot;https://www.influxdata.com/time-series-platform/telegraf/&quot;&gt;Telegraf&lt;/a&gt; and works perfectly fine!&lt;/p&gt;
&lt;p&gt;Some other improvements were made, such as enhancing the ACL filtering when using &lt;a href=&quot;https://docs.openstack.org/keystone/latest/&quot;&gt;Keystone&lt;/a&gt; for authentication, a new batch format for passing more information about non-existing metrics to create and tons of performance improvements!&lt;/p&gt;
&lt;p&gt;We already started working on the next version of Gnocchi! Come and join us on &lt;a href=&quot;http://github.com/gnocchixyz&quot;&gt;GitHub&lt;/a&gt;! Star us, and stay tuned for some more awesome news around metrics.&lt;/p&gt;
</content:encoded><category>gnocchi</category></item><item><title>Gnocchi 4.1 is out</title><link>https://julien.danjou.info/blog/gnocchi-4-1-release/</link><guid isPermaLink="true">https://julien.danjou.info/blog/gnocchi-4-1-release/</guid><description>We did it again. A bit more of our usual four months were needed to do it, but Gnocchi 4.1 has been released. This is a great news and another big milestone for the project!  As usual, we enhanced Gno</description><pubDate>Fri, 27 Oct 2017 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;We did it again. A bit more of our usual four months were needed to do it, but Gnocchi 4.1 has been released. This is a great news and another big milestone for the project!&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-logo.png&quot; alt=&quot;Gnocchi logo&quot; /&gt;&lt;/p&gt;
&lt;p&gt;As usual, we enhanced Gnocchi and added a bunch of new things that &lt;a href=&quot;http://gnocchi.xyz/releasenotes/4.1.html&quot;&gt;can all be seen in the online changelog&lt;/a&gt;. Nevertheless, I would like to talk of a few here!&lt;/p&gt;
&lt;p&gt;First, we added notification support to the Redis incoming driver. This feature makes sure that, when using Redis as an incoming measure driver, the metrics as processed as fast as possible, rather than waiting &lt;code&gt;metric_processing_delay&lt;/code&gt;. This moves the incoming driver toward more of a push model than a pull model – even if it still uses both. That feature decreases the latency between the time metrics are pushed, and metrics are processed by &lt;em&gt;metricd&lt;/em&gt;, which is transcendent.&lt;/p&gt;
&lt;p&gt;Secondly, the internal computing engine (measures aggregation) has entirely been ported from &lt;a href=&quot;http://pandas.pydata.org&quot;&gt;Pandas&lt;/a&gt; to &lt;a href=&quot;http://numpy.org&quot;&gt;Numpy&lt;/a&gt;. While Pandas is written using Numpy, it does some things more than Numpy itself when used. Those features are beneficial when quickly writing data analysis processes, but are not needed for Gnocchi. They take CPU time, which means less throughput for &lt;em&gt;metricd&lt;/em&gt;. Pandas is still needed for the old and deprecated dynamic aggregation feature and will be entirely removed as a dependency in the next version of Gnocchi.&lt;/p&gt;
&lt;p&gt;Finally, the biggest functionality that has landed is &lt;a href=&quot;http://gnocchi.xyz/rest.html#aggregates-on-the-fly-measurements-modification-and-aggregation&quot;&gt;the new &lt;code&gt;/v1/aggregates&lt;/code&gt; endpoint&lt;/a&gt;. This is a principal feature that allows to retrieve aggregates but also to do cross-aggregation in new ways that were not possible before. For example, you can request &quot;the absolute value of the average of two metrics being multiplied by&quot; writing: &lt;code&gt;(absolute (* (metric 32dd0731-c423-45aa-94f6-e4069989eb57 mean) (metric 942990de-b208-4bf7-a0ee-93e4890df73a mean)))&lt;/code&gt;. This endpoint supports fetching any metric from the database (by id or by search in resources) and applying any mathematics operation. The syntax is inspired from Lisp, which makes it easy to write both as a string or as JSON.&lt;/p&gt;
&lt;p&gt;Come and join us on &lt;a href=&quot;http://github.com/gnocchixyz&quot;&gt;GitHub&lt;/a&gt;! Star us, and stay tuned for some more awesome news around metrics.&lt;/p&gt;
</content:encoded><category>gnocchi</category></item><item><title>My interview with Cool Python Codes</title><link>https://julien.danjou.info/blog/interview-coolpythoncodes/</link><guid isPermaLink="true">https://julien.danjou.info/blog/interview-coolpythoncodes/</guid><description>A few days ago, I&apos;ve recently been contacted by Godson Rapture from Cool Python codes to answer a few questions about what I work on in open source.</description><pubDate>Thu, 05 Oct 2017 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;A few days ago, I&apos;ve recently been contacted by Godson Rapture from &lt;a href=&quot;http://coolpythoncodes.com/&quot;&gt;Cool Python codes&lt;/a&gt; to answer a few questions about what I work on in open source. Godson regularly interview developers and I invite you to check out his website!&lt;/p&gt;
&lt;p&gt;Here&apos;s a copy of &lt;a href=&quot;http://coolpythoncodes.com/julien-danjou/&quot;&gt;my original interview&lt;/a&gt;. Enjoy!&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Good day, Julien Danjou, welcome to Cool Python Codes. Thanks for taking your precious time to be here.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;You’re welcome!&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Could you kindly tell us about yourself like your full name, hobbies, nationality, education, and experience in programming?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Sure. I’m Julien Danjou, I’m French and live in Paris, France. I studied Computer science for 5 years around 15 years ago, and continued my career in that field since then, specializing in open source projects.&lt;/p&gt;
&lt;p&gt;Those last years, I’ve been working as a software engineer at Red Hat. I’ve spent the last 10 years working with the Python programming language. Now I work on the Gnocchi project which is a time series database.&lt;/p&gt;
&lt;p&gt;When I’m not coding, I enjoy running half-marathon and playing FPS games.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/pyconfr-2017-jd.jpg&quot; alt=&quot;Julien Danjou at PyCon France 2017&quot; /&gt;&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Can you narrate your first programming experience and what got you to start learning to program?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;I started programming around 2001, and my first serious programs were in Perl. I was contributing to a hosting platform for free software named VHFFS. It was a free software project itself, and I enjoyed being able to learn from other more experienced developers and being able to contribute back to it. That’s what got me stuck into that world of open source projects.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Which programming language do you know and which is your favorite?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;I know quite a few, I’ve been doing serious programming in Perl, C, Lua, Common Lisp, Emacs Lisp and Python.&lt;/p&gt;
&lt;p&gt;Obviously, my favorite is Common Lisp, but I was never able to use it for any serious project, for various reasons. So I spend most of my time hacking with Python, which I really enjoy as it is close to Lisp, in some ways. I see it as a small subset of Lisp.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;What inspired you to venture into the world of programming and drove you to learn a handful of programming languages?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;It was mostly scratching my own itches when I started. Each time I saw something I wanted to do or a feature I wanted in an existing software, I learned what I needed to get going and get it working.&lt;/p&gt;
&lt;p&gt;I studied C and Lua while writing awesome- the window manager that I created 10 years ago and used for a while. I learned Emacs Lisp while writing extensions that I wanted to see in Emacs, etc. It’s the best way to start.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;What is your blog about?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;My blog is a platform where I write about what I work on most of the time. Nowadays, it’s mostly about Python and the main project I contribute to,&lt;br /&gt;
Gnocchi.&lt;/p&gt;
&lt;p&gt;When writing about Gnocchi, I usually try to explain what part of the project I worked on, what new features we achieved, etc.&lt;/p&gt;
&lt;p&gt;On Python, I try to share solutions to common problems I encountered or identified while doing e.g. code reviews. Or presenting a new library I created!&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Tell us more about your book, The Hacker’s Guide to Python.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;It’s a compilation of everything I learned those last years building large Python applications. I spent the last 6 years developing on a large code base with thousands of other developers.&lt;/p&gt;
&lt;p&gt;I’ve reviewed tons of code and identified the biggest issues, mistakes, and bad practice that developers tend to have. I decided to compile that in a guide, helping developers that played a bit with Python to learn the stages to get really productive with Python.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;OpenStack is the biggest open source project in Python, Can you tell us more about OpenStack?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;OpenStack is a cloud computing platform, started 7 years ago now. Its goal is to provide a programmatic platform to manage your infrastructure while being open source and avoiding vendor lock-in.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Who uses OpenStack? Is it for programmers, website owners?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;It’s used by a lot of different organizations – not really by individuals. It’s a big piece of software. You can find it in some famous public cloud providers (Dreamhost, Rackspace…), and also as a private cloud in a lot of different organizations, from Bloomberg to eBay or the CERN in Switzerland, a big OpenStack user. Tons of telecom providers also leverages OpenStack for their own internal infrastructure.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Have you participated in any OpenStack conference? What did you speak on if&lt;br /&gt;
you did?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;I’ve attended the last 9 OpenStack summits and a few other OpenStack events around the world. I’ve been engaged in the upstream community for the last 6 years now.&lt;/p&gt;
&lt;p&gt;My area of expertise is telemetry, the stack of software that is in charge of collecting and storing metrics from the various OpenStack components. This is what I regularly talk about during those events.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;How can one join the OpenStack community?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;There’s an entire documentation about that, called the &lt;a href=&quot;https://docs.openstack.org/infra/manual/developers.html&quot;&gt;Developer’s Guide&lt;/a&gt;. It explains how to setup your environment to send patches, how to join the community using the mailing-lists or IRC.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;What makes your book, &lt;a href=&quot;https://thehackerguidetopython.com&quot;&gt;The Hacker’s Guide to Python&lt;/a&gt; stand out from other Python books? Also, who exactly did you write this book for?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;I wrote the book that I always wanted to read about Python, but never found. It’s not a book for people that want to learn Python from scratch. It’s a great guide for those who know the language but don’t know the details that experienced developers know and that make the difference. The best practice, the elegant solutions to common problems, etc. That’s why it also includes interviews with prominent Python developers, so they can share their advice on different areas.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;How can someone get your book?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;I’ve decided to self-publish my book, so he does not have an editor like you can be used to see. The best place to get it is online at where you can pick the format you want, electronic or paper.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;What do you mean when you say you hack with Python?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Unfortunately, most people refer to hacking as the activity of some bad guys trying to get access to whatever they’re not supposed to see. In the book title, I mean “hacking” as the elegant way of writing code and making things worse smoothly even when you were not expecting to make it.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;You mentioned earlier that Gnocchi is a time series database. Can you please be more elaborate about Gnocchi? Is there also any documentation about Gnocchi?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;So Gnocchi is a project I started a few years ago to store time series at large scale. Timeseries are basically a series of tuple composed of a timestamp and a value.&lt;/p&gt;
&lt;p&gt;Imagine you wanted to store the temperature of all the rooms of the world at any point of time. You’d need a dedicated database for that with the right data structure. This is what Gnocchi does: it provides this data structure storage at very, very large scale.&lt;/p&gt;
&lt;p&gt;The primary use case is infrastructure monitoring, so most people use it to store tons of metrics about their hardware, software, etc. It’s fully documented on &lt;a href=&quot;http://gnocchi.xyz&quot;&gt;its website&lt;/a&gt;.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;How can a programmer without much experience contribute to open source projects?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;The best way to start is to try to fix something that irritates you in some way. It might be a bug, it might be a missing feature. Start small. Don’t try big things first or you could be discouraged.&lt;/p&gt;
&lt;p&gt;Never stop.&lt;/p&gt;
&lt;p&gt;Also, don’t plunge right away in the community and start poking random people or spam them with questions. Do your homework, and listen to the community for a while to get a sense of how things are going. That can be joining IRC and lurking or following the mailing lists for example.&lt;/p&gt;
&lt;p&gt;Big open source communities dedicate programs to help you become engaged. It might be worth a try. Generic programs like Outreachy or Google Summer of Code are a great way to start if you don’t feel confident enough to jump by your own means in a community.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Just out of curiosity, do you write code in French?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Never ever. I think it’s acceptable to write in your language if you are sure that your code will never be open sourced and that your whole team is talking in that language, no matter what – but it’s a ballsy assumption, clearly.&lt;/p&gt;
&lt;p&gt;Truth is that if you do open source, English is the standard, so go with it. Be sad if you want, but please be pragmatic.&lt;/p&gt;
&lt;p&gt;I’ve seen projects being open sourced by companies where all the code source comments were in Korean. It was impossible for any non-Korean people to get a glance of what the code and the project was doing, so it just failed and disappeared.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;How does a team of programmers handle bugs in a large open source project?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;I wish there was some magic recipe, but I don’t think it’s the case. What you want is to have a place where your users can feel safe reporting bugs. Include a template so they don’t forget any details: how to reproduce the bugs, what they expected, etc. The worst thing is to have users reporting “That does not work.” with no details. It’s a waste of time.&lt;/p&gt;
&lt;p&gt;What tool to use to log all of that really depends on the team size and culture.&lt;/p&gt;
&lt;p&gt;Once that works, the actual fixing of bug doesn’t follow any rule. Most developers fix the bug they encounter or the ones that are the most critical for users. Smaller problems might not be fixed for a long time.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Can you tell us about the new book you are working on and when do we expect&lt;br /&gt;
to get it?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;That new book is entitled &lt;a href=&quot;https://scaling-python.com&quot;&gt;“Scaling Python”&lt;/a&gt; and it provides insight into how to build largely scalable and distributed applications using Python.&lt;/p&gt;
&lt;p&gt;It is also based on my experience in building this kind of software during the past years. This book also includes interviews of great Python hackers who work on scalable system or know a thing or two about writing applications for performance – an important point to have scalable applications.&lt;/p&gt;
&lt;p&gt;The book is in its final stage now, and it should be out at the beginning of 2018.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;How can someone get in contact with you?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;I’m reachable at &lt;a href=&quot;mailto:julien@danjou.info&quot;&gt;julien@danjou.info&lt;/a&gt; by email or via Twitter, &lt;a href=&quot;https://twitter.com/juldanjou&quot;&gt;@juldanjou&lt;/a&gt;.&lt;/p&gt;
</content:encoded><category>career</category><category>python</category><category>books</category><category>gnocchi</category><category>openstack</category></item><item><title>Gnocchi 4 performance</title><link>https://julien.danjou.info/blog/gnocchi-4-performance/</link><guid isPermaLink="true">https://julien.danjou.info/blog/gnocchi-4-performance/</guid><description>It has been a long time since I have tested Gnocchi performances. Last time was two years ago, on version 2. The current version for Gnocchi is 4.0, released a couple of months ago.</description><pubDate>Mon, 11 Sep 2017 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;It has been a long time since I have tested &lt;a href=&quot;http://gnocchi.xyz&quot;&gt;Gnocchi&lt;/a&gt; performances. &lt;a href=&quot;https://julien.danjou.info/blog/2015/gnocchi-benchmarks&quot;&gt;Last time was two years ago, on version 2&lt;/a&gt;. The current version for Gnocchi is 4.0, &lt;a href=&quot;https://julien.danjou.info/blog/2017/gnocchi-4-release&quot;&gt;released a couple of months ago&lt;/a&gt;. It adds a lot of new features, such as a &lt;a href=&quot;http://redis.org&quot;&gt;Redis&lt;/a&gt; incoming driver and a new job distribution method.&lt;/p&gt;
&lt;p&gt;Many of those features and improvement implemented over the last couple of years were made with performance in mind. It is time to check if this lives up to our expectation.&lt;/p&gt;
&lt;h3&gt;Test protocol&lt;/h3&gt;
&lt;p&gt;I have pulled the servers I used a couple of years ago out of the dust, updated them with latest RHEL 7 and installed Gnocchi 4.0.1 and Redis 4.0.1 on one of them. I used the other server as the benchmark client, in charge of generating a bunch of loads.&lt;/p&gt;
&lt;p&gt;The hardware configuration for each server is:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;2 × Intel(R) Xeon(R) CPU E5-2609 v3 @ 1.90GHz (6 cores each)&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;32 GB RAM&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;SanDisk Extreme Pro 240GB SSD&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I have installed Gnocchi using &lt;code&gt;pip install gnocchi[postgresql,file,redis]&lt;/code&gt;, created a PostgreSQL database and wrote the following configuration file:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;[indexer]
url = postgresql://root:@localhost/gnocchi

## Uncomment when testing with Redis
## [incoming]
## driver = redis

[storage]
file_basepath = /root/gnocchi-venv/data
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The perk of having good default values: you only to write a couple of configuration lines to get it working.&lt;/p&gt;
&lt;p&gt;I have used uWSGI as the Web server, using the configuration&lt;br /&gt;
file &lt;a href=&quot;http://gnocchi.xyz/operating.html#running-api-as-a-wsgi-application&quot;&gt;provided Gnocchi&apos;s documentation&lt;/a&gt; and configured it with 64 processes and 16 threads.&lt;/p&gt;
&lt;p&gt;Since the hardware configurations are identical, I allow myself in this article to compare the performances of Gnocchi 2 and Gnocchi 4 directly.&lt;/p&gt;
&lt;h3&gt;Benchmark tools&lt;/h3&gt;
&lt;p&gt;For generating loads, I have reused the code that I wrote and merged&lt;br /&gt;
in &lt;a href=&quot;http://pypi.python.org/gnocchiclient&quot;&gt;python-gnocchiclient&lt;/a&gt;. It is still not that easy to generate a lot of parallel loads in Python, though it is still the best tool I find available that was not too complicated to setup for things like CRUD operations.&lt;/p&gt;
&lt;p&gt;To benchmark measures, I needed something very fast to generate requests on the client side to be sure to be able to overload the server. I have leveraged &lt;a href=&quot;https://github.com/wg/wrk&quot;&gt;wrk&lt;/a&gt;, which is written in C++ and is fast. It is scriptable using Lua, so it made it easy to generate fake batches of data.&lt;/p&gt;
&lt;h3&gt;Metric CRUD operations&lt;/h3&gt;
&lt;p&gt;The first step is to benchmarks the CRUD operations for metrics. Here are the&lt;br /&gt;
results, compared to the benchmarks I did against Gnocchi 2.&lt;/p&gt;
&lt;p&gt;Without surprises (but with great pleasure), everything is between 13% and 26% faster. Those operations mostly consist of SQL operations for the backend and serialization on the API – nothing heavy.&lt;/p&gt;
&lt;h3&gt;Sending and getting measures&lt;/h3&gt;
&lt;p&gt;Writing measures is still the hottest topic! How fast can you push things into that time series database and how efficient it is at retrieving those?&lt;/p&gt;
&lt;p&gt;Gnocchi has been supporting various batching methods for a while, and here the tested one is the simplest case, i.e., batching for one metric at a time.&lt;/p&gt;
&lt;p&gt;I think the chart talks for itself. With Redis as a driver, I attained almost &lt;strong&gt;1 million measures per second&lt;/strong&gt;. I did not find a suitable tool to report performances with a payload bigger than 5000 points, so I stopped at that. Those results are inline with what &lt;a href=&quot;https://medium.com/@gord.chung/gnocchi-4-introspective-a83055e99776&quot;&gt;Gordon Chung measured recently on Gnocchi 4&lt;/a&gt; – though he achieved &lt;strong&gt;1.3 million measures per second&lt;/strong&gt; with his bigger hardware!&lt;/p&gt;
&lt;p&gt;These are performances using HTTP as a protocol – with all its overhead and JSON serialization going on. Gnocchi does not implement any custom protocol so far because we never had any requirement for more performances. However, that would certainly be a good path to follow for anyone wanting to go even faster.&lt;/p&gt;
&lt;p&gt;Reading metrics is 54% faster here again. You can retrieve up to 400 000 measures per second (around 150 Mbit/s of data). That means you can retrieve a metric with a whole year of measures with a one-minute aggregate in 1.3 seconds. More realistically, you can retrieve the last 24 hours of data with a one minute precision for 280 metrics in just one second. That is more data you could ever fit on your graph dashboard!&lt;/p&gt;
&lt;p&gt;Most of the time is spent serializing points in JSON – again, a different retrieving mechanism could be envisioned to achieve even higher performances.&lt;/p&gt;
&lt;h3&gt;&lt;em&gt;Metricd&lt;/em&gt; speed&lt;/h3&gt;
&lt;p&gt;I did not benchmark myself metricd speed, as &lt;a href=&quot;https://medium.com/@gord.chung/gnocchi-4-introspective-a83055e99776&quot;&gt;Gordon wrote a complete report in the meantime&lt;/a&gt;. Gnocchi 4 multiplies the processing speed from Gnocchi 2 by a factor of 2.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-metricd-speed.png&quot; alt=&quot;Chart showing Gnocchi metricd processing speed improvements from version 2 to 4&quot; /&gt;&lt;/p&gt;
&lt;p&gt;This speed is quite impressive and allows Gnocchi to ingest and pre-compute considerable amount of data in a short time span. Some of the changes Gordon tested here are not yet released and will be part of the next minor release (4.1).&lt;/p&gt;
&lt;p&gt;Being that efficient means that with only 1 CPU, Gnocchi can process (data aggregation) roughly 700 measures per second. If you have 70 servers and gather 10 metrics per server every second, Gnocchi can process them without any delay.&lt;/p&gt;
&lt;p&gt;If you scale back your polling to one minute instead of one second (the most common scenario) and use a single computer with 12 cores, that means Gnocchi can &lt;strong&gt;aggregate the metrics from 50 400 servers with only one server&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;Not that bad.&lt;/p&gt;
&lt;h3&gt;Conclusion&lt;/h3&gt;
&lt;p&gt;Our processing engine is getting now really mature. Hundreds of deployments are now using it for production purpose of gathering metrics. The recent improvements made for Gnocchi 4 are a compelling argument for users to upgrade, and we are pretty proud of our work! We still have a few ideas on how to improve some corner cases, but the general use case is getting well covered. Adding to that the native horizontal capability that Gnocchi provides since day one, it is getting hard to find a time series database that has those features with this level of performance (but of course I&apos;m biased, haha).&lt;/p&gt;
&lt;p&gt;And if you have any questions, feel free to shoot them in the comment section. 😉&lt;/p&gt;
&lt;p&gt;$(function () { var chartColors = { red: &apos;rgb(234, 18, 51)&apos;, orange: &apos;rgb(255, 159, 64)&apos;, yellow: &apos;rgb(255, 205, 86)&apos;, green: &apos;rgb(75, 192, 192)&apos;, blue: &apos;rgb(46, 106, 234)&apos;, purple: &apos;rgb(153, 102, 255)&apos;, grey: &apos;rgb(201, 203, 207)&apos; }; var color = Chart.helpers.color; var ctx = $(&quot;#metric_crud&quot;).get(0).getContext(&quot;2d&quot;); var metric_crud = new Chart(ctx, { type: &apos;bar&apos;, data: { labels: [&quot;Create&quot;, &quot;Read&quot;, &quot;Delete&quot;], datasets: [ { label: &quot;Gnocchi 2&quot;, backgroundColor: color(chartColors.blue).alpha(0.6).rgbString(), borderColor: chartColors.blue, borderWidth: 1, data: [1300, 670, 524] }, { label: &quot;Gnocchi 4&quot;, backgroundColor: color(chartColors.red).alpha(0.6).rgbString(), borderColor: chartColors.red, borderWidth: 1, data: [1473, 843, 708] } ] }, options: { responsive: true, legend: { position: &apos;top&apos;, }, scales: { yAxes: [ { ticks: { beginAtZero: true } } ] }, title: { display: true, text: &apos;CRUD operations&apos; } }}); var ctx = $(&quot;#metric_measures&quot;).get(0).getContext(&quot;2d&quot;); var metric_measures = new Chart(ctx, { type: &apos;bar&apos;, data: { labels: [&quot;1&quot;, &quot;10&quot;, &quot;100&quot;, &quot;500&quot;, &quot;1000&quot;, &quot;2000&quot;, &quot;3000&quot;, &quot;4000&quot;, &quot;5000&quot;], datasets: [ { label: &quot;Gnocchi 2 (file)&quot;, backgroundColor: color(chartColors.blue).alpha(0.6).rgbString(), borderColor: chartColors.blue, borderWidth: 1, data: [624, 6000, 45000, 98000, 113000, 121000, 123000, 125000, 122000] }, { label: &quot;Gnocchi 4 (file)&quot;, backgroundColor: color(chartColors.red).alpha(0.6).rgbString(), borderColor: chartColors.red, borderWidth: 1, data: [1 * 754.26, 10 * 770.16, 100 * 583.53, 500 * 522.88, 1000 * 406.38, 2000 * 273.03, 3000 * 215.11, 4000 * 185.08, 5000 * 176.11] }, { label: &quot;Gnocchi 4 (Redis)&quot;, backgroundColor: color(chartColors.purple).alpha(0.6).rgbString(), borderColor: chartColors.purple, borderWidth: 1, data: [1 * 674, 10 * 782.34, 100 * 600, 500 * 533.66, 1000 * 405.38, 2000 * 282, 3000 * 223, 4000 * 195, 5000 * 185.41] } ] }, options: { responsive: true, legend: { position: &apos;top&apos;, }, scales: { yAxes: [ { scaleLabel: { display: true, labelString: &quot;Measures per second&quot; }, ticks: { beginAtZero: true } } ], xAxes: [ { scaleLabel: { display: true, labelString: &quot;Measures per request&quot; }, } ] }, title: { display: true, text: &apos;Measures writing&apos; } }}); var ctx = $(&quot;#metric_measures_get&quot;).get(0).getContext(&quot;2d&quot;); var metric_measures_get = new Chart(ctx, { type: &apos;bar&apos;, data: { labels: [&quot;Get measures for metric&quot;], datasets: [ { label: &quot;Gnocchi 2 (file)&quot;, backgroundColor: color(chartColors.blue).alpha(0.6).rgbString(), borderColor: chartColors.blue, borderWidth: 1, data: [260000] }, { label: &quot;Gnocchi 4 (file)&quot;, backgroundColor: color(chartColors.red).alpha(0.6).rgbString(), borderColor: chartColors.red, borderWidth: 1, data: [46.43 * 8640] } ] }, options: { responsive: true, legend: { position: &apos;top&apos;, }, scales: { yAxes: [ { scaleLabel: { display: true, labelString: &quot;Measures per second&quot; }, ticks: { beginAtZero: true } } ] }, title: { display: true, text: &apos;Measures reading&apos; } }}); });&lt;/p&gt;
</content:encoded><category>gnocchi</category></item><item><title>Gnocchi or Prometheus?</title><link>https://julien.danjou.info/blog/gnocchi-or-prometheus/</link><guid isPermaLink="true">https://julien.danjou.info/blog/gnocchi-or-prometheus/</guid><description>The realm of time series database keeps expanding those last years. Now and then a new contender appears from the fog. People keep asking me about the difference between Gnocchi and Prometheus.</description><pubDate>Wed, 30 Aug 2017 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;The realm of time series database keeps expanding those last years. Now and then a new contender appears from the fog. People keep asking me about the difference between &lt;a href=&quot;http://gnocchi.xyz&quot;&gt;Gnocchi&lt;/a&gt; and &lt;a href=&quot;http://prometheus.io&quot;&gt;Prometheus&lt;/a&gt;. It&apos;s time to content them.&lt;/p&gt;
&lt;p&gt;Gnocchi and Prometheus are two open source projects evolving in the same expertise area, time series handling. They both are licensed under the &lt;strong&gt;Apache 2.0 license&lt;/strong&gt; (see &lt;a href=&quot;https://github.com/gnocchixyz/gnocchi/blob/master/LICENSE&quot;&gt;Gnocchi license file&lt;/a&gt; and &lt;a href=&quot;https://github.com/prometheus/prometheus/blob/master/LICENSE&quot;&gt;Prometheus license file&lt;/a&gt;. And that&apos;s a good thing!&lt;/p&gt;
&lt;p&gt;Both Gnocchi and Prometheus offers a bunch of features. Here&apos;s a table summary of the differences between the features they both offer – or not.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Feature&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Prometheus&lt;/p&gt;
&lt;p&gt;Gnocchi&lt;/p&gt;
&lt;p&gt;Multi-tenant&lt;/p&gt;
&lt;p&gt;❌&lt;/p&gt;
&lt;p&gt;✓&lt;/p&gt;
&lt;p&gt;User auth &amp;amp; ACL&lt;/p&gt;
&lt;p&gt;❌&lt;/p&gt;
&lt;p&gt;✓&lt;/p&gt;
&lt;p&gt;Resource history&lt;/p&gt;
&lt;p&gt;❌&lt;/p&gt;
&lt;p&gt;✓&lt;/p&gt;
&lt;p&gt;Metric polling&lt;/p&gt;
&lt;p&gt;✓&lt;/p&gt;
&lt;p&gt;❌&lt;/p&gt;
&lt;p&gt;Highly available&lt;/p&gt;
&lt;p&gt;❌&lt;/p&gt;
&lt;p&gt;✓&lt;/p&gt;
&lt;p&gt;Horizontal scalability&lt;/p&gt;
&lt;p&gt;❌&lt;/p&gt;
&lt;p&gt;✓&lt;/p&gt;
&lt;p&gt;Alerting engine&lt;/p&gt;
&lt;p&gt;✓&lt;/p&gt;
&lt;p&gt;❌&lt;/p&gt;
&lt;p&gt;Data compression&lt;/p&gt;
&lt;p&gt;✓&lt;/p&gt;
&lt;p&gt;✓&lt;/p&gt;
&lt;p&gt;Pre-computed aggregation&lt;/p&gt;
&lt;p&gt;✓&lt;/p&gt;
&lt;p&gt;✓&lt;/p&gt;
&lt;p&gt;Grafana support&lt;/p&gt;
&lt;p&gt;✓&lt;/p&gt;
&lt;p&gt;✓&lt;/p&gt;
&lt;p&gt;collectd support&lt;/p&gt;
&lt;p&gt;✓&lt;/p&gt;
&lt;p&gt;✓&lt;/p&gt;
&lt;p&gt;#comparison th, #comparison td + td { text-align: center; }&lt;/p&gt;
&lt;p&gt;There&apos;s a lot of overlap between the two projects, but there are also some major differences.&lt;/p&gt;
&lt;p&gt;First, Gnocchi does not try to solve the metric retrieval problem. Prometheus provides a pull mechanism and takes in charge of getting the measurements. Gnocchi developers estimate that they are plenty of tools already doing that and that work well, such as &lt;a href=&quot;http://collectd.org&quot;&gt;collectd&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/icon_siren.png&quot; alt=&quot;Alert siren icon&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Secondly, Prometheus offers an &lt;a href=&quot;https://prometheus.io/docs/alerting/overview/&quot;&gt;alerting engine&lt;/a&gt;, statically configured with&lt;br /&gt;
a YAML file. It is way better than Gnocchi which offers nothing in comparison – for now. Gnocchi developers &lt;a href=&quot;https://github.com/gnocchixyz/gnocchi/issues/71&quot;&gt;are discussing the feature&lt;/a&gt; and while it&apos;s not on the roadmap yet, it will happen. It will, however, leverage a REST API to be controlled, as it seems important to us to be able to define alerts&lt;br /&gt;
programmatically.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/icon_storage.png&quot; alt=&quot;Storage icon&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Then there is a bunch of features where Gnocchi shines compared to Prometheus, and it is the core of its function: storing metrics. Gnocchi has a great storage engine that supports many storage backends (plain&lt;br /&gt;
files, &lt;a href=&quot;https://docs.openstack.org/swift/latest/&quot;&gt;OpenStack Swift&lt;/a&gt;, &lt;a href=&quot;http://ceph.org&quot;&gt;Ceph&lt;/a&gt;…). It helps Gnocchi scaling horizontally and providing native high-availability, whereas Prometheus stays a single point of failure.&lt;/p&gt;
&lt;p&gt;Multi-tenant and authentication are also supported by Gnocchi, allowing a single instance to be shared by multiple accounts. System administrators do not commonly use this kind of feature, but applications developers usually need them.&lt;/p&gt;
&lt;p&gt;That brings me to the usage and querying of Prometheus and Gnocchi. Prometheus has its small DSL (referred to as &lt;a href=&quot;https://prometheus.io/docs/querying/basics/&quot;&gt;PromQL&lt;/a&gt;) whereas Gnocchi has a &lt;a href=&quot;http://gnocchi.xyz/rest.html&quot;&gt;fully featured REST API&lt;/a&gt; that tries to expose proper semantic. It does not seem there are major differences between the two in term of features.&lt;/p&gt;
&lt;p&gt;Both Prometheus and Gnocchi support aggregating values over time ranges on query time (&quot;give me the minimum value for every 5 minutes range over the last day&quot;). Gnocchi always aggregates metrics at writing time, and never at query time (unless doing it cross-metrics). This implies that Gnocchi needs a bit of CPU time at write time to pre-compute those aggregates, but it is blazingly fast at reading time as it has nothing to compute. Prometheus can do the same thing using &lt;a href=&quot;https://prometheus.io/docs/querying/rules/&quot;&gt;recording rules&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/icon_clock.png&quot; alt=&quot;Clock icon&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Prometheus has some limitations inherent to time series database designed around the notion of &quot;monitoring&quot;: they tend to compute everything relatively to &lt;code&gt;$NOW&lt;/code&gt;. For example, it seems impossible to inject data from the past. The timestamp for a value is the timestamp where Prometheus read that value. If Prometheus misses values for a few hours, don&apos;t think about importing it back.&lt;/p&gt;
&lt;p&gt;I&apos;m noting this here as it makes it harder to benchmark Prometheus for ingestion. You need tons of fake metrics to polls and build data. I did not find any reference of Prometheus performances online, though it is advertised to ingest &quot;millions of measures from thousands of sources&quot;.&lt;/p&gt;
&lt;p&gt;Query performances seem to vary on Prometheus, and I did not find any benchmark on that neither. Gnocchi leverages standard RDBMS (MySQL or PostgreSQL is supported) to query indexed data and the metrics retrieval is always &lt;em&gt;O(1)&lt;/em&gt;, making it &lt;strong&gt;always fast&lt;/strong&gt;.&lt;/p&gt;
&lt;h2&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;If you look in different and older areas, there never has been only one HTTP server. Many people use Apache HTTP server, but you&apos;ll find plenty of users of nginx, Tomcat, HAProxy, Node.js or uwsgi which are also common options nowadays. Same goes for RDBMS if you look at PostgreSQL, MySQL and other databases solution, etc. There will never be a project winning all the market share.&lt;/p&gt;
&lt;p&gt;It seems to me that time series storage and management is also growing in this category. There will probably be various projects that will enjoy some popularity and growth. Every project addresses the time series problem space with a different view and different trade-offs. There might never be a single project solving all problems at once.&lt;/p&gt;
&lt;p&gt;Prometheus seems to be oriented toward monitoring of live systems. Gnocchi is oriented to highly available time series storage at massive scale. Not considering performances (I was not able to compare anyway), both have different tradeoffs in term of features, philosophy, and orientation. Depending on your use cases, one might be a better fit than the other.&lt;/p&gt;
</content:encoded><category>gnocchi</category><category>monitoring</category></item><item><title>Using Gnocchi with Docker</title><link>https://julien.danjou.info/blog/using-gnocchi-with-docker/</link><guid isPermaLink="true">https://julien.danjou.info/blog/using-gnocchi-with-docker/</guid><description>I&apos;ve recently started to look into Docker to build images ready to be used with Gnocchi in it. I found it would be a great way to distribute a working instance of Gnocchi.</description><pubDate>Thu, 17 Aug 2017 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;I&apos;ve recently started to look into Docker to build images ready to be used with &lt;a href=&quot;http://gnocchi.xyz&quot;&gt;Gnocchi&lt;/a&gt; in it. I found it would be a great way to distribute a working instance of Gnocchi.&lt;/p&gt;
&lt;p&gt;To this end, we created the &lt;a href=&quot;https://github.com/gnocchixyz/gnocchi-docker&quot;&gt;gnocchi-docker&lt;/a&gt; repository on GitHub. It contains:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;a 11 lines long (only!) &lt;a href=&quot;https://github.com/gnocchixyz/gnocchi-docker/blob/master/gnocchi/Dockerfile&quot;&gt;Dockerfile&lt;/a&gt; to build a Linux image containing Gnocchi;&lt;/li&gt;
&lt;li&gt;a &lt;a href=&quot;https://github.com/gnocchixyz/gnocchi-docker/tree/master/grafana&quot;&gt;Dockerfile&lt;/a&gt; to create a &lt;a href=&quot;https://grafana.com/&quot;&gt;Grafana&lt;/a&gt; image that will use Gnocchi as datasource (preconfigured);&lt;/li&gt;
&lt;li&gt;a &lt;a href=&quot;https://github.com/gnocchixyz/gnocchi-docker/blob/master/collectd/Dockerfile&quot;&gt;Dockerfile&lt;/a&gt; to create a &lt;a href=&quot;http://collectd.org&quot;&gt;collectd&lt;/a&gt; image that gather various metrics for your container in order to feed Gnocchi and have something to display in Grafana;&lt;/li&gt;
&lt;li&gt;a &lt;a href=&quot;https://github.com/gnocchixyz/gnocchi-docker/blob/master/docker-compose.yaml&quot;&gt;docker-compose file&lt;/a&gt; that orchestrates and runs those containers.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;If you don&apos;t know &lt;a href=&quot;https://docs.docker.com/compose/&quot;&gt;docker-compose&lt;/a&gt;, it&apos;s a tool to define and run applications using multiple containers. This is very handy in our case, as we need to start a few services, and therefore a few containers, to have our whole stack running.&lt;/p&gt;
&lt;p&gt;If you just want to use and run Gnocchi in a snap using this, it&apos;s easy. First clone the repository:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ git clone https://github.com/gnocchixyz/gnocchi-docker.git
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Then, just ask docker-compose to start your stack of containers:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ cd gnocchi-docker
$ docker-compose up
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;On the first run, &lt;code&gt;docker-compose&lt;/code&gt; will build the various images (this should take only a few minutes) and then will start them.&lt;/p&gt;
&lt;p&gt;Once everything is started, you can connect to Grafana by typing the URL &lt;code&gt;http://&amp;lt;ip of your docker server&amp;gt;:3000&lt;/code&gt; in your browser and using &quot;admin&quot; as username and &quot;password&quot; as password. Just click on the dashboard entitled &quot;Gnocchi&quot; and wait a few minutes: you will see the chart being drawn in real time!&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-docker-grafana-screenshot.png&quot; alt=&quot;gnocchi-docker-grafana-screenshot&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The data fed into Gnocchi come from the &lt;code&gt;collectd&lt;/code&gt; container, which gathers various metrics (CPU, network interface statistics, etc).&lt;/p&gt;
&lt;p&gt;You can then edit the docker files as you like to add new features or test your code. The files are also a good basis if you want to deploy Gnocchi in production running Docker!&lt;/p&gt;
&lt;p&gt;If you want to access and play with Gnocchi in command line, just install &lt;a href=&quot;https://pypi.python.org/pypi/gnocchiclient&quot;&gt;gnocchiclient&lt;/a&gt; and do the following:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ export GNOCCHI_ENDPOINT=http://`docker-machine ip`:8041
$ gnocchi resource list
+----------+----------+------------+---------+----------------------+------------+----------+----------------+--------------+---------+
| id       | type     | project_id | user_id | original_resource_id | started_at | ended_at | revision_start | revision_end | creator |
+----------+----------+------------+---------+----------------------+------------+----------+----------------+--------------+---------+
| c31e4adc | collectd | None       | None    | collectd:fake-phy-   | 2017-08-17 | None     | 2017-08-17T12: | None         | admin   |
| -2cff-5f |          |            |         | host-719acbad336c    | T12:20:27. |          | 20:27.643790+0 |              |         |
| 78-8206- |          |            |         |                      | 643778+00: |          | 0:00           |              |         |
| f5ca66e4 |          |            |         |                      | 00         |          |                |              |         |
| 6cce     |          |            |         |                      |            |          |                |              |         |
+----------+----------+------------+---------+----------------------+------------+----------+----------------+--------------+---------+
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;You can now have fun creating new resources and metrics!&lt;/p&gt;
&lt;p&gt;Feel free to contribute patches to &lt;a href=&quot;http://github.com/gnocchixyz/gnocchi-docker&quot;&gt;the GitHub project&lt;/a&gt; too, obviously!&lt;/p&gt;
</content:encoded><category>gnocchi</category><category>linux</category></item><item><title>Gnocchi 4 is out</title><link>https://julien.danjou.info/blog/gnocchi-4-release/</link><guid isPermaLink="true">https://julien.danjou.info/blog/gnocchi-4-release/</guid><description>Finally! Four months ago we pushed the Gnocchi 3.1 release and here we are now, release the 4th major version of that timeseries database.  A lot happened in the last 4 months.First, as I already wrot</description><pubDate>Tue, 13 Jun 2017 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;Finally! Four months ago we pushed the Gnocchi 3.1 release and here we are now, release the 4th major version of that timeseries database.&lt;/p&gt;
&lt;p&gt;A lot happened in the last 4 months. &lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-logo.png&quot; alt=&quot;Gnocchi logo&quot; /&gt;&lt;/p&gt;
&lt;p&gt;First, as I already wrote about, &lt;a href=&quot;https://julien.danjou.info/blog/gnocchi-independence&quot;&gt;we moved to GitHub for hosting our project&lt;/a&gt;. This slowed down our development pace for a couple of weeks, but we&apos;re now almost back to normal! We were a bit sad to quit the great infrastructure that we used before, but it feels great to be hosted on a platform everyone knows about and is more straightforward to use.&lt;/p&gt;
&lt;p&gt;Second, we implemented some major changes that should improve performances &lt;em&gt;again&lt;/em&gt;. We tend to that in each release, I know, I know. As usual, the release notes contains most of &lt;a href=&quot;http://gnocchi.xyz/releasenotes/4.0.html&quot;&gt;the major changes we did and can be read online&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;But I&apos;d like to talk about few here that I find very exciting. The work and performances tests that Alex Krzos did (and&lt;br /&gt;
&lt;a href=&quot;https://julien.danjou.info/blog/2017/openstack-summit-pike-boston-recap&quot;&gt;we presented during the last OpenStack Summit&lt;/a&gt;) was of a great help for inspiration on where to improve performances.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;&lt;a href=&quot;http://redis.io&quot;&gt;Redis&lt;/a&gt;! We added a Redis driver which can store incoming measures and metric archives. Obviously, it&apos;s more meant for incoming measures. Remember, in Gnocchi 3.1 we split the storage driver into two parts: the incoming measure storage and the archive storage. Since you can use two different drivers for those different functions, with Gnocchi 4.0 you can use Redis to store your incoming measures in a very fast temporary storage service and then &lt;em&gt;metricd&lt;/em&gt; will process them and store the results in your favorite scalable storage such as &lt;a href=&quot;http://ceph.com&quot;&gt;Ceph&lt;/a&gt;, where it&apos;s mostly read.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Sacks! We rewrote the entire scheduling mechanism for &lt;em&gt;metricd&lt;/em&gt;. It now uses several &quot;sacks&quot; to store incoming measures in a distributed manner, instead of the previous one-sack-only storage for those incoming data. A hashring is then used to spread the processing workload on all the running &lt;em&gt;metricd&lt;/em&gt; daemon. Faster, simpler and more efficient scheduling should happen with this version!&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;S3! We fixed the S3 driver. It was a nice proof-of-concept in 3.1 and now it should work. For real.&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;That&apos;s mostly it. The rest of the changes are bug fixes there and there and some performance improvement, but this should be enough to get you excited to try it out.&lt;/p&gt;
&lt;p&gt;Come and join us on &lt;a href=&quot;http://github.com/gnocchixyz&quot;&gt;GitHub&lt;/a&gt;! Star us, and stay tuned for some more awesome news around metrics.&lt;/p&gt;
</content:encoded><category>gnocchi</category></item><item><title>OpenStack Summit Boston 2017 recap</title><link>https://julien.danjou.info/blog/openstack-summit-pike-boston-recap/</link><guid isPermaLink="true">https://julien.danjou.info/blog/openstack-summit-pike-boston-recap/</guid><description>The first OpenStack Summit of 2017 was last week, in Boston, MA, USA. I was able to attend as I&apos;ve been selected to give 3 talks, to help for a hands-on and to animate an on-boarding session.</description><pubDate>Mon, 15 May 2017 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;The &lt;a href=&quot;https://www.openstack.org/summit/boston-2017/&quot;&gt;first OpenStack Summit of 2017&lt;/a&gt; was last week, in Boston, MA, USA. I was able to attend as I&apos;ve been selected to give 3 talks, to help for a hands-on and to animate an on-boarding session. This made sure I was a bit busy every day, which was good.&lt;/p&gt;
&lt;p&gt;This is the first summit to happen since the new &lt;a href=&quot;https://www.openstack.org/ptg/&quot;&gt;Project Team Gathering (PTG)&lt;/a&gt; happened last February. I was unable to attend this first PTG back then, as there was no way to justify my presence there. The OpenStack Telemetry team that I lead is pretty small. People don&apos;t really need to talk to each other face to face to discuss: therefore we decided to not ask to be present during the last PTG event.&lt;/p&gt;
&lt;p&gt;The Telemetry on-boarding session that I organized with my fellow developer Gordon Chung on Tuesday had only 3 people showing up to ask a few questions about Telemetry. The session lasted 15 minutes on 90 planned. We shared that session with &lt;a href=&quot;https://wiki.openstack.org/wiki/CloudKitty&quot;&gt;CloudKitty&lt;/a&gt;, for which nobody showed up for. When you think about it, this was really disappointing but did not come as a surprise.&lt;/p&gt;
&lt;p&gt;First, the amount of company engaging developers into OpenStack has shrunk drastically during the last year. Secondly, since there&apos;s now another event (the PTG) twice a year, it seems pretty clear that every developer will not be able to attend all the 4 events every year, creating dispersion in the community.&lt;/p&gt;
&lt;p&gt;I personally was glad to attend the Summit rather than the PTG, as it is more valuable to meet operators and users than developers to gather feedback. However, meeting everyone at the same time would be great, especially for smaller teams. The PTG scattered some teams to a point that many of developers of those lineups won&apos;t go to either the PTG nor the OpenStack. As a consequence, I won&apos;t have any meeting point in the future with many of my fellow developers around OpenStack. I warned the Technical Committee last year about this when it was decided to reorganize the events. I&apos;m glad to be right but I&apos;m a bit sad that the Foundation did not listen.&lt;/p&gt;
&lt;p&gt;Though all the projects I work on tend to follow &lt;a href=&quot;https://julien.danjou.info/blog/foss-projects-management-bad-practice&quot;&gt;the good practice I wrote last year&lt;/a&gt;. Therefore I cannot say that it has huge consequences on the projects I work on. It&apos;s a loss as it makes it harder to reach users and operators for some of us. It also reduces our occasion for social interaction, which was a great benefit. But it will not prevent us from building great software anyway!&lt;/p&gt;
&lt;p&gt;The few other sessions of &lt;em&gt;&lt;a href=&quot;https://wiki.openstack.org/wiki/Forum&quot;&gt;The Forum&lt;/a&gt;&lt;/em&gt; (the space dedicated to developers during the Summit) that I attended discussed various technical things, and some sessions were pretty empty. I wonder if it was a lack of interest of people or if people were unable to travel to discuss those items. Anyhow, at this stage I am not sure it would have really mattered: this has been my 9th OpenStack Summit and many of the subjects discussed already have been discussed multiple time with barely any change since. Talk is cheap. Furthermore, most of the discussion were not made by stakeholders of the various projects involved, but by people on the side, or by members of the Technical Committee. There is just unfortunately too much of wishful thinking.&lt;/p&gt;
&lt;p&gt;On the talk side, my presentation with Alex Krzos entitled &lt;em&gt;Telemetry and the 10,000 instances&lt;/em&gt; went pretty well. We demonstrated what how we tested the performance of the telemetry stack.&lt;/p&gt;
&lt;p&gt;Same goes for my hands-on with the CloudKitty developers, where we managed to explain how Ceilometer, Gnocchi, and CloudKitty were able to work with each other to create nice billing reports. The last day was concluded with my talk on collectd and Gnocchi with Emma, which was short and to the point.&lt;/p&gt;
&lt;p&gt;My final talk was about the status and roadmap of the OpenStack Telemetry team where I tried to explain how the Telemetry works and what we might do (or not) in the next cycles. It was pretty short as we barely have a roadmap, the project having 3 developers doing 80% of the work.&lt;/p&gt;
&lt;p&gt;I was also able to catch up with Nubeliu about their Gnocchi usage. They &lt;a href=&quot;https://www.youtube.com/watch?v=Hlt3UwsvgjU&quot;&gt;presented a nice demo of the cloud monitoring solution&lt;/a&gt; they build on top of Gnocchi. They completely understood how to use Gnocchi to store a large number of metrics at scale and how to leverage the API to render what&apos;s happening in your infrastructure. It is pretty amazing.&lt;/p&gt;
&lt;p&gt;While I missed the energy and the drive that the design session used to have in the first summits, it has been a pretty good summit. I was especially happy to be able to discuss OpenStack Telemetry and Gnocchi. The feedback I gathered was tremendous and terrific and I&apos;m looking forward to the work we&apos;ll achieve in the next months!&lt;/p&gt;
</content:encoded><category>openstack</category><category>gnocchi</category><category>talks</category></item><item><title>Gnocchi independence</title><link>https://julien.danjou.info/blog/gnocchi-independence/</link><guid isPermaLink="true">https://julien.danjou.info/blog/gnocchi-independence/</guid><description>Three years have passed since Gnocchi started. After being incubated inside OpenStack, the project is now going independent.</description><pubDate>Sat, 06 May 2017 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;Three years have passed since I started working on &lt;a href=&quot;http://gnocchi.xyz&quot;&gt;Gnocchi&lt;/a&gt;. It&apos;s amazing to gaze at the path we wandered on.&lt;/p&gt;
&lt;p&gt;During all this time, Gnocchi has been &quot;incubated&quot; inside OpenStack. It has been created there and it grew with the rest of the ecosystem. But Gnocchi (developers) always stuck to some strange principles: autonomy and independence from the other OpenStack projects. This actually made the project a bit unpopular sometimes inside OpenStack, being stamped as some kind of &lt;em&gt;rebel&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;I&apos;ve spent the last years asserting that each project inside OpenStack should seek towards living its own life. It is a key success for any open source project to be able to be used in any context, not only the one it has been built for. Having to use large bundles of projects together is not a good user story. I wish OpenStack will be a set of more autonomous building blocks.&lt;/p&gt;
&lt;p&gt;One of the most used project by people not using an entire OpenStack installation has been &lt;a href=&quot;https://launchpad.net/swift&quot;&gt;Swift&lt;/a&gt;. That was possible because Swift always tried to be autonomous and to not depend on any other service. It is able to leverage external services but it can also work without any. And I feel that Swift is the most successful project if you measure that success by being used by people having zero knowledge about OpenStack.&lt;/p&gt;
&lt;p&gt;With the move toward the &lt;em&gt;Big Tent&lt;/em&gt;, it struck me that the OpenStack Foundation will end up as some sort of an Apache Foundation. And I am pretty sure nobody forces you to use the &lt;a href=&quot;https://httpd.apache.org/&quot;&gt;Apache HTTP server&lt;/a&gt; if you want to use e.g. &lt;a href=&quot;http://lucene.apache.org/&quot;&gt;Lucene&lt;/a&gt; or &lt;a href=&quot;http://hbase.apache.org/&quot;&gt;HBase&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Being part of OpenStack for Gnocchi has been a great advantage at the beginning of the project. The infrastructure provided is awesome. The support we had from the community was great. The Gerrit workflow suited us well.&lt;/p&gt;
&lt;p&gt;But unfortunately, now that the project is getting more and more mature, many of the requirements of being an OpenStack project has become a real burden. The various processes forced by OpenStack is hurting the development pace. The contribution workflow based around Gerrit and &lt;a href=&quot;https://launchpad.net&quot;&gt;Launchpad&lt;/a&gt; is too complicated for most external contributors and therefore prevents new users to participate to the development.&lt;br /&gt;
Worse, the bad image or reputation that OpenStack carries in certain situation or communities is preventing Gnocchi to be evaluated and, maybe, used.&lt;/p&gt;
&lt;p&gt;I think that many of those negative aspects are finally taken into account by the OpenStack Technical Committee, as can be seen in the &lt;a href=&quot;https://review.openstack.org/#/c/453262/&quot;&gt;proposed vision of 2 years from now for OpenStack&lt;/a&gt;.&lt;br /&gt;
Better late than never.&lt;/p&gt;
&lt;p&gt;So after spending a lot of time weighing the pros and the cons, we, Gnocchi&lt;br /&gt;
contributors, &lt;a href=&quot;http://lists.openstack.org/pipermail/openstack-dev/2017-March/114300.html&quot;&gt;finally decided to move Gnocchi out of OpenStack&lt;/a&gt;.&lt;br /&gt;
We started to move the project to a brand new &lt;a href=&quot;https://github.com/gnocchixyz&quot;&gt;Gnocchi organization on GitHub&lt;/a&gt;. At the time of this writing, only the main gnocchi repository is missing and should be moved soon after the OpenStack Summit happening next week.&lt;/p&gt;
&lt;p&gt;We also used that opportunity to make usage of the new Gnocchi logo, courtesy of my friend Thierry Ung!&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-logo.png&quot; alt=&quot;Gnocchi logo&quot; /&gt;&lt;/p&gt;
&lt;p&gt;We&apos;ll see how everything will turn out and if the project will gain more traction, as we hope. This will not change the consumption of Gnocchi made by projects such as &lt;a href=&quot;http://launchpad.net/ceilometer&quot;&gt;Ceilometer&lt;/a&gt;. and the project aims to remain a good friend of OpenStack. 😀&lt;/p&gt;
</content:encoded><category>gnocchi</category><category>openstack</category></item><item><title>Scalable metrics storage: Gnocchi on Amazon Web Services</title><link>https://julien.danjou.info/blog/metrics-on-amazon-with-gnocchi-s3-driver/</link><guid isPermaLink="true">https://julien.danjou.info/blog/metrics-on-amazon-with-gnocchi-s3-driver/</guid><description>As I wrote a few weeks ago in my post about Gnocchi 3.1 being released, one of the new feature available in this version it the S3 driver.</description><pubDate>Wed, 22 Feb 2017 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;As I wrote a few weeks ago in my &lt;a href=&quot;https://julien.danjou.info/blog/gnocchi-3-1-unleashed&quot;&gt;post about Gnocchi 3.1 being released&lt;/a&gt;, one of the new feature available in this version it the &lt;a href=&quot;https://aws.amazon.com/s3/&quot;&gt;S3&lt;/a&gt; driver. Today I would like to show you how easy it is to use it and store millions of metrics into the simple, durable and massively scalable object storage provided&lt;br /&gt;
by &lt;a href=&quot;https://aws.amazon.com/&quot;&gt;Amazon Web Services&lt;/a&gt;.&lt;/p&gt;
&lt;h2&gt;Installation&lt;/h2&gt;
&lt;p&gt;The installation of Gnocchi for this use case is not different than&lt;br /&gt;
the &lt;a href=&quot;http://gnocchi.xyz/install.html&quot;&gt;standard installation procedure described in the documentation&lt;/a&gt;. Simply install Gnocchi from &lt;a href=&quot;http://pypi.python.org&quot;&gt;PyPI&lt;/a&gt; using the following command:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ pip install gnocchi[s3,postgresql] gnocchiclient
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This will install Gnocchi with the dependencies for the S3 and PostgreSQL drivers and the command-line interface to talk with Gnocchi.&lt;/p&gt;
&lt;h2&gt;Configuring Amazon RDS&lt;/h2&gt;
&lt;p&gt;Since you need a SQL database for the indexer, the easiest way to get started is to create a database on &lt;a href=&quot;https://console.aws.amazon.com/rds/&quot;&gt;Amazon RDS&lt;/a&gt;. You can create a managed &lt;a href=&quot;http://postgresql.org&quot;&gt;PostgreSQL&lt;/a&gt; database instance in just a few clicks.&lt;/p&gt;
&lt;p&gt;Once you&apos;re on the homepage of &lt;a href=&quot;https://console.aws.amazon.com/rds/&quot;&gt;Amazon RDS&lt;/a&gt;, pick PostgreSQL as a&lt;br /&gt;
database:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-rds-postgresql.png&quot; alt=&quot;gnocchi-rds-postgresql&quot; /&gt;&lt;/p&gt;
&lt;p&gt;You can then configure your PostgreSQL instance: I&apos;ve picked a dev/test instance with the basic options available within the RDS Free Tier, but you can pick whatever you think is needed for your production use. Set a username and a password and note them for later: we&apos;ll need them to configure Gnocchi.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-rds-postgresql-conf.png&quot; alt=&quot;gnocchi-rds-postgresql-conf&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The next step is to configure the database in details. Just set the database name to &quot;gnocchi&quot; and leave the other options to their default values (I&apos;m lazy).&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-rds-postgresql-details.png&quot; alt=&quot;gnocchi-rds-postgresql-details&quot; /&gt;&lt;/p&gt;
&lt;p&gt;After a few minutes, your instance should be created and running. Note down the endpoint. In this case, my instance is &lt;code&gt;gnocchi.cywagbaxpert.us-east-1.rds.amazonaws.com&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-rds-postgresql-running.png&quot; alt=&quot;gnocchi-rds-postgresql-running&quot; /&gt;&lt;/p&gt;
&lt;h2&gt;Configuring Gnocchi for S3 access&lt;/h2&gt;
&lt;p&gt;In order to give Gnocchi an access to S3, you need to create access keys. The easiest way to create them is to go to &lt;a href=&quot;https://console.aws.amazon.com/iam&quot;&gt;IAM&lt;/a&gt; in your AWS console, pick a user with S3 access and click on the big gray button named &quot;Create access key&quot;.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-iam-create-keys.png&quot; alt=&quot;gnocchi-iam-create-keys&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Once you do that, you&apos;ll get the &lt;em&gt;access key id&lt;/em&gt; and &lt;em&gt;secret access key&lt;/em&gt;. Note them down, we will need these later.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-iam-get-keys.png&quot; alt=&quot;gnocchi-iam-get-keys&quot; /&gt;&lt;/p&gt;
&lt;h2&gt;Creating &lt;code&gt;gnocchi.conf&lt;/code&gt;&lt;/h2&gt;
&lt;p&gt;Now is time to create the &lt;code&gt;gnocchi.conf&lt;/code&gt; file. You can place it in &lt;code&gt;/etc/gnocchi&lt;/code&gt; if you want to deploy it system-wide, or in any other directory and add the &lt;code&gt;--config-file&lt;/code&gt; option to each Gnocchi command..&lt;/p&gt;
&lt;p&gt;Here are the values that you should retrieve and write in the configuration file:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;indexer.url&lt;/code&gt;: the PostgreSQL RDS instance endpoint and credentials (see above) to set into&lt;/li&gt;
&lt;li&gt;&lt;code&gt;storage.s3_endpoint_url&lt;/code&gt;: the S3 endpoint URL – that depends on the region you want to use and &lt;a href=&quot;http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region&quot;&gt;they are listed here&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;storage.s3_region_name&lt;/code&gt;: the S3 region name matching the endpoint you picked.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;storage.s3_access_key_id&lt;/code&gt; and &lt;code&gt;storage.s3_secret_acess_key&lt;/code&gt;: your AWS access key id and secret access key.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Your &lt;code&gt;gnocchi.conf&lt;/code&gt; file should then look like that:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;[indexer]
url = postgresql://gnocchi:gn0cch1rul3z@gnocchi.cywagbaxpert.us-east-1.rds.amazonaws.com:5432/gnocchi

[storage]
driver = s3
s3_endpoint_url = https://s3-eu-west-1.amazonaws.com
s3_region_name = eu-west-1
s3_access_key_id = &amp;lt;you access key id&amp;gt;
s3_secret_access_key = &amp;lt;your secret access key&amp;gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Once that&apos;s done, you can run &lt;code&gt;gnocchi-upgrade&lt;/code&gt; in order to initialize Gnocchi indexer (PostgreSQL) and storage (S3):&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ gnocchi-upgrade --config-file gnocchi.conf
2017-02-07 15:35:52.491 3660 INFO gnocchi.cli [-] Upgrading indexer &amp;lt;gnocchi.indexer.sqlalchemy.SQLAlchemyIndexer object at 0x108221950&amp;gt;
2017-02-07 15:36:04.127 3660 INFO gnocchi.cli [-] Upgrading storage &amp;lt;gnocchi.storage.s3.S3Storage object at 0x10ca943d0&amp;gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Then you can run the API endpoint using the test endpoint &lt;code&gt;gnocchi-api&lt;/code&gt; and specifying its default port 8041:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ gnocchi-api --port 8041 -- --config-file gnocchi.conf
2017-02-07 15:53:06.823 6290 INFO gnocchi.rest.app [-] WSGI config used: /Users/jd/Source/gnocchi/gnocchi/rest/api-paste.ini
********************************************************************************
STARTING test server gnocchi.rest.app.build_wsgi_app
Available at http://127.0.0.1:8041/
DANGER! For testing only, do not use in production
********************************************************************************
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The best way to run Gnocchi API is to use &lt;a href=&quot;http://gnocchi.xyz/master/running.html#running-api-as-a-wsgi-application&quot;&gt;uwsgi as documented&lt;/a&gt;, but in this case, using the testing daemon &lt;code&gt;gnocchi-api&lt;/code&gt; is good enough.&lt;/p&gt;
&lt;p&gt;Finally, in another terminal, you can start the &lt;code&gt;gnocchi-metricd&lt;/code&gt; daemon that will process metrics in background:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ gnocchi-metricd --config-file gnocchi.conf
2017-02-07 15:52:41.416 6262 INFO gnocchi.cli [-] 0 measurements bundles across 0 metrics wait to be processed.
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Once everything is running, you can use Gnocchi&apos;s client to query it and check that everything is OK. The backlog should be empty at this stage, obviously.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ gnocchi status
+-----------------------------------------------------+-------+
| Field                                               | Value |
+-----------------------------------------------------+-------+
| storage/number of metric having measures to process | 0     |
| storage/total number of measures to process         | 0     |
+-----------------------------------------------------+-------+
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Gnocchi is ready to be used!&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ # Create a generic resource &quot;foobar&quot; with a metric named &quot;visitor&quot;
$ gnocchi resource create foobar -n visitor
+-----------------------+-----------------------------------------------+
| Field                 | Value                                         |
+-----------------------+-----------------------------------------------+
| created_by_project_id |                                               |
| created_by_user_id    | admin                                         |
| creator               | admin                                         |
| ended_at              | None                                          |
| id                    | b4d568e4-7af1-5aec-ac3f-9c09fa3685a9          |
| metrics               | visitor: 05f45876-1a69-4a64-8575-03eea5b79407 |
| original_resource_id  | foobar                                        |
| project_id            | None                                          |
| revision_end          | None                                          |
| revision_start        | 2017-02-07T14:54:54.417447+00:00              |
| started_at            | 2017-02-07T14:54:54.417414+00:00              |
| type                  | generic                                       |
| user_id               | None                                          |
+-----------------------+-----------------------------------------------+

## Send the number of visitor at 2 different timestamps
$ gnocchi measures add --resource-id foobar -m 2017-02-07T15:56@23 visitor
$ gnocchi measures add --resource-id foobar -m 2017-02-07T15:57@42 visitor

## Check the average number of visitor
## (the --refresh option is given to be sure the measure are processed)
$ gnocchi measures show --resource-id foobar visitor --refresh
+---------------------------+-------------+-------+
| timestamp                 | granularity | value |
+---------------------------+-------------+-------+
| 2017-02-07T15:55:00+00:00 |       300.0 |  32.5 |
+---------------------------+-------------+-------+

## Now shows the minimum number of visitor
$ gnocchi measures show --aggregation min --resource-id foobar visitor
+---------------------------+-------------+-------+
| timestamp                 | granularity | value |
+---------------------------+-------------+-------+
| 2017-02-07T15:55:00+00:00 |       300.0 |  23.0 |
+---------------------------+-------------+-------+
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;And voilà! You&apos;re ready to store millions of metrics and measures on your Amazon Web Services cloud platform. I hope you&apos;ll enjoy it and feel free to ask any question in the comment section or by reaching me directly!&lt;/p&gt;
</content:encoded><category>gnocchi</category><category>web</category></item><item><title>Sending your collectd metrics to Gnocchi</title><link>https://julien.danjou.info/blog/gnocchi-collectd-setup/</link><guid isPermaLink="true">https://julien.danjou.info/blog/gnocchi-collectd-setup/</guid><description>Knowing that collectd is a daemon that collects system and applications metrics and that Gnocchi is a scalable timeseries database, it sounds like a good idea to combine them together.</description><pubDate>Thu, 16 Feb 2017 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;Knowing that &lt;a href=&quot;http://collectd.org/&quot;&gt;collectd&lt;/a&gt; is a daemon that collects system and applications metrics and that &lt;a href=&quot;http://gnocchi.xyz&quot;&gt;Gnocchi&lt;/a&gt; is a scalable timeseries database, it sounds like a good idea to combine them together. &lt;em&gt;Cherry on the cake&lt;/em&gt;: you can easily draw charts using &lt;a href=&quot;http://grafana.org&quot;&gt;Grafana&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;While it&apos;s true that Gnocchi is well integrated with &lt;a href=&quot;http://openstack.org&quot;&gt;OpenStack&lt;/a&gt;, as it orginally comes from this ecosystem, it actually works standalone by default. Starting with the 3.1 version, it is now easy to send metrics to &lt;em&gt;Gnocchi&lt;/em&gt; using &lt;em&gt;collectd&lt;/em&gt;.&lt;/p&gt;
&lt;h2&gt;Installation&lt;/h2&gt;
&lt;p&gt;What we&apos;ll need to install to accomplish this task is:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;collectd&lt;/li&gt;
&lt;li&gt;Gnocchi&lt;/li&gt;
&lt;li&gt;collectd-gnocchi&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;How you install them does not really matter. If they are packaged by your operating system, go ahead. For Gnocchi and collectd-gnocchi, you can also use &lt;em&gt;pip&lt;/em&gt;:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;## pip install gnocchi[file,postgresql]
[…]
Successfully installed gnocchi-3.1.0
## pip install collectd-gnocchi
Collecting collectd-gnocchi
  Using cached collectd-gnocchi-1.0.1.tar.gz
[…]
Installing collected packages: collectd-gnocchi
  Running setup.py install for collectd-gnocchi ... done
Successfully installed collectd-gnocchi-1.0.1
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The detailed installation procedure for Gnocchi is &lt;a href=&quot;http://gnocchi.xyz/install.html#id1&quot;&gt;detailed in the documentation&lt;/a&gt;. It among other things explains which flavors are available – here I picked PostgreSQL and the file driver to store the metrics.&lt;/p&gt;
&lt;h2&gt;Configuration&lt;/h2&gt;
&lt;h3&gt;Gnocchi&lt;/h3&gt;
&lt;p&gt;Gnocchi is simple to configure and is again &lt;a href=&quot;http://gnocchi.xyz/configuration.html&quot;&gt;documented&lt;/a&gt;. The default configuration file is &lt;code&gt;/etc/gnocchi/gnocchi.conf&lt;/code&gt; – you can generate it with &lt;code&gt;gnocchi-config-generator&lt;/code&gt; if needed. However, it also possible to specify another configuration file by appending the &lt;code&gt;--config-file&lt;/code&gt; option to any command line&lt;/p&gt;
&lt;p&gt;In Gnocchi&apos;s configuration file, you need to set the &lt;code&gt;indexer.url&lt;/code&gt; configuration option to point an existing PostgreSQL database and set &lt;code&gt;storage.file_basepath&lt;/code&gt; to an existing directory to store your metrics (the default is &lt;code&gt;/var/lib/gnocchi&lt;/code&gt;). That gives something like:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;[indexer]
url = postgresql://root:p4assw0rd@localhost/gnocchi

[storage]
file_basepath = /var/lib/gnocchi
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Once done, just run the &lt;code&gt;gnocchi-upgrade&lt;/code&gt; command to initialize the index and storage.&lt;/p&gt;
&lt;h3&gt;collectd&lt;/h3&gt;
&lt;p&gt;Collectd provides a default configuration file that loads a bunch of plugin by default, that will meter all sort of metrics on your computer. You can check the &lt;a href=&quot;http://collectd.org/documentation.shtml&quot;&gt;documentation&lt;/a&gt; online to see how to disable or enable plugins.&lt;/p&gt;
&lt;p&gt;As the &lt;em&gt;collectd-gnocchi&lt;/em&gt; plugin is written in Python, you&apos;ll need to enable the Python plugin and load the &lt;em&gt;collectd-gnocchi&lt;/em&gt; module:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;LoadPlugin python

&amp;lt;Plugin python&amp;gt;
  Import &quot;collectd_gnocchi&quot;
  &amp;lt;Module collectd_gnocchi&amp;gt;
      endpoint &quot;http://localhost:8041&quot;
  &amp;lt;/Module&amp;gt;
&amp;lt;/Plugin&amp;gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;That is enough to enable the storage of metrics in Gnocchi.&lt;/p&gt;
&lt;h2&gt;Running the daemons&lt;/h2&gt;
&lt;p&gt;Once everything is configured, you can launch &lt;code&gt;gnocchi-metricd&lt;/code&gt; and the &lt;code&gt;gnocchi-api&lt;/code&gt; daemon:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ gnocchi-metricd
2017-01-26 15:22:49.018 15971 INFO gnocchi.cli [-] 0 measurements bundles
across 0 metrics wait to be processed.
[…]
## In another terminal
$ gnocchi-api --port 8041
[…]
STARTING test server gnocchi.rest.app.build_wsgi_app
Available at http://127.0.0.1:8041/
[…]
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;It&apos;s not recommended to run Gnocchi using Gnocchi API (as&lt;br /&gt;
&lt;a href=&quot;http://gnocchi.xyz/running.html#running-as-a-wsgi-application&quot;&gt;written in the documentation&lt;/a&gt;): using &lt;a href=&quot;https://uwsgi-docs.readthedocs.io/&quot;&gt;uwsgi&lt;/a&gt; is a better option. However for rapid testing, the &lt;code&gt;gnocchi-api&lt;/code&gt; daemon is good enough.&lt;/p&gt;
&lt;p&gt;Once that&apos;s done, you can start &lt;code&gt;collectd&lt;/code&gt;:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ collectd
## Or to run in foreground with a different configuration file:
## $ collectd -C collectd.conf -f
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;If you have any problem launchding &lt;em&gt;colllectd&lt;/em&gt;, check syslog for more information: there might be an issue loading a module or plugin.&lt;/p&gt;
&lt;p&gt;If no error are printed, then everythin&apos;s working fine and you soon should see &lt;em&gt;gnocchi-api&lt;/em&gt; printing some requests such as:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;127.0.0.1 - - [26/Jan/2017 15:27:03] &quot;POST /v1/resource/collectd HTTP/1.1&quot; 409 113
127.0.0.1 - - [26/Jan/2017 15:27:03] &quot;POST /v1/batch/resources/metrics/measures?create_metrics=True HTTP/1.1&quot; 400 91
&lt;/code&gt;&lt;/pre&gt;
&lt;h2&gt;Enjoying the result&lt;/h2&gt;
&lt;p&gt;Once everything runs, you can access your newly created resources and metric by using the &lt;a href=&quot;http://pypi.python.org/pypi/gnocchiclient&quot;&gt;gnocchiclient&lt;/a&gt;. It should have been installed as a dependency of &lt;em&gt;collectd_gnocchi&lt;/em&gt;, but you can also install it manually using &lt;code&gt;pip install gnocchiclient&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;If you need to specify a different endpoint you can use the &lt;code&gt;--endpoint&lt;/code&gt; option (which default to &lt;a href=&quot;http://localhost:8041&quot;&gt;http://localhost:8041&lt;/a&gt;). Do not hesitate to check the &lt;code&gt;--help&lt;/code&gt; option for more information.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ gnocchi resource list --details
+---------------+----------+------------+---------+----------------------+---------------+----------+----------------+--------------+---------+-----------+
| id            | type     | project_id | user_id | original_resource_id | started_at    | ended_at | revision_start | revision_end | creator | host      |
+---------------+----------+------------+---------+----------------------+---------------+----------+----------------+--------------+---------+-----------+
| dd245138-00c7 | collectd | None       | None    | dd245138-00c7-5bdc-  | 2017-01-26T14 | None     | 2017-01-26T14: | None         | admin   | localhost |
| -5bdc-94f8-26 |          |            |         | 94f8-263e236812f7    | :21:02.297466 |          | 21:02.297483+0 |              |         |           |
| 3e236812f7    |          |            |         |                      | +00:00        |          | 0:00           |              |         |           |
+---------------+----------+------------+---------+----------------------+---------------+----------+----------------+--------------+---------+-----------+
$ gnocchi resource show collectd:localhost
+-----------------------+-----------------------------------------------------------------------+
| Field                 | Value                                                                 |
+-----------------------+-----------------------------------------------------------------------+
| created_by_project_id |                                                                       |
| created_by_user_id    | admin                                                                 |
| creator               | admin                                                                 |
| ended_at              | None                                                                  |
| host                  | localhost                                                             |
| id                    | dd245138-00c7-5bdc-94f8-263e236812f7                                  |
| metrics               | interface-en0@if_errors-0: 5d60f224-2e9e-4247-b415-64d567cf5866       |
|                       | interface-en0@if_errors-1: 1df8b08b-555a-4cab-9186-f9b79a814b03       |
|                       | interface-en0@if_octets-0: 491b7517-7219-4a04-bdb6-934d3bacb482       |
|                       | interface-en0@if_octets-1: 8b5264b8-03f3-4aba-a7f8-3cd4b559e162       |
|                       | interface-en0@if_packets-0: 12efc12b-2538-45e7-aa66-f8b9960b5fa3      |
|                       | interface-en0@if_packets-1: 39377ff7-06e8-454a-a22a-942c8f2bca56      |
|                       | interface-en1@if_errors-0: c3c7e9fc-f486-4d0c-9d36-55cea855596a       |
|                       | interface-en1@if_errors-1: a90f1bec-3a60-4f58-a1d1-b3c09dce4359       |
|                       | interface-en1@if_octets-0: c1ee8c75-95bf-4096-8055-8c0c4ec8cd47       |
|                       | interface-en1@if_octets-1: cbb90a94-e133-4deb-ac10-3f37770e32f0       |
|                       | interface-en1@if_packets-0: ac93b1b9-da71-4876-96aa-76067b35c6c9      |
|                       | interface-en1@if_packets-1: 2f8528b2-12ae-4c4d-bec7-8cc987e7487b      |
|                       | interface-en2@if_errors-0: ddcf7203-4c49-400b-9320-9d3e0a63c6d5       |
|                       | interface-en2@if_errors-1: b249ea42-01ad-4742-9452-2c834010df71       |
|                       | interface-en2@if_octets-0: 8c23013a-604e-40bf-a07a-e2dc4fc5cbd7       |
|                       | interface-en2@if_octets-1: 806c1452-0607-4b56-b184-c4ffd48f52c0       |
|                       | interface-en2@if_packets-0: c5bc6103-6313-4b8b-997d-01930d1d8af4      |
|                       | interface-en2@if_packets-1: 478ae87e-e56b-44e4-83b0-ed28d99ed280      |
|                       | load@load-0: 5db2248d-2dca-401e-b2e2-bbaee23b623e                     |
|                       | load@load-1: 6f74ac93-78fd-4a74-a47e-d2add487a30f                     |
|                       | load@load-2: 1897aca1-356e-4791-907f-512e516992b5                     |
|                       | memory@memory-active-0: 83944a85-9c84-4fe4-b471-1a6cf8dce858          |
|                       | memory@memory-free-0: 0ccc7cfa-26a5-4441-a15f-9ebb2aa82c6d            |
|                       | memory@memory-inactive-0: 63736026-94c4-47c5-8d6f-a9d89d65025b        |
|                       | memory@memory-wired-0: b7217fd6-2cdc-4efd-b1a8-a1edd52eaa2e           |
| original_resource_id  | dd245138-00c7-5bdc-94f8-263e236812f7                                  |
| project_id            | None                                                                  |
| revision_end          | None                                                                  |
| revision_start        | 2017-01-26T14:21:02.297483+00:00                                      |
| started_at            | 2017-01-26T14:21:02.297466+00:00                                      |
| type                  | collectd                                                              |
| user_id               | None                                                                  |
+-----------------------+-----------------------------------------------------------------------+
% gnocchi metric show -r collectd:localhost load@load-0
+------------------------------------+-----------------------------------------------------------------------+
| Field                              | Value                                                                 |
+------------------------------------+-----------------------------------------------------------------------+
| archive_policy/aggregation_methods | min, std, sum, median, mean, 95pct, count, max                        |
| archive_policy/back_window         | 0                                                                     |
| archive_policy/definition          | - timespan: 1:00:00, granularity: 0:05:00, points: 12                 |
|                                    | - timespan: 1 day, 0:00:00, granularity: 1:00:00, points: 24          |
|                                    | - timespan: 30 days, 0:00:00, granularity: 1 day, 0:00:00, points: 30 |
| archive_policy/name                | low                                                                   |
| created_by_project_id              |                                                                       |
| created_by_user_id                 | admin                                                                 |
| creator                            | admin                                                                 |
| id                                 | 5db2248d-2dca-401e-b2e2-bbaee23b623e                                  |
| name                               | load@load-0                                                           |
| resource/created_by_project_id     |                                                                       |
| resource/created_by_user_id        | admin                                                                 |
| resource/creator                   | admin                                                                 |
| resource/ended_at                  | None                                                                  |
| resource/id                        | dd245138-00c7-5bdc-94f8-263e236812f7                                  |
| resource/original_resource_id      | dd245138-00c7-5bdc-94f8-263e236812f7                                  |
| resource/project_id                | None                                                                  |
| resource/revision_end              | None                                                                  |
| resource/revision_start            | 2017-01-26T14:21:02.297483+00:00                                      |
| resource/started_at                | 2017-01-26T14:21:02.297466+00:00                                      |
| resource/type                      | collectd                                                              |
| resource/user_id                   | None                                                                  |
| unit                               | None                                                                  |
+------------------------------------+-----------------------------------------------------------------------+
$ gnocchi measures show -r collectd:localhost load@load-0
+---------------------------+-------------+--------------------+
| timestamp                 | granularity |              value |
+---------------------------+-------------+--------------------+
| 2017-01-26T00:00:00+00:00 |     86400.0 | 3.2705004391254193 |
| 2017-01-26T15:00:00+00:00 |      3600.0 | 3.2705004391254193 |
| 2017-01-26T15:00:00+00:00 |       300.0 | 2.6022800611413044 |
| 2017-01-26T15:05:00+00:00 |       300.0 |  3.561742940080275 |
| 2017-01-26T15:10:00+00:00 |       300.0 | 2.5605337960379466 |
| 2017-01-26T15:15:00+00:00 |       300.0 |  3.837517851142473 |
| 2017-01-26T15:20:00+00:00 |       300.0 | 3.9625948392427883 |
| 2017-01-26T15:25:00+00:00 |       300.0 | 3.2690042162698414 |
+---------------------------+-------------+--------------------+
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;As you can see, the command line works smoothly and can show you any kind of metric reported by &lt;em&gt;collectd&lt;/em&gt;. In this case, it was just running on my laptop, but you can imagine it&apos;s easy enough to poll thousands of hosts with &lt;em&gt;collectd&lt;/em&gt; and &lt;em&gt;Gnocchi&lt;/em&gt;.&lt;/p&gt;
&lt;h2&gt;Bonus: charting with Grafana&lt;/h2&gt;
&lt;p&gt;&lt;a href=&quot;http://grafana.org&quot;&gt;Grafana&lt;/a&gt;, a charting software, has a plugin for &lt;em&gt;Gnocchi&lt;/em&gt; as &lt;a href=&quot;http://gnocchi.xyz/grafana.html&quot;&gt;detailed in the documentation&lt;/a&gt;. Once installed, you can just configure &lt;em&gt;Grafana&lt;/em&gt; to point to &lt;em&gt;Gnocchi&lt;/em&gt; this way:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/grafana-config-screen-gnocchi.png&quot; alt=&quot;Screenshot of Grafana configuration screen pointing to Gnocchi as data source&quot; /&gt;&lt;/p&gt;
&lt;p&gt;You can then create a new dashboard by filling the forms as you wish. See this other screenshot for a nice example:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/grafana-gnocchi-load.png&quot; alt=&quot;Charts of my laptop&apos;s load average&quot; /&gt;&lt;/p&gt;
&lt;p&gt;I hope everything is clear and easy enough. If you have any question, feel free to write something in the comment section!&lt;/p&gt;
</content:encoded><category>gnocchi</category><category>monitoring</category></item><item><title>FOSDEM 2017, recap</title><link>https://julien.danjou.info/blog/fosdem-2017-recap/</link><guid isPermaLink="true">https://julien.danjou.info/blog/fosdem-2017-recap/</guid><description>Last week-end, I was in Brussels, Belgium for the 2017 edition of the FOSDEM, one of the greatest open source developer conference.</description><pubDate>Mon, 06 Feb 2017 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/fosdem.png&quot; alt=&quot;FOSDEM 2017 conference logo&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Last week-end, I was in Brussels, Belgium for the 2017 edition of the &lt;a href=&quot;http://fosdem.org&quot;&gt;FOSDEM&lt;/a&gt;, one of the greatest open source developer conference.&lt;/p&gt;
&lt;p&gt;This year, I decided to propose a talk about &lt;a href=&quot;http://gnocchi.xyz&quot;&gt;Gnocchi&lt;/a&gt; which was accepted in the &lt;a href=&quot;https://fosdem.org/2017/schedule/track/python/&quot;&gt;Python devroom&lt;/a&gt;. The track was very well organized (thanks to &lt;a href=&quot;https://wirtel.be/&quot;&gt;Stéphane Wirtel&lt;/a&gt;) and I was able to present Gnocchi to a room full of Python developers!&lt;/p&gt;
&lt;p&gt;I&apos;ve explained why we created Gnocchi and how we did it, and finally briefly explained how to use it with the command-line interface or in a Python application using the &lt;a href=&quot;http://gnocchi.xyz/gnocchiclient&quot;&gt;SDK&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;You can check the slides below and [the video of the talk (&lt;a href=&quot;https://video.fosdem.org/2017/UD2.120/storing_metrics_gnocchi.mp4&quot;&gt;https://video.fosdem.org/2017/UD2.120/storing_metrics_gnocchi.mp4&lt;/a&gt;).&lt;/p&gt;
</content:encoded><category>talks</category><category>python</category><category>gnocchi</category></item><item><title>Gnocchi 3.1 unleashed</title><link>https://julien.danjou.info/blog/gnocchi-3-1-unleashed/</link><guid isPermaLink="true">https://julien.danjou.info/blog/gnocchi-3-1-unleashed/</guid><description>It&apos;s always difficult to know when to release, and we really wanted to do it earlier. But it seems that each week more awesome work was being done in Gnocchi, so we kept delaying it while having no.</description><pubDate>Thu, 02 Feb 2017 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;It&apos;s always difficult to know when to release, and we really wanted to do it earlier. But it seems that each week more awesome work was being done in &lt;a href=&quot;http://gnocchi.xyz&quot;&gt;Gnocchi&lt;/a&gt;, so we kept delaying it while having no pressure to push it out.&lt;/p&gt;
&lt;p&gt;But now that the OpenStack cycle is finishing, even Gnocchi does not strictly follow it, it seemed to be a good time to cut the leash and leave this release be.&lt;/p&gt;
&lt;p&gt;There are again some major new changes coming from 3.0. The previous version 3.0 was tagged in October and had 90 changes merged from 13 authors since 2.2. This 3.1 version have 200 changes merged from 24 different authors. This is a great improvement of our contributor base and our rate of change – even if our delay to merge is very low. Once again, we pushed usage of release notes to document user visible changes, and &lt;a href=&quot;http://gnocchi.xyz/releasenotes/3.1.html&quot;&gt;they can be read online&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Therefore, I am going to summary quickly the major changes:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;The REST API authentication mechanism has been modularized. It&apos;s now simple to provide any authentication mechanism for Gnocchi as a plugin. The default is now a HTTP basic authentication mechanism that does not implement any kind of enforcement. The &lt;a href=&quot;http://docs.openstack.org/developer/keystone/&quot;&gt;Keystone&lt;/a&gt; authentication is still available, obviously.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Batching has been improved and can now create metrics on the fly, reducing the latency needed when pushing measures to non-existing metrics. This is leveraged by the &lt;a href=&quot;https://github.com/gnocchixyz/collectd-gnocchi&quot;&gt;collectd-gnoccchi&lt;/a&gt; plugin for example.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;The performance of Carbonara based backend has been largely improved. This is not really listed as a change as it&apos;s not user-visible, but an amazing work of profiling and rewriting code from &lt;a href=&quot;http://pandas.pydata.org/&quot;&gt;Pandas&lt;/a&gt; to &lt;a href=&quot;http://www.numpy.org/&quot;&gt;NumPy&lt;/a&gt; has been done. While Pandas is very developer-friendly and generic, using NumPy directly offers way more performance and should decrease &lt;code&gt;gnocchi-metricd&lt;/code&gt; CPU usage by a large factor.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;The storage has been split into two parts: the storage of incoming new measures to be processed, and the storage and archival of aggregated metrics. This allows to use e.g. file to store new measures being sent, and once processed store them into e.g. Ceph. Before that change, all the new measures had to go into Ceph. While there&apos;s no specific driver yet for incoming measures, it&apos;s easy to envision a driver for systems like &lt;a href=&quot;https://redis.io&quot;&gt;Redis&lt;/a&gt; or &lt;a href=&quot;https://memcached.org&quot;&gt;Memcached&lt;/a&gt;.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;A new &lt;a href=&quot;https://aws.amazon.com/s3/&quot;&gt;Amazon S3&lt;/a&gt; driver has been merged. It works in the same way than the file or &lt;a href=&quot;http://docs.openstack.org/developer/swift/&quot;&gt;OpenStack Swift&lt;/a&gt; drivers.&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-logo-old.jpg&quot; alt=&quot;Gnocchi logo&quot; /&gt;&lt;/p&gt;
&lt;p&gt;I will write more about some of these new features in the upcoming weeks, as they are very interesting for Gnocchi&apos;s users.&lt;/p&gt;
&lt;p&gt;We are planning to run a scalability test and benchmarks using the &lt;a href=&quot;http://scalelab.redhat.com/&quot;&gt;ScaleLab&lt;/a&gt; in a few weeks if everything goes as planned. I will obviously share the result here, but we also submitted a&lt;br /&gt;
talk for the next &lt;a href=&quot;https://www.openstack.org/summit/boston-2017/&quot;&gt;OpenStack Summit in Boston&lt;/a&gt; to present the results of our scalability and performance tests – hoping the session will be accepted.&lt;/p&gt;
&lt;p&gt;I will also be talking about Gnocchi &lt;a href=&quot;https://fosdem.org/2017/schedule/event/storing_metrics_gnocchi/&quot;&gt;this Sunday at FOSDEM&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;We don&apos;t have a very determined roadmap for Gnocchi during the next weeks. Sure we do have a few ideas on what we want to implement, but we are also very easily influenced by the requests of our user: therefore feel free to ask for anything!&lt;/p&gt;
</content:encoded><category>gnocchi</category></item><item><title>Attending OpenStack Summit Ocata</title><link>https://julien.danjou.info/blog/openstack-summit-ocata-barcelona-review/</link><guid isPermaLink="true">https://julien.danjou.info/blog/openstack-summit-ocata-barcelona-review/</guid><description>For the last time in 2016, I flew out to the OpenStack Summit in Barcelona, where I had the chance to meet (again) a lot of my fellow OpenStack contributors there.</description><pubDate>Mon, 31 Oct 2016 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;For the last time in 2016, I flew out to the &lt;a href=&quot;https://www.openstack.org/summit/barcelona-2016/&quot;&gt;OpenStack Summit in Barcelona&lt;/a&gt;, where I had the chance to meet (again) a lot of my fellow OpenStack contributors there.&lt;/p&gt;
&lt;h2&gt;How To Work Upstream with OpenStack&lt;/h2&gt;
&lt;p&gt;My week started by giving a talk about &lt;em&gt;How To Work Upstream with OpenStack&lt;/em&gt; where I explained, accompanied by Ryota and Ashiq, to the audience how to contribute upstream to OpenStack. It went well and was well received by the public – you can watch the video below or&lt;br /&gt;
download the slides.&lt;/p&gt;
&lt;h2&gt;Python 3 in telemetry projects&lt;/h2&gt;
&lt;p&gt;I&apos;ve attended a few interesting cross-project sessions, which helped me getting some prioritization for my work during the next few months.&lt;/p&gt;
&lt;p&gt;The Python 3 porting effort is blocked for a while in Nova and Swift for various (mostly non-technical) reasons, while almost all other projects are working correctly. On the other hand, we have committed the telemetry projects to be the first one to drop Python 2 support has soon as it is possible. The next steps are to be sure downstream is ready and enable functional testing in devstack with Python 3.&lt;/p&gt;
&lt;h2&gt;Ceilometer deprecation&lt;/h2&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gordon-gnocchi-talk.jpg&quot; alt=&quot;gordon-gnocchi-talk&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The Ceilometer sessions were really interesting, are we mainly discussed deprecating and removing old crufts that are not or should not be used anymore. The main change will be the depreciation of the Ceilometer API. It has been clear for more than a year that &lt;a href=&quot;http://gnocchi.xyz&quot;&gt;Gnocchi&lt;/a&gt; is the way-to-go to store and provide access to metrics, but we failed at announcing wildly. A lot of the people I talked to during the summit were not aware that the Ceilometer API was not a good pick, and that Gnocchi was the now recommended storage backend. Bad communication from our side – but we are going to fix it as of now.&lt;/p&gt;
&lt;p&gt;We also committed to simplify the current architecture by removing the collector, which has now be made obsolete by the agent based architecture that was implemented during the last development cycles.&lt;/p&gt;
&lt;h2&gt;Aodh alarm timeout&lt;/h2&gt;
&lt;p&gt;We had a feature proposal for a while in Aodh that we postponed for too long already: having timeout triggered after not having seen some events. This seems to be a functionality requested by NFV users – something we want Aodh to cover.&lt;/p&gt;
&lt;p&gt;We spent some time discussing this feature, and now that we all have a clear understanding of the use case, we&apos;ll work on having a clear path to the implementation.&lt;/p&gt;
&lt;p&gt;I&apos;ve also attended a session with the &lt;a href=&quot;https://wiki.openstack.org/wiki/Vitrage&quot;&gt;Vitrage&lt;/a&gt; developers in order to discuss how we could work better together, as they rely on Aodh. It seems there might be some convergence in the future, which would be very welcome. Wait&apos;n see.&lt;/p&gt;
&lt;h2&gt;Gnocchi improvement, past and future&lt;/h2&gt;
&lt;p&gt;The Gnocchi session ran smoothly, and everyone seemed happy with the work we have done so far. We&apos;ve made some impressive improvement in Gnocchi 3.0 – as &lt;a href=&quot;https://julien.danjou.info/blog/2016/gnocchi-3.0-release&quot;&gt;I already covered previously&lt;/a&gt; – and Gordon Chung presented a short talk about the performance difference metered while working on this new version of Gnocchi:&lt;/p&gt;
&lt;p&gt;The return of the InfluxDB driver is on the table, as Sam Morrison proposed a patch for that while back. While it&apos;s not as fast and scalable as other drivers, it offers a good alternative for people having to use it.&lt;/p&gt;
&lt;p&gt;Leandro Reox presented how to do capacity planning using Ceilometer and Gnocchi, presenting the projects at the same time:&lt;/p&gt;
&lt;p&gt;It is pretty impressive to see what they achieved with this project, and I&apos;m looking forward to being able to check how it works inside.&lt;/p&gt;
&lt;h2&gt;PTG and beyond&lt;/h2&gt;
&lt;p&gt;The next meeting is supposed to be the new &lt;a href=&quot;https://www.openstack.org/ptg/&quot;&gt;OpenStack PTG&lt;/a&gt; in February in Atlanta, though we did not request any specific space there. While the team love seeing each other face-to-face every few months, we achieved to follow &lt;a href=&quot;https://julien.danjou.info/blog/foss-projects-management-bad-practice&quot;&gt;all of the guidelines I listed recently&lt;/a&gt; on good open source project management, meaning we are able to work very well asynchronously and remotely. There is no need to put hard requirements on people wanting to participate in our community. Nevertheless, I expect cross-projects discussions that will happen to still concern the OpenStack Telemetry projects.&lt;/p&gt;
&lt;p&gt;In the end, we&apos;re all very happy with our past and future roadmaps and I&apos;m looking forward to achieving our next big milestones with our amazing telemetry team!&lt;/p&gt;
</content:encoded><category>openstack</category><category>gnocchi</category><category>open-source</category><category>talks</category></item><item><title>Gnocchi 3.0 release</title><link>https://julien.danjou.info/blog/gnocchi-3-0-release/</link><guid isPermaLink="true">https://julien.danjou.info/blog/gnocchi-3-0-release/</guid><description>After a few weeks of hard work with the team, here is the new major version of Gnocchi, stamped 3.0.0. It was very challenging, as we wanted to implement a few big changes in it.</description><pubDate>Mon, 03 Oct 2016 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;After a few weeks of hard work with the team, here is the new major version of Gnocchi, stamped &lt;a href=&quot;https://launchpad.net/gnocchi/3.0/3.0.0&quot;&gt;3.0.0&lt;/a&gt;. It was very challenging, as we wanted to implement a few big changes in it.&lt;/p&gt;
&lt;p&gt;Gnocchi is now using &lt;a href=&quot;http://docs.openstack.org/developer/reno/&quot;&gt;reno&lt;/a&gt; to its maximum and you can read &lt;a href=&quot;http://gnocchi.xyz/releasenotes/3.0.html&quot;&gt;the release notes of the 3.0 branch&lt;/a&gt; online. Some notes might be missing as it is our first release with it, but we are making good progress at writing changelogs for most of our user facing and impacting changes.&lt;/p&gt;
&lt;p&gt;Therefore, I&apos;ll only write here about our big major feature that made us bump the major version number.&lt;/p&gt;
&lt;h2&gt;New storage engine&lt;/h2&gt;
&lt;p&gt;And so the most interesting thing that went in the 3.0 release, is the new storage engine that has been built by me and Gordon Chung during those last months. The original approach of writing data in Gnocchi was really naive, so we had an iterative improvement process since version 1.0, and we&apos;re getting close to something very solid.&lt;/p&gt;
&lt;p&gt;This new version leverages several important features which increase performance by a large factor on Ceph (using &lt;code&gt;write(offset)&lt;/code&gt; rather than &lt;code&gt;read()+write()&lt;/code&gt; to append new points), our recommended back-end.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi3_processtime_readwrite_vs_offset.png&quot; alt=&quot;gnocchi3_processtime_readwrite_vs_offset&quot; /&gt;&lt;/p&gt;
&lt;p&gt;To summarize, since most data points are sent sequentially and ordered, we enhanced the data format to profit from that fact and be able to be appended without reading anything. That only works on Ceph though, which provides the needed features.&lt;/p&gt;
&lt;p&gt;We also enabled data compression on all storage drivers by enabling LZ4 compression (&lt;a href=&quot;https://julien.danjou.info/blog/gnocchi-carbonara-timeseries-compression&quot;&gt;see my previous article and research on the subject&lt;/a&gt;), which obviously offers its own set of challenges when using append-only write. The results are tremendous and decrease data usage by a huge factor:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi3_disksize.png&quot; alt=&quot;gnocchi3_disksize&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The rest of the processing pipeline also has been largely improved:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi3_processtime_post.png&quot; alt=&quot;gnocchi3_processtime_post&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi3_processtime_compress_offset.png&quot; alt=&quot;gnocchi3_processtime_compress_offset&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Overall, we&apos;re delighted with the performance improvement we achieved, and we&apos;re looking forward making even better more progress. Gnocchi is now one of the most performing and scalable timeseries databases out there.&lt;/p&gt;
&lt;h2&gt;Upcoming challenges&lt;/h2&gt;
&lt;p&gt;With that big change done, we&apos;re now heading toward a set of more lightweight improvements. Our &lt;a href=&quot;https://bugs.launchpad.net/gnocchi&quot;&gt;bug tracker&lt;/a&gt; is a good place to learn what might be on our mind (check for the &lt;em&gt;wishlist&lt;/em&gt; bugs).&lt;/p&gt;
&lt;p&gt;Improving our API features and offering a better experience for those coming outside of the real of OpenStack are now on my top priority list.&lt;/p&gt;
&lt;p&gt;But let me know if there&apos;s anything you have scratching you, obviously. 😎&lt;/p&gt;
</content:encoded><category>gnocchi</category><category>openstack</category></item><item><title>A retrospective of the OpenStack Telemetry project Newton cycle</title><link>https://julien.danjou.info/blog/retrospective-openstack-telemetry-newton/</link><guid isPermaLink="true">https://julien.danjou.info/blog/retrospective-openstack-telemetry-newton/</guid><description>A few weeks ago, I recorded an interview with Krishnan Raghuram about what was discussed for this development cycle for OpenStack Telemetry at the Austin summit.  It&apos;s interesting to look back at this</description><pubDate>Mon, 05 Sep 2016 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;A few weeks ago, I recorded an interview with Krishnan Raghuram about what was discussed for this development cycle for OpenStack Telemetry at the Austin summit.&lt;/p&gt;
&lt;p&gt;It&apos;s interesting to look back at this video more than 3 months after recording it, and see what actually happened to Telemetry. It turns out that some of the things that I think were going to happen did not happen yet. As the first release candidate version is approaching, it&apos;s very unlikely they happen.&lt;/p&gt;
&lt;p&gt;And on the other side, some new fancy features arrived suddenly without me having a clue about them.&lt;/p&gt;
&lt;p&gt;As far as &lt;strong&gt;Ceilometer&lt;/strong&gt; is concerned, here&apos;s the list of what really happened in terms of user features:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Added full support for SNMP v3 USM model&lt;/li&gt;
&lt;li&gt;Added support for batch measurement in Gnocchi dispatcher&lt;/li&gt;
&lt;li&gt;Set ended_at timestamp in Gnocchi dispatcher&lt;/li&gt;
&lt;li&gt;Allow Swift pollster to specify regions&lt;/li&gt;
&lt;li&gt;Add L3 cache usage and memory bandwidth meters&lt;/li&gt;
&lt;li&gt;Split out the event code (REST API and storage) to a new &lt;strong&gt;Panko&lt;/strong&gt; project&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;And a few other minor things. I planned none of them except Panko (which I was responsible for), and the ones we planned (documentation update, pipeline rework and polling enhancement) did not happen yet.&lt;/p&gt;
&lt;p&gt;For &lt;strong&gt;Aodh&lt;/strong&gt;, we expected to rework the documentation entirely too, and that did not happen either. What we did instead:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Deprecate and disable combination alarms&lt;/li&gt;
&lt;li&gt;Add pagination support in REST API&lt;/li&gt;
&lt;li&gt;Deprecated all non-SQL database store and provide a tool to migrate&lt;/li&gt;
&lt;li&gt;Support batch notification for aodh-notifier&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;It&apos;s definitely a good list of new features for Aodh, still small, but simplifying it, removing technical debt and continuing building momentum around it.&lt;/p&gt;
&lt;p&gt;For &lt;strong&gt;Gnocchi&lt;/strong&gt;, we really had no plan, except maybe a few small features (they&apos;re usually tracked in the Launchpad bug list). It turned out we had some fancy new idea with Gordon Chung on how to boost our storage engine, so we work on that. That kept us busy a few weeks in the end, though the preliminary results look tremendous – so it was definitely worth it. We also have a AWS S3 storage driver on its way.&lt;/p&gt;
&lt;p&gt;I find this exercise interesting, as it really emphasizes how you can&apos;t really control what&apos;s happening in any open source project, where your contributors come and go and work on their own agenda.&lt;/p&gt;
&lt;p&gt;That does not mean we&apos;re dropping the themes and ideas I&apos;ve laid out in that video. We&apos;re still pushing our &quot;documentation is mandatory&quot; policy and improving our &quot;work by default&quot; scenario. It&apos;s just a longer road that we expected.&lt;/p&gt;
</content:encoded><category>openstack</category><category>gnocchi</category></item><item><title>Gnocchi talk at the Paris Monitoring Meetup #6</title><link>https://julien.danjou.info/blog/paris-monitoring-6-gnocchi/</link><guid isPermaLink="true">https://julien.danjou.info/blog/paris-monitoring-6-gnocchi/</guid><description>Last week was the sixth edition of the Paris Monitoring Meetup, where I was invited as a speaker to present and talk about Gnocchi.</description><pubDate>Fri, 27 May 2016 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;Last week was the sixth edition of the &lt;a href=&quot;http://www.meetup.com/Paris-Monitoring/events/230515751/&quot;&gt;Paris Monitoring Meetup&lt;/a&gt;, where I was invited as a speaker to present and talk about &lt;a href=&quot;http://gnocchi.xyz&quot;&gt;Gnocchi&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/paris-monitoring.png&quot; alt=&quot;paris-monitoring&quot; /&gt;&lt;/p&gt;
&lt;p&gt;There was around 50 persons in the room, listening to my presentation of Gnocchi.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/jd-gnocchi-paris-monitoring-meetup-6.jpg&quot; alt=&quot;jd-gnocchi-paris-monitoring-meetup-6&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The talk went fine and I had a few interesting questions and feedback. One interesting point that keeps coming when talking about Gnocchi, is its OpenStack label, which scares away a lot of people. We definitely need to continue explaining that the project work stand-alone has a no dependency on OpenStack, just a great integration with it.&lt;/p&gt;
&lt;p&gt;The &lt;a href=&quot;http://www.monitoring-fr.org/&quot;&gt;Monitoring-fr&lt;/a&gt; organization also &lt;a href=&quot;http://www.monitoring-fr.org/2016/05/meetup-paris-monitoring-6-interview-de-julien-danjou-pour-gnocchi-metric-as-a-service/&quot;&gt;interviewed me&lt;/a&gt; after the meetup about Gnocchi. The interview is in French, obviously. I talk about Gnocchi, what it does, how it does it and why we started the project a couple of years ago. Enjoy, and let me know what you think!&lt;/p&gt;
</content:encoded><category>talks</category><category>monitoring</category><category>gnocchi</category><category>openstack</category></item><item><title>OpenStack Summit Newton from a Telemetry point of view</title><link>https://julien.danjou.info/blog/openstack-summit-newton-austin-telemetry/</link><guid isPermaLink="true">https://julien.danjou.info/blog/openstack-summit-newton-austin-telemetry/</guid><description>It&apos;s again that time of the year, where we all fly out to a different country to chat about OpenStack and what we&apos;ll do during the next 6 months.</description><pubDate>Mon, 02 May 2016 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;It&apos;s again that time of the year, where we all fly out to a different country to chat about OpenStack and what we&apos;ll do during the next 6 months. This time, it was in &lt;a href=&quot;https://en.wikipedia.org/wiki/Austin,_Texas&quot;&gt;Austin, TX&lt;/a&gt; and we chatted about the new Newton release that will be out in October.&lt;/p&gt;
&lt;p&gt;As the &lt;em&gt;Project Team Leader&lt;/em&gt; for the Telemetry project, I set up and animated the week for our team. We had 9 discussion slots of 40 minutes assigned, but finally only used 8. We also, somehow, canceled the contributor team meet-up on the last day, as only a few of us developers were there and available.&lt;/p&gt;
&lt;p&gt;We took &lt;a href=&quot;https://wiki.openstack.org/wiki/Design_Summit/Newton/Etherpads#Telemetry&quot;&gt;a few notes in our Etherpads&lt;/a&gt;, but I think most of them were pretty sparse, as there was nothing really important we talked about. Actually, many topics were already discussed and covered 6 months ago in Tokyo during the previous summit. We just did not have time to implement everything we wanted, so talking over it again would not have been of a great help.&lt;/p&gt;
&lt;h2&gt;Reference architecture&lt;/h2&gt;
&lt;p&gt;Unfortunately, nor Gordon Chung nor the &lt;a href=&quot;https://osic.org/&quot;&gt;OpenStack Innovation Center&lt;/a&gt; had time to run the tests and benchmarks they wanted to run before the summit. We still discussed their plan to run tests and benchmark of the whole Telemetry suite (Ceilometer, Gnocchi &amp;amp; Aodh). They should run their tests for 3 weeks, no more, in a few weeks. The window to run tests being narrow, they want to be sure they are prepared, and will reach to us for help, ideas, and validation.&lt;/p&gt;
&lt;p&gt;I&apos;ve also requested them to, if possible, provide us some profiling (e.g. cProfile) data so we can have better knowledge of the area we can optimize.&lt;/p&gt;
&lt;h2&gt;Gnocchi, next steps&lt;/h2&gt;
&lt;p&gt;This session was particularly smooth since most people in the room were not up-to-date with Gnocchi 2.1. Some people expressed concerned about the InfluxDB driver removal, though they were not aware of the bugs it had, and that Gnocchi was actually performing better – so they may very likely be testing Gnocchi directly instead.&lt;/p&gt;
&lt;p&gt;No particular fancy feature was requested, only a few bugs and ideas noted on Launchpad were discussed.&lt;/p&gt;
&lt;h2&gt;Enhancing Ceilometer polling&lt;/h2&gt;
&lt;p&gt;This session was not particularly productive, as everything was we wanted to discuss was already on the Etherpad from… Tokyo, 6 months ago. It turns out nobody had time to pursue this project, so we&apos;ll see what happens. There&apos;s definitely some work to do to pursue our goal of splitting the pipeline definition into smaller files.&lt;/p&gt;
&lt;h2&gt;Aodh roadmap &amp;amp; improvements&lt;/h2&gt;
&lt;p&gt;First, we decided to definitely kill the combination alarm in the future, in favor of the new composite alarms definition that we like better.&lt;/p&gt;
&lt;p&gt;We should switch to &lt;a href=&quot;http://docs.openstack.org/developer/python-openstackclient/&quot;&gt;OpenStackClient&lt;/a&gt; in the future for &lt;a href=&quot;http://docs.openstack.org/developer/python-aodhclient/&quot;&gt;aodhclient&lt;/a&gt;. The OSC team indicated they are willing to provide a way to keep the &quot;aodh&quot; CLI command on its own, which is something that blocked us to move to OSC.&lt;/p&gt;
&lt;p&gt;A bunch of people indicated that had support for alarms CRUD in the Horizon dashboard. They should work together with the Horizon team to complete what has been started in Horizon recently to add Aodh support.&lt;/p&gt;
&lt;h2&gt;Ceilometer splitting&lt;/h2&gt;
&lt;p&gt;A year ago, we decided to split Ceilometer and its alarm feature: Aodh was born. We did discuss doing it again 6 months ago, but nothing happened as we already had so many stuff on our plate.&lt;/p&gt;
&lt;p&gt;As far as I&apos;m concerned, I think it&apos;s now time to split some Ceilometer functionality again, so I&apos;m going to do that this time with the event part. Gordon found a name, and this new project will be named &lt;em&gt;Panko&lt;/em&gt;.&lt;/p&gt;
&lt;h2&gt;Documentation&lt;/h2&gt;
&lt;p&gt;We have then discussed our documentation. Users present in the room were particularly happy with the Gnocchi policy that we apply since the beginning: no doc = no merge of your patch. The consensus is to move forward on this policy for all Telemetry projects, especially since it&apos;s now clear that the documentation team is not going to help us more. Ildikó, our documentation&lt;br /&gt;
wizard, will take care of making some links between the official OpenStack documentation and our projects, avoid content duplication.&lt;/p&gt;
&lt;p&gt;For this cycle, my personal plan is to document Aodh up to roughly 80 %, and then force that policy on newly implemented changes.&lt;/p&gt;
&lt;h2&gt;Events management&lt;/h2&gt;
&lt;p&gt;The event management part of Ceilometer and API (soon to be split in its own project as stated above) was discussed in this session. Nothing really exciting coming here, as nobody is willing to enhance it for now. Which, again, makes it a great candidate for splitting it out of Ceilometer.&lt;/p&gt;
&lt;h2&gt;Vitrage&lt;/h2&gt;
&lt;p&gt;The last session was dedicated to &lt;a href=&quot;https://wiki.openstack.org/wiki/Vitrage&quot;&gt;Vitrage&lt;/a&gt;, a root cause analysis tool built on OpenStack. The Vitrage team had a few features that they wanted to see in Aodh, so we discussed that at length. Notably, more support for sending notifications on events (alarm creation, deletion…) should be added in this next release.&lt;/p&gt;
&lt;p&gt;Also, a new alarm type that would be entirely managed and triggered over HTTP would be very useful for external projects such as Vitrage. We&apos;ll try to make that happen during this cycle too.&lt;/p&gt;
&lt;h2&gt;Talks&lt;/h2&gt;
&lt;p&gt;There were a few interesting talks about our telemetry projects during this summit, among other I highly recommend watching:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=W5KT5GJKJw8&quot;&gt;OpenStack Ceilometer with Gnocchi and Aodh Feature&lt;/a&gt;, where Amol and Paul from Ericsson explain what Gnocchi and Aodh do and how they work, and then help people deploy it on their lab.&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=BdebhsBFEJs&quot;&gt;DPDK, Collectd &amp;amp; Ceilometer The Missing Link&lt;/a&gt;, where Ryota Mibu, one of the contributor to Aodh explains why he implemented the event alarm feature&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=-K8NI38LPtU&quot;&gt;Showback &amp;amp; Chargeback!! OpenStack Gnocchi + Cloudkitty as a Whole Billing System&lt;/a&gt;, where Maximiliano Venesio (Nubeliu) and Stéphane Albert (Objectif Libre) talk about how they built an amazing scalable billing solution using &lt;a href=&quot;http://gnocchi.xyz&quot;&gt;Gnocchi&lt;/a&gt; and &lt;a href=&quot;https://wiki.openstack.org/wiki/CloudKitty&quot;&gt;CloudKitty&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://www.youtube.com/watch?v=0Q8pfbwxMb8&quot;&gt;Using Ceilometer Data for Effective Witch-Hunting&lt;/a&gt;, where Mike explain how Overstock.com leveraged Ceilometer to track anomalies in their cloud.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;All of this should keep me and the team busy for the next cycle. If you have any question about what has been discussed or the future of our projects, don&apos;t hesitate to leave a comment or ask us on the &lt;a href=&quot;http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev&quot;&gt;OpenStack development mailing list&lt;/a&gt;.&lt;/p&gt;
</content:encoded><category>openstack</category><category>gnocchi</category></item><item><title>Gnocchi 2.1 release</title><link>https://julien.danjou.info/blog/gnocchi-2-1-release/</link><guid isPermaLink="true">https://julien.danjou.info/blog/gnocchi-2-1-release/</guid><description>A little less than 2 months after our latest major release, here is the new minor version of Gnocchi, stamped 2.1.0.</description><pubDate>Wed, 13 Apr 2016 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;A little less than 2 months after our latest major release, here is the new minor version of Gnocchi, stamped &lt;a href=&quot;https://launchpad.net/gnocchi/2.1/2.1.0&quot;&gt;2.1.0&lt;/a&gt;. It was a smooth release, but with one major feature implemented by my fellow fantastic developer Mehdi Abaakouk: the ability to create resource types dynamically.&lt;/p&gt;
&lt;h2&gt;Resource types REST API&lt;/h2&gt;
&lt;p&gt;This new version of Gnocchi offers the long-awaited ability to create resource types dynamically. What does that mean? Well, until version 2.0, the resources that you were able to create in Gnocchi had a particular type that was defined in the code: instance, volume, SNMP host, Swift account, etc. All of them were tied to OpenStack, since it was our primary use case.&lt;/p&gt;
&lt;p&gt;Now, &lt;a href=&quot;http://gnocchi.xyz/rest.html#resource-types&quot;&gt;the API allows to create resource types dynamically&lt;/a&gt;. This means you can create your own custom types to describe your own architecture. You then can exploit the same features that were offered before: history of your resources, searching through them, associating metrics, etc!&lt;/p&gt;
&lt;h2&gt;Performances improvement&lt;/h2&gt;
&lt;p&gt;We did some profiling on Gnocchi, and some benchmarks, and with the help of my fellow developer Gordon Chung, improved the metric performances.&lt;/p&gt;
&lt;p&gt;The API speed improved a bit, and I&apos;ve measured the Gnocchi API endpoint of being able to ingest up to 190k measures/s with only one node (the same as used in my &lt;a href=&quot;https://julien.danjou.info/blog/gnocchi-benchmarks&quot;&gt;previous benchmark&lt;/a&gt;) using &lt;a href=&quot;https://uwsgi-docs.readthedocs.org/&quot;&gt;uwsgi&lt;/a&gt;, so a 50 % improvement. The time required to compute aggregation on new measures is now also metered and displayed in the &lt;code&gt;gnocchi-metricd&lt;/code&gt; log in debug mode. Handy to have an idea of how fast your measures are treated.&lt;/p&gt;
&lt;h2&gt;Ceph backend optimization&lt;/h2&gt;
&lt;p&gt;The Ceph back-end has been improved again by Mehdi. We&apos;re now relying on OMAP rather than xattr for finer grained control and better performance.&lt;/p&gt;
&lt;p&gt;We already have a few new features being prepared for our next release, so stay tuned! And if you have any suggestion, feel free to say a word.&lt;/p&gt;
</content:encoded><category>gnocchi</category><category>openstack</category></item><item><title>Gnocchi 2.0 release</title><link>https://julien.danjou.info/blog/gnocchi-2-0-release/</link><guid isPermaLink="true">https://julien.danjou.info/blog/gnocchi-2-0-release/</guid><description>Gnocchi 2.0 is out with major new features including a Grafana datasource, Ceph storage driver, and a revamped REST API.</description><pubDate>Fri, 19 Feb 2016 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;A little more than 3 months after our latest minor release, here is the new major version of Gnocchi, stamped &lt;a href=&quot;https://launchpad.net/gnocchi/2.0/2.0.0&quot;&gt;2.0.0&lt;/a&gt;. It contains a lot of new and exciting features, and I&apos;d like to talk about some of them to celebrate!&lt;/p&gt;
&lt;p&gt;You may notice that this release happens in the middle of the OpenStack release cycle. Indeed, Gnocchi does not follow that 6-months cycle, and we release whenever our code is ready. That forces us to have a more iterative approach, less disruptive for other projects and allow us to achieve a higher velocity. Applying the good old mantra &lt;em&gt;release early, release often&lt;/em&gt;.&lt;/p&gt;
&lt;h2&gt;Documentation&lt;/h2&gt;
&lt;p&gt;This version features a large documentation update. Gnocchi is still the only OpenStack server project that implements a &quot;no doc, no merge&quot; policy, meaning any code must come with the documentation addition or change included in the patch. The full documentation is included in the source code and available online at &lt;a href=&quot;http://gnocchi.xyz/&quot;&gt;gnocchi.xyz&lt;/a&gt;.&lt;/p&gt;
&lt;h2&gt;Data split &amp;amp; compression&lt;/h2&gt;
&lt;p&gt;I&apos;ve already covered this change extensively in &lt;a href=&quot;https://julien.danjou.info/blog/gnocchi-carbonara-timeseries-compression&quot;&gt;my last blog about timeseries compression&lt;/a&gt;. Long story short, Gnocchi now splits timeseries archives in small chunks that are compressed, increasing speed and decreasing data size.&lt;/p&gt;
&lt;h2&gt;Measures batching support&lt;/h2&gt;
&lt;p&gt;Gnocchi now supports batching, which allow submitting several measures for different metric in a single request. This is especially useful in the context where your application tends to cache metrics for a while and is able to send them in a batch. Usage is &lt;a href=&quot;http://gnocchi.xyz/rest.html#measures-batching&quot;&gt;fully documented for the REST API&lt;/a&gt;.&lt;/p&gt;
&lt;h2&gt;Group by support in aggregation&lt;/h2&gt;
&lt;p&gt;One of the most demanded features was the ability to do measure aggregation no resource, using a group by type query. This is now possible using the &lt;a href=&quot;http://gnocchi.xyz/rest.html#aggregation-across-metrics&quot;&gt;new &lt;code&gt;groupby&lt;/code&gt; parameter to aggregation queries&lt;/a&gt;.&lt;/p&gt;
&lt;h2&gt;Ceph backend optimization&lt;/h2&gt;
&lt;p&gt;We improved the Ceph back-end a lot. Mehdi Abaakouk wrote a new Python binding for Ceph, called &lt;a href=&quot;https://github.com/sileht/pycradox&quot;&gt;Cradox&lt;/a&gt;, that is going to replace the current Python rados module in the subsequent Ceph releases. Gnocchi makes usage of this new module to speed things up, making the Ceph based driver really, really faster than before. We also implemented asynchronous data deletion, which improves performance a bit.&lt;/p&gt;
&lt;p&gt;The next step will be to run some new benchmarks &lt;a href=&quot;https://julien.danjou.info/blog/gnocchi-benchmarks&quot;&gt;like I did a few months ago&lt;/a&gt; and compare with the Gnocchi 1.3 series. Stay tuned!&lt;/p&gt;
</content:encoded><category>gnocchi</category><category>openstack</category></item><item><title>Timeseries storage and data compression</title><link>https://julien.danjou.info/blog/gnocchi-carbonara-timeseries-compression/</link><guid isPermaLink="true">https://julien.danjou.info/blog/gnocchi-carbonara-timeseries-compression/</guid><description>The first major version of the scalable timeserie database I work on, Gnocchi was a released a few months ago. In this first iteration, it took a rather naive approach to data storage.</description><pubDate>Mon, 15 Feb 2016 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;The first major version of the scalable timeserie database I work on, &lt;a href=&quot;http://gnocchi.xyz&quot;&gt;Gnocchi&lt;/a&gt; was a released a few months ago. In this first iteration, it took a rather naive approach to data storage. We had little ideas about if and how our distributed back-ends were going to be heavily used, so we stuck to the code of the first proof-of-concept written a couple of years ago.&lt;/p&gt;
&lt;p&gt;Recently we got more feedbacks from our users, ran a few &lt;a href=&quot;https://julien.danjou.info/blog/gnocchi-benchmarks&quot;&gt;benchmarks&lt;/a&gt;. That gave us enough feedback to start investigating in improving our storage strategy.&lt;/p&gt;
&lt;h2&gt;Data split&lt;/h2&gt;
&lt;p&gt;Up to Gnocchi 1.3, all data for a single metric are stored in a single gigantic file per aggregation method (&lt;em&gt;min&lt;/em&gt;, &lt;em&gt;max&lt;/em&gt;, &lt;em&gt;average&lt;/em&gt;…). This means that the file can grow to several megabytes in size, which make it slow to manipulate. For the next version of Gnocchi, our first work has been to rework that storage and split the data into smaller parts.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-carbonara-split.png&quot; alt=&quot;gnocchi-carbonara-split&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The diagram above shows how data are organized inside Gnocchi. Until version 1.3, there would have been only one file for each aggregation methods.&lt;/p&gt;
&lt;p&gt;In the upcoming 2.0 version, Gnocchi will split all these data into smaller parts, where each data split is stored in a file/object. This allows to manipulate smaller pieces of data and to increase the parallelism of the CRUD operations on the back-end – leading to large speed improvement.&lt;/p&gt;
&lt;p&gt;In order to split timeseries into several chunks, Gnocchi defines a maximum number of N points to keep per chunk, to limit their maximum size. It then defines a hash function that produces a non-unique key for any timestamp. It makes it easy to find in which chunk any timestamp should be stored or retrieved.&lt;/p&gt;
&lt;h2&gt;Data compression&lt;/h2&gt;
&lt;p&gt;Up to Gnocchi 1.3, the data stored for each metric is simply serialized using &lt;a href=&quot;http://msgpack.org&quot;&gt;msgpack&lt;/a&gt;, a fast and small serialization format. Though, this format does not provide any compression. That means that storing data points needs 8 bytes for a timestamp (64 bits timestamp with nanosecond precision) and 8 bytes for a value (64 bits double-precision floating-point), plus some overhead (extra information and &lt;em&gt;msgpack&lt;/em&gt; itself).&lt;/p&gt;
&lt;p&gt;After looking around on how to compress all these measures, I stumbled upon a paper from some &lt;a href=&quot;http://facebook&quot;&gt;Facebook&lt;/a&gt; engineers called about Gorilla, their in-memory timeserie database, entitled &quot;&lt;em&gt;&lt;a href=&quot;http://www.vldb.org/pvldb/vol8/p1816-teller.pdf&quot;&gt;Gorilla: A Fast, Scalable, In-Memory Time Series Database&lt;/a&gt;&lt;/em&gt;&quot;. For reference, part of this encoding is also used by &lt;a href=&quot;https://docs.influxdata.com/influxdb/v0.9/concepts/storage_engine/&quot;&gt;InfluxDB&lt;/a&gt; in its new storage engine.&lt;/p&gt;
&lt;p&gt;The first technique I implemented is easy enough, and it&apos;s inspired from delta-of-delta encoding. Instead of storing each timestamp for each data point, and since all the data points are aggregated on a regular interval, we transpose points to be the time difference divided by the interval. For example, the suite of timestamps &lt;code&gt;timestamps = [41230, 41235, 41240, 41250, 41255]&lt;/code&gt; is encoded into &lt;code&gt;timestamps = [41230, 1, 1, 2, 1], interval = 5&lt;/code&gt;. This allows regular compression algorithms to reduce the size of the integer list using &lt;a href=&quot;https://en.wikipedia.org/wiki/Run-length_encoding&quot;&gt;run-length encoding&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;To actually compress the values, I tried two different algorithms:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/LZ4_(compression_algorithm)&quot;&gt;LZ4&lt;/a&gt;, a fast compression/decompression algorithm&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;The XOR based compression scheme described in the Gorilla paper mentioned above – that &lt;a href=&quot;https://gist.github.com/jd/b0aa5cbfa42f4eb23eb9&quot;&gt;I had to implement myself&lt;/a&gt;. For reference, it also exists a &lt;a href=&quot;http://golang.org&quot;&gt;Go&lt;/a&gt; implementation in &lt;a href=&quot;https://github.com/dgryski/go-tsz&quot;&gt;go-tsz&lt;/a&gt;.&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I then benchmarked these solutions:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-carbonara-compression-speed.png&quot; alt=&quot;gnocchi-carbonara-compression-speed&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The XOR algorithm implemented in Python is pretty slow, compared to LZ4. Truth is that &lt;a href=&quot;https://github.com/steeve/python-lz4&quot;&gt;python-lz4&lt;/a&gt; is fully implemented in C, which makes it fast. I&apos;ve profiled my XOR implementation in Python, to discover that one operation took 20 % of the time: &lt;code&gt;count_lead_and_trail_zeroes&lt;/code&gt;, which is in charge of counting the number of leading and trailing zeroes in a binary number.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-carbonara-xor-profiling.png&quot; alt=&quot;gnocchi-carbonara-xor-profiling&quot; /&gt;&lt;/p&gt;
&lt;p&gt;I tried 2 Python implementations of the same algorithm (and submitted them to my friend and Python developer &lt;a href=&quot;http://haypo-notes.readthedocs.org/&quot;&gt;Victor Stinner&lt;/a&gt; by the way).&lt;/p&gt;
&lt;p&gt;The first version using string search with &lt;code&gt;.index()&lt;/code&gt; is 10× faster than the second one that only do integer computation. Ah, Python… As Victor explained, each Python operation is slow and there&apos;s a lot in the second version, whereas &lt;code&gt;.index()&lt;/code&gt; is implemented in C and really well optimized and only needs 2 Python operations.&lt;/p&gt;
&lt;p&gt;Finally, I ended up optimizing that code by leveraging &lt;a href=&quot;https://cffi.readthedocs.org/en/latest/&quot;&gt;cffi&lt;/a&gt; to use directly &lt;code&gt;ffsll()&lt;/code&gt; and &lt;code&gt;flsll()&lt;/code&gt;. That decreased the run-time of &lt;code&gt;count_lead_and_trail_zeroes&lt;/code&gt; by 45 %, making the entire XOR compression code speed increased by a small 7 %. This is not enough to catch up with LZ4 speed. At this stage, the only solution to achieve a high-speed would probably to go with a full C implementation.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-carbonara-compression-size.png&quot; alt=&quot;gnocchi-carbonara-compression-size&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Considering the compression ratio of the different algorithms, they are pretty much identical. The worst case scenario (random values) for LZ4 compress down to 9 bytes per data point, whereas XOR can go down to 7.38 bytes per data point. In general XOR encoding beats LZ4 by 15 %, except for cases where all values are 0 or 1. However, LZ4 is faster than XOR by a factor of 4×-70× depending on cases.&lt;/p&gt;
&lt;p&gt;That means that we&apos;ll use LZ4 for data compression in Gnocchi 2.0. It&apos;s possible that we could achieve as fast compression/decompression algorithm, but I don&apos;t think it&apos;s worth the effort right now – it&apos;d represent a lot of code to write and to maintain.&lt;/p&gt;
</content:encoded><category>gnocchi</category><category>python</category></item><item><title>Profiling Python using cProfile: a concrete case</title><link>https://julien.danjou.info/blog/guide-to-python-profiling-cprofile-concrete-case-carbonara/</link><guid isPermaLink="true">https://julien.danjou.info/blog/guide-to-python-profiling-cprofile-concrete-case-carbonara/</guid><description>Writing programs is fun, but making them fast can be a pain. Python programs are no exception to that, but the basic profiling toolchain is actually not that complicated to use. Here, I would like to</description><pubDate>Mon, 16 Nov 2015 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;Writing programs is fun, but making them fast can be a pain. Python programs are no exception to that, but the basic profiling toolchain is actually not that complicated to use. Here, I would like to show you how you can quickly profile and analyze your Python code to find what part of the code you should optimize.&lt;/p&gt;
&lt;h2&gt;What&apos;s profiling?&lt;/h2&gt;
&lt;p&gt;Profiling a Python program is doing a dynamic analysis that measures the execution time of the program and everything that compose it. That means measuring the time spent in each of its functions. This will give you data about where your program is spending time, and what area might be worth optimizing.&lt;/p&gt;
&lt;p&gt;It&apos;s a very interesting exercise. Many people focus on local optimizations, such as determining e.g. which of the Python functions &lt;code&gt;range&lt;/code&gt; or &lt;code&gt;xrange&lt;/code&gt; is going to be faster. It turns out that knowing which one is faster may never be an issue in your program, and that the time gained by one of the functions above might not be worth the time you spend researching that, or arguing about it with your colleague.&lt;/p&gt;
&lt;p&gt;Trying to blindly optimize a program without measuring where it is actually spending its time is a useless exercise. Following your guts alone is not always sufficient.&lt;/p&gt;
&lt;p&gt;There are many types of profiling, as there are many things you can measure. In this exercise, we&apos;ll focus on CPU utilization profiling, meaning the time spent by each function executing instructions. Obviously, we could do many more kind of profiling and optimizations, such as memory profiling which would measure the memory used by each piece of code – something I talk about in &lt;a href=&quot;https://thehackerguidetopython.com&quot;&gt;The Hacker&apos;s Guide to Python&lt;/a&gt;.&lt;/p&gt;
&lt;h2&gt;cProfile&lt;/h2&gt;
&lt;p&gt;Since Python 2.5, Python provides a C module called &lt;em&gt;&lt;a href=&quot;https://docs.python.org/2/library/profile.html&quot;&gt;cProfile&lt;/a&gt;&lt;/em&gt; which has a reasonable overhead and offers a good enough feature set. The basic usage goes down to:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;&amp;gt;&amp;gt;&amp;gt; import cProfile
&amp;gt;&amp;gt;&amp;gt; cProfile.run(&apos;2 + 2&apos;)
         2 function calls in 0.000 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 &amp;lt;string&amp;gt;:1(&amp;lt;module&amp;gt;)
        1    0.000    0.000    0.000    0.000 {method &apos;disable&apos; of &apos;_lsprof.Profiler&apos; objects}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Though you can also run a script with it, which turns out to be handy:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ python -m cProfile -s cumtime lwn2pocket.py
         72270 function calls (70640 primitive calls) in 4.481 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.004    0.004    4.481    4.481 lwn2pocket.py:2(&amp;lt;module&amp;gt;)
        1    0.001    0.001    4.296    4.296 lwn2pocket.py:51(main)
        3    0.000    0.000    4.286    1.429 api.py:17(request)
        3    0.000    0.000    4.268    1.423 sessions.py:386(request)
      4/3    0.000    0.000    3.816    1.272 sessions.py:539(send)
        4    0.000    0.000    2.965    0.741 adapters.py:323(send)
        4    0.000    0.000    2.962    0.740 connectionpool.py:421(urlopen)
        4    0.000    0.000    2.961    0.740 connectionpool.py:317(_make_request)
        2    0.000    0.000    2.675    1.338 api.py:98(post)
       30    0.000    0.000    1.621    0.054 ssl.py:727(recv)
       30    0.000    0.000    1.621    0.054 ssl.py:610(read)
       30    1.621    0.054    1.621    0.054 {method &apos;read&apos; of &apos;_ssl._SSLSocket&apos; objects}
        1    0.000    0.000    1.611    1.611 api.py:58(get)
        4    0.000    0.000    1.572    0.393 httplib.py:1095(getresponse)
        4    0.000    0.000    1.572    0.393 httplib.py:446(begin)
       60    0.000    0.000    1.571    0.026 socket.py:410(readline)
        4    0.000    0.000    1.571    0.393 httplib.py:407(_read_status)
        1    0.000    0.000    1.462    1.462 pocket.py:44(wrapped)
        1    0.000    0.000    1.462    1.462 pocket.py:152(make_request)
        1    0.000    0.000    1.462    1.462 pocket.py:139(_make_request)
        1    0.000    0.000    1.459    1.459 pocket.py:134(_post_request)
[…]
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This prints out all the function called, with the time spend in each and the number of times they have been called.&lt;/p&gt;
&lt;h3&gt;Advanced visualization with KCacheGrind&lt;/h3&gt;
&lt;p&gt;While being useful, the output format is very basic and does not make easy to grab knowledge for complete programs. For more advanced visualization, I leverage &lt;a href=&quot;https://kcachegrind.github.io/html/Home.html&quot;&gt;KCacheGrind&lt;/a&gt;. If you did any C programming and profiling these last years, you may have used it as it is primarily designed as front-end for &lt;a href=&quot;http://valgrind.org/&quot;&gt;Valgrind&lt;/a&gt; generated call-graphs.&lt;/p&gt;
&lt;p&gt;In order to use, you need to generate a &lt;em&gt;cProfile&lt;/em&gt; result file, then convert it to KCacheGrind format. To do that, I use &lt;em&gt;&lt;a href=&quot;https://pypi.python.org/pypi/pyprof2calltree&quot;&gt;pyprof2calltree&lt;/a&gt;&lt;/em&gt;.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ python -m cProfile -o myscript.cprof myscript.py
$ pyprof2calltree -k -i myscript.cprof
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;And the KCacheGrind window magically appears!&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/kcachegrind.png&quot; alt=&quot;kcachegrind&quot; /&gt;&lt;/p&gt;
&lt;h2&gt;Concrete case: Carbonara optimization&lt;/h2&gt;
&lt;p&gt;I was curious about the performances of &lt;a href=&quot;https://git.openstack.org/cgit/openstack/gnocchi/tree/gnocchi/carbonara.py&quot;&gt;Carbonara&lt;/a&gt;, the small timeseries library I wrote for &lt;a href=&quot;http://launchpad.net/gnocchi&quot;&gt;Gnocchi&lt;/a&gt;. I decided to do some basic profiling to see if there was any obvious optimization to do.&lt;/p&gt;
&lt;p&gt;In order to profile a program, you need to run it. But running the whole program in profiling mode can generate &lt;em&gt;a lot&lt;/em&gt; of data that you don&apos;t care about, and adds noise to what you&apos;re trying to understand. Since Gnocchi has thousands of unit tests and a few for Carbonara itself, I decided to profile the code used by these unit tests, as it&apos;s a good reflection of basic features of the library.&lt;/p&gt;
&lt;p&gt;Note that this is a good strategy for a curious and naive first-pass profiling.&lt;br /&gt;
There&apos;s no way that you can make sure that the hotspots you will see in the unit tests are the actual hotspots you will encounter in production. Therefore, a profiling in conditions and with a scenario that mimics what&apos;s seen in production is often a necessity if you need to push your program optimization further and want to achieve perceivable and valuable gain.&lt;/p&gt;
&lt;p&gt;I activated &lt;em&gt;cProfile&lt;/em&gt; using the method described above, creating a &lt;code&gt;cProfile.Profile&lt;/code&gt; object around my tests (I actually &lt;a href=&quot;https://github.com/testing-cabal/testtools/pull/163&quot;&gt;started to implement that in testtools&lt;/a&gt;). I then run &lt;em&gt;KCacheGrind&lt;/em&gt; as described above. Using &lt;em&gt;KCacheGrind&lt;/em&gt;, I generated the following figures.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/kcachegrind-carbonara-old-list.png&quot; alt=&quot;kcachegrind-carbonara-old-list&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The test I profiled here is called &lt;code&gt;test_fetch&lt;/code&gt; and is pretty easy to understand: it puts data in a timeserie object, and then fetch the aggregated result. The above list shows that 88 % of the ticks are spent in &lt;code&gt;set_values&lt;/code&gt; (44 ticks over 50). This function is used to insert values into the timeserie, not to fetch the values. That means that it&apos;s really slow to insert data, and pretty fast to actually retrieve them.&lt;/p&gt;
&lt;p&gt;Reading the rest of the list indicates that several functions share the rest of the ticks, &lt;code&gt;update&lt;/code&gt;, &lt;code&gt;_first_block_timestamp&lt;/code&gt;, &lt;code&gt;_truncate&lt;/code&gt;, &lt;code&gt;_resample&lt;/code&gt;, etc. Some of the functions in the list are not part of Carbonara, so there&apos;s no point in looking to optimize them. The only thing that can be optimized is, sometimes, the number of times they&apos;re called.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/kcachegrind-carbonara-old-graph.png&quot; alt=&quot;kcachegrind-carbonara-old-graph&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The call graph gives me a bit more insight about what&apos;s going on here. Using my knowledge about how Carbonara works, I don&apos;t think that the whole stack on the left for &lt;code&gt;_first_block_timestamp&lt;/code&gt; makes much sense. This function is supposed to find the first timestamp for an aggregate, e.g. with a timestamp of 13:34:45 and a period of 5 minutes, the function should return 13:30:00. The way it works currently is by calling the &lt;code&gt;resample&lt;/code&gt; function from Pandas on a timeserie with only one element, but that seems to be very slow. Indeed, currently this function represents 25 % of the time spent by &lt;code&gt;set_values&lt;/code&gt; (11 ticks on 44).&lt;/p&gt;
&lt;p&gt;Fortunately, I recently added a small function called &lt;code&gt;_round_timestamp&lt;/code&gt; that does exactly what &lt;code&gt;_first_block_timestamp&lt;/code&gt; needs that without calling any Pandas function, so no &lt;code&gt;resample&lt;/code&gt;. So I ended up rewriting that function this way:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;     def _first_block_timestamp(self):
-        ts = self.ts[-1:].resample(self.block_size)
-        return (ts.index[-1] - (self.block_size * self.back_window))
+        rounded = self._round_timestamp(self.ts.index[-1], self.block_size)
+        return rounded - (self.block_size * self.back_window)
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;And then I re-run the exact same test to compare the output of &lt;em&gt;cProfile&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/kcachegrind-carbonara-new-list.png&quot; alt=&quot;kcachegrind-carbonara-new-list&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The list of function seems quite different this time. The number of time spend used by &lt;code&gt;set_values&lt;/code&gt; dropped from 88 % to 71 %.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/kcachegrind-carbonara-new-graph.png&quot; alt=&quot;kcachegrind-carbonara-new-graph&quot; /&gt;&lt;/p&gt;
&lt;p&gt;The call stack for &lt;code&gt;set_values&lt;/code&gt; shows that pretty well: we can&apos;t even see the &lt;code&gt;_first_block_timestamp&lt;/code&gt; function as it is so fast that it totally disappeared from the display. It&apos;s now being considered insignificant by the profiler.&lt;/p&gt;
&lt;p&gt;So we just speed up the whole insertion process of values into Carbonara by a nice 25 % in a few minutes. Not that bad for a first naive pass, right?&lt;/p&gt;
&lt;p&gt;If you want to know more, I wrote a whole chapter about optimizing code in &lt;a href=&quot;https://scaling-python.com&quot;&gt;Scaling Python&lt;/a&gt;. Check it out!&lt;/p&gt;
</content:encoded><category>python</category><category>gnocchi</category></item><item><title>Gnocchi 1.3.0 release</title><link>https://julien.danjou.info/blog/gnocchi-1-3-0-released/</link><guid isPermaLink="true">https://julien.danjou.info/blog/gnocchi-1-3-0-released/</guid><description>Finally, Gnocchi 1.3.0 is out. This is our final release, more or less matching the OpenStack 6 months schedule, that concludes the Liberty development cycle.</description><pubDate>Wed, 04 Nov 2015 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;Finally, &lt;a href=&quot;https://launchpad.net/gnocchi/trunk/1.3.0&quot;&gt;Gnocchi 1.3.0&lt;/a&gt; is out. This is our final release, more or less matching the OpenStack 6 months schedule, that concludes the Liberty development cycle.&lt;/p&gt;
&lt;p&gt;This release was supposed to be released a few weeks earlier, but our integration test got completely blocked for several days just the week before the OpenStack Mitaka summit.&lt;/p&gt;
&lt;h2&gt;New website&lt;/h2&gt;
&lt;p&gt;We build a new dedicated website for Gnocchi at &lt;a href=&quot;http://gnocchi.xyz&quot;&gt;gnocchi.xyz&lt;/a&gt;. We want to promote Gnocchi outside of the &lt;a href=&quot;http://openstack.org&quot;&gt;OpenStack&lt;/a&gt; bubble, as it a useful timeseries database on its own that can work without the rest of the stack. We&apos;ll try to improve the documentation. If you&apos;re curious, feel free to check it out and report anything you miss!&lt;/p&gt;
&lt;h2&gt;The speed bump&lt;/h2&gt;
&lt;p&gt;Obviously, if it was a bug in Gnocchi that we have hit, it would have been quick to fix. However, we found &lt;a href=&quot;https://bugs.launchpad.net/python-keystoneclient/+bug/1508424&quot;&gt;a nasty bug&lt;/a&gt; in Swift caused by the evil monkey-patching of Eventlet (once again) blended with a mixed usage of native threads and Eventlet threads in Swift. Shake all of that, and you got yourself pretty race conditions when using the Keystone middleware authentication.&lt;/p&gt;
&lt;p&gt;In the meantime, we disabled Swift multi-threading by using mod_wsgi instead of Eventlet in devstack.&lt;/p&gt;
&lt;h2&gt;New features&lt;/h2&gt;
&lt;p&gt;So what&apos;s new in this new shiny release? A few interesting things:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;Metric deletion is now asynchronous. That&apos;s not the most used feature in the REST API – weirdly people do not often delete metrics – but it&apos;s now way faster and reliable by being asynchronous. &lt;em&gt;Metricd&lt;/em&gt; is now in charge of cleaning up things up.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Speed improvement. We are now confident to be even more faster than in the &lt;a href=&quot;https://julien.danjou.info/blog/gnocchi-benchmarks&quot;&gt;latest benchmarks I run&lt;/a&gt; (around 1.5-2× faster), which makes Gnocchi &lt;em&gt;really&lt;/em&gt; fast with its native storage back-ends. We profiled and optimized Carbonara and the REST API data validation.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Improve &lt;em&gt;metricd&lt;/em&gt; status report. It now reports the size of the backlog of the whole cluster both in its log and via the REST API. Easy monitoring!&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Ceph drivers enhancement. We had people testing the Ceph drivers in production, so we made a few changes and fixes to it to make it more solid.&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;And that&apos;s all we did in the last couple of months. We have a lot of things on the roadmap that are pretty exciting, and I&apos;ll sure talk about them in the next weeks.&lt;/p&gt;
</content:encoded><category>gnocchi</category><category>openstack</category></item><item><title>OpenStack Summit Mitaka from a Telemetry point of view</title><link>https://julien.danjou.info/blog/openstack-summit-mitaka-tokyo-telemetry/</link><guid isPermaLink="true">https://julien.danjou.info/blog/openstack-summit-mitaka-tokyo-telemetry/</guid><description>Last week I was in Tokyo, Japan for the OpenStack Summit, discussing the new Mitaka version that will be released in 6 months.</description><pubDate>Mon, 02 Nov 2015 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;Last week I was in Tokyo, Japan for the &lt;a href=&quot;https://www.openstack.org/summit/tokyo-2015/&quot;&gt;OpenStack Summit&lt;/a&gt;, discussing the new Mitaka version that will be released in 6 months.&lt;/p&gt;
&lt;p&gt;I&apos;ve attended the summit mainly to discuss and follow-up new developments on &lt;a href=&quot;http://launchpad.net/ceilometer&quot;&gt;Ceilometer&lt;/a&gt;, &lt;a href=&quot;http://launchpad.net/gnocchi&quot;&gt;Gnocchi&lt;/a&gt;, &lt;a href=&quot;http://launchpad.net/aodh&quot;&gt;Aodh&lt;/a&gt; and Oslo. It has been a pretty good week and we were able to discuss and plan a few interesting things. Below are what I found remarkable during this summit concerning those projects.&lt;/p&gt;
&lt;h2&gt;Distributed lock manager&lt;/h2&gt;
&lt;p&gt;I did not attend this session, but I need to write something about it.&lt;/p&gt;
&lt;p&gt;See, when working in a distributed environment like OpenStack, it&apos;s almost obvious that sooner or later you end up needing a distributed lock mechanism. It started to be pretty obvious and a serious problem for us 2 years ago in Ceilometer. Back then, we proposed the &lt;a href=&quot;https://wiki.openstack.org/wiki/Oslo/blueprints/service-sync&quot;&gt;service-sync&lt;/a&gt; blueprint and talked about it during the OpenStack Icehouse Design Summit in Hong-Kong. The session at that time was a success, and in 20 minutes I convinced everyone it was the good thing to do. The night following the session, we picked a named, Tooz, to name this new library. It was the first time I met Joshua Harlow, which became one of the biggest Tooz contributor since then.&lt;/p&gt;
&lt;p&gt;For the following months, we tried to move the lines in OpenStack. It was very hard to convince people that it was the solution to their problem. Most of the time, they did not seem to grasp the entirety of what was at stake.&lt;/p&gt;
&lt;p&gt;This time, it seems that we managed to convince everyone that a DLM is indeed needed. Joshua wrote an extensive specification called &lt;a href=&quot;https://review.openstack.org/#/c/209661/&quot;&gt;Chronicle of a DLM&lt;/a&gt;, which ended up being discussed and somehow adopted during that session in Tokyo.&lt;/p&gt;
&lt;p&gt;So yes, Tooz will be the weapon of choice for OpenStack. It will avoid a hard requirement on any DLM solution directly. The best driver right now is the &lt;a href=&quot;https://zookeeper.apache.org/&quot;&gt;ZooKeeper&lt;/a&gt; one, but it&apos;ll still be possible for operators to use e.g. Redis.&lt;/p&gt;
&lt;p&gt;This is a great achievement for us, after spending years trying to fix features such as the &lt;a href=&quot;https://blueprints.launchpad.net/nova/+spec/tooz-for-service-groups&quot;&gt;Nova service group subsystem&lt;/a&gt; and seeing our proposals postponed forever.&lt;/p&gt;
&lt;p&gt;(If you want to know more, &lt;a href=&quot;http://lwn.net&quot;&gt;LWN.net&lt;/a&gt; has&lt;br /&gt;
&lt;a href=&quot;https://lwn.net/Articles/662140/&quot;&gt;a great article about that session&lt;/a&gt;.)&lt;/p&gt;
&lt;h2&gt;Telemetry team name&lt;/h2&gt;
&lt;p&gt;With the new projects launched this last year, Aodh &amp;amp; Gnocchi, in parallel of the old Ceilometer, plus the change from programs to Big Tent in OpenSack, the team is having an identity issue. Being referred to as the &quot;Ceilometer team&quot; is not really accurate, as some of us only work on Aodh or on Gnocchi. So after discussing that, I &lt;a href=&quot;https://review.openstack.org/#/c/240809/&quot;&gt;proposed to rename the team to Telemetry&lt;/a&gt; instead. We&apos;ll see how it goes.&lt;/p&gt;
&lt;h2&gt;Alarms&lt;/h2&gt;
&lt;p&gt;The first session was about alarms and the Aodh project. It turns out that the project is in pretty good shape, but probably need some more love, which I hope I&apos;ll be able to provide in the next months.&lt;/p&gt;
&lt;p&gt;The need for a new &lt;em&gt;aodhclient&lt;/em&gt; based on the technologies we recently used building &lt;em&gt;gnocchiclient&lt;/em&gt; has been reasserted, so we might end up working on that pretty soon. The Tempest support also needs some improvement, and we have a plan to enhance that.&lt;/p&gt;
&lt;h2&gt;Data visualisation&lt;/h2&gt;
&lt;p&gt;We got David Lyle in this session, the Project Technical Leader for &lt;a href=&quot;http://openstack/horizon&quot;&gt;Horizon&lt;/a&gt;. It was an interesting discussion. It used to be technically challenging to draw charts from the data Ceilometer collects, but it&apos;s now very easy with Gnocchi and its API.&lt;/p&gt;
&lt;p&gt;While the technical side is resolved, the more political and user experience side of was to draw and how was discussed at length. We don&apos;t want to make people think that Ceilometer and Gnocchi are a full monitoring solution, so there&apos;s some precaution to take. Other than that, it would be pretty cool to have view of the data in Horizon.&lt;/p&gt;
&lt;h2&gt;Rolling upgrade&lt;/h2&gt;
&lt;p&gt;It turns out that Ceilometer has an architecture that makes it easy to have rolling upgrade. We just need to write a proper documentation explaining how to do it and in which order the services should be upgraded.&lt;/p&gt;
&lt;h2&gt;Ceilometer splitting&lt;/h2&gt;
&lt;p&gt;The split of the alarm feature of Ceilometer in its own project Aodh in the last cycle was a great success for the whole team. We want to split other pieces of Ceilometer, as they make sense on their own, makes it easier to manage. They are also some projects that want to use them without the whole stack, so that&apos;s a good idea to make it happen.&lt;/p&gt;
&lt;h2&gt;CloudKitty &amp;amp; Gnocchi&lt;/h2&gt;
&lt;p&gt;I attended the 2 sessions that were allocated to &lt;a href=&quot;https://wiki.openstack.org/wiki/CloudKitty&quot;&gt;CloudKitty&lt;/a&gt;. It was pretty interesting as they want to simplify their architecture and leverage what Gnocchi provides. I proposed my view of the project architecture and how they could leverage the more of Gnocchi to retrieve and store data. They want to go in that direction though it&apos;s a large amount of work and refactoring on their side, so it&apos;ll take time.&lt;/p&gt;
&lt;p&gt;We also need to enhance the support of extension for new resources in Gnocchi, and that&apos;s something I hope I&apos;ll work on in the next months.&lt;/p&gt;
&lt;p&gt;Overall, this summit was pretty good and I got a tremendous amount of good feedback on Gnocchi. I again managed to get enough ideas and tasks to tackle for the next 6 months. It really looks interesting to see where the whole team will go from that. Stay tuned!&lt;/p&gt;
</content:encoded><category>openstack</category><category>gnocchi</category></item><item><title>Benchmarking Gnocchi for fun &amp; profit</title><link>https://julien.danjou.info/blog/gnocchi-benchmarks/</link><guid isPermaLink="true">https://julien.danjou.info/blog/gnocchi-benchmarks/</guid><description>We got pretty good feedback on Gnocchi so far, even if we only had little. Recently, in order to have a better feeling of where we were at, we wanted to know how fast (or slow) Gnocchi was.</description><pubDate>Tue, 13 Oct 2015 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;We got pretty good feedback on &lt;a href=&quot;http://launchpad.net/gnocchi&quot;&gt;Gnocchi&lt;/a&gt; so far, even if we only had little. Recently, in order to have a better feeling of where we were at, we wanted to know how fast (or slow) Gnocchi was.&lt;/p&gt;
&lt;p&gt;The &lt;a href=&quot;https://julien.danjou.info/openstack-ceilometer-the-gnocchi-experiment.html&quot;&gt;early benchmarks that some of the Mirantis engineers ran last year&lt;/a&gt; showed pretty good signs. But a year later, it was time to get real numbers and have a good understanding of Gnocchi capacity.&lt;/p&gt;
&lt;h2&gt;Benchmark tools&lt;/h2&gt;
&lt;p&gt;The first thing I realized when starting that process, is that we were lacking of tools to run benchmarks. Therefore I started to write some benchmark tools in &lt;a href=&quot;https://launchpad.net/python-gnocchiclient&quot;&gt;python-gnocchiclient&lt;/a&gt;, which provides a command line tool to interrogate Gnocchi. I added a few basic commands to measure metric performance, such as:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ gnocchi benchmark metric create -w 48 -n 10000 -a low
+----------------------+------------------+
| Field                | Value            |
+----------------------+------------------+
| client workers       | 48               |
| create executed      | 10000            |
| create failures      | 0                |
| create failures rate | 0.00 %           |
| create runtime       | 8.80 seconds     |
| create speed         | 1136.96 create/s |
| delete executed      | 10000            |
| delete failures      | 0                |
| delete failures rate | 0.00 %           |
| delete runtime       | 39.56 seconds    |
| delete speed         | 252.75 delete/s  |
+----------------------+------------------+
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The command line tool supports the &lt;code&gt;--verbose&lt;/code&gt; switch to have detailed progress report on the benchmark progression. So far it supports metric operations only, but that&apos;s the most interesting part of Gnocchi.&lt;/p&gt;
&lt;h2&gt;Spinning up some hardware&lt;/h2&gt;
&lt;p&gt;I got a couple of bare metal servers to test Gnocchi on. I dedicated the first one to Gnocchi, and used the second one as the benchmark client, plugged on the same network. Each server is made of&lt;br /&gt;
2×&lt;a href=&quot;http://ark.intel.com/products/81897/Intel-Xeon-Processor-E5-2609-v3-15M-Cache-1_90-GHz&quot;&gt;Intel Xeon E5-2609 v3&lt;/a&gt; (12 cores in total) and 32 GB of RAM. That provides a lot of CPU to handle requests in parallel.&lt;/p&gt;
&lt;p&gt;Then I simply performed a basic &lt;a href=&quot;http://www.redhat.com/en/technologies/linux-platforms/enterprise-linux&quot;&gt;RHEL 7&lt;/a&gt; installation and ran &lt;a href=&quot;http://devstack.org&quot;&gt;devstack&lt;/a&gt; to spin up an installation of Gnocchi based on the master branch, disabling all of the others OpenStack components. I then tweaked the Apache httpd configuration to use the worker MPM and increased the maximum number of clients that can sent request simultaneously.&lt;/p&gt;
&lt;p&gt;I configured Gnocchi to use the &lt;em&gt;PostsgreSQL&lt;/em&gt; indexer, as it&apos;s the recommended one, and the &lt;em&gt;file&lt;/em&gt; storage driver, based on Carbonara (Gnocchi own storage engine). That means files were stored locally rather than in Ceph or Swift.&lt;/p&gt;
&lt;p&gt;Using the &lt;em&gt;file&lt;/em&gt; driver is less scalable (you have to run on only one node or uses a technology like NFS to share the files), but it was good enough for this benchmark and to have some numbers and profiling the beast.&lt;/p&gt;
&lt;p&gt;The OpenStack Keystone authentication middleware was not enabled in this setup, as it would add some delay validating the authentication token.&lt;/p&gt;
&lt;h2&gt;Metric CRUD operations&lt;/h2&gt;
&lt;p&gt;Metric creation is pretty fast. I managed to attain 1300 metric/s created pretty easily. Deletion is now asynchronous, which means it&apos;s faster than in Gnocchi 1.2, but it&apos;s still slower than creation: 500 metric/s can be deleted. That does not sound like a huge issue since metric deletion is actually barely used in production.&lt;/p&gt;
&lt;p&gt;Retrieving metric information is also pretty fast and goes up to 800 metric/s. It&apos;d be easy to achieve very higher throughput for this one, as it&apos;d be easy to cache, but we didn&apos;t feel the need to implement it so far.&lt;/p&gt;
&lt;p&gt;Another important thing is that all of these numbers are constant and barely depends on the number of the metric already managed by Gnocchi.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Operation&lt;/th&gt;
&lt;th&gt;Details&lt;/th&gt;
&lt;th&gt;Rate&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Create metric&lt;/td&gt;
&lt;td&gt;Created 100k metrics in 77 seconds&lt;/td&gt;
&lt;td&gt;1300 metric/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show metric&lt;/td&gt;
&lt;td&gt;Show a metric 100k times in 149 seconds&lt;/td&gt;
&lt;td&gt;670 metric/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Delete metric&lt;/td&gt;
&lt;td&gt;Deleted 100k metrics in 190 seconds&lt;/td&gt;
&lt;td&gt;524 metric/s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2&gt;Sending and getting measures&lt;/h2&gt;
&lt;p&gt;Pushing measures into metrics is one of the hottest topic. Starting with Gnocchi 1.1, the measures pushed are treated asynchronously, which makes it much faster to push new measures. Getting new numbers on that feature was pretty interesting.&lt;/p&gt;
&lt;p&gt;The number of metric per second you can push depends on the batch size, meaning the number of actual measurements you send per call. The naive approach is to push 1 measure per call, and in that case, Gnocchi is able to handle around 600 measures/s. With a batch containing 100 measures, the number of calls per second goes down to 450, but since you push 100 measures each time, that means 45k measures per second pushed into Gnocchi!&lt;/p&gt;
&lt;p&gt;I&apos;ve pushed the test further, inspired by the recent &lt;a href=&quot;https://influxdb.com/blog/2015/10/07/the_new_influxdb_storage_engine_a_time_structured_merge_tree.html&quot;&gt;blog post of InfluxDB claiming to achieve 300k points per second&lt;/a&gt; with their new engine. I ran the same benchmark on the hardware I had, which is roughly two times smaller than the one they used. I achieved to push Gnocchi to a little more than 120k measurement per second. If I had same hardware as they used, I could interpolate the results to achieve almost 250k measures/s pushed. Obviously, you can&apos;t strictly compare Gnocchi and InfluxDB since they are not doing exactly the same thing, but it still looks way better than what I expected.&lt;/p&gt;
&lt;p&gt;Using smaller batch sizes of 1k or 2k improve the throughput further to around 125k measures/s.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Operation&lt;/th&gt;
&lt;th&gt;Details&lt;/th&gt;
&lt;th&gt;Rate&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Push metric 5k&lt;/td&gt;
&lt;td&gt;Push 5M measures with batch of 5k measures in 40 seconds&lt;/td&gt;
&lt;td&gt;122k measures/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Push metric 4k&lt;/td&gt;
&lt;td&gt;Push 5M measures with batch of 4k measures in 40 seconds&lt;/td&gt;
&lt;td&gt;125k measures/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Push metric 3k&lt;/td&gt;
&lt;td&gt;Push 5M measures with batch of 3k measures in 40 seconds&lt;/td&gt;
&lt;td&gt;123k measures/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Push metric 2k&lt;/td&gt;
&lt;td&gt;Push 5M measures with batch of 2k measures in 41 seconds&lt;/td&gt;
&lt;td&gt;121k measures/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Push metric 1k&lt;/td&gt;
&lt;td&gt;Push 5M measures with batch of 1k measures in 44 seconds&lt;/td&gt;
&lt;td&gt;113k measures/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Push metric 500&lt;/td&gt;
&lt;td&gt;Push 5M measures with batch of 500 measures in 51 seconds&lt;/td&gt;
&lt;td&gt;98k measures/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Push metric 100&lt;/td&gt;
&lt;td&gt;Push 5M measures with batch of 100 measures in 112 seconds&lt;/td&gt;
&lt;td&gt;45k measures/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Push metric 10&lt;/td&gt;
&lt;td&gt;Push 5M measures with batch of 10 measures in 852 seconds&lt;/td&gt;
&lt;td&gt;6k measures/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Push metric 1&lt;/td&gt;
&lt;td&gt;Push 500k measures with batch of 1 measure in 800 seconds&lt;/td&gt;
&lt;td&gt;624 measures/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Get measures&lt;/td&gt;
&lt;td&gt;Push 43k measures of 1 metric&lt;/td&gt;
&lt;td&gt;260k measures/s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;What about getting measures? Well, it&apos;s actually pretty fast too. Retrieving a metric with 1 month of data with 1 minute interval (that&apos;s 43k points) takes less than 2 second.&lt;/p&gt;
&lt;p&gt;Though it&apos;s actually slower than what I expected. The reason seems to be that the JSON is 2 MB big and encoding it takes a lot of time for Python. I&apos;ll investigate that. Another point I discovered, is that by default Gnocchi returns all the datapoints for each granularities available for the asked period, which might double the size of the returned data for nothing if you don&apos;t need it. It&apos;ll be easy to add an option to the API to only retrieve what you need though!&lt;/p&gt;
&lt;p&gt;Once benchmarked, that meant I was able to retrieve 6 metric/s per second, which translates to around 260k measures/s.&lt;/p&gt;
&lt;h2&gt;&lt;em&gt;Metricd&lt;/em&gt; speed&lt;/h2&gt;
&lt;p&gt;New measures that are pushed into Gnocchi are processed asynchronously by the &lt;code&gt;gnocchi-metricd&lt;/code&gt; daemon. When doing the benchmarks above, I ran into a very interesting issue: sending 10k measures on a metric would make &lt;code&gt;gnocchi-metricd&lt;/code&gt; uses up to 2 GB RAM and 120 % CPU for more than 10 minutes.&lt;/p&gt;
&lt;p&gt;After further investigation, I found that the naive approach we used to resample datapoints in Carbonara using &lt;a href=&quot;http://pandas.pydata.org/&quot;&gt;Pandas&lt;/a&gt; was causing that. I &lt;a href=&quot;https://github.com/pydata/pandas/issues/11217&quot;&gt;reported a bug on Pandas&lt;/a&gt; and the upstream author was kind enough to provide a nice workaround, that I sent as &lt;a href=&quot;https://github.com/pydata/pandas/pull/11242&quot;&gt;a pull request&lt;/a&gt; to Pandas documentation.&lt;/p&gt;
&lt;p&gt;I wrote a fix for Gnocchi based on that, and started using it. Computing the standard aggregation methods set (std, count, 95pct, min, max, sum, median, mean) for 10k batches of 1 measure (worst case scenario) for one metric with 10k measures now takes only 20 seconds and uses 100 MB of RAM – 45× faster. That means that in normal operations, where only a few new measures are processed, the operation of updating a metric only takes a few milliseconds. Awesome!&lt;/p&gt;
&lt;h2&gt;Comparison with Ceilometer&lt;/h2&gt;
&lt;p&gt;For comparison sake, I&apos;ve quickly run some read operations benchmark in Ceilometer. I&apos;ve fed it with one month of samples for 100 instances polled every minute. That represents roughly 4.3M samples injected, and that took a while – almost 1 hour whereas it would have taken less than a minute in Gnocchi. Then I tried to retrieve some statistics in the same way that we provide them in Gnocchi, which mean aggregating them over a period of 60 seconds over a month.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Operation&lt;/th&gt;
&lt;th&gt;Details&lt;/th&gt;
&lt;th&gt;Rate&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Read metric SQL&lt;/td&gt;
&lt;td&gt;Read measures for 1 metric&lt;/td&gt;
&lt;td&gt;2min 58s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Read metric MongoDB&lt;/td&gt;
&lt;td&gt;Read measures for 1 metric&lt;/td&gt;
&lt;td&gt;28s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Read metric Gnocchi&lt;/td&gt;
&lt;td&gt;Read measures for 1 metric&lt;/td&gt;
&lt;td&gt;2s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Obviously, Ceilometer is very slow. It has to look into 4M of samples to compute and return the result, which takes a lot of time. Whereas Gnocchi just has to fetch a file and pass it over. That also means that the more samples you have (so the more time you collect data and the more resources you have), slower Ceilometer will become. This is not a problem with Gnocchi, as I emphasized when I started designing it.&lt;/p&gt;
&lt;p&gt;Most Gnocchi operations are &lt;em&gt;O(log R)&lt;/em&gt; where R is the number of metrics or resources, whereas most Ceilometer operations are &lt;em&gt;O(log S)&lt;/em&gt; where S is the number of samples (measures). Since is R millions of time smaller than S, Gnocchi gets to be much faster.&lt;/p&gt;
&lt;p&gt;And what&apos;s even more interesting, is that Gnocchi is entirely scalable horizontally. Adding more Gnocchi servers (for the API and its background processing worker &lt;em&gt;metricd&lt;/em&gt;) will multiply Gnocchi performances by the number of servers added.&lt;/p&gt;
&lt;h2&gt;Improvements&lt;/h2&gt;
&lt;p&gt;There are several things to improve in Gnocchi, such as splitting Carbonara archives to make them more efficient, especially from drivers such as Ceph and Swift. It&apos;s already on my plate, and I&apos;m looking forwarding to working on that!&lt;/p&gt;
&lt;p&gt;And if you have any questions, feel free to shoot them in the comment section. 😉&lt;/p&gt;
</content:encoded><category>gnocchi</category><category>openstack</category></item><item><title>Gnocchi talk at OpenStack Paris Meetup #16</title><link>https://julien.danjou.info/blog/openstack-france-paris-meetup-gnocchi-talk/</link><guid isPermaLink="true">https://julien.danjou.info/blog/openstack-france-paris-meetup-gnocchi-talk/</guid><description>Last week, I&apos;ve been invited to the OpenStack Paris meetup #16, whose subject was about metrics in OpenStack. Last time I spoke at this meetup was back in 2012, during the OpenStack Paris meetup #2.</description><pubDate>Mon, 05 Oct 2015 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;Last week, I&apos;ve been invited to the &lt;a href=&quot;http://www.meetup.com/OpenStack-France/events/225227112/&quot;&gt;OpenStack Paris meetup #16&lt;/a&gt;, whose subject was about metrics in OpenStack. Last time I spoke at this meetup was back in 2012, during the &lt;a href=&quot;https://julien.danjou.info/blog/openstack-france-meetup-2&quot;&gt;OpenStack Paris meetup #2&lt;/a&gt;. A very long time ago!&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-talk-2.jpg&quot; alt=&quot;gnocchi-talk-2&quot; /&gt;&lt;/p&gt;
&lt;p&gt;I talked for half an hour about &lt;a href=&quot;http://launchpad.net/gnocchi&quot;&gt;Gnocchi&lt;/a&gt;, the OpenStack project I&apos;ve been running for 18 months now. I started by explaining the story behind the project and why we needed to build it. Ceilometer has an interesting history and had a curious roadmap these last year, and I summarized that briefly. Then I talk about how Gnocchi works and what it offers to users and operators. The slides where full of JSON, but I imagine it offered a interesting view of what the API looks like and how easy it is to operate. This also allowed me to emphasize how many use cases are actually really covered and solved, contrary to what Ceilometer did so far. The talk has been well received and I got a few interesting questions at the end.&lt;/p&gt;
</content:encoded><category>talks</category><category>openstack</category><category>gnocchi</category></item><item><title>Visualize your OpenStack cloud: Gnocchi &amp; Grafana</title><link>https://julien.danjou.info/blog/openstack-gnocchi-grafana/</link><guid isPermaLink="true">https://julien.danjou.info/blog/openstack-gnocchi-grafana/</guid><description>We&apos;ve been hard working with the Gnocchi team these last months to store your metrics, and I guess it&apos;s time to show off a bit.  So far Gnocchi offers scalable metric storage and resource indexation,</description><pubDate>Mon, 14 Sep 2015 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;We&apos;ve been hard working with the Gnocchi team these last months to store your metrics, and I guess it&apos;s time to show off a bit.&lt;/p&gt;
&lt;p&gt;So far Gnocchi offers scalable metric storage and resource indexation, especially for OpenStack cloud – but not only, we&apos;re generic. It&apos;s cool to store metrics, but it can be even better to have a way to visualize them!&lt;/p&gt;
&lt;h2&gt;Prototyping&lt;/h2&gt;
&lt;p&gt;We very soon started to build a little HTML interface. Being REST-friendly guys, we enabled it on the same endpoints that were being used to retrieve information and measures about metric, sending back &lt;code&gt;text/html&lt;/code&gt; instead of &lt;code&gt;application/json&lt;/code&gt; if you were requesting those pages from a Web browser.&lt;/p&gt;
&lt;p&gt;But let&apos;s face it: we are back-end developers, we suck at any kind front-end development. CSS, HTML, JavaScript? Bwah! So what we built was a starting point, hoping some magical Web developer would jump in and finish the job.&lt;/p&gt;
&lt;p&gt;Obviously it never happened.&lt;/p&gt;
&lt;h2&gt;Ok, so what&apos;s out there?&lt;/h2&gt;
&lt;p&gt;It turns out there are back-end agnostic solutions out there, and we decided to pick &lt;a href=&quot;http://grafana.org&quot;&gt;Grafana&lt;/a&gt;. Grafana is a complete graphing dashboard solution that can be plugged on top of any back-end. It already supports timeseries databases such as Graphite, InfluxDB and OpenTSDB.&lt;/p&gt;
&lt;p&gt;That was largely enough for that my fellow developer &lt;a href=&quot;https://blog.sileht.net/&quot;&gt;Mehdi Abaakouk&lt;/a&gt; to jump in and start writing a Gnocchi plugin for Grafana! Consequently, there is now a basic but solid and working back-end for Grafana that lies in the &lt;em&gt;&lt;a href=&quot;https://github.com/grafana/grafana-plugins/tree/master/datasources/gnocchi&quot;&gt;grafana-plugins&lt;/a&gt;&lt;/em&gt;&lt;br /&gt;
repository.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-grafana.png&quot; alt=&quot;gnocchi-grafana&quot; /&gt;&lt;/p&gt;
&lt;p&gt;With that plugin, you can graph anything that is stored in Gnocchi, from raw metrics to metrics tied to resources. You can use templating, but no annotation yet.&lt;/p&gt;
&lt;p&gt;The back-end supports Gnocchi with or without Keystone involved, and any type of authentication (basic auth or Keystone token). So yes, it even works if you&apos;re not running Gnocchi with the rest of OpenStack.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-grafana-group.png&quot; alt=&quot;gnocchi-grafana-group&quot; /&gt;&lt;/p&gt;
&lt;p&gt;It also supports advanced queries, so you can search for resources based on some criterion and graphs their metrics.&lt;/p&gt;
&lt;h2&gt;I want to try it!&lt;/h2&gt;
&lt;p&gt;If you want to deploy it, all you need to do is to install Grafana and its plugins, and create a new datasource pointing to Gnocchi. It is that simple. There&apos;s some CORS middleware configuration involved if you&apos;re planning on using Keystone authentication, but it&apos;s pretty straightforward – just set the &lt;code&gt;cors.allowed_origin&lt;/code&gt; option to the URL of your Grafana dashboard.&lt;/p&gt;
&lt;p&gt;We added support of Grafana directly in Gnocchi devstack plugin. If you&apos;re running &lt;a href=&quot;http://devstack.org&quot;&gt;DevStack&lt;/a&gt; you can follow &lt;a href=&quot;http://docs.openstack.org/developer/gnocchi/devstack.html&quot;&gt;the instructions&lt;/a&gt; – which are basically adding the line &lt;code&gt;enable_service gnocchi-grafana&lt;/code&gt;.&lt;/p&gt;
&lt;h2&gt;Moving to Grafana core&lt;/h2&gt;
&lt;p&gt;[Mehdi just opened a pull request] (&lt;a href=&quot;https://github.com/grafana/grafana/pull/2716&quot;&gt;https://github.com/grafana/grafana/pull/2716&lt;/a&gt;) a few days ago to merge the plugin into Grafana core. It&apos;s actually one of the most unit-tested plugin in Grafana so far, so it should be on a good path to be merged in the future and have support of Gnocchi directly into Grafana without any plugin involved.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/grafana-gnocchi-unittests.png&quot; alt=&quot;grafana-gnocchi-unittests&quot; /&gt;&lt;/p&gt;
</content:encoded><category>gnocchi</category><category>openstack</category><category>monitoring</category></item><item><title>Ceilometer, Gnocchi &amp; Aodh: Liberty progress</title><link>https://julien.danjou.info/blog/ceilometer-gnocchi-aodh-liberty-progress/</link><guid isPermaLink="true">https://julien.danjou.info/blog/ceilometer-gnocchi-aodh-liberty-progress/</guid><description>It&apos;s been a while since I talked about Ceilometer and its companions, so I thought I&apos;d go ahead and write a bit about what&apos;s going on this side of OpenStack. I&apos;m not going to cover new features and fa</description><pubDate>Tue, 04 Aug 2015 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;It&apos;s been a while since I talked about Ceilometer and its companions, so I thought I&apos;d go ahead and write a bit about what&apos;s going on this side of OpenStack. I&apos;m not going to cover new features and fancy stuff today, but rather a shallow overview of the new project processes we initiated.&lt;/p&gt;
&lt;h2&gt;Ceilometer growing&lt;/h2&gt;
&lt;p&gt;&lt;a href=&quot;http://launchpad.net/ceilometer&quot;&gt;Ceilometer&lt;/a&gt; has grown a lot since that time when we started it 3 years ago. It has evolved from a system designed to fetch and store measurements, to a more complex system, with agents, alarms, events, databases, APIs, etc.&lt;/p&gt;
&lt;p&gt;All those features were needed and asked for by users and operators, but let&apos;s be honest, some of them should never have ended up in the Ceilometer code repository, especially not all at the same time.&lt;/p&gt;
&lt;p&gt;The reality is we picked a pragmatic approach due to the rigidity of the OpenStack Technical Committee in regards to new projects to become OpenStack integrated – and, therefore, blessed – projects. Ceilometer was actually the first project to be incubated and then integrated. We had to go through the very first issues of that process.&lt;/p&gt;
&lt;p&gt;Fortunately, now that time has passed, and all those constraints have been relaxed. To me, the &lt;a href=&quot;https://www.openstack.org/foundation&quot;&gt;OpenStack Foundation&lt;/a&gt; is turning into something that looks like the &lt;a href=&quot;http://www.apache.org/foundation/&quot;&gt;Apache Foundation&lt;/a&gt;, and there&apos;s, therefore, no need to tie technical solutions to political issues.&lt;/p&gt;
&lt;p&gt;Indeed, the &lt;a href=&quot;https://www.openstack.org/summit/vancouver-2015/summit-videos/presentation/the-big-tent-a-look-at-the-new-openstack-projects-governance&quot;&gt;Big Tent&lt;/a&gt; now allows much more flexibility to all of that. Back a year ago, we were afraid to bring Gnocchi into Ceilometer. Was the Technical Committee going to review the project? Was the project going to be in the scope of Ceilometer for the Technical Committee? Now we don&apos;t have to ask ourselves those questions, now that we have that freedom, it empowers us to actually do what we think is good in term of technical design without worrying too much about political issues.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/ceilometer-activity.png&quot; alt=&quot;ceilometer-activity&quot; /&gt;&lt;/p&gt;
&lt;h2&gt;Acknowledging Gnocchi&lt;/h2&gt;
&lt;p&gt;The first step in this new process was to continue working on &lt;a href=&quot;https://launchpad.net/gnocchi&quot;&gt;Gnocchi&lt;/a&gt; (a timeserie database and resource indexer designed to overcome historical Ceilometer storage issue) and to decide that it was not the right call to merge it into Ceilometer as some REST API v3, but that it was better to keep it standalone.&lt;/p&gt;
&lt;p&gt;We managed to get traction to Gnocchi, getting a few contributors and users. We&apos;re even seeing talks proposed to the next Tokyo Summit where people leverage Gnocchi, such as &quot;Service of predictive analytics on cost and performance in OpenStack&quot;, &quot;&lt;a href=&quot;https://wiki.openstack.org/wiki/Surveil&quot;&gt;Suveil&lt;/a&gt;&quot; and &quot;Cutting Edge NFV On OpenStack: Healing and Scaling Distributed Applications&quot;.&lt;/p&gt;
&lt;p&gt;We are also doing some progress on pushing Gnocchi outside of the OpenStack community, as it can be a self-sufficient timeserie and resource database that can be used without any OpenStack interaction.&lt;/p&gt;
&lt;h2&gt;Branching Aodh&lt;/h2&gt;
&lt;p&gt;Rather than continuing to grow Ceilometer, during the last summit we all decided that it was time to reorganize and split Ceilometer into the different components it is made of, leveraging a more &lt;a href=&quot;https://en.wikipedia.org/wiki/Service-oriented_architecture&quot;&gt;service-oriented architecture&lt;/a&gt;. The alarm subsystem of Ceilometer being mostly untied to the rest of Ceilometer, we decided it was the first and perfect candidate to do that. I personally engaged into doing the work and created a new repository with only the alarm code from Ceilometer, named &lt;a href=&quot;https://launchpad.net/aodh&quot;&gt;Aodh&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/woman-fire.jpg&quot; alt=&quot;woman-fire&quot; /&gt;&lt;/p&gt;
&lt;p&gt;This made sense for a lot of reason. First because Aodh can now work completely standalone, using either Ceilometer or Gnocchi as a backend – or any new plugin you&apos;d write. I love the idea that OpenStack projects can work standalone – like Swift does for example – without implying any other OpenStack component. I think it&apos;s a proof of good design. Secondly, because it allows us to resonate on a smaller chunk of software – a reason really under-estimated today in OpenStack. I believe that the size of your software should match a certain ratio to the size of your team.&lt;/p&gt;
&lt;p&gt;Aodh is, therefore, a new project under the OpenStack Telemetry program (or what remains of OpenStack programs now), alongside Ceilometer and Gnocchi, forked from the original Ceilometer alarm feature. We&apos;ll deprecate the latter with the Liberty release, and we&apos;ll remove it in the Mitaka release.&lt;/p&gt;
&lt;h2&gt;Lessons learned&lt;/h2&gt;
&lt;p&gt;Actually, moving that code out of Ceilometer (in the case of Aodh), or not merging it in (in the case of Gnocchi) had a few side effects that I admit I think we probably under-estimated back then.&lt;/p&gt;
&lt;p&gt;Indeed, the code size of Gnocchi or Aodh ended up being much smaller than the entire Ceilometer project – Gnocchi is 7× smaller and Aodh 5x smaller than Ceilometer – and therefore much more easy to manipulate and to hack on. That allowed us to merge dozens of patches in a few weeks, cleaning-up and enhancing a lot of small things in the code. Those tasks are very much harder in Ceilometer, due to the bigger size of the code base and the small size of our team. By having our small team working on smaller chunks of changes – even when it meant actually doing more reviews – greatly improved our general velocity and the number of bugs fixed and features implemented.&lt;/p&gt;
&lt;p&gt;On the more sociological side, I think it gave the team the sensation of finally owning the project. Ceilometer was huge, and it was impossible for people to know every side of it. Now, it&apos;s getting possible for people inside a team to cover a much larger portion of those smaller project, which gives them a greater sense of ownership and caring. Which ends up being good for the project quality overall.&lt;/p&gt;
&lt;p&gt;That also means that we technically decided to have different core teams by project (Ceilometer, Gnocchi, and Aodh) as they all serve different purposes and can all be used standalone or with each others. Meaning we could have contributors completely ignoring other projects.&lt;/p&gt;
&lt;p&gt;All of that reminds me some discussion I heard about projects such as Glance, trying to fit new features in - some that are really orthogonal to the original purpose. It&apos;s now clear to me that having different small components interacting together that can be completely owned and taken care of by a (small) team of contributors is the way to go. People that can therefore trust each others and easily bring new people in, makes a project really incredibly more powerful. Having a project covering a too wide set of features make things more difficult if you don&apos;t have enough manpower. This is clearly an issue that big projects inside OpenStack are facing now, such as Neutron or Nova.&lt;/p&gt;
</content:encoded><category>openstack</category><category>gnocchi</category></item><item><title>OpenStack Summit Liberty from a Ceilometer &amp; Gnocchi point of view</title><link>https://julien.danjou.info/blog/openstack-summit-liberty-vancouver-ceilometer-gnocchi/</link><guid isPermaLink="true">https://julien.danjou.info/blog/openstack-summit-liberty-vancouver-ceilometer-gnocchi/</guid><description>Last week I was in Vancouver, BC for the OpenStack Summit, discussing the new Liberty version that will be released in 6 months.</description><pubDate>Tue, 26 May 2015 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;Last week I was in &lt;a href=&quot;http://vancouver.ca/&quot;&gt;Vancouver, BC&lt;/a&gt; for the &lt;a href=&quot;https://www.openstack.org/summit/vancouver-2015/&quot;&gt;OpenStack Summit&lt;/a&gt;, discussing the new Liberty version that will be released in 6 months.&lt;/p&gt;
&lt;p&gt;I&apos;ve attended the summit mainly to discuss and follow-up new developments on Ceilometer, Gnocchi and Oslo. It has been a pretty good week and we were able to discuss and plan a few interesting things.&lt;/p&gt;
&lt;h2&gt;Ops feedback&lt;/h2&gt;
&lt;p&gt;We had half a dozen Ceilometer sessions, and the first one was dedicated to getting feedbacks from operators using Ceilometer. We had a few operators present, and a few of the Ceilometer team. We had constructive discussion, and my feeling is that operators struggles with 2 things so far: scaling Ceilometer storage and having Ceilometer not killing the rest of OpenStack.&lt;/p&gt;
&lt;p&gt;We discussed the first point as being addressed by &lt;a href=&quot;http://launchpad.net/gnocchi&quot;&gt;Gnocchi&lt;/a&gt;, and I presented a bit Gnocchi itself, as well as how and why it will fix the storage scalability issue operators encountered so far.&lt;/p&gt;
&lt;p&gt;Ceilometer putting down the OpenStack installation is more interesting problem. Ceilometer pollsters request information from Nova, Glance… to gather statistics. Until Kilo, Ceilometer used to do that regularly and at fixed interval, causing high pike load in OpenStack. With the &lt;a href=&quot;http://docs.openstack.org/developer/ceilometer/architecture.html#polling-agents-asking-for-data&quot;&gt;introduction of jitter&lt;/a&gt; in Kilo, this should be less of a problem. However, Ceilometer hits various endpoints in OpenStack that are poorly designed, and hitting those endpoints of Nova or other components triggers a lot of load on the platform. Unfortunately, this makes operators blame Ceilometer rather than blaming the components being guilty of poor designs. We&apos;d like to push forward improving these components, but it&apos;s probably going to take a long time.&lt;/p&gt;
&lt;h2&gt;Componentisation&lt;/h2&gt;
&lt;p&gt;When I started the Gnocchi project last year, I pretty soon realized that we would be able to split Ceilometer itself in different smaller components that could work independently, while being able to leverage each others. For example, Gnocchi can run standalone and store your metrics even if you don&apos;t use Ceilometer – nor even OpenStack itself.&lt;/p&gt;
&lt;p&gt;My fellow developer &lt;a href=&quot;http://burningchrome.com/&quot;&gt;Chris Dent&lt;/a&gt; had the same idea about splitting Ceilometer a few months ago and drafted a proposal. The idea is to have Ceilometer split in different parts that people could assemble together or run on their owns.&lt;/p&gt;
&lt;p&gt;Interestingly enough, we had three 40 minutes sessions planned to talk and debate about this division of Ceilometer, though we all agreed in 5 minutes that this was the good thing to do. Five more minutes later, we agreed on which part to split. The rest of the time was allocated to discuss various details of that split, and I engaged to start doing the work with Ceilometer alarming subsystem.&lt;/p&gt;
&lt;p&gt;I wrote a &lt;a href=&quot;https://review.openstack.org/#/c/184307/&quot;&gt;specification&lt;/a&gt; on the plane bringing me to Vancouver, that should be approved pretty soon now. I already started doing the implementation work. So fingers crossed, Ceilometer should have a new components in Liberty handling alarming on its own.&lt;/p&gt;
&lt;p&gt;This would allow users for example to only deploys Gnocchi and Ceilometer alarm. They would be able to feed data to Gnocchi using their own system, and build alarms using Ceilometer alarm subsystem relying on Gnocchi&apos;s data.&lt;/p&gt;
&lt;h2&gt;Gnocchi&lt;/h2&gt;
&lt;p&gt;We didn&apos;t have a Gnocchi dedicated slot – mainly because I indicated I didn&apos;t feel we needed one. We anyway discussed a few points around coffee, and I&apos;ve been able to draw a few new ideas and changes I&apos;d like to see in Gnocchi. Mainly changing the API contract to be more asynchronously so we can support &lt;a href=&quot;http://influxdb.com/&quot;&gt;InfluxDB&lt;/a&gt; more correctly, and improve Carbonara (the library we created to manipulate timeseries) based drivers to be faster.&lt;/p&gt;
&lt;p&gt;All of those should – plus a few Oslo tasks I&apos;d like to tackle – should keep me busy for the next cycle!&lt;/p&gt;
</content:encoded><category>openstack</category><category>gnocchi</category></item><item><title>Gnocchi 1.0: storing metrics and resources at scale</title><link>https://julien.danjou.info/blog/openstack-gnocchi-first-release/</link><guid isPermaLink="true">https://julien.danjou.info/blog/openstack-gnocchi-first-release/</guid><description>A few months ago, I wrote a long post about what I called back then the &quot; Gnocchi experiment &quot;. Time passed and we – me and the rest of the Gnocchi team – continued to work on that project, finalizing</description><pubDate>Tue, 21 Apr 2015 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;A few months ago, I wrote a long post about what I called back then the &quot;&lt;a href=&quot;https://julien.danjou.info/blog/openstack-ceilometer-the-gnocchi-experiment&quot;&gt;Gnocchi experiment&lt;/a&gt;&quot;. Time passed and we – me and the rest of the Gnocchi team – continued to work on that project, finalizing it.&lt;/p&gt;
&lt;p&gt;It&apos;s with a great pleasure that we are going to release our first &lt;em&gt;1.0&lt;/em&gt; version this month, roughly at the same time that the integrated &lt;a href=&quot;http://openstack.org&quot;&gt;OpenStack&lt;/a&gt; projects release their Kilo milestone. The &lt;a href=&quot;https://pypi.python.org/pypi/gnocchi&quot;&gt;first release candidate numbered 1.0.0rc1&lt;/a&gt; has been released this morning!&lt;/p&gt;
&lt;h2&gt;The problem to solve&lt;/h2&gt;
&lt;p&gt;Before I dive into Gnocchi details, it&apos;s important to have a good view of what problems Gnocchi is trying to solve.&lt;/p&gt;
&lt;p&gt;Most of the IT infrastructures out there consists of a set of resources. These resources have properties: some of them are simple attributes whereas others might be measurable quantities (also known as metrics).&lt;/p&gt;
&lt;p&gt;And in this context, the cloud infrastructures make no exception. We talk about instances, volumes, networks… which are all different kind of resources. The problems that are arising with the cloud trend is the scalability of storing all this data and being able to request them later, for whatever usage.&lt;/p&gt;
&lt;p&gt;What Gnocchi provides is a REST API that allows the user to manipulate resources (CRUD) and their attributes, while preserving the history of those resources and their attributes.&lt;/p&gt;
&lt;p&gt;Gnocchi is fully documented and the &lt;a href=&quot;http://gnocchi.xyz&quot;&gt;documentation is available online&lt;/a&gt;. We are the first OpenStack project to require patches to &lt;em&gt;integrate the documentation&lt;/em&gt;. We want to raise the bar, so we took a stand on that. That&apos;s part of our policy, the same way it&apos;s part of the OpenStack policy to require unit tests.&lt;/p&gt;
&lt;p&gt;I&apos;m not going to paraphrase the whole Gnocchi documentation, which covers things like installation (super easy), but I&apos;ll guide you through some basics of the features provided by the REST API. I will show you some example so you can have a better understanding of what you could leverage using Gnocchi!&lt;/p&gt;
&lt;h2&gt;Handling metrics&lt;/h2&gt;
&lt;p&gt;Gnocchi provides a full REST API to manipulate time-series that are called &lt;em&gt;metrics&lt;/em&gt;. You can easily create a metric using a simple HTTP request:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;POST /v1/metric HTTP/1.1
Content-Type: application/json

{
  &quot;archive_policy_name&quot;: &quot;low&quot;
}

HTTP/1.1 201 Created
Location: http://localhost/v1/metric/387101dc-e4b1-4602-8f40-e7be9f0ed46a
Content-Type: application/json; charset=UTF-8

{
  &quot;archive_policy&quot;: {
    &quot;aggregation_methods&quot;: [
      &quot;std&quot;,
      &quot;sum&quot;,
      &quot;mean&quot;,
      &quot;count&quot;,
      &quot;max&quot;,
      &quot;median&quot;,
      &quot;min&quot;,
      &quot;95pct&quot;
    ],
    &quot;back_window&quot;: 0,
    &quot;definition&quot;: [
      {
        &quot;granularity&quot;: &quot;0:00:01&quot;,
        &quot;points&quot;: 3600,
        &quot;timespan&quot;: &quot;1:00:00&quot;
      },
      {
        &quot;granularity&quot;: &quot;0:30:00&quot;,
        &quot;points&quot;: 48,
        &quot;timespan&quot;: &quot;1 day, 0:00:00&quot;
      }
    ],
    &quot;name&quot;: &quot;low&quot;
  },
  &quot;created_by_project_id&quot;: &quot;e8afeeb3-4ae6-4888-96f8-2fae69d24c01&quot;,
  &quot;created_by_user_id&quot;: &quot;c10829c6-48e2-4d14-ac2b-bfba3b17216a&quot;,
  &quot;id&quot;: &quot;387101dc-e4b1-4602-8f40-e7be9f0ed46a&quot;,
  &quot;name&quot;: null,
  &quot;resource_id&quot;: null
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The &lt;code&gt;archive_policy_name&lt;/code&gt; parameter defines how the measures that are being sent are going to be aggregated. You can also define archive policies using the API and specify what kind of aggregation period and granularity you want. In that case , the &lt;em&gt;low&lt;/em&gt; archive policy keeps 1 hour of data aggregated over 1 second and 1 day of data aggregated to 30 minutes. The functions used for aggregations are the mathematical functions standard deviation, minimum, maximum, … and even 95th percentile. All of that is obviously customizable and you can create your own archive policies.&lt;/p&gt;
&lt;p&gt;If you don&apos;t want to specify the archive policy manually for each metric, you can also create &lt;em&gt;archive policy rule&lt;/em&gt;, that will apply a specific archive policy based on the metric name, e.g. metrics matching &lt;code&gt;disk.*&lt;/code&gt; will be high resolution metrics so they will use the &lt;code&gt;high&lt;/code&gt; archive policy.&lt;/p&gt;
&lt;p&gt;It&apos;s also worth noting Gnocchi is precise up to the nanosecond and is not tied to the current time. You can manipulate and inject measures that are years old and precise to the nanosecond. You can also inject points with old timestamps (i.e. old compared to the most recent one in the timeseries) with an archive policy allowing it (see &lt;code&gt;back_window&lt;/code&gt; parameter).&lt;/p&gt;
&lt;p&gt;It&apos;s then possible to send measures to this metric:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;POST /v1/metric/387101dc-e4b1-4602-8f40-e7be9f0ed46a/measures HTTP/1.1
Content-Type: application/json

[
  {
    &quot;timestamp&quot;: &quot;2014-10-06T14:33:57&quot;,
    &quot;value&quot;: 43.1
  },
  {
    &quot;timestamp&quot;: &quot;2014-10-06T14:34:12&quot;,
    &quot;value&quot;: 12
  },
  {
    &quot;timestamp&quot;: &quot;2014-10-06T14:34:20&quot;,
    &quot;value&quot;: 2
  }
  ]
  
HTTP/1.1 204 No Content
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;These measures are synchronously aggregated and stored into the configured storage backend. Our most scalable storage drivers for now are either based on &lt;a href=&quot;http://launchpad.net/swift&quot;&gt;Swift&lt;/a&gt; or &lt;a href=&quot;http://ceph.com&quot;&gt;Ceph&lt;/a&gt; which are both scalable storage objects systems.&lt;/p&gt;
&lt;p&gt;It&apos;s then possible to retrieve these values:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;GET /v1/metric/387101dc-e4b1-4602-8f40-e7be9f0ed46a/measures HTTP/1.1

HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

[
  [
    &quot;2014-10-06T14:30:00.000000Z&quot;,
    1800.0,
    19.033333333333335
  ],
  [
    &quot;2014-10-06T14:33:57.000000Z&quot;,
    1.0,
    43.1
  ],
  [
    &quot;2014-10-06T14:34:12.000000Z&quot;,
    1.0,
    12.0
  ],
  [
    &quot;2014-10-06T14:34:20.000000Z&quot;,
    1.0,
    2.0
  ]
]
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;As older Ceilometer users might notice here, metrics are only storing points and values, nothing fancy such as metadata anymore.&lt;/p&gt;
&lt;p&gt;By default, values eagerly aggregated using mean are returned for all supported granularities. You can obviously specify a time range or a different aggregation function using the &lt;code&gt;aggregation&lt;/code&gt;, &lt;code&gt;start&lt;/code&gt; and &lt;code&gt;stop&lt;/code&gt; query parameter.&lt;/p&gt;
&lt;p&gt;Gnocchi also supports doing aggregation across aggregated metrics:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;GET /v1/aggregation/metric?metric=65071775-52a8-4d2e-abb3-1377c2fe5c55&amp;amp;metric=9ccdd0d6-f56a-4bba-93dc-154980b6e69a&amp;amp;start=2014-10-06T14:34&amp;amp;aggregation=mean HTTP/1.1

HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

[
  [
    &quot;2014-10-06T14:34:12.000000Z&quot;,
    1.0,
    12.25
  ],
  [
    &quot;2014-10-06T14:34:20.000000Z&quot;,
    1.0,
    11.6
  ]
]
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This computes the mean of mean for the metric &lt;code&gt;65071775-52a8-4d2e-abb3-1377c2fe5c55&lt;/code&gt; and &lt;code&gt;9ccdd0d6-f56a-4bba-93dc-154980b6e69a&lt;/code&gt; starting on 6th October 2014 at 14:34 UTC.&lt;/p&gt;
&lt;h2&gt;Indexing your resources&lt;/h2&gt;
&lt;p&gt;Another object and concept that Gnocchi provides is the ability to manipulate resources. There is a basic type of resource, called &lt;em&gt;generic&lt;/em&gt;, which has very few attributes. You can extend this type to specialize it, and that&apos;s what Gnocchi does by default by providing resource types known for OpenStack such as &lt;em&gt;instance&lt;/em&gt;, &lt;em&gt;volume&lt;/em&gt;, &lt;em&gt;network&lt;/em&gt; or even &lt;em&gt;image&lt;/em&gt;.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;POST /v1/resource/generic HTTP/1.1

Content-Type: application/json

{
  &quot;id&quot;: &quot;75C44741-CC60-4033-804E-2D3098C7D2E9&quot;,
  &quot;project_id&quot;: &quot;BD3A1E52-1C62-44CB-BF04-660BD88CD74D&quot;,
  &quot;user_id&quot;: &quot;BD3A1E52-1C62-44CB-BF04-660BD88CD74D&quot;
}

HTTP/1.1 201 Created
Location: http://localhost/v1/resource/generic/75c44741-cc60-4033-804e-2d3098c7d2e9
ETag: &quot;e3acd0681d73d85bfb8d180a7ecac75fce45a0dd&quot;
Last-Modified: Fri, 17 Apr 2015 11:18:48 GMT
Content-Type: application/json; charset=UTF-8

{
  &quot;created_by_project_id&quot;: &quot;ec181da1-25dd-4a55-aa18-109b19e7df3a&quot;,
  &quot;created_by_user_id&quot;: &quot;4543aa2a-6ebf-4edd-9ee0-f81abe6bb742&quot;,
  &quot;ended_at&quot;: null,
  &quot;id&quot;: &quot;75c44741-cc60-4033-804e-2d3098c7d2e9&quot;,
  &quot;metrics&quot;: {},
  &quot;project_id&quot;: &quot;bd3a1e52-1c62-44cb-bf04-660bd88cd74d&quot;,
  &quot;revision_end&quot;: null,
  &quot;revision_start&quot;: &quot;2015-04-17T11:18:48.696288Z&quot;,
  &quot;started_at&quot;: &quot;2015-04-17T11:18:48.696275Z&quot;,
  &quot;type&quot;: &quot;generic&quot;,
  &quot;user_id&quot;: &quot;bd3a1e52-1c62-44cb-bf04-660bd88cd74d&quot;
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The resource is created with the UUID provided by the user. Gnocchi handles the history of the resource, and that&apos;s what the &lt;code&gt;revision_start&lt;/code&gt; and &lt;code&gt;revision_end&lt;/code&gt; fields are for. They indicates the lifetime of this revision of the resource. The &lt;code&gt;ETag&lt;/code&gt; and &lt;code&gt;Last-Modified&lt;/code&gt; headers are also unique to this resource revision and can be used in a subsequent request using &lt;code&gt;If-Match&lt;/code&gt; or &lt;code&gt;If-Not-Match&lt;/code&gt; header, for example:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;GET /v1/resource/generic/75c44741-cc60-4033-804e-2d3098c7d2e9 HTTP/1.1
If-Not-Match: &quot;e3acd0681d73d85bfb8d180a7ecac75fce45a0dd&quot;

HTTP/1.1 304 Not Modified
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Which is useful to synchronize and update any view of the resources you might have in your application.&lt;/p&gt;
&lt;p&gt;You can use the &lt;code&gt;PATCH&lt;/code&gt; HTTP method to modify properties of the resource, which will create a new revision of the resource. The history of the resources are available via the REST API obviously.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;metrics&lt;/code&gt; properties of the resource allow you to link metrics to a resource. You can link existing metrics or create new ones dynamically:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;POST /v1/resource/generic HTTP/1.1
Content-Type: application/json

{
  &quot;id&quot;: &quot;AB68DA77-FA82-4E67-ABA9-270C5A98CBCB&quot;,
  &quot;metrics&quot;: {
    &quot;temperature&quot;: {
      &quot;archive_policy_name&quot;: &quot;low&quot;
    }
  },
  &quot;project_id&quot;: &quot;BD3A1E52-1C62-44CB-BF04-660BD88CD74D&quot;,
  &quot;user_id&quot;: &quot;BD3A1E52-1C62-44CB-BF04-660BD88CD74D&quot;
}

HTTP/1.1 201 Created
Location: http://localhost/v1/resource/generic/ab68da77-fa82-4e67-aba9-270c5a98cbcb
ETag: &quot;9f64c8890989565514eb50c5517ff01816d12ff6&quot;
Last-Modified: Fri, 17 Apr 2015 14:39:22 GMT
Content-Type: application/json; charset=UTF-8

{
  &quot;created_by_project_id&quot;: &quot;cfa2ebb5-bbf9-448f-8b65-2087fbecf6ad&quot;,
  &quot;created_by_user_id&quot;: &quot;6aadfc0a-da22-4e69-b614-4e1699d9e8eb&quot;,
  &quot;ended_at&quot;: null,
  &quot;id&quot;: &quot;ab68da77-fa82-4e67-aba9-270c5a98cbcb&quot;,
  &quot;metrics&quot;: {
    &quot;temperature&quot;: &quot;ad53cf29-6d23-48c5-87c1-f3bf5e8bb4a0&quot;
  },
  &quot;project_id&quot;: &quot;bd3a1e52-1c62-44cb-bf04-660bd88cd74d&quot;,
  &quot;revision_end&quot;: null,
  &quot;revision_start&quot;: &quot;2015-04-17T14:39:22.181615Z&quot;,
  &quot;started_at&quot;: &quot;2015-04-17T14:39:22.181601Z&quot;,
  &quot;type&quot;: &quot;generic&quot;,
  &quot;user_id&quot;: &quot;bd3a1e52-1c62-44cb-bf04-660bd88cd74d&quot;
}
&lt;/code&gt;&lt;/pre&gt;
&lt;h2&gt;Haystack, needle? Find!&lt;/h2&gt;
&lt;p&gt;With such a system, it becomes very easy to index all your resources, meter them and retrieve this data. What&apos;s even more interesting is to query the system to find and list the resources you are interested in!&lt;/p&gt;
&lt;p&gt;You can search for a resource based on any field, for example:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;POST /v1/search/resource/instance HTTP/1.1
Content-Type: application/json

{
  &quot;=&quot;: {
    &quot;user_id&quot;: &quot;bd3a1e52-1c62-44cb-bf04-660bd88cd74d&quot;
  }
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;That query will return a list of all resources owned by the &lt;code&gt;user_id&lt;/code&gt; &lt;code&gt;bd3a1e52-1c62-44cb-bf04-660bd88cd74d&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;You can do fancier queries such as retrieving all the instances started by a user this month:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;POST /v1/search/resource/instance HTTP/1.1
Content-Type: application/json
Content-Length: 113

{
  &quot;and&quot;: [
    {
      &quot;=&quot;: {
        &quot;user_id&quot;: &quot;bd3a1e52-1c62-44cb-bf04-660bd88cd74d&quot;
      }
    },
    {
      &quot;&amp;gt;=&quot;: {
        &quot;started_at&quot;: &quot;2015-04-01&quot;
      }
    }
  ]
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;And you can even do fancier queries than the fancier ones (still following?). What if we wanted to retrieve all the instances that were on host &lt;code&gt;foobar&lt;/code&gt; the 15th April and who had already 30 minutes of uptime? Let&apos;s ask Gnocchi to look in the history!&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;POST /v1/search/resource/instance?history=true HTTP/1.1
Content-Type: application/json
Content-Length: 113

{
  &quot;and&quot;: [
    {
      &quot;=&quot;: {
        &quot;host&quot;: &quot;foobar&quot;
      }
    },
    {
      &quot;&amp;gt;=&quot;: {
        &quot;lifespan&quot;: &quot;1 hour&quot;
      }
    },
    {
      &quot;&amp;lt;=&quot;: {
        &quot;revision_start&quot;: &quot;2015-04-15&quot;
      }
    }

  ]
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;I could also mention the fact that you can &lt;a href=&quot;http://docs.openstack.org/developer/gnocchi/rest.html#searching-for-values-in-metrics&quot;&gt;search for value in metrics&lt;/a&gt;.&lt;br /&gt;
One feature that I will very likely include in Gnocchi 1.1 is the ability to search for resource whose specific metrics matches some value. For example, having the ability to search for instances whose CPU consumption was over 80% during a month.&lt;/p&gt;
&lt;h2&gt;Cherries on the cake&lt;/h2&gt;
&lt;p&gt;While Gnocchi is well integrated and based on common OpenStack technology, please do note that it is completely able to function without any other OpenStack component and is pretty straight-forward to deploy.&lt;/p&gt;
&lt;p&gt;Gnocchi also implements a full RBAC system based on the &lt;a href=&quot;http://docs.openstack.org/developer/oslo.policy/&quot;&gt;OpenStack standard oslo.policy&lt;/a&gt; and which allows pretty fine grained control of permissions.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-resource-html.png&quot; alt=&quot;gnocchi-resource-html&quot; /&gt;&lt;/p&gt;
&lt;p&gt;There is also some work ongoing to have HTML rendering when browsing the API using a Web browser. While still simple, we&apos;d like to have a minimal Web interface served on top of the API for the same price!&lt;/p&gt;
&lt;p&gt;Ceilometer alarm subsystem supports Gnocchi with the Kilo release, meaning you can use it to trigger actions when a metric value crosses some threshold. And OpenStack &lt;a href=&quot;http://launchpad.net/heat&quot;&gt;Heat&lt;/a&gt; also supports auto-scaling your instances based on Ceilometer+Gnocchi alarms.&lt;/p&gt;
&lt;p&gt;And there are a few more API calls that I didn&apos;t talk about here, so don&apos;t hesitate to take a peek at the &lt;a href=&quot;http://gnocchi.xyz&quot;&gt;full documentation&lt;/a&gt;!&lt;/p&gt;
&lt;h2&gt;Towards Gnocchi 1.1!&lt;/h2&gt;
&lt;p&gt;Gnocchi is a different beast in the OpenStack community. It is under the umbrella of the Ceilometer program, but it&apos;s one of the first projects that is not part of the (old) integrated release. Therefore we decided to have a release schedule not directly linked to the OpenStack and we&apos;ll release more often that the rest of the old OpenStack components – probably once every 2 months or the like.&lt;/p&gt;
&lt;p&gt;What&apos;s coming next is a close integration with Ceilometer (e.g. moving the dispatcher code from Gnocchi to Ceilometer) and probably more features as we have more requests from our users. We are also exploring different backends such as InfluxDB (storage) or MongoDB (indexer).&lt;/p&gt;
&lt;p&gt;Stay tuned, and happy hacking!&lt;/p&gt;
</content:encoded><category>gnocchi</category><category>openstack</category></item><item><title>OpenStack Ceilometer and the Gnocchi experiment</title><link>https://julien.danjou.info/blog/openstack-ceilometer-the-gnocchi-experiment/</link><guid isPermaLink="true">https://julien.danjou.info/blog/openstack-ceilometer-the-gnocchi-experiment/</guid><description>A little more than 2 years ago, the Ceilometer project was launched inside the OpenStack ecosystem. Its main objective was to measure OpenStack cloud platforms in order to provide data and mechanisms.</description><pubDate>Mon, 18 Aug 2014 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;A little more than 2 years ago, the &lt;a href=&quot;http://launchpad.net/ceilometer&quot;&gt;Ceilometer&lt;/a&gt; project was launched inside the OpenStack ecosystem. Its main objective was to measure OpenStack cloud platforms in order to provide data and mechanisms for functionalities such as billing, alarming or capacity planning.&lt;/p&gt;
&lt;p&gt;In this article, I would like to relate what I&apos;ve been doing with other Ceilometer developers in the last 5 months. I&apos;ve lowered my involvement in Ceilometer itself directly to concentrate on solving one of its biggest issue at the source, and I think it&apos;s largely time to take a break and talk about it.&lt;/p&gt;
&lt;h2&gt;Ceilometer early design&lt;/h2&gt;
&lt;p&gt;For the last years, Ceilometer didn&apos;t change in its core architecture. Without diving too much in all its parts, one of the early design decision was to build the metering around a data structure we called &lt;strong&gt;samples&lt;/strong&gt;. A sample is generated each time Ceilometer measures something. It is composed of a few fields, such as the the resource id that is metered, the user and project id owning that resources, the meter name, the measured value, a timestamp and a few free-form metadata. Each time Ceilometer measures something, one of its components (an agent, a pollster…) constructs and emits a sample headed for the storage component that we call the &lt;strong&gt;collector&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;This collector is responsible for storing the samples into a database. The Ceilometer collector uses a pluggable storage system, meaning that you can pick any database system you prefer. Our original implementation has been based on MongoDB from the beginning, but we then added a SQL driver, and people contributed things such as HBase or DB2 support.&lt;/p&gt;
&lt;p&gt;The REST API exposed by Ceilometer allows to execute various reading requests on this data store. It can returns you the list of resources that have been measured for a particular project, or compute some statistics on metrics. Allowing such a large panel of possibilities and having such a flexible data structure allows to do a lot of different things with Ceilometer, as you can almost query the data in any mean you want.&lt;/p&gt;
&lt;h2&gt;The scalability issue&lt;/h2&gt;
&lt;p&gt;We soon started to encounter scalability issues in many of the read requests made via the REST API. A lot of the requests requires the data storage to do full scans of all the stored samples. Indeed, the fact that the API allows you to filter on any fields and also on the free-form metadata (meaning non indexed key/values tuples) has a terrible cost in terms of performance (as pointed before, the metadata are attached to each &lt;em&gt;sample&lt;/em&gt; generated by Ceilometer and is stored as is). That basically means that the &lt;em&gt;sample&lt;/em&gt; data structure is stored in most drivers in just one table or collection, in order to be able to scan them at once, and there&apos;s no good &quot;perfect&quot; sharding solution, making data storage scalability painful.&lt;/p&gt;
&lt;p&gt;It turns out that the Ceilometer REST API is unable to handle most of the requests in a timely manner as most operations are &lt;em&gt;O(n)&lt;/em&gt; where &lt;em&gt;n&lt;/em&gt; is the number of samples recorded (see &lt;a href=&quot;http://en.wikipedia.org/wiki/Big_O_notation&quot;&gt;big O notation&lt;/a&gt; if you&apos;re unfamiliar with it). That number of samples can grow very rapidly in an environment of thousands of metered nodes and with a data retention of several weeks. There is a few optimizations to make things smoother in general cases fortunately, but as soon as you run specific queries, the API gets barely usable.&lt;/p&gt;
&lt;p&gt;During this last year, as the Ceilometer PTL, I discovered these issues first hand since a lot of people were feeding me back with this kind of testimony. We engaged several blueprints to improve the situation, but it was soon clear to me that this was not going to be enough anyway.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/unacceptable.jpg&quot; alt=&quot;unacceptable&quot; /&gt;&lt;/p&gt;
&lt;h2&gt;Thinking outside the box&lt;/h2&gt;
&lt;p&gt;Unfortunately, the PTL job doesn&apos;t leave him enough time to work on the actual code nor to play with anything new. I was coping with most of the project bureaucracy and I wasn&apos;t able to work on any good solution to tackle the issue at its root. Still, I had a few ideas that I wanted to try and as soon as I stepped down from the PTL role, I stopped working on Ceilometer itself to try something new and to think a bit outside the box.&lt;/p&gt;
&lt;p&gt;When one takes a look at what have been brought recently in Ceilometer, they can see the idea that Ceilometer actually needs to handle 2 types of data: events and metrics.&lt;/p&gt;
&lt;p&gt;Events are data generated when something happens: an instance start, a volume is attached, or an HTTP request is sent to an REST API server. These are events that Ceilometer needs to collect and store. Most OpenStack components are able to send such events using the notification system built into &lt;em&gt;&lt;a href=&quot;https://wiki.openstack.org/wiki/Oslo/Messaging&quot;&gt;oslo.messaging&lt;/a&gt;&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;Metrics is what Ceilometer needs to store but that is not necessarily tied to an event. Think about an instance CPU usage, a router network bandwidth usage, the number of images that Glance is storing for you, etc… These are not events, since nothing is happening. These are facts, states we need to meter.&lt;/p&gt;
&lt;p&gt;Computing statistics for billing or capacity planning requires both of these data sources, but they should be distinct. Based on that assumption, and the fact that Ceilometer was getting support for storing events, I started to focus on getting the metric part right.&lt;/p&gt;
&lt;p&gt;I had been a system administrator for a decade before jumping into OpenStack development, so I know a thing or two on how monitoring is done in this area, and what kind of technology operators rely on. I also know that there&apos;s still no silver bullet – this made it a good challenge.&lt;/p&gt;
&lt;p&gt;The first thing that came to my mind was to use some kind of time-series database, and export its access via a REST API – as we do in all OpenStack services. This should cover the metric storage pretty well.&lt;/p&gt;
&lt;h2&gt;Cooking Gnocchi&lt;/h2&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-logo-old-2.jpg&quot; alt=&quot;gnocchi-logo-old-2&quot; /&gt;&lt;/p&gt;
&lt;p&gt;At the end of April 2014, this led met to start a new project code-named Gnocchi. For the record, the name was picked after confusing so many times the OpenStack Marconi project, reading OpenStack Macaroni instead. At least one OpenStack project should have a &quot;pasta&quot; name, right?&lt;/p&gt;
&lt;p&gt;The point of having a new project and not send patches on Ceilometer, was that first I had no clue if it was going to make something that would be any better, and second, being able to iterate more rapidly without being strongly coupled with the release process.&lt;/p&gt;
&lt;p&gt;The first prototype started around the following idea: what you want is to meter things. That means storing a list of tuples of (timestamp, value) for it. I&apos;ve named these things &quot;entities&quot;, as no assumption are made on what they are. An entity can represent the temperature in a room or the CPU usage of an instance. The service shouldn&apos;t care and should be agnostic in this regard.&lt;/p&gt;
&lt;p&gt;One feature that we discussed for several OpenStack summits in the Ceilometer sessions, was the idea of doing aggregation. Meaning, aggregating samples over a period of time to only store a smaller amount of them. These are things that time-series format such as the &lt;a href=&quot;http://oss.oetiker.ch/rrdtool/&quot;&gt;RRDtool&lt;/a&gt; have been doing for a long time on the fly, and I decided it was a good trail to follow.&lt;/p&gt;
&lt;p&gt;I assumed that this was going to be a requirement when storing metrics into Gnocchi. The user would need to provide what kind of archiving it would need: 1 second precision over a day, 1 hour precision over a year, or even both.&lt;/p&gt;
&lt;p&gt;The first driver written to achieve that and store those metrics inside Gnocchi was based on &lt;a href=&quot;http://graphite.wikidot.com/whisper&quot;&gt;whisper&lt;/a&gt;. Whisper is the file format used to store metrics for the &lt;a href=&quot;http://graphite.wikidot.com/&quot;&gt;Graphite&lt;/a&gt; project. For the actual storage, the driver uses Swift, which has the advantages to be part of OpenStack and scalable.&lt;/p&gt;
&lt;p&gt;Storing metrics for each entities in a different &lt;em&gt;whisper&lt;/em&gt; file and putting them in Swift turned out to have a fantastic algorithm complexity: it was &lt;em&gt;O(1)&lt;/em&gt;. Indeed, the complexity needed to store and retrieve metrics doesn&apos;t depends on the number of metrics you have nor on the number of things you are metering. Which is already a huge win compared to the current Ceilometer collector design.&lt;/p&gt;
&lt;p&gt;However, it turned out that &lt;em&gt;whisper&lt;/em&gt; has a few limitations that I was unable to circumvent in any manner. I needed to patch it to remove a lot of its assumption about manipulating file, or that everything is relative to now (&lt;code&gt;time.time()&lt;/code&gt;). I&apos;ve started to hack on that in my own fork, but… then everything broke. The &lt;em&gt;whisper&lt;/em&gt; project code base is, well, not the state of the art, and have 0 unit test. I was starring at a huge effort to transform &lt;em&gt;whisper&lt;/em&gt; into the time-series format I wanted, without being sure I wasn&apos;t going to break everything (remember, no test coverage).&lt;/p&gt;
&lt;p&gt;I decided to take a break and look into alternatives, and stumbled upon &lt;a href=&quot;http://pandas.pydata.org/&quot;&gt;Pandas&lt;/a&gt;, a data manipulation and statistics library for Python. Turns out that Pandas support time-series natively, and that it could do a lot of the smart computation needed in Gnocchi. I built a new file format leveraging Pandas for computing the time-series and named it &lt;strong&gt;carbonara&lt;/strong&gt; (a wink to both the &lt;a href=&quot;https://github.com/graphite-project/carbon&quot;&gt;Carbon&lt;/a&gt; project and pasta, how clever!). The code is quite small (a third of &lt;em&gt;whisper&lt;/em&gt;&apos;s, 200 SLOC vs 600 SLOC), does not have many of the &lt;em&gt;whisper&lt;/em&gt; limitations and… it has test coverage. These Carbonara files are then, in the same fashion, stored into Swift containers.&lt;/p&gt;
&lt;p&gt;Anyway, Gnocchi storage driver system is designed in the same spirit that the rest of OpenStack and Ceilometer storage driver system. It&apos;s a plug-in system with an API, so anyone can write their own driver. Eoghan Glynn has already started to write a &lt;a href=&quot;http://influxdb.com/&quot;&gt;InfluxDB&lt;/a&gt; driver, working closely with the upstream developer of that database. Dina Belova started to write an &lt;a href=&quot;http://opentsdb.net/&quot;&gt;OpenTSDB&lt;/a&gt; driver. This helps to make sure the API is designed directly in the right way.&lt;/p&gt;
&lt;h2&gt;Handling resources&lt;/h2&gt;
&lt;p&gt;Measuring individual entities is great and needed, but you also need to link them with resources. When measuring the temperature and the number of a people in a room, it is useful to link these 2 separate entities to a resource, in that case the room, and give a name to these relations, so one is able to identify what attribute of the resource is actually measured. It is also important to provide the possibility to store attributes on these resources, such as their owners, the time they started and ended their existence, etc.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-relationship.png&quot; alt=&quot;gnocchi-relationship&quot; /&gt;&lt;/p&gt;
&lt;p&gt;Once this list of resource is collected, the next step is to list and filter them, based on any criteria. One might want to retrieve the list of resources created last week or the list of instances hosted on a particular node right now.&lt;/p&gt;
&lt;p&gt;Resources also need to be specialized. Some resources have attributes that must be stored in order for filtering to be useful. Think about an instance name or a router network.&lt;/p&gt;
&lt;p&gt;All of these requirements led to to the design of what&apos;s called the &lt;em&gt;indexer&lt;/em&gt;. The indexer is responsible for indexing entities, resources, and link them together. The initial implementation is based on &lt;a href=&quot;http://sqlalchemy.org&quot;&gt;SQLAlchemy&lt;/a&gt; and should be pretty efficient. It&apos;s easy enough to index the most requested attributes (columns), and they are also correctly typed.&lt;/p&gt;
&lt;p&gt;We plan to establish a model for all known OpenStack resources (instances, volumes, networks, …) to store and index them into the Gnocchi indexer in order to request them in an efficient way from one place. The generic resource class can be used to handle generic resources that are not tied to OpenStack. It&apos;d be up to the users to store extra attributes.&lt;/p&gt;
&lt;p&gt;Dropping the free form metadata we used to have in Ceilometer makes sure that querying the indexer is going to be efficient and scalable.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-classes.png&quot; alt=&quot;gnocchi-classes&quot; /&gt;&lt;/p&gt;
&lt;h2&gt;REST API&lt;/h2&gt;
&lt;p&gt;All of this is exported via a REST API that was partially designed and documented in the &lt;a href=&quot;http://git.openstack.org/cgit/openstack/ceilometer-specs/tree/specs/juno/gnocchi.rst&quot;&gt;Gnocchi specification in the Ceilometer repository&lt;/a&gt;; though the spec is not up-to-date yet. We plan to auto-generate the documentation from the code as we are currently doing in Ceilometer.&lt;/p&gt;
&lt;p&gt;The REST API is pretty easy to use, and you can use it to manipulate entities and resources, and request the information back.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/gnocchi-architecture.png&quot; alt=&quot;gnocchi-architecture&quot; /&gt;&lt;/p&gt;
&lt;h2&gt;Roadmap &amp;amp; Ceilometer integration&lt;/h2&gt;
&lt;p&gt;All of this plan has been exposed and discussed with the Ceilometer team&lt;br /&gt;
during the last &lt;a href=&quot;https://www.openstack.org/summit/openstack-summit-atlanta-2014/&quot;&gt;OpenStack summit in Atlanta&lt;/a&gt; in May 2014, for the Juno release. I led a session about this entire concept, and convinced the team that using Gnocchi for our metric storage would be a good approach to solve the Ceilometer collector scalability issue.&lt;/p&gt;
&lt;p&gt;It was decided to conduct this project experiment in parallel of the current Ceilometer collector for the time being, and see where that would lead the project to.&lt;/p&gt;
&lt;h2&gt;Early benchmarks&lt;/h2&gt;
&lt;p&gt;Some engineers from Mirantis did a few benchmarks around Ceilometer and also against an early version of Gnocchi, and Dina Belova presented them to us during the mid-cycle sprint we organized in Paris in early July.&lt;/p&gt;
&lt;p&gt;The following graph sums up pretty well the current Ceilometer performance issue. The more you feed it with metrics, the more slow it becomes.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/image03.png&quot; alt=&quot;image03&quot; /&gt;&lt;/p&gt;
&lt;p&gt;For Gnocchi, while the numbers themselves are not fantastic, what is interesting is that all the graphs below show that the performances are stable without correlation with the number of resources, entities or measures. This proves that, indeed, most of the code is built around a complexity of &lt;em&gt;O(1)&lt;/em&gt;, and not &lt;em&gt;O(n)&lt;/em&gt; anymore.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/image00.png&quot; alt=&quot;image00&quot; /&gt;&lt;br /&gt;
&lt;img src=&quot;https://julien.danjou.info/content/images/03/image01.png&quot; alt=&quot;image01&quot; /&gt;&lt;br /&gt;
&lt;img src=&quot;https://julien.danjou.info/content/images/03/image04.png&quot; alt=&quot;image04&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/image05.png&quot; alt=&quot;image05&quot; /&gt;&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/image06.png&quot; alt=&quot;image06&quot; /&gt;&lt;/p&gt;
&lt;h2&gt;Next steps&lt;/h2&gt;
&lt;p&gt;&lt;img src=&quot;https://julien.danjou.info/content/images/03/clement-drawing-gnocchi.jpg&quot; alt=&quot;clement-drawing-gnocchi&quot; /&gt;&lt;/p&gt;
&lt;p&gt;While the Juno cycle is being wrapped-up for most projects, including Ceilometer, Gnocchi development is still ongoing. Fortunately, the composite architecture of Ceilometer allows a lot of its features to be replaced by some other code dynamically. That, for example, enables Gnocchi to provides a Ceilometer dispatcher plugin for its collector, without having to ship the actual code in Ceilometer itself. That should help the development of Gnocchi to not be slowed down by the release process for now.&lt;/p&gt;
&lt;p&gt;The Ceilometer team aims to provide Gnocchi as a sort of technology preview with the Juno release, allowing it to be deployed along and plugged with Ceilometer. We&apos;ll discuss how to integrate it in the project in a more permanent and strong manner probably during the &lt;a href=&quot;https://www.openstack.org/summit/openstack-paris-summit-2014/&quot;&gt;OpenStack Summit for Kilo&lt;/a&gt; that will take place next November in Paris.&lt;/p&gt;
</content:encoded><category>openstack</category><category>gnocchi</category></item></channel></rss>