Blog

GSoC 2015 - Wrap Up

Tue, 25 Aug 2015 00:00:00 +0000

How time flies by when you’re having fun. Still remember the day my GSoC proposal was selected, like it was yesterday. Now, here we are at the end of the awesome journey.

So what exactly was my project about?

My project was titled “Better Tooling for Baloo”. So, exactly is Baloo? Baloo is a framework responsible for file indexing and search for KDE. It is capable of full text indexing and blazing fast queries, while having a small foot print. My project involved writting and improving introspection and control tools for Baloo.

This invlolved improving balooctl the already existing commmand line tools to control baloo. I ended up adding various options to it:

balooctl status now shows size of the index and state of the indexer i.e. what baloo is upto right now.
Add option balooctl status [file..] which tells if a file is indexed or not and if it is not indexed, is it scheduled to be indexed. This option will be useful to debug cases in which user cannot find a specific file.
Add option balooctl monitor which prints out filepaths as they are being indexed. Useful for users which want to know what baloo is doing right now.
Add option balooctl index [file..] which indexes the specified files. Useful for indexing specific files manually for eg. in case they might be in an excluded directory.

Now comes the major part of the project baloo-monitor, I will explain the it’s various features and challenges I faced while implementing them below.

Baloo Monitor

![](/assets/article_images/gsoc/monitor_final.png)

As can be seen from the screenshot, the application shows the user baloo’s current state, file being indexed, total progress and estimated remaining time and a button to suspend/resume indexing.

Challenges

There were loads of challenges I had to face to get there. The first one being the baloo_file_extractor’s behaviour which I mentioned in my first post. Once that was refactored, I had to deal with the internal queue based architecture which was pretty complex and didn’t work well for introspection. So the next step was to come up with a new architecture which I failed miserably with and finally realizing how hard it is to design software. Then with loads of help of my mentor, a better architecture was thought of which is what baloo uses now and works quite well for introspection.

Following this it was a matter of figuring out how Inter-process communication works via D-Bus and writting the required code for the monitor, which as easy as it sounds now, was pretty hard for me to get working back then.

Predicting the future is hard. Why do I mention it? I had to come up with an estimated remaining time algorithm for the monitor. The only information available to calculate that without insane amount of overhead was time taken to index past files and files left. Therefore predicting the future. The problem with this is time taken to index different types of files varies wildly for instance if we’re indexing a complete e-book thingy vs a simple mp3 file, the former can take more than triple the amount of time.

I tried the simplest approach to begin with, simply average the time taken to index a single batch till now and multiply it with the number of batches. (we index file in batches). That didn’t work out quite well, because we could have a situation where the bathes in the beginning were super quick which would drive down the average and the we encounter batches of text heavy files which would slow down the indexing but wouldn’t have enough impact on the average time.

The approach I finally used was to keep track of the time taken to index 5 most recent batches of files and use that to calculate a weighted average giving the most recent batch the highest weight. This approach works better than the simple approach as recent batches have a higher impact on average time but can jump around quite a bit. Still this is the approach I currently use, maybe in the future I may come up with a better one.

The road ahead

We’ve decided that baloo-monitor belongs in KInfoCenter and I am in the process of making a KCM for it. All in all this was an amazing experience and the three months that I’ve learnt the most about various fields of Software Development, from design to implementation. I am going to continue working with KDE for the foreseeable future :).

Thank you KDE and Google for giving me the chance, and a big thanksto my mentor Vishesh Handa for dealing with my, sometimes sloppy, code patiently and poniting out ways to improve.

Edit: Forgot to mention all my work is in the master branch already as I worked in the master branch all along. It was somewhat scary, knowing that your changes can break the core functionality but so much fun as well.

Also, a somewhat alpha verison of the monitor is present in the latest version of baloo (5.13) with loads of fixes and improvements already in the master branch for the next release. Can be launched from the command line using “baloo-monitor”.

Akademy 2015

Mon, 17 Aug 2015 00:00:00 +0000

So, I had the opportunity to attend my first Akademy this year :). For those who don’t know what Akademy is here is a short description: It is the annual world summit of KDE. It includes a 2 day conference in which contributers present the latest developments regarding KDE, followed by 5 days of workshops and sessions in which intrested people hold discussions regarding various KDE projects.

What probably comes to your mind after reading the word contributers is developers, but that is just part of the story. The kde community is made of way more than that. I met artists, designers, translators and lots of other people as well at Akademy. This years Akademy was held at A Coruña, Galicia, Spain.

As I mentioned this was my first Akademy or for that matter my first international FOSS conference, I was pretty damn excited to go, after attending Akademy I must say that my excitement was justified.

So what is so special about Akademy? There are countless things which make it an amazing experience, at the top of the list is: finally getting to meet the people who were just IRC nicks till now.I finally met my GSoC mentor Vishesh Handa with whom I’ve been working for about a year now. When so many like minded, awesome and dedicated people get together there’s loads of stuff happening everywhere and plenty of new stuff to learn.

The First two days had talks related to various parts of KDE the highlight being the announcement of Plasma Mobile, KDE’s attempt at an open platform for mobile which is capable of running full fledged Qt applications and in the future will run android applications as well.

I gave a lightning talk about my GSoC project which I’ve been working on for the past couple of months.

![Me at my Akademy 2015](/assets/article_images/akademy/me.jpg)

The remaining days were followed by workshops, and diverse BoF’s planning a vision for KDE, VDG sessions to help people improve the design for their applications, Plasma mobile and loads more.

Akademy 2015 was the best week I’ve spent till now. A big thanks to KDE E.v. for sponsoring me :)

![](/assets/article_images/akademy/akademy.png)

GSoC 2015 - update 2

Sun, 21 Jun 2015 00:00:00 +0000

It has been a while since my last post, been busy working on my project :p. I must say it has been a great learning experience. With the refactoring done as mentioned in my previous post I began working on CLI tools baloo currently provides, we currently have a tool called balooctl which has loads of options to control baloo. I added 2 more options to the existing set

balooctl status [file..] it tells us about the status of the specified files i.e. if they are indexed or not and what level i.e. the basic indexing and content indexing.
balooctl index [file..] this option can be used to index the specified files manually.

Then I started working with the first major part of my project that is the Monitor. So I began by exporting URL of every file we start indexing over D-Bus after eventually figuring out how D-Bus works. Then I made a simple CLI utility which which on running prints out url of each file as it is indexed. This utility has been merged in the master branch and can be used by calling balooctl monitor.

Now on to the fun part working with a UI for the monitor, I decided to go with qml for the UI mainly because of my familiarity with it (or rather my “unfamiliarity” with QWidgets) and my love for shiny new stuff. I understand that there is a performance penalty by using qml but for now the application seems to be running without any perceivable slowdowns, but it is pretty basic as of now, we’ll see how it goes. Here’s the screenshot of the monitor in action.

![](/assets/article_images/gsoc/monitor.png)

For those of you who want to give it a shot, it has been merged in baloo’s master branch so if you build and install baloo from master you’ll get an application “Baloo Monitor”, it is pretty basic for now.

I’m working an a time estimation logic which will try and not keep the user in the dark as to how much more time the indexing will take it hasn’t been added to the monitor yet. Estimating time accurately is a bigger challenge than I initially anticipated. This is mainly because the varied nature of files on an average computer. Indexing a most of the stuff is almost instantaneous whereas time taken to index the pdf’s or other text heavy stuff largely depends on the amount of text in it, as we index it’s content too. Simply relying on the average time it takes to index a fixed number of files, which is the current state, somewhat works but results in jumpy and sometimes inaccurate results. So time estimation needs refinement. So in the coming weeks I’ll be working on:

Cleaning up the D-bus code (using D-bus interfaces with proper XML descriptions)
Adding more features to the monitor
Refining time estimation

GSoC 2015 Week - 1

Mon, 01 Jun 2015 00:00:00 +0000

As I mentioned before my GSoC proposal titled Better Tooling for Baloo has been accepted.

For the uninitiated, The Google Summer of Code (GSoC) is an international annual program in which Google awards stipends to all students who successfully complete a requested free and open-source software coding project during the summer. (Straight from Wikipedia because I’m lazy)

So, one week has passed since the official coding period began, and I’ve been working on laying the foundation for my project. To begin my project I have re-factored baloo_file_extractor.

Previous behaviour:

Start a KJob with a batch of documents, which started the extractor using QProcess giving all of the file ID’s as arguments when starting the QProcess.
If the extractor ran for more than a fixed amount of time we would assume it is stuck (happens with some files we have trouble processing, though rarely) and proceed by killing it, and restarting it with half the ID’s (keeping the other half for later processing), this was repeated till the we narrow down the file which caused it to get stuck (sort of a binary search)
Repeat the process for every batch of ID’s.

Problems with the approach:

This approach required us to restart the extractor for every batch of ID’s, which was quite frequent.
Narrowing down the file causing erratic behavior, was very tricky.

New behaviour:

We start a persistent extractor process and give it ID’s via stdin.
The extractor writes to stdout the ID it has started indexing and in case the extractor gets stuck we check the last ID written by the extractor, blacklist the ID (also log it somewhere) remove from the batch and resend the batch.

The first part i.e. sending files over stdin has been done, the second part i.e. error handling and logging needs a bit more testing.

As always my mentor Vishesh Handa has been a great help whenever I got stuck. I got stuck mainly, as pointed out by my mentor, because I was making lots of huge untested changes without actually planning the design properly which caused errors that were hard to trace. Lessons learnt:

Always keep testability in mind when designing new stuff.
Always try to make small changes and test them.

GSoC with KDE

Sun, 03 May 2015 00:00:00 +0000

Guess whose GSoC project proposal got accepted? I am so excited, to tell you all that I’ll be working on Baloo along with amazingly experienced KDE developers, this summer, getting to learn loads of stuff and be paid for it! Most importantly I’ll also get a Google Summer of code T-Shirt :) , can’t think of a better way to be spending my summers. Thank you KDE for giving me this chance,and a big thanks to Vishesh Handa for all his guidance and feedback on my proposal.

My project

So, my project is titled: Better Tooling for Baloo. Let me begin by explaining what Baloo is. According to its wiki page it is “Baloo is a metadata and search framework by KDE.” What exactly does it mean? Baloo is responsible for providing full text search capabilities to KDE applications. It doesn’t end there it also provides searching on basis of metadata of various types of files. To acomplish this it indexes file contents and metadata using various plugins ,called extractors, to handle different types of files. It then exposes the data it has indexed with the help of various API’s. So thats a very high level view of how it works. Now, my project, as the title states will provide better tools for Baloo. These tools will mainly be:

A UI (and some CLI tools too) to control and monitor Baloo’s progress and current status. This will give power users more control over Baloo and also make debugging erratic behaviour much easier. The UI will be something along these lines:

![Monitor Mockup](/assets/article_images/gsoc-intro/monitor_mockup.png)

there will be certain advanced features like logging, which will keep track of which files have been indexed, how much time it took to index the file and which files, if any, made the extractor crash.

A tool to visualize which types of files take up how much space in the system currently. This will be most likely a pie chart which would look like:

![Space Visualization Mockup](/assets/article_images/gsoc-intro/pie_mockup.png) These designs are simply mockups that I made for my proposal. They will be improved with feedback from KDE's Visual Design Group and the community. There's a long road ahead and loads of stuff to be learnt, awesome code to write! Looking forward to it. For those of you interested in more details about my project here's a link to my proposal: [Google Drive](https://drive.google.com/file/d/0BxyaYrqdfxSOYnNVUm9jNHVNWU0/view?usp=sharing)

conf.kde.in 2015 - The Initiation

Sun, 19 Apr 2015 00:00:00 +0000

So I finally have something blog worthy after a long time or at least writing this post will be a decent way to make it through the last 5 hours of journey back from Kerala. I feel a bit of context is required here, a while back I had submitted a talk proposal for conf.kde.in 2015, which, to my delight, was accepted. This years conference was held at Amritapuri, thanks to the insane amounts of effort put in by the FOSS club at Amrita Vishwa Vidyapeetham University, it went on smoothly. A special shout out to R.Harish Navnit. This was my first KDE conference and I must say it was amazing. I met awesome KDE India people, learnt a lot from the other talks and most importantly made new friends who love KDE.

Enough small talk talk a bit about the conference now. The conference opened with our keynote speaker Noufal Ibrahim, Founder of PyCon India, giving a demo of combining command line utilities to create a summary of Moby Dick from the book. Straight to demonstrations no boring stuff, that’s how we roll. Looking for clues like good old

![Sherlock Holmes](/assets/article_images/cki15/sherlock.gif)

Noufal, even though not a KDE user, did an amazing job of showing how powerful, small & reusable utilities can be, when combined creatively.

![Pradeepto and Shantanu](/assets/article_images/cki15/pradeepto.jpg)

Then Pradeepto and Shantanu took the stage to tell the students about what KDE is, being involved with KDE for ages now, these guys are obviously the best people for the job. They demonstrated lots of KDE software, told the students about the KDE community and motivated them to contribute. This was followed by Somsubhra’s talk on Krita in which he demonstrated it’s power with the help of tons of videos.

Then after a short break for lunch came the moment I’d been simultaneously dreading and looking forward to, my talk. As I already stated, this was my first conf.kde.in, it was also my first talk at such huge event. I was pretty anxious about it, but encouragement from the other the speakers specially Devaja Shah helped me calm my nerves, and I went on to give a decent talk which I hope motivated the students there to contribute to KDE. In my talk I shared my experience with the KDE community, told the students about my SoK project with Baloo, getting started with code contributions complete with demos of IRC, fetching, building, changing code and generating and submiting patches. One major aspect of my talk was to get the students to start using KDE and improving what they feel needs improving, scratching their own itch.

![me!](/assets/article_images/cki15/pinak.jpg)

My talk, was followed by a hands on session by Shantanu for QML. It was amazing to see students reading around the documentation and experimenting to do stuff they wanted to and not just sticking to what they were being taught. He started with basics and ended with animations. I have to say Shantanu is a good teacher, even I learnt a couple of things from the session. The first day concluded with students interacting with the speakers asking their queries.

Sadly I missed pre-lunch talks the second day by Sanjiban, Sinny, Jigar and Rishab as I was helping out Shantanu in hands on QML sessions for students who’d missed the opportunity on the first day. But I’m sure they did an amazing job looking at the students’ enthusiasm. After lunch, Devaja Shah took the students on a journey through the KDE galaxy giving a tour of lot of planets (read KDE projects).This followed by her presenting other ways of contributing to KDE apart from coding. This included participating in the Promo team, writing dot stories and helping in localization of KDE software.

The last two talks were by Ashish Madeti who gave impressive demo’s of his GSoC project with PMC, by playing some awesome music on Plasma Media center using MPRIS and Karan Luthra who gave an amazing presentation on Trojitá the IMAP e-mail client. His talk helped the students in understanding concepts of IMAP, what Trojitá exactly is and how they can contribute to it. All in all it was an amazing experience with enthusiastic students eager to learn new things. I hope we get tons of new contributers. I finally met Pradeepto Bhattacharya, the founder of KDE India, talked a lot about KDE and random stuff. It was pretty amazing meeting other KDE lovers. This was my indeed my initiation to the KDE India community and I hope to be there at future events we hold.

![Group photo](/assets/article_images/cki15/group.jpg)

A few photographs I took:

Now to the fun part, exploring Amritapuri. After the conference we went on to watch the sunset from 18th floor of a building along the coast. Amazing thing to witness. Amritapuri is a beautiful place situated along the western coastline of India. Finally saw the renowned Kerala backwaters from a bird’s eye view.

![Sunset](/assets/article_images/cki15/sunset.jpg) ![Backwaters](/assets/article_images/cki15/backwaters.jpg)

SoK conclusion

Sat, 14 Mar 2015 00:00:00 +0000

My SoK journey has come to an end. It was the best learning experience ever. As the say no better way to learn than to get your hands dirty. My project, even though it was more oriented towards research, gave me hands on experience with Baloo’s codebase. Looking for a better alternative to Xapian, which is after all a core part of Baloo, is no small task.

What I acomplished:

Read the code for current implementation which uses the Xapian backend and figured out why it falls short for our use case. Compiled a list with Vishesh Handa Xapaian Problems.
Read Lucene’s documentation and and implemented a proof of concept port of Baloo using Lucene. The code can be found here.

Outcome:

We will be benchmarking the various backends (Xapian, Lucene++, custom backend - more on that later) once I iron out some minor kinks with the lucene port. While working with Lucene I realized it is plagued with some of the same problems as Xapian, namely relies on exceptions, concurrency etc. Also one of the main reason that prevents us from using Lucene is that there is no properly maintained C++ port. For my project I worked with Lucene++ which lags behind the JAVA Lucene by a major Version.(Lucene++ is 3.0.x and JAVA lucene is at 4.10.x) and the developer has no plans in catching up. So it is very unlikely we will use Lucene++ in Baloo.

What Lies Ahead:

Vishesh Handa is working on a custom solution for replacing Xapian and it is looking very promising. Now that I have gotten really familiar with Baloo’s architecture maybe I can help around with that. Also I will continue contributing to KDE projects. A shout out to the wonderful KDE community which has, along with giving me an awesome Desktop Environment, has given me the power and knowledge to contribute back and help in shaping its future. Also a big thanks to my mentor Vishesh Handa who was always there to help me when I got stuck.

SSD Yay!

Tue, 20 Jan 2015 00:00:00 +0000

I had been thinking about a way to improve my laptops performance. Finally came to the conclusion that my 5400 RPM hard disk drive was bogging down my otherwise decent laptop. So I started looking around for a solid state drive and zeroed on to Samsung’s 840 evo (250gb). Just when I was about to order I came across an article announcing the release of its successor the 850 evo. Now this caused a dilemma, buy the tried and tested 840 evo or wait for a while and get the latest and greatest 850 evo. Considering the price difference was not that big I decided to wait. Why not get the most recent one? When I finally found a listing on ebay for the 850, checked the sellers feedback (important thing to consider when you’re ordering something this costly), placed my order as fast as I could. Then began the period of waiting. Check tracking number. Why the hell hasn’t he shipped it yet? After it was shipped, check the freaking tracking number 10 times a day to see the status. Excitement can make you insanely impatient. After 3-4 days when it finally arrived, ripped the packing open, grabbed my screw driver kit and replaced my HDD. But this wasn’t it, I had wait a little more. I had to install an OS. I am an arch Linux user and previously had windows 8 on dualboot. This time I decided windows wasn’t worth the precious space on my SSD (250 GB). Also this time I decided to switch to UEFI as it supposedly gives better boot speeds. The first challenge was booting the arch’s live USB in UEFI mode, it kept saying “failed to override security policy”. Turns out my laptop’s UEFI implementation doesn’t support secure boot, which I found out after searching around.

The fix was simply: replacing bootx64.efi with loader.efi in the installation media.

Decided to use gummiboot as my bootloader this time, as grub2 seemed a bit overkill for my needs, finally when everything was set up. Rebooted and then ‘holy shit’ boot was insanely fast ~10 seconds to fully responsive KDE plasma5 session, not the login screen the actual desktop! Goodbye suspending the laptop. The screenshot at the begging shows the boot time using systemd-analyze 7.553s, pretty impressive! Everything is snappy, chrome launches instantly from a cold start (previously took ages), other applications too launch instantly. I am pretty satisfied with the SSD’s performance.

The obligatory r/w performance benchmarks:

Write speed:

[pinak@asus_arch ~]$ dd if=/dev/zero of=tempfile bs=1M count=1024 conv=fdatasync,notrunc
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 2.14841 s, 500 MB/s

Read speed:

[root@asus_arch pinak]# echo 3 > /proc/sys/vm/drop_caches
[root@asus_arch pinak]# exit
[pinak@asus_arch ~]$ dd if=tempfile of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 2.00183 s, 536 MB/s

SoK Status Report part-2

Thu, 15 Jan 2015 00:00:00 +0000

Been a while since my last post. Exams came in and threw my momentum off. But now I’m back on track. I’ve been working on porting Baloo to use lucene instead of Xapain using Lucene++. Lucene++ is a C++ port of the java Lucene.

The first challenge I faced was adding the lucenePlusPlus library to the project. Which I first solved by adding static paths to the library and include directories. This was the wrong approach as told by my mentor Vishesh Handa, as this would only work on my system. The correct way to do this was to use a cmake finder module for LucenePlusPlus. I’m using one from here.

To use lucene++ a lot of string and other datatype conversions were required between formats Lucene++ uses and their Qt counterparts. (Mainly std::wstring <-> QString) So I decided to write a wrapper to provide clean API’s which will be used in my port (using the current Xapian wrapper we have as a reference). I’ll be adding more features to this wrapper as I need them going forward. Currently I’ve ported basicindexingjob in the baloo file component and written a test verify its working, there is a long way yet to go. My working code can be found in lucene branch of baloo’s git repo.

Edit (19th January 2015):

As Kais Hassan mentions that I should probably add an explanation as to how lucene can be utilized by baloo which is primarily a desktop searching solution, here It is: Baloo requires full text indexing and searching capabilities, for which we currently use xapian, but there are some problems with it as I previously stated in my Status report 1.

Lucene, which is primarily used for internet search solutions, can easily be used for desktop searching suites. It uses documents which are the basic unit for searching and indexing. A document consists of Terms which simply map Field -> value. Queries retrieve documents which contain the specified value for a field (Queries can be more complex though and involve many fields). We create a document for every file, when we index the files. These documents contain appropriate values (URL, name, type, content etc) for the file.

SoK Status Report part-1

Sat, 29 Nov 2014 00:00:00 +0000

As mentioned in my previous post I’d be writing about my journey through SoK in upcoming posts, here I am with my first status report.

So far what I’ve learnt/done:

Skimmed through xapian’s ‘getting started’ Documentation to get familiar with the basics of xapian.
Analyzed baloo’s code to know how and why are we using xapian.
Then I implemented a small program using xapian to add index and search. I did this to get familiar the basic API’s. The code can be found in my scratch repository

Note: It is necessary to explain what an xapian document is before proceeding further. I’ll try my best to keep it breif. Xapian Document is the basic unit returned by xapian after a search. The implementation in baloo is: every indexed file has an associated document which contains the terms relevant to the file. This is stored in xapian’s database and mapped to the file using and SQLite database.

Got familiar with xapian-inspect, which is a tool used to inspect contents of an xapian database table. I used this for inspecting tables of the database created by the basic utility I made to understand how an xapian database is organized internally. The main tables in the database are:
- Posting list table: This maps the indexed terms to the documents in which they occur. This is the main table that used in resolving queries.
- Record able: This stores the data associated with a document.
- Term list table: This table maps the documents to the to the terms that occur in the document.
- Position table: This table maps document + term to the position in the document. This information required for phrase queries.
Started looking for problems with xapian in accordance to our use case. I’ve made a wiki page for a list. The problems that I’ve currently figured out are:
- It heavily relies on exceptions. Exceptions are not well supported in Qt and might make the application crash as mentioned here. For example while locking a database Xapian expects the program to catch certain exceptions and retry if they are caught.
- If we want to read and write to an Xapian database simultaneously we need to keep separate copies for reading and writing, thus wasting memory.
- It does not handle data that is changing frequently, if data in document changes to frequently it can lead to a conditions in which locking the database for writing becomes impossible thus making baloo fail.
- Baloo needs support for normalizing text i.e. removing all diacritic marks and also needs to split words with ‘_’ to generate terms, Xapian’s term generator doesn’t provide support for either. So baloo uses its own term generator.
- While searching for something the user may not type complete words so we need to look for every possible expansion of the words in a query, xapian doesn’t provide this feature so we’re using our own query parser.

This list is not exhaustive and I’ll be adding more problems as I figure them out so keep a look out on the wiki page. Once again a big thanks to my mentor Vishesh Handa who guided me through all this.