Esteve Personal Blog

I’ve just emerged genlop, after reading about it in a tutorial about subversion. Genlop parses portage logfiles for information, as it is said in it’s man page.

Explicitily copying it’s man page :

- Nice colorful output.
- Full Portage merge and unmerge history.
- Display date, time and build time of every merge.
- Display total and average build time of selected package[s].
- Estimate upgrade time.
- Watching current merge progress.
- Use alternate portage logfile(s).
- Match package names using regular expressions.
- Log corruption detection.

The most useful function I found is being able to know how much time did an emerge take:
The syntax is easy:

Update:
As Pau noted in a comentary, another useful option is -i :

-i extra infos for the selected package (build specific USE and CFLAGS variables, average build time, etc)

Continuing with the indexing file system from last post, I’ve implemented parsing support for both MP3 metadata and PDF contents, it works in the same way. When you put a PDF or an MP3 file in the indexed directory, a parser gets the contents of the file and indexes them.

For the MP3 it indexes as words all info contained in the TAG and for PDF it converts them to text ( using pstotext ) and indexes every word. As I’ve commented in last post I’m not seeking performance, I know that there are better ways of indexing a PDF, but I just want to try how an indexed filesystem would look like

The next thing that needs to be done is indexing by document type, so being able to search by artist, song, pdf author and so on. There is also need to put more information on found words, like line on text where the word was found, following words, and so on.

Mini Bench:
The time for indexing a 1.3 Mb PDF file in my system is 55 seconds. After that searches for a word under MySql are reported in 0 seconds. This file contained 77241 words that where 7161 unique words.
From this little test it is obvious that the PDF parser is too slow for real use, but for testing is ok.

Code:
Code for the Indexer with MP3 metadata and PDF support

References:
Magic Python: Determines file type
Pstotext homepage

FUSE is a library with a set of function that let you reimplement the VFS layer operations, in other words it lets you write a userspace filesystem, not doing so in kernel space is really flexible, as we can use any external library.

I first became aware of Fuse when I read about GmailFs, which uses a Gmail account as a structured storage, thus becoming a normal mountpoint in your system. GmailFs uses Fuse-Python which are a set of bindings to use Fuse under python. AVFS is another example of Fuse filesystem, it uses tar files as if they were drives, so they can be mounted and accessed without having to untar them.

What I want to use Fuse for, is to make the file indexer more autonomous, with the help of Fuse one might be able to have a fuse mountpoint where all files are always indexed. Once you perform a write operation the indexer might be able to re index the changed portion, when you delete a file it’ll be automatically out of the indexer, and so on. This way there is no need to have the indexer as a daemon which checks for changed files to reindex.

To accomplish this I need to implement an indexing hook on the following operations:

Write
Unlink
Rename
Trunacte

At the end we would have a regular directory, but always synced with the indexing database.

Implementation:
The main goal is just to make a prototype of how would a searchable filesystem look like, is for this reason I have chosen Python ( and also because I’m in process of learning it ). The implementation will make as use of an existing filesystem ( aka reiser, ext, …). Fuse won’t have much to do, just hook all access from one directory to another.
Imagine we want to have an indexed directory called Documents, we would have a mirror in .Documents, where the files are stored using our normal filesystem. We would work normally under Documents, when doing a write operation the hook would catch it and reindex the file, after that the file is saved in the mirror directory. The mirror directory makes our implementation idependent from the filesystem that the user is more comfortable with, we also avoid any problem with data corruption because all the data is stored in a reliable filesystem, we are adding a layer on top of our filesystem, not reinventing the wheel.

Code:
Code for the Indexing Filesystem

References
Fuse SF project
LWN Article on Fuse
Fuse Documentation
GmailFs