portugalfuturista/elixir
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Converted MD to AsciiDoc

Browse source
This commit is contained in:
Carmeli Tamir 2020-04-18 10:46:44 -04:00
parent fded04ed0c
commit a62164a95f
1 changed files with 142 additions and 135 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

277
README.md → README.adoc
View file

@ -1,22 +1,25 @@
[![Build Status](https://travis-ci.com/bootlin/elixir.svg?branch=master)](https://travis-ci.com/bootlin/elixir)
:doctype: book
:pp: {plus}{plus}
# The Elixir Cross Referencer
image::https://travis-ci.com/bootlin/elixir.svg?branch=master[Build Status,link=https://travis-ci.com/bootlin/elixir]
= The Elixir Cross Referencer
Elixir is a source code cross-referencer inspired by
[LXR](https://en.wikipedia.org/wiki/LXR_Cross_Referencer). It's written
in Python and its main purpose is to index every release of a C or C++
https://en.wikipedia.org/wiki/LXR_Cross_Referencer[LXR]. It's written
in Python and its main purpose is to index every release of a C or C{pp}
project (like the Linux kernel) while keeping a minimal footprint.
It uses Git as a source-code file store and Berkeley DB for cross-reference
data. Internally, it indexes Git *blobs* rather than trees of files to avoid
data. Internally, it indexes Git _blobs_ rather than trees of files to avoid
duplicating work and data. It has a straightforward data structure
(reminiscent of older LXR releases) to keep queries simple and fast.
You can see it in action on https://elixir.bootlin.com/
Note: this documentation applies to version 1.0 of Elixir.
NOTE: this documentation applies to version 1.0 of Elixir.
# Requirements
= Requirements
* Python >= 3.6
* The Jinja2 and Pygments (>= 2.2) Python libraries
@ -25,19 +28,19 @@ Note: this documentation applies to version 1.0 of Elixir.
* Perl (for non-greedy regexes and automated testing)
* Falcon and `mod_wsgi` (for the REST api)
# Installation
= Installation
## Architecture
== Architecture
Elixir has the following architecture:
.---------------.----------------.
| CGI interface | REST interface |
|---------------|----------------.
| Query command | Update command |
|---------------|----------------|
| Shell script |
'--------------------------------'
.---------------.----------------.
| CGI interface | REST interface |
|---------------|----------------.
| Query command | Update command |
|---------------|----------------|
| Shell script |
'--------------------------------'
The shell script (`script.sh`) is the lower layer and provides commands
to interact with Git and other Unix utilities. The Python commands use
@ -47,45 +50,49 @@ databases (`update.py`). Finally, the CGI interface (`web.py`) and
the REST interface (`api.py`) use the query interface to generate HTML
pages and to answer REST queries, respectively.
When installing the system, you should test each layer manually and make
sure it works correctly before moving on to the next one.
## Install Manually
== Install Manually
### Install Dependences
=== Install Dependences
> For RedHat/CentOS
____
For RedHat/CentOS
____
```
----
yum install python36-jinja2 python36-pygments python36-bsddb3 python3-falcon global-ctags git httpd
```
> For Debian
----
```
____
For Debian
____
----
sudo apt install python3 python3-jinja2 python3-pygments python3-bsddb3 python3-falcon python3-pytest exuberant-ctags perl git apache2 libapache2-mod-wsgi-py3
```
----
To enable the REST api, follow the installation instructions on [`mod_wsgi`](https://github.com/GrahamDumpleton/mod_wsgi)
and connect it to the apache installation as detailed in <https://github.com/GrahamDumpleton/mod_wsgi#connecting-into-apache-installation>.
To enable the REST api, follow the installation instructions on https://github.com/GrahamDumpleton/mod_wsgi[`mod_wsgi`]
and connect it to the apache installation as detailed in https://github.com/GrahamDumpleton/mod_wsgi#connecting-into-apache-installation.
To know which packages to install, you can also read the Docker files in the `docker/` directory
to know what packages Elixir needs in your favorite distribution.
### Download Elixir Project
=== Download Elixir Project
```
----
git clone https://github.com/bootlin/elixir.git /usr/local/elixir/
```
----
### Create Directory
=== Create Directory
```
----
mkdir -p /path/elixir-data/linux/repo
mkdir -p /path/elixir-data/linux/data
```
----
### Set environment variables
=== Set environment variables
Two environment variables are used to tell Elixir where to find the project's
local git repository and its databases:
@ -95,58 +102,59 @@ local git repository and its databases:
Now open `/etc/profile` and append the following content.
```
----
export LXR_REPO_DIR=/path/elixir-data/linux/repo
export LXR_DATA_DIR=/path/elixir-data/linux/data
```
----
And then run `source /etc/profile`.
### Clone Kernel source code
=== Clone Kernel source code
First clone the master tree released by Linus Torvalds:
```
----
cd /pathy/elixir-data/linux
git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git repo
```
----
Then, you should also declare a `stable` remote branch corresponding to the `stable` tree, to get all release updates:
```
----
cd repo
git remote add stable git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
git fetch stable
```
----
Feel free to add more remote branches in this way, as Elixir will consider tags from all remote branches.
### First Test
=== First Test
```
----
cd /usr/local/elixir/
./script.sh list-tags
```
----
### Create Database
=== Create Database
```
----
./update.py
```
----
> Generating the full database can take a long time: it takes about 15 hours on a Xeon E3-1245 v5 to index 1800 tags in the Linux kernel. For that reason, you may want to tweak the script (for example, by limiting the number of tags with a "head") in order to test the update and query commands. You can even create a new Git repository and just create one tag instead of using the official kernel repository which is very large.
____
Generating the full database can take a long time: it takes about 15 hours on a Xeon E3-1245 v5 to index 1800 tags in the Linux kernel. For that reason, you may want to tweak the script (for example, by limiting the number of tags with a "head") in order to test the update and query commands. You can even create a new Git repository and just create one tag instead of using the official kernel repository which is very large.
____
### Second Test
=== Second Test
Verify that the queries work:
```
$ ./query.py v4.10 ident raw_spin_unlock_irq
$ ./query.py v4.10 file /kernel/sched/clock.c
```
$ ./query.py v4.10 ident raw_spin_unlock_irq
$ ./query.py v4.10 file /kernel/sched/clock.c
Note: `v4.10` can be replaced with any other tag.
NOTE: `v4.10` can be replaced with any other tag.
### Configure httpd
=== Configure httpd
The CGI interface (`web.py`) is meant to be called from your web
server. Since it includes support for indexing multiple projects,
@ -154,15 +162,15 @@ it expects a different variable (`LXR_PROJ_DIR`) which points to a
directory with a specific structure:
* `<LXR_PROJ_DIR>`
* `<project 1>`
* `data`
* `repo`
* `<project 2>`
* `data`
* `repo`
* `<project 3>`
* `data`
* `repo`
** `<project 1>`
*** `data`
*** `repo`
** `<project 2>`
*** `data`
*** `repo`
** `<project 3>`
*** `data`
*** `repo`
It will then generate the other two variables upon calling the query
command.
@ -171,7 +179,7 @@ Now open `/etc/httpd/conf.d/elixir.conf` and write the following content.
Note: If using apache2 (Ubuntu/Debian) instead of httpd (RedHat/Centos),
the default config file to edit is: `/etc/apache2/sites-enabled/000-default.conf`
```
----
HttpProtocolOptions Unsafe
# Required for HTTP
<Directory /usr/local/elixir/http/>
@ -203,47 +211,48 @@ AddHandler cgi-script .py
RewriteRule "^/$" "/linux/latest/source" [R]
RewriteRule "^/(?!api).*/(source|ident|search)" "/web.py" [PT]
</VirtualHost>
```
----
cgi and rewrite support has been enabled by default in RHEL/CentOS, but you should enable it manually if your distribution is Debian/Ubuntu.
```
----
a2enmod cgi rewrite
```
----
Finally, start the httpd server.
```
----
systemctl start httpd
```
----
### Configure lighthttpd
=== Configure lighthttpd
Here's a sample configuration for lighthttpd:
```
----
server.document-root = server_root + "/elixir/http"
url.redirect = ( "^/$" => "/linux/latest/source" )
url.rewrite = ( "^/.*/(source|ident|search)" => "/web.py/$1")
setenv.add-environment = ( "PYTHONIOENCODING" => "utf-8",
"LXR_PROJ_DIR" => "/path/to/elixir-data" )
```
----
### Using a cache to improve performance
=== Using a cache to improve performance
At Bootlin, we're using the [Varnish http cache](https://varnish-cache.org/)
At Bootlin, we're using the https://varnish-cache.org/[Varnish http cache]
as a front-end to reduce the load on the server running the Elixir code.
.-------------. .---------------. .-----------------------.
| Http client | --------> | Varnish cache | --------> | Apache running Elixir |
'-------------' '---------------' '-----------------------'
.-------------. .---------------. .-----------------------.
| Http client | --------> | Varnish cache | --------> | Apache running Elixir |
'-------------' '---------------' '-----------------------'
### Keeping Elixir databases up to date
=== Keeping Elixir databases up to date
To keep your Elixir databases up to date and index new versions that are released,
we're proposing to use a script like `utils/update-elixir-data` which is called
through a daily cron job.
### Keeping git repository disk usage under control
=== Keeping git repository disk usage under control
As you keep updating your git repositories, you may notice that some can become
considerably bigger than they originally were. This seems to happen when a `gc.log`
@ -252,12 +261,13 @@ to fail, and therefore causing the repository to consume disk space at a fast
pace every time new objects are fetched.
When this happens, you can save disk space by packing git directories as follows:
```
----
cd <bare-repo>
git prune
rm gc.log
git gc --aggressive
```
----
Actually, a second pass with the above commands will save even more space.
@ -265,23 +275,23 @@ To process multiple git repositories in a loop, you may use the
`utils/pack-repositories` that we are providing, run from the directory
where all repositories are found.
## Building Docker images
== Building Docker images
Docker files are provided in the `docker/` directory. To generate your own
Docker image for indexing the sources of a project (for example for the Musl
project which is much faster to index that Linux), download the `Dockerfile`
file for your target distribution and run:
$ docker build -t elixir --build-arg GIT_REPO_URL=git://git.musl-libc.org/musl --build-arg PROJECT=musl .
$ docker build -t elixir --build-arg GIT_REPO_URL=git://git.musl-libc.org/musl --build-arg PROJECT=musl .
Then you can use your new container as follows (you get the container id from the output of `docker build`):
$ docker run <container-id>
$ docker run <container-id>
You can the open the below URL in a browser on your host: http://172.17.0.2/musl/latest/source
(change the container IP address if you don't get the default one)
# Database design
= Database design
`./update.py` stores a bidirectionnal mapping between git object hashes ("blobs") and a sequential key.
The goal of indexing such hashes is to reduce their storage footprint (20 bytes for a SHA-1 hash
@ -289,7 +299,7 @@ versus 4 bytes for a 32 bit integer).
A detailed diagram of the databases will be provided. Until then, just use the Source, Luke.
# Hardware requirements
= Hardware requirements
Performance requirements depend mostly on the amount of traffic that you get
on your Elixir service. However, a fast server also helps for the initial
@ -301,12 +311,12 @@ git repositories.
At Bootlin, here are a few details about the server we're using:
* As of July 2019, our Elixir service consumes 17 GB of data (supporting all projects),
or for the Linux kernel alone (version 5.2 being the latest), 12 GB for indexing data,
and 2 GB for the git repository.
or for the Linux kernel alone (version 5.2 being the latest), 12 GB for indexing data,
and 2 GB for the git repository.
* We're using an LXD instance with 8 GB of RAM on a cloud server with 8 CPU cores
running at 3.1 GHz.
running at 3.1 GHz.
# Supporting a new project
= Supporting a new project
Elixir has a very simple modular architecture that allows to support
new source code projects by just adding a new file to the Elixir sources.
@ -315,7 +325,7 @@ Elixir's assumptions:
* Project sources have to be available in a git repository
* All project releases are associated to a given git tag. Elixir
only considers such tags.
only considers such tags.
First make an installation of Elixir by following the above instructions.
See the `projects` subdirectory for projects that are already supported.
@ -323,8 +333,8 @@ See the `projects` subdirectory for projects that are already supported.
Once Elixir works for at least one project, it's time to clone the git
repository for the project you want to support:
cd /srv/git
git clone --bare https://github.com/zephyrproject-rtos/zephyr
cd /srv/git
git clone --bare https://github.com/zephyrproject-rtos/zephyr
After doing this, you may also reference and fetch remote branches for this project,
for example corresponding to the `stable` tree for the Linux kernel (see the
@ -333,16 +343,16 @@ instructions for Linux earlier in this document).
Now, in your `LXR_PROJ_DIR` directory, create a new directory for the
new project:
cd $LXR_PROJ_DIR
mkdir -p zephyr/data
ln -s /srv/git/zephyr.git repo
export LXR_DATA_DIR=$LXR_PROJ_DIR/data
export LXR_REPO_DIR=$LXR_PROJ_DIR/repo
cd $LXR_PROJ_DIR
mkdir -p zephyr/data
ln -s /srv/git/zephyr.git repo
export LXR_DATA_DIR=$LXR_PROJ_DIR/data
export LXR_REPO_DIR=$LXR_PROJ_DIR/repo
Now, go back to the Elixir sources and test that tags are correctly
extracted:
./script.sh list-tags
./script.sh list-tags
Depending on how you want to show the available versions on the Elixir pages,
you may have to apply substitutions to each tag string, for example to add
@ -351,13 +361,13 @@ shown. You may also decide to ignore specific tags. All this can be done
by redefining the default `list_tags()` function in a new `project/<projectname>.sh`
file. Here's an example (`projects/zephyr.sh` file):
list_tags()
{
echo "$tags" |
grep -v '^zephyr-v'
}
list_tags()
{
echo "$tags" |
grep -v '^zephyr-v'
}
Note that `<project_name>` **must** match the name of the directory that
Note that `<project_name>` *must* match the name of the directory that
you created under `LXR_PROJ_DIR`.
The next step is to make sure that versions are classified as you wish
@ -365,15 +375,15 @@ in the version menu. This classification work is done through the
`list_tags_h()` function which generates the output of the `./scripts.sh list-tags -h`
command. Here's what you get for the Linux project:
v4 v4.16 v4.16
v4 v4.16 v4.16-rc7
v4 v4.16 v4.16-rc6
v4 v4.16 v4.16-rc5
v4 v4.16 v4.16-rc4
v4 v4.16 v4.16-rc3
v4 v4.16 v4.16-rc2
v4 v4.16 v4.16-rc1
...
v4 v4.16 v4.16
v4 v4.16 v4.16-rc7
v4 v4.16 v4.16-rc6
v4 v4.16 v4.16-rc5
v4 v4.16 v4.16-rc4
v4 v4.16 v4.16-rc3
v4 v4.16 v4.16-rc2
v4 v4.16 v4.16-rc1
...
The first column is the top level menu entry for versions.
The second one is the next level menu entry, and
@ -387,58 +397,55 @@ to customize the `list_tags_h` function.
You should also make sure that Elixir properly identifies
the most recent versions:
./script.sh get-latest
./script.sh get-latest
If needed, customize the `get_latest()` function.
You are now ready to generate Elixir's database for your
new project:
./update.py
./update.py
You can then check that Elixir works through your http server.
# REST api usage
= REST api usage
After configuring httpd, you can test the api usage:
## ident query
== ident query
Send a get request to ```/api/ident/<Project>/<Ident>?version=<version>```.
Send a get request to `/api/ident/<Project>/<Ident>?version=<version>`.
For example:
curl http://127.0.0.1/api/ident/barebox/cdev?version=latest
curl http://127.0.0.1/api/ident/barebox/cdev?version=latest
The response body is of the following structure:
```
----
{
"definitions":
[{"path": "commands/loadb.c", "line": 71, "type": "variable"}, ...],
"references":
[{"path": "arch/arm/boards/cm-fx6/board.c", "line": "64,64,71,72,75", "type": null}, ...]
}
```
----
# Automated testing
= Automated testing
Elixir includes a simple test suite in `t/`. To run it,
from the top-level Elixir directory, run:
prove
prove
The test suite uses code extracted from Linux v5.4 in `t/tree`.
## Licensing of code in `t/tree`
== Licensing of code in `t/tree`
The copied code is licensed as described in the [COPYING] file included with
The copied code is licensed as described in the https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/COPYING[COPYING] file included with
Linux. All the files copied carry SPDX license identifiers of `GPL-2.0+` or
`GPL-2.0-or-later`. Per [GNU's compatibility table], GPL 2.0+ code can be used
under GPLv3 provided the combination is under GPLv3. Moreover, [GNU's overview
`GPL-2.0-or-later`. Per https://www.gnu.org/licenses/gpl-faq.en.html#AllCompatibility[GNU's compatibility table], GPL 2.0+ code can be used
under GPLv3 provided the combination is under GPLv3. Moreover, https://www.gnu.org/licenses/license-list.en.html#AGPLv3.0[GNU's overview
of AGPLv3] indicates that its terms "effectively consist of the terms of GPLv3"
plus the network-use paragraph. Therefore, the developers have a good-faith
belief that licensing these files under AGPLv3 is authorized. (See also [this
belief that licensing these files under AGPLv3 is authorized. (See also https://github.com/Freemius/wordpress-sdk/issues/166#issuecomment-310561976[this
issue comment] for another example of a similar situation.)
[COPYING]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/COPYING
[GNU's compatibility table]: https://www.gnu.org/licenses/gpl-faq.en.html#AllCompatibility
[GNU's overview of AGPLv3]: https://www.gnu.org/licenses/license-list.en.html#AGPLv3.0
[this issue comment]: https://github.com/Freemius/wordpress-sdk/issues/166#issuecomment-310561976