$ELIXIR_THREADS fallback to nproc is straight forward code, much more
than the incantation to find the path to the Elixir install path.
Remove the incantation and replace by simple code:
if test -z "$ELIXIR_THREADS"; then
ELIXIR_THREADS="$(nproc)"
fi
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
Make utils/index-repository idempotent, meaning we can call it multiple
times on the same repo and same remotes without issues.
Also allow adding new remotes to an existing repo.
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
Simplify the script. We never `cd` into the directory, we instead use
`git -C`. Avoid repeating it by creating a $git variable.
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
This is pretty useful as update-elixir-data gets called often to check
for new updates. Most often, there are none, so checking all remotes at
the same time is useful. This only applies to the kernel, that is the
only project using multiple (three) remotes.
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
Crawl-Delay does have a big impact on the loadavg of the server,
meaning:
- (1) most requests are from crawlers and,
- (2) most crawlers listen to Crawl-Delay.
The prod server can handle the current loadavg just fine, let's let them
up their game.
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
lib.getFileFamily() returns None for many files. Our assert to ensure
the family is valid should only be done once we have checked the family
is NOT None.
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
Add a list (CACHED_DEFINITIONS_FAMILIES) that tells us which families
have their definitions cached. We use that to assert at DB.__init__()
and q.query('file') that everything is working as expected.
If someone modifies lib.getFileFamily() for example, we'll get an
explicit warning that we should add a definitions cache to that new
family.
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
By default it never restarts. It is a good idea to avoid leaks across
weeks of a Python process running.
Also remove useless comment about processes value (16 is probably higher
than CPU count and will be fine).
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
`./script.sh get-latest <offset>` gets the full list of tags, filters
it, sorts it then returns a single result. On the Python side, it gets
the first one. If that works, it uses it, else it tries the second one,
etc.
That is a weird implementation: modify get-latest to return all tags so
that Python code only has to spawn a single subprocess.
Also, rename it from `get-latest` to `get-latest-tags`. This makes
things more explicit (what latest?) and also explicits that more than
one tag is required.
Also, argument 1 is supposed to be an offset.
No custom implementation of get_latest() did implement that.
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
This code was required because we messed-up in the past regarding
caching headers. This is not required anymore because the caching set
has expired, so no well behaving user-agent should have remains.
This represents something like 240k requests to the backend (not the
cache) over two weeks. Server load was minimal because generating those
responses is really fast.
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
Apparently we might need this for some C code extensions:
> Forcing a WSGI application to run within the first interpreter can be
> necessary when a third party C extension module for Python has used
> the simplified threading API for manipulation of the Python GIL and
> thus will not run correctly within any additional sub interpreters
> created by Python.
https://www.modwsgi.org/en/latest/configuration-directives/WSGIApplicationGroup.html
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
Lookup if a definition exists is taking too long to render source code.
Generate small databases that only tell us if a definition exists for a
given family. Because the database is much smaller, it is faster to
query.
Many URLs could only be queried at 12 req/s. With that patch, I can do
>80 req/s on the same URLs, with the same config.
We generate the caches from update.py. We also add an edge-case to
generate the files (if they don't exist) even if no new tag exists.
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
This behavior isn't used at the moment, but is much more sensible that
the past that was to give None to whatever content-type lambda we had
(eg DefList that would call split on the None).
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
Before, we copied all source files at the start of the Dockerfile; that
meant we always rebuilt most steps. Only copy requirements.txt first,
then copy the rest after many steps.
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
Set env variable $ELIXIR_ROOT inside the Dockerfile. That is used by
utils/index-all-repositories to know where to put repo/data directories.
With that adjustment the docker setup is working as indicated in the
README.
Fix the following error:
$ docker exec -it -e PYTHONUNBUFFERED=1 elixir-container /bin/bash -c \
'export "PATH=/usr/local/elixir/venv/bin:$PATH" ; /usr/local/elixir/utils/index-all-repositories'
Error: ELIXIR_ROOT environment variable not set
It's where Elixir data are stored
It seems that now self.db.vers.exists('') returns True for some
projects, perhaps following the change from bsddb3 to berkeleydb.
Modify the logic to always call `./script.sh get-latest` at least once.
Fixes: #349
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
Following official documentation, move away from bsddb3 to berkeleydb.
https://www.jcea.es/programacion/pybsddb.htm
Main change: key and values are now bytes and not strings.
The code diff is really simple because all operations on keys/values is
done from elixir/data.py and that works fine with both bytes and
strings (it does explicit casting).
Add a cache for `exists` queries. Currently, `exists` calls `git
ls-tree` and parses the result to check if a file exists. A single
call takes around 20-30 ms.
It only gets used by Makefile filters. Large Makefiles cause a filter to
make hundreds of these calls, causing filter processing to take
seconds.
Statistics on 20 HTTP requests on /linux/v6.11.6/source/MAINTAINERS:
without: with:
avg 1160 843
median 951 790
75th perc 1289 861
95th perc 2452 1078
max 2874 1749
The cache is stored inside Query, of which there is one instance per
request. We do not risk cache invalidation issues.
About memory usage: on Linux v6.9.4, the cache is 12MB.
Important changes that should make reading the diff easier:
* Script now only works on line numbers. Elements are queried when
necessary
* handleLineRange was refactored into functions that only validate and
return line number(s), it's now the callers responsibility to
highlight the range
