Add a list (CACHED_DEFINITIONS_FAMILIES) that tells us which families
have their definitions cached. We use that to assert at DB.__init__()
and q.query('file') that everything is working as expected.
If someone modifies lib.getFileFamily() for example, we'll get an
explicit warning that we should add a definitions cache to that new
family.
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
By default it never restarts. It is a good idea to avoid leaks across
weeks of a Python process running.
Also remove useless comment about processes value (16 is probably higher
than CPU count and will be fine).
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
`./script.sh get-latest <offset>` gets the full list of tags, filters
it, sorts it then returns a single result. On the Python side, it gets
the first one. If that works, it uses it, else it tries the second one,
etc.
That is a weird implementation: modify get-latest to return all tags so
that Python code only has to spawn a single subprocess.
Also, rename it from `get-latest` to `get-latest-tags`. This makes
things more explicit (what latest?) and also explicits that more than
one tag is required.
Also, argument 1 is supposed to be an offset.
No custom implementation of get_latest() did implement that.
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
This code was required because we messed-up in the past regarding
caching headers. This is not required anymore because the caching set
has expired, so no well behaving user-agent should have remains.
This represents something like 240k requests to the backend (not the
cache) over two weeks. Server load was minimal because generating those
responses is really fast.
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
Apparently we might need this for some C code extensions:
> Forcing a WSGI application to run within the first interpreter can be
> necessary when a third party C extension module for Python has used
> the simplified threading API for manipulation of the Python GIL and
> thus will not run correctly within any additional sub interpreters
> created by Python.
https://www.modwsgi.org/en/latest/configuration-directives/WSGIApplicationGroup.html
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
Lookup if a definition exists is taking too long to render source code.
Generate small databases that only tell us if a definition exists for a
given family. Because the database is much smaller, it is faster to
query.
Many URLs could only be queried at 12 req/s. With that patch, I can do
>80 req/s on the same URLs, with the same config.
We generate the caches from update.py. We also add an edge-case to
generate the files (if they don't exist) even if no new tag exists.
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
This behavior isn't used at the moment, but is much more sensible that
the past that was to give None to whatever content-type lambda we had
(eg DefList that would call split on the None).
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
Before, we copied all source files at the start of the Dockerfile; that
meant we always rebuilt most steps. Only copy requirements.txt first,
then copy the rest after many steps.
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
Set env variable $ELIXIR_ROOT inside the Dockerfile. That is used by
utils/index-all-repositories to know where to put repo/data directories.
With that adjustment the docker setup is working as indicated in the
README.
Fix the following error:
$ docker exec -it -e PYTHONUNBUFFERED=1 elixir-container /bin/bash -c \
'export "PATH=/usr/local/elixir/venv/bin:$PATH" ; /usr/local/elixir/utils/index-all-repositories'
Error: ELIXIR_ROOT environment variable not set
It's where Elixir data are stored
It seems that now self.db.vers.exists('') returns True for some
projects, perhaps following the change from bsddb3 to berkeleydb.
Modify the logic to always call `./script.sh get-latest` at least once.
Fixes: #349
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
Following official documentation, move away from bsddb3 to berkeleydb.
https://www.jcea.es/programacion/pybsddb.htm
Main change: key and values are now bytes and not strings.
The code diff is really simple because all operations on keys/values is
done from elixir/data.py and that works fine with both bytes and
strings (it does explicit casting).
Add a cache for `exists` queries. Currently, `exists` calls `git
ls-tree` and parses the result to check if a file exists. A single
call takes around 20-30 ms.
It only gets used by Makefile filters. Large Makefiles cause a filter to
make hundreds of these calls, causing filter processing to take
seconds.
Statistics on 20 HTTP requests on /linux/v6.11.6/source/MAINTAINERS:
without: with:
avg 1160 843
median 951 790
75th perc 1289 861
95th perc 2452 1078
max 2874 1749
The cache is stored inside Query, of which there is one instance per
request. We do not risk cache invalidation issues.
About memory usage: on Linux v6.9.4, the cache is 12MB.
Important changes that should make reading the diff easier:
* Script now only works on line numbers. Elements are queried when
necessary
* handleLineRange was refactored into functions that only validate and
return line number(s), it's now the callers responsibility to
highlight the range
The current order for displaying references is:
- [definitions] prototype, label, typedef, variable, struct, member,
function
- documentation
- references
This order is not very practical when you are searching for information
about a symbol, as you often want to know if the symbol is documented.
By inverting the order and displaying the documentation first, the user
experience may be improved for finding the documentation. This should not
significantly change the UX for definitions, as there are already many
definitions, and you already have to search for what you want.
Tell to browsers the resources we will need, as soon as possible. This
means they do not need to fully download our CSS to know what they
should load next.
For some reason, crossorigin is required on fonts to avoid them being
loaded twice by Firefox.
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
Avoid many tiny requests. The issue is that browsers have a limited
amount of concurrent requests they can make. We do 19 for
loading /linux/v6.11.6/source. Reduce that to 14.
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
Previously, the web server was responsible for redirecting '/' to a
sensible URL. Most likely, the target URL was '/linux/latest/source'.
From there on, web did the redirect to the proper version.
Avoid a redirect by handling '/' directly from our application to the
correct version.
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
Pygments 2.12.0 slightly changed the order of HTML elements which
makes line numbers look weird with current CSS styles.
Quoting changelog:
> When linenos=table is used, the <table> itself is now wrapped
> with a <div class="highlight"> tag instead of placing it
> inside the <td class="code"> cell (#632.) With this change,
> the output matches the documented behavior.
https://pygments.org/docs/changelog/#version-2-12-0
Related issue: https://github.com/pygments/pygments/issues/632
This commit aims to fix this by further restricting CSS selectors
that selected everything under .highlight to only select source code
related elements - descendants of .code in the new layout.
This was also tested on Pygemnts 2.14.0, version currently packaged
for Debian Bookworm
https://packages.debian.org/bookworm/armel/python3-pygments
* Run pytest only for t, do not run tests for other libraries
* Add email and name git options to test repo init
* Change test repo permissions to avoid "dubious ownership" error
Current update script serializes database access using mutexes.
According to a user of Oracle support forums, this is not enough.
https://forums.oracle.com/ords/apexds/post/berkeley-db-file-corrupted-while-operating-for-hours-panic-4953
> if you are accessing the same database from multiple threads or
> multiple processes, they must share a cache (memory pool). In other
> words, it is not sufficient to just make sure no DB->put or DB->get
> operations are run simultaneously as you do with mutexes. Berkeley
> DB also maintains information about database files across calls in
> the cache, such as the list of free pages. If two threads accessing
> a database file have independent freelists, they will eventually
> both try to allocate the same page for different purposes, and the
> structure of the file will be compromised.
DB.open provides a flag that should be specified if database is to be
shared between threads
https://docs.oracle.com/cd/E17276_01/html/api_reference/C/dbopen.html
> DB_THREAD
> Cause the DB handle returned by DB->open() to be free-threaded; that
> is, concurrently usable by multiple threads in the address space.
> You should use this flag only in the absence of an encompassing
> environment.
While this probably won't solve all database concurrency issues (web
accessing the database during updates likely still will behave weird)
it could help with recent database corruption issues.
https://docs.oracle.com/cd/E17276_01/html/programmer_reference/program_mt.html
> The DB_THREAD flag must be specified to the DB_ENV->open() and
> DB->open() methods if the Berkeley DB handles returned by those
> interfaces will be used in the context of more than one thread.
> Setting the DB_THREAD flag inconsistently may result in database
> corruption.
> When using the non-cursor Berkeley DB calls to retrieve key/data
> items (for example, DB->get()), the memory to which the pointer
> stored into the Dbt refers is valid only until the next call using
> the DB handle returned by DB->open(). This includes any use of the
> returned DB handle, including by another thread within the process.
>
> For this reason, if the DB_THREAD handle was specified to the
> DB->open() method, either DB_DBT_MALLOC, DB_DBT_REALLOC or
> DB_DBT_USERMEM must be specified in the DBT when performing any
> non-cursor key or data retrieval.
It seems that bsddb3 sets appropriate flags in DBTs for us if DB_THREAD
is specified.
https://hg.jcea.es/pybsddb/file/tip/src/Module/berkeleydb.c#l2025
(ctrl+f for DB_THREAD)
I believe DBTs used in DB_put shouldn't require any extra flags because
the DBTs are only read by Berkeley DB (doesn't matter if they get
invalidated on the next call).
Linux is the only currently supported project for which we want three
remote URLs; see index-all-repositories for the list.
We could optimise the fetching by doing a single fetch call to all
remotes at the same time using --jobs (or fetch.parallel config).
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
