Lookup if a definition exists is taking too long to render source code.
Generate small databases that only tell us if a definition exists for a
given family. Because the database is much smaller, it is faster to
query.
Many URLs could only be queried at 12 req/s. With that patch, I can do
>80 req/s on the same URLs, with the same config.
We generate the caches from update.py. We also add an edge-case to
generate the files (if they don't exist) even if no new tag exists.
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
Current update script serializes database access using mutexes.
According to a user of Oracle support forums, this is not enough.
https://forums.oracle.com/ords/apexds/post/berkeley-db-file-corrupted-while-operating-for-hours-panic-4953
> if you are accessing the same database from multiple threads or
> multiple processes, they must share a cache (memory pool). In other
> words, it is not sufficient to just make sure no DB->put or DB->get
> operations are run simultaneously as you do with mutexes. Berkeley
> DB also maintains information about database files across calls in
> the cache, such as the list of free pages. If two threads accessing
> a database file have independent freelists, they will eventually
> both try to allocate the same page for different purposes, and the
> structure of the file will be compromised.
DB.open provides a flag that should be specified if database is to be
shared between threads
https://docs.oracle.com/cd/E17276_01/html/api_reference/C/dbopen.html
> DB_THREAD
> Cause the DB handle returned by DB->open() to be free-threaded; that
> is, concurrently usable by multiple threads in the address space.
> You should use this flag only in the absence of an encompassing
> environment.
While this probably won't solve all database concurrency issues (web
accessing the database during updates likely still will behave weird)
it could help with recent database corruption issues.
https://docs.oracle.com/cd/E17276_01/html/programmer_reference/program_mt.html
> The DB_THREAD flag must be specified to the DB_ENV->open() and
> DB->open() methods if the Berkeley DB handles returned by those
> interfaces will be used in the context of more than one thread.
> Setting the DB_THREAD flag inconsistently may result in database
> corruption.
> When using the non-cursor Berkeley DB calls to retrieve key/data
> items (for example, DB->get()), the memory to which the pointer
> stored into the Dbt refers is valid only until the next call using
> the DB handle returned by DB->open(). This includes any use of the
> returned DB handle, including by another thread within the process.
>
> For this reason, if the DB_THREAD handle was specified to the
> DB->open() method, either DB_DBT_MALLOC, DB_DBT_REALLOC or
> DB_DBT_USERMEM must be specified in the DBT when performing any
> non-cursor key or data retrieval.
It seems that bsddb3 sets appropriate flags in DBTs for us if DB_THREAD
is specified.
https://hg.jcea.es/pybsddb/file/tip/src/Module/berkeleydb.c#l2025
(ctrl+f for DB_THREAD)
I believe DBTs used in DB_put shouldn't require any extra flags because
the DBTs are only read by Berkeley DB (doesn't matter if they get
invalidated on the next call).
This gave suboptimal results in terms of indexing time,
plus a stuck job. Should be investigated further before
changing this.
This reverts commit e1dd2b4881.
Instead of having a complicated thread count scheme and
ending up wasting time waiting for the docs task to finish
with a too low thread count at the end, let's run all
tasks with same number of threads, which by default
is the number of CPUs in the system.
This way, the indexing work always uses as many CPUs
as possible, especially when same tasks are finished
before the others.
The OS shouldn't bother if we try to run more threads
than the actual number of CPUs.
Set the ELIXIR_THREADS environment variable
if you want to use a lower number of threads,
typically on a desktop machine on which you want some
responsiveness.
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
As Chris White said it, it's better to have only one line to change if
we have to update that value.
Signed-off-by: Maxime Chretien (MixLeNain) <mchretien@linuxmail.org>
If an ident is in db.defs it's already a valid ident.
So we need to test the idents with lib.isIdent before adding an ident in
db.defs and not while parsing references.
Signed-off-by: Maxime Chretien <maxime.chretien@bootlin.com>
This prevent the script from locking at line 578 because UpdateIds will never send an event to wake up the threads (line 91) if there is no tag to add.
Also change len(tag_buf) to num_tags in line 548, no need to compute the
length again as we already have it.
Signed-off-by: Maxime Chretien <maxime.chretien@bootlin.com>
We don't need multiple threads for that as it is significantly faster
than defs, refs, docs and comps
Signed-off-by: Maxime Chretien <maxime.chretien@bootlin.com>
Idxes needs to be sorted in RefLists but documentation idxes are lower
than others and we added them latter in the RefList so there was a
problem. The easiest solution to solve that is to create a separate
database for documentations.
Signed-off-by: Maxime Chretien <maxime.chretien@bootlin.com>
The threads are then distributed among the functions using a specific
rule (see code comments)
Signed-off-by: Maxime Chretien <maxime.chretien@bootlin.com>
It's now easier to change the number of threads for each functions
Refs are always what remains at the end so more threads will be helpfull
to finish faster
Signed-off-by: Maxime Chretien <maxime.chretien@bootlin.com>
Use 7 threads to do more things at once and spread the load
Lock file before the loop so we don't lock and unlock all the time
This will hopefully reduce database creation time
Signed-off-by: Maxime Chretien <maxime.chretien@bootlin.com>
This brings the idea of file families, each ident is identified by his
family and can be referenced in compatible families.
For exemple :
- A Kconfig ident can be referenced in a C file but not in a Devicetree file.
- A Devicetree ident is only referenced in Devicetree files.
- A C ident is only referenced in C files.
Kconfig idents are defined without the CONFIG_ at the beginning.
We add it while indexing for an easier processing of other files.
Signed-off-by: Maxime Chretien <maxime.chretien@bootlin.com>
This allows to index multiple things at the same time so it speeds up
database indexation.
Signed-off-by: Maxime Chretien <maxime.chretien@bootlin.com>
- find-file-doc-comments.pl: new file
- data.py: Add database to store doc-comment locations
- script.sh: Add parse-docs subcommand
- update.py:
- Add code to process doc comments
- Update some variable names in hopes of reducing confusion
- query.py:
- Add code to report doc comments
- Update some variable names in hopes of reducing confusion
Also:
- t/TestEnvironment.pm: Add find_doc attribute
- t/interact.pl: Don't die if update.py fails
- t/TestHelpers.pm: Permit checking specific sections of query.py output
- t/300: update regexes per the preceding
- gitignore tags (ctags output) and .cache (api_test.py output)
- For long runs, allows to see which project we are in, and the
percentage of processed tags
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Include C++ files when indexing and tokenizing files.
Unfortunately due to how C++ utilizes namespaces and classes, this
currently introduces a large amount of false positives for unqualified
identifiers that look the same.
Signed-off-by: Chen-Yu Tsai <wens@csie.org>
Python's standard library provides a function to split out file
extensions, and it also handles dot files correctly.
Use that instead of just retrieving the last two characters of
the file name.
This should help with issue #27.
Signed-off-by: Chen-Yu Tsai <wens@csie.org>
