Commit graph

47 commits

Author SHA1 Message Date
Théo Lebrun
d8ce24d244 utils/index-repository: refactor by creating project_init() function
Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
2025-01-31 19:27:17 +01:00
Théo Lebrun
a58e8b55bd utils/*: delete common.sh and inline $ELIXIR_THREADS fallback
$ELIXIR_THREADS fallback to nproc is straight forward code, much more
than the incantation to find the path to the Elixir install path.
Remove the incantation and replace by simple code:

    if test -z "$ELIXIR_THREADS"; then
        ELIXIR_THREADS="$(nproc)"
    fi

Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
2025-01-31 19:27:17 +01:00
Théo Lebrun
9d90b07201 utils/index-repository: support calling on existing repository
Make utils/index-repository idempotent, meaning we can call it multiple
times on the same repo and same remotes without issues.

Also allow adding new remotes to an existing repo.

Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
2025-01-31 19:27:17 +01:00
Théo Lebrun
a34a6c692a utils/index-repository: add alias for git -C ...
Simplify the script. We never `cd` into the directory, we instead use
`git -C`. Avoid repeating it by creating a $git variable.

Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
2025-01-31 19:27:17 +01:00
Théo Lebrun
a4b8cb4c02 utils/update-elixir-data: fetch in parallel
This is pretty useful as update-elixir-data gets called often to check
for new updates. Most often, there are none, so checking all remotes at
the same time is useful. This only applies to the kernel, that is the
only project using multiple (three) remotes.

Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
2025-01-31 19:27:17 +01:00
Théo Lebrun
50ef3cd46d utils/index-repository: fetch in parallel
Previous sequence:
 - git clone ...               # first fetch
 - git remote add remote0 ...
 - git fetch remote0           # second fetch
 - git remote add remote1 ...
 - git fetch remote1           # third fetch

Now:
 - git init
 - git remote add remote0 ...
 - git remote add remote1 ...
 - git remote add remote2 ...
 - git fetch --all -j4         # all fetches at the same time

Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
2025-01-31 19:27:17 +01:00
Théo Lebrun
55921f1957 utils/index-repository: support multiple remote URLs
Linux is the only currently supported project for which we want three
remote URLs; see index-all-repositories for the list.

We could optimise the fetching by doing a single fetch call to all
remotes at the same time using --jobs (or fetch.parallel config).

Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
2024-10-02 10:48:03 +02:00
Théo Lebrun
06d503e3d9 utils/index-repository: if $ELIXIR_THREADS is not passed, use nproc
Follow behavior of utils/index-all-repositories.

Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
2024-10-01 16:31:07 +02:00
Théo Lebrun
905e6ef657 utils/index-repository: pass $ELIXIR_THREADS to update.py
Follow behavior of utils/index-all-repositories.

Signed-off-by: Théo Lebrun <theo.lebrun@bootlin.com>
2024-10-01 16:27:10 +02:00
Franciszek Stachura
ffe8d79c40 Simplify and refactor Dockerfile
* Remove repository indexing from Dockerfile
* Copy Elixir code from the context instead of cloning it from Github
* Move Dockerfile from docker/debian/Dockerfile to docker/Dockerfile
* Add docs for the new Dockerfile
* Add an utility script that allows easier repository configuration
* Add .dockerignore
* Use relative paths in Docker section in README
2024-08-05 14:54:03 +02:00
Franciszek Stachura
76b7a8a659 Remove query.default_query and all related references
* Pass process env vars to script.sh ran from Query
2024-02-07 10:31:58 +01:00
Michael Opdenacker
bc4be252df utils/index-all-repositories: add freebsd
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
2024-01-11 09:56:08 +01:00
Michael Opdenacker
527ccae62f utils/index-all-repositories: add Xen project
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
2024-01-10 13:46:23 +01:00
Michael Opdenacker
c156433520 utils: set ELIXIR_THREADS to number of CPUs
This way, you have a correct behaviour even without setting
ELIXIR_THREADS

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
2023-05-05 15:17:27 +02:00
Michael Opdenacker
02ae89c917 utils: figure out Elixir install directory
No more need for ELIXIR_INSTALL variable

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
2023-05-05 15:11:51 +02:00
Artem Chernyshev
ddfc4e8d23 replace iterating over hardcoded project names
Self explanatory. Replacing it with env variable
2023-04-07 12:21:11 +02:00
Michael Opdenacker
c3b8106f85 utils/index-all-repositories: index projects sequentially
Instead of all projects in parallel.
Each indexing task runs parallel jobs anyway.
This makes errors easier to spot and the whole process easier to stop.

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
2023-03-23 15:14:41 +01:00
Michael Opdenacker
b5b0d720fb utils/index-all-repositories: update project repositories
- Replace git:// by https://, more secure, more common
- Update URLs when necessary

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
2023-03-23 15:05:08 +01:00
Michael Opdenacker
2fca2acfff utils/index-all-repositories: remove call to "filename"
No longer available on Ubuntu 22.04, and not necessary
anyway.

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
2023-03-23 12:01:16 +01:00
Ziyue Pan
a627571baf Style: fix redundant parentheses and typos 2023-03-23 09:54:14 +01:00
Michael Opdenacker
00f52df9dc dpdk: also index the "stable" release repository
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
2022-07-29 12:47:42 +02:00
Soenke Huster
f0842f8688
Add BlueZ 2021-10-16 22:51:14 +02:00
Michael Opdenacker
fc517b659d Rename op-tee project
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
2020-11-04 09:00:46 +01:00
Michael Walle
450b1d3a02 Add optee support
Fetch tags that look like a version number, i.e. "[0-9].".

Signed-off-by: Michael Walle <michael@walle.cc>
2020-11-03 14:58:25 +01:00
Michael Opdenacker
8eea9a20c4 utlis/index-all-repository: improve verbosity
- Allowing to know when we switch to the second indexing pass

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
2020-06-19 11:40:41 +02:00
Maxime Chretien
d3a6027e55 utils/index-all-repositories: Add missing parameters
Signed-off-by: Maxime Chretien <maxime.chretien@bootlin.com>
2020-06-19 09:31:59 +02:00
Maxime Chretien
e2d68e3f4b Add Grub support
Signed-off-by: Maxime Chretien <maxime.chretien@bootlin.com>
2020-06-11 17:29:03 +02:00
Maxime Chretien
2e4d967cdb index-all-repositories : Fetch all tags
This is needed for old linux versions

Signed-off-by: Maxime Chretien <maxime.chretien@bootlin.com>
2020-06-08 08:46:18 +02:00
Michael Opdenacker
100ef94665
Merge pull request #152 from MaximeChretien/db-creation-speed
Update.py : Improve threading
2020-06-05 10:30:51 +02:00
Maxime Chretien
f6838a3889 README & scripts : Update for threads argument of update.py
Signed-off-by: Maxime Chretien <maxime.chretien@bootlin.com>
2020-06-03 10:07:09 +02:00
Maxime Chretien
5626c39546 Add support for old Linux versions
This add support for linux 0.01 to 2.6.12-rc1

Some versions are missing like 0.02 to 0.10 (not included) but it looks like
they have been deleted over time.

Signed-off-by: Maxime Chretien <maxime.chretien@bootlin.com>
2020-06-02 13:46:14 +02:00
Chris White
5e7cab18ca Added/updated copyright notices; whitespace fixes
No functional changes
2020-05-23 15:14:21 -04:00
Maxime Chretien
fbe9fa0e72 tests: Add support for file families
And fix a problem in a test where an ident where wrongly identified by
ctags. It is fixed in the new ctags version.

Signed-off-by: Maxime Chretien <maxime.chretien@bootlin.com>
2020-05-22 10:29:24 +02:00
llccd
a03d0abdc3
Fix a typo in pack-repositories 2020-04-27 17:01:33 +08:00
Maxime Chretien
7e6f88c26e Add database performance test script
This solves issue https://github.com/bootlin/elixir/issues/109

This script can be used with every project and with any number of
versions, files or identifiers.

Signed-off-by: Maxime Chretien <maxime.chretien@bootlin.com>
2020-04-23 13:27:23 +02:00
Maxime Chretien
f25d1b3205 Add support for the Toybox project
Signed-off-by: Maxime Chretien <maxime.chretien@bootlin.com>
2020-04-21 10:25:19 +02:00
Chris White
f112ab0826 Regularize whitespace; add editorconfig
- Change the few files that use leading tabs into leading spaces.
  This way they match the other files in the repo.
- Add an EditorConfig file so that supported editors will automatically
  use spaces instead of tabs.
- (Extra) Add some common editor backup-file extensions to .gitignore.
2020-03-21 08:58:43 -04:00
Michael Opdenacker
11ed7b7684 utils/update-elixir-data: add "--tags" to "git fetch"
- Not having this was causing us to miss tags

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
2019-11-25 14:31:38 +01:00
Michael Opdenacker
287667d2a6 update-elixir-data: fix directory test
- "-d" wasn't working with symbolic links
  which we are using on our server to switch
  databases after updates

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
2019-11-07 14:51:31 +01:00
Michael Opdenacker
13738d8dca index-all-repositories: fix llvm project name
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
2019-10-13 10:06:40 +02:00
Michael Opdenacker
d55498ef77 index-all-repositories: add second pass
- Useful for big repositories that take a lot of time to
  index and are updated frequently

- Good to run this on the indexing machine (faster)
  than through an update on the server (slower in our case)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
2019-10-13 08:59:20 +02:00
Michael Opdenacker
285b6f5a3c Make update-elixir-data more generic
- Now supports a custom data directory
  (passed as a command line parameter)

- Now supports a custom elixir install
  director (passed through the ELIXIR_INSTALL
  environment variable

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
2019-10-11 10:17:42 +02:00
Michael Opdenacker
8b632abd68 index-all-repositories: fetch the "other" branch too
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
2019-10-11 09:54:58 +02:00
Michael Opdenacker
b205cda6db index-all-repositories: index all projects in parallel
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
2019-10-08 05:28:23 +02:00
Michael Opdenacker
54befeb96b Script to re-index all repositories
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
2019-10-02 08:21:46 +02:00
Michael Opdenacker
188b0eb1fc New script to pack repositories
- For maintenance use, run it when you have gc.log files

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
2019-09-30 13:42:37 +02:00
Michael Opdenacker
3b97c3a73b Share script to keep databases up to date
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
2019-09-30 10:28:19 +02:00