portugalfuturista/elixir
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Make Query.query('exist') lookup faster by adding a filename cache

Browse source 
Add a cache for `exists` queries. Currently, `exists` calls `git
ls-tree` and parses the result to check if a file exists. A single
call takes around 20-30 ms.

It only gets used by Makefile filters. Large Makefiles cause a filter to
make hundreds of these calls, causing filter processing to take
seconds.

Statistics on 20 HTTP requests on /linux/v6.11.6/source/MAINTAINERS:

                 without:     with:
    avg            1160        843
    median          951        790
    75th perc      1289        861
    95th perc      2452       1078
    max            2874       1749

The cache is stored inside Query, of which there is one instance per
request. We do not risk cache invalidation issues.

About memory usage: on Linux v6.9.4, the cache is 12MB.
This commit is contained in:
Franciszek Stachura 2024-08-12 20:07:23 +02:00 committed by Théo Lebrun
parent 81a1116013
commit 7df3543b2a
1 changed files with 12 additions and 11 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

23
elixir/query.py
View file

@ -61,6 +61,7 @@ class Query:
self.data_dir = data_dir
self.dts_comp_support = int(self.script('dts-comp'))
self.db = data.DB(data_dir, readonly=True, dtscomp=self.dts_comp_support)
self.file_cache = {}
def script(self, *args):
return script(*args, env=self.getEnv())
@ -136,22 +137,22 @@ class Query:
return decode(self.script('get-type', version, path)).strip()
elif cmd == 'exist':
# Returns True if the requested file exists, otherwise returns False
version = args[0]
path = args[1]
dirname, filename = os.path.split(path)
if version not in self.file_cache:
version_cache = set()
last_dir = None
for _, path in self.db.vers.get(version).iter():
dirname, filename = os.path.split(path)
if dirname != last_dir:
last_dir = dirname
version_cache.add(dirname)
version_cache.add(path)
entries = decode(self.script('get-dir', version, dirname)).split("\n")[:-1]
for entry in entries:
fname = entry.split(" ")[1]
self.file_cache[version] = version_cache
if fname == filename:
return True
return False
return path.strip('/') in self.file_cache[version]
elif cmd == 'dir':