Commit Graph
55 Commits
Author SHA1 Message Date
Cédric Krier b8294ceaf1 Clear cache just after starting to listen
When the thread is restarted, we do not know if we missed any payload. So we
must clear the cache and the pool to have a clean start.

issue11397
review431631003
2022-09-25 20:23:13 +02:00
Nicolas Évrard e46d5dbfc6 Catch get_connection errors in Cache listener
issue11563
review411271003
2022-09-17 19:00:48 +02:00
Cédric Krier 889539d5cc Use default selectors instead of select
issue11268
review364881002
2022-03-20 01:40:28 +01:00
Cédric Krier 57de8723dd Import only once datetime for cache
issue11245
review370871002
2022-03-07 00:32:40 +01:00
Cédric Krier 36d5bce9bc Avoid concurrency failure in cache with channel test
We need to ensure that the listener is running on the return of the transaction
start otherwise notification can be missed.
We must also just from time to time check that the listener is still the
current listener.
And finally we can not wait an arbitrary time for cache sync, instead we must
rely on the _clean_last being reset with the sync happens.

issue9413
review364831002
2022-02-14 23:12:00 +01:00
Cédric Krier 632deb43e5 Use dictionary as domain for Reference field
issue11074
review383911002
2022-02-03 23:05:47 +01:00
Cédric Krier a0bfbfa255 Follow isort formatting
issue10928
2021-12-11 17:58:45 +01:00
Cédric Krier 0e41c6940a Factorize the instantiation of LRUDict and LRUDictTransaction
We use default_factory to avoid create unnecessary instances. The
default_factory_with_key argument is added to LRUDict to pass the missing key
to the default_factory method. And we use an LRUDict also for the
Transaction.cache.

issue10726
review362531002
2021-09-26 22:55:52 +02:00
Cédric Krier 2de16cf0f4 Do not reset cache on commit and clear
When a transaction is committed indeed a new transaction is started. So we do
not need to use a transaction cache just because a commit was sent because the
started_at is reset to a higher time.
This same logic applies to a rollback.

issue10528
review365521003
2021-07-12 23:14:32 +02:00
Cédric Krier 04b66919f7 Set field context when validate domain
issue10386
review336171002
2021-06-06 09:27:28 +02:00
Cédric Krier 0b1041b4c5 Remove _check_access and _skip_warnings keys from cache's context
issue10377
review351911002
2021-05-16 17:52:30 +02:00
Cédric Krier 7a11f76cec Remove _request key from cache's context
issue10376
review351901002
2021-05-05 23:14:54 +02:00
Cédric Krier 646ba70965 Pop cache keys from a copy of the context
Otherwise it modifies the context for the all transaction.

issue10380
review358061003
2021-05-05 23:12:49 +02:00
Cédric Krier fb9b7a5ee8 Add statistics to Cache
issue10100
review335611003
2021-04-03 23:43:40 +02:00
Cédric Krier 9c81cb6a4a Replace cursor fetchall by iteration over cursor
issue5817
review335831003
2021-03-21 16:09:13 +01:00
Cédric Krier 3d632a8c8d Remove client key from the cache's context
issue10099
review331891002
2021-02-25 00:05:43 +01:00
Cédric Krier c587d74dc4 Refresh pool of other processes
When a process change the list of activated modules, the others must refresh
their pool. We do that by just stopping the pool for that database, the next
request will load the new pool.

issue8988
review302861002
2021-01-20 00:52:07 +01:00
Cédric Krier 421ec76c49 Clear all cache after module loaded
As we rollback the cache modification when loading a module, we must clear all
of them at the end to ensure to not keep outdated data.

issue9759
review339421002
2021-01-13 00:26:53 +01:00
Cédric Krier 18fc13fb15 Use autocommit connection to LISTEN
This is needed to receive notification in a timely manner.

issue9966
review333541002
2021-01-03 13:00:26 +01:00
Cédric Krier 7822844f52 Store listener threads per process
This ensure that if the process is forked, the child process will start new
listeners for the Cache and the Bus.

issue9832
review329601002
2020-11-27 22:19:56 +01:00
Cédric Krier b3f8d125f9 Use custom class to store record data
By using a class with slots instead of standard dict, we can reduce a lot the
memory usage to store the record data in the caches and values.
Also we can rely on the data in the cache to exist only for real field which
can be used to create a fast path in ModelStorage.__getattr__.
As now storing data in the cache has no extra cost, we can store data from the
transactional cache in the local. This allow also to have a shorter path in
ModelStorage.__getattr__.
For more memory efficiency, we remove the record from the cache instead of
clearing them. It was better to clear a dict but a record is less expensive to
re-instantiate than clearing.

issue9369
issue9225
review297861002
2020-06-13 09:48:03 +02:00
Cédric Krier cd97362e08 Limit size of cache to reset per notify call
issue8781
review309371003
2020-04-19 16:37:51 +02:00
Cédric Krier a99931c5c4 Do use translation cache if records were modified after last synchronisation
On distributed setup, the cache is not synchronized instantaneously so we can
not use the cache if the record was modified by another process since the last
synchronisation.

issue9013
review262841002
2020-02-15 01:36:48 +01:00
Nicolas Évrard 8fb13634ea Use direct access to backend classes
issue8836
review258421002
2019-12-06 17:33:47 +01:00
Cédric Krier 7dbcc96b8e Use separate connection to query cache table
The table is a hot spot for contention so we must minimize the duration lock
are kept on it. As at the start of a transaction, the cache may be sync and so
it locks all the row in the table, the transaction used for this query should
not be the main one that lasts a long time. Idem for the commit.

issue8447
review283591002
2019-07-08 21:24:33 +02:00
Cédric Krier 4b4c57a362 Add duration on Cache
issue8249
review283181002
2019-04-11 18:35:25 +02:00
Cédric Krier cbd72f3003 Clear all reset cache in the Cache commit loop
The correct indentation was lost in review68431002 after an update to tip.

issue8027
2019-03-10 19:37:03 +01:00
Cédric Krier e5e92ac81a Allow to use channel to synchronize Cache
When the clear timeout of the cache is zero, we launch a thread to synchronize
caches if the back-end support channels.
The thread is started on the first call to sync for the database.
The channel is also used to clear the cache of the process initiating the
clearing. This allows to test the mechanism is working with a single process.
So there will be a small delay even on single instance.

issue8027
review68431002
2019-03-09 17:03:38 +01:00
Cédric Krier c4ddbb064a Prevent filling cache with data from old transaction
Once the cache has been cleared, we must ensure that transaction started
before the clearing cannot fill the global cache.

issue8019
review56541002
2019-03-02 23:11:14 +01:00
Cédric Krier 9eaac0f3a3 Cast timestamp as datetime for sqlite back-end
SQLite backend returns a string for the datetime so the comparison between
timestamp fails.

issue8080
review48721002
2019-02-13 10:23:44 +01:00
Cédric Krier 4c7441ec74 Store timestamp on Cache per database
The timestamp defines when the cache of a database has been cleared.

issue8018
review50711002
2019-01-30 10:05:55 +01:00
Cédric Krier 572372d3c8 Raise NotImplementedError instead of NotImplemented
issue8022
review49731002
2019-01-30 10:00:49 +01:00
Cédric Krier da24071467 Make Cache transactional
We implement sync, commit and rollback to follow transaction semantic.
When a transaction clears a cache, the global cache is cleared only when the
transaction is committed meanwhile the transaction will use a cache for itself
only.
We remove _nocache option because the sync, commit and rollback operate only
for the transaction.

issue7975
review53521002
2019-01-18 11:57:11 +01:00
Cédric Krier 78b2811fff Prevent Cache to start too much new transactions
resets starts a transaction only if there is something to store.
clean is run at best every 5 minutes by default.

issue7533
review64261002
2018-07-10 17:03:49 +02:00
Cédric Krier d2a461390a Remove support for Python 2.7
issue7497
review53261002
2018-06-26 19:11:19 +02:00
Cédric Krier d813ac2090 Add limit to SQL queries consumed with a single fetchone call
This remove some load on the database server.
This is only needed for queries that may have more than one rows.

issue7040
review43001003
2018-01-09 18:31:00 +01:00
Cédric Krier cdc5952145 Manage Cache in Transaction
issue6216
review33021002
2017-02-02 12:04:27 +01:00
Sergi Almacellas Abellana 0de3ae0a51 Allow to override cache implementation
issue5962
review33491003
2016-10-26 15:55:05 +02:00
Cédric Krier 512808e151 Add LRUDictTransaction
issue5293
review22771002
2016-03-03 23:27:38 +01:00
Nicolas Évrard 2fef479575 Follow PEP-0249 for Transaction and backends
issue4949
review19781002
2016-03-03 21:51:57 +01:00
Cédric Krier 7459aab3d1 Use CURRENT_TIMESTAMP instead of NOW
CURRENT_TIMESTAMP is the correct SQL standard way.

issue4837
review15531002
2015-07-01 19:41:49 +02:00
Cédric Krier 34fb7d7471 Drop all cache of a database when it is dropped
issue4584
review11121002
2015-03-25 18:56:52 +01:00
Cédric Krier c144f8c349 Fix most flake8 errors
review8771002
2014-12-02 12:54:01 +01:00
Cédric Krier e0b995abbe Refactore configuration to allow easy extention
issue3627
review7401002
2014-09-03 23:32:01 +02:00
Cédric Krier bc6acd08ee Drop support of Python 2.6
review2691002
2014-01-31 00:02:03 +01:00
Nicolas Évrard 3215eab818 Cache.set returns the value in order to prevent using temporary variable
review543002
2013-09-16 12:29:41 +02:00
Cédric Krier fc6c6cfdda Migration to python-sql
* Use python-sql
* Add convert_domain method on Fields
* Add sql_format and sql_type methods on Fields
* Allow to return a full domain for Function.searcher
* Replace static backend by dynamic get method
* Replace order_field by order_<field name> method

review1001002
2013-09-06 22:32:37 +02:00
Cédric Krier 1e60085854 Use a better hashing method for context
repr doesn't guarantee to have always the same string

review642002
2013-02-04 19:59:05 +01:00
Cédric Krier d6b453a516 Fix Cache to use the local thread context 2012-12-11 22:57:04 +01:00
Cédric Krier 0fe1b93864 Active Record 2012-09-24 22:16:53 +02:00