New contains/icontains Query Backend
This release introduces a new method of gathering results from icontains and contains queries. The previous implementation suffered the following drawbacks:
- If a field was indexed for a contains-style query, the possible length of value of the field was limited. Saving a value which was too long would result in an error.
- The index data for such fields could become very large, this would make entities "heavy" and would result in performance issues across the site as this index data would be transferred with the returned instances.
- Index data was saved across several fields (due to a misunderstanding of the limits of list properties).
The new backend has none of these drawbacks. The limit for string value is the same as CharField (1500 bytes) and index data is stored on a descendent entity, meaning data isn't needlessly transferred with instances.
If you have been using contains/icontains fields on versions of Djangae before 0.9.10 you will need to migrate your data. One potential method for this is:
djangaeidx.yamland make a note of the models which have contains or icontains indexes.
- Deploy a version of your application to a non-default version (do not migrate traffic!). Make sure this
version uses Djangae 0.9.10 and has
- Open a shell using the remote sandbox (
./manage.py --sandbox=remote shell).
- Import each model in turn and use the
defer_iterationto resave your instances on the new non-default version: e.g.
defer_iteration(MyModel.objects.all(), MyModel.save, _target="your-new-app-version").
- Note that what you are doing here is using the remote shell to defer tasks onto the task queue of your production application, not to localhost.
- Test the version works as expected before migrating traffic to it.
Be aware that although this will add new index data, the original index data will still linger which will affect performance. You can use Djangae's migration support to write a migration file to clear this data. An example migration is:
from djangae.db.backends.appengine.indexing import ( LegacyContainsIndexer, LegacyIContainsIndexer, CHARACTERS_PER_COLUMN, MAX_COLUMNS_PER_SPECIAL_INDEX, ) from djangae.db.migrations.operations import RemoveFieldData from djangae.fields import CharField from django.db import migrations class Migration(migrations.Migration): """ Populates the Person model with the default value for the new `favourite_colour` field. """ dependencies = [ # Add dependent migration here (at least `('app_label', '0001_initial')`) ] # Set this to the model name of your model MODEL_NAME = "YourModel" # Set this to the name (or column name, if different) of the indexed field on your model FIELD_COLUMN = "your_field_column_here" # The legacy `contains` indexing supported indexing of up to 103 characters per column, but if # you know that your data doesn't contain any values this long then you can reduce this, which # may reduce the number of operations needed to delete your old index data MAX_LENGTH_OF_INDEXED_VALUES = 103 # Change this to LegacyIContainsIndexer if appropriate INDEXER = LegacyContainsIndexer() # END OF THINGS THAT YOU NEED TO EDIT # The legacy indexer(s) use multiple columns to split the indexed values across, so we need to # create some example values to get the correct column names for deletion values_for_lengths = [ "a" * x for i, x in enumerate(CHARACTERS_PER_COLUMN) if i < MAX_COLUMNS_PER_SPECIAL_INDEX ] index_column_names = [ INDEXER.indexed_column_name(FIELD_COLUMN, value, None) for value in values_for_lengths ] operations = [ RemoveFieldData( MODEL_NAME, index_column_name, CharField() ) for index_column_name in index_column_names ]
You will need one of these migrations for each
icontains entry in djangaeidx.yaml. Make sure you at least
have an initial migration for the app (which can be generated using the
Remember: You should not run a migration like this until you have resaved all your contains-using models and are successfully
DJANGAE_USE_LEGACY_CONTAINS_LOGIC = False
For more info on migrations, see the migrations documentation.